Skip to content

Conversation

@JulienLavocat
Copy link
Contributor

Description of Changes

Add Clerk tutorial

API and ABI breaking changes

None.

Expected complexity level and risk

1

Testing

Tested locally

@JulienLavocat JulienLavocat marked this pull request as draft January 19, 2026 16:55
@JulienLavocat
Copy link
Contributor Author

/update-llm-benchmark

@clockwork-labs-bot
Copy link
Collaborator

LLM Benchmark Results (ci-quickfix)

Language Mode Category Tests Passed Task Pass %
Rust rustdoc_json basics 26/27 91.7% ⬆️ +8.3%
Rust rustdoc_json schema 29/34 80.0%
Rust rustdoc_json total 55/61 86.4% ⬆️ +4.5%
C# docs basics 11/27 59.0% ⬇️ -2.1%
C# docs schema 10/34 31.5%
C# docs total 21/61 46.5% ⬇️ -1.1%

Compared against master branch baseline

Generated at: 2026-01-20T12:13:28.082Z

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

  • Total failures analyzed: 22

Analysis

Analysis of SpacetimeDB Benchmark Failures

Rust Failures

1. Root Causes

  • Compile/Publish Errors:

    • The errors publish_error: spacetime publish failed (exit=1) indicate issues related to improperly defined packages or dependencies during the publish phase.
    • Example rows show missing accessibility qualifiers like pub for struct fields which can cause issues.
  • Other Failures:

    • Issues like no such table for profiles, drawings, and accounts indicate missing definitions in the Rust test setup.
    • Warnings about "UNSTABLE" commands suggest code that may be relying on deprecated or unsupported features.

2. Recommendations

Specific Documentation Changes:

  • Update fields in Rust structs for visibility:

    • Section: Rust Struct Definitions
    • Change: Add pub visibility modifier for struct fields:
      pub struct Address {
          pub street: String,
          pub zip: i32,
      }
  • Provide clear setup instructions for test database:

    • Section: Test Setup
    • Change: Include examples of how to create required tables in the setup phase.
    ctx.db.create_table("profiles");
  • Clarify dependency versioning practices:

    • Section: Dependency Management
    • Change: Add guidance for locking versions to avoid issues when publishing via cargo.
    For stable builds, ensure to lock dependency versions in your `Cargo.toml`.

3. Priority

  1. Table visibility in Rust structs - High impact; affects struct accessibility across tests.
  2. Clear test database setup instructions - High impact; avoids common "no such table" errors.
  3. Dependency management guidance - Medium impact; helps stabilize the project during publish processes.

C# Failures

1. Root Causes

  • The majority of C# failures show no such table errors, suggesting that the test environment is not properly initializing the required tables.
  • The publish_error: spacetime build failed (exit=1) indicates compilation issues with improper annotations or methods.

2. Recommendations

Specific Documentation Changes:

  • Update table annotations for C# classes:

    • Section: C# Struct/Classes Definitions
    • Change: Ensure all tables have the correct attribute specifications:
    [Table(Name = "User", Public = true)]
    public partial struct User { ... }
  • Clarify initialization for the test setup:

    • Section: Initialization of Test Data
    • Change: Include specific setup steps to create necessary tables before tests run:
    public static void InitializeDatabase(ReducerContext ctx) {
        ctx.Db.User.Insert(new User{ ... });
    }
  • Ensure method annotations are clear:

    • Section: Reducer Methods
    • Change: Include notes on required attributes for methods to function correctly in different test setups.
    Methods defined as reducers must include specific annotations to be correctly parsed.

3. Priority

  1. Update table and method annotations - High impact; fixes multiple failure points immediately.
  2. Add clear initialization for database tests - High impact; prevents repeated "no such table" issues.
  3. Add method annotation guidelines - Medium impact; helps clarify binding for reducer functions.

By addressing these documentation root causes in high-priority areas, the robustness and reliability of benchmark tests in both Rust and C# can be significantly improved, leading to more reliable development and testing practices.

@JulienLavocat JulienLavocat marked this pull request as ready for review January 20, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants