Fixes C# benchmark test failures caused by table naming convention mismatches

[cloutiertyler]

Description of Changes

Adds TypeScript as a third language for LLM benchmark tests alongside Rust and C#, and fixes table naming convention mismatches.

TypeScript Support:

• Added Lang::TypeScript variant with camelCase naming conventions
• Created TypeScript project template (templates/typescript/server/) with package.json, tsconfig.json, and index.ts
• Added TypeScript publisher that uses spacetime build and spacetime publish
• Created 22 TypeScript task prompts and golden answer files for all benchmark tests
• Updated prompt discovery to find tasks/typescript.txt files

Table Naming Fix:

• Standardized on singular table names across all languages:
  • Rust: user (snake_case singular)
  • C#: User (PascalCase singular)
  • TypeScript: user (camelCase singular)
• Updated table_name() helper to convert singular names to appropriate case per language
• Updated all spec.rs files to use table_name("user", lang) instead of hardcoded "users"

CI/Hashing Improvements:

• Added compute_processed_context_hash() for language-specific hash computation after tab filtering
• Updated CI check to verify both rustdoc_json and docs modes for Rust
• Fixed --hash-only mode to skip golden builds

API and ABI breaking changes

None - these are internal benchmark tooling changes only.

Expected complexity level and risk

Complexity: 2

The changes add a new language following existing patterns for Rust and C#. The table naming fixes are straightforward find-and-replace style updates. Low risk since this only affects the benchmark tooling, not the core SpacetimeDB codebase.

Testing

• cargo build -p xtask-llm-benchmark compiles successfully
• All 22 TypeScript golden modules build and publish successfully
• Rust and C# benchmarks unaffected by changes

[cloutiertyler]

/update-llm-benchmark

[cloutiertyler]

/update-llm-benchmark

[cloutiertyler]

/update-llm-benchmark

[cloutiertyler]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	9/27	36.1% ⬇️ -47.2%
Rust	rustdoc_json	schema	9/34	26.5% ⬇️ -53.5%
Rust	rustdoc_json	total	18/61	31.7% ⬇️ -50.1%
C#	docs	basics	12/27	61.1%
C#	docs	schema	10/34	31.5%
C#	docs	total	22/61	47.7%

Compared against master branch baseline

_{Generated at: 2026-01-20T01:46:16.880Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 59

Analysis

Analysis of SpacetimeDB Benchmark Test Failures: Rust and C#

Rust Failures

1. Root Causes

• Inconsistent Table Names: Many failures arise from incorrect or inconsistent table names used in the code segments compared to what is expected in the benchmarks. Tables such as users, drawings, event, etc., are referenced with incorrect names leading to errors.
• Lack of pub Keyword: Many struct fields are missing the pub keyword, causing issues with accessing these fields outside the module.
• Unstable API Warnings: Numerous tests are failing due to reliance on unstable methods or API changes.
• Missing Error Handling: Functions that should return Result types do not, leading to issues when error handling is assumed.

2. Recommendations

• Update Table Names for Consistency:
  • In the table definitions and usages, ensure names are consistent throughout the documentation. For instance:
    • event should replace all instances of events.
    • primitive should replace all instances of primitives.
    • drawing should replace all instances of drawings.
• Add pub Keyword for Structs and Fields:
  • Documentation should specify that structs and their fields must be public for access.
• Document API Stability:
  • Clearly mark all APIs that are unstable and subject to change. Provide a dedicated section for upcoming breaking changes, if possible.
• Error Handling:
  • Example code should consistently include error handling to return results or handle errors gracefully. This should be highlighted in the documentation.

3. Priority

1. Fix Table Name Consistencies: This will directly resolve numerous failures and prevent potential confusion.
2. Add pub Keyword Requirement: Ensuring access to fields would significantly improve usability and reduce errors in testing.
3. Document API Stability: Prevent future issues arising from unexpected API changes.

---

C# Failures

1. Root Causes

• Table Name Consistency: Like Rust, several tests in C# fail due to improper table names, particularly for users, results, and accounts.
• Lack of public Modifiers: Many structs and fields are missing the public modifier, which can restrict access from the context in which they are used.
• API Instability Documentation: References to unstable API methods are rampant, causing uncertainty in method usage and expectations.

2. Recommendations

• Align Table Names:
  • Replace all instances of Users with User, both in definitions and references.
  • Standardize other table names like Results and Accounts similarly.
• Include public Modifiers:
  • The documentation should specify the need for public qualifiers on structs and fields explicitly, with examples demonstrating proper usage.
• Mark Unstable APIs:
  • Document all APIs that are unstable and document their expected changes. Provide clear notes in the sections where these APIs are discussed, especially around construction methods and reducer configurations.

3. Priority

1. Align Table Names: This will immediately tackle a significant number of test failures.
2. Public Modifier Guidelines: Enhancing understanding of access levels will greatly improve code quality and tests.
3. Unstable API Documentation: This is a long-term fix but is critical for preventing confusion around API usage in future versions.

---

These actionable insights, with specific attention to documentation updates, can help mitigate the current benchmark failures and improve user experience in both Rust and C# implementations of the SpacetimeDB.

[cloutiertyler]

/update-llm-benchmark

[cloutiertyler]

I am going to have to force merge this PR. The reason is that this PR changes how the context for the LLMs is calculated, in particular it now takes a mode (e.g. rust_doc vs docs) instead of just language as a way to compute the context. However, the current CI check uses the ci-check defined on the PR, whereas /update-llm-benchmark checks out master. Since the mechanism for generating the hash (/update-llm-benchmark) and the CI check (cargo llm ci-check) are on different branches, they do not match.

I have fixed this oversight in this PR, but since github runs the workflow from master only, there's no way for me to have it be fixed until this merges.

[cloutiertyler]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	9/27	36.1% ⬇️ -31.2%
Rust	rustdoc_json	schema	9/34	26.5% ⬇️ -63.5%
Rust	rustdoc_json	total	18/61	31.7% ⬇️ -45.9%
C#	docs	basics	10/24	56.3% ⬇️ -2.1%
C#	docs	schema	9/34	29.5% ⬇️ -2.0%
C#	docs	total	19/58	44.1% ⬇️ -2.0%

Compared against master branch baseline

_{Generated at: 2026-01-20T17:48:37.135Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 59

Analysis

SpacetimeDB Benchmark Failure Analysis

Rust

1. Root Causes

A. Compile/Publish Errors

• Limited visibility of pub visibility on struct fields and functions leads to errors in the code. Many structs/functions lack pub, causing access errors.

B. Timeout Issues

• Timeouts likely stem from inefficient queries in reducers or missing table structures.

C. Other Failures

• Schema Parity Issues: Many tests fail due to discrepancies between expected schema and implementation. Functions and fields lack pub visibility, leading to failures in accessing them.
• Data Parity Issues: Many tests indicate 'no such table' errors, indicating that certain tables are not correctly defined or initialized before tests are run.

2. Recommendations

A. Update Visibility in Documentation

1. Structs and Reducers
   • Update all struct fields and functions in documentation examples to include pub where applicable.
   • Example:

     #[table(name = user)]
     pub struct User {
         #[primary_key]
         pub id: i32,
         pub name: String,
         pub age: i32,
         pub active: bool,
     }

B. Initialization Documentation

2. Initialization of Tables
   • Documentation should clarify how to initialize tables before running tests. Include examples and note the importance of seeding the database.
   • Sections mentioning the initialization of tables should be explicit on calling the insert reducer before data access.

C. Schema and Data Parity

3. Schema Example Updates
   • Ensure that all schema examples are verified against common tests like CRUD operations, ensuring no missing fields or incorrect types.

3. Priority

• High priority to update the visibility of fields in structs, especially for those involved in common CRUD operations, as this will directly affect numerous tests.
• Secondly, ensure documentation includes a guide for initializing tables and seeding data.

---

C#

1. Root Causes

A. Timeout Issues

• Similar to Rust, timeouts most likely arise from inefficient queries or missing table structures before tests run.

B. Other Failures

• Tables not being accessible during the test indicate that examples may lack clarity on initialization or seeding of tables.

C. Publish Errors

• Errors during publishing point to missing configurations or misnamed tables.

2. Recommendations

A. Update Example Structures

1. Visibility Issues
   • Documentation must ensure that all struct fields and methods are marked with correct visibility (e.g., public).
   • Example:

     [Table(Name = "User")]
     public partial struct User
     {
         [PrimaryKey] public int Id;
         public string Name;
         public int Age;
         public bool Active;
     }

B. Initialization and Seed Documentation

2. Clarifying Table Initialization
   • Include detailed steps for initializing and seeding tables, prefacing that these steps are necessary before running tests.

C. Publishing and Configuration Examples

3. Ensuring Correct Naming and Configurations
   • Add a section explicitly mentioning the need for correct table names and indexing in documentation to avoid confusion during publishing.

3. Priority

• High priority to adjust visibility in struct definitions within examples, as this significantly impacts many tests.
• Secondly, improve clarity around table initialization and necessary configurations to mitigate publish errors.

---

Summary

Each language has similar root causes primarily related to visibility and initialization issues with structs and tables. The suggestions focus on adjusting the documentation to enhance clarity, provide better examples, and ensure correct struct accessibility, ultimately improving the robustness of the SpacetimeDB benchmarks and reducing common failure points.

[cloutiertyler]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	10/27	44.4% ⬇️ -22.9%
Rust	rustdoc_json	schema	8/34	24.5% ⬇️ -65.5%
Rust	rustdoc_json	total	18/61	35.4% ⬇️ -42.3%
C#	docs	basics	11/25	58.3%
C#	docs	schema	9/34	29.5% ⬇️ -2.0%
C#	docs	total	20/59	45.2% ⬇️ -0.9%

Compared against master branch baseline

_{Generated at: 2026-01-20T18:26:47.651Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 58

Analysis

Analysis of SpacetimeDB Benchmark Failures

Rust Failures (41 total)

1. Root Causes:

• Incomplete Code Samples: Many Rust tests failed due to missing pub modifiers for struct fields, which are necessary for the SpacetimeDB framework to access them.
• Inconsistent Method Names: Missing or inconsistent naming conventions for reducers and table methods (e.g., schedule vs. scheduled).
• Lack of Type Annotations: Several structs and fields lack type annotations, resulting in compile-time errors. For example, fields in structs should be marked pub.
• Incorrect Handling of Result Types: Functions that don't return results correctly when operations might fail, leading to runtime failure scenarios.
• Database Initialization Issues: Errors related to missing database schemas may stem from tests not running in the correct environment or configurations.

2. Recommendations:

• Documentation File Updates:

  1. Field Access in Structs: Update the section discussing struct definitions. Ensure all field definitions include the pub keyword, especially in examples.

     Example Change:

     pub struct User {
         #[primary_key]
         pub id: i32,
         pub name: String,
         pub age: i32,
         pub active: bool,
     }

  2. Correct Naming Conventions: Review the naming conventions for scheduled reducers. Ensure that all naming matches the documentation's defined specifications (e.g., scheduled vs. schedule).

  3. Result Handling: Add examples demonstrating how to properly handle results and error cases in reducer functions. Update existing reducer examples to show return types.

  4. Database Initialization: Provide a checklist or specific examples for initializing databases within tests. This includes correct environment variables, schemas, and config files.

3. Priority:

• High Priority: Update field access documentation (point 1) as it directly affects many tests and involves core language features.

• Medium Priority: Consistency in naming conventions (point 2) should be addressed to avoid confusion and ensure coherent implementation across different tests.

• Medium Priority: Add examples for result handling (point 3) to prevent runtime issues that might not be immediately obvious during development.

---

C# Failures (17 total)

1. Root Causes:

• Missing Field Modifiers: Similar to Rust, many struct fields in tests lack the public access modifier, leading to accessibility issues.
• Incorrect Attribute Usage: Misapplication of attributes like Public, PrimaryKey, and Table. These are critical for the Spacetime framework.
• Database Table Naming: Some tests refer to tables that do not exist or are incorrectly named, leading to SQL failures.
• Schema Issues: Many errors can be traced back to missing database schemas or tables that need to be defined for tests to pass.

2. Recommendations:

• Documentation File Updates:

  1. Field Accessibility: Ensure all struct fields in user-facing documentation include proper access modifiers like public.
     Example Change:

     [Table(Name = "User")]
     public partial struct User
     {
         [PrimaryKey] public int Id;
         public string Name;
         public int Age;
         public bool Active;
     }

  2. Correct Attribute Examples: Provide correct usage of C# attributes in struct examples to prevent misapplication. Clarify where and how to use them in examples.

  3. Consistency in Table Names: Ensure all documentation regarding table names matches the actual implementations used in test scenarios. This should include a clear mapping of expected names vs actual names.

  4. Database Initialization Examples: Include examples in the documentation for initializing databases correctly within tests.

3. Priority:

• High Priority: Update field accessibility documentation (point 1) immediately as it is fundamental to struct definitions.

• Medium Priority: Correct attribute usage guidance (point 2) should be detailed to prevent common misconfiguration mistakes in C# struct setups.

• Medium/Low Priority: Schema naming consistency (point 3) and database initialization examples (point 4) are important but can be somewhat less urgent than accessibility and attributes.

By addressing these root issues and recommendations, we can enhance the usability of SpacetimeDB across both Rust and C#, thereby improving the success rate of benchmark tests and reducing developer frustration.

[jdetter]

I reviewed all of the documentation changes - I didn't review the benchmark code + answer changes as I trust you've double-checked those.

All optional nits - let me know if you need another review 🙂

[cloutiertyler]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	10/27	44.4% ⬇️ -22.9%
Rust	rustdoc_json	schema	9/34	26.5% ⬇️ -63.5%
Rust	rustdoc_json	total	19/61	36.3% ⬇️ -41.4%
C#	docs	basics	12/27	61.1% ⬆️ +2.8%
C#	docs	schema	10/34	31.5%
C#	docs	total	22/61	47.7% ⬆️ +1.5%

Compared against master branch baseline

_{Generated at: 2026-01-21T01:34:57.629Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 58

Analysis

Analysis of SpacetimeDB Benchmark Failures

This analysis breaks down the failures encountered in the Rust and C# benchmarks of SpacetimeDB, highlighting the root causes, recommended documentation updates, and priorities.

Rust Failures

1. Root Causes

• Inconsistent Usage of Access Modifiers: Many structs lack pub on fields, which is needed for them to be accessible from outside their defining module, leading to compile errors.
• Missing Error Handling: Functions do not return Result<(), String>, which is necessary for indicating success or failure of operations.
• Incorrect Import Statements: Missing necessary imports, like for ScheduleAt or Duration, which are crucial for scheduled tables.
• Inconsistencies in Naming Conventions: Some expected types or names do not match the code, e.g., scheduled_timer vs tick_timer.

2. Recommendations

• Modify Struct Field Access Modifiers

  • Update documentation to emphasize the need for pub modifiers on struct fields.
  • Example Change: In the documentation for SpacetimeDB, clearly state:

    "Ensure to declare struct fields as pub for accessibility in reducers and outside modules."

• Add Error Handling Guidance

  • Include examples that demonstrate returning Result<(), String> in reducers.
  • Example Change: Add an entry under "Error Handling" in the documentation:

    "All reducer functions should return Result<(), String> to indicate the success or failure of the operation."

• Standardize Example Names and Imports

  • Ensure all example codes use consistent naming and include the necessary imports.
  • Example Change: Under the section for scheduled tables, provide a clear listing of required imports with an example:

    use spacetimedb::{ReducerContext, ScheduleAt, Table, Duration};

• Clarify Scheduling Mechanisms

  • Enhance explanations regarding how to set up scheduled tables correctly, including namespace requirements.
  • Example Change: In the scheduling section:

    "When defining a scheduled table, always ensure to utilize the full namespace for special types such as ScheduleAt."

3. Priority

1. Modify Struct Field Access Modifiers: This is critical as it causes fundamental compilation errors.
2. Add Error Handling Guidance: Important for improving reliability and user experience with the API.
3. Standardize Example Names and Imports: Lowers the barrier to consistent usage across examples.

---

C# Failures

1. Root Causes

• Inconsistent Usage of Access Modifiers: Similar to Rust, many properties lack the access modifiers, causing access issues externally.
• Missing Error Handling: Lack of robust return types for the reducer methods.
• Non-standardized Naming Conventions: Differences in names used for tables and classes cause confusion (e.g., "User" vs "user").
• Wrong Segmentation of Attributes: Attributes like [SpacetimeDB.Table] need to follow specific patterns which are sometimes inconsistent.

2. Recommendations

• Modify Access Modifiers

  • Emphasize the need to declare properties as public.
  • Example Change: Update the C# module documentation to clarify:

    "Ensure to declare the properties of the structs as public to allow proper access throughout your application."

• Implement robust Error Handling Examples

  • Illustrate the required return types of reducer methods clearly.
  • Example Change: Include in the reducer section:

    "Always define your reducer methods to signal success or failure effectively, returning void for successful execution."

• Consistent Naming and Attribute Usage

  • Provide consistent naming practices for tables and properties used often.
  • Example Change: Update the documentation to include a convention section:

    "Use PascalCase for struct and variable names consistently across your code."

• Clarify Use of Attributes in Classes

  • Guide users consistently on how to apply attributes correctly.
  • Example Change: Provide an example section on struct definition:

    [Table(Name = "User", Public = true)]
    public struct User
    {
        [PrimaryKey] public int Id;
        public string Name;
    }

3. Priority

1. Modify Access Modifiers: This will resolve the most frequent compilation issues at the core of struct definitions.
2. Consistent Naming and Attribute Usage: Establishing a standard will significantly reduce confusion among developers.
3. Implement Robust Error Handling Examples: Helps in building a more user-friendly API.

---

By implementing these specific documentation changes, SpacetimeDB can improve both its usability and reliability, addressing the issues highlighted in the benchmarks for both Rust and C#.

[cloutiertyler]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	10/27	44.4% ⬇️ -29.9%
Rust	rustdoc_json	schema	9/34	26.5% ⬇️ -53.5%
Rust	rustdoc_json	total	19/61	36.3% ⬇️ -40.6%
C#	docs	basics	12/27	61.1%
C#	docs	schema	10/34	31.5%
C#	docs	total	22/61	47.7%

Compared against master branch baseline

_{Generated at: 2026-01-21T03:05:32.938Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 58

Analysis

Analysis of SpacetimeDB Benchmark Test Failures

Rust Failures

Root Causes

1. Incomplete Visibility of Struct Fields

   • Many struct fields for tables (e.g., User) are not marked as pub, causing accessibility issues.

2. Inconsistent Usage of Table Names

   • There are discrepancies in the naming conventions between the code and the expected database tables. For example, using users instead of user, and results instead of result.

3. Missing Result Types on Reducer Functions

   • A significant number of reducer functions lack return types or proper error handling, leading to compilation and runtime errors.

4. Incorrect Scheduling Parameters in Scheduled Tables

   • The scheduling system is not consistently defined in the tests (e.g., ScheduleAt parameter management), leading to inconsistencies in functional expectations.

Recommendations

1. Make Struct Fields Public

   • Update all structs associated with SpacetimeDB tables to ensure fields are public (e.g., change id: i32 to pub id: i32).
   • Documentation Change: Update the sample code documentation sections that define table structures.

2. Standardize Naming Conventions

   • Ensure naming conventions for structs and database tables are aligned.
   • Documentation Change: Revise naming conventions section to specify standard naming practices clearly.

3. Include Result Types in Reducers

   • Add recommended return types for all reducer functions (e.g., return Result<(), String> instead of ()).
   • Documentation Change: Update reducer function examples in documentation to include this information.

4. Clarify Scheduling Table Configuration

   • Provide explicit instructions for scheduling parameters using accurate examples.
   • Documentation Change: Include a dedicated section on Scheduling with working code examples.

Priority

1. Make Struct Fields Public – This is the most critical fix as it directly impacts accessibility and usability.
2. Include Result Types in Reducers – This ensures proper error handling, which would eliminate many runtime issues.
3. Standardize Naming Conventions – Clear naming helps maintain consistency across multiple codebases.
4. Clarify Scheduling Table Configuration – Enhances clarity on functionality, but lesser impact compared to the first two.

---

C# Failures

Root Causes

1. Inconsistent Field Access Modifiers

   • Fields in structs for database tables are often not marked as public, leading to access issues.

2. Misalignment in Table Definitions

   • There are discrepancies between the expected structure of tables and provided definitions (e.g., different table names).

3. Reducer Function Formatting

   • Incomplete or incorrect formatting of reducer functions that may lead to improper execution.

4. Lack of Error Handling in Functions

   • Similar to Rust, many functions do not have meaningful return types or exceptions coded in for errors, which can lead to failures.

Recommendations

1. Ensure Fields are Public

   • Change all fields in database table structs to have public access.
   • Documentation Change: Update the database table definition examples to reflect this.

2. Standardize Naming in Table Definitions

   • Review and fix discrepancies in table definitions, specifying clear rules for naming and structuring.
   • Documentation Change: Provide clearer guidelines for naming conventions in structs.

3. Include Proper Formatting and Return Types for Reducers

   • Add return types to all reducer functions, following the expected pattern.
   • Documentation Change: Revise reducer function examples with complete signatures and return types.

4. Implement Exception Handling

   • Ensure all database interactions in reducers incorporate exception handling.
   • Documentation Change: Include a section on error handling in the reducers’ documentation.

Priority

1. Ensure Fields are Public – Ensuring accessibility across all struct fields is critical to prevent many failures.
2. Include Proper Formatting and Return Types for Reducers – Providing clear function signatures can greatly enhance functional reliability.
3. Standardize Naming in Table Definitions – Important for avoiding confusion and ensuring correctness.
4. Implement Exception Handling – Should be detailed but is less critical than the above issues since core access issues need to be prioritized.

---

Conclusion

Both languages suffer from structural and accessibility issues in their respective code samples, leading to a myriad of runtime and compilation problems. Prioritizing documentation fixes based on accessibility and naming conventions will significantly improve usability and reduce failures in benchmarks.