run all scenarios in CI tests

[zeroXbrock]

Is your feature request related to a problem? Please describe.

If there's ever a breaking change to the TOML spec, we have to update all the builtin scenarios. Running them in CI will make sure we don't miss any, and will also alert us if we don't realize we've made a breaking change.

Describe the solution you'd like

Add a rust test that loads the scenario configs from the files in ./scenarios/ and runs setup and a short spam run on a spawned anvil instance.

Describe alternatives you've considered

I considered writing a shell script that would invoke the CLI, but that would be kind of muddy, because now we'd be testing both the CLI and the scenario files. This test only needs to consider whether the scenario files are valid, both syntactically and functionally.

[0xriazaka]

can i work on that?

[arpitkarnatak]

I have a question about your solution @zeroXbrock

Since we only need to check if the toml files are valid syntactically (Compatible with TOML specs) and functionally (Compatible with Contender's scenarios), shouldn't we just check for these rather than spawning an instance of anvil and running these scenario files as you suggested, since running them will introduce 2 additional possible points of failure (some errors with Anvil or some misconfiguration in scenarios (Not the file but the env vars, something like starting with a balance less than min balance needed) could also fail the tests even if nothing's wrong with the scenario file itself

I guess a better solution could be two functions, like is_valid_toml(&testfile) -> bool and is_valid_contender_scenario(&testfile) -> bool could be better, and be faster than running all contender spam commands, let me know what you think

[zeroXbrock]

I guess a better solution could be two functions, like is_valid_toml(&testfile) -> bool and is_valid_contender_scenario(&testfile) -> bool could be better, and be faster than running all contender spam commands, let me know what you think

@arpitkarnatak I agree, we only really need to parse the files to make sure they're syntactically correct. Are you interested in working on this?

[zeroXbrock]

hey @0xriazaka @arpitkarnatak do you one of you want to take this?

[NehharShah]

@zeroXbrock Can I take this one?

[zeroXbrock]

@zeroXbrock Can I take this one?

all yours, thanks!

[Peponks9]

while looking for ways to contribute i saw this one and looks like it might be closed now, right?

[zeroXbrock]

while looking for ways to contribute i saw this one and looks like it might be closed now, right?

Welcome @Peponks9, thanks for flagging!
@NehharShah implemented a solution to test the syntax of every file that works great, but I'd still like to have some tests that run the scenarios against a node. We've had issues in the past where a scenario change might be syntactically correct, but introduces a bug that only manifests when the scenario is executed onchain. It should also help us avoid merging bugs introduced by any upstream components.

@Peponks9 Would you be interested in working on this?