Add ephemeral tests to waspc/e2e-tests

[FranjoMindek]

Description

What are "ephemeral" tests?
Before this PR waspc/e2e-tests only had the snapshot tests. They function in a way where we want to keep their artifacts (snapshots) as part of the git tracked files. Ephemeral tests on the other hand are just normal tests where we don't want to keep any artifacts after running them. So think of them as normal tests, we just use this name to keep them distinct from the snapshot tests.

Before going to deep into the solution be aware that we are aiming to refactor the whole waspc/e2e-tests by moving them away from the current unix shell commands approach. Instead we will write all of our logic in pure haskell. This is to make them cross platform compatible. That is why the tests which would require complex unix shell command implementations were only partially implemented. As the unix shell commands logic will be dropped, they are not worth the time investment.
More on cross platform compatible e2e tests here: #3404

In short, we want a net gain positive from these tests, not that they cover everything. Standard good code quality rules still apply.

The CLI commands table, which the implementation was based on can be found here: https://www.notion.so/wasp-lang/CLI-Testing-24518a74854c8051b0fef88533592504?source=copy_link
In there we mention which commands we are testing, which we are not, and why.

Note, as we now have additional variant of tests besides the snapshot tests, I decided to refactor the structure so that the snapshot tests logic is moved under the waspc/e2e-tests/SnapshotTest folder.
Changes related to moving snapshot tests folder are in a separate PR: #3468

Ephemeral tests consist of a name (which creates a same-named directory under EphemeralTest/temp) and ephemeral test cases. Each ephemeral test case is a unix shell command. Ephemeral test cases are executed sequentially. This is different from snapshot tests which only ever have a single unix shell command. This approach makes it much easier to build the unix shell commands (since we can split them into steps - test cases), and also we can validate the individual test cases separately. The validity of the test case is done by checking its exit status.

Sequentiality is enforced by executing the unix shell commands while creating the Hspec's Spec. Only for the captured exit status to be translated into SUCCESS/FAIL after running all of the tests. This is because I couldn't force Tasty to run them in sequence even while using the sequential function.

Blocked by #3480 which enables us to use the wasp db reset command non-interactively. This PR assumes the above PR is merged.

Type of change

• 🔧 Just code/docs improvement
• 🐞 Bug fix
• 🚀 New/improved feature
• 💥 Breaking change

[Martinsos]

Cool! I won't get into proper reviewing if others have time because I don't have enough time do to it properly, but I would potentially recommend @cprecioso as a reviewer, he is new to this piece of code and will both benefit from learning about it + offering a fresh perspective.

My only quick feedback is, do we have to call them "ephemeral"? Can't they just be named "tests", and "snapshot" tests are then kind of a specialization? Ephemeral sounds like there is sometihng unusually ephemeral happening, but then one realized "oh ok they are just normal tests" hah, so would be nice to avoid that "misdirection". I guess you maybe did so because you hvae Tests and then you have SnapshotTests and you needed to also somehow name these guys, but maybe that is then design issue (or maybe not).

Sequential part -> I hope when you said they have to execute sequentially, you meant a single test calls its content sequentially, not that all of them have to wait for each other? If actually all tests have to execute sequentially, that kind of sucks, doesn't it? Would be good to understand better in that case if we can not hvae that limitation.

Ok this is my quick take, I hope someobdy else can take this one over.

[cprecioso]

Hey, didn't take a look yet, just a heads-up that I won't be able to fit this review before I go on vacation. I can do it when I come back.

[FranjoMindek]

My only quick feedback is, do we have to call them "ephemeral"? Can't they just be named "tests", and "snapshot" tests are then kind of a specialization? Ephemeral sounds like there is sometihng unusually ephemeral happening, but then one realized "oh ok they are just normal tests" hah, so would be nice to avoid that "misdirection". I guess you maybe did so because you hvae Tests and then you have SnapshotTests and you needed to also somehow name these guys, but maybe that is then design issue (or maybe not).

Yes. It's simply giving them a name. I could name them just Tests instead. Though that makes me feel like it's something which should also affect snapshot tests, but it does not. It's a separate variant.

Sequential part -> I hope when you said they have to execute sequentially, you meant a single test calls its content sequentially, not that all of them have to wait for each other? If actually all tests have to execute sequentially, that kind of sucks, doesn't it? Would be good to understand better in that case if we can not hvae that limitation.

Sadly they are fully sequential because of Sequentiality is enforced by executing the unix shell commands while creating the Hspec's Spec. Because spec creation is not done in parallel, their execution ends up sequential. I only ever managed to do everything in parallel, or do everything sequentially, didn't manage to run the ephemeral test in parallel where their cases are run in sequence. Even when I use Hspec's sequential execution, they start the next shell command before the previous one was finished.

Do note that currently even snapshot tests are executed sequentially (the project generation part, which is also the most expensive part time wise, as it's done sequentially before running the TestTree).

So we do have a lot to gain by optimizing both of them.

Hey, didn't take a look yet, just a heads-up that I won't be able to fit this review before I go on vacation. I can do it when I come back.

That is fine, I still have to fix the issue that interactive tests (e.g. wasp db reset or wasp new) don't work on github runners.

[cprecioso]

Maybe "Regression tests" is what we want?

[Martinsos]

I don't think "regression tests" term is used in this way. I would just call them Tests then and that is ok, they are just normal, general tests.