Skip to content

Conversation

@alco
Copy link
Member

@alco alco commented Dec 12, 2025

Starting on December 10, the Test TS packages against sync-service (packages/typescript-client) CI job started hanging until cancelled by GH Actions after 6 hours of runtime.

image

This seems to be related to the fact that pnpm coverage and pnpm test run in interactive mode by default. Passing the --run command-line flag to those prevents this behaviour and makes sure the process exits after it has finished running the tests.

EDIT: found the actual culprit, it's not the way we run pnpm coverage but a certain test that got added in a recent PR causes the test runner to hang. The CI job on that PR is the first known occurrence of the "CI job stuck for 6 hours" problem.

I have reproduced it locally by running pnpm exec vitest --coverage "--run". The test runner was stuck running test/stream.test.ts for some minutes, then produced the following output and exited with exit code 1:

<--- Last few GCs --->

[166905:0x28ad6000]   201497 ms: Mark-Compact 4077.8 (4104.3) -> 4066.1 (4104.3) MB, pooled: 0 MB, 1656.56 / 0.00 ms  (average mu = 0.238, current mu = 0.184) allocation failure; scavenge might not succeed
[166905:0x28ad6000]   203588 ms: Mark-Compact 4074.2 (4104.8) -> 4066.1 (4100.8) MB, pooled: 4 MB, 2076.70 / 0.00 ms  (average mu = 0.120, current mu = 0.007) allocation failure; scavenge might not succeed

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x72be1c node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/home/alco/.asdf/installs/nodejs/24.11.1/bin/node]
 2: 0xb9dc10  [/home/alco/.asdf/installs/nodejs/24.11.1/bin/node]
 3: 0xb9dcff  [/home/alco/.asdf/installs/nodejs/24.11.1/bin/node]
 4: 0xe367e5  [/home/alco/.asdf/installs/nodejs/24.11.1/bin/node]
 5: 0xe36812  [/home/alco/.asdf/installs/nodejs/24.11.1/bin/node]
 6: 0xe36b0a  [/home/alco/.asdf/installs/nodejs/24.11.1/bin/node]
 7: 0xe4702a  [/home/alco/.asdf/installs/nodejs/24.11.1/bin/node]
 8: 0xe4b3d0  [/home/alco/.asdf/installs/nodejs/24.11.1/bin/node]
 9: 0x18de281  [/home/alco/.asdf/installs/nodejs/24.11.1/bin/node]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Vitest caught 1 unhandled error during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Error ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Error: [vitest-pool]: Worker forks emitted error.
 ❯ EventEmitter.<anonymous> ../../node_modules/.pnpm/[email protected]_@[email protected][email protected][email protected][email protected][email protected][email protected][email protected]/node_modules/vitest/dist/chunks/cli-api.C7sYjHmQ.js:8010:22
 ❯ EventEmitter.emit node:events:508:28
 ❯ ChildProcess.emitUnexpectedExit ../../node_modules/.pnpm/[email protected]_@[email protected][email protected][email protected][email protected][email protected][email protected][email protected]/node_modules/vitest/dist/chunks/cli-api.C7sYjHmQ.js:7583:22
 ❯ ChildProcess.emit node:events:508:28
 ❯ Process.ChildProcess._handle.onexit node:internal/child_process:294:12

Caused by: Error: Worker exited unexpectedly
 ❯ ChildProcess.emitUnexpectedExit ../../node_modules/.pnpm/[email protected]_@[email protected][email protected][email protected][email protected][email protected][email protected][email protected]/node_modules/vitest/dist/chunks/cli-api.C7sYjHmQ.js:7582:33
 ❯ ChildProcess.emit node:events:508:28
 ❯ Process.ChildProcess._handle.onexit node:internal/child_process:294:12

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯


 Test Files  13 passed (14)
      Tests  275 passed (283)
Type Errors  no errors
     Errors  1 error
   Start at  14:25:51
   Duration  261.75s (transform 696ms, setup 56ms, import 1.05s, tests 34.06s, environment 2.99s, typecheck 599ms)

JUNIT report written to /home/alco/code/electric-sql/electric/packages/typescript-client/junit/test-report.junit.xml
 % Coverage report from istanbul
-------------------------|---------|----------|---------|---------|------------------------------------------------------------------------------------------------------------------------------------------
File                     | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                                                                        
-------------------------|---------|----------|---------|---------|------------------------------------------------------------------------------------------------------------------------------------------
All files                |   92.76 |     86.2 |   89.61 |   92.99 |                                                                                                                                          
 client.ts               |   90.23 |    80.37 |   87.03 |   90.93 | 598-611,718-723,869,880,885,917,1052,1265,1286-1287,1316,1322,1342,1382-1383,1417,1459,1526,1572,1596,1599,1608,1623,1627-1628,1639-1641 
 column-mapper.ts        |   98.27 |    97.05 |     100 |   98.21 | 304                                                                                                                                      
 constants.ts            |     100 |      100 |     100 |     100 |                                                                                                                                          
 error.ts                |   74.35 |    83.33 |   58.33 |   74.35 | 57-81,103-104                                                                                                                            
 expired-shapes-cache.ts |   96.77 |    91.66 |     100 |   96.55 | 57                                                                                                                                       
 fetch.ts                |   96.79 |    91.39 |   96.29 |   96.55 | 123,149-150,182,334                                                                                                                      
 helpers.ts              |   91.66 |    85.71 |     100 |   91.66 | 71                                                                                                                                       
 parser.ts               |    98.8 |     87.3 |     100 |   98.73 | 153                                                                                                                                      
 shape.ts                |   91.34 |    91.17 |   82.75 |      92 | 78,108,123-128,269,282-284                                                                                                               
 snapshot-tracker.ts     |     100 |      100 |     100 |     100 |                                                                                                                                          
 up-to-date-tracker.ts   |   93.33 |    95.83 |      90 |    93.1 | 70-72,138                                                                                                                                
-------------------------|---------|----------|---------|---------|------------------------------------------------------------------------------------------------------------------------------------------

@codecov
Copy link

codecov bot commented Dec 12, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.65%. Comparing base (7f24b70) to head (e82672e).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3593   +/-   ##
=======================================
  Coverage   75.65%   75.65%           
=======================================
  Files          51       51           
  Lines        2752     2752           
  Branches      411      407    -4     
=======================================
  Hits         2082     2082           
  Misses        668      668           
  Partials        2        2           
Flag Coverage Δ
electric-telemetry 22.71% <ø> (ø)
elixir 57.38% <ø> (ø)
elixir-client 73.94% <ø> (ø)
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/typescript-client 93.80% <ø> (ø)
packages/y-electric 55.66% <ø> (ø)
typescript 88.13% <ø> (ø)
unit-tests 75.65% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alco alco force-pushed the alco/ci-pnpm-run branch 2 times, most recently from 1ae7b88 to 6dd0ada Compare December 12, 2025 13:36
@alco
Copy link
Member Author

alco commented Dec 12, 2025

I can still reproduce the problematic test locally after reverting the commit that upgraded Node to v24. So the problem seems to be in the test logic.

@alco alco changed the title Avoid running vitest in interactive/watch mode on CI ci: Avoid running vitest in interactive/watch mode on CI Dec 13, 2025
Copy link
Contributor

@msfstef msfstef left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

alco added a commit that referenced this pull request Dec 15, 2025
@alco
Copy link
Member Author

alco commented Dec 15, 2025

To summarize, this PR does not address the problem I thought was caused by stuck vitest process. That issue has been fixed in #3602.

Vitest by default checks for the presence of the CI env variable, which GitHub Actions set for jobs, and doesn't use the watch mode when it is set. However, for the sake of being explicit, I'm merging this change anyway.

@alco alco merged commit 9e932d2 into main Dec 15, 2025
44 checks passed
@alco alco deleted the alco/ci-pnpm-run branch December 15, 2025 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants