Transcoder crash (cannot transcode captions)

[eliteprox]

Describe the bug
A remote transcoder v0.8.8 running on the same system as orchestrator crashed and stopped receiving jobs from orchestrator

Expected behavior
Transcoder should keep running, perhaps reject the stream as invalid or other reasonable solution

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

See transcoder logs below

I1110 14:37:50.681192       1 ot_rpc.go:144] Transcoding taskId=7092779 url=https://178.63.87.148:8933/stream/2c856fdb/3948.tempfile
I1110 14:37:52.586910       1 ot_rpc.go:144] Transcoding taskId=7092780 url=https://178.63.87.148:8933/stream/4f9c8fff/10074.tempfile
[Parsed_fps_1 @ 0x71359c647880] cc_fifo cannot transcode captions fps=25/1
I1110 14:37:52.826454       1 ot_rpc.go:144] Transcoding taskId=7092781 url=https://178.63.87.148:8933/stream/2c856fdb/3949.tempfile
I1110 14:37:53.490514       1 ot_rpc.go:144] Transcoding taskId=7092782 url=https://178.63.87.148:8933/stream/4f9c8fff/10076.tempfile
I1110 14:37:54.738120       1 ot_rpc.go:144] Transcoding taskId=7092783 url=https://178.63.87.148:8933/stream/2c856fdb/3950.tempfile
I1110 14:37:56.840112       1 ot_rpc.go:144] Transcoding taskId=7092785 url=https://178.63.87.148:8933/stream/2c856fdb/3951.tempfile
I1110 14:37:58.777467       1 ot_rpc.go:144] Transcoding taskId=7092787 url=https://178.63.87.148:8933/stream/2c856fdb/3952.tempfile
I1110 14:38:00.731903       1 ot_rpc.go:144] Transcoding taskId=7092788 url=https://178.63.87.148:8933/stream/2c856fdb/3953.tempfile
I1110 14:38:01.491448       1 ot_rpc.go:125] End of stream receive cycle because of err="EOF", waiting for running transcode jobs to complete

GPU metrics show the encoder at 50% utilization. nvidia dcgm exporter reports both encoder and decoder at 50%

root@Ubuntu-2204-jammy-amd64-base ~ # nvidia-smi \
  --query-gpu=timestamp,name,index,uuid,pci.bus_id,driver_version,\
utilization.gpu,utilization.memory,memory.total,memory.used,memory.free,\
temperature.gpu,fan.speed,power.draw,power.limit,clocks.gr,clocks.sm,clocks.mem \
  --format=csv,noheader,nounits
2025/11/14 12:28:28.129, NVIDIA RTX 4000 SFF Ada Generation, 0, GPU-e36bc6be-3b37-08e7-e392-e90e29bfa784, 00000000:01:00.0, 550.54.15, 1, 0, 20475, 307, 19838, 47, 74, 34.97, 70.00, 2325, 2325, 6800

root@Ubuntu-2204-jammy-amd64-base ~ # nvidia-smi dmon
# gpu    pwr  gtemp  mtemp     sm    mem    enc    dec    jpg    ofa   mclk   pclk
# Idx      W      C      C      %      %      %      %      %      %    MHz    MHz
    0     35     47      -      1      0     50      0      0      0   6800   2325

[j0sh]

@eliteprox Are you sure that they are related? The logs you pasted show around 8 seconds and several segments between that log line (which seems like a warning) and the EOF. If so, this should be fairly easy to reproduce - can you do that?