WebRTC h264 video Latency

[ronPartuk]

I’m running a C++ WebRTC server and a minimal JS client on the same machine (127.0.0.1), sending H264 video from camera (or file) to the browser.

What Happens
The server always displays each frame before encoding. After encoding, the frame is sent immediately.
The client video in the browser is always exactly 16 frames behind the server.
This lag does not depend on FPS, or resolution—it’s always 16 frames.
If I pause the server stream, the client video immediately freezes—it does not drain any buffered frames.

Server-Side Timings
Encode time: I use FFmpeg (H264, zerolatency, no B-frames), encoding and pre-send processing take 8–14ms per frame (measured).

Send path: As soon as the frame is encoded, it’s pushed to the WebRTC track—there is no extra buffering before the network.

Server Code (frame send snippet)

Here’s how I set up the media track, encoder, and packetizer chain:

const uint8_t h264PayloadType = 96;
const uint32_t h264Ssrc = ssrc;
const std::string cname = "video-stream";
const std::string msid = "stream1";

// 1. Create media description first
rtc::Description::Video media(cname, rtc::Description::Direction::SendOnly);
media.addH264Codec(h264PayloadType);
media.addSSRC(ssrc, cname, msid, cname);

// 2. Add track to peer connection FIRST
track = peerConnection->addTrack(media);

// 3. Now create the packetizer chain
auto rtpConfig = std::make_shared<rtc::RtpPacketizationConfig>(
    h264Ssrc, cname, h264PayloadType, rtc::H264RtpPacketizer::ClockRate);
auto packetizer = std::make_shared<rtc::H264RtpPacketizer>(
    rtc::H264RtpPacketizer::Separator::LongStartSequence, rtpConfig);
auto srReporter = std::make_shared<rtc::RtcpSrReporter>(rtpConfig);
packetizer->addToChain(srReporter);
auto nackResponder = std::make_shared<rtc::RtcpNackResponder>();
packetizer->addToChain(nackResponder);

// 4. Now set the media handler (track is valid now)
track->setMediaHandler(packetizer);

And here’s the minimal frame send code (after encoding with FFmpeg):

    for (rtc::Track* track : currentTracks) {
        if (track && track->isOpen()) {
            rtc::binary sample(
                reinterpret_cast<std::byte*>(pkt->data),
                reinterpret_cast<std::byte*>(pkt->data + pkt->size)
            );
            auto now_us = std::chrono::duration_cast<std::chrono::microseconds>           (std::chrono::high_resolution_clock::now().time_since_epoch()).count();
            track->sendFrame(sample, now_us);
        }
    }

The frame is sent as soon as it’s ready. There is no artificial delay or buffering on the server.

Client Code

<video id="v" autoplay muted playsinline></video>

<script>
    const pc = new RTCPeerConnection({bundlePolicy:'max-bundle'});
    pc.ontrack = e => {
        v.srcObject = e.streams[0];
        v.play();
    };
    const ws = new WebSocket('ws://127.0.0.1:8000');
    ws.onmessage = async ev => {
      const m = JSON.parse(ev.data);
        if (m.type === 'offer') {
              await pc.setRemoteDescription(m);
              const ans = await pc.createAnswer();
              await pc.setLocalDescription(ans);
              ws.send(JSON.stringify({type: ans.type, sdp: ans.sdp}));
        }
    };
</script>

What I See
Encode and send: 8–14ms per frame (FFmpeg H264, zerolatency).

Network: Localhost, UDP, zero loss or jitter.

16-frame lag: Always present, even if I change FPS, resolution, or pause the server.

No draining on pause: Client video freezes immediately, content is always 16 frames behind.

Questions
Where is this 16-frame buffer coming from? Is it a WebRTC stack thing, the HTML video element, or something else?

Is there any way to reduce or bypass this buffer? I’ve tried all the “low latency” hints and tricks, nothing works.

Anyone else seeing this exact 16-frame lag? Is it baked into Chrome/Firefox/WebRTC?

More Than half a second is too much for real-time apps. Would really appreciate any insight, docs, or workarounds.

[paullouisageneau]

There is no buffer on media transport, with libdatachannel packets are actually sent synchronously. Half a second is a huge delay, it means something is wrong with the setup.

No draining on pause: Client video freezes immediately, content is always 16 frames behind.

This behavior hints at an issue on server side before sending. Given that the delay is constant, it sounds like the encoder is not properly setup for low latency. It could also be too slow, causing a buildup of frames to be encoded.

There could also be an issue with timestamping, as you should use the correct frame PTS instead of relying on wall clock time.