Multiple messages with different message ID on vercel stream with toolcalls

[apm-tobias-duesterhus]

Hey there,

I am unsing Prism (v 0.95.0) in combination with the Vercel SDK 5 for frontend. Since I added tool calls, I found that in case of a tool call, I get multiple text-messages per request with different message IDs. This seems to force Vercel SDK to show multiple messages with the same content. This seems to be a known issue in Vercel that was solved in their backend.

Here is a short excerpt from the stream response (the first message seems to be empty):

data: {"type":"start","messageId":"evt_01KB1XH56DBFTBR3WRPD2TR3C7"}

data: {"type":"tool-input-available","toolCallId":"fc_0633e23d579e07a2006927e586cc0c81949d56bb001d29de09","toolName":"google_web_search","input":{"query_string":"..."}}

data: {"type":"finish","messageMetadata":{"finishReason":"tool-calls","usage":{"promptTokens":348,"completionTokens":18}}}

data: {"type":"tool-output-available","toolCallId":"fc_0633e23d579e07a2006927e586cc0c81949d56bb001d29de09","output":"{...}"}

data: {"type":"start","messageId":"evt_01KB1XH7YKJRMYBT91XRAC8HMV"}

data: {"type":"text-start","id":"evt_01KB1XH7YE6ZHYA5CVC2VXET56"}

Is there a fix for Prism that I did not found yet?

[apm-tobias-duesterhus]

I started debugging it a little more. I copied the code from the DataProtocolAdapter and adopted the handleStreamStart Method:

protected function handleStreamStart(StreamStartEvent $event): array
{
    return [
        'type' => 'start',
        'messageId' => $event->id,
    ];
}

private ?string $stream_id = null;

...

protected function handleStreamStart(StreamStartEvent $event): array
{
    if ($this->stream_id === null) {
        $this->stream_id = $event->id;
    }
    return [
        'type' => 'start',
        'messageId' => $this->stream_id,
    ];
}

This is not perfect at all, but fixes duplicate messages in the chat frontend managed by Vercel.

The Problem seems to be that a "StreamStartEvent" is triggered multiple times (for what ever reason) within the same stream

[apm-tobias-duesterhus]

I think, I found the reason for all that:

As I already pointed: The problem is, that the StreamStartEvent is triggered multiple times within a stream in case of Prism-side toolcalls.
The reason for that seems to be within the Stream-Handler classes of the Providers. The following example comes from the OpenAI provider (Prism\Prism\Providers\OpenAI\Handlers\Stream):

The method processStream is called on a new request to the provider. On the very start of this method, one can finde the following line of code:

$this->state->reset()->withMessageId(EventID::generate());

This resets the stream und causes sending a StreamStartEvent with a new Event-ID as its Identifier.
The method then processes the stream from the provider until the stream returns eof. After this processing, in case of a toolcall is required, the method $this->handleToolCalls($request, $depth); is called. This however, does:

yield from $this->processStream($nextResponse, $request, $depth);

Which restarts the stream by calling $this->state->reset(), which in turn triggers a StreamStartEvent in the following process again.

This problem also leads to multiple calls to the method onComplete to store the messages e.g. in a Database.

[apm-tobias-duesterhus]

One more thought about this:

I think this "bug" is not really a bug in Prism. From my point of view, multiple messages with different message IDs in one stream are totally valid and should be displayed in Vercel UI accordingly. Merging all messages from one stream into one single message seems valid on the other hand but breaks the intention of the message-ID and causes some issues in message rendering (what should separate two messages? A line-break, space?

On the other hand, besides the bug in Vercel-Streaming, there still seems to be an issue in the method onComplete: It is called on every response from the LLM with all messages collected during the whole operation. So to work around this, I had to create a buffer to avoid duplicates.

So all-in-all there are probably 2 bugs - more or less.