Agent not handling imageurl from response of an MCP server

[dennisnotojr]

I configured fast-agent to create a web search agent that uses an mcp google search server tool. The response from the mcp server looks like this

{
searchParameters:{
q:"CNN"
type:"news"
num:5
page:1
engine:"google"
}
news:
[
0:{
title:"Trump considers targeting cocaine facilities inside Venezuela, CNN reports"
link:"https://www.reuters.com/world/americas/trump-considers-targeting-cocaine-facilities-inside-venezuela..."
snippet:"U.S. President Donald Trump is considering plans to target facilities producing cocaine and routes u..."
date:"4 hours ago"
source:"Reuters"
imageUrl:"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBwgHBgkIBwgKCgkLDRYPDQwMDRsUFRAWIB0iIiAdH..."
position: 1
}}

When the response does not have an imageurl the agent returns the summarized information. when the response contains imageurls the agent freaks out and responds with

The provided data appears to be a JSON object representing an image. It contains information such as:

• id: A unique identifier for the image.
• name: The name of the image, which is empty in this case.
• description: An HTML-formatted description of the image.
• author: Information about the author of the image, including their ID and username.
• width and height: The dimensions of the image (1000 pixels wide and 600 pixels tall).
• categories: An array containing two categories the image belongs to: "Dogs" and "Nature".
• tags: An empty array suggesting that no additional tags are associated with this image.
• downloadsLink: A URL link related to downloads for this image, which is also empty in this case.
• views: The number of times the image has been viewed (1500 views).
• imageLinks: Empty URLs where one might expect direct links to different sizes or variations of the original image.
• previewImageLink: A URL pointing directly to a preview version/photo sample associated with this entry; in your instance appears as base64-encoded data
providing immediate display without requiring an external request for download completion verification - typical during API responses involving
multimedia content handling efficiently ensuring quick retrieval alongside accompanying details rather than delaying until separate file requests
completion.
• credits: The amount of credits required to access or use/download the image, which is 1 credit in this instance.

This data structure seems standard for APIs that manage media content like images on stock photo platforms where various attributes about an asset provide
context necessary both programmatically through code interaction and visually within user interfaces designed around multimedia consumption experiences.

For a test, outside fast-agent, i fed the llm just the json and asked to summerize and the llm did fine.

Don't know what the agent is doing to causes the wierd answer?

[evalstate]

From the data supplied here, it looks like the image itselfhas been Base64 encoded and embedded in the URL, which will normally be a large amount of tokens. It's truncated in what's been pasted here, but I expect it's a lot of characters.

Ideally the MCP Server in question would return that in an ImageContent block so that the models vision capabilities could be used for it. If it's in a TextContent that will choke most LLMs (and if the copied/pasted version is the truncated snippet it will potentially give very different results).

If you can share more about the message (a save_history attached will help) or the MCP Server so I can run it and see that would be 👍

[dennisnotojr]

The mcp server can be found at https://github.com/garylab/serper-mcp-server. You will need to get a free api key at https://serper.dev/. The google_search_news tool returns imageurls with in the json return array which causes the problem. To get the agent to select the tool, use example, "what is the latest world news?"

The below file has the log file records on the return of the mcp server tool call.
message.txt

[evalstate]

I think this issue is more for the serper-mcp-server repo than fast-agent.

https://github.com/garylab/serper-mcp-server/blob/main/src/serper_mcp_server/server.py

The MCP Server is not doing any additional post-processing of the results for LLM Presentation, and is a minimal API wrapper.

I would suggest raising an issue in their repo to either strip ImageUrl content in the case it is Base64 content, or present it as the appropriate MCP type.