How to use prompt management with litellm's integration with langfuse?

[GildeshAbhay]

So for using prompt management, with openai we have this tutorial

https://langfuse.com/docs/prompts/get-started

in this, they use something called
"langfuse_prompt"
which doesn't work with litellm as we just have a successive_callback in litellm

I also tried proxy but the proxy thing doesn't work.
Is there some other solution?

[dosubot[bot]]

To integrate Langfuse's prompt management with LiteLLM, you can use the LiteLLM Python SDK to send logs to Langfuse by setting the appropriate environment variables. Here's an example of how you can configure this in your Python code:

import os
from litellm import completion

# Set environment variables
os.environ["LANGFUSE_PUBLIC_KEY"] = "<your_public_key>"
os.environ["LANGFUSE_SECRET_KEY"] = "<your_secret_key>"
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"  # or "https://us.cloud.langfuse.com" for US region
os.environ["OPENAI_API_KEY"] = "<your_openai_api_key>"
os.environ["COHERE_API_KEY"] = "<your_cohere_api_key>"

# Set callbacks
litellm.success_callback = ["langfuse"]
litellm.failure_callback = ["langfuse"]

# Example usage
response = completion(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "Hi 👋 - I'm openai"}
    ]
)

print(response)

Additionally, you can use the LiteLLM Proxy with the Langfuse OpenAI SDK Wrapper for a more comprehensive integration. This setup automatically captures token counts, latencies, streaming response times, API errors, and more. You can also configure the LiteLLM Proxy UI to send logs directly to Langfuse by setting the callback to Langfuse in the UI ^[1][2][3].

For more advanced usage, such as setting custom trace IDs, trace user IDs, and tags, you can use the metadata parameter in the completion function. This allows you to pass additional information to Langfuse, enhancing traceability and observability of your LLM interactions.

To continue talking to Dosu, mention @dosu.

[marcklingen]

To clarify, you want to link traces produces by the litellm python sdk to prompts within langfuse prompt management by passing the langfuse_prompt attribute?

[GildeshAbhay]

Yes. I want to track prompts and traces for litellm.
Openai one is easy. Litellm is giving issues

[marcklingen]

ok thanks for confirming. can you open an issue on the litellm repo for this? happy to contribute and later add this to the langfuse docs as well

[analyst3dev]

The LiteLLM docs does suggest there is a prompt metadata parameter that can be passed in, but it doesn't seem to do anything to link the generation to the prompts. https://litellm.vercel.app/docs/observability/langfuse_integration

Generation Specific Parameters
generation_id - Identifier for the generation, auto-generated by default
generation_name - Identifier for the generation, auto-generated by default
parent_observation_id - Identifier for the parent observation, defaults to None
prompt - Langfuse prompt object used for the generation, defaults to None

[hassiebp]

@GildeshAbhay @analyst3dev Could you please try the following with the latest version of the LiteLLM and Langfuse Python SDK?

from langfuse import Langfuse

langfuse_prompt = Langfuse().get_prompt('my_prompt')

response = completion(
  model="gpt-3.5-turbo",
  messages=langfuse_prompt.prompt,
  metadata={
      "prompt": langfuse_prompt
  },
)
 
print(response)

It looks like LiteLLM is looking in the metadata for the langfuse prompt: https://github.com/BerriAI/litellm/blob/134bd2cebb6b62c2dd5f314bdd6e8270b78ca332/litellm/integrations/langfuse/langfuse.py#L693

[JohanBekker]

Hey, I'm having the same problem where I cannot link the Langfuse prompt to a generation trace using LiteLLM. Following the url you've supplied I also found the following lines:
https://github.com/BerriAI/litellm/blob/main/litellm/integrations/langfuse/langfuse.py#L399
https://github.com/BerriAI/litellm/blob/main/litellm/integrations/langfuse/langfuse.py#L636

But it seems the latest version of Langfuse on pypi is 2.53.3

Seems like a new deployment to pypi is necessary. Would be great to get this to work!

[gambastyle]

im having the exact same problem,
im using both litellm and langchain chain, tried both ways, can't seem to link the generation to the prompt

[hassiebp]

@gambastyle Using Langchain is an additional complication. We have separate docs for linking prompts to Langchain runs (check out the Langchain tabs): https://langfuse.com/docs/prompts/get-started#link-with-langfuse-tracing-optional

[gambastyle]

Hi, Thank you for the reply,
Unfortunately, this method is also not working for me, not sure why.

prompt_data = self.langfuse.get_prompt(chain_name)

prompt = PromptTemplate.from_template(
template=prompt_data.get_langchain_prompt().strip(),
metadata={"langfuse_prompt": prompt_data}
)

            return prompt | self.llm | parser

[gambastyle]

Is there any requirement of the llm type or configuration? Does this have anything to do at all with the llm?
I am not defining any callback like when you use Langfuse. Can you explain how the prompt data is processed? I couldn't find the code for that processes this.

[tiagoefreitas]

I don't understand having to pass the prompt as a message and the prompt name as Metadata...what is the point of the prompt managent then?

…

On Tue, 29 Oct 2024 at 17:58, Hassieb Pakzad ***@***.***> wrote: @GildeshAbhay <https://github.com/GildeshAbhay> @analyst3dev <https://github.com/analyst3dev> Could you please try the following with the latest version of the LiteLLM and Langfuse Python SDK? from langfuse import Langfuse langfuse_prompt = Langfuse().get_prompt('my_prompt') response = completion( model="gpt-3.5-turbo", messages=langfuse_prompt.prompt, metadata={ "prompt": langfuse_prompt }, ) print(response) It looks like LiteLLM is looking in the metadata for the langfuse prompt: https://github.com/BerriAI/litellm/blob/134bd2cebb6b62c2dd5f314bdd6e8270b78ca332/litellm/integrations/langfuse/langfuse.py#L693 — Reply to this email directly, view it on GitHub <#3780 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXJNRXHOICOXZMYXSN6NT3Z57EEHAVCNFSM6AAAAABQB4XJ7WVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBZGA4DKMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[hassiebp]

The object shape of what LiteLLM accepts as a completion prompt (e.g. chat messages) differs to what Langfuse needs under the hood to link prompts.

Feel free to check out our prompt management docs for other benefits of prompt management. https://langfuse.com/docs/prompts/get-started

[JohanBekker]

As it seems this is a LiteLLM error, I've made a bug ticket there which should solve this problem. They do a version check on Langfuse > 2.73 but it seems they confused the version of the 'main' repo (already at 2.86) with the python repo(at 2.53.3).
https://github.com/BerriAI/litellm/blob/main/litellm/integrations/langfuse/langfuse.py#L399

BerriAI/litellm#6545

[hassiebp]

Great catch, thanks @JohanBekker !

[JohanBekker]

I was actually wrong, misread the versions. LiteLLM implementation expects a dict, not a ChatPromptClient. If you add ChatPromptClient.__dict__ instead, the prompt tracking works.

BerriAI/litellm#6545 (comment)

[logg926]

I am using Litellm Proxy, I've tried langefuse_prompt, prompt, langefusePrompt passing ChatPromptClient.dict in, but seems it is still not adding the trace to the prompt

[evris99]

Did you find a solution for this?

I am struggling to make liteLLM proxy pass the prompt to langfuse