Advanced Mode Prompt Editing

[lenaxia]

Description:
Allow advanced users to modify the core prompts to fit their needs more appropriately.

Ask:

• A page which allows the user to modify the core prompts with templating support
• A button to reset the prompt to default should the user need to.
• Bonus: Prompts are configurable via file or env var to allow for better gitops deployment (e.g. k8s)

[david1542]

Thanks for creating the issue, @lenaxia.

Can you elaborate a bit more about what would you hope to achieve by manipulating the prompts? Do you wish to "steer" the LLM during investigation/chat mode?

[lenaxia]

Yes, I have found projects that were promising but the built in prompts were less than great, or did not work well with the LLM backend I was using.

MemGPT is an example where they have a great idea, but their prompts are insanely long and cause some LLMs to lose context or eat up all the available context (e.g. on Llama3).

Also as prompting techniques evolve, its nice to be able to update without having to wait for the maintainer to update.

[david1542]

@lenaxia thanks for the clarification.

Do you have a specific flow in mind? I'm trying to understand how you view the "steering" ability, and make sure we'll build the right thing.

Would you want to specifically guide the LLM to specific places via prompt editing by editing the system prompt? Or would you like to build custom workflows for each incident/alerts?

We're thinking about creating a workflows feature where you can guide the agent explicitly (for each incident) by creating a workflow, and then you can edit the prompt in each step phase, kinda like n8n. We think that can make the tool extremely flexible.

I'm trying to understand how much control would you want over the investigation process :)

[lenaxia]

Sorry my house flooded so havent really been able to try anything.

The workflow feature would be great. If you're planning on doing that sooner rather than later then I would definily be intersted, especially if you support function calling.

In the issue I describe above, in its most simplest form, a single page with enables edits for all the prompts merlinn uses would be sufficient. it sounds like it only has a system prompt right now, but I imagine that shouldn't be the case long term.

[david1542]

@lenaxia So sorry to hear about the flood 😕 I hope it's okay now.
About what you said, it's definitely something we're thinking of doing. Thanks for clarifications.

I hope this isn't too much to ask, but could we schedule a 20-minute call? I'd really appreciate the chance to delve deeper into the directions we've discussed and hear more about the use case. If you're down, feel free to pick a slot 😊

[lenaxia]

Done.