ramalama service for xdg-desktop-portal

[swick]

Hey all,

I was investigating a Portal for LLM/AI workloads. The idea here is that sandboxed applications can re-use the same models and runtimes while the user gains control over which apps get access to the computationally expensive workloads.

xdg-desktop-portal is a user service that apps talk with, which would then talk with ramalama on their behalf. The sandboxed applications share the same network namespace as the host. Ideally, we (xdg-desktop-portal) would have some kind of service to talk to, which is not exposed on the network namespace, which would mean something like dbus, varlink or REST over a UNIX socket.

The service should provide a way to create some sort of session for a specific client which can internally spawn a new container or exec something in a container that's already running.

The portal would also not expose detailed information about the available models and runtimes but rather tell the portal what kind of use-case the model should be optimized for. For example, a coding assistant would request a different model than a translation tool. This kind of selection is something we could probably build into the portal but maybe that's also interesting for ramalama?

Anyway, this is more of a brain-dump than a specific issue. Is there any interest in figuring out how we can make this work?

[ericcurtin]

Open-minded about this for sure :)

[rhatdan]

Moving this to a discussion.

[castrojo]

Is there any interest in figuring out how we can make this work?

Thanks for starting this discussion, I've been thinking about this too! I've been shopping around this idea at KubeCon in a more general sense, I am dead tired but just got done with an interesting discussion with folks so please bear with me. I'll be talking in concepts:

I think the desktop is a great place to prototype local AI, many of these desktop AI products end up being privacy nightmares or products people don't want. I think a good win for the free desktop here is to be developer first, provide a common AI API for app developers to build their apps.

Just like podman brings every server workload to local devs, this could be the AI extension of that. Here's some use cases I've been kicking the tyres on:

Users are using chatgpt and friends to help teach themselves linux. It's awful, I don't have time to share the amount of terrible advice we get in the Universal Blue discord from these LLMs. We use dosu.dev as an AI assistant for our github, and are currently experimenting with it's RAG capabilities. We instructed it to ingest our official documentation, upstream docs, community posts, discourse and discord posts, manpages etc. gave it some parameters and turned it on, here are two examples:

• How do I pin my current image to a specific date? ublue-os/bluefin#2253
• How do I get the list of runtimes for each flatpak? ublue-os/bluefin#2287

There are some bad examples in there too, but not bad! I am reasonably certain that with a bit of work we could have something that at a minimum would be better than the average help post on the internet. I was thinking if we had an API endpoint then tools like Newelle could talk to we could start to combo out some things. Then we could ship hyper-specific project documentation. I envision a "Get some help" icon on the panel that let's the user chat with their local LLM to figure out how to use their computer. Indexed on official docs, and known good content.

I have tons of files like "IMG_0000012.png" on my computer, wouldn't it be neat if the photo/file manager had a way to go through that stuff and organize it for me, etc. And the IDE local integration story would be so cool too.

Then it'd be up to the app authors to innovate, just have the OS provide the endpoint.

cc @hanthor @axelquack

[swick]

I wrote down the result of my research into an AI portal. This includes ramalama specifics I think we have to talk about to make this a reality:

flatpak/xdg-desktop-portal#1743

I'm sure there are some inaccuracies, so I'd also appreciate feedback if you spot anything.

A bunch of things are a bit hand-wavy, but I think two things are immediately actionable:

• We need a way to expose an API from the container that can be used by the portal and not by flatpak instances (i.e. not bound to the network namespace)
• For integrating model and runtime management into desktops, an API that's not exec and parse output would be useful (varlink maybe?)