ScreenCast: CreateVirtualMonitor and RemoveVirtualMonitor

[ywh233]

We are currently using Mutter's private APIs for Remote Desktop/ScreenCast. We would like to eventually switch to cross-DE APIs, so I'm making some proposal to close the feature gaps.

One thing we want to support is the ability to create and remove virtual monitors after the remote desktop session has started. Currently you can only create one virtual monitor by calling ScreenCast.SelectSources before calling RemoteDesktop.Start or ScreenCast.Start. We would like something like:

CreateVirtualMonitor (
  IN session_handle o,
  IN options a{sv},
  OUT handle o
)

With the options vardict being:

• cursor_mode (u): See ScreenCast:AvailableCursorModes
• I'd also like to have an optional name option in the vardict, which is a connector name in GNOME and an output name in KDE. This will allows us to, e.g., associate the PipeWire stream with the virtual monitor in GNOME's DisplayConfig API. I can live with an initial implementation without it though.

It should return the following result via the org.freedesktop.portal.Request::Response signal:

• stream: This is pretty much just a single entry of streams of ScreenCast.Start's result

And the corresponding RemoveVirtualMonitor method:

RemoveVirtualMonitor (
  IN session_handle o,
  IN options a{sv},
  OUT handle o
)

With the options vardict being:

• id: The opaque identifier in the stream tuple of CreateVirtualMonitor's response

We can also introduce a new DYNAMIC_VIRTUAL = 8 type to AvailableSourceTypes. If only this source type is specified, no virtual monitors or streams should be created when Start is called. This eliminates the one off oddity of the first virtual monitor. If DYNAMIC_VIRTUAL is not specified, CreateVirtualMonitor and RemoveVirtualMonitor should fail. The portal backend could implement this as a checkbox in the confirmation/display selection dialog, if no permission has previously been granted.

Given that xdg-desktop-portal-gnome uses the private Mutter API under the hood, and KDE developer claims kwin already supports creating arbitrary number of virtual monitors, it sounds like the proposed change should be trivial to implement.

[jadahl]

Is this about the use case covering a session that is primarily accessed remotely? If so, the portal API is not suitable, since its use case is about allowing a user to explicitly provide a way to share their screen.

[ywh233]

Is this about the use case covering a session that is primarily accessed remotely?

Yes.

This may not be the best suited API for unattended remote access, but it's the closest to a universal solution nevertheless. In #471 it looks like you are not against the idea of pre-authorizing the remote desktop/screencast portal. If that ever gets implemented, then I'm sure developers will try to use it for unattended remote access.

I am aware that you are drafting a new remote desktop spec, which will work very well with our use case, but I'm not sure how far it is from materialization, and presumably it will take even longer for all major Wayland DEs to implement. The proposed change here on the contrary is trivial and incremental, and can be easily implemented.

[jadahl]

I am certainly much more fond of a few tweaks to the existing implementation than duplicating a lot more things.
Especially if clients are forced to potentially write two codepaths too.

There will always be two code paths if the same code base aims to cover two rather different use cases: screen sharing and headless remote desktop. Sharing the the pixel plumbing and input emulation is already addressed, which I'd say is the lion share of the complexty - the rest doesn't really have that much in common.

• pre-auth

We definitely want this in xdg portals for other things anyway. There's stuff happening in libei already for other use-cases.
That's a non-issue.

There is a major difference between flatpak (i.e. desktop applications) and headless remote desktop servers is that the trust is reversed - the software in question more or less replaces KMS/evdev, so the same kind of permission thinking doesn't apply anymore. This is software that needs to integrate with the display manager / login screen, it needs to run system services, it doesn't have any GUI to interact with.

The pre-auth for portals has a much more narrow use case in my mind.

• dynamic monitor creation/deletion
  This patch.

The problem I have with dynamic monitor creation/deletion is that it departs from what portals are which are ways for the user to be in control of what is shared. Now we support creating a virtual monitor, but it's a user action to do so; this would undermine that and give control to the application.

• ListMonitors

I can imagine us wanting this anyway for the flatpak case.

What desktop applications would want to list monitors, and for what use? If you care about presentation of some GUI, there is already RANDR/wl_output.

• Virtual monitor persistence.

I agree it's not something we want exposed to random Flatpak apps.

Though I don't fully understand it.
I understand the application requirement, but I don't get why we need new infra at a portal/desktop level. A VNC server can make one portal connection, and have a client connect/disconnect multiple times, the screen would implicitly persist.

The benefits is that it allows for remote desktop servers to only have services running when there are users connected. In theory one can indeed have a dummy service just to keep resources alive, but I don't believe one should make decisions about headless remote desktop platform API based on what desktop applications need or should have exposed to them.

• keymap control.

We definitely don't want something global expose to random flatpak apps.

But can it be on the libei device? We need to move to a world where each device has it's own keymap. Anything global is short-sighted.

How would it be on the libei device? There are ideas how to introduce sending keysyms instead of keycodes, but that doesn't really cover the use case that this is about. To some degree, this is about being able to have the same keyboard layout on the remote host as on the local one, so that changing the keyboard layout where the client is running allows switching the layout (and any indicator) to match it.

It's indeed not something that should ever be exposed to desktop applications, thus should not be part of the portal, but again that should not hinder creating a useful remote desktop platform API.

So while I see the benefit of "just" adding some headless remote desktop specific API to this API, it will never be a good enough replacement for the types of needs that exists.

[ywh233]

Let me put it this way. With GNOME ditching X11 in v49 and KDE planning to do so in Plasma 6.8, how can a Remote Desktop application developer continue to support both platforms? Before the new spec is finalized and implemented, we would need to integrate with both Portals and Mutter's private APIs. That is unnecessary fragmentation and a maintenance burden.

If the new spec is ready to use today, then I'm more than happy to integrate with it. Could you give me a rough timeline? When do you expect it to be implemented in Mutter?

The ScreenCast portal already supports selecting a virtual display before the session is started, so I don't see how the proposed change will philosophically change the portal. Do you think that it was a bad design that should have rather been reverted? Or is your concern about users not explicitly approving creations of new virtual monitors? I'm fine with only supporting the new DYNAMIC_VIRTUAL type if the application has been pre-approved for Remote Desktop/ScreenCast. Since this is for unattended remote access, so the app is presumably already pre-approved. Or you can have the user specify a maximum number of monitors that can be created during a Remote Desktop session.

[jadahl]

If the new spec is ready to use today, then I'm more than happy to integrate with it. Could you give me a rough timeline? When do you expect it to be implemented in Mutter?

I have started to implement it, but naturally it's hard to give an exact time frame. The same would apply to any portal API addition, as it requires code in various layers, and I'd prioritize working on a solution I see as a feasible way forward for anything but the most basic use case, than working towards a stop gap, but in my eyes dead end, solution. There are already features being discussed as definitely out of scope for a portal, that is expected to be available via a headless remote desktop API, so spending time adding "dynamic virtual" support would effectively work against the goal of creating a more powerful headless remote desktop API.

The ScreenCast portal already supports selecting a virtual display before the session is started, so I don't see how the proposed change will philosophically change the portal. Do you think that it was a bad design that should have rather been reverted? Or is your concern about users not explicitly approving creations of new virtual monitors? I'm fine with only supporting the new DYNAMIC_VIRTUAL type if the application has been pre-approved for Remote Desktop/ScreenCast. Since this is for unattended remote access, so the app is presumably already pre-approved. Or you can have the user specify a maximum number of monitors that can be created during a Remote Desktop session.

The main difference is that with virtual screens, the user right now is the one effectively creating it. I would consider adding/removal a more "privileged" feature, thus indeed something that depend on pre-auth infrastructure, to configure an "app" to actually be a trusted service, which I'd say is not something that'd just be a check box via some GUI. And that pre-auth infrastructure is something that doesn't exist yet.

[davidedmundson]

And that pre-auth infrastructure is something that doesn't exist yet.

The pre-auth infra does exist and is being used already.

[jadahl]

The pre-auth infra does exist and is being used already.

What I meant with pre-auth I'm was referring to the linked issue about a portal level pre-auth infrastructure, not about backend specific configurable policy.