Feature: Consider baking mcr.microsoft.com/windows/server:ltsc2022 into Windows AKS base image

[Jamie0]

Is your feature request related to a problem?/Why is this needed
We run a set of containers based off of server:ltsc2022 due to various dependencies missing in servercore:ltsc2022.

When a new node bootstraps, it can take up to 10-15 minutes to pull this image from mcr.microsoft.com

This also has an impact on node pool upgrades, where the backplane doesn't wait for containers to finish pulling images, before proceeding to upgrade the next node. Even if the new node is marked with a NoSchedule taint, this can cause an outage.

Describe the solution you'd like in detail
It would be great for server:ltsc2022 to be baked into the AKS base image, like nanoserver + servercore, to avoid this heavy dependency when a node bootstraps.

This change would for sure increase the image size of the VM image, but I don't think it would have much of an impact on provisioning speed in the grand scheme. I think virtually all OS disks in Azure would have plenty of space capacity for this.

Describe alternatives you've considered
We've considered trying to make our application run on plain servercore, but we have too many dependencies (e.g. MediaFoundation)

Additional context

I opened an issue related to the upgrade issue, but it was closed for being stale (despite me bumping the issue, albeit a bit late)

Our cloud-based application shares lots of common code with a .NET 4.8 desktop application. We also depend on MediaFoundation which isn't available in servercore. There are many other applications and use cases where servercore isn't sufficient - for example, software that automates/tests web browsers.

[djsly]

@timmy-wright what do you think a bout this ? server:ltsc2022 is huge thougt and we favor servercore:ltsc2022 based containers

[timmy-wright]

I've made a PR for this. It's tricky, because we do actually cache servercore:ltsc2022 (or 2019 or 2025 depending on the windows version), but we cache it under the versioned tag (ie: 10.0.20348.4297) as that lets our renovate tool cache the latest servercore and the latest-1 servercore (which means customer workloads based on latest and latest-1 have the servercore image cached).

However, this caching doesn't actually include the servercore:ltsc2022 label, so if a container is based on that, and then the ltsc2022 label is updated, containerd will always download the new servercore rather than using the cached container. Which is a bit crap. So I've made a PR to cache the ltsc2022 tag as well as our precisely versioned tags so that there is less impact to customer workloads.

[timmy-wright]

Oh wait. server:ltsc2022 not servercore:ltsc2022. My bad misreading the message. At least I found a subtle caching bug in our approach as a result!

This one is problematic. As @djsly mentioned, the server container is huge and our AKS VHDs are already pushing the limit that we can use due to product and other constraints. We can't fit it in.

However, we are currently working on a more generic solution to this that I can't really talk about yet, but stay tuned for an announcement in a few months.