Modifying GPU requirement with replica count change did not reflect in existing replica/ didn't succeed the deployment

[bhks]

What happened:

I have an EKS cluster with LWS operator deployed. I deployed a single replica with 8 GPU as resource requirement. When I made a change to the yaml with 2 replica and reduce my GPU requirement to 4 the existing deployed replica and the pod didn't change.

This caused the second replica to not get deployed even if the total GPU requirement was satisfied.

What you expected to happen:
The existing replica and its pods to get recreated and both replica and its pods to be deployed successfully.

How to reproduce it (as minimally and precisely as possible):

• Should be easy to repro with first creating a single replica of model deployment and then making changes with 2 replica count and reducing GPU resource from 8 to 4 for each replica.

Anything else we need to know?:
No

Environment:

• Kubernetes version (use kubectl version): 1.32
• LWS version (use git describe --tags --dirty --always): v0.7.0
• Cloud provider or hardware configuration: AWS/EKS + GPU
• OS (e.g: cat /etc/os-release): AL2023

[ahg-g]

Thanks for reporting, are you able to consistently reproduce the issue? can you check if the worker statefulset was updated or not (I expect not, but it would be great if you can check)?

We should add an integration test to reproduce and address this bug.

[bhks]

can you check if the worker statefulset was updated or not (I expect not, but it would be great if you can check)?

We have since redeployed, however when I checked at the time, you are right that it did not touch the existing replica.

We should add an integration test to reproduce and address this bug.

I can help to address this if you are ok with it.

[ahg-g]

I can help to address this if you are ok with it.

That would be great, this will help us root cause the issue.

[bhks]

I am going to look at how we are processing the events for updates and see if we have gaps where we are not making updates to existing running replica even if we find the specs of the replica changed.

[bhks]

Hi @ahg-g ,

Looking into the [code and comment], it seems we are prioritizing the scale up first and then rollingupdate to pods. Scale down is not impacted by this behavior.

In scenarios where we have limited GPUs lets say 1 node with 8 GPU, I assigned 8 GPU to 1 replica for a model. Then realized I can actually create 2 replica and my model will be working fine with 4 GPU. This deployment will not succeed right ?

Because it can't move forward as we don't have another 8 GPU node and it can't update the pod as our logic does not allow it to happen.

This can happen when we have say 4 * 8 GPU node as well, we initially created 4 replica and later realize we could have 8 replica where each replica with 4 GPU instead of 8 GPU. The deployment will never move forward and it will get stuck in this decision and ask for more capacity even if we have capacity to satisfy the deployment.

Is there a way we could process the pod spec update first meaning somehow delete and recreate the statefulset with new spec safely and then perform the replica scale up if we don't have capacity.

Looking for your input on this.