Enable Reactive Scaling by supporting Application mode along with Session Cluster

[JayGhiya]

To manage cloud costs effectively for customers, it is important to have reactive scaling across all sort of services residing in kubernetes. We have been active users of lyft's flink operator from the start and are extremely grateful for the operator's current features.

To enable reactive scaling in task managers from 0...n:
a) The operator has to support application mode cluster along with session mode.
b) The keda scaled object file has to be deployed by the operator and the trigger has to be externalized in the crd.

i and some members would love to contribute to this feature and some other pressing issues as well.

@anandswaminathan please let me know how to take this forward in terms of contribution .

[vipinshreyaskumar]

thank you @JayGhiya for opening this req.

[anandswaminathan]

I believe we looked into this. There were several simpler options. Will need to dig a bit.

[JayGhiya]

Okay. Can you please share those options? As flink operator creates a session cluster and apache flink does not support Reactive scaling on session cluster as of now.

…

On Mon, 14 Mar, 2022, 11:02 pm Anand Swaminathan, ***@***.***> wrote: I believe we looked into this. There were several simpler options. Will need to dig a bit. — Reply to this email directly, view it on GitHub <#249 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF5IO3LYB2ZM4UFWCMR7ZSDU75ZZVANCNFSM5QUWO7ZA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[anandswaminathan]

@JayGhiya Can you share or add relevant links as well? Let me take a look

[maghamravi]

We are actually thinking of the same. Happy to collaborate on !

[JayGhiya]

@anandswaminathan sure Here is the link for reactive scaling on flink through hpa -> Flink Reactive Scaling. Here is the link to the poc that was done with keda. Flink Keda Poc. Keda Reference

[Yang-LI-CS]

Hello @JayGhiya Can you share some feed back about your Flink Keda Poc, I am searching for this solution also. And I am in flink operator deployment, hope to find a ways to coexists flink operator autoscaler with keda 🙏

[JayGhiya]

sure @Yang-LI-CS . Here is the repo where I had run a experiment with keda using flink's official Kubernetes operator (https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/) ->https://github.com/JayGhiya/Flink-AutoScale-Poc . The experience is consistent and stable. what exactly are you looking for in terms of type of scalers as it depends on sources of Flink pipeline