MON-4059: Update TelemeterClientFailures alert #550

slashpai · 2024-11-19T08:19:14Z

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Ref: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md#diagnosis

Use the new `metricsclient_http_requests_total` metric which would tell the difference between 4xx errors (e.g. bad pull secret) and 5xx (issue on Red Hat side). Signed-off-by: Jayapriya Pai <[email protected]>

openshift-ci-robot · 2024-11-19T08:19:19Z

@slashpai: This pull request references MON-4059 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

Details

In response to this:

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-11-19T08:21:20Z

@slashpai: This pull request references MON-4059 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

Details

In response to this:

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Ref: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md#diagnosis

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

simonpasquier

I'd recommend to create 2 alerting rules: one for the federate_from client and another one for the federate_to client. They can have the same name but different descriptions.

simonpasquier · 2024-11-19T10:17:32Z

jsonnet/telemeter/client/kubernetes.libsonnet

-                  sum by (namespace) (
-                    rate(federate_requests_total{job="telemeter-client"}[15m])
-                  ) > 0.2
+                  sum by(client, status_code) (rate(metricsclient_http_requests_total{status_code!~"200"}[15m])) > 0


(suggestion) we need to preserve the namespace label. Let's also evaluate the rate of failed requests.

Suggested change

sum by(client, status_code) (rate(metricsclient_http_requests_total{status_code!~"200"}[15m])) > 0

sum by(client, status_code,namespace) (rate(metricsclient_http_requests_total{status_code!~"2..",job="telemeter-client"}[15m]))

/

on(client, namespace) group_left() sum by(client, namespace) (rate(metricsclient_http_requests_total{job="telemeter-client"}[15m])) > 0.2

openshift-bot · 2025-04-22T01:00:35Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2025-05-31T00:30:41Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2025-06-30T08:00:34Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci · 2025-06-30T08:01:17Z

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

simonpasquier · 2025-06-30T08:28:37Z

/reopen

openshift-ci · 2025-06-30T08:28:48Z

@simonpasquier: Reopened this PR.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot · 2025-06-30T08:28:51Z

@slashpai: This pull request references MON-4059 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Ref: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md#diagnosis

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

simonpasquier · 2025-06-30T08:28:52Z

remove-lifecycle rotten

simonpasquier · 2025-06-30T08:28:57Z

/remove-lifecycle rotten

openshift-ci · 2025-06-30T08:29:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: slashpai

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [slashpai]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2025-09-29T01:01:25Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2025-10-29T08:30:16Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2025-11-29T00:01:12Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci · 2025-11-29T00:01:39Z

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

simonpasquier · 2025-12-01T08:52:28Z

/reopen
/remove-lifecycle rotten

simonpasquier · 2025-12-01T08:52:48Z

/lifecycle frozen

openshift-ci · 2025-12-01T08:52:58Z

@simonpasquier: The lifecycle/frozen label cannot be applied to Pull Requests.

Details

In response to this:

/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2025-12-01T08:54:08Z

@simonpasquier: Reopened this PR.

Details

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot · 2025-12-01T08:54:09Z

@slashpai: This pull request references MON-4059 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Ref: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md#diagnosis

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-12-01T11:28:43Z

@slashpai: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn	`6ff5685`	link	true	`/test e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

MON-4059: Update TelemeterClientFailures alert

6ff5685

Use the new `metricsclient_http_requests_total` metric which would tell the difference between 4xx errors (e.g. bad pull secret) and 5xx (issue on Red Hat side). Signed-off-by: Jayapriya Pai <[email protected]>

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 19, 2024

openshift-ci bot requested review from simonpasquier and thibaultmg November 19, 2024 08:21

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 19, 2024

simonpasquier reviewed Nov 19, 2024

View reviewed changes

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 22, 2025

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 31, 2025

openshift-ci bot closed this Jun 30, 2025

openshift-ci bot reopened this Jun 30, 2025

openshift-ci bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jun 30, 2025

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 29, 2025

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 29, 2025

openshift-ci bot closed this Nov 29, 2025

openshift-ci bot reopened this Dec 1, 2025

openshift-ci bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Dec 1, 2025

-                  sum by(client, status_code) (rate(metricsclient_http_requests_total{status_code!~"200"}[15m])) > 0
+                  sum by(client, status_code,namespace) (rate(metricsclient_http_requests_total{status_code!~"2..",job="telemeter-client"}[15m]))
+                  /
+                  on(client, namespace) group_left() sum by(client, namespace) (rate(metricsclient_http_requests_total{job="telemeter-client"}[15m])) > 0.2

MON-4059: Update TelemeterClientFailures alert #550

Are you sure you want to change the base?

MON-4059: Update TelemeterClientFailures alert #550

Uh oh!

Conversation

slashpai commented Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 19, 2024 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 19, 2024 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonpasquier left a comment

Choose a reason for hiding this comment

Uh oh!

simonpasquier Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

openshift-bot commented Apr 22, 2025

Uh oh!

openshift-bot commented May 31, 2025

Uh oh!

openshift-bot commented Jun 30, 2025

Uh oh!

openshift-ci bot commented Jun 30, 2025

Uh oh!

simonpasquier commented Jun 30, 2025

Uh oh!

openshift-ci bot commented Jun 30, 2025

Uh oh!

openshift-ci-robot commented Jun 30, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonpasquier commented Jun 30, 2025

Uh oh!

simonpasquier commented Jun 30, 2025

Uh oh!

openshift-ci bot commented Jun 30, 2025

Uh oh!

openshift-bot commented Sep 29, 2025

Uh oh!

openshift-bot commented Oct 29, 2025

Uh oh!

openshift-bot commented Nov 29, 2025

Uh oh!

openshift-ci bot commented Nov 29, 2025

Uh oh!

simonpasquier commented Dec 1, 2025

Uh oh!

simonpasquier commented Dec 1, 2025

Uh oh!

openshift-ci bot commented Dec 1, 2025

Uh oh!

openshift-ci bot commented Dec 1, 2025

Uh oh!

openshift-ci-robot commented Dec 1, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Dec 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

slashpai commented Nov 19, 2024 •

edited

Loading

openshift-ci-robot commented Nov 19, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Nov 19, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Jun 30, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Dec 1, 2025 •

edited by openshift-ci bot

Loading