Add OpenTelemetry instrumentation to API/Agent gRPC interface

[Salazar-99]

Overview

To make debugging in development and production easier we would like start instrumenting our internal code with OpenTelemetry data. To start with, we will add instrumentation to the Cluster API/Cluster Agent gRPC interface and use that as a model for the future. As part of this project we will also write up how to set up SigNoz locally to visualize the otel data and post it to the GitHub wiki.

Motivation

In the process of debugging #330 we realized that the distributed nature of the request made it difficult to debug so we want to take a step towards making these types of issues easier to debug by instrumenting our code with OpenTelementry data. Rather than instrumenting all of the code all at once we want to add it in once place to serve as a proof-of-concept and as a reference implementation for the future.

Goals

• Add OpenTelemetry instrumentation to the Cluster API gRPC client
• Add OpenTelemetry instrumentation to the Cluster Agent gRPC server
• Write up how to use SigNoz to visualize the data locally
• Identify other places to add OpenTelementry instrumentation
• Create GitHub issues if appropriate

Non-Goals

We don't want to instrument everything everywhere all at once.

Design Overview

• We can use the logging.level config flag to detect "debug" mode:
  https://github.com/kubetail-org/kubetail/blob/main/modules/shared/config/config.go#L113
• We can use this reference implementation to instrument the client/server https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/google.golang.org/grpc/otelgrpc/example

Open Questions

No response

Future Work

No response

[Salazar-99]

I think the general approach should be to isolate the client-server interaction between the Cluster API and the Cluster Agent as a starting point. There is a gRPC Otel library with examples here: https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/google.golang.org/grpc/otelgrpc/example.

SigNoz can be deployed either via Docker or Helm as shown here: https://github.com/SigNoz/signoz?tab=readme-ov-file#deploy-using-dockerself-hosted

I'll give this a shot sometime this week.

[amorey]

Sounds great. I updated the spec with your suggestions.

[Salazar-99]

It may be worth investigating ClickStack, which came out after starting this discussion, as an alternative to SigNoz.
https://clickhouse.com/blog/clickstack-a-high-performance-oss-observability-stack-on-clickhouse