Skip to content

Provide documentation how to handle producer backpressure in Pulsar client applications #40

@lhotari

Description

@lhotari

When the client application produces messages faster than the Pulsar broker (Astra Streaming service) can accept, the broker will create back pressure on message sending. It is the responsibility of the client application to handle this.

Astra Streaming will throttle by default:
https://docs.datastax.com/en/streaming/astra-streaming/operations/astream-limits.html

The client application cannot assume that the broker has accepted the message for delivery until the send operation completes successfully. It isn't a data loss caused by our service when the broker throttles message sending. The client application must handle this type of scenario.

Based on our experience, one possible way to handle back pressure for a Pulsar producer is to set the Pulsar client's producer send timeout setting to infinite or a high value, and to enable the blockIfQueueFull setting.

When looking into this, I noticed that in the Apache Pulsar community, this type of knowledge hasn't been well-documented or shared.

How are our customers supposed to know that client applications are responsible for handling back pressure?
How could they understand what actions are necessary to manage back pressure properly?

The expected outcome of addressing this issue is to have answers to the above questions.
This documentation should be also contributed to Apache Pulsar upstream documentation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions