-
Notifications
You must be signed in to change notification settings - Fork 60
Description
Is your feature request related to a problem? Please describe.
When the gateway returns a RESOURCE_EXHAUSTED error, the worker repeatedly retries to activate jobs without any delay. This "thundering herd" behavior overloads the gateway and worsens the resource exhaustion issue.
Describe the solution you'd like
Implement a configurable exponential backoff strategy for the job worker, specifically for RESOURCE_EXHAUSTED errors. The backoff should reset after a successful activation.
The configuration should include:
backoffFactorminDelaymaxDelayjitter(optional)
Describe alternatives you've considered
Keeping the current fixed retry mechanism is insufficient as it does not address the core problem of overwhelming the gateway under load.
Additional context
This feature aligns with best practices for resilient systems and is already present in other clients, such as the Java client.
https://docs.camunda.io/docs/apis-tools/java-client/job-worker/#backoff-configuration
https://github.com/camunda/camunda/blob/b9165e9759143e80e7e3bd2a884837cf141276a1/clients/java/src/main/java/io/camunda/zeebe/client/api/worker/BackoffSupplier.java