(2.14) Support durable and flow-controled stream sourcing/mirroring #7613
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements durable stream sourcing/mirroring as described in this ADR: nats-io/nats-architecture-and-design#389. Built on top of the consumer reset API: #7489.
A durable push consumer can be created with
AckPolicy: AckFlowControland a value forHeartbeat, to be used for stream sourcing and mirroring as an alternative for the current ephemeral consumer. Messages will be acknowledged automatically by the receiving end based on flow control, or triggered by the heartbeat. Additionally,MaxAckPendingspecifies the upper bound for pending messages, if this is reached a flow control message will also be sent, if not already, to acknowledge messages.AckPolicy: AckFlowControlis special in that it only allows for infinite retries forMaxDeliverand has noAckWaitorBackOff. Messages are meant to reliably flow from one place to the other, the heartbeat somewhat functions likeAckWaitbut acknowledgements may happen either earlier or later depending on how many messages are sent, how large they are (larger means flow control happens sooner), or whenMaxAckPendingis hit.Importantly, this durable consumer can reliably source and mirror messages from Interest and WorkQueue retention streams, whereas this is not fully reliable with the current ephemeral consumer approach.
Resolves #4109
Signed-off-by: Maurice van Veen [email protected]