lib/gis: new G_progress_* API with concurrency support#7350
Open
nilason wants to merge 2 commits intoOSGeo:mainfrom
Open
lib/gis: new G_progress_* API with concurrency support#7350nilason wants to merge 2 commits intoOSGeo:mainfrom
nilason wants to merge 2 commits intoOSGeo:mainfrom
Conversation
This adds new non-locking G_progress_* API, which is used to report incremental work completion for long-running operations. It supports percentage-, count- and time-based progress contexts. The implementation is organized as a telemetry pipeline. Producer-side API calls update atomic progress state and enqueue `EV_PROGRESS` or `EV_LOG` records into a bounded ring buffer. A single consumer thread drains that buffer, converts raw records into `event_t` values, and forwards them either to installed sink callbacks or to default renderers selected from the current G_info_format() mode. Concurrency is designed as multi-producer, single-consumer per telemetry stream. Producers reserve slots with an atomic `write_index`, publish events by setting a per-slot `ready` flag with release semantics, and use atomic compare-and-swap to ensure that only one producer emits a given percent threshold or time-gated update. The consumer advances a non-atomic `read_index`, waits for published slots, processes events in FIFO order, and then marks slots free again. Typical use pattern is: 1. Creation of `GProgressContext` with one of `G_progress_context_create*` 2. Call `G_progress_update` inside working loop 3. Finalize and free resources with `G_progress_context_destroy`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds new non-locking
G_progress_{*}API, which is used to report incremental work completion for long-running operations. It supports percentage-, count- and time-based progress contexts.Typical use pattern is:
or
This is a somewhat streamlined version of #7259, to be used explicitly with loops that may be run in parallel. This could still be considered "complicated code" compared to current
G_percent, but it is what I could come up with to achieve my goals of concurrent, lock-free, fast implementation with deterministic output (FIFO). In addition, it has a time based progress context (updating progress every X ms).Requirement: C11 atomic operations and POSIX threads. When those prerequisites are available,
G_USE_PROGRESS_NGis defined and this implementation is enabled. There should be a solution for adopting for MSVC, but I can't test that.Future possible improvements:
G_progress_debugto be able to properly debug concurrent code.Testing: I have added support to
r.horizon,r.mapcalc,r.sunandr.texture, so please run any of those tools, in particular with data sets that make good use of concurrent code (large regions etc.).Implementation details
The implementation is organized as a telemetry pipeline. Producer-side API calls update atomic progress state and enqueue
EV_PROGRESSorEV_LOGrecords into a bounded ring buffer. A single consumer thread drains that buffer, converts raw records intoevent_tvalues, and forwards them either to installed sink callbacks or to default renderers selected from the current G_info_format() mode.Concurrency is designed as multi-producer, single-consumer per telemetry stream. Producers reserve slots with an atomic
write_index, publish events by setting a per-slotreadyflag with release semantics, and use atomic compare-and-swap to ensure that only one producer emits a given percent threshold or time-gated update. The consumer advances a non-atomicread_index, waits for published slots, processes events in FIFO order, and then marks slots free again.References
https://medium.com/@s.g.manikandan/building-high-performance-lock-free-multi-producer-multi-consumer-queues-using-ring-buffer-for-8441205e80d9
https://www.joewood.me/posts/lock-free-ring-buffer
https://h4x0r.org/ring/