Skip to content

lib/gis: new G_progress_* API with concurrency support#7350

Open
nilason wants to merge 2 commits intoOSGeo:mainfrom
nilason:upd_percent_multithread_new_API
Open

lib/gis: new G_progress_* API with concurrency support#7350
nilason wants to merge 2 commits intoOSGeo:mainfrom
nilason:upd_percent_multithread_new_API

Conversation

@nilason
Copy link
Copy Markdown
Contributor

@nilason nilason commented Apr 28, 2026

This adds new non-locking G_progress_{*} API, which is used to report incremental work completion for long-running operations. It supports percentage-, count- and time-based progress contexts.

Typical use pattern is:

GProgressContext *ctx = G_progress_context_create(n_rows, step);
for (row = 0; row < window.rows; row++) {
    // costly calculation ...
     G_progress_update(ctx);
}
G_progress_context_destroy(ctx);

or

GProgressContext *ctx = G_progress_context_create_counter(100);
while (TRUE) {
     G_progress_update(ctx);
}
G_progress_context_destroy(ctx);

This is a somewhat streamlined version of #7259, to be used explicitly with loops that may be run in parallel. This could still be considered "complicated code" compared to current G_percent, but it is what I could come up with to achieve my goals of concurrent, lock-free, fast implementation with deterministic output (FIFO). In addition, it has a time based progress context (updating progress every X ms).

Requirement: C11 atomic operations and POSIX threads. When those prerequisites are available, G_USE_PROGRESS_NG is defined and this implementation is enabled. There should be a solution for adopting for MSVC, but I can't test that.

Future possible improvements:

  • The time based progress context may be expanded to report estimated time (ETA).
  • It could perhaps be advantageous to add a G_progress_debug to be able to properly debug concurrent code.

Testing: I have added support to r.horizon, r.mapcalc, r.sun and r.texture, so please run any of those tools, in particular with data sets that make good use of concurrent code (large regions etc.).

Implementation details

The implementation is organized as a telemetry pipeline. Producer-side API calls update atomic progress state and enqueue EV_PROGRESS or EV_LOG records into a bounded ring buffer. A single consumer thread drains that buffer, converts raw records into event_t values, and forwards them either to installed sink callbacks or to default renderers selected from the current G_info_format() mode.

Concurrency is designed as multi-producer, single-consumer per telemetry stream. Producers reserve slots with an atomic write_index, publish events by setting a per-slot ready flag with release semantics, and use atomic compare-and-swap to ensure that only one producer emits a given percent threshold or time-gated update. The consumer advances a non-atomic read_index, waits for published slots, processes events in FIFO order, and then marks slots free again.

References

https://medium.com/@s.g.manikandan/building-high-performance-lock-free-multi-producer-multi-consumer-queues-using-ring-buffer-for-8441205e80d9

https://www.joewood.me/posts/lock-free-ring-buffer

https://h4x0r.org/ring/

nilason added 2 commits April 28, 2026 14:54
This adds new non-locking G_progress_* API, which is used to report
incremental work completion for long-running operations. It supports
percentage-, count- and time-based progress contexts.

The implementation is organized as a telemetry pipeline. Producer-side
API calls update atomic progress state and enqueue `EV_PROGRESS`
or `EV_LOG` records into a bounded ring buffer. A single consumer thread
drains that buffer, converts raw records into `event_t` values, and
forwards them either to installed sink callbacks or to default renderers
selected from the current G_info_format() mode.

Concurrency is designed as multi-producer, single-consumer per telemetry
stream. Producers reserve slots with an atomic `write_index`, publish
events by setting a per-slot `ready` flag with release semantics, and
use atomic compare-and-swap to ensure that only one producer emits a
given percent threshold or time-gated update. The consumer advances a
non-atomic `read_index`, waits for published slots, processes events
in FIFO order, and then marks slots free again.

Typical use pattern is:

1. Creation of `GProgressContext` with one of `G_progress_context_create*`
2. Call `G_progress_update` inside working loop
3. Finalize and free resources with `G_progress_context_destroy`
@nilason nilason added this to the 8.6.0 milestone Apr 28, 2026
@nilason nilason added the enhancement New feature or request label Apr 28, 2026
@github-actions github-actions Bot added raster Related to raster data processing C Related code is in C libraries module CMake labels Apr 28, 2026
@nilason nilason linked an issue Apr 28, 2026 that may be closed by this pull request
@nilason nilason requested a review from marisn April 28, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C Related code is in C CMake enhancement New feature or request libraries module raster Related to raster data processing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] G_percent is not safe to be called from parallel code

1 participant