time: delay the cancellation of timers #7467

ADD-SP · 2025-07-19T08:50:37Z

Review guide

Please checkout Reduce contention in timer driver #6504 for the lock contention issue.
Please checkout RFC: Delayed cancellation of timer #7384 for high-level design.

Design document

See https://gist.github.com/ADD-SP/4037e25256b8dc8a4956962415de2356.

Benchmarks

See https://gist.github.com/ADD-SP/4037e25256b8dc8a4956962415de2356.

Signed-off-by: ADD-SP [email protected]

ADD-SP · 2025-07-19T13:31:47Z

tokio-util/tests/time_delay_queue.rs

 }

-#[tokio::test]
-async fn reset_later_after_slot_starts() {


Note

This test was removed because it was testing the behavior of tokio::runtime::time::entry::reset, which is removed in this PR.

ADD-SP · 2025-07-19T13:31:54Z

tokio-util/tests/time_delay_queue.rs

 }

-#[tokio::test]
-async fn reset_earlier_after_slot_starts() {


Note

This test was removed because it was testing the behavior of tokio::runtime::time::entry::reset, which is removed in this PR.

ADD-SP · 2025-07-19T13:41:16Z

tokio-util/tests/time_delay_queue.rs

    sleep(ms(20)).await;

-    assert!(queue.is_woken());
+    assert_ready_some!(poll!(queue));


Note

queue.reset_at resets inner Sleep, however, the new implementation will drop the inner timer and create a new one. So the waker will not be called, we have to poll manually.

ADD-SP · 2025-07-19T13:41:37Z

tokio-util/tests/time_delay_queue.rs

    sleep(ms(20)).await;

-    assert!(queue.is_woken());
+    assert_ready_some!(poll!(queue));


Note

queue.reset_at resets inner Sleep, however, the new implementation will drop the inner timer and create a new one. So the waker will not be called, we have to poll manually.

tokio/src/runtime/scheduler/current_thread/mod.rs

tokio/src/runtime/scheduler/multi_thread/worker.rs

ADD-SP · 2025-07-20T09:18:06Z

tokio/src/util/mod.rs

    feature = "signal",
-    feature = "time",
 ))]
 pub(crate) use wake_list::WakeList;


Note

We no long use WakeList in the time subsystem.

tokio/src/runtime/time/mod.rs

base is d23a838

base is 5dacc2e

Signed-off-by: ADD-SP <[email protected]>

base is d060401

Signed-off-by: ADD-SP <[email protected]>

ADD-SP · 2025-10-18T12:54:03Z

.github/workflows/ci.yml

          CARGO_TARGET_WASM32_WASIP1_RUNNER: "wasmtime run --"
          CARGO_TARGET_WASM32_WASIP1_THREADS_RUNNER: "wasmtime run -W bulk-memory=y -W threads=y -S threads=y --"
          RUSTFLAGS: --cfg tokio_unstable -Dwarnings -C target-feature=+atomics,+bulk-memory -C link-args=--max-memory=67108864
+          RUSTDOCFLAGS: -C link-args=--max-memory=67108864


Note

tokio/tokio-util/src/io/sync_bridge.rs

Line 91 in d060401

/// let mut data = vec![0; 16 * 1024];

Signed-off-by: ADD-SP <[email protected]>

base is d25778f

Signed-off-by: ADD-SP <[email protected]>

base is f490029

Signed-off-by: ADD-SP <[email protected]>

Remove debug print statement for new entry creation.

tokio/src/runtime/time/wheel/mod.rs

tokio/src/runtime/context.rs

Signed-off-by: ADD-SP <[email protected]>

base is 5efb1c3

Signed-off-by: ADD-SP <[email protected]>

base is 5a709e3

Signed-off-by: ADD-SP <[email protected]>

Darksonn · 2025-11-13T08:30:54Z

tokio/src/runtime/scheduler/multi_thread/worker.rs

+            let (core, ()) = self.enter_with_time_context(core, |time_cx| {
+                util::time::post_auto_advance(&handle.driver, auto_advance_duration);
+                util::time::process_expired_timers(&mut time_cx.wheel, &handle.driver);


Can we avoid the RefCell like this?

Obtain linked list of timers to wake up, while holding RefMut on core.

Release RefMut on core.

Iterate linked list to wake up tasks, without RefMut.

Darksonn · 2025-11-13T08:31:25Z

tokio/src/runtime/scheduler/multi_thread/worker.rs

+                    util::time::remove_cancelled_timers(&mut time_cx.wheel, &mut time_cx.canc_rx);
+                    let should_yield = util::time::insert_inject_timers(
+                        &mut time_cx.wheel,
+                        &time_cx.canc_tx,
+                        handle.take_remote_timers(),
+                    );


For cancellation, we probably also have to move timers to linked list, and cancel after releasing RefMut. (I think.)

Darksonn · 2025-11-13T08:32:51Z

tokio/src/runtime/scheduler/multi_thread/worker.rs

+        pub(crate) fn push_remote_timer(&self, hdl: EntryHandle) {
+            {
+                let mut synced = self.shared.synced.lock();
+                synced.inject_timers.push(hdl);
+            }
+            self.notify_parked_remote();


Maybe we should use a different mutex from task injectoin queue for contention?

Darksonn · 2025-11-13T08:38:53Z

tokio/src/runtime/time/wheel/entry.rs

+    /// The entry is in the pending queue of the timer wheel,
+    /// and not in any wheel level, which means that
+    /// the entry is reached its deadline and waiting to be woken up.
+    Pending(Sender, Waker, ThreadId),


Since block_in_place may move the Core from one thread to another, using the thread id may be a problem.

Darksonn · 2025-11-13T08:43:43Z

tokio/src/runtime/time/wheel/entry.rs

+        }
+    }
+
+    pub(crate) fn register_waker(&self, waker: &Waker) {


Consider dropping old waker after releasing lock.

Darksonn · 2025-11-13T08:56:41Z

tokio/src/runtime/time/timer.rs

+                    if thread_id == cur_thread_id {
+                        // Safety:
+                        // 1. entry is either in slots or pending list
+                        // 2. entry is registered in this thread
+                        unsafe {
+                            wheel.remove(entry);
+                        }


Maybe we can just call entry.transition_to_cancelling() always here to avoid relying on thread id?

ADD-SP changed the title ~~[WIP]~~ [WIP] time: delayed cancellation of timers Jul 19, 2025

ADD-SP force-pushed the add_sp/time-local-wheel branch 11 times, most recently from c2d5790 to d04c22f Compare July 19, 2025 13:19

ADD-SP changed the title ~~[WIP] time: delayed cancellation of timers~~ [WIP] time: delay the cancellation of timers Jul 19, 2025

ADD-SP commented Jul 19, 2025

View reviewed changes

tokio/src/runtime/scheduler/current_thread/mod.rs Outdated Show resolved Hide resolved

ADD-SP commented Jul 19, 2025

View reviewed changes

tokio/src/runtime/scheduler/multi_thread/worker.rs Outdated Show resolved Hide resolved

ADD-SP force-pushed the add_sp/time-local-wheel branch 2 times, most recently from 695eea8 to e9255ee Compare July 20, 2025 08:21

ADD-SP mentioned this pull request Jul 20, 2025

time: delay the Arc::clone of the scheduler::Handle until registering timer #7461

Closed

ADD-SP commented Jul 20, 2025

View reviewed changes

tokio/src/runtime/time/mod.rs Outdated Show resolved Hide resolved

ADD-SP added 7 commits October 16, 2025 21:55

merge: sync changes from the base branch

9bf117d

base is d23a838

fixup! cancel the timer locally if possible

df81bbd

fixup! cancel the timer locally if possible

88d3596

merge: sync changes from the base branch

ef78794

base is 5dacc2e

small refactor of cancellation code path

d8b47b9

Signed-off-by: ADD-SP <[email protected]>

merge: sync changes from the base branch

9cddfda

base is d060401

bump the wasm memory limit for doc-tests

63a93c2

Signed-off-by: ADD-SP <[email protected]>

ADD-SP commented Oct 18, 2025

View reviewed changes

ADD-SP and others added 9 commits October 21, 2025 08:15

fix potenial mem leak when dropping timer

aa413dd

Signed-off-by: ADD-SP <[email protected]>

runtime: improve style by reducing indent level

5e5312a

Signed-off-by: ADD-SP <[email protected]>

merge: sync changes from the base branch

1c59461

base is d25778f

fix memory leak when draining the cancel queue

94d40e3

Signed-off-by: ADD-SP <[email protected]>

merge: sync changes from the base branch

a5c3026

base is f490029

remove debugging code

17922c2

Signed-off-by: ADD-SP <[email protected]>

fix unused import error

c47aa24

Signed-off-by: ADD-SP <[email protected]>

remove debug print

47f4036

Remove debug print statement for new entry creation.

fix heap-use-after-free issue

5b53771

martin-g reviewed Oct 24, 2025

View reviewed changes

tokio/src/runtime/time/wheel/mod.rs Outdated Show resolved Hide resolved

tokio/src/runtime/context.rs Show resolved Hide resolved

ADD-SP added 11 commits October 24, 2025 23:23

remove legacy comments of Wheel::insert()

47e04e8

Signed-off-by: ADD-SP <[email protected]>

merge: sync changes from the base branch

fca1e42

base is 5efb1c3

handle the spurious wakeup of condvar in multi-thread runtime

15b0563

Signed-off-by: ADD-SP <[email protected]>

merge: sync changes from the base branch

d9ff6b8

base is 5a709e3

merge: sync changes from the base branch

22a3771

merge: sync changes from the base branch

a6a0a85

merge: sync changes from the base branch

ba627cf

merge: sync changes from the base branch

89b71f3

merge: sync changes from the base branch

522252f

fix reports of unsafe_op_in_unsafe_fn

e3102f7

Signed-off-by: ADD-SP <[email protected]>

merge: sync changes from the base branch

4082cba

Darksonn reviewed Nov 13, 2025

View reviewed changes

Uh oh!

time: delay the cancellation of timers #7467

Are you sure you want to change the base?

time: delay the cancellation of timers #7467

Conversation

ADD-SP commented Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review guide

Design document

Benchmarks

Uh oh!

Choose a reason for hiding this comment

Note

Uh oh!

Choose a reason for hiding this comment

Note

Uh oh!

ADD-SP Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Note

Uh oh!

ADD-SP Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Note

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Note

Uh oh!

Uh oh!

ADD-SP Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Note

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ADD-SP commented Jul 19, 2025 •

edited

Loading

ADD-SP Jul 19, 2025 •

edited

Loading

ADD-SP Jul 19, 2025 •

edited

Loading

ADD-SP Oct 18, 2025 •

edited

Loading