Skip to content

Add throttle queue to speed up processing #14791

@shuangkun

Description

@shuangkun

Summary

Add throttle queue to speed up processing,move wfc.throttler.Add from events handler to workqueue.

Use Cases

I've been having performance issues lately. with many logs like this appearing every day. There's also the issue of inconsistencies between the local informer and the remote apiserver.

   1 trace.go:205] Trace[1468444633]: "DeltaFIFO Pop Process" ID:simulation-data/xxxxxxxxx,Depth:139805,Reason:slow event handlers blocking the queue (19-Aug-2025 14:11:39.233) (total time: 123ms):

When would you use this?

Large-scale senario, to Slowing down the performance issue.

I believe one reason is that wfc.throttler.Add is taking quite a while. Its operation is more complex than before, requiring it to iterate through the queue and calculate the number of running requests in each namespace.

				UpdateFunc: func(obj interface{}) {
					key, err := cache.MetaNamespaceKeyFunc(obj)
					if err == nil {
						// for a new workflow, we do not want to rate limit its execution using AddRateLimited
						wfc.wfQueue.AddAfter(key, wfc.Config.InitialDelay.Duration)
						priority, creation := getWfPriority(obj)
						wfc.throttler.Add(key, priority, creation)  -》 wfc.throttlerQueue.Add(newThrottlerKey(key,action))
					}
				},

When would you use this?


Message from the maintainers:

Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions