Skip to content

Commit d3fff80

Browse files
authored
feat(queue): add removeGlobalRateLimit method (#3481)
1 parent f1573b3 commit d3fff80

File tree

5 files changed

+109
-26
lines changed

5 files changed

+109
-26
lines changed

docs/gitbook/guide/queues/global-concurrency.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,16 @@ const globalConcurrency = await queue.getGlobalConcurrency();
1818
Note that if you choose a concurrency level in your workers, it will not override the global one, it will just be the maximum jobs a given worker can process in parallel but never more than the global one.
1919
{% endhint %}
2020

21+
### Remove Global Concurrency
22+
23+
It can be done using the following method:
24+
25+
```typescript
26+
await queue.removeGlobalConcurrency();
27+
```
28+
2129
## Read more:
2230

2331
- 💡 [Set Global Concurrency API Reference](https://api.docs.bullmq.io/classes/v5.Queue.html#setglobalconcurrency)
2432
- 💡 [Get Global Concurrency API Reference](https://api.docs.bullmq.io/classes/v5.Queue.html#getglobalconcurrency)
33+
- 💡 [Remove Global Concurrency API Reference](https://api.docs.bullmq.io/classes/v5.Queue.html#removeglobalconcurrency)

docs/gitbook/guide/queues/global-rate-limit.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,16 @@ const globalConcurrency = await queue.getRateLimitTtl();
1919
Note that if you choose a rate limit level in your workers, it won't override the global one.
2020
{% endhint %}
2121

22+
### Remove Global Rate Limit
23+
24+
It can be done using the following method:
25+
26+
```typescript
27+
await queue.removeGlobalRateLimit();
28+
```
29+
2230
## Read more:
2331

2432
- 💡 [Set Global Rate Limit API Reference](https://api.docs.bullmq.io/classes/v5.Queue.html#setglobalratelimit)
2533
- 💡 [Get Rate Limit Ttl API Reference](https://api.docs.bullmq.io/classes/v5.Queue.html#getratelimitttl)
34+
- 💡 [Remove Global Rate Limit API Reference](https://api.docs.bullmq.io/classes/v5.Queue.html#removeglobalratelimit)

docs/gitbook/python/changelogs/changelog-v1.md

Lines changed: 29 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -33,54 +33,57 @@
3333

3434
* **job:** add isWaiting method [python] ([#2328](https://github.com/taskforcesh/bullmq/issues/2328)) ([5db9f95](https://github.com/taskforcesh/bullmq/commit/5db9f957939cd873eea0224d34569189e5520e84))
3535

36-
## v1.19.0 (2023-12-12)
36+
# [1.19.0](https://github.com/taskforcesh/bullmq/compare/vpy1.18.0...vpy1.19.0) (2023-12-12)
3737

38-
### Feature
3938

40-
- **job:** Add promote method [python] ([#2323](https://github.com/taskforcesh/bullmq/issues/2323)) ([`61f4ba3`](https://github.com/taskforcesh/bullmq/commit/61f4ba3e99486aa36e5cc3d9b448b8080c567eb1))
39+
### Features
4140

42-
## v1.18.0 (2023-12-10)
41+
* **job:** add promote method [python] ([#2323](https://github.com/taskforcesh/bullmq/issues/2323)) ([61f4ba3](https://github.com/taskforcesh/bullmq/commit/61f4ba3e99486aa36e5cc3d9b448b8080c567eb1))
4342

44-
### Fix
43+
# [1.18.0](https://github.com/taskforcesh/bullmq/compare/vpy1.17.0...vpy1.18.0) (2023-12-10)
4544

46-
- **retry:** Pass right redis command name into retryJob script (#2321) [python] ([`6bb21a0`](https://github.com/taskforcesh/bullmq/commit/6bb21a07c9754659fa5aa1734df1046a6da5d16a))
47-
- **flows:** Add meta key to queues created with flows ([`272ec69`](https://github.com/taskforcesh/bullmq/commit/272ec69557f601a138e1aaba739f7e7878d5344b))
48-
- **update-progress:** Remove old updateProgress script to prevent conflict (#2298) (python) ([`e65b819`](https://github.com/taskforcesh/bullmq/commit/e65b819101f8e0e8fdef8c51cfdf9a52f5e73f13))
49-
- **worker:** Should cap update progress events ([`2cab9e9`](https://github.com/taskforcesh/bullmq/commit/2cab9e94f65c7bdd053e3fb5944bcda6e3ebaa39))
5045

51-
## v1.17.0 (2023-11-24)
46+
### Bug Fixes
5247

53-
### Feature
48+
* **retry:** pass right redis command name into retryJob script ([#2321](https://github.com/taskforcesh/bullmq/issues/2321)) [python] ([6bb21a0](https://github.com/taskforcesh/bullmq/commit/6bb21a07c9754659fa5aa1734df1046a6da5d16a))
49+
* **flows:** add meta key to queues created with flows ([272ec69](https://github.com/taskforcesh/bullmq/commit/272ec69557f601a138e1aaba739f7e7878d5344b))
50+
* **update-progress:** remove old updateProgress script to prevent conflict ([#2298](https://github.com/taskforcesh/bullmq/issues/2298)) (python) ([e65b819](https://github.com/taskforcesh/bullmq/commit/e65b819101f8e0e8fdef8c51cfdf9a52f5e73f13))
51+
* **worker:** should cap update progress events ([2cab9e9](https://github.com/taskforcesh/bullmq/commit/2cab9e94f65c7bdd053e3fb5944bcda6e3ebaa39))
5452

55-
- **worker:** Better handling of concurrency when fetching jobs ([#2242](https://github.com/taskforcesh/bullmq/issues/2242)) ([`d2e2035`](https://github.com/taskforcesh/bullmq/commit/d2e203588878ee64cb21e67141f73b32867dfb40))
53+
# [1.17.0](https://github.com/taskforcesh/bullmq/compare/vpy1.16.1...vpy1.17.0) (2023-11-24)
5654

57-
### Fix
5855

59-
- **worker:** Do not wait for slow jobs fixes #2290 ([`568d758`](https://github.com/taskforcesh/bullmq/commit/568d7585edb1f2ef15991d4ae4a2425e6834046a))
56+
### Features
6057

61-
## v1.16.1 (2023-11-09)
58+
* **worker:** better handling of concurrency when fetching jobs ([#2242](https://github.com/taskforcesh/bullmq/issues/2242)) ([d2e2035](https://github.com/taskforcesh/bullmq/commit/d2e203588878ee64cb21e67141f73b32867dfb40))
6259

63-
### Fix
60+
## [1.16.1](https://github.com/taskforcesh/bullmq/compare/vpy1.16.0...vpy1.16.1) (2023-11-09)
6461

65-
- **job:** Set delay value on current job instance when it is retried (#2266) (python) ([`76e075f`](https://github.com/taskforcesh/bullmq/commit/76e075f54d5745b6cec3cb11305bf3110d963eae))
6662

67-
## v1.16.0 (2023-11-08)
63+
### Bug Fixes
6864

69-
### Fix
65+
* **job:** set delay value on current job instance when it is retried ([#2266](https://github.com/taskforcesh/bullmq/issues/2266)) (python) ([76e075f](https://github.com/taskforcesh/bullmq/commit/76e075f54d5745b6cec3cb11305bf3110d963eae))
7066

71-
- **backoff:** Fix builtin backoff type (#2265) [python] ([`76959eb`](https://github.com/taskforcesh/bullmq/commit/76959eb9d9495eb1b6d2d31fab93c8951b5d3b93))
67+
# [1.16.0](https://github.com/taskforcesh/bullmq/compare/vpy1.15.4...vpy1.16.0) (2023-11-08)
7268

73-
## v1.15.4 (2023-11-05)
7469

75-
### Fix
70+
### Bug Fixes
7671

77-
- Update delay job property when moving to delayed set ([#2261](https://github.com/taskforcesh/bullmq/issues/2261)) ([`69ece08`](https://github.com/taskforcesh/bullmq/commit/69ece08babd7716c14c38c3dd50630b44c7c1897))
72+
* **backoff:** fix builtin backoff type ([#2265](https://github.com/taskforcesh/bullmq/issues/2265)) [python] ([76959eb](https://github.com/taskforcesh/bullmq/commit/76959eb9d9495eb1b6d2d31fab93c8951b5d3b93))
7873

79-
## v1.15.3 (2023-11-05)
74+
## [1.15.4](https://github.com/taskforcesh/bullmq/compare/vpy1.15.3...vpy1.15.4) (2023-11-05)
8075

81-
### Fix
8276

83-
- **add-job:** Trim events when waiting-children event is published (#2262) (python) ([`198bf05`](https://github.com/taskforcesh/bullmq/commit/198bf05fa5a4e1ce50081296033a2e0f26ece498))
77+
### Bug Fixes
78+
79+
* update delay job property when moving to delayed set ([#2261](https://github.com/taskforcesh/bullmq/issues/2261)) ([69ece08](https://github.com/taskforcesh/bullmq/commit/69ece08babd7716c14c38c3dd50630b44c7c1897))
80+
81+
## [1.15.3](https://github.com/taskforcesh/bullmq/compare/vpy1.15.2...vpy1.15.3) (2023-11-05)
82+
83+
84+
### Bug Fixes
85+
86+
* **add-job:** trim events when waiting-children event is published ([#2262](https://github.com/taskforcesh/bullmq/issues/2262)) (python) ([198bf05](https://github.com/taskforcesh/bullmq/commit/198bf05fa5a4e1ce50081296033a2e0f26ece498))
8487

8588
## v1.15.2 (2023-10-18)
8689

src/classes/queue.ts

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -309,6 +309,14 @@ export class Queue<
309309
return client.hdel(this.keys.meta, 'concurrency');
310310
}
311311

312+
/**
313+
* Remove global rate limit values.
314+
*/
315+
async removeGlobalRateLimit() {
316+
const client = await this.client;
317+
return client.hdel(this.keys.meta, 'max', 'duration');
318+
}
319+
312320
/**
313321
* Adds a new job to the queue.
314322
*

tests/test_rate_limiter.ts

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,60 @@ describe('Rate Limiter', function () {
130130
await result;
131131
await worker.close();
132132
});
133+
134+
describe('when rate limit is removed', () => {
135+
it('should execute jobs without rate limit', async function () {
136+
this.timeout(2000);
137+
138+
const numJobs = 10;
139+
140+
const worker = new Worker(
141+
queueName,
142+
async () => {
143+
const currentTtl = await queue.getRateLimitTtl();
144+
expect(currentTtl).to.be.equal(-2); // -2 means no rate limit
145+
},
146+
{
147+
connection,
148+
prefix,
149+
},
150+
);
151+
152+
await queue.setGlobalRateLimit(1, 500);
153+
await queue.removeGlobalRateLimit();
154+
155+
const result = new Promise<void>((resolve, reject) => {
156+
queueEvents.on(
157+
'completed',
158+
// after every job has been completed
159+
after(numJobs, async () => {
160+
try {
161+
const timeDiff = new Date().getTime() - startTime;
162+
expect(timeDiff).to.be.lte(150);
163+
resolve();
164+
} catch (err) {
165+
reject(err);
166+
}
167+
}),
168+
);
169+
170+
queueEvents.on('failed', async err => {
171+
await worker.close();
172+
reject(err);
173+
});
174+
});
175+
176+
const jobs = Array.from(Array(numJobs).keys()).map(() => ({
177+
name: 'rate test',
178+
data: {},
179+
}));
180+
const startTime = new Date().getTime();
181+
await queue.addBulk(jobs);
182+
183+
await result;
184+
await worker.close();
185+
});
186+
});
133187
});
134188

135189
it('should obey the rate limit', async function () {

0 commit comments

Comments
 (0)