gh-141335: Deque as a Growable Ring Buffer #141337
Conversation
matthiasgoergens
requested review from
AA-Turner and
rhettinger
as code owners
| Unix. They are also useful for tracking transactions and other pools of data | ||
| where only the most recent activity is of interest. | ||
| where only the most recent activity is of interest. Passing a *maxlen* | ||
| greater than :data:`sys.maxsize` raises :exc:`ValueError`. |
There was a problem hiding this comment.
We could also leave the behaviour the same as before (raise OverflowError), or fix it to be in line with what eg slicing for lists does (which handles huge numbers gracefully.)
There was a problem hiding this comment.
Pull Request Overview
This PR reimplements the collections.deque data structure from a doubly-linked list of blocks to a growable ring buffer (circular array), authored by Matthias Goergens. The primary goals are to improve performance for indexing operations and add slicing support.
Key Changes
- Replaced the block-based doubly-linked list implementation with a power-of-2 sized ring buffer using a single dynamically allocated array
- Added O(1) indexing support (previously O(n) in the middle) and O(k) slicing support for deques
- Updated documentation to reflect the new capabilities and performance characteristics
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| Modules/_collectionsmodule.c | Complete rewrite of deque internals from block-based to ring buffer implementation, including all operations (append, pop, rotate, etc.) |
| Modules/clinic/_collectionsmodule.c.h | Updated generated clinic code for deque_rotate to accept PyObject* instead of Py_ssize_t |
| Lib/test/test_deque.py | Added extensive hypothesis-based property tests and new test cases for slicing functionality |
| Lib/collections/init.py | Added import for reverse iterator type needed for pickle support |
| Doc/whatsnew/3.14.rst | Documented new slicing support and maxlen validation change |
| Doc/library/collections.rst | Updated documentation to reflect O(1) indexing, slicing support, and maxlen validation behavior |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| /* collections module implementation of a deque() datatype | ||
| Written and maintained by Raymond D. Hettinger <[email protected]> | ||
| Originally written by Raymond D. Hettinger <[email protected]> |
There was a problem hiding this comment.
@rhettinger I'm happy to handle this however you feel like, including leaving the whole comment as is.
matthiasgoergens
force-pushed
the
growable-ring-buffer
branch
from
c735b03 to
8aeb1d7
Compare
Co-authored-by: Copilot <[email protected]>
…wable-ring-buffer
|
Hmm, looks like I broke some of the tests. Will fix. |
matthiasgoergens
force-pushed
the
growable-ring-buffer
branch
from
f8212b3 to
99784a8
Compare
|
The remaining two CI failures seem to be flakiness? |
1 participant
Inspired by Rust's VecDeque, we have rewritten Python's
collections.dequeto use a growable ring buffer implementation. This change improves memory efficiency and performance for various operations, including indexing and slicing.Specifically, indexing is now O(1), and slicing is supported at all in O(k) time, where k is the number of elements in the slice.
collections.dequeto a growable ring buffer for fast random access #141335