RFC: Unified Cursor Format #642
Description
Unified Cursor Format
Problem
Two separate cursor implementations with duplicated base62 encoding:
- LogStore:
v1:{sortBy}:{sortOrder}:{position}(not yet on main) - Tenants:
tntv01:{position}
Design Decision
Cursor format: {resource}v{version}:{data} → base62 encoded
| Resource | Format | Current Data |
|---|---|---|
| Events | evtv01:{data} |
Position string: 1737123456789_evt_abc |
| Deliveries | dlvv01:{data} |
Position string: 1737123456789_dlv_xyz |
| Tenants | tntv01:{data} |
Timestamp: 1737123456789 |
Key insight: Cursors only need opaque data. Sort direction comes from the request. The v1:{sortBy}:{sortOrder}:{position} format in the logstore branch is unnecessary.
Evolution
Currently, data is just a position string. If a driver needs richer data, bump the version:
// v01: data = position string
"evtv01:1737123456789_evt_abc"
// v02: data = structured (hypothetical)
"evtv02:1737123456789_evt_abc|extra"For structured data, avoid JSON (parsing overhead). Use a custom struct with MarshalBinary/UnmarshalBinary for efficient encoding.
Driver handling multiple versions:
data, err := cursor.Decode(req.Next, "evt", 2)
if errors.Is(err, cursor.ErrVersionMismatch) {
// Fallback to v1
data, err = cursor.Decode(req.Next, "evt", 1)
if err != nil {
return cursor.ErrInvalidCursor
}
position = data // v1: data is just position
} else {
// v2: custom struct with additional fields
var c CursorDataV2
if err := c.UnmarshalBinary([]byte(data)); err != nil {
return cursor.ErrInvalidCursor
}
position = c.Position
extra = c.Extra
}Each driver owns its data format. The cursor package is format-agnostic - just handles {resource}v{version}:{data} encoding.
Shared Package
New: internal/cursor/cursor.go
package cursor
var (
ErrInvalidCursor = errors.New("invalid cursor")
ErrVersionMismatch = errors.New("cursor version mismatch")
)
func Encode(resource string, version int, data string) string
func Decode(encoded, resource string, version int) (data string, err error)
func Base62Encode(s string) string
func Base62Decode(s string) (string, error)Backward Compatibility
Not strictly necessary (beta, few users, cursors are ephemeral), but cheap to include.
Legacy logstore cursors (raw base62'd data, no prefix):
data, err := cursor.Decode(req.Next, "evt", 1)
if errors.Is(err, cursor.ErrVersionMismatch) {
data, err = cursor.Base62Decode(req.Next) // legacy: no prefix
}