feat: SimpleQ v2.0 — ground-up rewrite#8
Open
rdegges wants to merge 7 commits into
Open
Conversation
Remove Python 2 + boto v1 source code (jobs.py, queues.py, workers.py), old tests, setup.py, requirements.txt, and Travis CI config. This is a ground-up rewrite — none of the v1 code is reusable.
pyproject.toml with Python 3.10+, boto3, pydantic, structlog, typer, rich. Dev deps: pytest, moto, ruff, mypy. Updated .gitignore for modern Python.
- exceptions.py: 7-class hierarchy (SimpleQError → specific errors) - config.py: Config dataclass with env auto-detection - job.py: Job model with pickle serialization + 256KB size check - cost.py: CostTracker for SQS API call accounting
- queue.py: boto3 SQS wrapper with auto-create, FIFO, DLQ, batch ops - task.py: @sq.task decorator with .delay() and .apply() - app.py: SimpleQ entry point with task registry and queue cache
- worker.py: polling loop, burst mode, async tasks, retries with backoff, SIGINT/SIGTERM graceful shutdown - cli.py: typer CLI with worker, queue, and stats commands - __init__.py: public API exports for SimpleQ, Queue, Task, Worker, Job
Tests cover: job serialization, queue CRUD, FIFO, DLQ, task decorator, worker processing, burst mode, async tasks, retries, graceful shutdown, backoff calculation, CLI commands, cost tracking, config env overrides, and error cases (256KB limit, unpicklable args, auto_create=False).
README is the spec (README-driven development): quickstart, features (tasks, queues, worker, retries, DLQ, FIFO, async, cost tracking), CLI reference, configuration, Lambda mode teaser.
Owner
Author
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Review Summary by QodoSimpleQ v2.0 — Complete rewrite with Python 3.10+, boto3, and comprehensive test suite
WalkthroughsDescription• Complete ground-up rewrite from Python 2 + boto to Python 3.10+ + boto3 • Core library: SimpleQ app, @task decorator, Queue class, Worker with retries • CLI: worker, queue CRUD, stats, DLQ management commands • 104 comprehensive tests with moto covering all major features • Modern packaging: pyproject.toml, ruff, mypy, pytest, structured logging Diagramflowchart LR
A["Python 2 + boto v1"] -->|"Remove legacy code"| B["Clean slate"]
B -->|"Add core modules"| C["Job, Config, Cost, Exceptions"]
C -->|"Add Queue + Task"| D["Queue class, Task decorator"]
D -->|"Add Worker + CLI"| E["Worker polling, CLI commands"]
E -->|"Add 104 tests"| F["SimpleQ v2.0 complete"]
F -->|"Modern stack"| G["Python 3.10+, boto3, pydantic, typer, rich"]
File Changes2. simpleq/exceptions.py
|
Code Review by Qodo
1. Unbounded dependency versions
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
+2
to
+45
| requires = ["setuptools>=70.0"] | ||
| build-backend = "setuptools.build_meta" | ||
|
|
||
| [project] | ||
| name = "simpleq" | ||
| version = "2.0.0" | ||
| description = "The Python task queue for AWS. Dead simple. SQS native. Zero infrastructure." | ||
| readme = "README.md" | ||
| license = "ISC" | ||
| requires-python = ">=3.10" | ||
| authors = [ | ||
| { name = "Randall Degges", email = "r@rdegges.com" }, | ||
| ] | ||
| keywords = ["sqs", "aws", "queue", "worker", "tasks", "background-jobs"] | ||
| classifiers = [ | ||
| "Development Status :: 4 - Beta", | ||
| "Environment :: Console", | ||
| "Intended Audience :: Developers", | ||
| "Operating System :: OS Independent", | ||
| "Programming Language :: Python :: 3", | ||
| "Programming Language :: Python :: 3.10", | ||
| "Programming Language :: Python :: 3.11", | ||
| "Programming Language :: Python :: 3.12", | ||
| "Programming Language :: Python :: 3.13", | ||
| "Topic :: Software Development :: Libraries :: Python Modules", | ||
| "Topic :: System :: Distributed Computing", | ||
| ] | ||
| dependencies = [ | ||
| "boto3>=1.35.0", | ||
| "pydantic>=2.10.0", | ||
| "structlog>=24.4.0", | ||
| "typer>=0.15.0", | ||
| "rich>=13.9.0", | ||
| ] | ||
|
|
||
| [project.optional-dependencies] | ||
| dev = [ | ||
| "pytest>=8.3.0", | ||
| "pytest-cov>=6.0.0", | ||
| "moto[sqs]>=5.0.0", | ||
| "mypy>=1.13.0", | ||
| "ruff>=0.8.0", | ||
| "boto3-stubs[sqs]>=1.35.0", | ||
| ] |
There was a problem hiding this comment.
1. Unbounded dependency versions 📘 Rule violation ⛯ Reliability
pyproject.toml uses lower-bound-only constraints (e.g., boto3>=...) instead of bounded version ranges. This can introduce breaking changes via transitive/major upgrades and violates the dependency range policy.
Agent Prompt
## Issue description
`pyproject.toml` dependency constraints are unbounded (lower-bound only), which violates the requirement to use bounded version ranges.
## Issue Context
The manifest currently contains entries like `boto3>=1.35.0` and `setuptools>=70.0` with no `<` upper bound.
## Fix Focus Areas
- pyproject.toml[2-45]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+82
to
+88
| [tool.mypy] | ||
| python_version = "3.10" | ||
| warn_return_any = true | ||
| warn_unused_configs = true | ||
| disallow_untyped_defs = true | ||
| check_untyped_defs = true | ||
|
|
There was a problem hiding this comment.
2. Mypy strict mode missing 📘 Rule violation ⚙ Maintainability
The mypy configuration does not enable strict = true and omits many required strictness flags. This weakens type-safety guarantees required by the checklist.
Agent Prompt
## Issue description
Mypy is not configured per the required strict-mode policy (missing `strict = true` and multiple mandated flags).
## Issue Context
The project-level mypy config is defined in `pyproject.toml` under `[tool.mypy]`.
## Fix Focus Areas
- pyproject.toml[82-88]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+58
to
+81
| [tool.ruff] | ||
| target-version = "py310" | ||
| line-length = 88 | ||
|
|
||
| [tool.ruff.lint] | ||
| select = [ | ||
| "E", | ||
| "W", | ||
| "F", | ||
| "I", | ||
| "B", | ||
| "C4", | ||
| "UP", | ||
| "ARG", | ||
| "SIM", | ||
| "PERF", | ||
| ] | ||
| ignore = [ | ||
| "E501", | ||
| ] | ||
|
|
||
| [tool.ruff.lint.per-file-ignores] | ||
| "tests/*" = ["ARG", "S101"] | ||
|
|
There was a problem hiding this comment.
3. Ruff config missing required rules 📘 Rule violation ⚙ Maintainability
The Ruff configuration is missing required selections (e.g., TCH) and required global ignores (e.g., B008), and the tests per-file ignore glob is too narrow. This violates the mandated Ruff configuration policy.
Agent Prompt
## Issue description
Ruff is configured, but it does not match the required baseline (missing required rule groups and ignores, and tests per-file ignores are not applied broadly enough).
## Issue Context
The Ruff config lives in `pyproject.toml` under `[tool.ruff]`, `[tool.ruff.lint]`, and `[tool.ruff.lint.per-file-ignores]`.
## Fix Focus Areas
- pyproject.toml[58-81]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| @@ -0,0 +1,93 @@ | |||
| """Tests for the SimpleQ application class.""" | |||
There was a problem hiding this comment.
4. Tests not in required directories 📘 Rule violation ⚙ Maintainability
Tests are placed directly under tests/ instead of tests/unit/ or tests/integration/. This violates the required test directory layout policy.
Agent Prompt
## Issue description
Test files are located directly under `tests/` rather than the mandated `tests/unit/` and `tests/integration/` directories.
## Issue Context
Multiple new test modules use the `tests/test_*.py` pattern.
## Fix Focus Areas
- tests/test_app.py[1-1]
- tests/test_queue.py[1-1]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+34
to
+38
| self.fn = fn | ||
| self.app = app | ||
| self.name = f"{fn.__module__}.{fn.__qualname__}" | ||
| self.retries = retries | ||
| self.backoff = backoff |
There was a problem hiding this comment.
5. Task name violates allowed charset 📘 Rule violation ✓ Correctness
Task.name is constructed using module.qualname which includes dots and can include <locals>, violating the required [A-Za-z0-9_] task-name constraint and length policy. Invalid task names can break downstream assumptions and validation requirements.
Agent Prompt
## Issue description
Task names are generated in a format that violates the required task-name character set and length constraints.
## Issue Context
`fn.__module__` and `fn.__qualname__` produce dotted names and may include `<locals>`.
## Fix Focus Areas
- simpleq/task.py[34-38]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+172
to
+185
| try: | ||
| self.client.send_message(**kwargs) | ||
| except Exception as e: | ||
| if "NoCredential" in type(e).__name__ or "NoCredential" in str(e): | ||
| raise ConfigurationError( | ||
| "AWS credentials not found.\n\n" | ||
| " Set credentials using one of:\n" | ||
| " export AWS_ACCESS_KEY_ID=xxx\n" | ||
| " export AWS_SECRET_ACCESS_KEY=xxx\n\n" | ||
| " Or configure the AWS CLI:\n" | ||
| " aws configure" | ||
| ) from e | ||
| raise | ||
| self.cost_tracker.track_send(self.name) |
There was a problem hiding this comment.
9. Clienterror not mapped to simpleqerror 📘 Rule violation ⛯ Reliability
Boto3 SQS operations can raise ClientError, but the code does not consistently catch and wrap it into domain-specific SimpleQError subclasses. This can leak AWS exceptions through public SimpleQ APIs.
Agent Prompt
## Issue description
AWS/boto3 exceptions (notably `ClientError`) may escape SimpleQ public APIs because they are not translated into domain-specific exceptions.
## Issue Context
`send_job` currently re-raises the raw exception for non-credential errors.
## Fix Focus Areas
- simpleq/queue.py[172-185]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+123
to
+126
| @sq.task | ||
| def add(a, b): | ||
| return a + b | ||
|
|
There was a problem hiding this comment.
10. Untyped helper functions in tests 📘 Rule violation ⚙ Maintainability
Some new test-defined functions lack parameter and return type annotations. This violates the requirement for complete type hints on all function definitions.
Agent Prompt
## Issue description
Test-defined functions are missing required parameter and return type annotations.
## Issue Context
The policy applies to all function definitions in the change set, including those defined inside tests.
## Fix Focus Areas
- tests/test_task.py[123-126]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+161
to
+170
| kwargs: dict[str, Any] = { | ||
| "QueueUrl": self.url, | ||
| "MessageBody": job.serialize(), | ||
| } | ||
| if delay_seconds: | ||
| kwargs["DelaySeconds"] = delay_seconds | ||
| if message_group_id and self.fifo: | ||
| kwargs["MessageGroupId"] = message_group_id | ||
| if deduplication_id and self.fifo: | ||
| kwargs["MessageDeduplicationId"] = deduplication_id |
There was a problem hiding this comment.
11. Fifo messagegroupid missing 🐞 Bug ✓ Correctness
For FIFO queues, Task.delay() can call Queue.send_job() with message_group_id=None, and Queue.send_job() will omit MessageGroupId from the send_message request. This breaks FIFO enqueue (and FIFO DLQ redrive) because FIFO messages require a group id.
Agent Prompt
### Issue description
FIFO queues can be used without a MessageGroupId because `Task.delay()` may pass `message_group_id=None`, and `Queue.send_job()` conditionally omits `MessageGroupId`. This causes FIFO enqueue (and FIFO DLQ redrive) to fail at runtime.
### Issue Context
- `Task.delay()` only sets `group_id` when `self.message_group_id` is not `None`.
- `Queue.send_job()` only adds `MessageGroupId` when `message_group_id` is truthy.
- `Queue.redrive_dlq()` re-sends jobs without any group id.
### Fix Focus Areas
- simpleq/task.py[57-80]
- simpleq/queue.py[153-170]
- simpleq/queue.py[292-314]
### Suggested fix approach
- Enforce FIFO correctness by either:
1) raising a clear `ConfigurationError` when `fifo=True` and no `message_group_id` is provided, OR
2) providing a deterministic default group id (e.g., a constant like "default" or derived from task name) when `fifo=True`.
- For FIFO DLQ redrive, ensure a group id is available when re-sending:
- Option A (best): add an optional `message_group_id` field to `Job`, serialize it, set it in `Task.delay()`, and have `Queue.send_job()` use it (or pass it explicitly in `redrive_dlq`).
- Option B: choose a documented fallback group id for redriven FIFO messages if the original is unavailable (less ideal for ordering semantics).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+266
to
+276
| """List messages in a queue's dead letter queue.""" | ||
| client = _get_sqs_client(region) | ||
| dlq_name = f"{name}-dlq" | ||
|
|
||
| try: | ||
| url = client.get_queue_url(QueueName=dlq_name)["QueueUrl"] | ||
| except client.exceptions.QueueDoesNotExist: | ||
| console.print( | ||
| f"[red]✗[/red] No DLQ found for queue '{name}'.\n\n" | ||
| f" DLQ name expected: {dlq_name}\n" | ||
| f" Enable DLQ with: SimpleQ(dead_letter_queue=True)" |
There was a problem hiding this comment.
12. Fifo dlq name wrong 🐞 Bug ✓ Correctness
simpleq queue dlq always looks for a DLQ named {queue}-dlq, but the Queue implementation creates
FIFO DLQs as {queue}-dlq.fifo (by replacing .fifo). This makes the CLI DLQ command fail for FIFO
queues and prints the wrong “expected DLQ name”.
Agent Prompt
### Issue description
The CLI DLQ command uses a different DLQ naming scheme than the core Queue implementation for FIFO queues, causing `simpleq queue dlq <fifo-queue>` to fail.
### Issue Context
- CLI uses `{name}-dlq` unconditionally.
- Queue uses `{name}.replace('.fifo', '-dlq.fifo')` when FIFO.
### Fix Focus Areas
- simpleq/cli.py[261-279]
- simpleq/queue.py[118-123]
### Suggested fix approach
- Implement a shared helper for DLQ naming (e.g., `_dlq_name(queue_name: str) -> str`) and use it in:
- `queue_dlq`
- `queue_redrive` output text (it currently prints `'{name}-dlq'` even when FIFO)
- Ensure behavior matches Queue’s FIFO naming: `orders.fifo` -> `orders-dlq.fifo`.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+110
to
+139
| @app.command() | ||
| def worker( | ||
| app_module: str = typer.Option( | ||
| ..., "-a", "--app", help="Python module containing the SimpleQ app." | ||
| ), | ||
| queues: Optional[str] = typer.Option( | ||
| None, "-q", "--queues", help="Comma-separated queue names to listen to." | ||
| ), | ||
| concurrency: int = typer.Option( | ||
| 10, "-c", "--concurrency", help="Number of concurrent jobs." | ||
| ), | ||
| ) -> None: | ||
| """Start a worker that processes background jobs.""" | ||
| sq = _discover_app(app_module) | ||
|
|
||
| if queues: | ||
| queue_names = [q.strip() for q in queues.split(",")] | ||
| else: | ||
| # Use all queues from registered tasks | ||
| queue_names = list( | ||
| {t.queue_name for t in sq.tasks.values()} or {"default"} | ||
| ) | ||
|
|
||
| queue_objects = [sq.queue(name) for name in queue_names] | ||
|
|
||
| _print_banner(sq, queue_names) | ||
|
|
||
| w = Worker(queues=queue_objects, tasks=sq.tasks) | ||
| w.work() | ||
|
|
There was a problem hiding this comment.
13. Concurrency flag unused 🐞 Bug ✓ Correctness
The CLI exposes --concurrency but the value is never used; Worker is created without it and processes jobs sequentially. This is a silent behavioral bug that makes the CLI throughput options misleading.
Agent Prompt
### Issue description
`simpleq worker --concurrency/-c` is accepted but ignored; the worker runs single-threaded and sequential.
### Issue Context
- CLI defines `concurrency` but does not pass it to `Worker`.
- `Worker.work()` processes jobs in a simple nested loop.
### Fix Focus Areas
- simpleq/cli.py[110-139]
- simpleq/worker.py[47-85]
- README.md[236-242]
### Suggested fix approach
Choose one:
1) **Implement concurrency**:
- Add a `concurrency: int` parameter to `Worker`.
- Use a bounded executor (e.g., `concurrent.futures.ThreadPoolExecutor(max_workers=concurrency)`) for sync tasks and/or an asyncio-based runner for async tasks.
- Ensure deletion/visibility changes happen after task completion.
2) **Remove the flag**:
- Drop `--concurrency` from CLI and remove it from README until concurrency exists.
- This avoids a silent misconfiguration.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete rewrite of SimpleQ from Python 2 + boto to Python 3.10+ + boto3.
SimpleQ()app,@sq.taskdecorator with.delay()/.apply(),Queueclass with auto-create,Workerwith burst mode and graceful shutdownsimpleq worker,simpleq queue list/create/stats/purge/delete/dlq/redrive,simpleq statsTest Coverage
104 tests across 9 test files covering:
.delay(),.apply()Overall coverage: 84% (100% on core modules, lower on CLI/error paths)
Pre-Landing Review
4 issues found — all resolved:
logger.info()in signal handler moved to main loop (async-signal-safety)-aflagTODOS
Items completed in this PR:
Items remaining:
Test plan
🤖 Generated with Claude Code