Prowlarr parallel requests and rank torrent name fix#362
Prowlarr parallel requests and rank torrent name fix#362lucamaia9 wants to merge 5 commits intog0ldyy:mainfrom
Conversation
WalkthroughThe pull request refactors Prowlarr search execution from a single aggregated HTTP request to parallel per-indexer concurrent requests with individual timeouts, adds explicit Changes
Sequence DiagramsequenceDiagram
participant User
participant get_prowlarr
participant Fetch as fetch_prowlarr_results
participant Prowlarr as Prowlarr API
participant Results as Result Aggregation
Note over User,Results: Previous Flow (Single Request)
User->>get_prowlarr: search query
get_prowlarr->>Prowlarr: single aggregated search
Prowlarr-->>get_prowlarr: combined results
get_prowlarr->>Results: process & filter
Results-->>User: final results
Note over User,Results: New Flow (Parallel Requests)
User->>get_prowlarr: search query
get_prowlarr->>get_prowlarr: collect indexer IDs
rect rgb(200, 220, 240)
Note over get_prowlarr,Prowlarr: asyncio.gather() — parallel per-indexer requests
par Indexer 1
get_prowlarr->>Fetch: indexer_id=1, query
Fetch->>Prowlarr: search (timeout)
Prowlarr-->>Fetch: results
Fetch-->>get_prowlarr: results[1]
and Indexer 2
get_prowlarr->>Fetch: indexer_id=2, query
Fetch->>Prowlarr: search (timeout)
Prowlarr-->>Fetch: results
Fetch-->>get_prowlarr: results[2]
and Indexer N
get_prowlarr->>Fetch: indexer_id=N, query
Fetch->>Prowlarr: search (timeout)
Prowlarr-->>Fetch: results
Fetch-->>get_prowlarr: results[N]
end
end
get_prowlarr->>Results: flatten & aggregate
get_prowlarr->>Results: process & filter
Results-->>User: final results
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes The changes introduce new async concurrency logic in Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
comet/scrapers/prowlarr.py(2 hunks)comet/utils/torrent.py(1 hunks)pyproject.toml(1 hunks)
🔇 Additional comments (3)
comet/utils/torrent.py (1)
53-56: LGTM! Explicit timeout handling improves observability.The dedicated
asyncio.TimeoutErrorhandler provides clear logging for timeout scenarios, which complements the parallel request changes in the Prowlarr scraper.comet/scrapers/prowlarr.py (1)
80-102: LGTM! Well-structured per-indexer fetch with proper timeout and error handling.The function correctly isolates failures to individual indexers and returns partial results when some indexers timeout or fail.
pyproject.toml (1)
26-26: Pinned version is valid and secure.Verification confirms that rank-torrent-name version 1.9.0 exists on PyPI and has no known security vulnerabilities.
| get_indexers_response = await session.get( | ||
| f"{settings.INDEXER_MANAGER_URL}/api/v1/indexer", | ||
| headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY}, | ||
| timeout=aiohttp.ClientTimeout(total=settings.INDEXER_MANAGER_TIMEOUT), | ||
| ) | ||
| get_indexers = await get_indexers.json() | ||
| get_indexers = await get_indexers_response.json() |
There was a problem hiding this comment.
Add status check before parsing JSON response.
The code calls json() without verifying the HTTP response status. If the indexer list fetch fails with a non-2xx status, this could raise an exception.
Apply this diff to check the status:
get_indexers_response = await session.get(
f"{settings.INDEXER_MANAGER_URL}/api/v1/indexer",
headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY},
timeout=aiohttp.ClientTimeout(total=settings.INDEXER_MANAGER_TIMEOUT),
)
+get_indexers_response.raise_for_status()
get_indexers = await get_indexers_response.json()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| get_indexers_response = await session.get( | |
| f"{settings.INDEXER_MANAGER_URL}/api/v1/indexer", | |
| headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY}, | |
| timeout=aiohttp.ClientTimeout(total=settings.INDEXER_MANAGER_TIMEOUT), | |
| ) | |
| get_indexers = await get_indexers.json() | |
| get_indexers = await get_indexers_response.json() | |
| get_indexers_response = await session.get( | |
| f"{settings.INDEXER_MANAGER_URL}/api/v1/indexer", | |
| headers={"X-Api-Key": settings.INDEXER_MANAGER_API_KEY}, | |
| timeout=aiohttp.ClientTimeout(total=settings.INDEXER_MANAGER_TIMEOUT), | |
| ) | |
| get_indexers_response.raise_for_status() | |
| get_indexers = await get_indexers_response.json() |
🤖 Prompt for AI Agents
In comet/scrapers/prowlarr.py around lines 110 to 115, the code calls
get_indexers_response.json() without checking the HTTP status; add a status
check on get_indexers_response.status and handle non-2xx responses before
parsing JSON (e.g., raise or log an error and return/raise an exception) so you
only call .json() when the response is successful; ensure you include the status
and response text/body in the error handling for debugging and use the same
timeout and headers already present.
| tasks = [ | ||
| fetch_prowlarr_results(session, indexer_id, title) | ||
| for indexer_id in indexers_id | ||
| ] | ||
| all_results_lists = await asyncio.gather(*tasks) | ||
|
|
||
| # Flatten the list of lists into a single list of results | ||
| response = [result for sublist in all_results_lists for result in sublist] | ||
|
|
There was a problem hiding this comment.
Add return_exceptions=True to asyncio.gather for graceful degradation.
Without return_exceptions=True, an unexpected exception from any indexer task (not caught by fetch_prowlarr_results) will cancel all other tasks and lose partial results, defeating the purpose of parallel requests with timeout tolerance.
Apply this diff:
tasks = [
fetch_prowlarr_results(session, indexer_id, title)
for indexer_id in indexers_id
]
-all_results_lists = await asyncio.gather(*tasks)
+all_results_lists = await asyncio.gather(*tasks, return_exceptions=True)
# Flatten the list of lists into a single list of results
-response = [result for sublist in all_results_lists for result in sublist]
+response = [
+ result
+ for sublist in all_results_lists
+ if not isinstance(sublist, Exception)
+ for result in sublist
+]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| tasks = [ | |
| fetch_prowlarr_results(session, indexer_id, title) | |
| for indexer_id in indexers_id | |
| ] | |
| all_results_lists = await asyncio.gather(*tasks) | |
| # Flatten the list of lists into a single list of results | |
| response = [result for sublist in all_results_lists for result in sublist] | |
| tasks = [ | |
| fetch_prowlarr_results(session, indexer_id, title) | |
| for indexer_id in indexers_id | |
| ] | |
| all_results_lists = await asyncio.gather(*tasks, return_exceptions=True) | |
| # Flatten the list of lists into a single list of results | |
| response = [ | |
| result | |
| for sublist in all_results_lists | |
| if not isinstance(sublist, Exception) | |
| for result in sublist | |
| ] |
🤖 Prompt for AI Agents
In comet/scrapers/prowlarr.py around lines 126 to 134, asyncio.gather is called
without return_exceptions=True which will cancel all tasks if any task raises;
change the gather call to asyncio.gather(*tasks, return_exceptions=True) and
then post-process all_results_lists to filter out exceptions (e.g., iterate
results, if isinstance(item, Exception) log or ignore it, otherwise treat as the
expected list), finally flatten only the successful result lists into response
so partial results are preserved.
|
rtn fixed in 831a0cf |
2 participants
Prowlarr scrapper can use parallel requests so the timeout works returning the indexers that successfully returned values. Also fix to "rank-torrent-name==1.9.0", as the new version (1.9.2) causes import issues and would require further code changes.
Summary by CodeRabbit
Bug Fixes
Refactor
Chores