Skip to content

feat(taps): New RESTStream.get_http_request method to construct the HTTP request for a stream#3483

Open
edgarrmondragon wants to merge 34 commits intomainfrom
feat/get_http_request
Open

feat(taps): New RESTStream.get_http_request method to construct the HTTP request for a stream#3483
edgarrmondragon wants to merge 34 commits intomainfrom
feat/get_http_request

Conversation

@edgarrmondragon
Copy link
Collaborator

@edgarrmondragon edgarrmondragon commented Feb 12, 2026

Related

Summary by Sourcery

Introduce a dedicated HTTPRequest abstraction in the REST stream layer and update tap-dummyjson to use it for request construction and parameter handling.

New Features:

  • Add an HTTPRequest dataclass for REST streams, including customizable query parameter encoding and reusable request construction via RESTStream.get_http_request.

Enhancements:

  • Refactor RESTStream.prepare_request to build requests from HTTPRequest instances and standardize URL parameter typing and encoding.
  • Simplify DummyJSONAuthenticator to always set the Authorization header after ensuring a valid token.
  • Update tap-dummyjson client pagination to override get_http_request instead of get_url_params for setting paging parameters.

Tests:

  • Add a unit test covering custom HTTPRequest.encode_params behavior with safe characters in query parameters.

Summary by Sourcery

Introduce an HTTPRequest abstraction and context for RESTStream to construct and customize HTTP requests, and migrate pagination and client patterns to use it instead of get_url_params while deprecating prepare_request overrides.

New Features:

  • Add an HTTPRequest dataclass and HTTPRequestContext for representing REST stream HTTP requests and their pagination/context.
  • Expose RESTStream.get_http_request for building per-request URL, headers, params, and payload from stream context and paginator state.

Enhancements:

  • Refactor RESTStream request preparation to build PreparedRequest objects from HTTPRequest instances with standardized payload typing and parameter encoding.
  • Update built-in taps (dummyjson, GitLab) and the cookiecutter tap template to implement pagination by overriding get_http_request instead of get_url_params.
  • Improve paginator typing for RESTStream and BaseAPIPaginator implementations.
  • Emit deprecation warnings when prepare_request is overridden, including when the override is in an intermediate base class.
  • Adjust documentation examples to show customizing pagination by modifying HTTPRequest rather than URL parameter dicts.

Documentation:

  • Add reference stubs for HTTPRequest and HTTPRequestContext and update pagination and incremental replication guides to document get_http_request-based pagination patterns.

Tests:

  • Add unit tests for HTTPRequest parameter encoding, including custom safe characters and non-string parameter values.
  • Extend REST pagination and metrics tests to cover the new get_http_request usage and deprecation warnings for prepare_request overrides.

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
@edgarrmondragon edgarrmondragon self-assigned this Feb 12, 2026
@sourcery-ai
Copy link
Contributor

sourcery-ai bot commented Feb 12, 2026

Reviewer's Guide

Introduce an HTTPRequest/HTTPRequestContext abstraction for REST streams, refactor RESTStream request preparation to build PreparedRequest objects from HTTPRequest instances, migrate pagination/request examples and taps to override get_http_request instead of get_url_params, and add tests and docs for the new behavior and deprecation of prepare_request overrides.

Sequence diagram for RESTStream.request_records with HTTPRequest

sequenceDiagram
    actor TapRunner
    participant RESTStream
    participant Paginator as BaseAPIPaginator
    participant Ctx as HTTPRequestContext
    participant Req as HTTPRequest
    participant Session as RequestsSession
    participant API

    TapRunner->>RESTStream: request_records(context)
    RESTStream->>RESTStream: get_new_paginator()
    RESTStream-->>RESTStream: Paginator instance

    loop until Paginator.finished
        alt prepare_request overridden (deprecated path)
            RESTStream->>RESTStream: prepare_request(context,next_page_token)
            RESTStream-->>Session: PreparedRequest
        else default path using HTTPRequest
            RESTStream->>RESTStream: _prepare_request(context,paginator)
            RESTStream->>Ctx: construct HTTPRequestContext(stream_context,paginator)
            RESTStream->>RESTStream: get_http_request(context:HTTPRequestContext)
            RESTStream-->>Req: HTTPRequest
            Req->>Req: encode_params()
            RESTStream->>Session: build_prepared_request(method,url,encoded_params,headers,data or json)
            Session-->>RESTStream: PreparedRequest
        end

        RESTStream->>API: decorated_request(PreparedRequest,context)
        API-->>RESTStream: Response
        RESTStream->>Paginator: advance(response)
    end

    RESTStream-->>TapRunner: records iterator
Loading

Updated class diagram for RESTStream HTTPRequest abstraction

classDiagram
    class HTTPRequest {
        +str url
        +str method
        +dict~str,str~ headers
        +dict~str,any or list~any~ or None~ params
        +JSONPayload data
        +str safe_query_chars
        +list_tuple_str_any_ _get_url_params()
        +str encode_params()
    }

    class HTTPRequestContext~TPag~ {
        +Context or None stream_context
        +TPag paginator
    }

    class JSONPayload {
    }

    class _HTTPStream~TToken~ {
        <<abstract>>
        +requests.Session requests_session
        +any authenticator
        +PreparedRequest build_prepared_request(method,url,params,headers,json,data)
    }

    class RESTStream~TToken~ {
        +str http_method
        +dict~str,str~ http_headers
        +bool payload_as_json
        +BaseAPIPaginator~TToken~ or None get_new_paginator()
        +HTTPRequest get_http_request(context:HTTPRequestContext)
        +PreparedRequest _prepare_request(context:Context or None,paginator:BaseAPIPaginator)
        +PreparedRequest prepare_request(context:Context or None,next_page_token:TToken or None)
        +JSONPayload prepare_request_payload(context:Context or None,next_page_token:TToken or None)
        +Iterable~dict~ request_records(context:Context or None)
    }

    class BaseAPIPaginator~TToken~ {
        <<abstract>>
        +TToken or None current_value
        +bool finished
        +void advance(response)
    }

    class LegacyStreamPaginator~TToken~ {
    }

    class JSONPathPaginator {
    }

    class SimpleHeaderPaginator {
    }

    class OffsetPaginator {
        +int page_size
    }

    HTTPRequestContext~TPag~ --> HTTPRequest : used_by_get_http_request
    RESTStream~TToken~ --> HTTPRequest : returns
    RESTStream~TToken~ --> HTTPRequestContext~BaseAPIPaginator~ : constructs

    _HTTPStream~TToken~ <|-- RESTStream~TToken~

    BaseAPIPaginator~TToken~ <|-- LegacyStreamPaginator~TToken~
    BaseAPIPaginator~TToken~ <|-- JSONPathPaginator
    BaseAPIPaginator~TToken~ <|-- SimpleHeaderPaginator
    BaseAPIPaginator~TToken~ <|-- OffsetPaginator

    RESTStream~TToken~ --> BaseAPIPaginator~TToken~ : get_new_paginator
    HTTPRequestContext~TPag~ --> BaseAPIPaginator~TToken~ : paginator
Loading

File-Level Changes

Change Details Files
Introduce HTTPRequest and HTTPRequestContext abstractions and integrate them into RESTStream request construction.
  • Add JSONPayload alias and HTTPRequest dataclass with url, method, headers, params, data, and safe_query_chars plus helpers to encode query parameters.
  • Add generic HTTPRequestContext carrying stream_context and a typed paginator instance for use when building requests.
  • Expose RESTStream.get_http_request method that builds an HTTPRequest from get_url, http_method, http_headers, get_url_params, and prepare_request_payload, using paginator.current_value for paging state.
  • Refactor RESTStream.prepare_request to delegate to a new _prepare_request helper that uses HTTPRequest.encode_params and conditionally sets json or data based on payload_as_json.
singer_sdk/streams/rest.py
Tighten paginator typing and wire the new HTTPRequest flow into the request loop with deprecation handling for prepare_request overrides.
  • Introduce generic _TPag bound to BaseAPIPaginator and use it across HTTPRequestContext and internal helpers.
  • Update RESTStream.get_new_paginator signature to return BaseAPIPaginator[_TToken] and refine concrete implementation return types to specific paginator subclasses.
  • Change request_records to detect any prepare_request override in the MRO, emit a SingerSDKDeprecationWarning, and fall back to the legacy prepare_request path; otherwise use the new _prepare_request/HTTPRequest flow.
  • Ensure request metrics and logging still operate on PreparedRequest objects built via build_prepared_request.
singer_sdk/streams/rest.py
singer_sdk/pagination.py
tests/core/rest/test_pagination.py
tests/core/rest/test_request_metrics.py
Migrate templates, example taps, and documentation from get_url_params to get_http_request-based pagination patterns.
  • Update cookiecutter tap REST client template to override get_http_request, modify request.params and optionally request.data, instead of get_url_params/prepare_request_payload.
  • Refactor meltano-tap-gitlab streams to override get_http_request, using HTTPRequestContext.paginator and stream_context instead of raw context/next_page_token and returning modified HTTPRequest objects.
  • Refactor meltano-tap-dummyjson client to set pagination params via get_http_request using OffsetPaginator values and page_size.
  • Update pagination and incremental replication docs, including HATEOAS and offset examples, to show get_http_request/HTTPRequestContext usage instead of get_url_params/parse_qsl patterns, and adjust porting guide text accordingly.
cookiecutter/tap-template/{{cookiecutter.tap_id}}/{{cookiecutter.library_name}}/rest-client.py
packages/meltano-tap-gitlab/tap_gitlab/streams.py
packages/meltano-tap-dummyjson/tap_dummyjson/client.py
docs/guides/pagination-classes.md
docs/incremental_replication.md
docs/guides/porting.md
docs/reference.rst
Add tests and reference docs for HTTPRequest behavior and prepare_request deprecation.
  • Add unit tests for HTTPRequest.encode_params covering custom safe_query_chars and non-string/iterable parameter encoding with doseq semantics.
  • Extend REST pagination tests to use HTTPRequest/HTTPRequestContext, including typed paginator contexts and a regression test ensuring prepare_request overrides on intermediate base classes trigger deprecation warnings in concrete streams.
  • Extend request metrics tests to work with get_http_request overrides and assert log argument types remain as expected.
  • Add API reference stubs for HTTPRequest and HTTPRequestContext so they appear in generated documentation.
tests/core/rest/test_request.py
tests/core/rest/test_pagination.py
tests/core/rest/test_request_metrics.py
docs/classes/singer_sdk.streams.rest.HTTPRequest.rst
docs/classes/singer_sdk.streams.rest.HTTPRequestContext.rst

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@edgarrmondragon edgarrmondragon added Type/Tap Singer taps HTTP HTTP based taps and targets such (REST, XML, etc.) labels Feb 12, 2026
@edgarrmondragon edgarrmondragon changed the title refactor: Update tap-dummyjson feat(taps): New RESTStream.get_http_request Feb 12, 2026
@codecov
Copy link

codecov bot commented Feb 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.43%. Comparing base (3cea93a) to head (ef52f17).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3483   +/-   ##
=======================================
  Coverage   94.43%   94.43%           
=======================================
  Files          72       72           
  Lines        5890     5926   +36     
  Branches      727      730    +3     
=======================================
+ Hits         5562     5596   +34     
- Misses        231      232    +1     
- Partials       97       98    +1     
Flag Coverage Δ
core 82.60% <100.00%> (+0.07%) ⬆️
end-to-end 76.05% <79.48%> (-0.13%) ⬇️
optional-components 43.60% <53.84%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq
Copy link

codspeed-hq bot commented Feb 12, 2026

Merging this PR will not alter performance

✅ 8 untouched benchmarks


Comparing feat/get_http_request (ef52f17) with main (3cea93a)

Open in CodSpeed

@read-the-docs-community
Copy link

read-the-docs-community bot commented Feb 12, 2026

Documentation build overview

📚 Meltano SDK | 🛠️ Build #31591821 | 📁 Comparing ef52f17 against latest (3cea93a)


🔍 Preview build

Show files changed (10 files in total): 📝 8 modified | ➕ 2 added | ➖ 0 deleted
File Status
genindex.html 📝 modified
incremental_replication.html 📝 modified
reference.html 📝 modified
classes/singer_sdk.GraphQLStream.html 📝 modified
classes/singer_sdk.RESTStream.html 📝 modified
classes/singer_sdk.pagination.BaseHATEOASPaginator.html 📝 modified
classes/singer_sdk.streams.rest.HTTPRequest.html ➕ added
classes/singer_sdk.streams.rest.HTTPRequestContext.html ➕ added
guides/pagination-classes.html 📝 modified
guides/porting.html 📝 modified

@edgarrmondragon edgarrmondragon changed the title feat(taps): New RESTStream.get_http_request feat(taps): New RESTStream.get_http_request method to construct the HTTP request for a stream Feb 12, 2026
…pagination

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
…headers

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
… in case developer is overridin `build_prepared_request()`

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
@edgarrmondragon edgarrmondragon force-pushed the feat/get_http_request branch 2 times, most recently from a1b8929 to 5f35372 Compare February 12, 2026 20:59
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
@edgarrmondragon edgarrmondragon added this to the v0.54 milestone Feb 12, 2026
github-merge-queue bot pushed a commit that referenced this pull request Feb 18, 2026
…rs` (#3498)

- Extracted from #3483

## Summary by Sourcery

Bug Fixes:
- Preserve the SDK's default User-Agent header when users override
http_headers so requests consistently include a User-Agent value.

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
…eam.get_http_request` (#3509)

SSIA

## Summary by Sourcery

Deprecate RESTStream.prepare_request in favor of
RESTStream.get_http_request and update pagination handling to operate on
paginator instances instead of raw page tokens.

New Features:
- Allow HTTPRequestContext to carry a paginator instance, enabling
richer pagination-aware request construction.
- Expose the page_size attribute on BaseOffsetPaginator for use in
request parameter building.

Enhancements:
- Adjust REST stream request construction to use paginator.current_value
via the new HTTPRequestContext paginator field.
- Introduce an internal helper to build PreparedRequest objects from
HTTPRequestContext while keeping backward compatibility with overridden
prepare_request implementations.
- Tighten typing on get_new_paginator and related HTTPRequestContext
generics to bind them to concrete paginator types across the SDK and
example taps.
- Update example taps and tests to construct HTTP requests using
paginator instances rather than raw next_page tokens.

Tests:
- Adapt REST pagination tests to the new HTTPRequestContext paginator
API and ensure backward compatibility paths are covered.

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
edgarrmondragon added a commit that referenced this pull request Feb 24, 2026
…n favor of `PageNumberPaginator` and `OffsetPaginator`, respectively (#3510)

These were made "base" classes with the expectation that users would
override them, but #1918 made that
redundant.

## Related

- #1918
- #2989
- #3483

## Summary by Sourcery

Deprecate the base paginator classes in favor of concrete
PageNumberPaginator and OffsetPaginator and update usages, tests, and
docs accordingly.

New Features:
- Introduce PageNumberPaginator and OffsetPaginator as the primary
paginator implementations for page-number and offset-based pagination.
- Add a shared singer_sdk_deprecated decorator helper preconfigured with
the SDK's deprecation warning type.

Bug Fixes:
- Tighten paginator-related tests by asserting concrete types and
cleaning up header handling in header-based pagination tests.

Enhancements:
- Mark BasePageNumberPaginator and BaseOffsetPaginator as deprecated
wrappers around the new paginator classes, exposing page_size as a
property on OffsetPaginator.
- Update REST stream examples, cookiecutter templates, and dummyjson tap
client to use the new paginator classes and type hints, including
explicit override annotations and typed Context usage.

Documentation:
- Revise pagination documentation and cookiecutter guidance to recommend
PageNumberPaginator and OffsetPaginator, and add reference stubs for
their API documentation.

Tests:
- Extend and adjust pagination tests to cover deprecation warnings for
the base paginator classes and to use the new PageNumberPaginator and
OffsetPaginator implementations.

## Summary by Sourcery

Deprecate legacy base pagination classes in favor of concrete
page-number and offset paginator implementations and update usages,
tests, and docs accordingly.

New Features:
- Introduce PageNumberPaginator and OffsetPaginator as the primary
page-number and offset-based paginator implementations.
- Add a shared singer_sdk_deprecated helper preconfigured with the SDK's
deprecation warning type for consistent deprecation messaging.

Bug Fixes:
- Tighten header-based and link-based pagination tests by asserting
concrete types and correcting header handling edge cases.

Enhancements:
- Mark BasePageNumberPaginator and BaseOffsetPaginator as deprecated
wrappers around the new paginator classes, exposing page_size as a
property on OffsetPaginator.
- Update REST stream examples, cookiecutter templates, and the dummyjson
tap client to use the new paginator classes, override annotations, and
typed Context parameters.

Documentation:
- Revise pagination guides and cookiecutter documentation to recommend
PageNumberPaginator and OffsetPaginator and add API reference stubs for
their classes.

Tests:
- Extend pagination tests to cover deprecation warnings for the base
paginator classes and to exercise the new concrete paginator
implementations.

---------

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
@edgarrmondragon edgarrmondragon marked this pull request as ready for review February 24, 2026 20:29
@edgarrmondragon edgarrmondragon requested review from a team as code owners February 24, 2026 20:29
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 5 issues, and left some high level feedback:

  • The new HTTPRequest.encode_params() assumes self.params is a mapping, which breaks the legacy get_url_params contract that allowed returning a pre-encoded str; consider either preserving the old prepare_request behavior for taps that only override get_url_params, or teaching HTTPRequest/get_http_request to handle a string params value for backward compatibility.
  • HTTPRequestContext requires a non-optional paginator type parameter (_TPag bounded to BaseAPIPaginator), but some usages conceptually allow None (e.g. unpaginated streams/tests); you may want to make paginator optional in the context or relax the generic bound so type hints accurately reflect valid runtime states.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The new `HTTPRequest.encode_params()` assumes `self.params` is a mapping, which breaks the legacy `get_url_params` contract that allowed returning a pre-encoded `str`; consider either preserving the old `prepare_request` behavior for taps that only override `get_url_params`, or teaching `HTTPRequest`/`get_http_request` to handle a string `params` value for backward compatibility.
- `HTTPRequestContext` requires a non-optional paginator type parameter (`_TPag` bounded to `BaseAPIPaginator`), but some usages conceptually allow `None` (e.g. unpaginated streams/tests); you may want to make `paginator` optional in the context or relax the generic bound so type hints accurately reflect valid runtime states.

## Individual Comments

### Comment 1
<location path="singer_sdk/streams/rest.py" line_range="102-110" />
<code_context>
+    #: ASCII characters that should not be quoted when encoding URL parameters.
+    safe_query_chars: str = ""
+
+    def _get_url_params(self) -> list[tuple[str, t.Any]]:
+        """Encode parameters as a list of tuples.
+
+        Returns:
+            A list of tuples of encoded parameters.
+        """
+        result: list[tuple[str, t.Any]] = []
+        for k, v in self.params.items():
+            values = [v] if not isinstance(v, list) else v
+            result.extend((k, v) for v in values if v is not None)
+        return result
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Broaden handling of multi-valued parameters beyond just `list`.

`_get_url_params` currently special-cases only `list` for multi-valued params. If a caller uses another sequence type like a `tuple`, it will be treated as a scalar and encoded via `str()`, leading to different behavior from other parts of the SDK that already accept tuples (e.g. `JSONPayload`). Consider treating any non-string `Sequence` (or at least adding `tuple`) as multi-valued to keep encoding consistent and avoid subtle issues for consumers passing tuples.

Suggested implementation:

```python
    def _get_url_params(self) -> list[tuple[str, t.Any]]:
        """Encode parameters as a list of tuples.

        Returns:
            A list of tuples of encoded parameters.
        """
        result: list[tuple[str, t.Any]] = []
        for k, v in self.params.items():
            if v is None:
                continue

            # Treat any non-string sequence as multi-valued (e.g. list, tuple).
            try:
                from collections.abc import Sequence  # type: ignore[import]
            except Exception:  # pragma: no cover - fallback if import location changes
                Sequence = tuple  # type: ignore[assignment]

            if isinstance(v, Sequence) and not isinstance(v, (str, bytes, bytearray)):
                values = v
            else:
                values = [v]

            result.extend((k, item) for item in values if item is not None)
        return result

```

To keep imports clean and avoid the fallback import inside the loop, you may want to:
1. Add `from collections.abc import Sequence` near the top of `rest.py` with the other imports.
2. After that, simplify the function by removing the `try/except` and using `Sequence` directly:
   - Remove the `try/except` block and `Sequence = tuple` fallback.
   - Use just: `if isinstance(v, Sequence) and not isinstance(v, (str, bytes, bytearray)):`.

If the codebase has a shared typing/import convention (e.g. already importing `Sequence` elsewhere), align this change with that convention instead of the inline import.
</issue_to_address>

### Comment 2
<location path="packages/meltano-tap-dummyjson/tap_dummyjson/client.py" line_range="63-74" />
<code_context>

     @override
-    def get_url_params(
+    def get_http_request(
         self,
-        context: Context | None,
-        next_page_token: int | None,
-    ) -> dict[str, int | None]:
-        return {
-            "skip": next_page_token,
-            "limit": PAGE_SIZE,
+        *,
+        context: HTTPRequestContext[OffsetPaginator],
+    ) -> HTTPRequest:
+        request = super().get_http_request(context=context)
+        request.params = {
+            "skip": context.paginator.current_value,
+            "limit": context.paginator.page_size,
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Avoid clobbering base query params when customizing pagination.

This override replaces `request.params` instead of extending it, so any defaults from the base class (or future additions) are lost. To keep base behavior while adjusting pagination, update the existing params instead, e.g. `request.params.update({"skip": ..., "limit": ...})`.

```suggestion
    @override
    def get_http_request(
        self,
        *,
        context: HTTPRequestContext[OffsetPaginator],
    ) -> HTTPRequest:
        request = super().get_http_request(context=context)

        # Ensure params is a dict so we can safely update it
        if request.params is None:
            request.params = {}

        request.params.update(
            {
                "skip": context.paginator.current_value,
                "limit": context.paginator.page_size,
            }
        )
        return request
```
</issue_to_address>

### Comment 3
<location path="packages/meltano-tap-dummyjson/tap_dummyjson/client.py" line_range="77-74" />
<code_context>

     @override
-    def post_process(
+    def prepare_request(
         self,
-        row: Record,
-        context: Context | None = None,
-    ) -> Record | None:
-        return row
+        context: Context | None,
+        next_page_token: int | None,
+    ) -> requests.PreparedRequest:
+        request = super().prepare_request(context, next_page_token)
+        if next_page_token is not None:
+            request.headers["X-Custom-Header"] = "value"
+        return request
</code_context>
<issue_to_address>
**issue:** Overriding `prepare_request` defeats the new `get_http_request` flow and triggers deprecation warnings.

With `request_records`, overriding `prepare_request` keeps you on the deprecated path and bypasses `_prepare_request`/`get_http_request`, so your `get_http_request` override is never used. To follow the new API and avoid deprecation warnings, move the header logic into `get_http_request` (e.g. `request.headers["X-Custom-Header"] = "value"`) and drop the `prepare_request` override.
</issue_to_address>

### Comment 4
<location path="tests/core/rest/test_request.py" line_range="6-15" />
<code_context>
+from singer_sdk.streams.rest import HTTPRequest
+
+
+def test_custom_safe_chars():
+    request = HTTPRequest(
+        method="GET",
+        url="https://example.com/api",
+        headers={"Authorization": "Bearer token"},
+        params={"key": "(abc)"},
+        safe_query_chars="()",
+    )
+
+    assert request.encode_params() == "key=(abc)"
</code_context>
<issue_to_address>
**suggestion (testing):** Broaden coverage of `HTTPRequest.encode_params` to include lists, `None` values, and default quoting behavior.

Consider expanding the tests to cover:

- Multiple params including some with `None` values, asserting `None` entries are omitted and ordering is preserved.
- A param whose value is a list (e.g., `{"key": ["a", "b"]}`) to verify `doseq=True` behavior.
- A call without `safe_query_chars`, asserting characters like `(` and `)` are percent-encoded.

This will exercise `_get_url_params`/`encode_params` across key edge cases.
</issue_to_address>

### Comment 5
<location path="docs/guides/pagination-classes.md" line_range="43-45" />
<code_context>
+    def get_http_request(self, *, context):
+        request = super().get_http_request(context=context)
+        if context.paginator.current_value:
+            # Next page token is a URL, so we can to set the request URL directly
+            request.url = context.paginator.current_value.geturl()
+        return request
</code_context>
<issue_to_address>
**issue (typo):** Fix small grammar issue in this comment text.

"so we can to set" is ungrammatical; please change it to "so we can set the request URL directly".

```suggestion
        if context.paginator.current_value:
            # Next page token is a URL, so we can set the request URL directly
            request.url = context.paginator.current_value.geturl()
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

edgarrmondragon and others added 2 commits February 24, 2026 14:35
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • The HTTPRequestContext generic is bounded to BaseAPIPaginator, but several usages (e.g. HTTPRequestContext[None] in tests) violate that bound and will upset static type checkers; consider either loosening the bound or updating those call sites to use a paginator-compatible type parameter.
  • In EpicIssuesStream.get_http_request, you now read context.stream_context directly; it might be worth adding a short comment or helper to make it clearer that stream_context can be None and that the method assumes an epic_id-bearing context is always provided for this stream.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The `HTTPRequestContext` generic is bounded to `BaseAPIPaginator`, but several usages (e.g. `HTTPRequestContext[None]` in tests) violate that bound and will upset static type checkers; consider either loosening the bound or updating those call sites to use a paginator-compatible type parameter.
- In `EpicIssuesStream.get_http_request`, you now read `context.stream_context` directly; it might be worth adding a short comment or helper to make it clearer that `stream_context` can be `None` and that the method assumes an `epic_id`-bearing context is always provided for this stream.

## Individual Comments

### Comment 1
<location path="docs/guides/pagination-classes.md" line_range="44" />
<code_context>
-        return params
+    def get_http_request(self, *, context):
+        request = super().get_http_request(context=context)
+        if context.paginator.current_value:
+            # Next page token is a URL, so we can to set the request URL directly
+            request.url = context.paginator.current_value.geturl()
+        return request
</code_context>
<issue_to_address>
**issue (typo):** Fix the grammatical error "can to" in this comment (appears twice in this file).

The phrase is currently "so we can to set the request URL directly"; update it to "so we can set the request URL directly" in both occurrences in this file.

```suggestion
            # Next page token is a URL, so we can set the request URL directly
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • In request_records, the deprecation check for prepare_request is performed on every pagination loop iteration, which will emit a warning per page; consider moving the type(self).prepare_request is not _HTTPStream.prepare_request check (and warning) outside the loop so each stream only warns once per sync.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `request_records`, the deprecation check for `prepare_request` is performed on every pagination loop iteration, which will emit a warning per page; consider moving the `type(self).prepare_request is not _HTTPStream.prepare_request` check (and warning) outside the loop so each stream only warns once per sync.

## Individual Comments

### Comment 1
<location path="tests/core/rest/test_pagination.py" line_range="578-587" />
<code_context>
+def test_prepare_request_override_emits_deprecation(tap: Tap):
</code_context>
<issue_to_address>
**suggestion (testing):** Also assert on the returned records to ensure the deprecated path still behaves correctly functionally.

You can mirror the pattern from `test_prepare_request_override_in_intermediate_class_emits_deprecation`, e.g.:

```python
with pytest.warns(SingerSDKDeprecationWarning, match="prepare_request"):
    records = list(stream.request_records(context=None))

assert records == [{"id": 1}]
```

This ensures the deprecated path not only warns but still issues the request and parses the response correctly.

Suggested implementation:

```python
def test_prepare_request_override_emits_deprecation(tap: Tap):
    """Validate that overriding prepare_request emits a deprecation warning."""

    class MyAPIStream(RESTStream):
        name = "my-api-stream"
        url_base = "https://my.api.test"
        path = "/path/to/resource"
        schema: t.ClassVar[dict] = {
            "type": "object",
            "properties": {"id": {"type": "integer"}},
        }

    stream = MyAPIStream(tap=tap)

    with pytest.warns(SingerSDKDeprecationWarning, match="prepare_request"):
        records = list(stream.request_records(context=None))

    assert records == [{"id": 1}]

```

This change assumes:
1. `pytest` and `SingerSDKDeprecationWarning` are already imported in `tests/core/rest/test_pagination.py`. If not, add them to the imports at the top of the file.
2. The test or shared fixtures already configure the HTTP response for `MyAPIStream` so that `request_records` yields a single record `{"id": 1}`. If not, mirror the response mocking setup used in `test_prepare_request_override_in_intermediate_class_emits_deprecation` so that the functional assertion on `records` passes.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Signed-off-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
@edgarrmondragon
Copy link
Collaborator Author

@dependabot rebase

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

HTTP HTTP based taps and targets such (REST, XML, etc.) Type/Tap Singer taps

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant