NMS-19751: Support for Token Based Authentication with XML/JSON Collector (and HTTP collector)

[marshallmassengill]

This adds support for token-based authentication against external HTTP APIs during data collection. Operators describe an API login flow once in etc/token-auth-configuration.xml and reference the resulting token from any XML/JSON or HTTP collector header via the ${token:} metadata placeholder. The runtime performs the login call, caches the token, refreshes on TTL, invalidates on 401/403, and supports reload via the standard daemon-reload event.

The motivating use case is vendor APIs like Cisco Catalyst Center, F5 BIG-IP, and VMware vSphere, where the collectors themselves can't perform a login flow.

What's in here:

• JAXB model + XSD for token-auth-configuration.xml with three token-extraction modes (jsonpath, response header, body-as-token)
• TokenAcquirer, TokenCache, TokenProviderImpl runtime, plus a TokenAuthCollectorAdaptor that resolves ${token:} placeholders on the controller before the collector dispatch
• Two adaptor entry points so ${token:...} works wherever auth headers live:
• beforeCollect substitutes inside the basic collection-parameter map (e.g. collectd-configuration.xml values)
• beforeRuntimeInterpolation substitutes inside the JAXB-deserialized runtime-attribute tree (e.g. xml-datacollection-config.xml values, http-datacollection-config.xml values). Mate's standard pass then resolves the remaining ${node:...}, ${scv:...}, ${env:...} etc. without touching core/mate.
• Per-node placeholder support inside blocks: ${node:label}, ${scv:alias:key}, ${env:NAME} all resolve before the auth call. Cache identity widens to (auth-name, SHA-256 fingerprint of resolved fields) so one auth definition can serve many regional endpoints with one cache entry per distinct resolution.
• Early-refresh window of max(600s, 5% of ) on the cache so the controller never hands back a token within ten minutes of upstream expiry.
• Passive invalidation on 401/403: the failing request isn't retried in-cycle; the adaptor invalidates the matching cache entry, and the next scheduled collection re-acquires.
• Reload listener: opennms:send-event -p daemonName=TokenAuth uei.opennms.org/internal/reloadDaemonConfig flushes the cache and re-reads the file; reload failures publish reloadDaemonConfigFailed and leave the previous config in effect.
• Custom block on in http-datacollection-config.xml so Authorization: Bearer ${token:foo} works in the HTTP collector too.
• Reference docs at Operation → Deep Dive → Administration → Configuration → Collector Token Authentication (docs/.../token-authentication.adoc).

Validated end-to-end against Hashicorp Vault (auth/client_token JSON), PocketBase (superuser token JSON), and WireMock — including a Cisco Catalyst Center-style stub that exchanges HTTP Basic for a
Token JSON and gates data endpoints behind X-Auth-Token. Each test exercises SCV-resolved credentials plus per-node metadata in the auth definition (URL path, basic-auth, request body, and request
headers).

I'd love feedback on this. We were asked by a potential customer the other day about our capability to scrape some specific endpoints that use this style of authentication and I think this fills a
pretty big gap for us in the XML/JSON Collector today.

Assisted-by: Claude Code:Opus 4.7

[marshallmassengill]

So this is passing except for the WmiPeerFactoryTest in CircleCI as far as I can tell. There were indeed some integration test failures that I needed to fix. They should be now.

[dino2gnt]

So this is passing except for the WmiPeerFactoryTest in CircleCI as far as I can tell.

The WmiPeerFactoryTest is failing to instantiate the EntityScopeProvider, which seems like it could be related to your metadata changes.

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.context.support.ClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityScopeProvider' defined in class path resource [test-dao-context.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.*******.core.mate.api.EntityScopeProvider]: Specified class is an interface

[marshallmassengill]

I'll give it a look again.

[marshallmassengill]

Integration tests all seem to be passing now. There were definitely a couple things in there. I've made a note to myself to run those locally first now.

[dino2gnt]

Smooth-brain thought: Couldn't you accomplish this same thing using a shell script in cron that does a curl and feeds the token response to scvcli?

[marshallmassengill]

Smooth-brain thought: Couldn't you accomplish this same thing using a shell script in cron that does a curl and feeds the token response to scvcli?

I think that can/will work... but there are some misses to me... It could definitely cover a single long lived token use case... I can't argue that really but I don't think it covers the shorter lived ones as nicely.

That being said, I think the biggest issue is that any 401/403 recoveries have to happen on the cron schedule and can't happen in-line with a response (granted, I guess you could use the failed collection in OpenNMS and have it do that with scripting but then you've got 3 things to mangle).

There is also something to be said that this works with metadata (and SCV) whereas cron wouldn't so you could do this across a fleet of systems instead of having to probably create a separate script for each one and having to manage other secure vault stuff.

I also think this keeps everything in OpenNMS and that wayt tokens live in memory (RAM) as a result so I think that fits with the idea of tokens being short lived and not meant to be permanent (semi-permanent?) and stored in a vault.

[Copilot]

Pull request overview

Adds a dynamic-auth runtime to OpenNMS that can acquire/cache/refresh tokens from external HTTP login flows defined in etc/auth-configuration.xml, expose them via the metadata DSL ${auth:<name>}, and enable collector retry-on-401/403 behavior (XML/JSON collectors and HTTP collector headers).

Changes:

• Introduces auth-configuration.xml schema/model, parsing/validation, token acquisition and caching runtime, and a reload listener to re-read config + flush cache.
• Wires ${auth:<name>} into the metadata DSL via a new AuthScope, TokenProvider API, and EntityScopeProvider integration.
• Adds collector-side support: HTTP collector <headers> block, and 401/403 retry logic in XML/JSON collectors (with token invalidation + refresh).

Reviewed changes

Copilot reviewed 51 out of 52 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
protocols/xml/src/test/java/org/opennms/protocols/xml/collector/RefreshAuthTokensTest.java	Adds unit tests for request-header token refresh behavior on retry.
protocols/xml/src/main/java/org/opennms/protocols/xml/collector/AbstractXmlCollectionHandler.java	Adds 401/403 retry-once path and token refresh/invalidation logic for XML collector requests.
protocols/xml/src/main/java/org/opennms/protocols/json/collector/AbstractJsonCollectionHandler.java	Adds 401/403 retry-once path using shared token refresh logic for JSON collector requests.
protocols/xml/src/main/java/org/opennms/protocols/http/HttpUrlConnection.java	Throws typed AuthFailureException on 401/403 to enable auth-aware retry upstream.
protocols/xml/src/main/java/org/opennms/protocols/http/AuthFailureException.java	Adds typed exception carrying HTTP status for auth failures.
opennms-config/src/test/resources/JUnitHttpServer.keystore	Adds test keystore for HTTPS acquisition tests.
opennms-config/src/test/java/org/opennms/netmgt/config/auth/TokenProviderImplTest.java	Unit tests for TokenProviderImpl behaviors.
opennms-config/src/test/java/org/opennms/netmgt/config/auth/TokenCacheTest.java	Unit tests for token caching, expiry, invalidation, and scope-fingerprinting behavior.
opennms-config/src/test/java/org/opennms/netmgt/config/auth/TokenAcquirerTest.java	Tests token acquisition and extraction modes (jsonpath/header/body-as-token) plus placeholder interpolation.
opennms-config/src/test/java/org/opennms/netmgt/config/auth/TokenAcquirerHttpsTest.java	Verifies disable-SSL-verification behavior against self-signed certs.
opennms-config/src/test/java/org/opennms/netmgt/config/auth/AuthRuntimeIntegrationTest.java	Integration test covering the full dynamic-auth stack composition.
opennms-config/src/test/java/org/opennms/netmgt/config/auth/AuthConfigReloadListenerTest.java	Tests daemon reload event handling and cache flush behavior.
opennms-config/src/test/java/org/opennms/netmgt/config/AuthConfigFactoryTest.java	Tests parsing/validation rules for auth-configuration.xml.
opennms-config/src/main/resources/META-INF/opennms/component-dao.xml	Imports auth runtime Spring context and registers auth-configuration.xml as a JAXB config resource.
opennms-config/src/main/resources/META-INF/opennms/applicationContext-auth.xml	Wires TokenAcquirer/TokenCache/TokenProvider + OSGi service + reload listener.
opennms-config/src/main/java/org/opennms/netmgt/config/auth/TokenProviderImpl.java	Implements TokenProvider backed by AuthConfigFactory + TokenCache.
opennms-config/src/main/java/org/opennms/netmgt/config/auth/TokenCache.java	Adds in-memory token cache keyed by (authName, resolved fingerprint), plus reverse-lookup invalidation.
opennms-config/src/main/java/org/opennms/netmgt/config/auth/TokenAcquirerScopeBinder.java	Late-binds EntityScopeProvider onto TokenAcquirer when available to avoid wiring cycles.
opennms-config/src/main/java/org/opennms/netmgt/config/auth/TokenAcquirer.java	Implements HTTP login flow execution, placeholder interpolation, token extraction, and fingerprinting.
opennms-config/src/main/java/org/opennms/netmgt/config/auth/CachedToken.java	Introduces token value + optional expiry container for caching.
opennms-config/src/main/java/org/opennms/netmgt/config/auth/AuthConfigReloadListener.java	Implements daemon-reload listener that reloads auth config and flushes token cache.
opennms-config/src/main/java/org/opennms/netmgt/config/AuthConfigFactory.java	Adds singleton config loader/validator for auth-configuration.xml.
opennms-config/pom.xml	Adds runtime deps needed for auth acquisition (httpclient, core.web wrapper, jackson-databind).
opennms-config-tester/src/test/java/org/opennms/netmgt/config/tester/ConfigTesterTest.java	Ensures config-tester validates auth-configuration.xml.
opennms-config-tester/src/main/resources/META-INF/opennms/applicationContext-configTester.xml	Wires AuthConfigFactory for config-tester context.
opennms-config-model/src/test/java/org/opennms/netmgt/config/httpdatacollection/HttpDatacollectionConfigTest.java	Updates HTTP DC model test to include new <headers> element.
opennms-config-model/src/test/java/org/opennms/netmgt/config/auth/AuthConfigurationTest.java	Adds JAXB/XSD round-trip tests for new auth config model.
opennms-config-model/src/main/resources/xsds/http-datacollection-config.xsd	Adds <headers> block support under <url> for HTTP collector.
opennms-config-model/src/main/resources/xsds/auth-configuration.xsd	Adds schema for auth-configuration.xml.
opennms-config-model/src/main/java/org/opennms/netmgt/config/httpdatacollection/Url.java	Adds headers list to HTTP DC Url model.
opennms-config-model/src/main/java/org/opennms/netmgt/config/httpdatacollection/Header.java	Adds new HTTP DC header model type.
opennms-config-model/src/main/java/org/opennms/netmgt/config/auth/package-info.java	Defines JAXB namespace for new auth config model package.
opennms-config-model/src/main/java/org/opennms/netmgt/config/auth/TokenFrom.java	Adds token extraction configuration model.
opennms-config-model/src/main/java/org/opennms/netmgt/config/auth/Header.java	Adds auth-request header model type.
opennms-config-model/src/main/java/org/opennms/netmgt/config/auth/Content.java	Adds auth-request content/body model type.
opennms-config-model/src/main/java/org/opennms/netmgt/config/auth/BasicAuth.java	Adds basic-auth model for auth request.
opennms-config-model/src/main/java/org/opennms/netmgt/config/auth/AuthConfiguration.java	Adds top-level auth configuration model.
opennms-config-model/src/main/java/org/opennms/netmgt/config/auth/Auth.java	Adds per-auth definition model (URL/method/headers/content/token-from/TTL/flags).
opennms-base-assembly/src/main/filtered/etc/auth-configuration.xml	Adds default stub config file shipped in base assembly.
integration-tests/config/src/test/java/org/opennms/netmgt/config/WillItUnmarshalIT.java	Adds unmarshal IT coverage for auth-configuration.xml.
features/collection/collectors/src/test/java/org/opennms/netmgt/collectd/HttpCollectorHeaderTest.java	Adds focused unit tests for applying configured headers to outgoing HTTP requests.
features/collection/collectors/src/main/java/org/opennms/netmgt/collectd/HttpCollector.java	Applies configured URL headers to outgoing requests via new helper.
core/mate/model/src/main/java/org/opennms/core/mate/model/EntityScopeProviderImpl.java	Wires optional TokenProvider to provide AuthScope in entity scope chains.
core/mate/api/src/test/java/org/opennms/core/mate/api/AuthScopeTest.java	Adds unit tests for AuthScope behavior.
core/mate/api/src/test/java/org/opennms/core/mate/api/AuthScopeInterpolatorTest.java	Adds tests confirming ${auth:...} flows through Interpolator with AuthScope.
core/mate/api/src/main/java/org/opennms/core/mate/api/TokenProvider.java	Introduces TokenProvider API used by metadata DSL auth namespace.
core/mate/api/src/main/java/org/opennms/core/mate/api/Interpolator.java	Tracks calling scope stack for auth placeholder resolution to inherit per-node context.
core/mate/api/src/main/java/org/opennms/core/mate/api/EntityScopeProvider.java	Adds default getScopeForAuth() hook for ${auth:...} support.
core/mate/api/src/main/java/org/opennms/core/mate/api/AuthScope.java	Adds new metadata DSL scope implementation for ${auth:<name>}.
core/lib/src/main/java/org/opennms/core/utils/ConfigFileConstants.java	Registers auth-configuration.xml in config file constants mapping.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[marshallmassengill]

@cgorantla I added the minion 401/403 feature. Wasn't too bad to add but it does alter the HTTPCollector a bit given that those need to be retries but I could not come up with a reason someone would be using that collector for testing that and there is an alternative through PSM. Tests all seem to be passing. Let me know about feedback. I can adjust accordingly.

[cgorantla]

Reviewed few modules. Need to actually validate this by running this code.

[marshallmassengill]

Converted to draft for now. Working on a revised plan based on the feedback.

[cgorantla]

I have added few more comments on what I think should be considered.

[marshallmassengill]

This PR has been rewritten to address issues raised:

• "Don't update core modules like metadata and RPC collections for the small benefit" — no changes to core/mate/api/Interpolator, EntityScopeProviderImpl, CollectorClientRpcModule, or CollectorResponseDTO.
• "Consider similar thing like ServiceMonitorAdaptor for collectors" — new CollectorAdaptor interface in features/collection/api modeled on ServiceMonitorAdaptor. Token-auth logic lives in a single CollectorTokenAuthAdaptor class that runs on the OpenNMS controller (never on the Minion).
• "BeanUtils.getBean will not work on Minion" — token resolution happens on the controller before RPC dispatch. The Minion sees already-substituted parameter values; no token-aware code on the Minion side.
• "Interpolator ThreadLocal recursive lookup — clever but not intuitive" — removed entirely. The auth runtime no longer participates in the metadata DSL scope chain.
• "ApplicationListener doesn't fit OpenNMS's multi-context model" — replaced with standard Spring DI of EventIpcManager plus an init-method, same pattern as Collectd/Statsd/etc.
• "TokenAcquirerScopeBinder is a hacky workaround" — class deleted; the binding cycle it papered over is gone after AuthScope removal.

Other changes:

• Feature renamed to "Collector Token Authentication" to match the docs page title. Placeholder syntax is now ${token:}. Config file is etc/collector-token-auth-configuration.xml. Daemon-reload name is CollectorTokenAuth.
• PR base changed from develop to release-36.x
• Passive cache invalidation on the adaptor: when the controller sees a FAILED CollectionSet whose request used a token, it invalidates that cache entry. Next cycle re-acquires naturally. No retry orchestration.

I'm going to kick off a copilot review @cgorantla but will wait to address anything until you've looked this over.

I also still need to run integration tests locally.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated 17 comments.

[Copilot]

Pull request overview

Copilot reviewed 46 out of 47 changed files in this pull request and generated 6 comments.

[cgorantla]

Suggested few comments, I still need to go through whole code one more time

It is difficult to go through the code as there is too much commenting for every method.

Only add comments that are strictly necessary.

Also lot of places, package naming is expanded like java.util.List<org.opennms.netmgt.collection.api.CollectorAdaptor> which should be avoided.

[cgorantla]

Looks like we are adding tests for every source file.
Not sure if all of them are worth it. May be worth looking at reducing them.

[Copilot]

Pull request overview

Copilot reviewed 50 out of 51 changed files in this pull request and generated 4 comments.

[cgorantla]

One improvement we can do otherwise LGTM!

[cgorantla]

There is one more bug that might still be there.

[cgorantla]

LGTM!