decrypt error while trying to connect periphery to core (outbound)

[diegovilar]

I'm trying to connect a Periphery agent to Core (both using docker).

Configured PERIPHERY_CORE_ADDRESS, PERIPHERY_ONBOARDING_KEY and PERIPHERY_CONNECT_AS on Periphery side.

Core is behind a reverse proxy (Traefik), both https and http (for testing only).

When I start Periphery, I get this on the it's logs:

INFO PeripheryStartup: Komodo Periphery version: v2.1.2
(...)
INFO PeripheryStartup: Public Key: MCow**********************=
INFO PeripheryStartup:StartCoreConnection{address="https://komodo.lab"}: Initiating outbound connection to wss://komodo.lab/ws/periphery?server=sesrv77
DEBUG [CORE AUTH] Zero trust identifiers host="komodo.lab" query="server=sesrv77" sec_websocket_accept="M8qeoqppLi0A4nnHK5UEWQ32B2M="
ERROR Onboarding failed: [Client] Failed to read handshake_m2: decrypt error

And on Core's logs:

DEBUG Server sesrv77 failed to onboard | [Server] Failed to get handshake_m3: Decoded error message over Core-Periphery communication channel: [Client] Failed to read handshake_m2: decrypt error

Tried to use http instead of https to connect to Core, but the same thing happens.

Ideas, anyone?

[mbecker20]

Some address mismatch, seeing both the .lab and .home addresses in those logs, just use one

[diegovilar]

Some address mismatch, seeing both the .lab and .home addresses in those logs, just use one

That was actually a typo while creating the issue here. That's not the real address. I just didn't want to make it public. The address is correct and so are the certificates. I don't think that's the problem.

[mbecker20]

The address should match the Core configured host as well. The zero trust identifiers need to be the same on both sides for decrypt to pass

[diegovilar]

So, explicitly setting the hostname of the Core's container to match the host's should do it?

[mbecker20]

Not the container hostname, the host configuration field

[diegovilar]

Not the container hostname, the host configuration field

On the Periphery side, the env var PERIPHERY_CORE_ADDRESS is set to the exact same value as KOMODO_HOST env var on the Core side, protocol and everything.

[diegovilar]

I appreciate you trying to help, @mbecker20.

Gonna try and draw a more detailed picture bellow.

# Core VM
inbound trafic -> host -> traefik (docker swarm) -> core container

Traefik is rounting 80 (just for testing) and 443 to Core's port: 9120. The VM's host is "manager.company.com", but our DNS has aliases for it: "ops.company.com" and wildcard "*.ops.company.com"

Traefik proxies http[s]://komodo.ops.company.com to the core's container on port 9120.

Trying to connect a Periphery Agent on another VM using outbound mode:

# Periphery VM
outbound trafic <- host <- (docker) <- periphery container

In this setup, doesn't matter if I use "http://komodo.ops.company.com" or "https://komodo.ops.company.com" to reach core, it reaches, but i get this on Peripherys logs:

ERROR Onboarding failed: [Client] Failed to read handshake_m2: decrypt error

and this on core´s logs:

DEBUG Server node-77.company.com failed to onboard | [Server] Failed to get handshake_m3: Decoded error message over Core-Periphery communication channel: [Client] Failed to read handshake_m2: decrypt error

If I map core's port to the host and try connecting periphery through "http://komodo.ops.company.com:9120", it works. I can actually use any hostname that reaches the core container (ops.company.com:9120, manager.company.com:9120, whatever.company.com:9120...). The host configuration (KOMODO_HOST) doesn't seem to matter here. I even commented it out now for testing...

So, it seems Traefik is getting in the way somehow. Using http, Traefik would not be doing termination since there is no TLS involved, but it seems it still messes with the messages somehow and breaks whatever is expected on the other end. Bypass traefik and it just works.

The thing is, while I'm testing using periphery on another VM that has lateral access to the VM core is on, and so I can test other ports, that will not work on our real infra: all other nodes will only be able to reach core through HTTPS on 443, and Traefik is the one listening there, so there is no way around that...

I tried making core contact periphery, but the problem persists since periphery instances will themselves be on docker and reachable only through their node's own traefik instances, so... and many nodes will just not be reachable at all from core (hence outbound mode being needed here).

Bottom line: out to find out what the heck is traefik doing to the messages...

[mbecker20]

Enable debug logs on Core so you can see what zero trust identifiers it gets, needs to match periphery debug log