DoT issue with multiple Internet connections

[romainl63]

Hi,

I noticed something that prevents me from using DoH properly on a corporate network.
I have a router/firewall with two fiber Internet connections on it.
If I put my two Internet connections in load balancing mode, DoH to the forwarders doesn't work very well, because I understand that it opens an encrypted tunnel between Technitium and the resolver in order to go faster.
I have the same problem if I put one Internet connection active and the other on standby. If the main link goes down, DNS resolution in DoT makes errors.
Every time this happens, my various Technitium servers are lost and make errors continuously until I restart them so that they relaunch their connections via my other Internet link. Basically, my entire network goes down completely, and no DNS resolution works, even if the entries are cached.
To solve the problem, I know I can force my various Technitiums to go through a single fiber connection and manage that way, but this is not ideal for load balancing and maintaining activity.
The classic DNS mode on port 53 does not pose any particular problems since there is no control.

If you have any ideas for solving this problem technically, perhaps we need a way to declare our various public IP addresses in Technitium so that it can establish a connection at the DoH level when an internet connection is no longer working. (At least for active/passive connections; I have little hope for load balancing with DoH technology.)

Regards

[ShreyasZare]

Thanks for the post. I am assuming that the load balancing is statefull. The DoH upstream connection is a TCP connection over port 443 to the upstream so it should work well with a statefull load balancer. So, not sure why it is causing an issue in this mode.

With failover mode for Internet connection, when one connection goes down, an existing TCP connection to the upstream should timeout in 16 sec. This is done using TCP keep alive feature which you can see here.

So, the DoH will try to reconnect to the upstream when the keep alive finds that the existing connection is broken. The reconnection attempt should go through your fallback connection since that is active. There too I am not sure why it would not go through immediately. I feel that the load balancer is causing the new connection attempt go the the older connection again for a while and this only stops occurring after a few mins.

For DNS over standard UDP transport, being a stateless protocol is probably the reason why it is not affected by this issue.

Basically, my entire network goes down completely, and no DNS resolution works, even if the entries are cached.

Do you have Serve Stale option enabled in Settings > Cache tab? If not then do enable it since this will ensure that the DNS server responds with expired data in cache in such cases. For cached data that is not expired, it should work even if the DNS server goes offline. You can test it with DNS Client tab on the admin panel during such event. It could be that the clients too are facing similar issues with their TCP connections and thus it looks like that the DNS is not responding. Again not sure about it but do check if something like this is occurring by querying using the DNS Client tool or using nslookup command from any of the clients in the network during such event.

[romainl63]

Thank you for your reply. I'm really sorry, I made a mistake. I'm not talking about DoH but DoT (DNS-over-TLS) on port 853. (Does that change anything for Technitium?)
Normally, my firewall is supposed to maintain connections on the gateway where they were started (in the case of active/passive, if the main link comes back, the connections still remain on the backup).
In the case of load balancing, it's “weighted round-robin” on the gateways, and I can adjust session persistence settings if necessary.
I have the “Serve Stale” option enabled on my entire Technitium cluster.
I ran some rough tests by disabling the gateway network cards at the firewall level to see how Technitium would behave, which responded with “server failure” for several minutes until it restarted. During that time, it was impossible to access our company websites, which I am 100% sure are cached.
I also tried to enable load balancing with specific SD-WAN routes for the machines hosting Technitium, but encountered the same problem when a link crashed.
I would need to conduct more in-depth tests, but this is complicated in production.

[ShreyasZare]

The DoT client implementation also works the same way and has the same TCP keep alive set. Note that DoT uses DNS-over-TCP client with just with added SSL layer so it uses the same code base and works the same.

So, not sure why it would give this issue. The DoT client will detect connection breakage and will attempt to make a new TCP connection on port 853. I guess that the new connection packets are not getting routed to the fallback/second WAN connection for some reason.

I have the “Serve Stale” option enabled on my entire Technitium cluster.
I ran some rough tests by disabling the gateway network cards at the firewall level to see how Technitium would behave, which responded with “server failure” for several minutes until it restarted. During that time, it was impossible to access our company websites, which I am 100% sure are cached.

The DNS server will respond with ServerFailure only when it does not have an answer in its cache. If there is a record in cache with 1 day TTL for a domain name, you will be able to resolve the record for the domain name for the next 4 days (+3 due to Serve Stale default TTL) even if the DNS server did not have access to the internet. You need to test this using DNS tools and not with web browsers to get correct results. Which is why I suggested earlier to use either the DNS Client tool on the panel or nslookup command from the client on the network. You can also use the Cache tab on the admin panel to look for a records for a domain name and see their TTL and try to resolve them when DNS server is offline just so that you can be sure about it.