Socks server can become unresponsive after a while

[C-Pro]

At least mine definitely was.
Logs are full of this stuff when I try to connect:

2019/12/25 10:45:49 [ERR] socks: Failed to handle request: Connect to 2001:xxx:4e8:f004::b:443 failed: dial tcp [2001:xxx:4e8:f004::b]:443: connect: cannot assign requested address
2019/12/25 10:45:49 [ERR] socks: Failed to handle request: Connect to 2001:xxx:4e8:f004::b:443 failed: dial tcp [2001:xxx:4e8:f004::b]:443: connect: cannot assign requested address
2019/12/25 10:45:51 [ERR] socks: Failed to handle request: Connect to 2001:xxx:4e8:f004::b:443 failed: dial tcp [2001:xxx:4e8:f004::b]:443: connect: cannot assign requested address
2019/12/25 10:45:51 [ERR] socks: Failed to handle request: Connect to 2001:xxx:4e8:f004::b:443 failed: dial tcp [2001:xxx:4e8:f004::b]:443: connect: cannot assign requested address

Looks like if we reached the limit of open file descriptors or something. It can mean we do not cleanup them properly of fast enough.