Terribly slow curl (TLS request) performance inside Docker

Technology Tech Support
1

Hello Guys!

The issue I am looking for help with, together with some troubleshooting steps alredy taken is described at r/docker, but it doesn’t seem to be interesting enough to get any attention :wink:

My current setup:

  • PiHole running in Docker ( official image pihole/pihole:latest )
  • Docker home dir is on mdadm raid1 )
  • PiHole home and configs ( /etc/pihole/ and /etc/dnsmasq.d directories) are located on mirror ZFS dataset (different physical device than Docker storage backend)
  • host is HP DL380e G8 with 2x E5-2450L and 64 GB of RAM, so I doubt HW performance is the issue
  • host is running Debian Testing

Effect of the issue:

root@1068cc07c7b9:/# curl https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -vvv
* Expire in 0 ms for 6 (transfer 0x55762eb3aec0)
[ ... lots of those ... ]
* Expire in 14 ms for 1 (transfer 0x55762eb3aec0)
*   Trying 151.101.0.133...
* TCP_NODELAY set
* Expire in 149978 ms for 3 (transfer 0x55762eb3aec0)
* Expire in 200 ms for 4 (transfer 0x55762eb3aec0)
* Connected to raw.githubusercontent.com (151.101.0.133) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* Operation timed out after 300216 milliseconds with 0 out of 0 bytes received
* Closing connection 0
curl: (28) Operation timed out after 300216 milliseconds with 0 out of 0 bytes received

curl times out (randomly, there are times that it passes). It never timeouts inside a container in host networking mode.

What I tried so far:

  • tried curling the blocklist from pihole container /bin/bash. Went as slow as gravity update.
  • tried different container - same case.
  • tried curling the blocklist from container with host network mode - blazingly fast
  • changing to various MTU - especially setting lower MTU for docker interface then the main ethernet MTU → didn’t help
  • I made sure I have installed docker from official docker repository
  • I flushed hosts iptables rules
  • I checked the connection from different machines in the same network to exclude network firewall/routing → excluded

What surprises me the most is that another instance of PiHole set up and maintained in the same way (though on different host) works perfectly fine and snappy

Have you ever faced similar issue or have seen it anywhere? Do you maybe have any ideas what might be causing this behavior?

2

Fixed it! Turned out that MTU was all good on my docker host, but the VPN server had unusually low MTU on vpn interface. The second instance worked well because the VPN wasn’t the default gateway…

1 Like