A while back (possibly after an update is the theory) I started to experience proxmox servers going down hard at our datacenter, no response on network, and the only thing that would get them back was a hard reset. This would happen up to several times a day at seemingly random!
After investigating the logs, these stood to me:
Especially line #7
... Aug 9 22:10:36 vm6 kernel: [613016.212489] e1000e 0000:00:1f.6 enp0s31f6: Reset adapter unexpectedly Aug 9 22:10:36 vm6 kernel: [613016.212596] vmbr0: port 1(enp0s31f6) entered disabled state Aug 9 22:10:40 vm6 kernel: [613020.202972] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx Aug 9 22:10:40 vm6 kernel: [613020.203105] vmbr0: port 1(enp0s31f6) entered blocking state Aug 9 22:10:40 vm6 kernel: [613020.203183] vmbr0: port 1(enp0s31f6) entered forwarding state Aug 9 22:10:42 vm6 kernel: [613022.133940] e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:0 ...
The issue seems that after a kernel/driver update, the interface going into a perpetual hang state, and crashes..
The solution i have found to work across multiple different servers is using the “ethtool” to set some values to prevent this from happening.*
Disable the following:
- GSO (generic-segmentation-offload)
- GRO (generic-receive-offload)
- TSO (tcp-segmentation-offload)
- TX (tx-checksumming)
- RX (rx-checksumming)
ethtool -K <INTERFACE> gso off gro off tso off tx off rx off
and also disable “pcie power saver”:
Create a script to automate this for us
Since this has to be done for the WAN facing interfaces (not loopback etc) and only (and everytime) an interface connects, we can utilize the scripts in /etc/network/if-up.d/ to do the job 🙂
Now run the following command to create a new script file:
nano /etc/network/if-up.d/hangfix-ifup && chmod +x /etc/network/if-up.d/hangfix-ifup
Insert this into the file:
#!/bin/sh -e if [ "$IFACE" = "YOUR-INTERFACE-NAME-HERE" ]; then /sbin/ethtool -K $IFACE gso off gro off tso off tx off rx off pcie_aspm=off fi exit 0