Random   •   Archives   •   RSS   •   About   •   Contact   •  

Brown Out Reboots Servers: NVIDIA Drivers Vanish, Ethernet Dies – Sneaker-Net Rescue with a Bare USB Stick

Two identical Ubuntu 24.04.2 boxes, both running ML workloads on NVIDIA GPUs. A brownout rolls through the neighborhood—lights dim & both machines abruptly power-cycle. Unlike my other systems with UPS protection, these servers lack battery backup & go down hard. They come back up, login prompt intact, but nvidia-smi returns nothing. Drivers gone. No problem—sudo ubuntu-drivers autoinstall, reboot, and… oh no. Ethernet vanishes. ip link shows only lo. No eth0, no enp3s0, no nothing. Both servers, perfectly synchronized in failure.

The culprit? The NVIDIA install (or the kernel update it pulled) silently removed linux-modules-extra-6.14.0-35-generic. Without it, the Intel igc NIC driver refuses to load. No network, no apt, no recovery—classic isolation trap.

Physical access: check. Internet on another machine: check. Ubuntu installer USB? Nope—this time, I went full sneaker-net with a blank USB stick & one precious .deb.

Use uname -a to get the exact kernel version for the deb file.

On a working laptop:

wget http://archive.ubuntu.com/ubuntu/pool/main/l/linux-hwe-6.14/linux-modules-extra-6.14.0-35-generic_6.14.0-35.35~24.04.1_amd64.deb
cp linux-modules-extra-*.deb /media/user/USB/

Eject. Run down to the rack. Plug USB into Server #1.

Boot normally (no live disk—just the regular system). Log in at console.

lsblk
# Spot the USB: probably /dev/sdb1, mounted under /media/ubuntu/USB or similar
sudo mkdir -p /mnt/usb
sudo mount /dev/sdb1 /mnt/usb
ls /mnt/usb
# → linux-modules-extra-6.14.0-35-generic_6.14.0-35.35~24.04.1_amd64.deb

Install it:

sudo dpkg -i /mnt/usb/linux-modules-extra-*.deb
sudo depmod -a

Reload the NIC driver:

lspci -nnk | grep -i ethernet
# → shows "Kernel driver in use: igc" (or should)
sudo modprobe -r igc
sudo modprobe igc
ip link
# → enp3s0 appears!

nvidia-smi is back. Server #1: saved.

Realtek RTL8125 NIC on the Second Server

Ah, the classic "identical" servers with a twist—one Intel, one Realtek. The RTL8125B (common on modern boards) uses the r8169 kernel module in Ubuntu 24.04.2. The fix is nearly identical to the Intel one, just swap the driver name in the reload step.

Reload the Realtek NIC driver:

lspci -nnk | grep -i ethernet
# → shows "Kernel driver in use: r8169" (or should)
sudo modprobe -r r8169 2>/dev/null || true
sudo modprobe r8169
ip link
# → enp3s0 appears!

Both servers: fully recovered. One USB stick, two different NICs, same root cause. The missing linux-modules-extra package strikes again.

Bonus: Mouse Drivers Too

It's not just NICs. The linux-modules-extra package also contains drivers for various USB mice & peripherals. If you boot into a system where your mouse suddenly stops working after an NVIDIA driver update, you're likely hitting the same issue. The HID drivers for many Logitech, Razer, & other gaming mice live in linux-modules-extra.

After installing the package & running sudo depmod -a, either reboot or manually reload the HID modules:

sudo modprobe -r usbhid
sudo modprobe usbhid

Your mouse should spring back to life. Same root cause, different symptom—the NVIDIA installer's collateral damage extends beyond networking.




Want comments on your site?

Remarkbox — is a free SaaS comment service which embeds into your pages to keep the conversation in the same place as your content. It works everywhere, even static HTML sites like this one!

Remarks: Brown Out Reboots Servers: NVIDIA Drivers Vanish, Ethernet Dies – Sneaker-Net Rescue with a Bare USB Stick

© Russell Ballestrini.