There’s a delicate balance between keeping a server up-to-date versus having a server that’s actually working. As someone who enjoys testing the ‘bleeding edge’ or ‘beta’ versions of software, I have also come to realize that uptime is more important than having the latest functionality. Of course, this needs to be balanced with applying security updates as soon as possible.

On two previous occasions, I have updated my bioinformatics server with a sudo dnf update, whose kernel updates unfortunately led to breaking changes in my experiment’s packages. This notably causes the most issues with Nvidia CUDA / drivers.

When a new kernel is installed, it becomes the default which GRUB boots into, essentially meaning that I cannot boot into the ‘last working kernel’ without having physical access to the machine. A KVM would be helpful here, but I currently do not have one.

Below are some notes on my server’s current setup and how I rolled back the kernel update.

Rolling back the Linux Kernel

Identify Boot Loader Specification (BLS) entries

• Kernel entries located in `/boot/loader/entries/`

ls -1 /boot/loader/entries/

# Output
# 1e9791396405438c8d655ac31992dd75-0-rescue.conf
# 1e9791396405438c8d655ac31992dd75-5.14.0-503.16.1.el9_5.x86_64.conf
# 1e9791396405438c8d655ac31992dd75-5.14.0-503.35.1.el9_5.x86_64.conf
# 1e9791396405438c8d655ac31992dd75-5.14.0-503.40.1.el9_5.x86_64.conf

• Confirmed current running kernel with:

uname -r

# Output
# 5.14.0-503.35.1.el9_5.x86_64

Set default kernel to a specific version

• Used `grub2-editenv` to set default kernel:

grub2-editenv set saved_entry=1e9791396405438c8d655ac31992dd75-5.14.0-503.35.1.el9_5.x86_64

Hide problematic kernel entries (optional)

• Edited `.conf` file(s) in `/boot/loader/entries/`
• Added `hidden` directive with a comment, for example:

sudo nano /boot/loader/entries/1e9791396405438c8d655ac31992dd75-5.14.0-503.40.1.el9_5.x86_64.conf

title Red Hat Enterprise Linux (5.14.0-503.40.1.el9_5.x86_64) 9.5 (Plow)
version 5.14.0-503.40.1.el9_5.x86_64
linux /vmlinuz-5.14.0-503.40.1.el9_5.x86_64
initrd /initramfs-5.14.0-503.40.1.el9_5.x86_64.img
options root=/dev/mapper/rhel-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet rd.driver.blacklist=nouveau
grub_users $grub_users
grub_arg --unrestricted
grub_class rhel


# Hidden because this kernel causes issues after update. cannot boot with GUI, use nvidia, etc
hidden

Verify GRUB configuration and environment

• Checked saved entry:

grub2-editenv list

• Listed GRUB menu entries:

awk -F\' '/menuentry / {print i++ ": " $2}' /boot/grub2/grub.cfg

Important Notes

• Changes via grub2-editenv control the default boot kernel.
• Kernel hiding done by adding `hidden` in the BLS entry `.conf` file.
• /boot/grub2/grub.cfg is auto-generated — avoid manual edits.
• Reboot to confirm the changes take effect.