How to recover a system rebooted during an update

Some actions should never be done during a system update. Rebooting or shutting down the system is one of them.

So, what's more stupid than trying to remove a USB keyboard during the update, except unplug the power cable instead? Yes, I did it, and my machine couldn't boot afterward...

starting Switch Root...
[  !!  ] Failed to execute /sbin/init
[!!!!!!] Failed to execute fallback shell.

Well, it's possible to recover without reinstalling everything!

Rebooting in rescue mode

The RHEL installation media offers a troubleshooting mode. It can be launched by booting with the media, and choose "Troubleshooting" in the grub menu.

You’ll then see the following screen:

Starting installer, one moment...
libreport is not available in this environment - bug reporting disabled
anaconda 40.22.3.26-1.el10_0 for Red Hat Enterprise Linux 10.0 started.
* installation log files are stored in tmp during the installation
* shell is available on TTY2 and in second TMUX pane (ctrl+b, then press 2)
* when reporting a bug add logs from /tmp as separate text/plain attachments
===============================================================================
===============================================================================
Rescue

The rescue environment will now attempt to find your Linux installation and 
mount it under the directory : /mnt/sysroot.  You can then make any changes 
required to your system. Choose '1' to proceed with this step.
You can choose to mount your file systems read-only instead of read-write by 
choosing '2'
If for some reason this process does not work choose '3' to skip directly to a 
shell


1) Continue
2) Read-only mount
3) Skip to shell
4) Quit (Reboot)
Please make a selection from the above: _

Type 1 and validate by pressing enter

After a short time, the installation is discovered:

Rescue Shell

Your system has been mounted under /mnt/sysroot.

If you would like to make the root of your system the root of the active system, 
run the command:
      chroot /mnt/sysroot
Warning: The rescue shell will trigger SELinux autorelabel on the subsequent 
boot. Add "enforcing-0" on the kernel command line for autorelabel to work 
properly.

When finished, please exit from the shell and your system will reboot.

Please press ENTER to get a shell:
bash-5.2#

As the message said, the system to repair is mounted in /mnt/sysroot

A nice troubleshooting session starts with a pleasant environment

If needed, set the keyboard mapping. For example, for a french keyboard:

localectl set-keymap fr

Every time the tab key is used to invoke the autocompletion, the buzzer is activated. It's possible to disable the buzzer totally with the command:

rmmod pcspkr

Have a look at what happened

The error message was about /sbin/init, so let's examine this file:

# ls -l /mnt/sysroot/sbin/init 
lrwxrwxrwx. 1 root root 22 Jan 20 11:11 /sbin/init -> ../lib/systemd/systemd

init is a symbolic link to /lib/systemd/systemd. Let's list the files in this directory:

# ls -l 
drwxr-xr-x.  2 root root      6 Jan 28 11:11 boot
drwxr-xr-x.  2 root root   4096 May 30 08:44 catalog
-rw-r--r--.  1 root root   9551 Oct 31 2022  import-pubring-gpg
-rw-r--r--.  1 root root      0 Jan 28 11:11 libsystemd-shared.abignore
drwxr-xr-x.  2 root root     29 May 30 08:44 network
drwxr-xr-x.  2 root root     29 Mar  3 11:54 ntp-units.d
-rwxr-xr-х.  1 root root      0 Jan 28 11:11 purge-nobody-user
-rw-r--r--.  1 root root      0 Oct 31 2022  resolv.conf
drwxr-xr-x. 37 root root  24976 May 30 08:44 system
-rwxr-xr-х.  1 root root      0 Jan 28 11:11 systemd            <=======
-rwxr-xr-х.  1 root root      0 Jan 28 11:11 systemd-ac-power
-rwxr-xr-х.  1 root root      0 Jan 28 11:11 systemd-backlight
-rwxr-xr-х.  1 root root      0 Jan 28 11:11 systemd-binfmt
[...]

Many other files in this directory are 0 bytes in size. An easy way to fix it is to reinstall the packages.

Reinstalling the packages

... but only those of the update transaction!

Dnf stores the rpm to install in /var/cache/dnf, so, it's possible to get them and reinstall it using the rpm command

To use the rpm command without bothering with many options, it's possible to chroot in the "physical" system. We can do it as the others binaries seems not to be broken:

# chroot /mnt/sysroot

Search for the directories containing the latest transaction:

# cd  /var/cache/dnf
# ls -ltr | grep rpms | tail -n 2
druxr-xr-x. 4 root root   38 May 30 08:41 rhel-9-for-x86_64-baseos-rpms-9ab91850ea154141
druxr-xr-x. 4 root root   38 May 30 08:41 rhel-9-for-x86_64-appstream-rpms-2111d588453ffe4c

Although only the systemd binaries appear to be broken (the system seemed to be unplugged during the systemd package update), it's a good idea to reinstall all the packages: the rpm scriptlets and the checks were not run. By reinstalling all the packages, we ensure that all the steps are executed properly.

Reinstall all the packages cached for this transaction:

# rpm -Uv --force rhel-9-for-x86_64-baseos-rpms-9ab91850ea154141/* rhel-9-for-x86_64-appstream-rpms-2111d588453ffe4c/* 
Verifying packages...
Preparing packages...
alternatives-1.24-2.e19.x86_64.rpm
alsa-1ib-1.2.13-2.el9.x86_64.rpm
alsa-ucm-1.2.13-2.el9.noarch.rpm
[...]

Reboot

Once the packages are reinstalled, reboot the system. The SELinux contexts will be relabeled. It can take some time.

The system is working again! 🎉

Note that the console snippets can have typos. I took photos of the screen during the process and used OCR to get the text