Linux system recovery


This method of system recovery is done on the console of your V-Colo. The normal startup system will be bypassed to get a root shell.

These instructions are generic and the details may not match the O/S on your V-Colo. If there are specific instructions provided by the vendor of your O/S, it is best to use those.

Summary of this procedure:


Connect to the console of your V-Colo:
Please enter one of the following to connect to, or reboot a system:

 Enter this               To do this
 ---------------------    ----------------------------
 vc293                    Connect to vc293 "My V-Colo"

 logout                   Disconnect from this console server

? vc293
*** Trying 166.84.3.146.

*** Connected to console server. ***

Basic instructions:

   Control-^ C                  close connection

   Control-^ may be either Control-6 or Control-Shift-6 on your keyboard.


Reboot your V-Colo.

If you are unable to reboot from within your V-Colo, you can use our Web control panel.

As your system reboots, the console session will be disconnected:


[  OK  ] Reached target System Reboot.
[340185.304841][    T1] reboot: Restarting system


*** Disconnected from console server. ***

Please enter one of the following to connect to, or reboot a system:

 Enter this               To do this
 ---------------------    ----------------------------
 vc293                    Connect to vc293 "My V-Colo"

 logout                   Disconnect from this console server

?
Reconnect again:

? vc293
*** Trying 166.84.3.146.

*** Connected to console server. ***

Basic instructions:

   Control-^ C                  close connection

   Control-^ may be either Control-6 or Control-Shift-6 on your keyboard.


When the GRUB menu appears, press "e" to edit the default entry. In some cases, you may not see the menu, but will still see that it is counting down. Press "e" if you see this.

                             GNU GRUB  version 2.04

 +----------------------------------------------------------------------------+
 |*My distribution, with Linux 5.13.0-51-generic                              |
 | My distribution, with Linux 5.13.0-51-generic (recovery mode)              |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 |                                                                            |
 +----------------------------------------------------------------------------+

      Use the ^ and v keys to select which entry is highlighted.
      Press enter to boot the selected OS, `e' to edit the commands
      before booting or `c' for a command-line. ESC to return
      previous menu.




Use the arrow keys to move down and find the line which specifies the kernel to load. The keyword at the beginning of the line may be "linux", "linux2", or "kernel":

                             GNU GRUB  version 2.04

 +----------------------------------------------------------------------------+
 |                  search --no-floppy --fs-uuid --set=root  12c373d5-b234-4d\|^
 |7e-a2d4-9d7a6b1879ca                                                        |
 |                else                                                        |
 |                  search --no-floppy --fs-uuid --set=root 12c373d5-b234-4d7\|
 |e-a2d4-9d7a6b1879ca                                                         |
 |                fi                                                          |
 |                echo        'Loading Linux 5.13.0-51-generic ...'           |
 |                linux        /boot/vmlinuz-5.13.0-51-generic root=UUID=12c3\|
 |73d5-b234-4d7e-a2d4-9d7a6b1879ca ro  console=ttyS0 quiet splash $vt_handoff |
 |                echo        'Loading initial ramdisk ...'                   |
 |                initrd        /boot/initrd.img-5.13.0-51-generic            |
 |                                                                            |
 +----------------------------------------------------------------------------+

      Minimum Emacs-like screen editing is supported. TAB lists
      completions. Press Ctrl-x or F10 to boot, Ctrl-c or F2 for
      a command-line or ESC to discard edits and return to the GRUB menu.




Move the cursor to the end of the line and append "rw init=/bin/bash".

                             GNU GRUB  version 2.04

 +----------------------------------------------------------------------------+
 |        insmod ext2                                                         |^
 |        if [ x$feature_platform_search_hint = xy ]; then                    |
 |          search --no-floppy --fs-uuid --set=root  12c373d5-b234-4d7e-a2d4-\|
 |9d7a6b1879ca                                                                |
 |        else                                                                |
 |          search --no-floppy --fs-uuid --set=root 12c373d5-b234-4d7e-a2d4-9\|
 |d7a6b1879ca                                                                 |
 |        fi                                                                  |
 |        linux        /boot/vmlinuz-5.13.0-51-generic root=UUID=12c373d5-b23\|
 |4-4d7e-a2d4-9d7a6b1879ca ro  console=ttyS0 quiet splash $vt_handoff rw init\|
 |=/bin/bash                                                                  |
 |        initrd        /boot/initrd.img-5.13.0-51-generic                    |v
 +----------------------------------------------------------------------------+

      Minimum Emacs-like screen editing is supported. TAB lists
      completions. Press Ctrl-x or F10 to boot, Ctrl-c or F2 for
      a command-line or ESC to discard edits and return to the GRUB menu.



Press Control-X to boot.

When it's finished booting, you should get a root shell prompt similar to one of the following:

bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
root@(none):/#

         [...]
         Starting Switch Root...
[   14.765310] systemd-journald[248]: Received SIGTERM from PID 1 (systemd).
bash-5.1#


Enter the commands needed to fix your system. In this example, the root password is changed:

root@(none):/# passwd root
New password:
Retype new password:
passwd: password updated successfully

You can also change the password of any other user or edit the SSH ".ssh/authorized_keys" file of root or any other user.

For systems with selinux enabled, security context labels have to be reset. This is common for Redhat derived systems like Fedora, Centos, and others.
Check if selinux is enabled on this system:

bash-5.1# grep ^SELINUX= /etc/selinux/config
SELINUX=enforcing

If the file is present and SELINUX=enforcing, then tell the system to do the relabling on next boot:

bash-5.1# touch /.autorelabel

On next boot, the system will re-label. This may take some time depending on the size of the filesystem. When finished, the system will reboot again.

Reboot the system:

bash-5.1# /sbin/reboot -f
Rebooting.