Candela Technologies Logo
Network Testing and Emulation Solutions

Recovering Filesystems

Goal: Recover a LANforge system that boots into Emergency Mode

If your LANforge boots into Emergency Mode, your system is experiencing file-system corruption. Follow these instructions to check the filesystems. This process will require a monitor and keyboard or a serial cable connected to the LANforge.
File system corruptions are caused by power-off events without properly shutting down the system
 
  1. The emergency mode prompt looks similar to this:
    Give root password for maintenance
    (or type Control-D to continue):
    screenshot
    1. The messages beforehand that begin with [ OK ] can be ignored.
    2. Messages beginning with ath10k_pci can be ignored.
  2. We assume you'll press the Enter (↲) key when executing the commands in further examples.
  3. Provide the root password: lanforge
  4. The we next need to determine where our filesystems are mounted. You might have filesystems on partitions, or on LVM volumes. Use the command mount | grep ext4 to find your partitions:
    mount | grep ext4
    /dev/sda2 on / type ext4 (rw,relatime,nodelalloc)
    /dev/sda4 on /home type ext4 (rw,relatime,nodelalloc)
    /dev/sda1 on /boot type ext4 (rw,relatime,nodelalloc)
    This example shows partitions.
    Note that inspecting /etc/fstab often does not show you device partitions, rather it shows you volume or partition UUIDs.
  5. Example of filesystems on volumes:
    mount | grep ext4
    /dev/mapper/fedora-root on / type ext4 (rw,relatime,nodelalloc)
    /dev/mapper/fedora-home on /home type ext4 (rw,relatime,nodelalloc)
    /dev/sda1 on /boot type ext4 (rw,relatime,nodelalloc)
    Notice that /boot is typically not a LVM volume.

  6. Issuing the fsck commands

  7. For each of those filesystems in the partition example above, you will execute the below commands:
    1. fsck -fy /dev/sda2 (for /)
    2. fsck -fy /dev/sda4 (for /home)
    3. fsck -fy /dev/sda1 (for /boot)
  8. For the volumes example above, you will execute the below commands:
    1. fsck -fy /dev/mapper/fedora-root (for /)
    2. fsck -fy /dev/mapper/fedora-home (for /home)
    3. fsck -fy /dev/sda1 (for /boot)

  9. Configuring Filesystems Check on Reboot

  10. The commands below involve setting the 'mount count' parameter to 1. Every time the filesystem is mounted, it will perform a check. The following examples ensure that full journaling is enabled on the filesystem. This ensures the most amount of recovery. And after that, modern versions of Fedora have metadata checksumming available. Older LANforge systems do not have this.

    1. Filesystem Check on Every Mount

      The filesystems do not need to be unmounted to set this parameter. Only the partition example is shown, the volumes example uses similar commands.
      1. touch /forcefsck (this technique works with any model of filesystem)
      2. tune2fs -c 1 /dev/sda2 (for /)
      3. tune2fs -c 1 /dev/sda4 (for /home)
      4. tune2fs -c 1 /dev/sda1 (for /boot)

    2. Tune the Filesystem for Full Journaling

      The filesystems do not need to be unmounted to set this parameter. The commands for the volumes version is similar to the partition example below.
      1. tune2fs -o journal_data /dev/sda2 (for /)
      2. tune2fs -o journal_data /dev/sda4 (for /home)
      3. tune2fs -o journal_data /dev/sda1 (for /boot)
    3. Note: Inspect /etc/fstab for conflicting mount options. The folling example shows mount options applied in the fstab file:
      mount | grep ext4
      /dev/sda2 on / type ext4 (rw,relatime,nodelalloc)
      /dev/sda4 on /home type ext4 (rw,relatime,nodelalloc)
      /dev/sda1 on /boot type ext4 (rw,relatime,data=writeback)

      grep data= /etc/fstab
      UUID=1c1b4732-653f-47dd-a106-ae17cf5b12a9 /boot ext4 data=writeback  1 2
      Notice the fstab entry for /boot? It has overridden the data journaling mode. Erase that setting from the fstab mount options.

    4. Enable Metadata Checksumming (optional)

      It is only practical to apply metadata checksumming in the below conditions. You will not be able to apply it to the root filesystem / because it cannot be unmounted when you boot the system.
      1. Fedora 27 or more recent
      2. tune2fs 1.43 or more recent
      3. crc32c or libcrc32c modules loaded
      4. you can unmount the filesystem.
    5. To Boot into Emergency Mode, follow these steps:
      1. Reboot system into Emergency Mode by appending the word emergency at the end of your grub boot option.
      2. umount -f /home
      3. umount -f /boot
      4. fsck -fD /dev/sda1 (remember that's /boot)
      5. fsck -fD /dev/mapper/fedora-home
      6. tune2fs -O metadata_csum /dev/sda1
      7. tune2fs -O metadata_csum /dev/mapper/fedora-home

    6. Disable Drive Write Caching (optional)

      Drives typically ship with write caching enabled for a performance boost. SSDs have this as well. By turning it off you trade some performance for increased data safety. LANforge systems do not ship with write caching disabled. To make this setting enabled every boot, it needs to be added to /etc/rc.local. LANforge systems have rc-local.service enabled by defualt.
      Only disable write caching if your machine powers off without shutdown frequently
      1. vi /etc/rc.local
      2. Add the line /sbin/hdparm -W0 /dev/sda
      3. Save the file.
      4. Reboot, or issue the command for this booted session:
        /sbin/hdparm -W0 /dev/sda
Candela  Technologies, 2417 Main Street, Suite 201, Ferndale, WA 98248, USA
www.candelatech.com | sales@candelatech.com | +1.360.380.1618
Facebook | LinkedIn | Blog