19.4 RAID1 - Mirroring

Mirroring is a technology used by many corporations and home users to back up data without interruption. When a mirror exists, it simply means that diskB replicates diskA. Or, perhaps diskC+D replicates diskA+B. Regardless of the disk configuration, the important aspect is that information on one disk or partition is being replicated. Later, that information could be more easily restored, backed up without causing service or access interruption, and even be physically stored in a data safe.

To begin, ensure the system has two disk drives of equal size, these exercises assume they are direct access (da(4)) SCSI disks.

19.4.1 Mirroring Primary Disks

Assuming FreeBSD has been installed on the first, da0 disk device, gmirror(8) should be told to store its primary data there.

Before building the mirror, enable additional debugging information and opening access to the device by setting the kern.geom.debugflags sysctl(8) option to the following value:

# sysctl kern.geom.debugflags=17

Now create the mirror. Begin the process by storing meta-data information on the primary disk device, effectively creating the /dev/mirror/gm device using the following command:

Warning: Creating a mirror out of the boot drive may result in data loss if any data has been stored on the last sector of the disk. This risk is reduced if creating the mirror is done promptly after a fresh install of FreeBSD.

# gmirror label -vb round-robin gm0 /dev/da0

The system should respond with:

Metadata value stored on /dev/da0.
Done.

Initialize GEOM, this will load the /boot/kernel/geom_mirror.ko kernel module:

# gmirror load

Note: When this command completes successfully, it creates the gm0 device node under the /dev/mirror directory.

Enable loading of the geom_mirror.ko kernel module during system initialization:

# echo 'geom_mirror_load="YES"' >> /boot/loader.conf

Edit the /etc/fstab file, replacing references to the old da0 with the new device nodes of the gm0 mirror device.

Note: If vi(1) is your preferred editor, the following is an easy way to accomplish this task:

# vi /etc/fstab

In vi(1) back up the current contents of fstab by typing :w /etc/fstab.bak. Then replace all old da0 references with gm0 by typing :%s/da/mirror\/gm/g.

The resulting fstab file should look similar to the following. It does not matter if the disk drives are SCSI or ATA, the RAID device will be gm regardless.

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/mirror/gm0s1b                       none            swap    sw              0       0
/dev/mirror/gm0s1a                      /               ufs     rw              1       1
/dev/mirror/gm0s1d              /usr        ufs rw      0   0
/dev/mirror/gm0s1f              /home       ufs rw      2   2
#/dev/mirror/gm0s2d                     /store          ufs     rw              2       2
/dev/mirror/gm0s1e                      /var            ufs     rw              2       2
/dev/acd0               /cdrom          cd9660  ro,noauto       0       0

Reboot the system:

# shutdown -r now

During system initialization, the gm0 should be used in place of the da0 device. Once fully initialized, this may be checked by visually inspecting the output from the mount command:

# mount
Filesystem         1K-blocks    Used    Avail Capacity  Mounted on
/dev/mirror/gm0s1a   1012974  224604   707334    24%    /
devfs                      1       1        0   100%    /dev
/dev/mirror/gm0s1f  45970182   28596 42263972     0%    /home
/dev/mirror/gm0s1d   6090094 1348356  4254532    24%    /usr
/dev/mirror/gm0s1e   3045006 2241420   559986    80%    /var
devfs                      1       1        0   100%    /var/named/dev

The output looks good, as expected. Finally, to begin synchronization, insert the da1 disk into the mirror using the following command:

# gmirror insert gm0 /dev/da1

As the mirror is built the status may be checked using the following command:

# gmirror status

Once the mirror has been built and all current data has been synchronized, the output from the above command should look like:

      Name    Status  Components
mirror/gm0  COMPLETE  da0
                      da1

If there are any issues, or the mirror is still completing the build process, the example will show DEGRADED in place of COMPLETE.

19.4.2 Mirroring individual partitions

With contributions from Anton Shterenlikht.

Note: in some partitioning schemes mirroring a whole disk is not possible at all. The most notable example is GPT, used in ia64 architecture. Secondary GPT header is stored in the last sector on disk. Since gmirror writes its data in the last sector, it destroys secondary GPT. A very good introduction to GPT (GUID Partition Table) can be found on Wikipedia.

Mirroring individual partitions is a more flexible way of achieving RAID1 compared to mirroring a whole disk. Below is a step by step guide.

  1. Check partition of the boot disk using gpart(8):

    # gpart show
    =>       34  143374671  da0  GPT  (68G)
             34     819200    1  efi  (400M)
         819234    1048576    2  freebsd-ufs  (512M)
        1867810    4194304    3  freebsd-swap  (2.0G)
        6062114    2097152    4  freebsd-ufs  (1.0G)
        8159266    2097152    5  freebsd-ufs  (1.0G)
       10256418  133118287    6  freebsd-ufs  (63G)
    

    Note: this example is taken from ia64 architecture, which includes EFI boot partition. Depending on the architecture of your system you might or might not have this partition present.

  2. Partition a spare disk, da1, exactly as the boot disk. First create a partitioning scheme on da1, identical to that on da0, in this example this is GPT:

    # gpart create -s gpt da1
    

    Add partitions exactly as on da0:

    # gpart add -b 34 -s 819200 -t freebsd-efi da1
    # gpart add -b 819234 -s 1048576 -t freebsd-ufs da1
    # gpart add -b 1867810 -s 4194304 -t freebsd-swap da1
    # gpart add -b 6062114 -s 2097152 -t freebsd-ufs da1
    # gpart add -b 8159266 -s 2097152 -t freebsd-ufs da1
    # gpart add -b 10256418 -s 133118287 -t freebsd-ufs da1
    

    Note: -b sets the starting block of a new partition, -s specifies partition size, in blocks, and -t is partition type. On ia64 systems the boot partition is of EFI type, hence the first partition added with gpart is freebsd-efi. Likewise, freebsd-swap is used to add a swap partition.

    Tip: do gpart show before each gpart add. This will tell you the starting block.

    When you are done both disks should be partitioned identically:

    # gpart show
    =>       34  143374671  da0  GPT  (68G)
             34     819200    1  efi  (400M)
         819234    1048576    2  freebsd-ufs  (512M)
        1867810    4194304    3  freebsd-swap  (2.0G)
        6062114    2097152    4  freebsd-ufs  (1.0G)
        8159266    2097152    5  freebsd-ufs  (1.0G)
       10256418  133118287    6  freebsd-ufs  (63G)
    
    =>       34  143374671  da1  GPT  (68G)
             34     819200    1  efi  (400M)
         819234    1048576    2  freebsd-ufs  (512M)
        1867810    4194304    3  freebsd-swap  (2.0G)
        6062114    2097152    4  freebsd-ufs  (1.0G)
        8159266    2097152    5  freebsd-ufs  (1.0G)
       10256418  133118287    6  freebsd-ufs  (63G)
    
  3. load gmirror kernel module:

    # gmirror load
    
  4. Create mirror for EFI partition.

    Note: If you do not have an EFI partition, skip this step.



    Unmount /efi because GEOM manipulations can be performed only on unmounted, not in use, partition:

    # umount /efi
    

    create EFI mirror on the boot disk, da0, in our example:

    # gmirror label -vb round-robin efi da0p1
    


    This would create /dev/mirror/efi device.

    Add EFI partition of the spare disk, da1 to the mirror:

    # gmirror insert efi da1p1
    

    The EFI partition in this example is only 400MB, so it rebuilds quickly:

    # gmirror status
          Name    Status  Components
    mirror/efi  COMPLETE  da0p1
                          da1p1
    

    mount EFI mirror and check:

    # mount -t msdosfs /dev/mirror/efi /efi
    # df
    Filesystem       512-blocks    Used     Avail Capacity  Mounted on
    ...
    /dev/mirror/efi      819008  141728    677280    17%    /efi
    
  5. Create mirror for root (/) partition. This involves extra steps since / cannot be unmounted.

    Create mirror on the spare disk, da1:

    # gmirror label -vb round-robin root da1p2
    

    Create ufs filesystem on this mirror:

    # newfs /dev/mirror/root
    

    Mount root mirror temporarily, say under /mnt:

    # mount /dev/mirror/root /mnt
    

    Copy / onto /mnt (which is the root mirror, /dev/mirror/root), using a combination of dump(8) and restore(8):

    # cd /mnt
    # dump 0aLf - / | restore rf -
    

    Warning: dump(8) is the only safe way to copy root partition. Any other copying tool is not guaranteed to do it right.

  6. Update fstab on the mirror. Edit /mnt/etc/fstab and change da0p1 into mirror/efi and da0p2 into mirror/root:

    # cat /mnt/etc/fstab
    # Device                Mountpoint      FStype  Options         Dump    Pass#
    /dev/da0p3              none            swap    sw              0       0
    /dev/mirror/root        /               ufs     rw              1       1
    /dev/mirror/efi         /efi            msdosfs rw              0       0
    /dev/da0p5              /tmp            ufs     rw              2       2
    /dev/da0p6              /usr            ufs     rw              2       2
    /dev/da0p4              /var            ufs     rw              2       2
    /dev/acd0               /cdrom          cd9660  ro,noauto       0       0
    
  7. Enable loading of the geom_mirror.ko kernel module during system initialization. For this do either:

    # echo 'geom_mirror_load="YES"' >> /boot/loader.conf
    

    or add

    options         GEOM_MIRROR
    
    to your kernel configuration file.

  8. Change the root device in /boot/loader.conf. The root device is specified with vfs.root.mountfrom option. It should point to the root mirror. For example, change

    vfs.root.mountfrom="ufs:/dev/da0p2"
    
    into
    vfs.root.mountfrom="ufs:/dev/mirror/root"
    


  9. Reboot into single user mode.

    Tip: On ia64 type boot -s on the boot prompt.

  10. At boot you should see gmirror loaded, and then these lines:

    GEOM_MIRROR: Device mirror/efi launched (2/2).
    GEOM_MIRROR: Device mirror/root launched (1/1).
    Trying to mount root from ufs:/dev/mirror/root
    
  11. Now that da0p2 is not mounted, it can be inserted into root mirror:

    # gmirror insert root da0p2
    
    and after mirror rebuild is complete you should see:
    # gmirror status
           Name    Status  Components
     mirror/efi  COMPLETE  da0p1
                           da1p1
    mirror/root  COMPLETE  da1p2
                           da0p2
    


  12. Create mirrors for all other partitions of da0, which are now not mounted:

    # gmirror label -vb round-robin swap da0p3
    # gmirror label -vb round-robin var da0p4
    # gmirror label -vb round-robin tmp da0p5
    # gmirror label -vb round-robin usr da0p6
    
  13. Edit /etc/fstab and change each remaining da0 partition into its mirror:

    # cat /etc/fstab
    # Device                Mountpoint      FStype  Options         Dump    Pass#
    /dev/mirror/swap        none            swap    sw              0       0
    /dev/mirror/root        /               ufs     rw              1       1
    /dev/mirror/efi         /efi            msdosfs rw              0       0
    /dev/mirror/tmp         /tmp            ufs     rw              2       2
    /dev/mirror/usr         /usr            ufs     rw              2       2
    /dev/mirror/var         /var            ufs     rw              2       2
    /dev/acd0               /cdrom          cd9660  ro,noauto       0       0
    
  14. Reboot.

  15. Add remaining da1 partitions to mirrors:

    # gmirror insert swap da1p3
    # gmirror insert var da1p4
    # gmirror insert tmp da1p5
    # gmirror insert usr da1p6
    
  16. Done!:

    # gmirror status
           Name    Status  Components
     mirror/efi  COMPLETE  da0p1
                           da1p1
    mirror/root  COMPLETE  da0p2
                           da1p2
    mirror/swap  COMPLETE  da0p3
                           da1p3
     mirror/var  COMPLETE  da0p4
                           da1p4
     mirror/tmp  COMPLETE  da0p5
                           da1p5
     mirror/usr  COMPLETE  da0p6
                           da1p6
    

19.4.3 Troubleshooting

19.4.3.1 System refuses to boot

If the system boots up to a prompt similar to:

ffs_mountroot: can't find rootvp
Root mount failed: 6
mountroot>

Reboot the machine using the power or reset button. At the boot menu, select option six (6). This will drop the system to a loader(8) prompt. Load the kernel module manually:

OK? load geom_mirror
OK? boot

If this works then for whatever reason the module was not being loaded properly. Check whether the relevant entry in /boot/loader.conf is correct. If the problem persists, place:

options    GEOM_MIRROR

in the kernel configuration file, rebuild and reinstall. That should remedy this issue.

19.4.3.2 System cannot find the boot device

In that case the system boot process would stop with a prompt similar to:

Loader variables:
vfs.root.mountfrom=
vfs.root.mountfrom.options=

Manual root filesystem specification:
  <fstype>:<device>  Mount <device> using filesystem <fstype>
                       eg. ufs:/dev/da0s1a
                       eg. cd9660:/dev/acd0
                       This is equivalent to: mount -t cd9660 /dev/acd0 /

  ?                  List valid disk boot devices
  <empty line>       Abort manual input

mountroot>

You would get this message most probably if you forget to add the correct boot device in /boot/loader.conf. The hints given by the system are clear. If your root device is /dev/mirror/root, and the filesystem is ufs, enter:

mountroot> ufs:/dev/mirror/root

The boot process would then continue:

Trying to mount root from ufs:/dev/mirror/root

When the system is up remember to edit your root device in /boot/loader.conf.

19.4.4 Recovering From Disk Failure

The wonderful part about disk mirroring is that when a disk fails, it may be replaced, presumably, without losing any data.

Considering the previous RAID1 configuration, assume that da1 has failed and now needs to be replaced. To replace it, determine which disk has failed and power down the system. At this point, the disk may be swapped with a new one and the system brought back up. After the system has restarted, the following commands may be used to replace the disk:

# gmirror forget gm0
# gmirror insert gm0 /dev/da1

Use the gmirror status command to monitor the progress of the rebuild. It is that simple.

This, and other documents, can be downloaded from ftp://ftp.FreeBSD.org/pub/FreeBSD/doc/.

For questions about FreeBSD, read the documentation before contacting <questions@FreeBSD.org>.
For questions about this documentation, e-mail <doc@FreeBSD.org>.