Mirroring is a technology used by many corporations and home users to back up data without interruption. When a mirror exists, it simply means that diskB replicates diskA. Or, perhaps diskC+D replicates diskA+B. Regardless of the disk configuration, the important aspect is that information on one disk or partition is being replicated. Later, that information could be more easily restored, backed up without causing service or access interruption, and even be physically stored in a data safe.
To begin, ensure the system has two disk drives of equal size, these exercises assume they are direct access (da(4)) SCSI disks.
Assuming FreeBSD has been installed on the first, da0 disk device, gmirror(8) should be told to store its primary data there.
Before building the mirror, enable additional debugging information and opening access
to the device by setting the kern.geom.debugflags sysctl(8) option to
the following value:
# sysctl kern.geom.debugflags=17
Now create the mirror. Begin the process by storing meta-data information on the primary disk device, effectively creating the /dev/mirror/gm device using the following command:
Warning: Creating a mirror out of the boot drive may result in data loss if any data has been stored on the last sector of the disk. This risk is reduced if creating the mirror is done promptly after a fresh install of FreeBSD.
# gmirror label -vb round-robin gm0 /dev/da0
The system should respond with:
Metadata value stored on /dev/da0. Done.
Initialize GEOM, this will load the /boot/kernel/geom_mirror.ko kernel module:
# gmirror load
Note: When this command completes successfully, it creates the gm0 device node under the /dev/mirror directory.
Enable loading of the geom_mirror.ko kernel module during system initialization:
# echo 'geom_mirror_load="YES"' >> /boot/loader.conf
Edit the /etc/fstab file, replacing references to the old da0 with the new device nodes of the gm0 mirror device.
Note: If vi(1) is your preferred editor, the following is an easy way to accomplish this task:
# vi /etc/fstabIn vi(1) back up the current contents of fstab by typing :w /etc/fstab.bak. Then replace all old da0 references with gm0 by typing :%s/da/mirror\/gm/g.
The resulting fstab file should look similar to the following. It does not matter if the disk drives are SCSI or ATA, the RAID device will be gm regardless.
# Device Mountpoint FStype Options Dump Pass# /dev/mirror/gm0s1b none swap sw 0 0 /dev/mirror/gm0s1a / ufs rw 1 1 /dev/mirror/gm0s1d /usr ufs rw 0 0 /dev/mirror/gm0s1f /home ufs rw 2 2 #/dev/mirror/gm0s2d /store ufs rw 2 2 /dev/mirror/gm0s1e /var ufs rw 2 2 /dev/acd0 /cdrom cd9660 ro,noauto 0 0
Reboot the system:
# shutdown -r now
During system initialization, the gm0 should be used in place of the da0 device. Once fully initialized, this may be checked by visually inspecting the output from the mount command:
# mount Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/mirror/gm0s1a 1012974 224604 707334 24% / devfs 1 1 0 100% /dev /dev/mirror/gm0s1f 45970182 28596 42263972 0% /home /dev/mirror/gm0s1d 6090094 1348356 4254532 24% /usr /dev/mirror/gm0s1e 3045006 2241420 559986 80% /var devfs 1 1 0 100% /var/named/dev
The output looks good, as expected. Finally, to begin synchronization, insert the da1 disk into the mirror using the following command:
# gmirror insert gm0 /dev/da1
As the mirror is built the status may be checked using the following command:
# gmirror status
Once the mirror has been built and all current data has been synchronized, the output from the above command should look like:
Name Status Components
mirror/gm0 COMPLETE da0
da1
If there are any issues, or the mirror is still completing the build process, the example will show DEGRADED in place of COMPLETE.
Note: in some partitioning schemes mirroring a whole disk is not possible at all. The most notable example is GPT, used in ia64 architecture. Secondary GPT header is stored in the last sector on disk. Since gmirror writes its data in the last sector, it destroys secondary GPT. A very good introduction to GPT (GUID Partition Table) can be found on Wikipedia.
Mirroring individual partitions is a more flexible way of achieving RAID1 compared to mirroring a whole disk. Below is a step by step guide.
Check partition of the boot disk using gpart(8):
# gpart show
=> 34 143374671 da0 GPT (68G)
34 819200 1 efi (400M)
819234 1048576 2 freebsd-ufs (512M)
1867810 4194304 3 freebsd-swap (2.0G)
6062114 2097152 4 freebsd-ufs (1.0G)
8159266 2097152 5 freebsd-ufs (1.0G)
10256418 133118287 6 freebsd-ufs (63G)
Note: this example is taken from ia64 architecture, which includes EFI boot partition. Depending on the architecture of your system you might or might not have this partition present.
Partition a spare disk, da1, exactly as the boot disk. First create a partitioning scheme on da1, identical to that on da0, in this example this is GPT:
# gpart create -s gpt da1
Add partitions exactly as on da0:
# gpart add -b 34 -s 819200 -t freebsd-efi da1 # gpart add -b 819234 -s 1048576 -t freebsd-ufs da1 # gpart add -b 1867810 -s 4194304 -t freebsd-swap da1 # gpart add -b 6062114 -s 2097152 -t freebsd-ufs da1 # gpart add -b 8159266 -s 2097152 -t freebsd-ufs da1 # gpart add -b 10256418 -s 133118287 -t freebsd-ufs da1
Note: -b sets the starting block of a new partition, -s specifies partition size, in blocks, and -t is partition type. On ia64 systems the boot partition is of EFI type, hence the first partition added with gpart is freebsd-efi. Likewise, freebsd-swap is used to add a swap partition.
Tip: do gpart show before each gpart add. This will tell you the starting block.
When you are done both disks should be partitioned identically:
# gpart show
=> 34 143374671 da0 GPT (68G)
34 819200 1 efi (400M)
819234 1048576 2 freebsd-ufs (512M)
1867810 4194304 3 freebsd-swap (2.0G)
6062114 2097152 4 freebsd-ufs (1.0G)
8159266 2097152 5 freebsd-ufs (1.0G)
10256418 133118287 6 freebsd-ufs (63G)
=> 34 143374671 da1 GPT (68G)
34 819200 1 efi (400M)
819234 1048576 2 freebsd-ufs (512M)
1867810 4194304 3 freebsd-swap (2.0G)
6062114 2097152 4 freebsd-ufs (1.0G)
8159266 2097152 5 freebsd-ufs (1.0G)
10256418 133118287 6 freebsd-ufs (63G)
load gmirror kernel module:
# gmirror load
Create mirror for EFI partition.
Note: If you do not have an EFI partition, skip this step.
Unmount /efi because GEOM manipulations can be performed only on unmounted, not in use, partition:
# umount /efi
create EFI mirror on the boot disk, da0, in our example:
# gmirror label -vb round-robin efi da0p1
This would create /dev/mirror/efi device.
Add EFI partition of the spare disk, da1 to the mirror:
# gmirror insert efi da1p1
The EFI partition in this example is only 400MB, so it rebuilds quickly:
# gmirror status
Name Status Components
mirror/efi COMPLETE da0p1
da1p1
mount EFI mirror and check:
# mount -t msdosfs /dev/mirror/efi /efi # df Filesystem 512-blocks Used Avail Capacity Mounted on ... /dev/mirror/efi 819008 141728 677280 17% /efi
Create mirror for root (/) partition. This involves extra steps since / cannot be unmounted.
Create mirror on the spare disk, da1:
# gmirror label -vb round-robin root da1p2
Create ufs filesystem on this mirror:
# newfs /dev/mirror/root
Mount root mirror temporarily, say under /mnt:
# mount /dev/mirror/root /mnt
Copy / onto /mnt (which is the root mirror, /dev/mirror/root), using a combination of dump(8) and restore(8):
# cd /mnt # dump 0aLf - / | restore rf -
Warning: dump(8) is the only safe way to copy root partition. Any other copying tool is not guaranteed to do it right.
Update fstab on the mirror. Edit /mnt/etc/fstab and change da0p1 into mirror/efi and da0p2 into mirror/root:
# cat /mnt/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/da0p3 none swap sw 0 0 /dev/mirror/root / ufs rw 1 1 /dev/mirror/efi /efi msdosfs rw 0 0 /dev/da0p5 /tmp ufs rw 2 2 /dev/da0p6 /usr ufs rw 2 2 /dev/da0p4 /var ufs rw 2 2 /dev/acd0 /cdrom cd9660 ro,noauto 0 0
Enable loading of the geom_mirror.ko kernel module during system initialization. For this do either:
# echo 'geom_mirror_load="YES"' >> /boot/loader.conf
or add
options GEOM_MIRRORto your kernel configuration file.
Change the root device in /boot/loader.conf. The root device is specified with vfs.root.mountfrom option. It should point to the root mirror. For example, change
vfs.root.mountfrom="ufs:/dev/da0p2"into
vfs.root.mountfrom="ufs:/dev/mirror/root"
Reboot into single user mode.
Tip: On ia64 type boot -s on the boot prompt.
At boot you should see gmirror loaded, and then these lines:
GEOM_MIRROR: Device mirror/efi launched (2/2). GEOM_MIRROR: Device mirror/root launched (1/1). Trying to mount root from ufs:/dev/mirror/root
Now that da0p2 is not mounted, it can be inserted into root mirror:
# gmirror insert root da0p2and after mirror rebuild is complete you should see:
# gmirror status
Name Status Components
mirror/efi COMPLETE da0p1
da1p1
mirror/root COMPLETE da1p2
da0p2
Create mirrors for all other partitions of da0, which are now not mounted:
# gmirror label -vb round-robin swap da0p3 # gmirror label -vb round-robin var da0p4 # gmirror label -vb round-robin tmp da0p5 # gmirror label -vb round-robin usr da0p6
Edit /etc/fstab and change each remaining da0 partition into its mirror:
# cat /etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/mirror/swap none swap sw 0 0 /dev/mirror/root / ufs rw 1 1 /dev/mirror/efi /efi msdosfs rw 0 0 /dev/mirror/tmp /tmp ufs rw 2 2 /dev/mirror/usr /usr ufs rw 2 2 /dev/mirror/var /var ufs rw 2 2 /dev/acd0 /cdrom cd9660 ro,noauto 0 0
Reboot.
Add remaining da1 partitions to mirrors:
# gmirror insert swap da1p3 # gmirror insert var da1p4 # gmirror insert tmp da1p5 # gmirror insert usr da1p6
Done!:
# gmirror status
Name Status Components
mirror/efi COMPLETE da0p1
da1p1
mirror/root COMPLETE da0p2
da1p2
mirror/swap COMPLETE da0p3
da1p3
mirror/var COMPLETE da0p4
da1p4
mirror/tmp COMPLETE da0p5
da1p5
mirror/usr COMPLETE da0p6
da1p6
If the system boots up to a prompt similar to:
ffs_mountroot: can't find rootvp Root mount failed: 6 mountroot>
Reboot the machine using the power or reset button. At the boot menu, select option six (6). This will drop the system to a loader(8) prompt. Load the kernel module manually:
OK? load geom_mirror OK? boot
If this works then for whatever reason the module was not being loaded properly. Check whether the relevant entry in /boot/loader.conf is correct. If the problem persists, place:
options GEOM_MIRROR
in the kernel configuration file, rebuild and reinstall. That should remedy this issue.
In that case the system boot process would stop with a prompt similar to:
Loader variables:
vfs.root.mountfrom=
vfs.root.mountfrom.options=
Manual root filesystem specification:
<fstype>:<device> Mount <device> using filesystem <fstype>
eg. ufs:/dev/da0s1a
eg. cd9660:/dev/acd0
This is equivalent to: mount -t cd9660 /dev/acd0 /
? List valid disk boot devices
<empty line> Abort manual input
mountroot>
You would get this message most probably if you forget to add the correct boot device in /boot/loader.conf. The hints given by the system are clear. If your root device is /dev/mirror/root, and the filesystem is ufs, enter:
mountroot> ufs:/dev/mirror/root
The boot process would then continue:
Trying to mount root from ufs:/dev/mirror/root
When the system is up remember to edit your root device in /boot/loader.conf.
The wonderful part about disk mirroring is that when a disk fails, it may be replaced, presumably, without losing any data.
Considering the previous RAID1 configuration, assume that da1 has failed and now needs to be replaced. To replace it, determine which disk has failed and power down the system. At this point, the disk may be swapped with a new one and the system brought back up. After the system has restarted, the following commands may be used to replace the disk:
# gmirror forget gm0
# gmirror insert gm0 /dev/da1
Use the gmirror status command to
monitor the progress of the rebuild. It is that simple.
This, and other documents, can be downloaded from ftp://ftp.FreeBSD.org/pub/FreeBSD/doc/.
For questions about FreeBSD, read the documentation before contacting <questions@FreeBSD.org>.
For questions about this documentation, e-mail <doc@FreeBSD.org>.