|
page 1 (home) page 2 (back)
Use 'cat /etc/fstab' and make sure all your md devices are shown. If they are not, something went wrong. You might consider starting over from the beginning: cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/md2 / ext3 defaults,errors=remount-ro 0 1 /dev/md0 /boot ext3 defaults 0 2 /dev/md1 none swap sw 0 0 /dev/hdd /media/cdrom0 udf,iso9660 user,noauto 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0Use 'cat /proc/mdstat' to see what is going on. cat /proc/mdstatOutput on my machine:
Personalities : [raid0] [raid1] [raid5]
md0 : active raid1 sda1[0] sdb1[1]
240832 blocks [2/2] [UU]
md1 : active raid1 sda2[0] sdb2[1]
682688 blocks [2/2] [UU]
md2 : active raid1 sda3[0] sdb3[1]
16747648 blocks [2/2] [UU]
unused devices: <none>
This shows both drives are up ([UU]) for all three md devices. At this point if drive
/dev/sdb failed (pie), we would continue to function. We could shut
down, replace pie with (a cleaned) spare,
and we would still be able to boot up using /dev/sda (apple).
Since we have a bootable system, we could get the spare drive back on
line with a few well chosen commands. Sounds like everything is OK then, right?
Well it is if /dev/sda
is never the one that fails. So our immediate challenge is to get a GRUB MBR installed on
/dev/sdb, so if /dev/sda fails, we can boot from the secondary drive. Grub
sees our secondary drive as (hd1) and we have set the boot flag on the first
partition of that drive. Grub starts counting from zero, so it sees that partition
as 0. We will install grub on (hd1,0).
We are going to reconfigure grub's menu.lst. Start by making a backup copy:
cp /boot/grub/menu.lst /boot/grub/menu.lst-backup
Edit menu.lst: vi /boot/grub/menu.lst
We are going to add a new menu item that tells grub to boot from our secondary drive. Make a duplicate of your existing top menu stanza, place the duplicate above the existing stanza, and modify it in the same manner I have. This example shows partition 0 is the partition flagged as bootable on my system. You can run something like 'fdisk -l /dev/sda' to determine which partition is bootable on your system but your original stanza will be correct: title Debian GNU/Linux, kernel 2.6.8-2-386 (hd1) root (hd1,0) kernel /vmlinuz-2.6.8-2-386 root=/dev/md2 ro initrd /initrd.img-2.6.8-2-386 savedefault boot title Debian GNU/Linux, kernel 2.6.8-2-386 root (hd0,0) kernel /vmlinuz-2.6.8-2-386 root=/dev/md2 ro initrd /initrd.img-2.6.8-2-386 savedefault bootSave and exit the file. Just a note: because we mount a /boot partition, you will not see the above entries in the form "/boot/initrd.img-2.6.8-2-386". If you do not mount a separate /boot partition, you will see the entries in that form. To install grub on both hard drives, start the grub command shell:
grub
At the 'grub>' shell prompt create new boot records for both drives:
root (hd0,0)
OK. We can test by rebooting. Do this at the console so if should fail you can simply choose another item from the menu.:
reboot
If all went well, once again we will edit menu.lst: vi /boot/grub/menu.lst
We will add a fallback entry that will (hopefully) automatically choose the next item in the menu if the first item fails. So, just below "default 0", add this entry: fallback 1
OK, now I'm going to simulate a failed drive. I don't recommend you try this (your system may explode), but at least you can learn from my system. I am carefully going to remove the power cable from the primary drive, apple. Once I do this, it will be "dirty" and should not be used again in this system without first being cleaned. This is what 'cat /proc/mdstat' shows as a result: sda1 and sda2 still show they are up because we have not had any read/write operations on them recently, sda3 shows it has failed.
md0 : active raid1 sda1[0] sdb1[1]
240832 blocks [2/2] [UU]
md1 : active raid1 sda2[0] sdb2[1]
682688 blocks [2/2] [UU]
md2 : active raid1 sda3[2](F) sdb3[1]
16747648 blocks [2/1] [_U]
If your hardware supports hot swappable drives I think you should mark the remaining
two devices faulty, then use mdadm (multi-disk administrator)
to remove all three faulty devices from our array
before inserting the new drive. You cannot use "mdadm --remove" on drives
that are in use, so they need to be set as faulty first.
You do not need to do this if you are going
to power down the system and replace the drive with a clean drive.
Make doubly sure you are failing the partitions on the
drive that has failed!
mdadm --set-faulty /dev/md0 /dev/sda1Shut it down: shutdown -h now
To clean a drive, I install it in a computer by itself, boot up using a DBAN disk and change the Method to Quick. If you would like a tomsrtbt disk, insert a blank floppy, then: cd /usr/local/src
For consistency (to keep my sanity) I always move the good drive to the primary position (if it is not already there) and place the new clean drive in the secondary position. We have shut down, so disconnect the good drive, clean apple, move pie (the good drive) into the primary position, place the cleaned apple in the secondary position and bring the system back up. On my system all I have to do to swap the two SCSI drives is to move the jumper from one drive to the other. OK, my system did boot up. First we see what's going on (cat /proc/mdstat). As you can see, sdb1, sdb2 and sdb3 are missing:
md0 : active raid1 sda1[1]
240832 blocks [2/1] [_U]
md1 : active raid1 sda2[1]
682688 blocks [2/1] [_U]
md2 : active raid1 sda3[1]
16747648 blocks [2/1] [_U]
We start by copying the partition structure from /dev/sda to /dev/sdb. We do this for what
should now be an obvious reason: the secondary drive is empty, but it needs to have the same
structure as the primary drive. If the disk was first cleaned, and is large enough,
you should have no errors:
sfdisk -d /dev/sda | sfdisk /dev/sdb
To further clean the drive we make sure the superblocks are zeroed out on the new drive (be careful you do this to the correct drive). Superblocks are used by RAID for identification purposes. Edit as needed:
mdadm --zero-superblock /dev/sdb1
Now we add our three sdb partitions to the corresponding md's. Understand what you are doing here before you do it, edit as needed:
mdadm --add /dev/md0 /dev/sdb1
They should start to synchronize. To watch them sync: watch -n 6 cat /proc/mdstat
Of course it's [Ctrl]+c to cancel 'watch'. NEVER reboot during synchronization. Once the recovery is complete (and not until then), create a new boot record on our secondary (replacement) drive:
grub
From the grub> prompt (edit partition number if needed):
root (hd1,0)
We are working again. You might want to reboot from the console to make sure you actually boot from the secondary drive. In A RAID system it is a good idea to avoid kernel version upgrades (security upgrades should be performed of course). If you are currently running Linux kernel 2.6.8 or older, installing a kernel newer than 2.6.8 may replace devfs with udev. If this happens you could be in big trouble. I have not been able to repair a system once it migrates from devfs to udev. If you are already using udev, upgrading the kernel to a newer kernel that also uses udev should not be an issue. Some of the new 2.6 kernels (from 2.6.12) no longer use mkinitrd to create the initrd.img. The set of programs now used to create the ramdisk image (initramfs-tools - run 'man mkinitramfs') for some reason may not create an initrd.img that is able to boot into our md devices. As a result, after an upgrade to one of the newer kernels, your system may not boot to the new kernel. This bug may be resolved at some point in the future (and may be resolved when you read this) but to work around the problem I was able to use mkinitrd to create the ramdisk image. The use of mkinitrd is deprecated when used with kernels 2.6.12 or newer. I don't know why this worked for me. I would not be surprised if it does not work for you. I am using Sarge but as a test I will install a kernel from 'testing' (Etch) to illustrate. I installed a 'testing' source in /etc/apt/sources.list and ran 'apt-get update'. We need a newer version of initrd-tools that can be used with the newer kernel:
apt-get -t testing install initrd-tools
Now I determine the correct kernel for my architecture:
apt-cache search linux-image
I am going to install: apt-get install linux-image-686
Running this command installed version 2.6.15-1-686 on my system. At this point the kernel may not boot into our md devices. You can try if you like to see if the bug has been fixed. If not then you will have to boot up using the old kernel. Go ahead and reboot. If it fails to boot using the new kernel then we will again compile a new /boot/initrd.img using mkinitrd. Here I create a backup copy of the current initrd.img, compile a new image to a temp file using the new kernel modules (/lib/modules/2.6.15-1-686/), then copy the temp file on top of the current initrd.img as we did before:
cp /boot/initrd.img-2.6.15-1-686 /boot/initrd.img-2.6.15-1-686-backup
mkinitrd -o /boot/initrd.img-2.6.15-1-686-temp /lib/modules/2.6.15-1-686/
We are now going to reconfigure grub once again.
vi /boot/grub/menu.lst
Make a duplicate of your existing top menu stanza, place the duplicate above the existing stanza, and modify it in the same manner I have (so we boot off of hd1): title Debian GNU/Linux, kernel 2.6.15-1-686 (hd1) root (hd1,0) kernel /vmlinuz-2.6.15-1-686 root=/dev/md6 ro initrd /initrd.img-2.6.15-1-686 savedefault boot title Debian GNU/Linux, kernel 2.6.15-1-686 (hd0) root (hd0,0) kernel /vmlinuz-2.6.15-1-686 root=/dev/md6 ro initrd /initrd.img-2.6.15-1-686 savedefault bootSave and exit the file, then start up the grub command prompt: grub
at the grub> prompt enter these commands to reinstall grub on the secondary drive (edit partition number if needed):
root (hd1,0)
Reboot to make sure it works. The Debian installer did a lot of work for us when we created our RAID devices. To appreciate the difficulty of building a RAID array on a system that is currently running and is not configured as one please read http://verchick.com/mecham/public_html/raid/raid1-degraded.html. The introduction substantially duplicates this document but the remainder explains important additional aspects. Start reading at the paragraph just above the illustration. References (alphabetical order). Not all of these are good, but all were interesting to me in one way or another. Trust me, there are a lot more documents similar to these out there: http://alioth.debian.org/project/showfiles.php?group_id=30283&release_id=288 http://deb.riseup.net/storage/software-raid/ http://forums.whirlpool.net.au/forum-replies-archive.cfm/471585.html http://nepotismia.com/debian/raidinstall/ http://nst.sourceforge.net/nst/docs/user/ch14.html http://piirakka.com/misc_help/Linux/raid_starts_degraded.txt http://thegoldenear.org/toolbox/unices/server-setup-debian.html http://togami.com/~warren/guides/remoteraidcrazies/ http://www.debian-administration.org/articles/238 http://www.debian-administration.org/users/philcore/weblog/4 http://www.doorbot.com/guides/linux/x86/grubraid/ http://www.epimetrics.com/topics/one-page?page_id=421&topic=Bit-head%20Stuff&page_topic_id=120 http://www.james.rcpt.to/programs/debian/raid1/ http://www.linuxjournal.com/article/5898 http://www.linuxsa.org.au/mailing-list/2003-07/1270.html http://www.linux-sxs.org/storage/raid_setup.html http://www.parisc-linux.org/faq/raidboot-howto.html http://www.tldp.org/HOWTO/Software-RAID-HOWTO.html http://trinityhome.org/misc/bootable-raid1.html http://www.vermittlungsprovision.net/367.html http://xtronics.com/reference/SATA-RAID-debian-for-2.6.html |