Now that I created a CentOS system with RAID-5 and RAID-1in my previous post, it’s time to test whether the RAID will actually save my system in case of disk failure.
A simple way to simulate a disk failure without actually beating the crap out of one of your disks (which would be especially difficult with the virtual disks in this case) is by using the mdadm utility included in CentOS and other Red Hat-based linux distributions.
1) First, confirm that your RAID includes a spare drive that the system will rebuild the array with and determine which disk is going to “fail” by checking the output of ’cat /proc/mdstat’:
Personalities : [raid6] [raid5] [raid4] [raid1]
md1 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3](S)
104320 blocks [3/3] [UUU]
md0 : active raid5 sdd2[3](S) sdc2[2] sdb2[1] sda2[0]
20755456 blocks level 5, 256k chunk, algorithm 2 [3/3] [UUU]
unused devices: <none>
As you can see, I have split each of the 4 disks in my virtual machine (VM) into 2 partitions each – a small RAID-1 partition for /boot and a large raid-5 partition for / (everything else). The (S) next to some of them signifies that that partition is a Spare, to be used in case of disk/partition failure.
2) To simulate disk failure, we will pass ’fail’ argument to one of the partitions:
mdadm /dev/md0 --manage --fail /dev/sdb2
Make sure the partition you’re marking is part of the correct array, or it will just tell you it can’t find it.
The array should automatically start rebuilding at this point. You can check the progress by checking mdstat again:
[root@rsheyd ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md1 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3](S)
104320 blocks [3/3] [UUU]
md0 : active raid5 sdd2[3] sdc2[2] sdb2[4](F) sda2[0]
20755456 blocks level 5, 256k chunk, algorithm 2 [3/2] [U_U]
[>....................] recovery = 2.6% (278172/10377728) finish=6.0min speed=27817K/sec
unused devices: <none>
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
3) After you have verified that the array rebuild has completed, you now know your system is safe from virtual hammers! To re-add the virtually destroyed partition back, just run:
mdadm /dev/md0 --manage --remove /dev/sdb2 mdadm /dev/md0 --manage --add /dev/sdb2
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }You might have to reboot before being able to add the partition back into your array.
2 thoughts on “Testing RAID in CentOS”