Ubuntu Server & AMI MegaRAID Express 300 Insanity
I picked up some new (used) goodies yesterday quite cheaply from a friend.
One of those, was a new 2U server box sporting a hardware RAID controller that supposedly linux supports. Excellent. Even better, this box is quieter than the one in the rack. Soooo… I’m hoping to (someday soon) migrate the existing server (Moose) to the new server (Squirrel) and have some hardware RAID goodness.
This will open up some nice opportunities.
So Linux supposedly supports this MegaRAID device with the legacy “megaraid” driver. Booting from the ubuntu-server installation the linux-generic kernel properly initialized the RAID device, the whole thing installed cleanly, and the world was a very happy place.
Then I rebooted to the new install. GRUB loads just fine, the kernel loads, then… nothing. If you’re in rescue mode, you can see the errors coming from the kernel that it can’t find the disk by ID… the ‘megaraid’ module isn’t loading.
Poop.
After hours of trying to figure out how to insmod to the linux-server kernel when booting from the linux-generic kernel on the install cd, I got a great idea. Switch to the linux-generic kernel with the ubuntu-server distribution on top.
Here’s how you do it.
Install the server distribution to your disk. The install disk will boot with kernel-generic, and the MegaRAID should be working properly.
Boot with the install CD to a rescule console. Open a new shell with your root device (in my case, sda1).
In the command line, issue the following commands.
apt-get update
apt-get install linux-generic
apt-get remove linux-server
apt-get remove linux-image-2.6.20-15-server
exit
Reboot the computer without the CD. You should be all set.
July 30th, 2007 at 10:25 am
Lovely post, although it all goes completely over my head.. luckily I have perfected the “smile and nod” response. It’s Wendy by the way.. been awhile I know… but my reason for writing is that I currently work at H-P Products and happened to be down in the MIS department this morning. Chris Slackford played me the “This is John (Greenfield), did someone just page me” from rock 107 cage match clip and then Tana was like yeahhh varner someone set that up. At this point, I was like BRYAN VARNER? and promptly decided I needed to say hi and see what’s up. Plus, I find it amusing that you use to work at MIS (there’s a hot chick working at your desk now).. not sure if I knew this before or not, but if I did, I promptly forgot it.
Aight, that’s it for now. Hit me up on aim or email me if you’d like. Have a good one!
August 1st, 2007 at 10:57 am
You could always use /usr/src/linux and make menuconfig. :P
Let me know how that runs. I want to build a NAS box.
I for one have not ventured very far into the wonderful world of raid. Would you prefer hw or sw raid for a centralized NAS used for laptop and desktop backups? I’m thinking of doing Raid-1 or 5 which both are supported by the Linux kernel, and then to decide the FS’s for it. Which FS did you use?
August 1st, 2007 at 3:04 pm
In response to cd /usr/src/linux; make menuconfig;
I don’t want to have to deal with manually updating my kernel every time a new patchset or point release is made for my distribution. I want to plug the box in, walk away from it, and have it automatically pull security / functionality updates for any installed packages. I have a very good reason for this.
Now, on to the RAID recommendations:
Software RAID does work.
In fact, the sever this is being served from is using software RAID-5.
But, it has it’s catches and problems that I’m really not happy with.
For example: You cannot use a RAID-5 array as a boot volume. The only software RAID you can boot from is RAID-1. To get around that, I created a 100MB partition at the end of disc 0 in my array, and set it’s mount point to /boot. Every other disc now has a 100MB hole at the end of the disc as well, so I’m wasting available disk space. If the disc with /boot ever gets whacked, I’m going to have to boot from another rescue disc, run the array in degraded mode, (no hot-spare) backup everything, and hope that nothing gives out while I’m doing that.
If you RAID-5 you really ought to have a hot spare, so you’ll need at least four discs.
Software RAID-5 eats a little bit of CPU while it’s figuring out how to write the data to disc (gotta love striped data + parity locations). Typically, this is less than 0.5% of a single 733mhz P3 on this box. It peaks around 1% when I’m really putting it under load (multi gigabyte file copies).
Software RAID-1 operates nicely with two discs, lets you boot, and doesn’t eat CPU time while it figures out how to stripe the data across the drives. It simply duplicates the bus commands to a redirected target. Minimal overhead, good performance.
Hardware RAID eliminates every problem software RAID introduces, and has a few very nice extras to boot.
Hardware RAID shows up as a single device to the OS. You format it, and you’re good to go. You can boot from it, with any RAID level. Array rebuilding (recovery from disc failure) is faster, less intrusive, and you won’t notice the degraded performance during a rebuild nearly as badly. Oh, and Hardware RAID cards come with with a nice, loud alarm so you know when a disc fails.
Hardware RAID also totally eliminates the CPU overhead. All the read/write commands are handed off to the RAID card. In addition, if you’ve got a good RAID controller, chances are it comes with on-board RAM. This is where hardware RAID gets totally insane.
You can enable read-ahead caching at the card level, so that if you’re dealing with RAID-0 or RAID-5, which can suffer from slow read performance due to disc seek times, it’ll go ahead and read the next block and cache it in the RAID controllers RAM. If the OS requests it, it doesn’t have to wait for the disc to seek and read.
You can also enable write caching. This does great things to your drive performance & may improve the MTBF. The RAID controller will re-sort pending read / write operations according to physical proximity on the disc. Letting the drive heads swing nicely back and forth, reducing the amount of Tourettes-like twitching across the platter.
If you can afford it, get a good RAID controller. If you want redundancy cheap, get a couple of big discs and go software RAID-1.
The setup I have right now on the ubuntu server at home is RAID 1 + 0.
http://en.wikipedia.org/wiki/RAID#Common_nested_RAID_levels