Emidio Planamente 's Homepage

Home GNU/Linux Programming Technology Sport Others Contact

Search

  Google linux


Debian

  Apt-get
  Debian Anatomy
  Debian Backports
  Debian Help
  Debian Planet
  Debian Swiss
  History of Debian
  Getdeb
  Kernel
  Kernel 2.6
  Packages
  Refcard for newbie
  Reference book
  SATA raid
  Weekly News


Official Docs

  Distrowatch
  Firewire
  Gimp
  Gimp Photo Archive
  Linuxdoc
  Linuxdoc Reference
  Linux Focus
  Linux From Scratch
  Linux Hacks
  Linux Hardware
  Linux Printing
  MySQL
  O'Reilly
  Rute
  Source Forge
  USB guide


Installation

  Instalinux
  Preseed


Laptop

  Linux on Laptops
  Linux Toshiba
  Tux Mobil


Live-CDs

  Create it Your self
  Knoppix
  Kororaa XGL 3


Pictures

  Bay13
  Gnomelook
  Kuznetsov


Security

  GNU/Linux
  PortsDB


Linux based product

  Dreambox


Free web service

  S5 presents


Against Micro$oft

  Combatti Micro$oft
  Microsuck


HTML validator CSS validator

 

Root on LVM on software RAID howto [Debian Sarge]


1. Introduction

2. Requirements

3. Installation

4. Using self compiled kernel

5. Tips and tricks

6. Trouble shooting

7. Testing

8. About


1. Introduction


This howto briefly describes how to install Debian Sarge on LVM2 on software RAID 1, directly from official Sarge DVD (without having to install it on a classical partition and then move to LVM on RAID).


Two words before starting.

This task is not something for beginners but for advanced users.

Don't forget to make a backup of your datas (possibly on an external HD that will be unplugged before starting the installation, or on a CD/DVD).


1.1. Restrictions


1.1.1. Restriction one


There is a limitation installing the system on LV: the "vgscan" and "vgchange" commands must be available to the system in order to mount LVs.


There are the following possibilities to accomplish this:


1) Use of an initrd image.


2) Don't install the whole system on LVs.


With the first solution, the whole system can be placed on LVs (either in a single LV or in more LVs). I'm not a fan of initrd images, but this is the easiest way.


With the second solution, the /sbin containing the vg tools must be placed outside the LVs, in a non-LVM partition.

At least other directories must be also outside the LVs (e.g.: /, /etc, /dev/, ...). The great advantage is that only the necessary LVs are mounted read-write, all the rest are mounted read-only!


Since the second solution is the right way to configure a server, but is a little bit expensive for a desktop PC, I only treat the first one in this document.


1.1.2. Restriction two


This resctriction is caused by booting a system installed on LVM.


The /boot can not be in LVM because the bootloader cannot read it!


1.2. Layout


Saying we have 2 IDE harddisks: dev/hda and /dev/hdb. Each one will be the mirror of the other.


Since /boot have to be in non LVM partition, 2 RAID devices are needed: /dev/md0 and /dev/md1.

The first for /boot, the second for LVM (/, /home, ...).


Therefore, this is the situation:


  /dev/hda1   /dev/hdb1   used for /dev/md0

  /dev/hda2   /dev/hdb2   used for /dev/md1


  /dev/md0  used for /boot

  /dev/md1  used for LVM


For the volume group name I have choosen "raid", but you can use whatelse do you prefer.

For the logical volume name of the / I have choosen "sarge", but you can use whatelse do you prefer.


  /dev/raid/sarge  used for /


2. Requirements


2.1. Hardware


The minimal requirements are:


- 2 harddisks (it doesn't make sense to have raid 1 on the same harddisk!)


2.2. Software


The minimal requirementes are:


- kernel 2.6 (maybe works also with 2.4, but the document is based on 2.6)

- lvm2

- mdadm


3. Installation


3.1. Booting the DVD


Power on with the Sarge DVD1 in the DVD reader.


Once Debian logo appears, type


  expert26


This will choose a 2.6 kernel.


3.2. Installing the system


Select the following items and configure it as normally:


  "Choose language"

  "Choose country or region"

  "Select a keyboard layout"

  "Detect and mount CD-ROM"


Now, the first important step for our purpose. Select


  "Load installer components from CD"


and then mark the following items:


  lvmcfg

  mdcfg


This will load full (kernel + applications) LVM and RAID support.


Proceed as normally with:


  "Detect network hardware"

  "Configure the network"

  "Detect hardware"


3.2.1. Creating physical partitions


Now, the second important step for our purpose: Select


  "Partition disks"


then


  "Manually edit partition table"


With the following steps, the content of the harddisk will be permanently erased. Have you made the backup? If not, run Forrest!


Create /dev/hda1 and /dev/hdb1 with a size of 64M, type 0xFD (Linux RAID autodectection) and bootable flag on.


Create /dev/hda2 and /dev/hdb2 with the remaining size and type 0xFD (Linux RAID autodectection).


These partitions will be used for the 2 RAID1 devices.


3.2.2. Creating RAID devices


Select


  "Configure software RAID"


You will asked for writing to the storage devices. Answer yes.


Create /dev/md0 using /dev/hda1 and /dev/hdb1.


This device will be used for /boot.


Create /dev/md1 using /dev/hda2 and /dev/hdb2.


This device will be used for LVM.


Now, the RAID arrays have been created and are currently syncronizing them self. You can verify this going on the second console (ALT + F2) and repeatedly calling


  cat /proc/mdstat


It would be nice to simply call


  watch -n 1 cat /proc/mdstat


but this last has not been integrated in the Debian installer.


3.2.3. Creating LVM devices


In the "Partition disks" window, select the just created RAID1 device of xxM (RAID1 device #1 / #1) and then:


Use as: physical volume for LVM


Select


  "Configure the Logical Volume Manager"


You will asked for writing changes to disks. Answer yes and ignore eventually warning about the kernel.


Create volume group "raid" on /dev/md1.


Create logical volume "sarge" with a size of 5.5G and of group "raid". This will be used for /.


Depending on your configuration, create additional volumes in the group "raid" for /home, /var/www, ...

If you prefer, you can do this later, but the process will be a little bit more complicated.


3.2.4. Creating /


In the "Partition disks" window, select the just created LV device of 5.5G (LVM VG raid, LV sarge / #1) and then:


  Use as: ReiserFS journaling file system

  Mount point: /

  Mount options: notail

  Label: sarge


3.2.5. Creating /boot


In the "Partition disks" window, select the RAID1 device of 64M (RAID1 device #0 / #1) and then:


  Use as: Ext3 journaling file system

  Mount point: /boot

  Mount options: defaults

  Label: boot

  Reserved blocks: 5%

  Typical usage: standard


3.2.6. Creating additional LVs before rebooting


For each additional logical volume you have created before, repeat the same process like for the / partition.


Here an example for /home:


In the "Partition disks" window, select the LV device created for /home and then:


  Use as: ReiserFS journaling file system

  Mount point: /home

  Mount options: notail

  Label: home


3.2.7. Finish partitioning


Write all changes to disk and leave this step.


3.2.8. Installing the base system


You are now ready to proceed with the system installation.


Install it exactly as you would make without LVM and RAID. The only difference now, is that /target is mounted on /dev/raid/sarge and /target/boot is mounted on /dev/md0. You can verify this by going on the second console (ALT + F2) and typing


  mount


When you are asked for the kernel to install, choose the proposed one:


  kernel-image-2.6.8-686-2


3.2.9. Installing the boot loader on the first harddisk


I prefer GRUB instead of LILO, because it is newer, more flexible and has some nice features like upgrading it self when a new kernel is installed/removed.


If you want, you can try with LILO, but at your own risk.


Once promped if installing GRUB in the MBR, answer yes.


Note the boot loader has only be installed on the first harddisk. In case this harddisk has a failure, you will not be able to boot the system. Therefore, the boot loader must also be installed on the second harddisk. Since this step cannot be accomplished here, you have to do it later.


A detailed documentation about GRUB can be found here: http://www.gnu.org/software/grub/manual/html_node/index.html.


3.2.10. Rebooting


Not sure if really needed, but for safety, wait until RAID has finished resyncing by repeatedly calling


  cat /proc/mdstat


on the second console (ALT + F2).


When done, switch back to the first console (ALT + F1) and select "Finish the installation" to reboot.


If your system does not boot correctly, you have probably done something wrong in one of the steps above.

In this case, refer you to chapter "Trouble shooting".


3.2.11. Installing the boot loader on the second harddisk


Since the bootloader has been installed only on the first harddisk, you have to manually install it on the second one.

Therefore, once the system is up:


  grub

  device (hd0) /dev/hdb

  root (hd0,0)

  setup (hd0)


Logically, replace the device path with them of your second harddisk!


3.3. Installing and configuring additional packages


You can now proceed installing all kind of packages do you need and logically configuring them.


3.4. Creating additional LVs after rebooting


If you have already done this step before, just skip to the next chapter.


Don't forget to create all the LVs you need (e.g.: /home, /var/www, ...) and to update the /etc/fstab file.


Logically, before mounting them with the correct mount point, move the content to the LV.


Here a complete example for /home.


3.4.1. Creating /home


Create LV of 2G:


  lvcreate -A -n home -L 2G raid


Create filesystem:


  mkreiserfs --label home /dev/raid/home


Temporarily mount /dev/raid/home and transfer the content of /home:


  mount /dev/raid/home /mnt

  mv /home/* /mnt

  umount /mnt


3.4.2. Updating /etc/fstab


Add an entry for /home:


  /dev/raid/home  /home  reiserfs  noatime,notail


3.4.3. Mounting /home


  mount /home


4. Using self compiled kernel


4.1. Introduction


A self compiled kernel is not a MUST, but it could be necessary for performance optimization.

In my case, for example, I would hyperthreading enabled for my Pentium IV, but there is not such an official kernel.


In such a case, it is important to create the initrd image, otherwise the system will not boot.


4.2. Compilation


In order to have a initrd image, it's enough to specify the parameter


  --initrd


Therefore:


  fakeroot make-kpkg --append_to_version -yourHost --initrd --revision=yourRevisionNumber kernel _image modules_image


Don't panic! This will not create a package with the initrd image! But after the installation the initrd will be in /boot.

I think, it is created during the installation it self.


4.3. Installation


Just install your self compiled kernel:


  dpkg -i /usr/src/kernel-image-2.6.x-yourHost_yourRevisionNumber.deb


If you have followed my suggestion and have installed grub, you are ready to reboot and testing your self compiled kernel.


5. Tips and tricks


5.1. Making backup of RAID configuration


In case the system becomes unbootable, you need the RAID configuration to be sure to can start the RAID in every condition.


Therefore, once the system is working, type:


  cd /etc/mdadm

  echo 'DEVICE /dev/hd*[0-9] /dev/sd*[0-9]' > mdadm.conf

  mdadm --detail --scan >> mdadm.conf


Now, make a backup of this file out of your RAID system!!!. It is very important if you have to solve a problem in your system using a rescue CD/DVD.


5.2. How to prevent the system becomes unusable


This system is safe enough from failure, but it is not safe enough against you! ;-)


It is very easy to break the system trying something new, maybe installing a kernel without the currect support (LVM, RAID, ...) and having the same name of the working kernel. In this way, the working kernel will be removed and the buggy kernel installed.


To prevent you cannot access your system if it becomes unbootable, procure you a rescue system.


5.2.1. Rescue CD/DVD


Procure you a rescue CD/DVD with kernel 2.6, LVM2 and RAID support.


You can download my Emi's rescue CD here: http://emidio.planamente.ch/rescuecd.


If you prefer, you can use the Debian DVD it self, but it needs much more time until you can have access to your damaged system.


5.2.2. Rescue system


An alternativ solution to the rescue CD/DVD is to install a rescue system on an other HD (maybe an external one).


Be sure to install it in a way you can always boot it, for example on a little partition on the first harddisk.


Be also sure to have installed all what is needed (kernel + tools).


5.3. Backup


Don't forget, data and system backups are always important, also if you use RAID system. An erased file is erased for the whole array and this is irreversible!!!


Take a look at http://www.planamente.ch/emidio/pages/linux_howto_backup.php.


6. Trouble shooting


6.1. Duplicate PV


By creating LVM on RAID 1 device, it could happen PVs are not created on the RAID device but on the physical partitions (e.g.: /dev/sda1 and /dev/sdb1 if there are part of /dev/md0).

This will result as an error by doing LVM scanning (pvscan, vgscan, ...) like


  Found duplicate PV 9w3TIxKZ6lFRqWUmQm9tlV5nsdUkTi4i: using /dev/sda1 not /dev/sdb1


and will make impossible to create a new volume group on /dev/md0.


In this case, you have to create a filter for LVM.

In the /etc/lvm/lvm.conf file, you have to add such a line:


  filter = [ "r|/dev/cdrom|","r|hd[ab]|" ]


Logically, replace  "hd" with "sd" for SCSI devices and "ab" with the correct one.


This will prevent such devices are scanned or used for LVM.


You have to reload LVM with


  /etc/init.d/lvm force-reload


but since your system is on LVM, this is not possible. Therefore you have to reboot.


6.2. System hangs up


The current description is for using the Sarge DVD1 as rescue DVD.


Boot with Sarge DVD1.


6.2.1. Starting RAID


Once the rescue system has booted, the RAID devices are not started yet, because the md driver (raid driver) is compiled as module and not built in.

I have taken a look in the source code and it seems the raid autodetection is explicity disabled if the driver is compiled as module. Don't ask me why.


Therefore, you have to know your exactly RAID configuration: using the mdadm.conf file of just your head!


Load drivers:


  modprobe md raid1


Assemble devices, without configuration file:


  mdadm --assemble /dev/md/0 /dev/scsi/host0/bus0/target0/lun0/part1 /dev/scsi/host0/bus0/target1/lun0/part1

  mdadm --assemble /dev/md/1 /dev/scsi/host0/bus0/target0/lun0/part2 /dev/scsi/host0/bus0/target1/lun0/part2


Assemble devices, with configuration file:


  mdadm --assemble --scan --config=myConfigFile


Logically, in both cases, replace the partition paths with yours.


If the arrays are not degraded, they should also automatically be started. If not, do it:


  mdadm --run /dev/md/0

  mdadm --run /dev/md/1


Verify they have been started:


  cat /proc/mdstat


6.2.2. Starting LVM


The logical volumes are easier to start.


Load device mapper driver:


  modprobe dm-mod


If you forget to load this driver, you will get a terrible error like:


  /proc/misc: No entry for device-mapper found

  Is device-mapper driver missing from kernel?

  Failure to communicate with kernel device-mapper driver.

  Incompatible libdevmapper 1.01.00-ioctl (2005-01-17)(compat) md kernel driver


Don't panic, just load dm-mod!!!


Search and activate all volume groups:


  vgscan

  vgchange -a y


Verify all LVs are active with:


  lvscan


6.2.3. Mounting broken system


Make mount point


  mkdir /target


and mount / of the broken system:


  mount /dev/raid/sarge /target


6.2.4. Changing root


This is one of the most interessting command of GNU/Linux: chroot.


This command will change the root in the shell where it has been invoked. Therefore, call


  chroot /target


and you will transfered in the broken system. If you type


  ls -l /


you will see the content of /target, but all what is outside it is absolutely not visible.


6.2.5. Mounting /proc


In order to restore your broken system, you have to mount the /proc directory, otherwise your kernel won't have info about the chrooted system.


Therefore, just type:


  mount /proc


6.2.6. Mounting /boot


Since the /boot is not in the same partition, it has to be mounted. Therefore, call


  mount /boot


6.2.7. Solving the problem


Here I can't help you a lot. There could be milion of problems and it is your responsability to solve it. I have warned you. Root on LVM on RAID is not for everyone! ;-)


If you didn't follow my instructions and installed LILO instead of GRUB, don't forget to call


  lilo


when you are done.


6.2.8. Unmounting /proc


In theory, you are ready to reboot, but you have to exit from the chrooted directory before.


Since normally it is important to unmount /proc before exiting, we do it (just for pedagogical level) also if in this case it would not be necessary, because you want to reboot and not work anymore on the rescue system.


Anyway, type:


  umount /proc


6.2.9. Exiting


Now, we are ready for leaving the chrooted environment.


Just type:


  exit


6.2.10. Rebooting


Finally, we can reboot and hope the problem is solved.


7. Testing


7.1. Simulating disk failure


7.1.1. Second disk fails


I have made this test by physically removing the second HD from my system (PC was turned off!!!) and booting.


The system booted without any problem and once logged in, I have received a mail informing me there was a degraded array.


Taking a look at


  cat /proc/mdstat


I had confirm.


Interessting thing, is that after have reputting the second drive on the system, the system still says the array is degraded.

I had to manually readd the second harddisk by calling:


  mdadm --add /dev/md0 /dev/hdb1

  mdadm --add /dev/md1 /dev/hdb2


and wait until resync was done, to restore the original situation.


In case of a real failure, the MBR has to be reinstalled after have replaced the broken disk:


  grub

  device (hd0) /dev/hdb

  root (hd0,0)

  setup (hd0)


7.1.2. First disk fails


I have made this test by physically removing the first HD from my system (PC was turned off!!!) and booting.


The system could not boot, but not because the RAID was not working but for the following reason.

I have both SCSI and IDE harddisks and the system is installed on the SCSI.

The BIOS can map the sequence of all the harddisk. By default, it assigns IDE before SCSI, but I have changed this order.

If the BIOS detects a change in the harddisk configuration, it reassigns the position of the devices by putting the IDE before SCSI.

In this case, the system becomes unbootable. I had just to reassign the correct order and the system booted correctly.


Also in this case, after a few minutes, I received a mail from the system , informing me that the array was degraded.

Taking a loot at


  cat /proc/mdstat


I had confirm.


Note: In this case, the second harddisk is called /dev/hda and not /dev/hdb, because the first one is missing!!!


After shuttind down, reputting the first harddisk and booting, I could re-add it to the array with:


  mdadm --add /dev/md0 /dev/hda1

  mdadm --add /dev/md1 /dev/hda2


Note: Now, the second harddisk is again called /dev/hdb, because the first one has been readded!!!


After a few minutes, the raid has been restored.


In case of a real failure, the MBR has to be reinstalled after have replaced the broken disk:


  grub

  device (hd0) /dev/hda

  root (hd0,0)

  setup (hd0)


7.2. Partition fails


One after the other, I have set all the single partitions making part of the array to faulty.

Every time I have set only one partition to faulty, I have verified the root has received a warning email and I have rebooted the system to verify if it still could come up.


To set a partition to faulty:


  mdadm --fail /dev/md1 /dev/hda2


To add the faulty partition to the array:


  mdadm --add /dev/md1 /dev/hda2


7.2.1. Data corruption


RAID has not been implemented against data corruption. If you try to simulate data corruption, you will have data corruption.


Therefore, don't try such a test unless you will really destroy your data!


8. About


8.1. Author


Emidio Planamente <eplanamente@gmx.ch>


8.2. Feedback


Please let me know if you could successfully install your system on LVM2 on RAID1 following this document description.


Any other feedback is also welcome.


8.3. History


Version 2.2 / 2006-02-06

  Changed "Creating RAID devices"

  Changed "Installing the boot loader on the first harddisk"

  Changed "Installing the boot loader on the second harddisk"


Version 2.1 / 2006-01-25

  Changed "Restriction one"

  Changed "Creating LVM devices"


Version 2.0 / 2006-01-22

  Changed "Rescue CD/DVD"


Version 1.9

  Changed "Creating RAID devices"

  Changed "Rebooting"


Version 1.8

  Fixed "Creating physical partitions"

  Fixed "Creating RAID devices"


Version 1.7

  Fixed "System hangs up"


Version 1.6

  Changed "Partition fails"


Version 1.5

  Added "Using self compiled kernel"


Version 1.4

  Changed "First disk fails"

  Changed "Second disk fails"


Version 1.3

  Changed "First disk fails"


Version 1.2

  Changed "Starting RAID" "without config file"


Version 1.1

  Changed "5.2 System hangs up"

  Changed "Installing the boot loader"

  Added "6. Testing"


Version 1

  First public release



Emidio Planamente

Last modified on 2006-02-06