From SIPfoundry sipXecs IP PBX, The Open Source SIP PBX for Linux - Calivia

Setting up a Test Environment for sipX HA using Xen

Introduction

In release 3.2 the sipX project is adding high-availability as one of the main new features. That means that two servers can be synchronized to run redundant proxy / registrar components, which avoids any single point of failure for SIP call routing functionality. In addition to high-availability this provides flexible scalability using DNS SRV for load balancing.

A test setup would typically have to consist of two independent servers. The following describes how the Xen Hyypervisor virtualization technology on Fedora Core 4 can be used to run the test setup on one physical server. We will be using three virtual machines (domU): One build server and the two redundant sipX servers.

HW Requirements: Sufficient memory is required to run two virtual machines with sipX. We recommend a minimum of 512MB per virtual machine and 128MB for dom0, which means a minimum system RAM of 1.5GB.

Setting up the Host OS (dom0) on Fedora Core 4

Xen is included in the standard FC4 base repository. However, we are not going to install it that way, because only select kernel pairs actually work and also because the FC4 repository still distributes Xen 2.2. Instead, we will install Xen as it is offered by Xensource.

Installing Fedora Core 4

The following describes how to install FC4 from scratch. Download the installation CD for Fedora Core 4 (only first CD required) and boot from it. The following configuration options are recommended:

Create disk partitions using LVM:

/boot partition: 256 MB
Physical LVM: All disk
Create "VolGroup00" using the rest of the available space
   Inside VolGroup00:
     swap partition: 1984MB
     /    partition: 12GB

Leave the rest of the disk (remaining capacity in VolGroup00) free. We will use this for virtual machines later.

Complete the setup as follows:

Fixed IP (recommended)
hostname.domain.com
no firewall
SELinux disabled
Customize software packages to be installed: Minimal system

Note: If you choose a dynamically assigned IP address (DHCP) create the file /etc/dhclient-eth0.conf as follows in order to enable dynamic DNS (DDNS) updates:

send host-name "hostname";

After installing FC4 login as root and run yum update.

Installing Xen 3.0

More detailed information is available in the release notes from Xensource. The Fedora Wiki can also be helpful, however, it describes an install process using the FC4 base repository, which is not suitable for our purposes. Below is a description of how we installed Xen 3.0 on FC4:

Note: We are using an install script provided by Xensosurce that installs a new yum repository from which all the necessary RPMs are downloaded. In addition, the script provides the necessary configuration. Alternatively, you can download the RPMs directly.

mkdir ~/xen
cd ~/xen
wget http://tx.downloads.xensource.com/downloads/xen-install-fc4
chmod +x xen-install-fc4

./xen-install-fc4 install

Note: The install script automatically configures Grub by adding a new entry to /boot/grub/grub.conf. Memory allocation is done automatically in Xen 3.0 and therefore no memory specification is added to the kernel line in the grub configuration file. The installer by default inserts the maxcpus=1 parameter into grub.conf. If you want to start dom0 with all available virtual CPUs you can remove it.

Now restart your system and boot into the Xen0 kernel. (Currently the provided kernel is vmlinuz-2.6.12.6-xen3_7.1_fc4).

Note: The same kernel image is used for the dom0 and domU machine. This is different as compared to Xen 2.x where a special domU kernel was required in addition to the dom0 kernel.

Post-Install Tasks and Verification

The xend deamon should be started automatically as the system comes back up. Verify that Xen0 (dom0) is operating correctly with the xm command.

xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0     2704     4 r-----   118.4

Manually reduce memory: Xen 3.0 uses a memory balloon driver that automatically frees memory for VMs (dom0 and domU) that do not use it and re-assignes it to VMs that need more. You can cap max memory that will be used by a VM with the command below. Actual memory assigned can be lower than that.

xm mem-max 0 384

Manually assigning virtual CPUs: In a system with Hyper Threading (HT) or multiple cores virtual CPUs can be assigned to VMs.

xm vcpu-set 0 2

Verify network connectivity and hostname / domainname settings: The host on which you plan to install sipX (dom0 or domU) needs to have a properly configured host and domain name as described in the sipX installation instructions. This name has to resolve in DNS.

Debugging:

xm dmesg

Note: During boot a warning is displayed "Currently emulating unsupported memory accesses in /lib/tls glibc libraries". After applying the suggested fix mv /lib/tls /lib/tls.disabled the warning still displays, so we chose to ignore it.

Installing a First FC4 Guest Operating System (domU)

We are going to install three guest operating systems, all running Fedora Core 4. In order to speed-up the creation of these virtual machines we will first create a template with a base configuration in an image file that then can be replicated and reused easily. The final virtual machines will run in their respective LVM partiions; however, the template will be image file based.

Note: Ready-to-use Images for different distributions are available from Jailtime.org.

Create the Image File

Create an image file (we chose a size of 1GB) to be used as the file system storage for the template FC4 guest operating system. The file acts as a virtual hard drive for the guest OS.

dd if=/dev/zero of=fc4-template.img bs=1M count=1 seek=1024

Format the image file creating an ext3 filesystem:

/sbin/mke2fs -F -j fc4-template.img

Mount the image on the /mnt mountpoint:

mount -o loop fc4-template.img /mnt

Create the /dev directory on the image with the required device nodes:

for i in console null zero ; do /sbin/MAKEDEV -d /mnt/dev -x $i ; done

Then create the /etc directory in the image that contains a valid /etc/fstab file for the virtual machine.

 /dev/sda1               /                       ext3    defaults 1 1
 none                    /dev/pts                devpts  gid=5,mode=620 0 0
 none                    /dev/shm                tmpfs   defaults 0 0
 none                    /proc                   proc    defaults 0 0
 none                    /sys                    sysfs   defaults 0 0

Note: A Swap file can be created with the mkswap command. This works for either an Image Swap file or a Swap partition. Make sure - of course - that every running VM needs its own swap space. dd if=/dev/zero of=swap.img bs=1M count=1 seek=1024 and mkswap swap.img. Then add it to the /mnt/etc/fstab file.

Install the Base Fedora Core Installation Files

We use yum's group install capability to install a vanilla base system. A suitable yum configuration file can be accessed directly on the Web. Alternatively you can copy the file below into a suitable directory (e.g. your home directory) and reference it on the yum command line. This file can be adapted if you want to install a different version of FC such as FC3 or the upcoming FC5.

yum -c http://people.redhat.com/~katzj/yum-xen.conf --installroot=/mnt -y groupinstall Base

 [main]
 cachedir=/var/cache/yum
 debuglevel=2
 logfile=/var/log/yum.log
 exclude=*-debuginfo
 gpgcheck=0
 obsoletes=1
 reposdir=/dev/null

 [base]
 name=Fedora Core 4 - $basearch - Base
 mirrorlist=http://fedora.redhat.com/download/mirrors/fedora-core-4
 enabled=1

 [updates-released]
 name=Fedora Core 4 - $basearch - Released Updates
 mirrorlist=http://fedora.redhat.com/download/mirrors/updates-released-fc4
 enabled=1

Copy the kernel modules from the host to the image file. This is necessary because the kernel installed by the yum command in the image file is release 2.6.15-1, but the Xen0 kernel used to boot the virtual machine is release 2.6.12-6. In order to boot the kernel needs a compatible set of modules installed inside the VM.

cp -r /lib/modules/2.6.12.6-xen3_7.1_fc4 /mnt/lib/modules

Enable Network Connectivity (DHCP)

After a standard install the networking services are not enabled by default. Create the following two files in order to define network interfaces eth0 and lo (loopback).

 DEVICE=eth0
 BOOTPROTO=dhcp
 ONBOOT=yes

 DEVICE=lo
 IPADDR=127.0.0.1
 NETMASK=255.0.0.0
 NETWORK=127.0.0.0
 # If you're having problems with gated making 127.0.0.0/8 a martian,
 # you can change this to something else (255.255.255.255, for example)
 BROADCAST=127.255.255.255
 ONBOOT=yes
 NAME=loopback

Create the file below in order to enable networking services at boot time automatically. This also defines a default gateway and a route to that gateway.

 NETWORKING=yes
 HOSTNAME=template
 GATEWAY=192.168.5.1

Tell the DHCP client to send its host name to the DHCP server when asking for a lease. This is important if you are using dynamic DNS (DDNS).

send host-name "hostname";

Alternative: Enable Networking using a Fixed IP Address

Replace the file ifcfg-eth0 with this one and create the file /etc/resolv.conf:

 DEVICE=eth0
 BOOTPROTO=static
 IPADDR=192.168.5.177
 NETMASK=255.255.255.0
 ONBOOT=yes

 # Optional:
 BROADCAST=192.168.5.255
 HWADDR=00:0F:1F:FA:CE:95
 NETWORK=192.168.5.0
 TYPE=Ethernet

search domain.com
nameserver 192.168.5.14

Note: File /mnt/etc/sysconfig/network is identical to above.

Unmount the Image File

umount /mnt

Note: We have not yet defined a host and domain name. We will do this as we customize this first template virtual machine.

Starting the template VM

The following is the Xen configuration file for the template virtual machine.

 kernel ="/boot/vmlinuz-2.6.12.6-xen3_7.1_fc4"
 memory = 512
 vcpus=2
 name = "template"
 nics=1
 vif = ['mac=aa:00:00:00:00:11, bridge=xenbr0']
 dhcp="dhcp"
 disk = ['file:/root/fc4-template.img,sda1,w']
 root = "/dev/sda1"
 extra = "ro selinux=0 3"

Note: Make sure the path to the image file is correct. By default a random MAC address is assigned to the virtual network interface. If you use DHCP a fixed MAC address should be defined so that the DHCP server does not get confused: vif = ['mac=aa:00:00:00:00:11, bridge=xenbr0']. Make sure a different MAC address is used for different virtual machines.

Start the template virtual machine:

xm create -c template

If everything went well you should see the virtual machine start up and you should end up with a login prompt. Enter root to login. Now would be a good time to verify that network connectivity works.

Some useful commands:

• You can exit the console by pressing Ctrl+]
• You can return to the console by running: xm console template
• You can shutdown your domain by running: xm shutdown -H template
• You can destroy (won't do a clean shutdown!) your domain by running: xm destroy template

Note: You can login to the virtual machine directly using ssh provided you set a root password.

Replicating the Template domU into an LVM Partition

Execute the following commands in dom0:

Creating a new LVM partition

First, let's look at what is already defined:

pvdisplay   (shows physical disks)
lvdisplay   (shows logical volumes)

Create a new partition of size 15GB. We call this master.

lvcreate -L 15G -n master VolGroup00
mke2fs -j /dev/VolGroup00/master

mount -t ext3 /dev/VolGroup00/master /mnt

Copy the template file system into the new partition:

mkdir /template
mount -o loop fc4-template.img /template
cp --archive /template/* /mnt
umount /template

Define Host and Domain Name

The host on which you plan to install sipX (dom0 or domU) needs to have a properly configured host and domain name as described in the sipX installation instructions. This name has to resolve in DNS.

Create the Xen Configuration File

 kernel ="/boot/vmlinuz-2.6.12.6-xen3_7.1_fc4"
 memory = 768
 vcpus=2
 name = "master"
 nics=1
 vif = ['mac=aa:00:00:00:00:22, bridge=xenbr0']
 dhcp="dhcp"
 disk = ['phy:/dev/VolGroup00/master,sda1,w']
 root = "/dev/sda1"
 extra = "ro selinux=0 3"

Note: Make sure the path to the partition is correct. By default a random MAC address is assigned to the virtual network interface. If you use DHCP a fixed MAC address should be defined so that the DHCP server does not get confused: vif = ['mac=aa:00:00:00:00:22, bridge=xenbr0']. Make sure a different MAC address is used for different virtual machines.

Start the virtual machine using xm create -c master.

Install sipX in a Virtual Machine (domU)

Installing sipX in a virtual machine is no different from installing it otherwise. Hight Availability (HA) capability is not yet in the stable release, so that you have to use a nighly build. Verifying hostname, domainname as well as DNS setup could be a good idea at this point. Also, don't forget to create an SSL certificate after installing sipX.

Repeat the process above and create a second virtual machine slave, install sipX and start it.

Old Content (to be deleted or moved)

Automatically start the FC4 virtual machine when Xen is started:

# cd /etc/xen/auto
# ln -s ../fc4 fc4

Enable the Network Manually:

ifconfig eth0 192.168.5.199 netmask 255.255.255.0
/sbin/route add default gw 192.168.5.1

ifconfig lo 127.0.0.1 netmask 255.255.255.0

 nano /etc/resolv.conf
   <add nameserver>
 

 DEVICE=eth0
 BOOTPROTO=static
 BROADCAST=192.168.5.255
 IPADDR=192.168.5.199
 NETMASK=255.255.255.0
 NETWORK=192.168.5.0
 ONBOOT=yes
 TYPE=Ethernet

Host and domain name:

echo sipx > /etc/hostname
/bin/hostname -F /etc/hostname

 /etc/sysconfig/network
 /etc/sysconfig/network-scripts/route-eth0 
 ADDRESS0=10.255.255.1 
 NETMASK0=255.255.255.255
 ADDRESS1=0.0.0.0
 NETMASK1=0.0.0.0
 GATEWAY1=10.255.255.1

Guest OS: CentOS 4 

# rpm --import http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-4
# yum -c yum-xen-centos.conf --installroot=/mnt/centos -y groupinstall Base

 [main]
 cachedir=/var/cache/yum
 debuglevel=2
 logfile=/var/log/yum.log
 exclude=*-debuginfo
 gpgcheck=0
 obsoletes=1
 reposdir=/dev/null

 [base]
 name=CentOS 4 - $basearch - Base
 baseurl=http://mirror.centos.org/centos-4/4/os/$basearch/
 enabled=1

 [updates-released]
 name=CentOS 4 - $basearch - Released Updates
 baseurl=http://mirror.centos.org/centos-4/4/updates/$basearch/
 enabled=1