Search This Blog

Thursday, May 1, 2008

Creating your own 10g RAC cluster at home using virtualbox and SAN targets (using openfiler + with/without USB hard disk)


After playing around with some open source technologies like virtualbox and openfiler, we are finally at a point when it is possible to create your own network lab of Oracle 10gR2 RAC cluster and ASM database using iscsi targets in an external hard disk drive. The last time I checked, USB 2.0 hard disk drives are cheaply available in the market.
If you do not want to use an external USB hard disk, then please go through the article A poor man's guide for creating iscsi targets without using external USB hard disks

All it takes is a little persistence and willingness to follow easy to follow tutorials. In the previous articles on this blog, I have talked about the building blocks of using virtualbox and openfiler open source softwares. If we combine that knowledge with Oracle 10g RAC setup, the following end configuration can be achieved:

Click here for a bigger view

The building blocks..

The operating system

First of all, you need to install a linux distribution of your choice using virtualbox. In this example, I chose SuSE linux 9.3. I feel that SuSE linux will be embraced by more and more enterprises in the near future, hence this decision.

For doing the installation, just boot from the Linux install media CD and then keep switching CDs as per request of the install process. It is always better if you have an ISO file of the installable media. You can get the SuSE media from

Always keep a decent amount of swap space for the OS installation. Also it is better to have a /tmp mount point as a separate mount point from the root (/) partition. These are some evergreen tips that a sysadmin friend of mine (Erik Niklas) had suggested to me a long time back.
A word of caution while deciding the VDI disk size for virtualbox machine (especially relevant for Version 1.5.6). The virtualbox disks cannot be resized!!

The dynamically expanding option means that the disk can grow UPTO the maximum limit and it will grow as per usage till the maximum limit. After that, it is not possible to resize it using some Vbox utility.A workaround may be to copy the device using partition editors, but till virtualbox comes out with an enhancement on this feature, better to be safe than sorry and keep a very high disk size - as per your judgement.

For building a Oracle 10g RAC configuration, it is better to focus all your installation efforts on one virtual machine and then clone the virtualbox disks (including the bootable disk) using the VboxManage clonevdi command. This way, you save a lot of duplicate effort.

The networking configuration - Public and Private IPs

The other networking building blocks of this configuration can be achieved by the following:

1) RAC needs a public IP, which is usually (but not limited to) eth0. It is advantageous to configure the public IP as host only networking interface when using virtualbox because it gives you the dual advantage of being able to SSH into the machine from outside (e.g. other rac node or another machine on the LAN) and also make internet work.
I tried to keep this as a static IP, but the downside is that the default gateway was somehow not working for me. However, if I kept this as DHCP interface, the default gateway and nameservers were being accessible out of the box.

A detailed example of how to do this is shown in the article Virtualbox Case Study: Making host only networking work between two Ubuntu Guest OS (virtual machine) on Windows Vista host

2) A RAC setup also needs a private network between the RAC nodes for global communication system (GCS) and synchronization. This is also usually referred to as the private interconnects. This is usually configured on a private subnet like 10.10.x.x or 192.168.x.x, wherein other network traffic would not impinge on the sacrosanct synchronization traffic between the RAC nodes.

A detailed example of how to do this is shown in the article Case study: Making Internal networking for talking between two linux guest OS (Ubuntu) on windows vista host

The Network Attached Storage or Storage Area Network setup..

NAS is hot! Well, what i mean is that its like configure once, use everywhere kind of thing. You setup iSCSI target logical volumes and then discover and access them from other machines. There are packages like linux-iscsi and open-iscsi which will help you do that. As per the latest announcements, both these open source projects have been merged into open-iscsi now.
Why should we choose Ubuntu for hard disk drivers? The partitions from a /dev/sd* devices gets discovered as separate devices, whereas all partitions in /dev/hd* get discovered as a single device.

There are quite promising open source softwares in this area too. openfiler and freenas are a few of them. For no particular reason, I picked openfiler and did a few experiments for discovering iscsi targets using it.

A detailed example of how to do this has been shown in the article Case Study: How to discover iscsi targets with linux-iscsi initiator package — Suse linux 9 (scsi Initiator) and openfiler (scsi target)
If you are using USB external hard disks for this exercise, there are many gotchas involved in their proper detection when using virtualbox on windows host. But some golden rules mentioned in this article can be useful: Virtualbox How to: Gotchas involved in making a USB external hard disk device work on windows host

A problem with openfiler 2.2 is that it forgets the previous logical volumes etc after reboots. This situation can result in frustration, especially when you see no LUNs being discovered from the initiators. If you read the following article, there is a good chance that you will be fine: iSCSI: no LUNs detected for session when using openfiler

The Oracle 10gR2 RAC setup

With the above components in place, the stage is now set for you to install the Oracle 10g CRS software. Needless to say, there are several detailed guides available on Oracle Technology Network on how to do this. Each Operating system has unique package pre-requisites for the Oracle software to be installed properly.
Some more guides area available on the website at

Check out the guides under SUSE Enterprise Linux Server 9 tab. They are pretty detailed.

Some more guides are available at and

Suggested reading..

It is very likely that you would have come across the widely read article by Jeffrey Hunter on a similar topic. In, he deals with creating a 10g RAC on Redhat linux.

If you do not want to use an external USB hard disk for iscsi SAN targets, then please go through the article A poor man's guide for creating iscsi targets without using external USB hard disks

No comments:

Post a Comment