Virtualbox shared folder mounting issue: Missing module on opensuse 10.3

The Version combination for this scenario..

This problem was encountered with the following combination:

• Virtualbox 1.6.4


• Host OS: windows XP service pack 2


• Guest OS: OpenSuSE 10.3

A shared virtual folder was defined for the opensuse virtual machine like this:

In the guest OS, a mount point was defined like this:

linux-jo4o:~ # mkdir -p /ddrive

Now when the mount of the shared folder was attempted, this error was received:

inux-jo4o:~ # mount -t vboxsf d_drive /ddrive
/sbin/mount.vboxsf: mounting failed with the error: No such device

An unlikely solution...

It seems that a critical module required for shared mount points was not loaded in memory. Here, we can see that only one module with the vbox pattern in it's name is loaded:

linux-jo4o:~ # lsmod | grep vbox
vboxadd                56232  3

So lets load the vboxsvfs module into memory:

linux-jo4o:~ # modprobe vboxvfs

After that, we can verify that it is really loaded:

linux-jo4o:~ # lsmod | grep vbox
vboxvfs                38184  0
vboxadd                56232  4 vboxvfs

Now, let us mount the shared folder:

linux-jo4o:~ #  mount -t vboxsf d_drive /ddrive

Awesome! That was smooth. Lets check if the shared folder really got mounted or not:

linux-jo4o:~ # mount
/dev/sda2 on / type ext3 (rw,acl,user_xattr)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
udev on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
/dev/sda3 on /home type ext3 (rw,acl,user_xattr)
/dev/sr0 on /media/VBOXADDITIONS_1.6.4_33808 type iso9660 (ro,nosuid,nodev,uid=0)
d_drive on /ddrive type vboxsf (rw)
linux-jo4o:~ #

Yippee!

I can share files with my windows laptop now. Now that was some adventure, eh :)

To make this more permanent, there has to be a way of loading this module automatically. Maybe I can put the modprobe command in /etc/rc.d/boot.local file, but then maybe there is a better way. I leave that to you to figure out.

Package dependency for installing virtualbox guest additions on Open SuSE 10.3

The problem at hand..

Well, I had to get the shared folder feature working on Open SuSE 10.3 virtual machine for a variety of reasons, and in turn, I had to install guest additions on virtualbox 1.6.4. There were a bunch of package dependencies even before coming to this error, and unfortunately, I dont have the entire series of command that I issued, but in this posting, I will attempt to present the solution of the immediate error that I was facing.

The KERN_INCL variable had to be set to the /usr/include directory:

linux-jo4o:/usr/src/linux # env | grep KER
KERN_INCL=/usr/include

After mounting the guest additions CD and running the VboxAdditions.run command as root, the kernel was not able to be compiled and this was visible in the install log:

linux-jo4o:~/addons # more /var/log/vboxadd-install.log

Installing VirtualBox 1.6.4 Guest Additions, built Tue Jul 29 20:13:59 CEST 2008

Testing the setup of the guest system

Building a test kernel module...

make KBUILD_VERBOSE=1 -C /lib/modules/2.6.22.5-31-default/build SUBDIRS=/tmp/sel
fgz1883915647/module/test SRCROOT=/tmp/selfgz1883915647/module/test modules
make[1]: *** No rule to make target `modules'.  Stop.
make: *** [vboxadd_test] Error 2

Building the test kernel module failed.
Giving up due to the problems mentioned above.
linux-jo4o:~/addons #

Initial thoughts and a potential solution

Initially, I thought that maybe it was a problem due to missing symbolic link to the modules directory (??), but when I posted this question on the vbox-users mailing list (which I think is more effective than the forums at times), I got a response saying that maybe I should check if the kernel-source package was installed or not.

To even find the package, we needed the main SOURCE repository added in the YAST setup. This is what the installed online repositories looked like:



When I ran the following command as root, it was clear that this was the missing link:

linux-jo4o:~ # zypper install kernel-source
* Reading repository 'Main Repository (NON-OSS)' cache
* Reading repository 'openSUSE-10.3-OSS-KDE 10.3' cache
* Reading repository 'Main Repository (Sources)' cache
* Reading repository 'Main Repository (OSS)' cache
* Reading installed packages [100%]

The following NEW package is going to be installed:
  kernel-source 

Overall download size: 49.2 M. After the operation, additional 243.4 M will be used.
Continue? [yes/no]: yes
Downloading package kernel-source-2.6.22.5-31.i586, 49.2 M (243.4 M unpacked)
Downloading: media
* Downloading [100%]
Downloading: kernel-source-2.6.22.5-31.i586.rpm
* Downloading [100%]
* Installing: kernel-source-2.6.22.5-31 [100%]
linux-jo4o:~ #

Now, when I tried installing the guest additions, it was able to install them and recompile the kernel JUST fine. Sweet!

linux-jo4o:~ # /media/VBOXADDITIONS_1.6.4_33808/VboxAdditions.run
Verifying archive integrity... All good.
Uncompressing
VirtualBox 1.6.4 Guest Additions for Linux
installation.............................................................................................................................................................................
VirtualBox 1.6.4 Guest Additions installation
Building the VirtualBox Guest Additions kernel module...
Building the shared folder support kernel module...
Installing the VirtualBox Guest Additions...

Successfully installed the VirtualBox Guest Additions.
You must restart your guest system in order to complete the installation.
linux-jo4o:~ #

How to get the enhanced Rich text editing buttons in wordpress.com editor?

The why -- Bloggers need Rich text editing..

If you are a newbie wordpress.com blogger, chances are that at some point you would have pulled your hair out trying to figure out how the heck to display the rich text editing buttons that would give you the MS word like editing features. I definitely remember doing that, it was an uphill experience. Believe me, I had "seriously" tried to look for it in Google and wordpress forums -- only to be told that you are probably mistaking wordpress.com for wordpress.org (arrgh!) and that its not part of the free wordpress.com blog (that was support talking -- can you believe that?).

I am posting this in the hope that a fellow newbie like me would not be in the same boat and gets productive right off the bat.

This is what you might originally see when you write a new post..

Life was difficult with this kind of editor. To put heading tags, I had to use the HTML editor and put these tags in manually, which was a royal pain:

Then one fine day...

It was Serendipity. Nothing less. While editing a post, I hit a key by mistake, and Viola!, there it was, the rich text palette, right in front of it. I had really no idea what had caused the second half of rich text editing buttons to come up, but I could see that it was not disappearing even after many shutdown and startups of my laptop.

Strangely, it would not come up on my other computers even though I was logged into the same wordpress account.

Its the kitchen sink, silly..

Serendipity - Part 2. God has his ways of revealing his glories. This fine day, I accidently clicked on the color palette icon on the extreme right end in the hope of getting some colors, but instead, I got the rich text editing buttons in the editor! The so called color palette was called the Kitchen sink. Duh! How ignorant of me. Thank you Almighty, for delivering me from ignorance.

So this is how it looked now:

Life was beautiful now...

I could do different kind of formatting, underlining, paste from word, erase, indenting, undo, redo, heading, color change etc.



Hope you find this useful too. Do leave me a message if you do.

An assessment of CiviCRM for non-profits

Introduction

Lately, we have been evaluating open source CRM softwares that are available in the market. Remarkably speaking, the CRM software market seems to really on the move with SugarCRM and CiviCRM being some of the most popular and widely used open source softwares.

Our Requirements in a nutshell

We are a small non profit service organization and have the need of maintaining contacts ( volunteers, donors, media contacts, partner organizations, NGOs etc), accounting/donor/fund raising information and potentially a document knowledge base and project management/event planning functionality. In addition, we needed mass personalized mailing functionality as well and an ability to define internal hierarchy or relationship between volunteers or organizations.

Till now, we were maintaining all our information in Yahoo! groups Files section or not even that in some cases. Contacts were kept in different forms, most of the times misplaced and rarely followed up on properly. Out of sight is out of mind and thats what happened.

CiviCRM

Well, I was looking at online articles and blogs about similar non-profit organizations who may implemented open source CRM softwares, and that's when I came to know about Civicrm. I spent about a day installing it on Drupal 5.x (civicrm 2.0 does not work with Drupal 6.x yet) and exploring it.

The scope of this article is limited to CiviCRM and observations around it. But as all things, there is good news and there is bad news. Let me start with the good news.

Good features:

1) The GUI is quite simple and intuitive for administrators. Most of the features and projected functionality can be figured out easily.

2) Contact management and defining different kinds of relationships is simple. You can define relationships between individuals, groups and organizations, which is quite extensible. For a contact, you can define associated grants, notes, meetings, groups, memberships, relationships, mails etc, kind of like a central view.

3) Contribute and Grants module are good for Integration with online payment system like Googlc checking or Paypal and for tracking and pre-scheduling donations

4) CiviMail can do personalized mailings. Attachments dont seem to be possible (at least I could not see it work). Messages can be tracked etc, but the reporting interface does not compare with SugarCRM (more evolved). For actually triggering the mail delivery a cron job has to be run and for bounce processing, some extra unix utilies (e.g. AMaViS,postfix) have to be configured

5) Customization to current forms or tables is easy. When we upgrade to a newer version of CiviCRM, the Custom Data Fields and Custom Data Groups will be maintained. Specialized customizations using the PHP programming language will need to be re-created when you upgrade, however.

6) The data model is simpler and easy to understand. The entity relationship diagram is available at
http://wiki.civicrm.org/confluence/display/CRMDOC/CiviCRM+ERD+2.0

7) It has a planner features too -- activities, meetings etc (types can be defined).

8) Good user integration/sync with Drupal and Joomla

9) Although there is no menu link for CiviReports, the reports are available through http://wiki.civicrm.org/confluence/display/CRMDOC/CiviReport. There are a lot of sql queries also that are available online on wiki.civicrm.org

Some other features -- but not essential for us:

1) CiviEvent is useful for putting signup pages for events. I thought it was kind of interesting though. Its good for promoting events and signing up people into your contact database.

2) CiviMember module can be used for tracking memberships to the organization (e.g. lifemember, goldmember etc). We did not need this functionality as we do not charge anything for membership, but it could be used at some point of time.

3) Can Configure Google or Yahoo Maps to work with CiviCRM. This is actually kind of cool. I always wanted to implement something like for our website wherein a project's location in the world could be shown through yahoo or google map.

4) The CitizenSpeak module allows the creation of email petition campaign nodes. Users enter the email recipients and the a message to create a campaign. When someone visits the node, they see a form that displays a preview of the message and allows them to enter their contact information and a personal statement. When they submit the form, the target of the campaign is notified. The version currently in CVS is built against Drupal 4.6. (http://drupal.org/project/citizenspeak)

Features lacking:

1) project Management -- This feature is present in sugarcrm Community Edition and we though we could use it for managing events (their tasks) and sharing event templates between chapters

2) Document management -- This feature would have been done to implement a knowledge base system for our volunteers and organization in general.

3) Can't create a new module through GUI, but APIs are published.

4) Leads and Opportunity Management are not present and not differentiated

Some Interesting Facts:

Amnesty International uses CiviCRM to collect website registration data and process online donations

Overall thoughts

I felt that the civicrm product is in initial stages and will see a lot of change or value addition in the near future. There are a lot of things coming and they will come slowly. From the http://civicrm.org/architecture, I could see that there was already talk of more performance or caching features in 2.1 version.

All in all, It still has some ways to go before it can catch up with sugarCRM. But then, civicrm is focused exclusively on non-profit organizations and is designed for them. It will have its own growth path: one that would may be less spectacular than sugarcrm for a little while, but it sure packs a lot of potential in it.

Sharing windows folders on unix hosts using samba filesystem (smbfs) using virtualbox

The need for windows sharing..

A few days ago, I wanted to do a Oracle 12i installation on a x86 linux distribution using virtualbox. The media was available on an external hard disk, but I thought that instead of copying the 30+ G of media on unix virtual machine, why not just make it shared between the windows host OS and unix guest OS.

The quest for sharing..

And so began my quest for sharing files between windows and virtualbox unix OS (SuSE linux). I discovered that for sharing folders using virtualbox, the guest OS additions have to be installed.

I have seen that upgrading the kernel breaks the guest OS additions at times. After upgrading the guest OS from SuSE linux 9 to SuSE linux 9.3, the guest additions stopped working. The same problem also exists with vmware tools.

So first, I created a shared folder in the virtual machine's definition:

Click here for a bigger view



After starting the Guest OS (Ubuntu),

root@gverma-laptop:/home/gverma# mkdir /mnt/winshare

root@gverma-laptop:/home/gverma# mount -t vboxsf public /mnt/winshare

Mind you, the filesystem type is NOT vboxfs, it is vboxsf (s and f are interchanged). If you do not take note of this, you might end up wasting your time for nothing.

Now, we can check that the shared folder did indeed mount:

root@gverma-laptop:/home/gverma# mount | grep vbox
public on /mnt/winshare type vboxsf (rw)

root@gverma-laptop:/home/gverma# ls /mnt/winshare
CyberLink  desktop.ini  Downloads  Music     Recorded TV         Videos
Desktop    Documents    Favorites  Pictures  StarzEntertainment  Vongo

Let it be mentioned here that the vboxadd and vboxvfs (note that it has vfs after vbox) modules needed to be loaded for this to work:

root@gverma-laptop:/home/gverma# lsmod | grep vb
vboxvfs                 42432  1
vboxadd                 24872  9 vboxvfs

The Caveat..

But, as every cloud has a silver lining, there was a caveat here for my problem.

The above case was just a demonstration of how the shared folder could have worked fine. I actually then went on to mount some linux media for Oracle Applications 12i for x86 intel platform. After the directory mounted all right, I had a really hard time adding the execute permissions to the rapidwiz files. Even after doing persistent # chmod -R +x commands, the shell scripts under the media were not showing the "x" execute flag.

This was very frustrating. It was probably just another virtualbox bug.

So, then I started looking around for more options. I had heard that many people had used samba with success and hence I started scouring the internet (Google Zindabaad! - Long live, Google!). Came across a link that explained how to do it in a succinct manner.

The good thing about sharing files from windows to unix using samba file system (smbfs) was that was that I did not need to run samba on unix at all.

Doing it using samba filesystem..

So here are some quick and easy steps for sharing a windows share using smbfs:

First things first. You need to create a network share for the folder that you want to share. In my case, I wanted to share an external hard disk that had the media for Oracle Applications 12i. So, I did a network share of windows folder (e.g. \\GAURAV-PC\media)

On the guest OS (Ubuntu), did the following:

# mkdir -p /mnt/media

# smbmount //GAURAV-PC/media /mnt/media -o username=gaurav,password=welcome

Please note that // (forward slashes) worked for me. Some guides on the internet say that you need to use \\ (back slashes)

Note here that GAURAV-PC hostname was defined in the local hosts file to a resolvable IP address in the LAN. The Ubuntu Virtual machine was connected to the windows Host through a host only networking type interface, that allowed the host and guest OS to talk to each other.

# more /etc/hosts
GAURAV-PC 192.168.0.2

Please note here that a ping to GAURAV-PC HAS TO work when you give the smbmount command. I wasted many hours trying to run strace on this command, just because it was hanging. The issue was that due to some strange happenstance, the ping to GAURAV-PC (windows host) was hanging.

The workaround was to ping the Ubuntu virtual machine (say 192.168.0.7) first from the host OS (windows), after which the ping to GAURAV-PC 192.168.0.2 started working from Ubuntu Guest OS. Strange, strange, I agree, but I am just a guy looking for solutions! I have seen this behaviour in Windows Vista home premium edition - host OS.

Make sure the smbfs module is loaded in the guest OS. If not, then load it using # modprobe smbfs:

root@gverma-laptop:/mnt/media/oracle_12.0.4/startCD/Disk1/rapidwiz# lsmod | grep smbfs
smbfs                  66296  0

Note that the Samba server does not need to run:

root@gverma-laptop:/mnt/media/oracle_12.0.4/startCD/Disk1/rapidwiz# ps -ef | grep smb
root      6116  5616  0 00:00:00 grep smb

This was my samba configuration file:

root@gverma-laptop:/mnt/media/oracle_12.0.4/startCD/Disk1/rapidwiz# more /usr/share/samba/smb.conf
[global]
workgroup = WORKGROUP
security = share
usershare path =  /var/lib/samba/usershare
usershare max shares = 100
usershare allow guests = yes
usershare owner only = yes
wins support = no
netbios name = GAURAV-PC

I am not exactly sure if setting the netbios name parameter was a major factor in making it work or not, but I guess it certainly did not hurt.

Now when I run mount command, the windows network share was showing up there just fine:

root@gverma-laptop:/mnt/media/oracle_12.0.4/startCD/Disk1/rapidwiz# mount
/dev/sda1 on / type ext3 (rw,relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-16-generic/volatile type tmpfs (rw)
securityfs on /sys/kernel/security type securityfs (rw)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
gvfs-fuse-daemon on /home/gverma/.gvfs type  fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=gverma)
//GAURAV-PC/media on /mnt/media type cifs (rw,mand)

Thankfully, I was able to make the unix shell scripts executable after doing this and launch the rapid wizard for 12i installation. That's all I wanted.

Conclusion

Well, so there you have it. If you are using windows host OS and you upgraded your guest OS kernel, the guest additions may not work due to compilation errors. In such a case, you are better off using the samba file system as its much easier to deal with and not too much of a pain to setup.

A poor man's guide for creating iscsi targets without using external USB hard disks

A special need..

Those of you who have a need for exposing iscsi targets to other machines for discovery, but who do not want to invest in an external USB hard disk OR who do not want to have a USB external hard disk connected to a desktop 24x7, there is a better way of creating and exposing iscsi target logical volumes.

This method does not need installing openfiler either as noted in my earlier article Combining Openfiler and Virtualbox (Ubuntu guest OS on windows host).

A simple solution..

The method is simple: create logical volumes based on SCSI devices on a Ubuntu Hardy Heron 8.04 installation using virtualbox.

The advantage of using Ubuntu is that it creates the hard disk devices as scsi devices (with the naming convention /dev/sd*, instead of /dev/hd*). I have seen that in other unix OS like SuSE linux, the local disks are listed as /dev/hd* (IDE). Ubuntu seems to have better disk drivers.

If you use Hardy Heron 8.04 Ubuntu, it is *really* easy to get iscsitarget package working. If you try to make the iscsi-target package work with Gutsy Gibbon 7.x release of Ubuntu, there is a very good chance that you will run into lot of compilation issues. I went down this path myself and later realized that a lot of bugs existed for Gutsy Gibbon release. (e.g. https://bugs.launchpad.net/ubuntu/gutsy/+source/iscsitarget/+bug/160104, http://ubuntuforums.org/showthread.php?t=692651) and eventually found https://bugs.launchpad.net/ubuntu/+source/iscsitarget/+bug/145539 which said that the module was fixed in Hardy Heron 8.04.

So instead of breaking my head over making iscsi-target package work in Gutsy Gibbon 2.6.22, I decided to give Hardy Heron 8.04 (still beta) a try.

Thankfully, with a little effort, I was able to make it work. In this article, I present a simplistic scenario in which we create three logical volumes that can be easily discovered by another virtualbox iscsi initiator machine(s) using open-iscsi package. This is the beauty of the entire approach.

Here is a block diagram of the end configuration:

Click here for a bigger view

Iscsi support in Hardy Heron 8.04 Ubuntu..

The first thing to understand is that as per the published features of Hardy Heron 8.04 Ubuntu at https://wiki.ubuntu.com/HardyHeron/Beta#head-da07b62e1e43afd0bef06ab8b60d2502c734a0f9, the iscsi support is enabled out of the box if we add iscsi=true in the boot options during the installation.

Click here for a bigger view



While installing Hardy 8.04 OS using virtualbox or any other virtualization software being used, remember to add iscsi=true in the boot options (after pressing F6)

Click here for a bigger view

Configuring iscsi targets..

I would like to give credit to a really awesome link I found through google: http://www.linuxconfig.org/Linux_lvm_-_Logical_Volume_Manager, which I rate as a one of the best article s for doing this!

Some other relevant links are:

https://help.ubuntu.com/community/SettingUpLVM-WithoutACleanInstall
http://t3flyers.wordpress.com/2007/04/24/logical-volume-manager-on-ubuntu-feisty-704/
http://www.howtoforge.com/linux_lvm

After Ubuntu installation, the output of uname -a looks like this:

HARDY# uname -a
Linux gverma-laptop 2.6.24-16-generic #1 SMP Thu Apr 10 13:23:42 UTC 2008 i686 GNU/Linux

And the output of fdisk -l looks like this:

HARDY# fdisk -l

Disk /dev/sda: 16.5 GB, 16592666624 bytes
255 heads, 63 sectors/track, 2017 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0004ccc2
Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1927    15478596   83  Linux
/dev/sda2            1928        2017      722925    5  Extended
/dev/sda5            1928        2017      722893+  82  Linux swap / Solaris

At this point, I added two more hard disks to the virtualbox virtual machine. Virtualbox 1.5.6 allows upto three hard disk devices to be attached to a machine.

Click here for a bigger view



After booting up HARDY again, /dev/sdb and /dev/sdc were also visible in the output of fdisk -l. Using fdisk, I created a single partition in both devices that spanned the entire device, resulting in /dev/sdb1 and /dev/sdc1:

HARDY# fdisk -l

...
Disk /dev/sdb: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xd310045f

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        2610    20964793+  83  Linux

Disk /dev/sdc: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x93d4c692

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1        2610    20964793+  83  Linux

Also make sure that you configure host only networking so that other virtual machines (scsi initiators) are able to ssh/telnet into HARDY. Please look at article to understand how host only networking can be setup: Virtualbox Case Study: Making host only networking work between two Ubuntu Guest OS (virtual machine) on Windows Vista host

Getting the right packages installed..

Now, we need to create the target physical volume, virtual group, and logical volume, in that order. These logical volumes will serve as SCSI targets that can be discovered by other iscsi initiator machines or virtual machines as per your configuration.

Make sure you reload the package definitions from the configured repositories (get the latest packages that are published from the ubuntu repositories). You can do this from the Synaptic Package Manager:

Click here for a bigger view



Now, you should install the lvm2 package in Synaptic Package Manager:

Click here for a bigger view

Create the physical volumes:

HARDY# lvm

lvm> pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created
lvm> pvcreate /dev/sdc1
Physical volume "/dev/sdc1" successfully created
lvm> pvdisplay
--- NEW Physical volume ---
PV Name               /dev/sdb1
VG Name
PV Size               19.99 GB
Allocatable           NO
PE Size (KByte)       0
Total PE              0
Free PE               0
Allocated PE          0
PV UUID               OHKhOq-AMG7-XtYz-7CrP-q2VH-2b53-tW0yMO

--- NEW Physical volume ---
PV Name               /dev/sdc1
VG Name
PV Size               19.99 GB
Allocatable           NO
PE Size (KByte)       0
Total PE              0
Free PE               0
Allocated PE          0
PV UUID               M9qXXB-GVxC-FZjI-9Zb7-IhcH-rbhb-vQ3aek

Create the virtual group:

lvm> vgcreate vg /dev/sdb1 /dev/sdc1
Volume group "vg" successfully created
lvm> vgdisplay
--- Volume group ---
VG Name               vg
System ID
Format                lvm2
Metadata Areas        2
Metadata Sequence No  1
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                0
Open LV               0
Max PV                0
Cur PV                2
Act PV                2
VG Size               39.98 GB
PE Size               4.00 MB
Total PE              10236
Alloc PE / Size       0 / 0
Free  PE / Size       10236 / 39.98 GB
VG UUID               GkUMNq-3atR-qKTK-lG0b-gM0n-budV-Uc4lH8

lvm>

Create logial volumes:

lvm> lvcreate -L 1G -n ocr vg
Logical volume "ocr" created
lvm> lvcreate -L 1G -n vote vg
Logical volume "vote" created
lvm> lvcreate -L 35G -n asm vg
Logical volume "asm" created

If you get an error message saying:
/proc/misc: No entry for device-mapper
you can get around it by issuing sudo modprobe dm-mod

lvm> lvdisplay
--- Logical volume ---
LV Name                /dev/vg/ocr
VG Name                vg
LV UUID                a2ARIQ-11dn-kdoB-cHCd-gtoR-aokO-HGqoo4
LV Write Access        read/write
LV Status              available
# open                 0
LV Size                1.00 GB
Current LE             256
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           254:0

--- Logical volume ---
LV Name                /dev/vg/vote
VG Name                vg
LV UUID                1DqYL0-Ptsx-9kmE-UHmn-PmE9-ebVm-bfr0r0
LV Write Access        read/write
LV Status              available
# open                 0
LV Size                1.00 GB
Current LE             256
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           254:1

--- Logical volume ---
LV Name                /dev/vg/asm
VG Name                vg
LV UUID                UKWcPz-aNGl-VdPm-51wa-Ewvm-ahvD-Y7N57v
LV Write Access        read/write
LV Status              available
# open                 0
LV Size                35.00 GB
Current LE             8960
Segments               2
Allocation             inherit
Read ahead sectors     0
Block device           254:2

lvm>

Now, you can check the logical volume devices like this:

HARDY# ls -l /dev/vg
total 0
lrwxrwxrwx 1 root root 18 2008-05-03 02:42 asm -> /dev/mapper/vg-asm
lrwxrwxrwx 1 root root 18 2008-05-03 02:41 ocr -> /dev/mapper/vg-ocr
lrwxrwxrwx 1 root root 19 2008-05-03 02:41 vote -> /dev/mapper/vg-vote

You can also check the newly created devices in the /dev/disk/* directories on HARDY:

HARDY# ls -l /dev/disk/*
/dev/disk/by-id:
..
lrwxrwxrwx 1 root root 19 2008-05-03 02:42 dm-name-vg-asm -> ../../mapper/vg-asm
lrwxrwxrwx 1 root root 19 2008-05-03 02:41 dm-name-vg-ocr -> ../../mapper/vg-ocr
lrwxrwxrwx 1 root root 20 2008-05-03 02:41 dm-name-vg-vote -> ../../mapper/vg-vote
..

/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root  9 2008-05-02 22:30 pci-0000:00:01.1-scsi-0:0:1:0 -> ../../sdb
lrwxrwxrwx 1 root root 10 2008-05-03 02:34 pci-0000:00:01.1-scsi-0:0:1:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 2008-05-02 22:30 pci-0000:00:01.1-scsi-1:0:1:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 2008-05-03 02:35 pci-0000:00:01.1-scsi-1:0:1:0-part1 -> ../../sdc1

You need to install the iscsitarget package using Synaptic Package Manager:

Click here for a bigger view



Now, you need to configure the /etc/ietd.conf file with the Target names and entries for each logical volume. The target names just need to unique within the network and are totally upto your imagination.

HARDY# more /etc/ietd.conf
Target iqn.2001-04.com.ubuntu:scsi.disk.vg.vote
Lun 0 Path=/dev/vg/vote,Type=fileio
Target iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr
Lun 0 Path=/dev/vg/ocr,Type=fileio
Target iqn.2001-04.com.ubuntu:scsi.disk.vg.asm
Lun 0 Path=/dev/vg/asm,Type=fileio

To enforce these entries, we need to restart the iscsitarget service manually now.

Keep in mind that there is no dash (-) between iscsi and target in the name of the service, as in openfiler 2.2 (openfiler's IET implementation service has the name iscsi-target)

HARDY# /etc/init.d/iscsitarget restart
Removing iSCSI enterprise target devices: succeeded.
Stopping iSCSI enterprise target service: succeeded.
Removing iSCSI enterprise target modules: succeeded.
Starting iSCSI enterprise target service: succeeded.

To double check if the logical volumes have been discovered and published, you can check content of /proc/net/iet/volume on HARDY:

::::::::::::::
/proc/net/iet/volume
::::::::::::::
tid:3 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.asm
lun:0 state:0 iotype:fileio iomode:wt path:/dev/vg/asm
tid:2 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr
lun:0 state:0 iotype:fileio iomode:wt path:/dev/vg/ocr
tid:1 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.vote
lun:0 state:0 iotype:fileio iomode:wt path:/dev/vg/vote

If you do a more of /proc/net/iet/session, you can clearly see that no iscsi-initiator machine have connected to HARDY yet (there are no session sub-entries under the volume name):

HARDY# more /proc/net/iet/session
tid:3 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.asm
tid:2 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr
tid:1 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.vote

Here is a quick check of the physical volume, volume group, and logical volume :

HARDY# lvm
lvm> lvscan
ACTIVE '/dev/vg/ocr' [1.00 GB] inherit
ACTIVE '/dev/vg/vote' [1.00 GB] inherit
ACTIVE '/dev/vg/asm' [35.00 GB] inherit

lvm> pvscan
PV /dev/sdb1 VG vg lvm2 [19.99 GB / 0 free]
PV /dev/sdc1 VG vg lvm2 [19.99 GB / 2.98 GB free]
Total: 2 [39.98 GB] / in use: 2 [39.98 GB] / in no VG: 0 [0 ]

lvm> vgscan
Reading all physical volumes. This may take a while...
Found volume group "vg" using metadata type lvm2
lvm>

Discovering the iscsi targets from another iscsi initiator machine..

We will assume that the iscsi initiator machine name is GUTSY and it has the open-iscsi package installed (this also installs iscsiadm utility along with it). We will also assume that the IP of HARDY (iscsi target) is 192.168.0.7.

You can leave the majority default values in /etc/iscsi.conf. Make sure you disable the CHAP authentication parameter values since we have not configured them in the scsi target.

Discover the new iscsi targets in HARDY:

GUTSY# iscsiadm -m discovery -t st -p 192.168.0.7
192.168.0.7:3260,1 iqn.2001-04.com.ubuntu:scsi.disk.vg.asm
192.168.0.7:3260,1 iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr
192.168.0.7:3260,1 iqn.2001-04.com.ubuntu:scsi.disk.vg.vote

Verify the newly discovered target machine:

GUTSY# sudo iscsiadm -m discovery
192.168.0.6:3260 via sendtargets
192.168.0.7:3260 via sendtargets  --> the new iscsi target got added to the local database

Check the combination of discovered iscsi-target and logical volumes (the new ones are in BLUE color):

GUTSY# sudo iscsiadm -m node
192.168.0.6:3260,1 iqn.2006-01.com.openfiler:openfiler.ocr
192.168.0.6:3260,1 iqn.2006-01.com.openfiler:openfiler.vote
192.168.0.6:3260,1 iqn.2006-01.com.openfiler:openfiler.asm
192.168.0.7:3260,1 iqn.2001-04.com.ubuntu:scsi.disk.vg.asm
192.168.0.7:3260,1 iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr
192.168.0.7:3260,1 iqn.2001-04.com.ubuntu:scsi.disk.vg.vote

But, at this point, we have no active sessions connected to these targets:

GUTSY# iscsiadm -m session
iscsiadm: No active sessions.

Establish connection to the scsi targets from the initiator:

GUTSY# iscsiadm -m node \
-T iqn.2001-04.com.ubuntu:scsi.disk.vg.asm \
-p 192.168.0.7 -l
Login session [iface: default, target: iqn.2001-04.com.ubuntu:scsi.disk.vg.asm, portal: 192.168.0.7,3260]
GUTSY# iscsiadm -m node \
-T iqn.2001-04.com.ubuntu:scsi.disk.vg.vote \
-p 192.168.0.7 -l
Login session [iface: default, target: iqn.2001-04.com.ubuntu:scsi.disk.vg.vote, portal: 192.168.0.7,3260]
GUTSY# iscsiadm -m node \
-T iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr \
-p 192.168.0.7 -l
Login session [iface: default, target: iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr, portal: 192.168.0.7,3260]

Verify the newly formed connections:

GUTSY# iscsiadm -m session
tcp: [1] 192.168.0.7:3260,1 iqn.2001-04.com.ubuntu:scsi.disk.vg.asm
tcp: [2] 192.168.0.7:3260,1 iqn.2001-04.com.ubuntu:scsi.disk.vg.vote
tcp: [3] 192.168.0.7:3260,1 iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr

It would be useful to enable automatic startup/discovery of these volumes on server and client. For this, you need to do the following:

• On server(where the logical volumes are created):

HARDY# sudo gedit /etc/iscsi/iscsid.conf

Change node.startup=manual to: node.startup = Automatic

GUTSY# sudo /etc/init.d/open-iscsi restart

• On client

GUTSY# sudo iscsiadm -m discovery -t st -p <ipaddress>
GUTSY# sudo iscsiadm -m node
-T iqn.2001-04.com.ubuntu:scsi.disk.vg.asm
-p <ipaddress> -o update
-n node.conn[0].startup -v automatic
GUTSY# sudo /etc/init.d/open-iscsi restart

• or reboot both the systems if required (make sure server boots first). To check if the scsi devices are automatically setup and list all your scsi iqns:

GUTSY # sudo iscsiadm -m session

Now assumed that the TCP scsi connections have been formed, let us check if the scsi devices are visible from fdisk -l:

GUTSY# fdisk -l

..
Disk /dev/sdb: 37.5 GB, 37580963840 bytes
64 heads, 32 sectors/track, 35840 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table

Disk /dev/sdd: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Disk identifier: 0x00000000

Disk /dev/sdd doesn't contain a valid partition table
GUTSY#

The newly discovered devices can also be checked under the contents of /dev/disk/*:

GUTSY# ls -l /dev/disk/*
/dev/disk/by-id:
total 0
..
lrwxrwxrwx 1 root root  9 2008-05-02 23:05 scsi-149455400000000000000000001000000795600000e000000 -> ../../sdc
lrwxrwxrwx 1 root root  9 2008-05-02 23:05 scsi-1494554000000000000000000020000004f5600000e000000 -> ../../sdd
lrwxrwxrwx 1 root root  9 2008-05-02 23:05 scsi-149455400000000000000000003000000a45600000e000000 -> ../../sdb
..

/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root  9 2008-05-02 23:05 ip-192.168.0.7:3260-iscsi-iqn.2001-04.com.ubuntu:scsi.disk.vg.asm-lun-0 -> ../../sdb
lrwxrwxrwx 1 root root  9 2008-05-02 23:05 ip-192.168.0.7:3260-iscsi-iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr-lun-0 -> ../../sdd
lrwxrwxrwx 1 root root  9 2008-05-02 23:05 ip-192.168.0.7:3260-iscsi-iqn.2001-04.com.ubuntu:scsi.disk.vg.vote-lun-0 -> ../../sdc
..

Relevant messages in the system log of scsi initiator (GUTSY)

# dmesg
...
[ 1156.290868] scsi2 : iSCSI Initiator over TCP/IP
[ 1156.607026] scsi 2:0:0:0: Direct-Access     IET      VIRTUAL-DISK     0    PQ: 0 ANSI: 4
[ 1156.607026] sd 2:0:0:0: [sdb] 73400320 512-byte hardware sectors (37581 MB)
[ 1156.607026] sd 2:0:0:0: [sdb] Write Protect is off
[ 1156.607026] sd 2:0:0:0: [sdb] Mode Sense: 77 00 00 08
[ 1156.607026] sd 2:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 1156.607111] sd 2:0:0:0: [sdb] 73400320 512-byte hardware sectors (37581 MB)
[ 1156.608275] sd 2:0:0:0: [sdb] Write Protect is off
[ 1156.608293] sd 2:0:0:0: [sdb] Mode Sense: 77 00 00 08
[ 1156.610635] sd 2:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 1156.610665]  sdb: unknown partition table
[ 1156.619320] sd 2:0:0:0: [sdb] Attached SCSI disk
[ 1156.619398] sd 2:0:0:0: Attached scsi generic sg2 type 0
[ 1163.865083] scsi3 : iSCSI Initiator over TCP/IP
[ 1164.132972] scsi 3:0:0:0: Direct-Access     IET      VIRTUAL-DISK     0    PQ: 0 ANSI: 4
[ 1164.135287] sd 3:0:0:0: [sdc] 2097152 512-byte hardware sectors (1074 MB)
[ 1164.136424] sd 3:0:0:0: [sdc] Write Protect is off
[ 1164.136438] sd 3:0:0:0: [sdc] Mode Sense: 77 00 00 08
[ 1164.138002] sd 3:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 1164.140341] sd 3:0:0:0: [sdc] 2097152 512-byte hardware sectors (1074 MB)
[ 1164.141443] sd 3:0:0:0: [sdc] Write Protect is off
[ 1164.141460] sd 3:0:0:0: [sdc] Mode Sense: 77 00 00 08
[ 1164.144746] sd 3:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 1164.144809]  sdc: unknown partition table
[ 1164.154241] sd 3:0:0:0: [sdc] Attached SCSI disk
[ 1164.154290] sd 3:0:0:0: Attached scsi generic sg3 type 0
[ 1170.533158] scsi4 : iSCSI Initiator over TCP/IP
[ 1170.798343] scsi 4:0:0:0: Direct-Access     IET      VIRTUAL-DISK     0    PQ: 0 ANSI: 4
[ 1170.798343] sd 4:0:0:0: [sdd] 2097152 512-byte hardware sectors (1074 MB)
[ 1170.801229] sd 4:0:0:0: [sdd] Write Protect is off
[ 1170.801263] sd 4:0:0:0: [sdd] Mode Sense: 77 00 00 08
[ 1170.806093] sd 4:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 1170.809875] sd 4:0:0:0: [sdd] 2097152 512-byte hardware sectors (1074 MB)
[ 1170.812347] sd 4:0:0:0: [sdd] Write Protect is off
[ 1170.812380] sd 4:0:0:0: [sdd] Mode Sense: 77 00 00 08
[ 1170.816757] sd 4:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 1170.816757]  sdd: unknown partition table
[ 1170.828375] sd 4:0:0:0: [sdd] Attached SCSI disk
[ 1170.828446] sd 4:0:0:0: Attached scsi generic sg4 type 0

Cross checking incoming sessions at scsi target (HARDY)..

In the meanwhile, in HARDY, the incoming sessions can be checked by:

HARDY# more /proc/net/iet/*
::::::::::::::
/proc/net/iet/session
::::::::::::::
tid:3 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.asm
sid:281474997486080 initiator:iqn.1993-08.org.debian:01:950a218cdd1
cid:0 ip:192.168.0.6 state:active hd:none dd:none
tid:2 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr
sid:844424984461824 initiator:iqn.1993-08.org.debian:01:950a218cdd1
cid:0 ip:192.168.0.6 state:active hd:none dd:none
tid:1 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.vote
sid:562949990973952 initiator:iqn.1993-08.org.debian:01:950a218cdd1
cid:0 ip:192.168.0.6 state:active hd:none dd:none
::::::::::::::
/proc/net/iet/volume
::::::::::::::
tid:3 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.asm
lun:0 state:0 iotype:fileio iomode:wt path:/dev/vg/asm
tid:2 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.ocr
lun:0 state:0 iotype:fileio iomode:wt path:/dev/vg/ocr
tid:1 name:iqn.2001-04.com.ubuntu:scsi.disk.vg.vote
lun:0 state:0 iotype:fileio iomode:wt path:/dev/vg/vote
root@gverma-laptop:/home/gverma#

Conclusion..

Congratulations! You have been successful in creating a poor man's scsi target and discovered them using open source softwares in an easy manner. Now GUTSY can read/write to /dev/vg/asm etc logical volumes. For more machines to do the same thing, similar setup is needed on them and they should also be access these volumes in a clustered fashion. This kind of setup is most beneficial for creating your own Oracle 10g RAC sandbox environment or for doing installs over shared disk.

In this case, we did not format the discovered logical volumes in ext3 or OCFS2 file system, because it is assumed that these volumes will be used as raw devices only.

Creating your own 10g RAC cluster at home using virtualbox and SAN targets (using openfiler + with/without USB hard disk)

Preface

After playing around with some open source technologies like virtualbox and openfiler, we are finally at a point when it is possible to create your own network lab of Oracle 10gR2 RAC cluster and ASM database using iscsi targets in an external hard disk drive. The last time I checked, USB 2.0 hard disk drives are cheaply available in the market.

If you do not want to use an external USB hard disk, then please go through the article A poor man's guide for creating iscsi targets without using external USB hard disks

All it takes is a little persistence and willingness to follow easy to follow tutorials. In the previous articles on this blog, I have talked about the building blocks of using virtualbox and openfiler open source softwares. If we combine that knowledge with Oracle 10g RAC setup, the following end configuration can be achieved:

Click here for a bigger view

The building blocks..

The operating system

First of all, you need to install a linux distribution of your choice using virtualbox. In this example, I chose SuSE linux 9.3. I feel that SuSE linux will be embraced by more and more enterprises in the near future, hence this decision.

For doing the installation, just boot from the Linux install media CD and then keep switching CDs as per request of the install process. It is always better if you have an ISO file of the installable media. You can get the SuSE media from www.opensuse.org

Always keep a decent amount of swap space for the OS installation. Also it is better to have a /tmp mount point as a separate mount point from the root (/) partition. These are some evergreen tips that a sysadmin friend of mine (Erik Niklas) had suggested to me a long time back.

A word of caution while deciding the VDI disk size for virtualbox machine (especially relevant for Version 1.5.6). The virtualbox disks cannot be resized!!

The dynamically expanding option means that the disk can grow UPTO the maximum limit and it will grow as per usage till the maximum limit. After that, it is not possible to resize it using some Vbox utility.A workaround may be to copy the device using partition editors, but till virtualbox comes out with an enhancement on this feature, better to be safe than sorry and keep a very high disk size - as per your judgement.

For building a Oracle 10g RAC configuration, it is better to focus all your installation efforts on one virtual machine and then clone the virtualbox disks (including the bootable disk) using the VboxManage clonevdi command. This way, you save a lot of duplicate effort.

The networking configuration - Public and Private IPs

The other networking building blocks of this configuration can be achieved by the following:

1) RAC needs a public IP, which is usually (but not limited to) eth0. It is advantageous to configure the public IP as host only networking interface when using virtualbox because it gives you the dual advantage of being able to SSH into the machine from outside (e.g. other rac node or another machine on the LAN) and also make internet work.

I tried to keep this as a static IP, but the downside is that the default gateway was somehow not working for me. However, if I kept this as DHCP interface, the default gateway and nameservers were being accessible out of the box.

A detailed example of how to do this is shown in the article Virtualbox Case Study: Making host only networking work between two Ubuntu Guest OS (virtual machine) on Windows Vista host

2) A RAC setup also needs a private network between the RAC nodes for global communication system (GCS) and synchronization. This is also usually referred to as the private interconnects. This is usually configured on a private subnet like 10.10.x.x or 192.168.x.x, wherein other network traffic would not impinge on the sacrosanct synchronization traffic between the RAC nodes.

A detailed example of how to do this is shown in the article Case study: Making Internal networking for talking between two linux guest OS (Ubuntu) on windows vista host

The Network Attached Storage or Storage Area Network setup..

NAS is hot! Well, what i mean is that its like configure once, use everywhere kind of thing. You setup iSCSI target logical volumes and then discover and access them from other machines. There are packages like linux-iscsi and open-iscsi which will help you do that. As per the latest announcements, both these open source projects have been merged into open-iscsi now.

Why should we choose Ubuntu for hard disk drivers? The partitions from a /dev/sd* devices gets discovered as separate devices, whereas all partitions in /dev/hd* get discovered as a single device.

There are quite promising open source softwares in this area too. openfiler and freenas are a few of them. For no particular reason, I picked openfiler and did a few experiments for discovering iscsi targets using it.

A detailed example of how to do this has been shown in the article Case Study: How to discover iscsi targets with linux-iscsi initiator package — Suse linux 9 (scsi Initiator) and openfiler (scsi target)

If you are using USB external hard disks for this exercise, there are many gotchas involved in their proper detection when using virtualbox on windows host. But some golden rules mentioned in this article can be useful: Virtualbox How to: Gotchas involved in making a USB external hard disk device work on windows host

A problem with openfiler 2.2 is that it forgets the previous logical volumes etc after reboots. This situation can result in frustration, especially when you see no LUNs being discovered from the initiators. If you read the following article, there is a good chance that you will be fine: iSCSI: no LUNs detected for session when using openfiler

The Oracle 10gR2 RAC setup

With the above components in place, the stage is now set for you to install the Oracle 10g CRS software. Needless to say, there are several detailed guides available on Oracle Technology Network on how to do this. Each Operating system has unique package pre-requisites for the Oracle software to be installed properly.

Some more guides area available on the novell.com website at http://www.novell.com/products/server/oracle/documents.html

Check out the guides under SUSE Enterprise Linux Server 9 tab. They are pretty detailed.

Some more guides are available at http://www.nextre.it/oracledocs/rac10gonsles9.html and http://www.oracle.com/technology/pub/articles/smiley_rac10g_install.html#oracle

Suggested reading..

It is very likely that you would have come across the widely read article by Jeffrey Hunter on a similar topic. In http://www.oracle.com/technology/pub/articles/hunter_rac10gr2_iscsi.html, he deals with creating a 10g RAC on Redhat linux.

If you do not want to use an external USB hard disk for iscsi SAN targets, then please go through the article A poor man's guide for creating iscsi targets without using external USB hard disks

iSCSI: no LUNs detected for session when using openfiler

A common problem...

If you are using openfiler, at some point of time, you are bound to reboot the openfiler OS/machine. After the reboot, a common problem is that openfiler forgets or loses the logical volume/volume group information.

So, as a workaround, you need to issue the following commands:

################################
# to discover the volume groups
################################
# vgscan   

#################################
# to discover the logical volumes
#################################
# lvscan  

##################################
# to discover the physical volumes
##################################
# pvscan

Then, you can display them using the commands vgdisplay, lvdisplay, pvdisplay. Similarly, there is a whole suite of commands in the same location.

Then, you STILL need to activate the discovered logical volumes using the -ay switch of lvchange:

# lvchange -ay openfiler/asm
# lvchange -ay openfiler/ocr
# lvchange -ay openfiler/vote

Now, if you run the lvscan or lvdisplay command, the status of the Logical volumes will show as ACTIVATED.

Remember..

AFTER you do this, You will STILL need to restart the iscsi-target service on openfiler machine.

openfiler $> service iscsi-target restart

If you fail to do this, you will see these messages in the dmesg output on the iscsi-initiator node (not the openfiler machine) - iSCSI: no LUNs detected for session. This message can be very confusing, especially since you will see that the session to the iscsi target WAS established:

openfiler $> dmesg
...
...
iSCSI: bus 0 target 0 = iqn.2006-01.com.openfiler:openfiler.vote
iSCSI: bus 0 target 0 portal 0 = address 10.143.213.248 port 3260 group 1
iSCSI: starting timer thread at 286340
iSCSI: bus 0 target 0 trying to establish session to portal 0, address 10.143.213.248 port 3260 group 1
iSCSI: bus 0 target 1 = iqn.2006-01.com.openfiler:openfiler.ocr
iSCSI: bus 0 target 1 portal 0 = address 10.143.213.248 port 3260 group 1
iSCSI: bus 0 target 1 trying to establish session to portal 0, address 10.143.213.248 port 3260 group 1
iSCSI: bus 0 target 0 established session #1, portal 0, address 10.143.213.248 port 3260 group 1
iSCSI: bus 0 target 2 = iqn.2006-01.com.openfiler:openfiler.asm
iSCSI: bus 0 target 2 portal 0 = address 10.143.213.248 port 3260 group 1
iSCSI: bus 0 target 1 established session #1, portal 0, address 10.143.213.248 port 3260 group 1
iSCSI: bus 0 target 2 trying to establish session to portal 0, address 10.143.213.248 port 3260 group 1
iSCSI: bus 0 target 3 = iqn.2006-01.com.openfiler:openfiler.test
iSCSI: bus 0 target 3 portal 0 = address 10.143.213.248 port 3260 group 1
iSCSI: bus 0 target 3 trying to establish session to portal 0, address 10.143.213.248 port 3260 group 1
iSCSI: bus 0 target 2 established session #1, portal 0, address 10.143.213.248 port 3260 group 1
iSCSI: no LUNs detected for session (bus 0 target 0) to iqn.2006-01.com.openfiler:openfiler.vote
iSCSI: no LUNs detected for session (bus 0 target 1) to iqn.2006-01.com.openfiler:openfiler.ocr
iSCSI: no LUNs detected for session (bus 0 target 2) to iqn.2006-01.com.openfiler:openfiler.asm
iSCSI: bus 0 target 3 established session #1, portal 0, address 10.143.213.248 port 3260 group 1

Case Study: How to discover iscsi targets with linux-iscsi initiator package -- Suse linux 9 (scsi Initiator) and openfiler (scsi target)

Birds eye view of the end configuration...

Here is the bird's eye view of the end configuration that we will achieve at the end of this article:

Click here for a bigger view..

My Grumblings with openfiler..

Those of you who have started experimented with openfiler may have started liking its features already. One of the biggest concerns that I have with openfiler is that it's administrative GUI is full of bugs or makes a lot of assumptions while working.

For example, it wouldn't show up logical volumes which were created in a iscsi external hard disk from another installation of openfiler virtual machine.

Somehow, CLI commands like pvscan, lvscan and vgscan are able to discover previously created physcial volumes, logical volumes and volume groups; but the front end GUI (http://<openfiler IP>:446) fails to do the same.

Although there is another open source product called freenas, I resisted to temptation to switch my loyalties too soon as no product is without bugs as such.

My requirement..

Anyways, my real requirement was to create a homegrown 10g RAC cluster using virtualbox virtual machines. For better or worse, I had chosen Suse Linux 9 (SP3) as the base operating system for the 10g RAC installation. A great reason was that many big customers like officedepot have chosen to implement 10g RAC on Suse Linux 9.3 . As time goes by, I feel that SuSE linux will become a more popular platform. Hence my persistence with this distribution.

There are ways to spoof the 10g RAC installation with 1 node too, but I wanted to simulate the real thing and be able to drive a Train-The-Trainer session for my teammates.

Looking back at my initial struggles...

I now realize that figuring out how to discover iscsi targets on Ubuntu was much easier. The experience is documented here: Combining Openfiler and Virtualbox (Ubuntu guest OS on windows host

My initial struggles were torn with anguish, especially because I realized very soon that I could not use open-iscsi linux package with SuSE linux 9 (2.6.5-7-244 kernel) at all. SuSE linux 10.x seems to have great support for it, though. This is simply because open-iscsi package works with kernels 2.6.14 and above only.

Tough luck there.

So, whats available for SuSE Linux 9 if you want to discover iscsi target devices?

Well, there are some options. The linux-iscsi package is very much available and with a little configuration, which is quite simplistic, it works great. A lot of people tried to woo me with other distributions like Oracle Enterprise Linux 5, which has iscsi-initiator-utils package built into it, but I stuck to my ground.

Here are some important distinctions between linux-iscsi and open-iscsi:

- The linux-iscsi package (aka iscsi-sfnet) reads /etc/iscsi.conf
- The open-iscsi package reads /etc/iscsid.conf. This package has an additional iscsiadm utility for discovering targets.

As of now, linux-iscsi and open-iscsi projects have been merged (as from their announcement) into one open-iscsi project.

Now, the difficult part: figuring out the setup ..

The most difficult part was figuring out the setup that worked. Eventually, after umpteen tries, it did work. On more than one occasion, I thought if it was even worth trying linux-iscsi initiator package with openfiler as iscsi target, iscsi-target drivers seemed more compatible with open-iscsi initiator package (this was the Ubuntu experience dominating my thinking).

However, I now realize that this perception was delusional. All I really needed was to a proper configuration of linux-iscsi package as iscsi initiator.

i will assume that the reader is conversant with the terms iscsi iniatiator/target.
If not, here is a crash course: iscsi targets are the LUNs or logical volumes in your NAS device , iscsi initiator is the client machine which wants to use these LUNs or Logical volumes. You dig?

With debug level 10 of iscsid process (# iscsid -d 10 &), i was getting the following error while discovering targets:

.. >> iscsid[17946]: connecting to 10.143.213.233:446
.. >> iscsid[17946]: connected local port 33785 to 10.143.213.233:446
.. >> iscsid[17946]: discovery session to 10.143.213.233:446 starting iSCSI login on fd 1
.. >> iscsid[17946]: sending login PDU with current stage 1, next stage 3, transit 0x80, isid 0x00023d000001
.. >> iscsid[17946]: >   InitiatorName=iqn.1987-05.com.cisco:01.51f06557c68
.. >> iscsid[17946]: >   InitiatorAlias=raclinux1
.. >> iscsid[17946]: >   SessionType=Discovery
.. >> iscsid[17946]: >   HeaderDigest=None
.. >> iscsid[17946]: >   DataDigest=None
.. >> iscsid[17946]: >   MaxRecvDataSegmentLength=8192
.. >> iscsid[17946]: >   X-com.cisco.PingTimeout=5
.. >> iscsid[17946]: >   X-com.cisco.sendAsyncText=Yes
.. >> iscsid[17946]: >   X-com.cisco.protocol=draft20
.. >> iscsid[17946]: wrote 48 bytes of PDU header
.. >> iscsid[17946]: wrote 248 bytes of PDU data
.. >> iscsid[17946]: socket 1 closed by target
.. >> iscsid[17946]: login I/O error, failed to receive a PDU
.. >> iscsid[17946]: retrying discovery login to 10.143.213.233
.. >> iscsid[17946]: disconnecting session 0x80b4890, fd 1
.. >> iscsid[17946]: discovery session to 10.143.213.233:446 sleeping for 2 seconds before next login attempt

I saw light at the end of the tunnel after trying a simple setup mentioned in http://www-941.ibm.com/collaboration/wiki/display/LinuxP/iSCSI

Lets talk about the experience in more detail now.

The setup on iscsi target (Openfiler) side..

[root@openfiler~]# uname -a
Linux openfiler.usdhcp.example.com 2.6.19.4-0.1.x86.i686.cmov #1 ..

I did not setup a network or subnet of allowed initiators for LUNs (as
can be seen here that the /etc/initiators.allow and /etc/initiators.deny files are
non-existent):

[root@openfiler~]# ls /etc/initiators.allow
ls: /etc/initiators.allow: No such file or directory 

[root@openfiler~]# ls /etc/initiators.deny
ls: /etc/initiators.deny: No such file or directory 

[root@openfiler~]# more /etc/ietd.conf
Target iqn.2006-01.com.openfiler:openfiler.testLun0Path=/dev/openfiler/test,Type=fileio

[root@openfiler~]# service iscsi-target status
ietd (pid 4164) is running...

Checking if the device drivers are loaded:

[root@openfiler~]# lsmod | grep scsi
iscsi_trgt             61788  4
scsi_mod              111756  2 sd_mod,usb_storage

Checking if the NAS device is discovered:

[root@openfiler~]# more /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: ST332083 Model: 3A               Rev: 3.AA
Type:   Direct-Access                    ANSI SCSI revision: 02

Checking what logical volumes have been discovered:

[root@openfiler~]# cat /proc/net/iet/session
tid:1 name:iqn.2006-01.com.openfiler:openfiler.test

Discover the volume groups, logical volumes and physical volumes:

[root@openfiler~]# vgscan
Reading all physical volumes.  This may take a while...
Found volume group "openfiler" using metadata type lvm2

[root@openfiler~]# lvscan
ACTIVE            '/dev/openfiler/ocr' [1.00 GB] inherit
ACTIVE            '/dev/openfiler/vote' [1.00 GB] inherit
ACTIVE            '/dev/openfiler/asm' [60.00 GB] inherit
ACTIVE            '/dev/openfiler/test' [32.00 MB] inherit

[root@openfiler~]# pvscan
PV /dev/sda2   VG openfiler   lvm2 [122.30 GB / 60.27 GB free]
Total: 1 [122.30 GB] / in use: 1 [122.30 GB] / in no VG: 0 [0   ]

As you can see, there are actually three more logical volumes that were discovered than
what we have configured in /etc/ietd.conf. We will deal with this later:

[root@openfiler~]# ls -l /dev/openfiler
total 0
lrwxrwxrwx  1 root root 25 Apr 24 09:58 asm -> /dev/mapper/openfiler-asm
lrwxrwxrwx  1 root root 25 Apr 24 09:58 ocr -> /dev/mapper/openfiler-ocr
lrwxrwxrwx  1 root root 26 Apr 24 12:07 test -> /dev/mapper/openfiler-test
lrwxrwxrwx  1 root root 26 Apr 24 09:59 vote -> /dev/mapper/openfiler-vote

The real deal-iscsi Initiator setup using linux-iscsi package on Suse Linux 9.3

raclinux1:~ # uname -a
Linux raclinux1 2.6.5-7.244-default #1 Mon Dec 12 18:32:25 UTC 2005 i686 i686 i386 GNU/Linux

Make sure the linux-iscsi package is installed:

raclinux1:/etc # rpm -qa | grep linux-iscsi
linux-iscsi-4.0.1-98

Show the discovered iscsi devices as of yet:

raclinux1:/etc # iscsi-ls
###############################################################################

iSCSI driver is not loaded

###############################################################################

Since the iscsi driver is missing, Load the iscsi driver (which is also known as the iscsi-Sfnet driver)

raclinux1:/etc # modprobe iscsi

Verify that the iscsi driver was loaded:

raclinux1:/etc # lsmod | grep scsi
iscsi                 182192  0
scsi_mod              112972  5 iscsi,sg,st,sd_mod,sr_mod
raclinux1:/etc #

Check what devices have been configured. Right now, no iscsi devices have been discovered:

raclinux1:/etc # iscsi-ls
*******************************************************************************
Cisco iSCSI Driver Version ... 4.0.198 ( 21-May-2004 )
*******************************************************************************
raclinux1:/etc #

Configure the /etc/iscsi.conf file for linux-iscsi - the most simplistic case -- This is SO the key.

Trivia
Initially, I had given port 446 in the DiscoveryAddress too and that was causing a very cryptic 'login I/O error, failed to receive a PDU' error.

I had searched all over the internet to resolve this error, including openfiler forums, only to find out that a few people resolved this by doing a firmware upgrade! Unfortunately, there is very little literature on the internet on this error. That is why I hope this article helps someone out there facing the same situation.

raclinux1:~ # more /etc/iscsi.conf
# this is the IP of the openfiler iscsi target machine
DiscoveryAddress=10.143.213.233

Verify that we have a unique IQN name for the initiator node (SuSE Linux 9.3):

raclinux1:~ # more /etc/initiatorname.iscsi
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1987-05.com.cisco:01.51f06557c68

Now, start up iscsid process with a high debug level to see what goes on behind the scenes.
I chose debug level 10 for no particular reason:

raclinux1:/etc # iscsid -d 10 &
[1] 30332
raclinux1:/etc # 1209056895.780916 >> iscsid[30332]: iSCSI debug level 10
1209056895.781428 >> iscsid[30332]: InitiatorName=iqn.1987-05.com.cisco:01.51f06557c68
1209056895.781790 >> iscsid[30332]: InitiatorAlias=raclinux1
1209056895.782101 >> iscsid[30332]: version 4.0.198 ( 21-May-2004)
1209056895.785327 >> iscsid[30333]: pid file fd 0
1209056895.785694 >> iscsid[30333]: locked pid file /var/run/iscsid.pid
1209056895.795251 >> iscsid[30333]: updating config 0xbfffeb10 from /etc/iscsi.conf
...
....
1209056895.799724 >> iscsid[30334]: sendtargets discovery process 0x80a80c0 starting, address 10.143.213.233:3260, continuous 1
1209056895.800315 >> iscsid[30334]: sendtargets discovery process 0x80a80c0 to 10.143.213.233:3260 using isid 0x00023d0000011209056895.802181 >> iscsid[30334]: connecting to 10.143.213.233:3260
1209056895.803657 >> iscsid[30334]: connected local port 34261 to 10.143.213.233:3260
1209056895.804189 >> iscsid[30334]: discovery session to 10.143.213.233:3260 starting iSCSI login on fd 1
1209056895.805081 >> iscsid[30334]: sending login PDU with current stage 1, next stage 3, transit 0x80, isid 0x00023d000001
1209056895.805415 >> iscsid[30334]: >    InitiatorName=iqn.1987-05.com.cisco:01.51f06557c68
1209056895.805807 >> iscsid[30334]: >    InitiatorAlias=raclinux1
1209056895.806120 >> iscsid[30334]: >    SessionType=Discovery
1209056895.806535 >> iscsid[30334]: >    HeaderDigest=None
1209056895.806918 >> iscsid[30334]: >    DataDigest=None
1209056895.807213 >> iscsid[30334]: >    MaxRecvDataSegmentLength=8192
1209056895.807515 >> iscsid[30334]: >    X-com.cisco.PingTimeout=5
1209056895.807910 >> iscsid[30334]: >    X-com.cisco.sendAsyncText=Yes
1209056895.808217 >> iscsid[30334]: >    X-com.cisco.protocol=draft20
1209056895.808555 >> iscsid[30334]: wrote 48 bytes of PDU header
1209056895.809044 >> iscsid[30334]: wrote 248 bytes of PDU data
1209056895.810896 >> iscsid[30333]: done starting discovery processes
...
...
1209056895.825881 >> iscsid[30334]: discovery login success to 10.143.213.233
1209056895.800928 >> iscsid[30334]: resolved 10.143.213.233 to 10.4294967183.4294967253.4294967273
...
...
TargetName=iqn.2006-01.com.openfiler:openfiler.test
1209056895.831110 >> iscsid[30334]: >    TargetAddress=10.143.213.233:3260,1
1209056895.831416 >> iscsid[30334]: discovery session to 10.143.213.233:3260 received text response, 88 data bytes, ttt 0xffffffff, final 0x80
...
...
1209056895.849821 >> iscsid[30333]: mkdir /var/lib
1209056895.850134 >> iscsid[30333]: mkdir /var/lib/iscsi
1209056895.850439 >> iscsid[30333]: opened bindings file /var/lib/iscsi/bindings
1209056895.850769 >> iscsid[30333]: locked bindings file /var/lib/iscsi/bindings
1209056895.851143 >> iscsid[30333]: scanning bindings file for 1 unbound sessions
1209056895.851580 >> iscsid[30333]: iSCSI bus 0 target 0 bound to session #1 to iqn.2006-01.com.openfiler:openfiler.test
1209056895.851906 >> iscsid[30333]: done scanning bindings file at line 11
1209056895.852320 >> iscsid[30333]: unlocked bindings file /var/lib/iscsi/bindings

Voila! A new virtual disk is discovered!

Paydirt! The iscsi targets are detected as per messages in /var/log/messages

iSCSI: 4.0.188.26 ( 21-May-2004) built for Linux 2.6.5-7.244-default
iSCSI: will translate deferred sense to current sense on disk command responses
iSCSI: control device major number 254 scsi15 : SFNet iSCSI driver
iSCSI:detected HBA host #15 iSCSI:
bus 0 target 0 = iqn.2006-01.com.openfiler:openfiler.test
iSCSI: bus 0 target 0 portal 0 = address
10.143.213.233 port 3260 group 1iSCSI: bus 0 target 0 established session #1, portal
0, address 10.143.213.233 port 3260 group 1
Vendor: Openfile  Model: Virtual disk      Rev: 0
Type:   Direct-Access                      ANSI SCSI
revision: 04
SCSI device sda: 65536 512-byte hdwr sectors (34 MB)
iSCSI: starting timer thread at 11948918
iSCSI: bus 0 target 0 trying to establish session to
portal 0, address 10.143.213.233 port 3260 group 1
SCSI device sda: drive cache: write through
sda: unknown partition table
Attached scsi disk sda at scsi15, channel 0, id 0, lun 0
Attached scsi generic sg0 at scsi15, channel 0, id 0, lun 0,  type 0
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.

Verifying that the discovery of target LUNs was indeed made:

raclinux1:~ # more /var/lib/iscsi/bindings
# iSCSI bindings, file format version 1.0.
# NOTE: this file is automatically maintained by the iSCSI daemon.
# You should not need to edit this file under most circumstances.
# If iSCSI targets in this file have been permanently deleted, you
# may wish to delete the bindings for the deleted targets.
#
# Format:
# bus   target  iSCSI
# id    id      TargetName
#
0       0       iqn.2006-01.com.openfiler:openfiler.test

Lets restart the linux-iscsi service again (without the debug this time):

raclinux1:/etc # rciscsi stop
Stopping iSCSI: sync umount sync iscsid

raclinux1:/etc # rciscsi start
Starting iSCSI: iscsi iscsid fsck/mount     done

raclinux1:/etc # rciscsi status
Checking for service iSCSI iSCSI driver is loaded
running

Check what devices were discovered:

raclinux1:/etc # iscsi-ls
*******************************************************************************
Cisco iSCSI Driver Version ... 4.0.198 ( 21-May-2004 )
*******************************************************************************
TARGET NAME             : iqn.2006-01.com.openfiler:openfiler.test
TARGET ALIAS            :
HOST NO                 : 18
BUS NO                  : 0
TARGET ID               : 0
TARGET ADDRESS          : 1.1.3923087114.0:0
SESSION STATUS          : DROPPED AT Thu Apr 24 10:19:16 2008
NO. OF PORTALS          : 1
Segmentation fault

raclinux1:/etc # fdisk -l /dev/sda

Disk /dev/sda: 33 MB, 33554432 bytes
2 heads, 32 sectors/track, 1024 cylinders
Units = cylinders of 64 * 512 = 32768 bytes

Disk /dev/sda doesn't contain a valid partition table

You can now partition the iscsi device using fdisk:

raclinux1:/etc # fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e   extended
p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1024, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1024, default 1024):
Using default value 1024

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
raclinux1:/etc #

raclinux1:/etc # fdisk -l /dev/sda

Disk /dev/sda: 33 MB, 33554432 bytes
2 heads, 32 sectors/track, 1024 cylinders
Units = cylinders of 64 * 512 = 32768 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1        1024       32752   83  Linux

raclinux1:/etc # ls -l /dev/disk
total 132
drwxr-xr-x   4 root root   4096 Apr 10 08:44 .
drwxr-xr-x  33 root root 118784 Apr 24 10:19 ..
drwxr-xr-x   2 root root   4096 Apr 24 10:21 by-id
drwxr-xr-x   2 root root   4096 Apr 24 10:21 by-path

raclinux1:/etc # ls -l /dev/disk/by-id
total 8
...
.. iscsi-iqn.2006-01.com.openfiler:openfiler.test-0 -> ../../sda
.. iscsi-iqn.2006-01.com.openfiler:openfiler.test-0-generic -> ../../sg0
.. iscsi-iqn.2006-01.com.openfiler:openfiler.test-0p1 -> ../../sda1

raclinux1:/etc # ls -l /dev/disk/by-path
total 8
.. ip-10.143.213.233-iscsi-iqn.2006-01.com.openfiler:openfiler.test-0 -> ../../sda
.. ip-10.143.213.233-iscsi-iqn.2006-01.com.openfiler:openfiler.test-0-generic -> ../../sg0
.. ip-10.143.213.233-iscsi-iqn.2006-01.com.openfiler:openfiler.test-0p1 -> ../../sda1
...

In the meanwhile, lets look at the sessions on openfiler server:

[root@openfiler~]# cat /proc/net/iet/session
tid:1 name:iqn.2006-01.com.openfiler:openfiler.test
sid:564049469047296 initiator:iqn.1987-05.com.cisco:01.51f06557c68
cid:0 ip:10.143.213.238 state:active hd:none dd:none

[root@openfiler~]# more /proc/net/iet/*
::::::::::::::
/proc/net/iet/session
::::::::::::::
tid:1 name:iqn.2006-01.com.openfiler:openfiler.test
sid:564049469047296 initiator:iqn.1987-05.com.cisco:01.51f06557c68
cid:0 ip:10.143.213.238 state:active hd:none dd:none
::::::::::::::
/proc/net/iet/session.xml
::::::::::::::
<?xml version="1.0" ?>

<info>

<target id="1" name="iqn.2006-01.com.openfiler:openfiler.test">
<session id="564049469047296" initiator="iqn.1987-05.com.cisco:01.51f06557c68">
<connection id="0" ip="10.143.213.238" state="active" hd="none" dd="none" />
</session>
</target>

</info>

::::::::::::::
/proc/net/iet/volume
::::::::::::::
tid:1 name:iqn.2006-01.com.openfiler:openfiler.test
lun:0 state:0 iotype:fileio iomode:wt path:/dev/openfiler/test
::::::::::::::
/proc/net/iet/volume.xml
::::::::::::::
<?xml version="1.0" ?>

<info>

<target id="1" name="iqn.2006-01.com.openfiler:openfiler.test">
<lun number="0" state= "0"iotype="fileio"iomode="wt" path="/dev/openfiler/test" />
</target>

</info>

Adding all the discovered LUNs to openfiler's published iscsi targets:

[root@openfiler~]# more /etc/ietd.conf
Target iqn.2006-01.com.openfiler:openfiler.test
Lun 0 Path=/dev/openfiler/test,Type=fileio
Targetiqn.2006-01.com.openfiler:openfiler.asm
Lun 1 Path=/dev/openfiler/asm,Type=fileio
Targetiqn.2006-01.com.openfiler:openfiler.ocr
Lun 2 Path=/dev/openfiler/ocr,Type=fileio
Targetiqn.2006-01.com.openfiler:openfiler.vote
Lun 3 Path=/dev/openfiler/vote,Type=fileio

[root@openfiler~]# service iscsi-target restart
Stopping iSCSI target service:                             [  OK  ]
Starting iSCSI target service:                             [  OK  ]

[root@openfiler~]# more /proc/net/iet/*
::::::::::::::
/proc/net/iet/session
::::::::::::::
tid:4 name:iqn.2006-01.com.openfiler:openfiler.vote
sid:282574492336640 initiator:iqn.1987-05.com.cisco:01.51f06557c68
cid:0 ip:10.143.213.238 state:active hd:none dd:none
tid:3 name:iqn.2006-01.com.openfiler:openfiler.ocr
sid:564049469047296 initiator:iqn.1987-05.com.cisco:01.51f06557c68
cid:0 ip:10.143.213.238 state:active hd:none dd:none
tid:2 name:iqn.2006-01.com.openfiler:openfiler.asm
sid:845524445757952 initiator:iqn.1987-05.com.cisco:01.51f06557c68
cid:0 ip:10.143.213.238 state:active hd:none dd:none
tid:1 name:iqn.2006-01.com.openfiler:openfiler.test
sid:1126999422468608 initiator:iqn.1987-05.com.cisco:01.51f06557c68
cid:0 ip:10.143.213.238 state:active hd:none dd:none
::::::::::::::
/proc/net/iet/session.xml
::::::::::::::
<?xml version="1.0" ?>

<info>

<target id="4" name="iqn.2006-01.com.openfiler:openfiler.vote">
<session id="282574492336640" initiator="iqn.1987-05.com.cisco:01.51f06557c68">
<connection id="0" ip="10.143.213.238" state="active" hd="none" dd="none" />
</session>
</target>

<target id="3" name="iqn.2006-01.com.openfiler:openfiler.ocr">
<session id="564049469047296" initiator="iqn.1987-05.com.cisco:01.51f06557c68">
<connection id="0" ip="10.143.213.238" state="active" hd="none" dd="none" />
</session>
</target>

<target id="2" name="iqn.2006-01.com.openfiler:openfiler.asm">
<session id="845524445757952" initiator="iqn.1987-05.com.cisco:01.51f06557c68">
<connection id="0" ip="10.143.213.238" state="active" hd="none" dd="none" />
</session>
</target>

<target id="1" name="iqn.2006-01.com.openfiler:openfiler.test">
<session id="1126999422468608" initiator="iqn.1987-05.com.cisco:01.51f06557c68">
<connection id="0" ip="10.143.213.238" state="active" hd="none" dd="none" />
</session>
</target>

</info>

::::::::::::::
/proc/net/iet/volume
::::::::::::::
tid:4 name:iqn.2006-01.com.openfiler:openfiler.vote
lun:0 state:0 iotype:fileio iomode:wt path:/dev/openfiler/asm
tid:3 name:iqn.2006-01.com.openfiler:openfiler.ocr
lun:0 state:0 iotype:fileio iomode:wt path:/dev/openfiler/asm
tid:2 name:iqn.2006-01.com.openfiler:openfiler.asm
lun:0 state:0 iotype:fileio iomode:wt path:/dev/openfiler/asm
tid:1 name:iqn.2006-01.com.openfiler:openfiler.test
lun:0 state:0 iotype:fileio iomode:wt path:/dev/openfiler/test
::::::::::::::
/proc/net/iet/volume.xml
::::::::::::::
<?xml version="1.0" ?>

<info>

<target id="4" name="iqn.2006-01.com.openfiler:openfiler.vote">
<lun number="0" state= "0"iotype="fileio"iomode="wt" path="/dev/openfiler/vote" />
</target>

<target id="3" name="iqn.2006-01.com.openfiler:openfiler.ocr">
<lun number="0" state= "0"iotype="fileio"iomode="wt" path="/dev/openfiler/ocr" />
</target>

<target id="2" name="iqn.2006-01.com.openfiler:openfiler.asm">
<lun number="0" state= "0"iotype="fileio"iomode="wt" path="/dev/openfiler/asm" />
</target>

<target id="1" name="iqn.2006-01.com.openfiler:openfiler.test">
<lun number="0" state= "0"iotype="fileio"iomode="wt" path="/dev/openfiler/test" />
</target>

</info>

In the meanwhile, On the initiator:

Now, let us check the devices detected (the iscsi-device command works more reliably):

raclinux1:/etc # iscsi-device /dev/sda
/dev/sda: 0   0   0       10.143.213.233   3260  iqn.2006-01.com.openfiler:openfiler.test
raclinux1:/etc # iscsi-device /dev/sdb
/dev/sdb: 0   1   0       10.143.213.233   3260  iqn.2006-01.com.openfiler:openfiler.asm
raclinux1:/etc # iscsi-device /dev/sdc
/dev/sdc: 0   2   0       10.143.213.233   3260  iqn.2006-01.com.openfiler:openfiler.vote
raclinux1:/etc # iscsi-device /dev/sdd
/dev/sdd: 0   3   0       10.143.213.233   3260  iqn.2006-01.com.openfiler:openfiler.ocr

raclinux1:/etc # fdisk -l /dev/sd*

Disk /dev/sda: 33 MB, 33554432 bytes
2 heads, 32 sectors/track, 1024 cylinders
Units = cylinders of 64 * 512 = 32768 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1        1024       32752   83  Linux

Disk /dev/sda1: 33 MB, 33538048 bytes
2 heads, 32 sectors/track, 1023 cylinders
Units = cylinders of 64 * 512 = 32768 bytes

Disk /dev/sdb: 64.4 GB, 64424509440 bytes
64 heads, 32 sectors/track, 61440 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 64.4 GB, 64424509440 bytes
64 heads, 32 sectors/track, 61440 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdc doesn't contain a valid partition table

Disk /dev/sdd: 64.4 GB, 64424509440 bytes
64 heads, 32 sectors/track, 61440 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdd doesn't contain a valid partition table

raclinux1:/etc # more /var/lib/iscsi/bindings
# iSCSI bindings, file format version 1.0.
# NOTE: this file is automatically maintained by the iSCSI daemon.
# You should not need to edit this file under most circumstances.
# If iSCSI targets in this file have been permanently deleted, you
# may wish to delete the bindings for the deleted targets.
#
# Format:
# bus   target  iSCSI
# id    id      TargetName
#
0       0       iqn.2006-01.com.openfiler:openfiler.test
0       1       iqn.2006-01.com.openfiler:openfiler.asm
0       2       iqn.2006-01.com.openfiler:openfiler.vote
0       3       iqn.2006-01.com.openfiler:openfiler.ocr

************************************************************************************************
Caveat:

Somehow, the iscsi-ls utility was not working. However, the devices were accesible all right.

Instead, the iscsi-device command works beautifully.
************************************************************************************************

raclinux1:/etc # iscsi-ls
*******************************************************************************
Cisco iSCSI Driver Version ... 4.0.198 ( 21-May-2004 )
*******************************************************************************
TARGET NAME             : iqn.2006-01.com.openfiler:openfiler.test
TARGET ALIAS            :
HOST NO                 : 20
BUS NO                  : 0
TARGET ID               : 0
TARGET ADDRESS          : 1.1.3923087114.0:0
SESSION STATUS          : DROPPED AT Thu Apr 24 10:43:41 2008
NO. OF PORTALS          : 1
Segmentation fault

raclinux1:/etc # echo $?
139

Conclusion..

This proves that linux-iscsi package can be made to work on 2.6.5-7.x distributions or for any other linux distribution less than 2.6.14. So if open-iscsi was not meant to compile on your distribution, do not despair. There are other avenues. This article also serves to demonstrate how to use the command line interface of the openfiler product better, as compared to the GUI console.

It also professes that there are some caveats in openfiler, but if we know our way around them, life is good.

Virtualbox How to: Gotchas involved in making a USB external hard disk device work on windows host

Preface

Sometime back, I installed Openfiler on a external hard disk (manufacturer: Iomega)that I had at home (running Windows Vista Home Premium). This was meant to be a proof of concept and pretty soon, I had to bring it to office desktop (running Windows XP SP2) to make it work with some other virtualbox virtual machines that I had built in my spare time.

Troubles in recognizing the USB external hard disk on windows XP

However, the problem that I faced on Windows XP SP2 host operating system was that the openfiler Virtual machine (using virtualbox 1.5.6 of course) would not recognize the USB external hard disk at all! The same USB device was working fine on windows vista.

After struggling with this problem for a while, I found a quick workaround (Not sure if it is documented in the user manual or not) and also tested some reset steps on windows, which seem to work with the Iomega external USB hard disk.

These steps should also be relevant to other hard disk models in general.

Some more details...

In my case, windows had detected the USB drive and had even installed the Virtualbox driver for it. USB support was enabled for the openfiler virtual machine in the settings. The external hard disk was selected, but still no iscsi drive showed up on doing a fdisk -l

When to know a device is being accessed..

Well, if the device makes a whirring noise and the light blinks or comes on, it is enough to know that the device is being accessed internally and something good happened in terms of making the USB device work.

The setup that worked..

However, this setup worked after a little tinkering:

Firstly, make sure the USB controller and EHCI controller is enabled. Then choose the USB external hard disk device from the list of all USB devices and enabled it.

Click here for a bigger view



Make sure no other Virtual machine is accessing the same USB device. From what I have seen, if the same device is accessed by other virtualbox VMs, an exclusive lock is acquired on it, which makes it difficult to be used by other virtual machines.

This is true with windows too. Windows will stop seeing the USB hard disk when it is being used by a virtualbox virtual machine. At least, I have seen this behaviour with virtualbox 1.5.6

Click here for a bigger view



Make sure that Windows is able to detect the USB device first. If not, then you need to troubleshoot that first. Usually, powering off the device, unplugging it from the USB port and restarting the windows PC/desktop/laptop works great. (Hey, its windows).

You want to see something like this in your system tray to be sure of this.



Now you know for a fact that windows, your guest Operating system, detected the USB drive fine. So now, there is a good chance that virtualbox will detect it too.

Fire it up..

Power up the virtual machine and check if the USB device got detected. A simple fdisk -l will show it as a iscsi device, with names like /dev/sda, /dev/sdb or so. If you do not see anything of the sort, you can be certain that the device did not get detected. Another visual way to know is to see if the light on USB device comes up or not. No light, no access.

The manual workaround if the device is still now seen after powering up the virtual machine ..

However, there is still another trick up our sleeve that can be tried.

Click here for a bigger view



At the bottom message bar of the virtual machine, right click on the USB plug icon. You will see that a list of the detected USB devices comes up. Select the external hard disk. Unless another virtual machine is running that is using the same USB device, there is a very good chance that the light on the hard disk will light up like this:



At this point, you may also see a windows dialog box saying that a virtualbox USB device was detected and the driver needs to be loaded:

Double check before you congratulate yourself..

All right! So it looks like your device got accessed after all.

There are many ways to confirm if it really got detected by the virtual machine as a scsi device or not:

1) Check the fdisk -l command's output:

Click here for a bigger view



2) Check the contents of /proc/scsi/scsi file:

Click here for a bigger view



3) Check the content of /var/log/messages or the output of dmesg command:

Click here for a bigger view

A quick Recap...

Once again, if this does not work, just go back to the basics:

1) Power off the USB device (when applicable) and take it out

2) Reboot windows

3) Plug in the USB device

4) Make sure windows detects it

5) Make sure the virtual machine has the USB device enabled in its setup

6) Start up the virtual machine. Make sure no other virtual machine is using the same USB drive.

7) If you do not see the USB device, force it to be attached by right click on USB plug icon in bottom bar of virtualbox VM application window

Hope that helps.