OpenSolaris as SCSI target

Ceri Davies — Mon, 19 Nov 2007 21:53:00 +0000

The OpenSolaris COMSTAR project released their first set of bits at the end of last week.

The code drop, SUNWsmft, comprises support for using an OpenSolaris system as a SCSI target, which means that OpenSolaris now supports native CIFS, iSCSI, SCSI and NFS with a transactional COW filesystem, runnable on a system that costs less than £20k (half that in certain industries) for 24TB, explaining why NetApp are very scared indeed.

I had to try it out as soon as possible. Luckily the code drop works on Solaris 10 as well as OpenSolaris, and I had Solaris 10 systems and HBAs available for testing. In order to isolate this from our real SANs, I just ran a fibre channel cable between the HBAs, so was somewhat wary that I had no idea how to get a HBA to do arbitrated loop. Turns out that isn’t a problem, and here’s all I had to do to get this working.

Install SUNWsmft

Grab the package and install it.

 # fetch http://opensolaris.org/os/project/comstar/files/SUNWstmf.sparc.tar.gz
 # gzip -dc SUNWstmf.sparc.tar.gz | tar xf -
 # yes | pkgadd -d . SUNWstmf

Choose target ports

The postinstall script loads the kernel modules straight away, so now you just have to choose a HBA port to put in target mode using the qlt driver. My test target system has two dual port HBAs and I couldn’t remember which one had the cable connected, so I just did them all and rebooted:

# update_drv -d -i '"pciex1077,2432"' qlc
# update_drv -a -i '"pciex1077,2432"' qlt
# init 6

When the machine comes back up, qlt attaches and the HBAs automatically negotiate an arbitrated loop. Great.

Create LU backing

I’m going to use ZFS zvols because I don’t want to wait for mkfile to finish (I have to use a zpool with a funny name since I’m doing this in a rush and this is what I have):

# zfs create archive/stmf
# zfs create archive/stmf/lun
# for i in `jot 10 0`; do zfs create -V 1g archive/stmf/lun/000$i; done

Register LUs

Now you have to tell the COMSTAR bits about the LUs you created.

# for i in `jot 10 0`; do sbdadm lu-create -r /dev/zvol/dsk/archive/stmf/lun/000$i; done
# sbdadm lu-list

Found 10 LU(s)

              GUID                  DATA SIZE     SOURCE
--------------------------------  -------------  ----------------
6000ae4000144f86b97e4741f80e000a     1073676288  /dev/zvol/dsk/archive/stmf/lun/0009
6000ae4000144f86b97e4741f80d0009     1073676288  /dev/zvol/dsk/archive/stmf/lun/0008
6000ae4000144f86b97e4741f80c0008     1073676288  /dev/zvol/dsk/archive/stmf/lun/0007
6000ae4000144f86b97e4741f80a0007     1073676288  /dev/zvol/dsk/archive/stmf/lun/0006
6000ae4000144f86b97e4741f8070006     1073676288  /dev/zvol/dsk/archive/stmf/lun/0005
6000ae4000144f86b97e4741f8060005     1073676288  /dev/zvol/dsk/archive/stmf/lun/0004
6000ae4000144f86b97e4741f8060004     1073676288  /dev/zvol/dsk/archive/stmf/lun/0003
6000ae4000144f86b97e4741f8050003     1073676288  /dev/zvol/dsk/archive/stmf/lun/0002
6000ae4000144f86b97e4741f8040002     1073676288  /dev/zvol/dsk/archive/stmf/lun/0001
6000ae4000144f86b97e4741f8030001     1073676288  /dev/zvol/dsk/archive/stmf/lun/0000

Create views

Now I have to create a view to allow the initiators access to the LUs I’ve created. Since I don’t know what I’m doing at this point, I just open them all; in production we’ll use host groups (“stmfadm create-view”) but I’m just throwing it open here:

# stmfadm add-view 6000AE4000144F86B97E4741F8030001
# stmfadm list-view -l 6000AE4000144F86B97E4741F8030001
View Entry: 0
    Host group   : All
    Target group : All
    LUN          : 0

Reinitialize the link on the initiator

In order to get the initiator to see the new LU, I have to do the following (one or other of these three commands always works, but no single one ever works 100%):

# luxadm -e forcelip /devices/pci@1d,700000/QLGC,qlc@2/fp@0,0:devctl
# cfgadm -al
# devfsadm -C -v
#  format
Searching for disks...done

c3t210000E08B945E9Fd0: configured with capacity of 1023.88MB


AVAILABLE DISK SELECTIONS:
       0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@1f,700000/scsi@2/sd@0,0
       1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@1f,700000/scsi@2/sd@1,0
       2. c1t2d0 <SEAGATE-ST373207LSUN72G-045A-68.37GB>
          /pci@1f,700000/scsi@2/sd@2,0
       3. c1t3d0 <SEAGATE-ST373207LSUN72G-045A-68.37GB>
          /pci@1f,700000/scsi@2/sd@3,0
       4. c3t210000E08B945E9Fd0 <SUN-COMSTAR-1.0 cyl 32764 alt 2 hd 2 sec 32>
          /pci@1d,700000/QLGC,qlc@2/fp@0,0/ssd@w210000e08b945e9f,0
Specify disk (enter its number):

yay!

"OpenSolaris as SCSI target" by Ceri

Tue, 11 Dec 2007 22:03:06 +0000

Hi Ceri, Yes, it's a little weird :) These are not my binaries, they're the ones provided by the COMSTAR project and yes, they do work on Solaris 10. Specifically, it's Solaris 10 update 3 with a reasonable number of patches. I'm not sure what compiler the COMSTAR team used, but these ones definitely work. PS I've emailed this response as well.

"OpenSolaris as SCSI target" by CeriDavies@Hotmail.com

Tue, 11 Dec 2007 21:34:53 +0000

Hi, my name is Ceri (Ifor) Davies, I am writing a qlogic target mode driver for solaris 10, and am thinking of switching to Comstar. Are you recompiling for solaris 10, or are you saying your nevada binaries worked with solaris 10? That would surprise me, since nevada uses stdio 11 (I think). The qlogic qlc driver panics on solaris 10 if compiled with studio 11, and has to be compiled with studio 10. P.S. Your having the same name is weird

Within Reason: OpenSolaris as SCSI target