OpenSolaris as SCSI target

Posted by Ceri Davies Mon, 19 Nov 2007 21:53:00 GMT

The OpenSolaris COMSTAR project released their first set of bits at the end of last week.

The code drop, SUNWsmft, comprises support for using an OpenSolaris system as a SCSI target, which means that OpenSolaris now supports native CIFS, iSCSI, SCSI and NFS with a transactional COW filesystem, runnable on a system that costs less than £20k (half that in certain industries) for 24TB, explaining why NetApp are very scared indeed.

I had to try it out as soon as possible. Luckily the code drop works on Solaris 10 as well as OpenSolaris, and I had Solaris 10 systems and HBAs available for testing. In order to isolate this from our real SANs, I just ran a fibre channel cable between the HBAs, so was somewhat wary that I had no idea how to get a HBA to do arbitrated loop. Turns out that isn’t a problem, and here’s all I had to do to get this working.

Install SUNWsmft

Grab the package and install it.

 # fetch http://opensolaris.org/os/project/comstar/files/SUNWstmf.sparc.tar.gz
 # gzip -dc SUNWstmf.sparc.tar.gz | tar xf -
 # yes | pkgadd -d . SUNWstmf

Choose target ports

The postinstall script loads the kernel modules straight away, so now you just have to choose a HBA port to put in target mode using the qlt driver. My test target system has two dual port HBAs and I couldn’t remember which one had the cable connected, so I just did them all and rebooted:

# update_drv -d -i '"pciex1077,2432"' qlc
# update_drv -a -i '"pciex1077,2432"' qlt
# init 6

When the machine comes back up, qlt attaches and the HBAs automatically negotiate an arbitrated loop. Great.

Create LU backing

I’m going to use ZFS zvols because I don’t want to wait for mkfile to finish (I have to use a zpool with a funny name since I’m doing this in a rush and this is what I have):

# zfs create archive/stmf
# zfs create archive/stmf/lun
# for i in `jot 10 0`; do zfs create -V 1g archive/stmf/lun/000$i; done

Register LUs

Now you have to tell the COMSTAR bits about the LUs you created.

# for i in `jot 10 0`; do sbdadm lu-create -r /dev/zvol/dsk/archive/stmf/lun/000$i; done
# sbdadm lu-list

Found 10 LU(s)

              GUID                  DATA SIZE     SOURCE
--------------------------------  -------------  ----------------
6000ae4000144f86b97e4741f80e000a     1073676288  /dev/zvol/dsk/archive/stmf/lun/0009
6000ae4000144f86b97e4741f80d0009     1073676288  /dev/zvol/dsk/archive/stmf/lun/0008
6000ae4000144f86b97e4741f80c0008     1073676288  /dev/zvol/dsk/archive/stmf/lun/0007
6000ae4000144f86b97e4741f80a0007     1073676288  /dev/zvol/dsk/archive/stmf/lun/0006
6000ae4000144f86b97e4741f8070006     1073676288  /dev/zvol/dsk/archive/stmf/lun/0005
6000ae4000144f86b97e4741f8060005     1073676288  /dev/zvol/dsk/archive/stmf/lun/0004
6000ae4000144f86b97e4741f8060004     1073676288  /dev/zvol/dsk/archive/stmf/lun/0003
6000ae4000144f86b97e4741f8050003     1073676288  /dev/zvol/dsk/archive/stmf/lun/0002
6000ae4000144f86b97e4741f8040002     1073676288  /dev/zvol/dsk/archive/stmf/lun/0001
6000ae4000144f86b97e4741f8030001     1073676288  /dev/zvol/dsk/archive/stmf/lun/0000

Create views

Now I have to create a view to allow the initiators access to the LUs I’ve created. Since I don’t know what I’m doing at this point, I just open them all; in production we’ll use host groups (“stmfadm create-view”) but I’m just throwing it open here:

# stmfadm add-view 6000AE4000144F86B97E4741F8030001
# stmfadm list-view -l 6000AE4000144F86B97E4741F8030001
View Entry: 0
    Host group   : All
    Target group : All
    LUN          : 0

Reinitialize the link on the initiator

In order to get the initiator to see the new LU, I have to do the following (one or other of these three commands always works, but no single one ever works 100%):

# luxadm -e forcelip /devices/pci@1d,700000/QLGC,qlc@2/fp@0,0:devctl
# cfgadm -al
# devfsadm -C -v
#  format
Searching for disks...done

c3t210000E08B945E9Fd0: configured with capacity of 1023.88MB


AVAILABLE DISK SELECTIONS:
       0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@1f,700000/scsi@2/sd@0,0
       1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@1f,700000/scsi@2/sd@1,0
       2. c1t2d0 <SEAGATE-ST373207LSUN72G-045A-68.37GB>
          /pci@1f,700000/scsi@2/sd@2,0
       3. c1t3d0 <SEAGATE-ST373207LSUN72G-045A-68.37GB>
          /pci@1f,700000/scsi@2/sd@3,0
       4. c3t210000E08B945E9Fd0 <SUN-COMSTAR-1.0 cyl 32764 alt 2 hd 2 sec 32>
          /pci@1d,700000/QLGC,qlc@2/fp@0,0/ssd@w210000e08b945e9f,0
Specify disk (enter its number):

yay!