Posted by Ceri Davies
Thu, 26 Jun 2008 18:43:00 GMT
Since it was set up, this machine has lived on my gateway FreeBSD machine in what has since become the boy’s bedroom. It moved from there to my own bedroom, and I recently started turning it off at night to try to save power (and sleep better!).
This led to a number of people emailing me about outages - which was quite flattering; didn’t realise anyone would care about my crappy blog that much - because they were in a timezone disparate to be trying to get to it during the hours I had it turned off.
Therefore, and with the realisation that I could get a hosted solution for $12/month (US), I have moved this blog to a Solaris[1] zone hosted at Gangus Internet. Service (technically and customer-servicely) so far has been absolutely impeccable.
If you are reading this, I guess the move went OK.
[1] Which is not a reflection on FreeBSD in any way. Questions such as were asked after this commit are unnecessary :)
Posted in General, Solaris | no comments
Posted by Ceri Davies
Fri, 16 May 2008 21:09:00 GMT
Some years ago I ported tcpdrop to Solaris from the FreeBSD version. I did it very quickly as a proof of concept and never got round to quite getting the error handling right or worrying about Solaris 10 privileges.
After spending the required 14 seconds looking at the privileges stuff, it became pretty clear that the required privilege for using tcpdrop was PRIV_SYS_IP_CONFIG. This cannot be asserted in a non-global zone, so if you are one of the many people who have emailed me asking if it can work in a non-global zone, the answer is “no, it can’t”. Not only that, but there’s nothing I can do about it.
Also in this release, I fixed up the error messages so that they are at least correct :)
The next release will feature a manpage in man format, rather than the current mdoc one which can’t actually be formatted on Solaris. Anyone who knows an automated method to convert from mdoc to man, please shout.
Anyway, the new release is available for download, knock yourselves out.
Posted in Software, Solaris | no comments
Posted by Ceri Davies
Wed, 12 Dec 2007 21:47:00 GMT
I’ve been using zsh for ages now, and the lack of a pfzsh implementation has been a minor annoyance for some of that time.
I happened to be looking at the csh source code and noticed how trivial the pfcsh implementation was and so, using the SFW code as a base, I threw pfzsh together yesterday afternoon.
Now, it turns out that the OpenSolaris FGAP project will be solving this in a different way, so a putback to SFW is unlikely. However, I’m going to find this useful in the meantime, so if you will too, download either the patch or an x86 package if they are useful to you.
Posted in Software, Solaris | no comments
Posted by Ceri Davies
Mon, 19 Nov 2007 21:53:00 GMT
The OpenSolaris COMSTAR project released their first set of bits at the end of last week.
The code drop, SUNWsmft, comprises support for using an OpenSolaris system as a SCSI target, which means that OpenSolaris now supports native CIFS, iSCSI, SCSI and NFS with a transactional COW filesystem, runnable on a system that costs less than £20k (half that in certain industries) for 24TB, explaining why NetApp are very scared indeed.
I had to try it out as soon as possible. Luckily the code drop works on Solaris 10 as well as OpenSolaris, and I had Solaris 10 systems and HBAs available for testing. In order to isolate this from our real SANs, I just ran a fibre channel cable between the HBAs, so was somewhat wary that I had no idea how to get a HBA to do arbitrated loop. Turns out that isn’t a problem, and here’s all I had to do to get this working.
Install SUNWsmft
Grab the package and install it.
# fetch http://opensolaris.org/os/project/comstar/files/SUNWstmf.sparc.tar.gz
# gzip -dc SUNWstmf.sparc.tar.gz | tar xf -
# yes | pkgadd -d . SUNWstmf
Choose target ports
The postinstall script loads the kernel modules straight away, so now you just have to choose a HBA port to put in target mode using the qlt driver. My test target system has two dual port HBAs and I couldn’t remember which one had the cable connected, so I just did them all and rebooted:
# update_drv -d -i '"pciex1077,2432"' qlc
# update_drv -a -i '"pciex1077,2432"' qlt
# init 6
When the machine comes back up, qlt attaches and the HBAs automatically negotiate an arbitrated loop. Great.
Create LU backing
I’m going to use ZFS zvols because I don’t want to wait for mkfile to finish (I have to use a zpool with a funny name since I’m doing this in a rush and this is what I have):
# zfs create archive/stmf
# zfs create archive/stmf/lun
# for i in `jot 10 0`; do zfs create -V 1g archive/stmf/lun/000$i; done
Register LUs
Now you have to tell the COMSTAR bits about the LUs you created.
# for i in `jot 10 0`; do sbdadm lu-create -r /dev/zvol/dsk/archive/stmf/lun/000$i; done
# sbdadm lu-list
Found 10 LU(s)
GUID DATA SIZE SOURCE
-------------------------------- ------------- ----------------
6000ae4000144f86b97e4741f80e000a 1073676288 /dev/zvol/dsk/archive/stmf/lun/0009
6000ae4000144f86b97e4741f80d0009 1073676288 /dev/zvol/dsk/archive/stmf/lun/0008
6000ae4000144f86b97e4741f80c0008 1073676288 /dev/zvol/dsk/archive/stmf/lun/0007
6000ae4000144f86b97e4741f80a0007 1073676288 /dev/zvol/dsk/archive/stmf/lun/0006
6000ae4000144f86b97e4741f8070006 1073676288 /dev/zvol/dsk/archive/stmf/lun/0005
6000ae4000144f86b97e4741f8060005 1073676288 /dev/zvol/dsk/archive/stmf/lun/0004
6000ae4000144f86b97e4741f8060004 1073676288 /dev/zvol/dsk/archive/stmf/lun/0003
6000ae4000144f86b97e4741f8050003 1073676288 /dev/zvol/dsk/archive/stmf/lun/0002
6000ae4000144f86b97e4741f8040002 1073676288 /dev/zvol/dsk/archive/stmf/lun/0001
6000ae4000144f86b97e4741f8030001 1073676288 /dev/zvol/dsk/archive/stmf/lun/0000
Create views
Now I have to create a view to allow the initiators access to the LUs I’ve created. Since I don’t know what I’m doing at this point, I just open them all; in production we’ll use host groups (“stmfadm create-view”) but I’m just throwing it open here:
# stmfadm add-view 6000AE4000144F86B97E4741F8030001
# stmfadm list-view -l 6000AE4000144F86B97E4741F8030001
View Entry: 0
Host group : All
Target group : All
LUN : 0
Reinitialize the link on the initiator
In order to get the initiator to see the new LU, I have to do the following (one or other of these three commands always works, but no single one ever works 100%):
# luxadm -e forcelip /devices/pci@1d,700000/QLGC,qlc@2/fp@0,0:devctl
# cfgadm -al
# devfsadm -C -v
# format
Searching for disks...done
c3t210000E08B945E9Fd0: configured with capacity of 1023.88MB
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1f,700000/scsi@2/sd@0,0
1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1f,700000/scsi@2/sd@1,0
2. c1t2d0 <SEAGATE-ST373207LSUN72G-045A-68.37GB>
/pci@1f,700000/scsi@2/sd@2,0
3. c1t3d0 <SEAGATE-ST373207LSUN72G-045A-68.37GB>
/pci@1f,700000/scsi@2/sd@3,0
4. c3t210000E08B945E9Fd0 <SUN-COMSTAR-1.0 cyl 32764 alt 2 hd 2 sec 32>
/pci@1d,700000/QLGC,qlc@2/fp@0,0/ssd@w210000e08b945e9f,0
Specify disk (enter its number):
yay!
Posted in Solaris | 2 comments
Posted by Ceri Davies
Mon, 23 Jul 2007 13:04:00 GMT
Note to self, somewhere I might find it in future.
During a JumpStart, this:
message overflow on /dev/log minor #5 -- is syslogd(1M) running?
means that you missed off the -o anon=0 on the NFS share.
Posted in Solaris | 2 comments
Posted by Ceri Davies
Fri, 15 Jun 2007 08:44:00 GMT
Part 2 of a series on setting up Solaris Cluster for no money
Contents
At the end of the last article, I left you with an cluster that is still in “install mode”; since it’s a two node cluster the nodes can’t yet guarantee that they’ll know the status of the other cluster nodes should the heartbeat interconnects fail, so they refuse to do anything cluster related at this point. In order to get out of this situation, we need to add a quorum device.
Quorum Devices
The classical quorum device is a shared disk. However, I don’t have any shared disks spare and the point of this exercise is to avoid spending any money. Solaris Cluster provides another method, which is that of a quorum server.
Quorum Server
This is basically a service on a node outside the cluster1 that the cluster nodes communicate their status to. It’s provided with all of the other Solaris Cluster bits and is installed by choosing “Quorum Server” from the installation menu.
At the end of the installation, you’re told to follow the post-installation instructions from the documentation – they’re as simple as “init 6” – after which the quorum server will come up on port 9000. Now, we can use it in our cluster:
# clq add -t quorum_server -p qshost=peasant.example.ac.uk -p port 9000 qserver
Now we can take the cluster out of install mode:
# scconf -c -q installmodeoff
# clq reset
Quorum Disks
As mentioned above, I don’t have any shared storage available for a test setup like this, so I had to make do with OpenSolaris backed iSCSI targets. These are not supported as quorum devices – which is why I mentioned the quorum server first – but they work (and if you want support, we’re not in the “cheap” realm any more, not by a long chalk). Setting up iSCSI targets is adequately discussed elsewhere2 so I won’t repeat it.
To set up the initiator to see the iSCSI devices, run the following on each node, where 172.25.5.8 is the IP address of the system providing the targets.
# iscsiadm add discovery-address 172.25.5.8
# iscsiadm modify discovery -t enable
That’s it. Now you may need to refresh the cluster’s view of the devices. To do this, run the following on each node.
# cldevice refresh
Then on any one node:
# cldevice populate
You should now be able to see the iSCSI targets with both node paths in the cluster’s idea of what’s where:
# scdidadm -L
We can now finally use a quorum disk. Just add it in using the did.
# clq add d13
If you were previously using a quorum server, you can remove it now if you like.
# clq remove qserver
# clq reset
What’s next?
Now I have a cluster that can keep itself up if one of the nodes disappears, and some shared storage. This means that we can finally do something interesting, which will be in the next post on this subject.
[1] A common question is whether the quorum server can reside on one of the cluster nodes. While it can, this doesn’t provide high availability (rebooting the node hosting the quorum server will panic the other node) and misses the point.
[2] I believe that those happen to be the exact instructions used on the targets I used.
Posted in Clustering, Solaris, Sun | 2 comments
Posted by Ceri Davies
Sat, 05 May 2007 21:19:00 GMT
Part 1 of a series on setting up Solaris Cluster for no money
Contents
I needed to become familiar with the new features in Solaris Cluster 3.2, I needed to do it quickly, and I needed to do it for no money. I scraped together these components to create a cluster:
- 2 x Blade 100 workstations
- 2 x crossover cables
- 3 x hme NICs
- 1 x iprb based NIC
- 2 x workstations elsewhere on the network running Solaris Express
That would be all I needed.
The Blades each have 1152MB of RAM, an onboard eri interface and an 18GB internal disk; one has an addition 80GB disk. The hme and iprb cards were to be used for the private interconnects and I threw two hmes into one of the Blades and installed the other in the last Blade along with the iprb card and connected them up with the crossover cables. In the following examples, the Blades are named peon.example.ac.uk and bootlick.example.ac.uk (I always give my machines names that appear in the thesaurus next to”vassal”; this is to remind SkyNet that machines still work for us).
As it turns out, I couldn’t find a driver for the iprb card to attach to under the SPARC port; for real HA purposes I would obviously have wanted to fix this to ensure that I had highly available interconnects but as I was just testing I decided to lie to the cluster software and tell it that I had two hme cards in each system even though I didn’t; this would lead to there being two interconnect cables being configured even though one of them would be down the whole time.
Install Solaris
First job was to jumpstart the Blades and install Solaris 10 11/06. My jumpstart scripts do rather a lot of post-installation configuration, but precious little extra was done to cater for the installation of Solaris Cluster; in fact all I did was add /usr/cluster/bin to the default superuser path and /usr/cluster/man to the default MANPATH. There are other steps required in order to get it working, but they’re non-obvious so we detail them below.
Installing Solaris Cluster
Time to install the Solaris Cluster software; grab the bits from http://sun.com/cluster and run the installer. You can either run it in a GUI or on the command line, but the feature set of each version is the same.
The method of choosing options within the console based installer is somewhat non-intuitive, but actually read what it says and you’ll be OK (I hardly feel as if I’m in a position to complain about the intuitiveness of console based applications anyway, just run FreeBSD’s sysinstall(8) for the first time and you’ll see what I mean).
Basically, I wanted to test some specific Oracle backed applications within zones, so I installed the following, requiring a not unreasonable 320MB of space. Be sure to choose “Configure Later” if you’re playing along at home.
Java DB
Java DB Server
Java DB Client
Sun Cluster 3.2
Sun Cluster Core
Sun Cluster Manager
Sun Cluster Agents 3.2
Sun Cluster HA for Apache Tomcat
Sun Cluster HA for Apache
Sun Cluster HA for Oracle
Sun Cluster HA for Solaris Containers
Post-installation steps
When the installation finishes, run PCA to get the latest patches for the Cluster software.
# pca -i
Run “catman -w” to get the manpages into the windex databases for whatis(1).
# catman -w &
I’ll be running ipfilter on the public eri interfaces. At the time of my testing this was not supported but it worked. A note to the product QA manager sorted this out though. I needed to edit /etc/iu.ap and add pfil to the eri line, then edit /etc/ipf/pfil.ap and comment out the eri entry there (this was enabled by my JumpStart postinstall scripts):
peon% grep eri /etc/iu.ap
eri -1 0 clhbsndr pfil
peon% grep eri /etc/ipf/pfil.ap
#eri -1 0 pfil
Also, we need to configure IP Filter to allow communication between the cluster nodes; specifically we need to allow SSH and portmapper traffic; that’s ports tcp/22, tcp/111 and udp/111. Note that pfil isn’t configured on the hme interfaces, so if you have rules of the type “via if”, don’t bother to add any entries for hme.
As mentioned above, we want SSH traffic between the nodes and we particularly want to log in as root over SSH. This isn’t as insecure as people might have you believe, as long as it’s configured properly. Without further knowledge of exactly what the Solaris Cluster product is doing, this is as secure as I went.
Edit /etc/ssh/sshd_config, and set PermitRootLogin to without-password. Then, on one node, generate a key for root.
# ssh-keygen -t dsa -b 2048 -C "Cluster Key"
# cp /.ssh/id_dsa.pub /.ssh/authorized_keys
# vi /.ssh/authorized_keys
Add 'from=bootlick.example.ac.uk,peon.example.ac.uk' to the beginning of the line.
Copy /.ssh to the other nodes and now, like a caveman, ssh as root from each node to every other node to ensure that they are all aware of each other’s host keys (yes, I could use ssh-keyscan, but I didn’t).
Finally, I needed to do some extra configuration to keep IPMP happy; Solaris Cluster will insist on putting your public interfaces into an IPMP group and it turns out that when you only have one interface in an IPMP group, in.mpathd will always use ICMP probes to monitor the interface rather then relying on the link status, even if you do not configure test addresses. Now this particular test network’s default router was too crappy or too busy to provide a decent response time to the ICMP probes and so my eri interfaces were being marked down all the time. Therefore, I added some static host routes for in.mpathd to choose as additional test addresses. The correct place to do this on Solaris 10 is /etc/inet/static_routes:
# cat >> /etc/inet/static_routes <<-EOF
# List of static routes. These are added by /lib/svc/method/net-init
# and each non-empty, non-comment line is prefixed with "/usr/sbin/route add ".
# For keeping IPMP happy.
-host 172.25.0.177 172.25.0.177
-host 172.25.5.21 172.25.5.21
-host 172.25.5.30 172.25.5.30
-host 172.25.50.186 172.25.50.186
EOF
Note that since I’m now reliant on at least one of those other hosts being up, I’m not quite getting HA here again. No mind, let’s plough on, reboot the system for all of the changes above to take effect (the systems should now reboot in non-cluster mode) and I’ll see you in the next installment to get the cluster up and running.
Posted in Clustering, Solaris, Sun | 2 comments
Posted by Ceri Davies
Wed, 02 May 2007 20:22:00 GMT
Part 1 of a series on setting up Solaris Cluster for no money
Contents
As I’ve written before, I was hoping to deploy Solaris Cluster at work.
I never did manage to find out what the recurring support costs for it were, but it turns out that there is another flaw; the documentation explicitly disallows running different major versions of Solaris within the same cluster, something that Veritas’ Cluster Services explicitly does allow. That’s fine for some projects, but not having an upgrade path for the project under consideration is unacceptable, so we stuck with VCS; I’ll write about this project at a later date, as it’s essentially a massive consolidation project on Niagara boxes which is quite fun.
However, I’m not down on Solaris Cluster. I’m a little annoyed (and I’d be fucking appalled if I were a shareholder) that Sun took a long time to fail to find me recurrent costs, a little more annoyed that, when I pointed out that this wasn’t even the reason we were choosing a different product and would they like to sell me 200 Sun Ray clients instead, I didn’t get so much as a response, and just plain disappointed that there are no X4500s available in the UK for Try and Buy at the moment.
With that off my chest, I’ll proceed in the next post to discuss how to use Solaris Cluster and Solaris Express to set up a high-ish availability cluster for no money, probably with GlassFish in there somewhere so that I can win a huge TV.
Posted in Clustering, Solaris, Sun, Veritas | 1 comment
Posted by Ceri Davies
Sat, 03 Mar 2007 15:52:00 GMT
Where I work, I look after three highly available clusters running Veritas Cluster Services on Solaris. The hardware is old enough that maintenance is becoming prohibitively expensive and we’re therefore planning to buy new hardware over the next six months or so. Veritas tried to hold us over a barrel over support costs not so long ago, and so this seemed to be a good time to investigate moving to a different HA system.
The obvious choice for me was Solaris Cluster 3.2 (or Sun Cluster 3.2 as it was called at release). It had originally seemed that what I wanted to do would be suboptimal, although the release of 3.2 completely fixed all of the issues that existed with the setup that I had wanted to implement.
Mmm, free
Additionally good is that Solaris Cluster is free to run, even in production, although it must be relicensed (at a reasonably rate) if you wish to buy support. It also supports a large variety of server and storage hardware. Therefore it was no hassle to just download the software and crack on with testing; one barrier to adoption nipped in the bud.
What, no host-based packet filter?
After testing out the design that I had envisioned, it seemed that everything that I had wanted the software to do was in there; the only fly in the ointment was that the IP Filter packet filter was not supported. It worked for some scenarios, but the lack of official support would have been a problem for us.
At around this time, the QA manager for Solaris Cluster, John Blair, happened to post a blog entry introducing himself on the Sun Cluster Oasis[1]. So I asked him about the IP Filter situation.
Ask, and ye shall receive
Less than six weeks later, IP Filter is officially supported for failover services. That’s an amazing response time. I’m not even a paying customer.
Professional services
Even before discovering this little nugget, I proceeded to obtain quotes for the licensing and support costs for Solaris Cluster 3.2. As I mentioned above, they’re quite reasonable.
However, I was told at this point that there was a requirement for Sun Professional Services to come in perform the installation and configuration of the cluster before support could be obtained, and this was far from reasonably priced. At this point I was pretty angry and a little disappointed; I’m a big fan of Sun and couldn’t see why they would throw away customers like this.
I went far enough to complain about it publicly, although it was later pointed out that there was an option for a simple installation validation which is much more reasonable and by pointing this out I hereby absolve myself from the FUD-spreading.
You coming or what?
At this point it’s still not clear that we’ll end up running Solaris Cluster on these platforms, but I’m hopeful that we will. The design that I want to implement slots right in to the Solaris Cluster design and the implementation is therefore very simple and easy to understand (and, by extension, it’s easier to document, which is more the point for me!).
The title of this post is a small admission that I may be starting to sound like this around the office, sorry guys :)
[1] Note to Sun Marketing; there’s some rebranding to be done here :)
Posted in Clustering, Software, Solaris, Sun, Veritas | Tags cluster, ha, solaris, sun | 2 comments
Posted by Ceri Davies
Sat, 24 Feb 2007 13:27:00 GMT
I wrote an article for Sun’s BigAdmin a few months ago.
Due to a backlog of articles, it was only published a couple of weeks ago, but Sun gave me double the usual number of free stuff points in compensation for the delay.
What’s extra nice is that my article has been up on the front page of Sun Developer Network for a couple of days now. Not because it’s particularly brilliant, I think it’s just “put a BigAdmin article on SDN” week.
Still cool though :)
Posted in Software, Solaris, Sun | Tags sun | no comments
