Linux-Blog 3.0

LiveJournal user suppressingfire pointed out that 11.0.0.1 isn't a private network address. My first reaction was that it didn't matter, since it was just for cluster interconnect. Still, best to keep things by the book, so we'll change it to keep it in side the 10 network.

# ifdown eth1

Then edit the /etc/network/interfaces file so the eth1 section now says:

# The cluster interconnect interface
auto eth1
iface eth1 inet static
address 10.1.0.1
netmask 255.255.255.0
broadcast 10.1.0.255

then bring the interface back up:

# ifup eth1

In the TruCluster world, we used to have a configuration option to totally block anything coming from the memory channel cards (the cluster interconnect) from getting out onto the outside network. I don't know for sure whether OpenSSI has such an option, so better safe than sorry.

Posted by YatPundit at 4:26 PM | Permalink

The OpenSSI group has done the bulk of their development on RedHat Linux. OpenSSI currently supports Fedora Core 2 and 3, as well as a deprecated version for RH9. Additionally, OpenSSI is distributed for Knoppix 3.6 and Debian Sarge. Others in the community are working with SuSe 9.x, but there's no official distro out for SuSe as of now.

I wanted to stay away from the source and use a packaged version of OpenSSI, so that put the choice down to FC or Debian. I've never been a big fan of RedHat--I've used SuSe for years now, and am comfortable with it. That meant going with Debian.

I downloaded the ISOs for Sarge from debian.org, burned the CDs, put in the first one, and fired up snowmane. Instead of a Linux boot/install screen, I saw Win2K come up (the seller left winders on to show that the system boots). Ugh, problem with the CD. Jumping back to hasufel, my XP workstation, I made the three-floppy install set needed to fire up Sarge. I used rawrite for windows (rwwrtwin.exe) to create the floppies and went back to snowmane. The boot floppy started fine, led to the kernel floppy, then the driver floppy, which mounted the CD. Go figure, I'm not going to puzzle out the boot issue on a low-rent machine, I'm just gonna boot it.

In terms of file system configuration, the box had W2K on it, so I let Sarge blow away the internal drive. The attached firewire drive was from an older box runing SuSe8, and was formatted as a reiserfs, so I let Sarg blow that one away as well.

I did a straightforward Debian install, having it install apache, BIND, NFS, samba, and local mail. No desktop stuff, no surprises for now, since the cluster is going to deprecate some things and modify others. Once the install was complete, the system re-booted, and I was logged in as root at the console, I manually did an apt-get to install sshd, and re-configured the network from the default DHCP setup the installation routine used to a static IP config for the two ethernet interfaces.

(rant)Side digresson here: One of the more frustrating things about shifting gears between flavors of UNIX has always been the use of each system's high-level utilities. If you're ever interviewing a UNIX/Linux geek for a job, don't worry that they've only used RedHat most of their career and you're a SuSe shop, or you've got the chance to hire a FreeBSD geek for your HP-UX shop. Just give the new guy a couple of weeks and a test box to play with your environment's high-level config stuff. Let him transition his thinking from YAST to sysinstall. Give him a chance to play with ifdown and ifup safely. Factor it into the new employee's start-up time. In the training biz, our best customers were the employers who know that it's OK to hire a Solaris geek for a Tru64 gig if you send them for a week to a System Administration class.(/rant)

OK, this is Debian, so that means ifdown/ifup, and the config file they use.

Here's what the /etc/network/interfaces file looks like:

snowmane:/mnt/fw# cat /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
address 10.0.0.2
netmask 255.255.255.0
gateway 10.0.0.1
broadcast 10.0.0.255
dns-search muffeletta.com
dns-nameservers 68.1.16.100 68.1.16.101
# The cluster interconnect interface
auto eth1
iface eth1 inet static
address 11.0.0.1
netmask 255.255.255.0
broadcast 11.0.0.255

Once the file was edited, I did:

# ifdown eth0
# ifup eth0
# ifup eth1

Since eth1 didn't exist in the config until I edited the /etc/network/interfaces file, there was no need to bring it down.

Time to switch back to hasufel and do the rest via ssh, since sitting at my desk is easier than standing in front of the rack. OK, putty connects with 10.0.0.2 on the office LAN no problem. Let's verify eth1 while we're at it:

snowmane:/home/edward# ping 11.0.0.1
PING 11.0.0.1 (11.0.0.1) 56(84) bytes of data.
64 bytes from 11.0.0.1: icmp_seq=1 ttl=64 time=0.101 ms
64 bytes from 11.0.0.1: icmp_seq=2 ttl=64 time=0.021 ms
64 bytes from 11.0.0.1: icmp_seq=3 ttl=64 time=0.020 ms
64 bytes from 11.0.0.1: icmp_seq=4 ttl=64 time=0.018 ms

--- 11.0.0.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2997ms
rtt min/avg/max/mdev = 0.018/0.040/0.101/0.035 ms
snowmane:/home/edward#

OK, we'll come back to that interface later.

Next step was to confirm that the firewire drive was attached and mount it:

# mkdir /mnt/fw
# mount /dev/sda1 /mnt/fw
mount: you must specify the filesystem type

Huh? SCSI bus weirdness, no doubt. We always tell TruCluster students that, if you can get the shared SCSI bus to work, whether it was copper or fibre, you've done 80% of the class. I'm taking the same attitude here.

I remembered that Kurt Garloff from SuSe has a neat script to re-scan the scsi bus on his personal website. It should work OK on Sarge, so here goes. I put the script on snowmane, in /usr/local/bin, made it executable, and ran it:

# chmod 755 /usr/local/bin/rescan-scsi-bus.sh
# cd /usr/local/bin
snowmane:/usr/local/bin# ./rescan-scsi-bus.sh
Host adapter 1 (ide-scsi) found.
Host adapter 0 (sbp2_0) found.
Scanning hosts 1 0 channels 0 for
SCSI target IDs 0 1 2 3 4 5 6 7 , LUNs 0
Scanning for device 1 0 0 0 ...
OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: E-IDE Model: CD-ROM 52X/AKH Rev: A63
Type: CD-ROM ANSI SCSI revision: 02
Scanning for device 0 0 0 0 ...
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: QUANTUM Model: FIREBALLP AS20.5 Rev: 0.01
Type: Direct-Access ANSI SCSI revision: 02
0 new device(s) found.
0 device(s) removed.

snowmane:/usr/local/bin#

Lo and behold, the 20GB drive is now visible on the bus. Let's try the mount again:

snowmane:/usr/local/bin# mount /dev/sda1 /mnt/fw
snowmane:/usr/local/bin# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 9471868 740936 8249788 9% /
tmpfs 63724 0 63724 0% /dev/shm
/dev/sda1 19378244 131212 18262652 1% /mnt/fw
snowmane:/usr/local/bin#

Voila, we have an external drive with an ext3 filesystem, since that's what we told the install program to put there.

Allright, we have a box with a base OS installed and set up, now we can proceed to the cluster install.

Next - OpenSSI initial install.

Posted by YatPundit at 9:57 AM | Permalink

I picked up a 800mhz Pentium III server in a 1U case on eBay for $175+shipping to kick this project off. It's got 256MB of RAM and a 10GB hard drive. I already had an external firewire drive enclosure, which currently has a 20GB IDE drive in it. Keeping with my tradition of server naming, this box was christened "snowmane."

Here's what snowmane looks like in the rack:

The box has two USB ports on the motherboard, but no firewire. I picked up a basic PCI firewire card from the local CompUSA. The system's single PCI slot had a Freevision capture card in it, so that came out, in goes the firewire card.

There are two Intel-chipset ethernet ports built into the box. I'll use eth0 for the regular connection, and eth1 will be for cluster interconnect. Video is basic XGA, and snowmane is hooked up to my kvm in the rack.

next - OS install

Posted by YatPundit at 9:52 AM | Permalink

One of the problems with working with AlphaServers for so long is that one ends up with champagne tastes. The catch is that I operate my office network in a beer budget. Relatively speaking, it's more like a Milwaukee's Best budget. Linux is good about working on servers that fit my budget, particularly since I don't plan on using X.

Running standalone Linux systems on low-rent equipment is one thing. Making that low-rent equipment highly available is a totally different issue. For openers, older, inexpensive systems often need upgrades to get them to a point where they can be clustered. Then there's the issue of shared storage. Storage Area Networks (SANs) aren't supposed to be low-rent. SAN users want good quality equipment and are willing to pay for it. I want something that will work in my rack here in my home office, and I'm not willing to pay for it.

This isn't an original thought on my part. Others have looked at what can be done to make systems highly available without a true SAN. The obvious solution for low-rent is to forget doing a shared bus configuration in the first place, opting for a Distributed Replicated Block Device (DRBD) configuration. But that's where my champagne tastes come in. I'm used to TruClusters, and a TruCluster requires a storage system on a shared SCSI or fibre-channel. Both of those are way out of my beer budget. My two tried-and-true BA350 storage shelves got trashed in the flood, so starting a shared SCSI configuration from scratch was going to be expensive. In the grand scheme of things, iSCSI isn't expensive, particularly when compared to a fibre SAN, but it's still more than I wanted to pay.

I put the dilemma to the folks in the Debian community on Live Journal, and got back an interesting suggestion: use firewire! Sure, why not, I've got an external firewire enclosure (I use it with my Acer TravelMate C104 tablet PC), so it was worth a try.

The choice of which cluster solution to go with for this project was a no-brainer. I want a low-rent TruCluster, so that means an OpenSSI system. OpenSSI's Cluster File System (CFS) is a kindred spirit to TruCluster's CFS; it's no surprise that the leaders in the OpenSSI community are HP employees. There are downsides to OpenSSI, most notably that it's not as mature a product as the Linux-HA project. As a result, my choices for Linux distro are much more limited. That may present more challenges, but we'll see.

Posted by YatPundit at 10:17 AM | Permalink