Outage: Building redundant setup

I'm taking the database down today for some time. I need to take a new consistent copy of it to enable transaction logging and replication to the new web server and a backup server. Sorry for the trouble.

The new server should be delivered today, by the way.

Running SBS-1 BaseStation on Linux

This is slightly off-topic, but you can probably guess why I'm interested in this stuff. There's been discussion on the forums whether you can run BaseStation on Linux, but I couldn't find the answer on the web, so I'm posting it here after finding out myself.

I bought myself an SBS-1 (by Kinetic Avionic Products) for a Christmas present. I've been running the BaseStation software on Windows so far, but that's not fun if you want to run it remotely.

So, tonight I tried to run it in Linux on top of Wine. It worked. Here's how:

1. Install a recent version of wine, and practically any versions of the old but incredibly useful tools nc and cu. On my Debian system this can be done using the command apt-get update && apt-get install wine nc cu. I'm runnig debian testing (lenny, in January 2008). This probably applies to Ubuntu, too.

2. Run some other Windows software with wine first, get it configured on your account and see that it actually runs something successfully.

3. Download the BaseStation CD which contains the driver DLLs. The driver DLLs are not really used, but they have to be present on the system so that BaseStation.exe will run.

4. Extract the DLLs from the CD in a temporary directory, and copy them to the system32 directory of your wine installation:

mkdir tmp && cd tmp
unzip ../BaseStation-1-1-1-119-CD.ZIP '*.dll'
mv *.dll ~/.wine/drive_c/windows/system32/
cd .. && rm -rf tmp

5. Download the latest version of BaseStation (I tried 1.1.1.125).

When wine is properly configured on your system, Firefox will run the wine-safe installer for the EXE file automatically, and basestation will be installed. If this doesn't happen, you might have to run the installer by hand (wine-safe setup.exe). After the installation, the installer will quit. Don't run BaseStation yet.

6. The application apparently cannot talk to the USB bus under wine, so we'll have to do a trick here. Connect the SBS-1 to the USB port of your Linux system, and observe the output of the dmesg command - it should notify you that an USB serial port has been found. On my system it looks like this:

usb 2-2: new full speed USB device using uhci_hcd and address 5
usb 2-2: configuration #1 chosen from 1 choice
ftdi_sio 2-2:1.0: FTDI USB Serial Device converter detected
drivers/usb/serial/ftdi_sio.c: Detected FT232RL
usb 2-2: FTDI USB Serial Device converter now attached to ttyUSB0

Now, the SBS-1 hardware can be reached behind /dev/ttyUSB0 at 921600 bits per second. We could connect to it using the very simple Unix "call up" program cu --nostop -s 921600 -l /dev/ttyUSB0, but we cannot talk the proprietary binary protocol used by the SBS-1 and the BaseStation software. Yet.

7. So, we run cu under another very simple and extremely useful Unix tool, nc, which is the swiss army knife of TCP/IP. We tell nc tolisten on TCP port 19251 using the localhost IP address of 127.0.0.1 (so that it cannot be connected by anyone from the Internet), and to start up the cu program when a connection arrives:

nc -l -s 127.0.0.1 -p 19251 -c "cu --nostop -s 921600 -l
/dev/ttyUSB0"

8. Now, while we have nc waiting for the connection, we can start up BaseStation in another window:

wine .wine/drive_c/Program\ Files/Kinetic/BaseStation/BaseStation.exe

Tell it to connect over the network, the address is 127.0.0.1, and the port is 19251.

There's no reason why you couldn't do this over the Internet, either. You might want to run cu under inetd instead of nc, though, since nc will quit when the connection closes. Remember to set up your firewall / tcp wrappers properly!

The 2-hour outage last night

After adding 4G of memory on the server (which went well, as usual) I had to do some disk arrangements, which took some time to complete. After booting up the Apache web server didn't return web pages correctly any more - web browsers would show the HTML source code instead of nicely formatted pages. After playing with a network analyzer (thanks Wireshark) and getting a little help from an ex-colleague I finally got it fixed.

I had upgraded a good bunch of libraries on the system earlier, but hadn't really restarted the web server processes afterwards (I'm only running a graceful restart after log rotation and config changes), so the web server was still using the old libraries. Now, after the reboot, new libraries were linked in, and it broke. There was an extra line break (\r\n) after the Server: Apache... HTTP header, and web browsers promptly stopped parsing the headers, and assumed a text/plain content type, and showed the HTML.

The strange thing was that enabling content-encoding compression (SetOutputFilter DEFLATE) fixed the problem. While it did work, the header was:

Server: Apache/2.2.6 (Unix) mod_ssl/2.2.6 OpenSSL/0.9.8g

While it didn't work, the header was:

Server: Apache/2.2.6 (Unix) mod_ssl/2.2.6
(+ the extra CRLF, then Connection: close and the following headers)

The workaround was to set ServerTokens Prod, so that the header only says Apache, and doesn't give any hints of version numbers or modules. My best guess is that disabling DEFLATE changed library linking order so that the openssl version query returned "\r\n". gnutls maybe?

The sysadmin lesson to learn here is probably that you should restart services depending on libraries which you have upgraded, so that incompatibility is revealed right away. It's nice to know which upgrade broke what. It's not fun to see applications break after "just a reboot".

On the positive side, the system isn't swapping any more.

Maintenance outage: adding memory

The aprs.fi server is going to be down for some time tonight or tomorrow, while I'm adding 4G of memory in it. Sorry for the inconvenience.

Lost: Find 815: Christiane I on APRS

ABC studios, an Australian company called Hoodlum and the creators of the TV series Lost have created an alternative reality game called Find 815, which produces content for fans of Lost during January. The first episode of Season 4 will be aired on January 31st. Lost is pretty popular, and so is the Find 815 game. The game is described in more detail on Lostpedia, a wiki site dedicated to Lost.

The story briefly mentions amateur radio, and involves a ship called Christiane I. The fun part is that Christiane I has an APRS tracker onboard and is now sailing somewhere near the Christmas Island! There has been some speculation whether ABC has actually hired a real ship, which could be tracked using APRS, and a few popular blogs and forums have linked to findu and aprs.fi.

I cannot comment on whether the ship tracker is authentic, although the TCPXX path does mean something to APRS experts.

Rotating APRS symbols

I've just installed the rotating APRS symbol code on the production server. I hope it won't slow your browsers down too much. It will cause some slowness, and eat a bit more memory, but it might be worth it.

The symbols have been rotated in 10° steps (0° to 350°) using Gimp's scripting language (Scheme / Script-Fu) and a little Perl. This was my first piece of Scheme, and my favorite quote from the Script-Fu tutorials was "Debugging Script-Fu scripts is only slightly less painful than stabbing yourself in the eye with a dull spoon". It's so true. I would have used imagemagick, but it broke the transparency. So I learned a new programming language. :)

Turkish APRS

TA2T spent 5½ hours last night translating aprs.fi in Turkish! Thank you.

Over the weekend I tried to make the map icons rotate and point to the course of a moving station. Got it basically working, but it still looks too ugly to be published. Need to do some graphics conversion work first. It might have a performance impact, I hope it won't be too bad.