Outage on Sunday (502 Bad Gateway)

One of the two aprs.fi web servers got stuck on Sunday afternoon (around 1300 UTC 2010-09-26) and started giving out 502 Bad Gateway responses. About half of the web requests for aprs.fi ended up on the stuck server, so sometimes you got the correct response page and sometimes you didn't. For some reason the automatic failover did not work – in exactly this sort of situation it should magically disable the second server and point all requests to the working one.

I happened to be driving back from the countryside when the problem appeared. After getting home in the evening and analyzing the situation I disconnected the server from the network remotely by shutting down the switch port, so that it would not interfere and that the other web server could handle all of the site visitors. At this point, when the server disappeared completely from the network, the other server automatically took over it's IP address and the service started working perfectly.

The next day, 15 hours after the disconnection, I went to visit the hosting site to bring the server back online, and took a couple of iphone video clips of the servers. First, the primary server (at the moment running the whole aprs.fi site and serving all visitors), and then the secondary server which had just been brought up and was receiving the past 15 hours worth of APRS data from the primary server. It was pretty busy at the time, but it took only an hour to copy over all the missing data from the primary.

After the replay was completed I started up the web service on the second server, too, and the service is now again running in a redundant configuration.

Root cause analysis continues. It was definitely a software glitch, probably something to do with me running a system process under gdb to analyze a SIGSEGV it gets every now and then. It might well have caused some services to freeze. The database didn't hang and continued to replicate until the disconnection, but the web service got stuck and I couldn't log in to the box using ssh, or through the serial console.

I'll also have to fix the health check used to trigger the automatic failover procedure, so that it will work next time this sort of problem appears.

The trip to the computer room gave me an excuse to quickly try out Apple's iMovie for composing a little video. Appears to work, but I guess I still prefer Vegas. There isn't much point in this silly video, but maybe it's interesting for some aprs.fi fans. :)

Ham::APRS::DeviceID published

The Ham::APRS::DeviceID module, written by myself and used by aprs.fi, has been uploaded to the CPAN. The perl module makes a fierce attempt to detect the make and model of the APRS device which has transmitted an APRS packet.

The detection is based on the APRS destination callsigns listed at http://aprs.org/aprs11/tocalls.txt and mic-e type codes documented at http://www.aprs.org/aprs12/mic-e-types.txt. If you wish to have your APRS software or tracker device detected by this module (and aprs.fi), please get in touch with Bob Bruninga to have it added in the relevant lists, and after that, please notify me that they've been updated. Thanks.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

IPv6 support in APRS packet parser - FAP version 1.17

Version 1.17 of the Ham::APRS::FAP APRS packet parser has been released and uploaded to the CPAN. It should appear at the mirrors during the next 24 hours or so. The main feature is IPv6 support in APRS packet path parsing.

What?!? IPv6 support in APRS?

aprsd, and sometimes other software too, sends position packets to the APRS-IS using the qAI Q construct. qAI triggers packet path tracing in the APRS-IS. Each APRS-IS server inserts the hexadecimal source IP address of the previous server to the packet. Now that quite a few Tier2 APRS-IS servers have IPv6 connectivity, even some T2 hubs, IPv6 hex addresses can appear, and they are a bit longer than regular callsigns, and were getting rejected by the parser.

SRCCALL>DSTCALL,qAI,IQ3VQ,THIRD,92E5A2B6,T2HUB1,200106F8020204020000000000000002,T2FINLAND

In this case 92E5A2B6 is the IP address of THIRD, and 200106F8020204020000000000000002 (2001:6f8:202:402::2 in normal IPv6 printed representation) is the IP address of T2HUB1 (T2FINLAND happened to be connected to T2HUB1 at the time).

Here's the complete list of changes:

• Remove / or ' ' from beginning of comment after parsing away PHG, altitude and other optional data
• Allow hexadecimal IPv6 addresses in APRS-IS paths after Q construct
• Added tests for uncompressed packet altitude (negative, too)
• Fixed destination callsign based symbol selection for 'BC'
• Updated URLs to aprs.org, etc

aprs.fi is also available using IPv6 at ipv6.aprs.fi.

API URLs changed

The aprs.fi API URLs have been updated. They all now point to http://api.aprs.fi/api/get instead of aprs.fi.

If you have written an application using the API, you need to upgrade it to use the new links. The old URLs pointing to aprs.fi will stop working in December 2010. This change has been made to make it possible to move the API code to another server or web service process set, if necessary.

Thank you, and happy API hacking!

Mobile signup and password recovery fixed

I've fixed the captcha code (recaptcha) to be valid xhtml, so that mobile browsers with proper strict XHTML parsing (such as the iPhone/iPad family) accept it. It's now possible to sign up and do password recovery using those devices.

blog.aprs.fi !

This aprs.fi news blog has, as of today, moved to the address where it should have been born in the first place: http://blog.aprs.fi/ !

Please update your links and bookmarks, if you have any. Thanks.

IE6 no more.

Users of Internet Explorer 6 get to see, since yesterday evening, a slightly annoying popup telling them to upgrade to one of the more modern browsers. The popup is shown on the live map page, and can be hidden by clicking a little 'x' button in the corner.

The concept, graphics and layout come from the IE6 no more site. It also addresses corporate users who cannot upgrade their own browsers. Believe me, I know your pain, I also have a corporate IT supported laptop at work. Luckily they offer a recent Firefox, too. Supporting IE6 is simply too much of a pain in the arse, it generates way too much extra work for me. I'd rather use that time for developing something new.

Besides, Google has dropped IE6 from the list of supported browsers, too, for Google Maps API v3 and some other products. It will stop working sooner or later, so at least the users should be warned.

Sorry!

IE6 and IE7 broken for a day

Sorry, I broke the real-time map yesterday for the users of Internet Explorer 6 and 7. It did work using IE8 and all other browsers. Fixed this morning.

My excuse is that my home vmware installation, which I use for running IE6 and IE7, was somehow broken, so I didn't test those browsers.

Thanks to Phillip ZL2TZE for bringing this to my attention.