Magic UTF-8 support, weighted callsigns and other updates

The aprs.fi service now supports UTF-8 in most APRS packet types, such as messages, comments and status messages. Using UTF-8 enables proper universal messaging for APRS users. The UTF-8 encoding of Unicode is backwards compatible with ASCII, and it can be transmitted without problems on the APRS-IS and the APRS radio frequencies. It does not require the end-user to manually switch between "compatible ASCII" and "unicode" mode depending on the language or destination, since English text (like this) is transmitted in exactly the same way in both ASCII and UTF-8. It also supports all of the special characters in other languages (åäöæø ÅÄÖÆØ ßüÜ). UTF-8 works on findu.com, too.

I strongly recommend all APRS authors to implement UTF-8 encoding in their APRS software for proper internationalization support! To get some UTF-8 testing packets, send an APRS message to the callsign UTF-8, and my testing responder will respond with four messages containing English text, Scandinavian and German special characters, and a few Japanese words.

In addition to UTF-8, I also added a magic recode feature (inspired by the irssi IRC client), which tries to guess the character set of the original packet based on the character distribution. It currently attempts to support ISO-8859-7, ISO-8859-15, CP437 and CP850. Because of the overlaps between these character sets, this feature can not be made to work perfectly - the task is impossible. It's guesswork. To get your characters to show correctly, please transmit them in UTF-8.


Inspired by weighted tag clouds, callsigns shown in the Other SSIDs list are now weighted according to the time of the last position update from that callsign. If the position was updated within 24 hours, the callsign becomes bigger than usual, and the other way around. For example: KJ4ERJ, WB4APR

The position history database table was migrated to a partitioned table. Or rather, the migration was started - new data is being inserted in a partitioned table, and old data is still in the unpartitioned one. This is a very technical change which does not make a lot of difference to most of you folks, but improves things a lot on the server side. I can now delete old history data in 1-day chunks very quickly with a minimal performance hit. Inserting new data requires less disk I/O and smaller memory buffers (thanks to the partitioned indexes).

I've also upgraded aprs.fi to use Ham::APRS::FAP version 1.13, upgraded the web server software, fixed some XHTML syntax, done some performance tweaks on the real-time map, and added support for APRS targets without a current known position (required for receiving telemetry from a station without a known position).

This upgrade was a big one, and required about an hour of downtime. Sorry for the trouble. I hope I didn't break anything this time.

Maintenance break on Saturday 5th of December

On Saturday, roughly between 18 and 19 UTC, I upgraded operating system components and installed security patches. These upgrades required taking the service down for some time, and even data collection was affected for a short while. Sorry for the trouble!

Before the reboot, the master server had an uptime of 681 days. Thanks go to the UPS and the diesel generator - there have been a few small power outages in Espoo during that time.

In the near future I'll be installing a new master server for aprs.fi, the hardware is already waiting for the OS installation in the basement.

I've also bought an 80 GB Intel X25-M SATA solid-state drive (SSD), a very quick flash-based hard disk-like device. The amount of disk I/O operations is starting to become an issue on aprs.fi, so I'm going to try moving the busiest database tables (which are also the smaller ones) on SSD. We'll see how that improves things! It might even allow me to implement some new features (which require additional I/O capacity).

Silja Europa steering & comments

Silja Europa (live position), a rather large passenger ferry (photos), has some steering issues, and is currently doing circles near Åland. News in Finnish: hs.fi, iltasanomat.fi. There are 1659 people onboard, but there is no immediate danger, since the ship can still steer using the propellers, although in a rather coarse manner.

Another small bit of news is that I've disabled anonymous commenting on this blog due to comment spam - I'm getting tired of manually deleting all the spam. From now on, you'll need a Google account to comment, or an account on one of the supported OpenID-enabled services. Sorry for the trouble.

During the last couple weeks I've been working on making the service UTF-8 clean for proper universal text messaging. It almost works in my testing installation, but there are still some quirks I need to clean up.

I've also installed a new database replica server at another physical site in downtown Helsinki, so that I'm able to bring the service alive without significant data loss in case of a complete hosting site loss (fire, flooding, extended network outage). The main servers are located in Espoo, Finland.

Ham::APRS::FAP 1.13 released

Version 1.13 of Ham::APRS::FAP was uploaded to CPAN today. The tests seem to be passing nicely, and the code has been running on aprs.fi for a long while, so the upgrade should be safe.

Ham::APRS::FAP is the APRS packet parser used by aprs.fi. If you wish to decode APRS packets in Perl, this is what you need. It's pretty fast, it's stable, and it can report it's errors in a developer-friendly way. And it's free (as in free speech).

Version 1.13 contains the following small fixes:

• Allow anyone to update telemetry parameters, skip the source callsign check
• Allow a PHG of 0000 for deleting PHG
• Added new error code sym_inv_table for invalid symbol table char
• Added local time zone parsing to object timestamps
• Fixed comments parsing for last resort !-location packets
• Parse APRS message rejects (negative acknowledgments)

Some of the improvements were implemented by Tapio, OH2KKU, while others were done by myself.

Embedded maps tracking fixed

The embedded map broke with the upgrade on Saturday - it stopped tracking the configured station (he_track parameter). That's fixed now. Sorry about that, I need to focus a bit more on testing before releasing...

Statistics after the upgrade

On 31 Oct I switched from Apache to nginx. Here are some graphs from one frontend. Which would you prefer, based on these graphs?




I only added the temperature plugin yesterday (before the upgrade), but ... the CPU core temperatures did drop by 2°C. Now, does nginx reduce global warming?

Multiple targets on live map (and other small stuff)

On Saturday I did the planned upgrade. The big new user-visible feature is that you can now track multiple targets on the map. Just type in multiple callsigns separated with a comma: Rautauoma, OH2RDK, or click on 'start tracking' more than once. This code has been running on the beta site for a week now.

There are also some smaller improvements, like symbols being shown in the "other SSIDs" list, callsign links in APRS paths shown in the info balloons, fixed web search, and a JavaScript memory leak workaround for IE.

Oh yeah, referenced digipeaters and igates are now shown when you look up a day in the history. There is little bug in there, though - it shows their current positions, instead of their positions on the selected day. But hey, they rarely move.

There were also some rather large changes under the hood, causing a longer outage than usually. I replaced the Apache web server software with nginx, which allows me to run with HTTP keep-alive again (without running out of memory), and should generally handle larger amounts of users without that much trouble. It also helps with resource management - I'm now running separate web server processes for each service instead of a single Apache process pool for a large number of virtual servers. Now, if one service gets a beating, the others won't be affected so easily. Upgrades and configuration errors in one service won't break others, either. The downside is that name-based vhosts are out of the question.

Today's photo was taken on 11th of August at the summer cottage in Maaninka. EXIF says 9:39:17 PM. It's freezing and very dark in Helsinki today, so I'd rather go back a few months.

PS. I'll be in Bordeaux, France for the next 2 weeks or so.