New version of aprs.fi's APRS packet parser released: Ham::APRS::FAP v1.21

Yesterday I uploaded a new version of the APRS packet parser used by aprs.fi, Ham::APRS::FAP, to the CPAN. It's open source, so if you're a little bit weird like me and enjoy programming in Perl, this might be useful. Here's a full list of changes:

• Improve make_position() to support HMS UTC timestamp. make_position() now returns the packet type character so that it can signal the presence of a timestamp.
• Improve make_position() to support comment string, !DAO! extension, altitude encoding. Fix rounding errors in lat/lon/speed. Support generating packets with no speed or course. Take optional parameters in a hash parameter. Implement unit tests for make_position().
• Set up Travis automatic running of the (existing) unit test runs.
• Additional character escaping in regular expressions to deal with deprecated functionality in Perl 5.22.

The packet parser is one of the more complete APRS decoders, although I'm not sure if any of the parsers deal with all of the APRS features and packet variants yet.

Device identification database updated, available for other apps

Most APRS devices and applications transmit an unique AX.25 destination callsign in all their packets, so that receiving stations can figure out which application or device is transmitting each packet. Bob Bruninga maintains a tocalls.txt index file, which lists all the assigned destination callsigns.

Those devices which use the Mic-E encoding to transmit position packets encode the latitude and a little bit of the longitude within the destination callsign, in which case something else has to be done for device identification. Mic-E device IDs are encoded around the comment text, with one character in the beginning of the comment text and zero to two characters in the end of the comment. The Mic-E type codes are indexed in mic-e-types.txt.

Now, when an application such as aprs.fi wishes to automatically decode the destination callsigns and type codes to readable application names, as seen on the aprs.fi station information page, the author of that application needs to collect all the device identifiers from those two files, and somehow convert them to application source code, or a configuration file that can be read by the application. The master files are written with human interpretation in mind, and it's rather hard to make an application automatically parse out the identifiers from them. It needs to be done manually, and whenever new devices or applications are published, all the applications wishing to detect the new ones need to get an update. All softwares authors need to get notified that there are some new devices, and then somehow add the devices in their respective configs. That's quite a lot of extra work that would be better spent writing some fancy new features instead.

OH7LZB-7, correctly identified as a Kenwood TH-D72,
at Mikkeli International yesterday afternoon.
Aircraft and training provided by MIK at Helsinki-Malmi.

For aprs.fi, I initially made a Perl module, Ham::APRS::DeviceID, and published it as open source, so that other software authors could use the index too, and skip the manual labour-intensive part of parsing the master text files. For some obscure reasons a number of programmers have chosen to use other programming languages than Perl, and the module was of limited usefulness for them.

To improve the situation, I converted the device index in YAML (Yet Another Markup Language), which is easy to read and edit by humans, and also easy to read and edit by computer programs. I also wrote a little converter program which parses the tocalls.yaml file and outputs the same data in JSON and XML formats, which are popular file formats for passing data between computer systems.

I then updated the original Perl module to read the YAML file, and removed the old database that was embedded in the code. DeviceID.pm 2.00 and later use tocalls.yaml. I also updated aprs.fi to use the new version of the Perl module, and tocalls.yaml. The update brought a number of new devices to aprs.fi, including the newer Yaesu radios.

Other programmers who wish to do device identification are welcome to download the YAML, JSON or XML files and use those. It should be straightforward to automatically update the device index during an application build, or even automatically update the index directly in the application.

New devices should still be first added to Bob's files, and only then to tocalls.yaml. If you have a new device which has already been added to Bob's index, please make a pull request or an issue ticket to update tocalls.yaml accordingly.

Parser updates and new delayed packet filtering feature

I've updated the APRS device identification module Ham::APRS::DeviceID to include detections for new APRS devices (KissOZ, anyfrog, unknown mic-e, SARTrack, Altus Metrum, SM2APRS, aprsc, NW Digital Radio UDR56K). The new version, 1.06, has been published on CPAN for your open source pleasure, and installed on aprs.fi.

I also published version 1.19 of the Ham::APRS::FAP packet parser used by aprs.fi. It only includes a small fix to the binary value telemetry bit order in Base91 comment telemetry. But at least that keeps me in the regular 1 release per year schedule! Just in time!

The larger feature in the aprs.fi upgrade is that it now utilizes the sequence number present in Base91 comment telemetry for detecting delayed packets.

APRS packets notably lack any sort of sequence number that could be used to detect old duplicate packets arriving late, or to place them in the correct order. Newer Byonics trackers can transmit telemetry (battery voltage, temperature, etc) within position packets using the new Base91 comment telemetry format, which is quite tightly packed, and also includes a sequence number in a range of 0...8280. This makes it really easy to detect a packet having an older sequence number than the previously received packet. There are already a lot of those devices in use.

aprs.fi has now been updated to make use of the sequence number when available. It's more reliable and faster than the old methods of detecting too high speed or duplicate packet content. Due to the amount of broken igates and digipeaters delaying packets for minutes or tens of minutes (often due to a buggy Kantronics KPC3+ in KISS mode) it can make sense for a tracker to transmit the sequence number alone without any actual telemetry! Packets dropped due to this will be shown with this error message:

Delayed or out-of-order packet (sequence number)

In practice it will usually be shown together with the error message about dropped telemetry, since the telemetry content is also ignored due to the duplicate sequence number: 

2012-12-31 00:24:51 EET: N0CALL-9>SY0UWY,WIDE1-1,WIDE2*,qAR,N00CALL:
`pOnqgd>/'"KQ}MT-RTG|!D&='a|!w;a!|3
[Duplicate telemetry sequence, Delayed or out-of-order packet (sequence number)]

You can use the Decoded mode of the raw packets view to see sequence numbers and telemetry values decoded from each packet.

Ham::APRS::FAP version 1.18

A new version of our APRS packet parser has recently been released on CPAN. It contains new features and bugfixes made by both myself and Tapio, OH2KKU. Here's a list of changes:

• Added Ham::APRS::IS module - an APRS-IS client library
• Fixed position resolution calculation for whole degree cases, and added tests
• Implemented a 'raw_timestamp' option to return undecoded timestamps in all but status packets
• Added tests for timestamp and status message parsing
• Added support for new base91 comment telemetry

Ham::APRS::DeviceID published

The Ham::APRS::DeviceID module, written by myself and used by aprs.fi, has been uploaded to the CPAN. The perl module makes a fierce attempt to detect the make and model of the APRS device which has transmitted an APRS packet.

The detection is based on the APRS destination callsigns listed at http://aprs.org/aprs11/tocalls.txt and mic-e type codes documented at http://www.aprs.org/aprs12/mic-e-types.txt. If you wish to have your APRS software or tracker device detected by this module (and aprs.fi), please get in touch with Bob Bruninga to have it added in the relevant lists, and after that, please notify me that they've been updated. Thanks.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

IPv6 support in APRS packet parser - FAP version 1.17

Version 1.17 of the Ham::APRS::FAP APRS packet parser has been released and uploaded to the CPAN. It should appear at the mirrors during the next 24 hours or so. The main feature is IPv6 support in APRS packet path parsing.

What?!? IPv6 support in APRS?

aprsd, and sometimes other software too, sends position packets to the APRS-IS using the qAI Q construct. qAI triggers packet path tracing in the APRS-IS. Each APRS-IS server inserts the hexadecimal source IP address of the previous server to the packet. Now that quite a few Tier2 APRS-IS servers have IPv6 connectivity, even some T2 hubs, IPv6 hex addresses can appear, and they are a bit longer than regular callsigns, and were getting rejected by the parser.

SRCCALL>DSTCALL,qAI,IQ3VQ,THIRD,92E5A2B6,T2HUB1,200106F8020204020000000000000002,T2FINLAND

In this case 92E5A2B6 is the IP address of THIRD, and 200106F8020204020000000000000002 (2001:6f8:202:402::2 in normal IPv6 printed representation) is the IP address of T2HUB1 (T2FINLAND happened to be connected to T2HUB1 at the time).

Here's the complete list of changes:

• Remove / or ' ' from beginning of comment after parsing away PHG, altitude and other optional data
• Allow hexadecimal IPv6 addresses in APRS-IS paths after Q construct
• Added tests for uncompressed packet altitude (negative, too)
• Fixed destination callsign based symbol selection for 'BC'
• Updated URLs to aprs.org, etc

aprs.fi is also available using IPv6 at ipv6.aprs.fi.

Ham::APRS::FAP 1.16 released

Version 1.16 of the APRS packet parser was uploaded to CPAN last night, and according to the first few CPAN tester reports it seems to be working as expected. Here's a list of changes:

• Added a couple of test cases
• Added an example script to parse a whole file of timestamped APRS-IS packets
• Fixed peet WX packet parsing for Perl 5.8 and older, which don't support n! packing
• Minor speedups by reducing amount of regular expressions, extra variables, reordering things to catch the most common case first, and tightening a loop
• Put '' around hash key strings (style issue)

The main change was the Perl 5.8 wx parsing fix. I couldn't have noticed it without the CPAN testers and the unit tests included with the module, since I don't run on 5.8 myself any more.

Ham::APRS::FAP 1.15 released

Version 1.15 of Ham::APRS::FAP, the APRS packet parser module used by aprs.fi, was uploaded to CPAN yesterday evening. It should appear on mirrors today. This update contains all the changes implemented since November 2009.

Like most of you know by now, the parser is released in Open Source terms (GPL or Artistic license, like most of Perl) so that others can freely use it to implement APRS systems. I believe it is the most complete APRS decoder available as open source.

Here's the change log:

1.14 Tue May 4 20:31:33 EEST 2010 - Hessu, OH7LZB

• Fixed humidity parsing, h0 means 100%
• Fixed peet bros $ULTW packet parsing, integers are signed (not unsigned), negative fahrenheit temperatures are now correctly parsed (and converted to Celsius)
• Parse comment from end of weather reports - if it's short enough treat it as the wx station/software type string
• Parse weather data sent after a compressed position
• Parse snowfall from normal wx packet
• Added test for positionless wx packets
• Added position packet format in hash ('format' => 'compressed')

1.15 Tue May 4 20:44:31 EEST 2010 - Hessu, OH7LZB

• Added a couple of missing tests and 'format' => 'nmea'

The time to test has arrived

Okay, I'm now officially fed up. Fed up with my own bugs caused by the complexity of the aprs.fi software. Every now and then I change something in one corner, maybe to fix a bug or to add a little feature, and that breaks something small in another corner of the project. Because I fail to notice the bug, it might be broken for days until someone actually tells me. Sometimes it's the embedded maps, sometimes it's the Facebook integration, sometimes it's AIS feeding using one of the three methods. And usually they're broken because I changed something that was quite far from the broken part.

It's time to do some automatic testing. It's no longer feasible to manually verify that things work after making a change and before installing the software on the production servers - there are too many things to test. It takes too long, and something is easily forgotten.

Writing automatic tests in hobby projects like this one is usually not done, because it generally feels like the time spent on writing testing code is wasted - hey, I could be implementing useful features during that time. But on the other hand, once some testing infrastructure is in place, it's much quicker and safer to implement changes since it takes only one or two commands to run the test suite and to see that the change didn't break anything.

A little terminology:

Unit tests execute small parts of the code base (usually a single function/method, or a single module/unit/class). They feed stuff to that little piece of code and see that the expected results come out. They're often run before actually compiling and building the whole application. As an example, I can write a test to run the APRS packet digipeater path parser with different example paths, and check that the correct stations are identified as the igates and the digipeaters.

System tests run the complete application, feeding data from the input side (for example, APRS or AIS packets using a simulated APRS-IS or JSON AIS connection) and checking that the right stuff comes out at the output side (icons pop up on the map, updated messages show up on the generated web pages).

The open-source Ham::APRS::FAP packet parser, which is used by aprs.fi, already has a fairly complete set of unit tests. After changing something, we can just run the command "make test" and within seconds we know if the change broke any existing functionality. If you follow the previous link to CPAN, and click View Reports (on the CPAN Testers row) you'll get a nice automatically generated report from the CPAN testers network. The volunteer testers run different versions of Perl on different operating systems and hardware platforms, automatically download all new modules which are submitted to the CPAN, run the unit tests included with the modules, and send the results to the cpantesters.org web site. Thanks to them, I can happily claim that the parser works fine on 8 different operating systems (including Windows), a number of different processor architectures (including less common ones like DEC Alpha and MIPS R14000 in addition to the usual 32-bit and 64-bit Intels), and with all current versions of Perl, even though I only run it on Linux and Solaris myself.

Last Friday SP3LYR reported on the aprs.fi discussion group that negative Fahrenheit temperatures reported by an Ultimeter weather station were displayed incorrectly by aprs.fi: -1F came up as 1831.8F and 999.9C. I copied a problematic packet from the aprs.fi raw packets display and pasted it to the testing code file in the FAP sources (t/31decode-wx-ultw.t), and added a few check lines which verify the results. Sure enough, the parsed temperature was incorrect, and "make test" failed after adding a test with a low enough temperature. There were a couple of test packets in there before, but none of them had a temperature below 0 Fahrenheit.

Only after adding a test case for this bug I started figuring out where the actual bug was. After fixing the bug the "make test" command passed and didn't complain about the wrong parsing result any more. I committed the changes to the SVN revision control system, and then installed the fixed FAP.pm module on aprs.fi. Because none of the other tests broke after the fix I can be sure that I didn't break anything else with the fix. And because there's now a test in the unit test suite for this potential bug, I'm sure that same bug will not accidentally reappear later.

This is called test-driven development, and it can be applied to normal feature development just as well. First write a piece of code which verifies if the new feature works, and then write the code which actually implements the functionality. When the test passes, you're done. You need to write a bit more code, but it's much more certain that the piece of code works, and won't break later on during the development cycle.

None of this is news to a professional programmer. But from now on I'll try to apply this approach to this hobby project too, at least to some degree. Yesterday I added a few unit tests to the code to get started:

$ make testperl
--- perl tests ---
PERL_DL_NONLAZY=1 /usr/bin/perl \
"-MExtUtils::Command::MM" "-e" \
"test_harness(0, 'libperl', 'libperl')" \
tests/pl/*.t
tests/pl/00load-module.......ok
tests/pl/11encoding..........ok
tests/pl/20aprs-path-tids....ok
All tests successful.
Files=3, Tests=83, 1 wallclock secs ( 0.26 cusr + 0.06 csys = 0.32 CPU)

It ran tests from 3 test files, and the files contained 83 different checks in total. The first file makes sure all Perl modules compile and load. The second file tests the magic character set converter using input strings in different languages and character sets, checking that the correct UTF-8 comes out. The third one runs 24 example APRS packets through the digipeater path inspector. By comparison, the Ham::APRS::FAP module's test suite has 18 files and 1760 tests, and it's just one component being used by aprs.fi.

In the near future I'll try to implement a few system tests which automatically reinstall the whole aprs.fi software in a testing sandbox, feed some APRS and AIS data in from the different interfaces, and see that they pop up on the presented web pages after a few seconds. I want to know that the live map API works, the embedded maps and info pages load, and that the Facebook integration runs. With a single 'make test' command, in 30 seconds, before installing the new version on the servers.

But now, some laundry and cleaning up the apartment... first things first.

Ham::APRS::FAP 1.13 released

Version 1.13 of Ham::APRS::FAP was uploaded to CPAN today. The tests seem to be passing nicely, and the code has been running on aprs.fi for a long while, so the upgrade should be safe.

Ham::APRS::FAP is the APRS packet parser used by aprs.fi. If you wish to decode APRS packets in Perl, this is what you need. It's pretty fast, it's stable, and it can report it's errors in a developer-friendly way. And it's free (as in free speech).

Version 1.13 contains the following small fixes:

• Allow anyone to update telemetry parameters, skip the source callsign check
• Allow a PHG of 0000 for deleting PHG
• Added new error code sym_inv_table for invalid symbol table char
• Added local time zone parsing to object timestamps
• Fixed comments parsing for last resort !-location packets
• Parse APRS message rejects (negative acknowledgments)

Some of the improvements were implemented by Tapio, OH2KKU, while others were done by myself.

Ham::APRS::FAP 1.12 released

On monday evening I uploaded the new version of our APRS parser on the Comprehensive Perl Archive Network.

Version 1.12 includes a parser for the !DAO! extension, which can be used in uncompressed and mic-e APRS packets to report the datum and an additional digit of position resolution. It also reports the position resolution (in meters) for parsed position packets, depending on the type of the packet (compressed, mic-e, uncompressed), the presence of DAO and the amount of position ambiguity. These improvements were implemented by Tapio, OH2KKU.

I felt an urge to do some slight improvements on it myself before releasing it, and couldn't come with anything that would be both quick to implement and actually useful, so I just added some tests, cleaned up the API and packaging slightly, fixed POD documentation style, upgraded MakeMaker, and added a little example script, just to make CPANTS happier and increase the Kwalitee rating a bit. :)

!DAO! support for greater precision

Tapio Sokura, OH2KKU, implemented the APRS precision and datum option (DAO) in the Ham::APRS::FAP parser last night. Looking at the IRC channel log, he started working on it at around 01:40 AM local time, and finished at 04:33 AM.

A few minutes ago I installed the new parser code on aprs.fi, so those of you transmitting uncompressed or mic-e packets with the DAO extension should now see the improved precision.

Compressed format packets have enough precision (around 1 foot) already, so DAO is not useful with it. aprs.fi properly decodes compressed packets (including speed and course), and compressed packets are the smallest ones (they help conserve valuable channel time on 144.39 and 144.800 MHz), so I would recommend using the compressed format instead of uncompressed or mic-e with DAO.

The only supported datum is still WGS84, no conversion from other datums specified by DAO is done.

Ham::APRS::FAP 1.11 released

I uploaded version 1.11 of the APRS parser module on CPAN today. It includes a telemetry parser and a small fix in the automatic tests (a timestamp check started failing after a certain date, since the APRS timestamp in the tested packet doesn't specify a full date+time). This is the version currently used by aprs.fi.

APRS parser module used by aprs.fi released

Here's a little Christmas present for the programmers out there:

Last night the APRS parser library used by aprs.fi was uploaded to CPAN. It's written in pure Perl, and it's reasonably complete, stable and fast. It's licensed under the same open terms as Perl itself (you pick: Artistic or GPL). It is written by Tapio Sokura, OH2KKU, with some additions by myself (weather packet parsing, machine-readable return messages, unit tests). Without this module, and the time Tapio took to write it in the first place, I wouldn't have gotten this far with aprs.fi. Parsing APRS packets is not easy.

It does miss some features (like telemetry parsing, and a few other smaller things), but I believe it to be the best parser currently published as open source. It validates input packets in a rather strict way and actually tells you what is wrong with each rejected packet. It does not crash when it sees something unexpected or corrupted. It returns numeric data in SI units. It's good enough to parse the APRS-IS feed in real time on any modern computer. It might even run on Windows with Perl (cygwin?) installed.

To install it on an Unix-like system (like Linux) with Perl using CPAN (as root):

root@proge:~#> perl -MCPAN -e shell
... if you're running the CPAN client for the first time, answer the questions, and once you're in the CPAN shell: ...
cpan> install Ham::APRS::FAP

You can also download the Ham::APRS::FAP module and browse it's contents on the CPAN web site. Installation instructions can be found in the README file.

Oh yeah. FAP stands for Finnish APRS Parser (or Fabulous APRS Parser). Ham::APRS::Parser was taken.