Important Hints to iGate developers

By connorgoodwolf - Own work, CC BY-SA 4.0

There are a lot of APRS iGate implementations out there, and new ones are popping up every few months these days. A lot of the new ones are made for various microcontroller chips and development boards (Arduino, ESP8266 and friends). Some are made for SDR sticks, some are built in mobile phone applications. Many of them reduce the amount of hardware needed to run an iGate by fully removing the need for a TNC, a full-blown computer, or an actual amateur radio transceiver.

Very cool stuff!


It would seem like writing an iGate would be very easy: receive packets off the air, put them in TNC2 format, and pass them to the APRS-IS servers over TCP (specifications: connecting, design, details). But it turns out that there are quite a few things that can go wrong in the implementation, and in fact, some things seem to be going wrong the same way fairly often. Some bugs are fairly common and some seem to pop up in multiple different iGates, old and new.

Currently the most popular ones are due to the false assumption that all packets are either plain 7-bit English ASCII text with no NUL bytes or other binary stuff, or some specific Unicode encoding such as UTF-8. These assumptions lead to the packets getting truncated at the first NUL (binary 0) byte, or otherwise modified in a way that scändic and øthér ïnternátiønal characters get mutilated if they don't happen to use the same encoding. Sometimes a truncation of a packet at the first NUL may lead to the packet being concatenated together with the next packet.

I get to decode the corrupted packets on aprs.fi, and I also get to process the corrupted ones in the aprsc APRS-IS server software which is fairly common these days. If the bugged packets are not detected and handled, there usually are some folks complaining to me about the odd results, since they are often received from aprsc and viewed on aprs.fi.

I can't blame the authors of the new applications – I have to admit that APRS documentation is not great, and it's easy to stumble across the same bugs if you haven't fixed them once before.

Every now and then I spend an evening trying to identify some of these bugs in the APRS-IS client applications and file bug tickets to the software authors. Last night I filed 3 packet corruption bug reports on Github for 3 different iGate applications. These are not the first ones, I've probably filed 30 in the past. It's not hard to spot these once you know what to look for.

I'd like to take this chance to thank to all the authors who fix the bugs in a timely manner and release the fixed versions!

Based on this experience I wrote a little document listing common iGate bugs: IGATE-HINTS.md. It'd be great if all APRS software authors could glance through it and make sure these bugs are not present in their own software. If you spot someone writing a new iGate, please pass on the link to the new author – hopefully these bugs will then be avoided right from the beginning.

Unfortunately some buggy iGate applications are orphaned and no longer developed. A few of those applications are fairly popular, too. It'd be great for the network if iGate operators could gradually upgrade to applications which still get updates, and don't exhibit any of the packet-corrupting bugs.

A lot of iGates are also not getting updated, even though they run software which would have updates available. Some iGate operators feel that the iGates work, sort of, since most packets go through them just fine, most of the time. I think I'll try to make aprs.fi highlight some of the more common issues on the station info page in the future.

If you know a little Python or C, and you've got a little extra time, let me know – there are some bugs out there in the Open Source Land which could use a helping hand.

New version of aprs.fi's APRS packet parser released: Ham::APRS::FAP v1.21

Yesterday I uploaded a new version of the APRS packet parser used by aprs.fi, Ham::APRS::FAP, to the CPAN. It's open source, so if you're a little bit weird like me and enjoy programming in Perl, this might be useful. Here's a full list of changes:

• Improve make_position() to support HMS UTC timestamp. make_position() now returns the packet type character so that it can signal the presence of a timestamp.
• Improve make_position() to support comment string, !DAO! extension, altitude encoding. Fix rounding errors in lat/lon/speed. Support generating packets with no speed or course. Take optional parameters in a hash parameter. Implement unit tests for make_position().
• Set up Travis automatic running of the (existing) unit test runs.
• Additional character escaping in regular expressions to deal with deprecated functionality in Perl 5.22.

The packet parser is one of the more complete APRS decoders, although I'm not sure if any of the parsers deal with all of the APRS features and packet variants yet.

Mic-E mangled packet parsing improvements

One of my APRS transmitters
using the Mic-E packet format.
Fresh photo for the blog.

Mic-E encoded packets, as commonly transmitted by Kenwood & Yaesu radios, Byonics trackers and others, are often corrupted by broken APRS igates. The speed/course bytes in these packets,  which often have binary values in the 0x1C - 0x1F range (unprintable control codes in the ASCII table), or the 0x7F DEL "character", are commonly removed or replaced with spaces (0x20, space bar) by certain iGate setups.

Mangling packets in transit is wrong, and by default it would make these position packets, which are completely valid according the protocol specification, fail to decode, and fail to plot on aprs.fi. To get around this aprs.fi and many other decoders try to detect these broken packets and extract whatever information is left in them after the broken igate.

Yesterday evening I fixed a few bugs in the Mic-E packet decoder used by aprs.fi. This improves the handling of mangled packets a bit further.

• The correct symbol table identifier (primary or secondary) is now successfully decoded from mangled Mic-E packets. The table identifier byte was incorrectly extracted before the demangling trick - from the wrong offset due to the mangling. Thanks to KD0KZE for the bug report.
• Speed and course information is no longer decoded from Mic-E packet if demangling was applied. That information has been lost in transit when the packet got mangled by the igate.
• Mangled Mic-E packets are now indicated with a mice_mangled flag in the decoded raw packets. Switch from Normal to Decoded mode in the raw packets list to see it in action. Expect to see broken igates highlighted on aprs.fi later on.

Today I released version 1.20 of the open source APRS packet parser, which contains the above improvements. There are a few other fixes in there, which were already installed on aprs.fi earlier last year.

Back button should now work!

Ocicat kitten, 5 weeks, (c) Heikki Hannikainen

Last night I spent a couple hours trying to improve URL navigation on the aprs.fi real-time map. The back and forward buttons have behaved quite weirdly for some time, and that simply needed to be fixed. The upgrade went in just a few minutes ago.

With the help of the HTML5 history API I think I mostly got it right now. It is of course likely that I missed some corners, so keep the feedback coming - either in blog comments or posts on the discussion group.

Basically, the browser history now only gets a new entry when the user initiates a "major change" in the map status: a new callsign or a new address is searched. Those new history entries can be navigated using the web browser's back and forward buttons.

Minor state changes, such as changing the time range / tail length parameters, and any changes not initiated by the user, should only update the current browser history entry. After first looking up station A, then looking up station B, and then changing time range, clicking back will take you to station A. Clicking forward will then take you to station B - with the changed time range! Try it out, you'll see what I mean.

Naturally this won't work so nicely with steam-powered web browsers, or the oldest boat anchors out there.

Christmas is coming, and postcards need to be printed, so I was required to take some kitten photos with seasonal props yesterday.

aprs.fi upgraded on 2013-10-20

Photo unrelated to upgrade! Some floppies we found at the
OH2K club station while cleaning and discarding old stuff.
Quite nostalgic stuff for me.

I just finished upgrading aprs.fi. It's been a while since the last full upgrade (a few small patches have been applied in the meanwhile). Here's a summary of the changes:

A fix for Internet Explorer 11. The real-time map view now loads and works on IE11, which has just become available to the general public.

When running on a mobile device (ipad, android tablets, etc), the callsign input form field no longer gets focus automatically during the map load process. It tends to pop up the on-screen keyboard, which is quite annoying. On non-mobile devices the automatic focus still happens, so that you can start typing in a callsign right after opening up the site.

Google Maps API code upgraded to version 3.13. I somehow suddenly get flashbacks from Donald Duck comics. Odd.

Some internal code refactoring was done in the API and authentication code. If all went well, no visible changes to anyone.

aprs.fi upgraded: track tail length adjustable

Kukkatarhan Cissus sings:
"We wish you a merry Christmas!"

I upgraded the aprs.fi service yesterday evening. I rebooted both main servers after upgrading operating system and database components. After that I upgraded the aprs.fi software itself.

It's been a long while since the last significant upgrade. There have been small patches and quiet bug fixes here and there. My excuse is the time I've put into completing and deploying aprsc, a new open-source APRS-IS software written in C together with OH2MQK. It's now in use on about 1/3rd of all APRS-IS servers and appears to work fine.

There is now a separate track tail length drop-down selector on the right side of the map. It's now possible to select "show last positions of all stations in the are for the past 24 hours" but still only show a track tail for the past 30 minutes, for example. Drawing tracks for 24 hours tends to clutter the map a lot and it can also be very slow on many devices.

The old wildcards-on-initial-navigation bug is finally fixed. If you go to http://aprs.fi/OH7* (use a wildcard character, or otherwise multiple stations come up from the initial search) clicking on the callsigns within the popup did not work. Fixed!

Fixed exporting of stations which have transmitted telemetry.

Lots of improvements on Tetra protocol support and closed enterprise service model (not visible on aprs.fi).

Improvements in command line tools for service maintenance.

Small performance improvements here and there.

I also upgraded the operating system and database (the usual security fixes and some small performance improvements).

We've got new kittens too. Been taking a lot of kitten photos!

Linking parameters fixed and proper "Show all" button

The "others" and "timerange" parameters described on the Linking page should now work again. They were broken since the anchor navigation upgrade. Thanks to Oba JA7UDE for the bug report!

To properly fix how the "others" parameter  works I also added a new "Show all" button which is now quite a bit more  sensible than the old "Show all" link:

• The button is disabled and greyed out when it really doesn't do anything (like, when you're not tracking a station and all stations are displayed anyway)
• "Show all" can now be toggled between on and off, not just on
• The button visually indicates whether it's on or off (depressed or not)

aprs.fi upgrade on Monday 2012-03-19

aprs.fi will be upgraded on Monday 2012-03-19. Some downtime should be expected, starting around 9 AM local time (0700 UTC). If you can find some bugs in beta.aprs.fi, this is the time to report and get them fixed before they make their way to aprs.fi! Here's the long list of changes included in the upgrade. It's so long that, as usual, some new bugs will probably pop up. Please report them on the discussion group and I'll try to get them fixed right away.

Anchor navigation and more AJAX

Achilles (left) and Ajax (right) play a board game with
knucklebones on this late 6th-century lekythos, a type of
oil-storing vessel associated with funeral rites. Photo taken
in Musée du Louvre by Marie-Lan Nguyen, 2011
(CC by attribution, Wikipedia/Wikimedia).

The technical implementation of the navigation in the real-time map has been completely changed. In the new model the map page is not reloaded from the servers and initialized again from scratch every time an user changes the view by searching for a new callsign or an address. This AJAX magic will make callsign searches and other view modifications considerably faster, especially on slow computers and slow connections such as mobile devices. It'll also reduce load on my and Google's servers. And make the browser's bookmark functions and the back button work better than before.

All of the links used by aprs.fi to refer to specific map views have changed, but the traditional ones documented on the linking instructions page will keep working, and I encourage you to use them as before - they're not going anywhere.

Try searching for OH2K and OH2TI one by one again and again, and you'll notice both the improved loading speed, and a nice panning effect. It'll pan whenever the newly searched station is close enough to the current view.

Improved address search

Address search has been improved to make a better use of the data returned by Google's API. A marker for the result is now only shown if the result is accurate (such as a street, or a house number on a specific street).

Address search will also adjust the zoom level - "Finland" or "Pohjois-Karjala" or "USA" should actually fit the specified region in view. The zoom levels come from Google and I can not make it any smarter than that (yes, "USA" will zoom out a bit too much). But it's certainly better than it was (fixed zoom level after address search).

Improved response times

When you make changes to the real-time map view, for example by zooming out or selecting a time range, aprs.fi will now react much more quickly.

New time range back/forward buttons

When a station is tracked, there are now two new handy additional arrow buttons to jump the time range back and forward. If you have selected an arbitrary range, it will jump the same amount back (select a week, and it'll jump to the previous week). Otherwise it'll select the current whole day (UTC 00:00:00 to 23:59:59 - sorry, no local time support yet) on the first click, and the following clicks will jump by 24 hours.

Sharper map graphics on iPad/iPhone

The scaling of the web page was tuned to switch automatically so that map graphics are not blurred on the iOS devices, especially when the device switches between landscape and portrait orientation.

Other small things

Searching for "OH2RDK" will now give a proper "there are other SSIDs available although this one doesn't exist" response.

aprs.fi now uses a new version of the Google Maps API, bringing in some visual updates and speedups from Google.

An old bug which makes stations disappear after a quick zoom-in-and-back-out operation (and some other cases) has been fixed.

The "street view is off by 200 meters" bug is fixed. Also, the map should stay centered when Street View is enabled or disabled.

The ruler tool now displays distances shorter than 1 km in meters to allow measuring short distances.

When a station is tracked on a mobile device, the info balloon does not automatically pop up and block the whole view.

Raw packets view in decoded mode displays the position packet's type (compressed, mic-e...).

Tuned digipeater/igate "heard" map to collect more data and leave less gaps in the map.

Support for new major version of the database server (SQL syntax changes).

CSRF security fixes were implemented in many places.

Some rough corners have been rounded up (literally), and a few shadows have been cast (again, literally).

Does aprs.fi feel slow?

Be sure to try it with a modern, quick browser such as Google Chrome, Safari or Internet Explorer 9. If you're upgrading from Firefox or an older Internet Explorer, you'll be surprised by the difference it makes.

Embedded map backlinks, API upgrades, speedups and new bat-toys

The embedded map displays a link back to aprs.fi in the low right corner again, as it used to do until July 2010, when aprs.fi switched to Google Maps API v3 and got Street View support. It should be much less annoying than the old ugly box, though. The new link is smaller and slightly transparent (on modern browsers).

I also upgraded to Google Maps API from version 3.4 to version 3.6, which brought in several enhancements and bug fixes. The API has it's own change log. Noticeable changes include fade transitions for map tiles when loading and changing the map type, Street View pegman previews, some speedups, and several bug fixes for iOS and Android.

I have also recently made some database access optimisations making many map displays and some raw packet searches load much faster.

Since 28th of August most, if not all, APRS symbols now have hover-on tooltips. Keep the mouse cursor on top of a station symbol and a description text should magically appear!

And, as you can see from the attached photo, I have some new toys for 70 cm. Built in GPS, text messaging, trunking support, rugged construction, submersible, digital and analog voice, and a few other nice features. And surprisingly inexpensive on ebay, when compared to new amateur HTs with APRS.

Small fixes on Monday morning

On Saturday and Sunday I did a little coding, and installed the new version before leaving for work this morning.

When an user tries to save a filter list twice with the same name, and error message is now displayed. It turns out that a lot of users press the "Save as new list" button after making any changes to the existing list, and end up with 20 lists (or more!). That button is only for creating a new list. Any changes to existing lists are saved immediately without pressing any additional buttons.

The Preferences view got a new hint text pointing users towards the new Filter tool button. A lot of users are complaining about the "missing filters".

When an user account is deleted, data related to that account is now more thoroughly deleted. That includes web stations, alerts, favourites, and so on.

Alert configuration changes should now actually do all the necessary database changes to enable new alarms and disable old ones.

The real-time map should now more reliably draw a symbol on the last point of a track. An old bug sometimes left the last point without a symbol.

When a client computer wakes up in the morning from a good night's sleep mode, the data reload should now be a quick one, instead of a very slow one.

Session cookie processing was also improved, and memcached was upgraded.

Again, it doesn't make sense to take a photo of software, and a blog needs to have some photos, so kittens it is again. This is FIN*Kukkatarhan Bellium, a chocolate-spotted Ocicat, posing in Mikkeli a few weeks back.

Clickable track lines, center & zoom links, etc

I've just installed a few small improvements on aprs.fi.

The track lines are now clickable, allowing for easier track line identification and jumping to the current position of that station.

The info balloons of stations (and the track lines) now have "center" and "zoom" links. There's also a new separator line in the info balloon, indicating the track line color used by that station.

The track waypoint hover-on callsign labels now also show the symbol of the station.

The track line drawing code received a nice little bug fix: the station symbol image is only really drawn at the last position within the selected time range. The symbol image will no longer jump and run away when you scroll the map along a track line towards the current position of the station.

IE7 bookmarks fixed, and help texts translation

The bookmark feature now works on IE7. There was a little bug in there, or maybe it actually is in Microsoft's Internet Explorer 7, since it worked with all other browsers I tried. Workaround implemented.

Additionally, I've finally moved the few help texts which can be opened by the '?' links to templates, so that they can be translated to other languages. This has been requested quite a few times. They're quite long, and I should probably write more of those, which will mean more work for the volunteer translators. Again, a big THANK YOU to all of you – translations are a very valuable contribution and they do help a lot of users to understand the site!

IPv6 support in APRS packet parser - FAP version 1.17

Version 1.17 of the Ham::APRS::FAP APRS packet parser has been released and uploaded to the CPAN. It should appear at the mirrors during the next 24 hours or so. The main feature is IPv6 support in APRS packet path parsing.

What?!? IPv6 support in APRS?

aprsd, and sometimes other software too, sends position packets to the APRS-IS using the qAI Q construct. qAI triggers packet path tracing in the APRS-IS. Each APRS-IS server inserts the hexadecimal source IP address of the previous server to the packet. Now that quite a few Tier2 APRS-IS servers have IPv6 connectivity, even some T2 hubs, IPv6 hex addresses can appear, and they are a bit longer than regular callsigns, and were getting rejected by the parser.

SRCCALL>DSTCALL,qAI,IQ3VQ,THIRD,92E5A2B6,T2HUB1,200106F8020204020000000000000002,T2FINLAND

In this case 92E5A2B6 is the IP address of THIRD, and 200106F8020204020000000000000002 (2001:6f8:202:402::2 in normal IPv6 printed representation) is the IP address of T2HUB1 (T2FINLAND happened to be connected to T2HUB1 at the time).

Here's the complete list of changes:

• Remove / or ' ' from beginning of comment after parsing away PHG, altitude and other optional data
• Allow hexadecimal IPv6 addresses in APRS-IS paths after Q construct
• Added tests for uncompressed packet altitude (negative, too)
• Fixed destination callsign based symbol selection for 'BC'
• Updated URLs to aprs.org, etc

aprs.fi is also available using IPv6 at ipv6.aprs.fi.

Mobile signup and password recovery fixed

I've fixed the captcha code (recaptcha) to be valid xhtml, so that mobile browsers with proper strict XHTML parsing (such as the iPhone/iPad family) accept it. It's now possible to sign up and do password recovery using those devices.

IE6 and IE7 broken for a day

Sorry, I broke the real-time map yesterday for the users of Internet Explorer 6 and 7. It did work using IE8 and all other browsers. Fixed this morning.

My excuse is that my home vmware installation, which I use for running IE6 and IE7, was somehow broken, so I didn't test those browsers.

Thanks to Phillip ZL2TZE for bringing this to my attention.

iPad location sharing, bugfixes and Android support

I've installed a couple of fixes in the web locations updating code now. It failed to work for about 6 unlucky users due to a little bug which didn't take database replication delays into account. But that's sorted now.

I've also put the 'share location' button on the non-mobile site, so that it can be used on the iPad and other larger devices with GPS. We'll see what effect that has!

The good news is that it seems to work on several Android phones:

• Google Nexus One
• HTC Dream
• HTC Espresso
• Samsung Galaxy S
• Motorola Droid
• LG Ally

It'll probably work on the rest of the new ones with GPS, too.
The new tool buttons are now documented in the manual.

Translating fixed

The aprs.fi translation tool was broken when the Google Maps API v3 integration was installed on July 5th. I've fixed it now to not use some API functions which were used from the maps API, and translation work can continue. There are some new strings which have been added lately for Street View integration, tooltips, data exporting and other new features, so most languages need some updating.

Again, I wish to thank everyone who has submitted translations – they are really very useful for a lot of the users around the world!

Unfortunately I did a small mistake during the installation of this bug fix, and the real-time map would not load for several minutes right after the initial installation tonight. Sorry about that.

Google fixed the Opera issues

I wrote about the Opera issues recently.

It turns out that Google has fixed polylines with opera yesterday, and track lines are drawn just fine now. Heard maps, phg circles, rulers and locator lookups seem to work, too. Opera is still not officially supported, so further issues may appear, but at least a bunch of very visible bugs were fixed.

Tooltips, center map here, quick refresh, and some bugfixes

Here are a few improvements from the past couple days which were installed this evening:

The top right corner buttons of the real-time map now have hover-on tooltips. The right-click context menu of the real-time map now has a 'center map here' option.

The station info balloon will only pop up automatically in the real-time (and embedded) maps, if the map is at least 300 pixels wide and 260 pixels high. This should keep the tracked station in view even on a small view.

The directional PHG cardioids on the real-time map were fixed, they were broken for a while due to a little bug. Same goes for clicking on the Google Earth KML tracking link when tracking a single station.

The static map positioning was fixed on the info pages with Internet Explorer.

The real-time map refreshes quickly after panning the map, instead of waiting for the regular update timer.

AIS ship callsign display is now more correct in the info balloon (no separate tracking link for the callsign).

Map on info page, with symbols, and other small things

The info pages now have a little static map showing the station's position. I hope you like it this way. The static maps have also been upgraded to show proper APRS symbol graphics! For example: http://aprs.fi/info/SP3WBX-9

Some of the pages now have an "add this" button which allows you to easily share the page on your favourite social networks.

The raw packet display was fixed to be really, really binary clean. It used to choke on some high binary values.

Weather packets with an unrealistic high or low temperature are now ignored.