Friday, March 13, 2015

Device identification database updated, available for other apps

Most APRS devices and applications transmit an unique AX.25 destination callsign in all their packets, so that receiving stations can figure out which application or device is transmitting each packet. Bob Bruninga maintains a tocalls.txt index file, which lists all the assigned destination callsigns.

Those devices which use the Mic-E encoding to transmit position packets encode the latitude and a little bit of the longitude within the destination callsign, in which case something else has to be done for device identification. Mic-E device IDs are encoded around the comment text, with one character in the beginning of the comment text and zero to two characters in the end of the comment. The Mic-E type codes are indexed in mic-e-types.txt.

Now, when an application such as aprs.fi wishes to automatically decode the destination callsigns and type codes to readable application names, as seen on the aprs.fi station information page, the author of that application needs to collect all the device identifiers from those two files, and somehow convert them to application source code, or a configuration file that can be read by the application. The master files are written with human interpretation in mind, and it's rather hard to make an application automatically parse out the identifiers from them. It needs to be done manually, and whenever new devices or applications are published, all the applications wishing to detect the new ones need to get an update. All softwares authors need to get notified that there are some new devices, and then somehow add the devices in their respective configs. That's quite a lot of extra work that would be better spent writing some fancy new features instead.

OH7LZB-7, correctly identified as a Kenwood TH-D72,
at Mikkeli International yesterday afternoon.
Aircraft and training provided by MIK at Helsinki-Malmi.

For aprs.fi, I initially made a Perl module, Ham::APRS::DeviceID, and published it as open source, so that other software authors could use the index too, and skip the manual labour-intensive part of parsing the master text files. For some obscure reasons a number of programmers have chosen to use other programming languages than Perl, and the module was of limited usefulness for them.

To improve the situation, I converted the device index in YAML (Yet Another Markup Language), which is easy to read and edit by humans, and also easy to read and edit by computer programs. I also wrote a little converter program which parses the tocalls.yaml file and outputs the same data in JSON and XML formats, which are popular file formats for passing data between computer systems.

I then updated the original Perl module to read the YAML file, and removed the old database that was embedded in the code. DeviceID.pm 2.00 and later use tocalls.yaml. I also updated aprs.fi to use the new version of the Perl module, and tocalls.yaml. The update brought a number of new devices to aprs.fi, including the newer Yaesu radios.

Other programmers who wish to do device identification are welcome to download the YAML, JSON or XML files and use those. It should be straightforward to automatically update the device index during an application build, or even automatically update the index directly in the application.

New devices should still be first added to Bob's files, and only then to tocalls.yaml. If you have a new device which has already been added to Bob's index, please make a pull request or an issue ticket to update tocalls.yaml accordingly.

Monday, February 17, 2014

Mic-E mangled packet parsing improvements

One of my APRS transmitters
using the Mic-E packet format.
Fresh photo for the blog.
Mic-E encoded packets, as commonly transmitted by Kenwood & Yaesu radios, Byonics trackers and others, are often corrupted by broken APRS igates. The speed/course bytes in these packets,  which often have binary values in the 0x1C - 0x1F range (unprintable control codes in the ASCII table), or the 0x7F DEL "character", are commonly removed or replaced with spaces (0x20, space bar) by certain iGate setups.

Mangling packets in transit is wrong, and by default it would make these position packets, which are completely valid according the protocol specification, fail to decode, and fail to plot on aprs.fi. To get around this aprs.fi and many other decoders try to detect these broken packets and extract whatever information is left in them after the broken igate.

Yesterday evening I fixed a few bugs in the Mic-E packet decoder used by aprs.fi. This improves the handling of mangled packets a bit further.
  • The correct symbol table identifier (primary or secondary) is now successfully decoded from mangled Mic-E packets. The table identifier byte was incorrectly extracted before the demangling trick - from the wrong offset due to the mangling. Thanks to KD0KZE for the bug report.
  • Speed and course information is no longer decoded from Mic-E packet if demangling was applied. That information has been lost in transit when the packet got mangled by the igate.
  • Mangled Mic-E packets are now indicated with a mice_mangled flag in the decoded raw packets. Switch from Normal to Decoded mode in the raw packets list to see it in action. Expect to see broken igates highlighted on aprs.fi later on.
Today I released version 1.20 of the open source APRS packet parser, which contains the above improvements. There are a few other fixes in there, which were already installed on aprs.fi earlier last year.

Monday, December 2, 2013

Back button should now work!

Ocicat kitten, 5 weeks, (c) Heikki Hannikainen
Last night I spent a couple hours trying to improve URL navigation on the aprs.fi real-time map. The back and forward buttons have behaved quite weirdly for some time, and that simply needed to be fixed. The upgrade went in just a few minutes ago.

With the help of the HTML5 history API I think I mostly got it right now. It is of course likely that I missed some corners, so keep the feedback coming - either in blog comments or posts on the discussion group.

Basically, the browser history now only gets a new entry when the user initiates a "major change" in the map status: a new callsign or a new address is searched. Those new history entries can be navigated using the web browser's back and forward buttons.

Minor state changes, such as changing the time range / tail length parameters, and any changes not initiated by the user, should only update the current browser history entry. After first looking up station A, then looking up station B, and then changing time range, clicking back will take you to station A. Clicking forward will then take you to station B - with the changed time range! Try it out, you'll see what I mean.

Naturally this won't work so nicely with steam-powered web browsers, or the oldest boat anchors out there.

Christmas is coming, and postcards need to be printed, so I was required to take some kitten photos with seasonal props yesterday.

Wednesday, November 6, 2013

Amateur Radio Questions and Answers

Amateur Radio Stack Exchange is a question and answer site for amateur radio enthusiasts. It's currently in public beta, and a lot of good questions already have great answers.

The Stack Exchange Q&A site format is very popular in programmer circles - take a look at Stack Overflow for example - good questions and answers get voted up, and the best answers are prominently displayed, unlike in traditional discussion groups and mailing list archives where the good bits are often buried in the middle of a lot of uninformed answers and chatter that is only slightly related to the original question.

If you've got an amateur radio related question, please post it there. If not, try answering one of the unanswered questions. It's actually a lot of fun!

The site is currently in a beta, and needs a little push (questions, answers, votes) to keep going.

Start with the instructions on the About page.

Sunday, October 20, 2013

aprs.fi upgraded on 2013-10-20

Photo unrelated to upgrade! Some floppies we found at the
OH2K club station while cleaning and discarding old stuff.
Quite nostalgic stuff for me.
I just finished upgrading aprs.fi. It's been a while since the last full upgrade (a few small patches have been applied in the meanwhile). Here's a summary of the changes:

A fix for Internet Explorer 11. The real-time map view now loads and works on IE11, which has just become available to the general public.

When running on a mobile device (ipad, android tablets, etc), the callsign input form field no longer gets focus automatically during the map load process. It tends to pop up the on-screen keyboard, which is quite annoying. On non-mobile devices the automatic focus still happens, so that you can start typing in a callsign right after opening up the site.

Google Maps API code upgraded to version 3.13. I somehow suddenly get flashbacks from Donald Duck comics. Odd.

Some internal code refactoring was done in the API and authentication code. If all went well, no visible changes to anyone.

Monday, September 23, 2013

aprs.fi within iOS 7 Safari web browser

When opening aprs.fi using the iOS 7 Safari browser on an iPad Mini, the browser is quite consistently killed by the operating system due to an out-of-memory situation, especially if there are a lot of stations in your area to show. This looks pretty much like the browser crashed, but the device's console log displays the OOM errors quite nicely. The same happens when I try to use Chrome. It worked pretty fine on iOS 6 on the same device.

It seems to me that iOS 7 grabs more memory itself, leaving less memory available for the applications, and the devices with 512 MB memory (or less) don't have enough to load aprs.fi in the browser when a lot of stations are brought to view. iPad Mini and iPad 2 have 512 megabytes, the original iPad 1 has 256 MB, the 3rd- and 4th-generation iPads ("new iPad") have 1 GB (specifications).

Sep 22 19:53:51 Minipad MobileSafari[1407] : Received memory warning.
Sep 22 19:53:53 Minipad MobileSafari[1407] : Received memory warning.
Sep 22 19:53:54 Minipad com.apple.launchd[1] (UIKitApplication:com.apple.mobilesafari[0xe421][1407]) : (UIKitApplication:com.apple.mobilesafari[0xe421]) Exited: Killed: 9

The same probably applies to iPhones, but I didn't upgrade to 7 yet.

There's not much I can do about this, other than make it somehow load less stations and lines on the map initially.

Or release a native iOS application which doesn't process the map data in Javascript, and will be able to keep its memory use lower that way.

Wednesday, June 26, 2013

Presentation at Ham Radio 2013, Friedrichshafen

I've prepared a presentation for the Ham Radio 2013 event taking place this weekend in Friedrichshafen. The title of the presentation is "Providing authenticated amateur radio services on the Internet", and it'll be held at Conference Center East, room Paris, at 14:00. Same time and place as last year!

How can we provide Internet services restricted to amateur radio users with reasonable guarantee that non-hams do not gain access to the services, without an awful lot of manual work by the service provider and mailing license copies for every service? Current practices, alternatives, a proposal for better future implementation of distributed trust using X.509 certificates, and a demonstration of an implementation for a web service.

If you're an user of authenticated amateur radio web services such as the APRS-IS network, or Echolink, or Logbook of the World, this should be interesting. If you're developing a service like this, or would like to develop one, it should be especially interesting.

There's a catch, though. Due to a surprising event I won't be able to fly to Germany myself this weekend. Fortunately Erik Finskas, OH2LAK, has agreed to act as a proxy and present my material there! Not a bad deal, since he happens to be a seasoned professional in the right field, and has applied the presented technology both at work and in the linked UHF repeater network in Finland.