Tuesday, November 3, 2015

New symbol graphics and better support for mobile devices

Old symbols, scaled up, pixels obvious
I've just upgraded aprs.fi to use my new APRS symbol graphics set. The new symbols are drawn in vector format (as opposed to a raster format at a fixed resolution), allowing them to be rendered at larger and smaller sizes without distortion or blurriness. The new symbols are slightly larger than the old ones, making them easier to recognise on modern displays having smaller pixels than the old ones. They're also available in double resolution so that they're properly sharp on the 4K/retina displays found on many modern tablets, phones and computers!

New symbols, scaled up - no pixelation!
One downside is that they are just a little bit bigger than the old ones, taking up more space on the screen. On the other hand, the old ones were a bit too small on modern displays, and the very small resolution did make it hard to understand what the symbol tried to mimic.

The new aprs.fi symbol set will be soon be published as open source on GitHub, in both vector (Adobe Illustrator/PDF) and raster (PNG) formats. Other APRS applications may then use them, too, at no cost. Raster renderings will be made available in 24x24, 48x48, 64x64 and 128x128 pixel resolutions - drawing from raster sprites in apps is usually quicker and easier than working with the vector source material. Having the vector sources makes it possible to improve them and and replace individual symbols easily. If you need to render other resolutions or make some other fine tuning, you can run Illustrator for free for 30 days. The symbol set release even comes with a little piece of javascript which crunches out the 3 PNG files (primary, secondary, overlay characters) at the 4 resolutions in a few seconds.

Naturally I did not draw all of the symbols myself. Many are loosely or strongly based on the original symbol graphics, primarily to keep the familiar and consistent look. Some symbols I obtained from other sources, such as Wikipedia. In those cases I picked SVG versions which allow commercial reuse (source known, and the work is placed on public domain, or with a CC license which allows adaptation and commercial reuse). In any case, the source and copyright information is documented separately for each symbol.

The aprs.fi symbol graphics set does not contain additional symbols for overlays yet, mostly because it takes lots and lots of time to draw them, and the effort it took to create this set was pretty high already. Maybe later!

There is one obvious difference in the new symbol set: the "ham store" symbol has been replaced with a more generic "store" shopping cart, reflecting the current \h symbol definition in the master index. Please use the 'H' overlay character to specify an amateur radio shop.

To complement the symbol graphics, I've previously published a machine-readable (CSV/JSON/XML/YAML) APRS symbol description index, which is easier to integrate in applications than Bob's master list.

Improved mobile device support

aprs.fi will now work better than before on mobile devices. I fixed the signup/login flow and most of the text and data table views to scale more nicely on small devices, allowing  horizontal scrolling of tables. It'll need some more work to make it very nice, but this is a good start.

Friday, October 9, 2015

OpenStreetMap available again

Earlier this year aprs.fi was unavailable for a short while after Google disabled Maps API access due to mixed OSM/Google content being visible on the site.

I now spent a couple evenings setting up OSM again in a way that would not interfere with Google's policy of not allowing Google content such as Street View or address search results to be shown on top of non-Google maps. The result of that work is now live on aprs.fi.

OpenStreetMap maps are again visible, but Street View buttons and controls are hidden while in OSM mode.

While using OSM, address searches are done by using the GeoNames database. City lookups work quite well, but address searches do not seem to work, at least not for Finland. City/country lookups do not return information on how large the found place is, so zoom level does not adjust automatically to cover the place.

Wednesday, October 7, 2015

Beware: Cookies.

Credit: star5112 / Flickr Creative Commons
This is not the feature I actually wanted to implement next, but here goes.

aprs.fi now displays one of those oh-so-common warnings saying "This site uses cookies". There is some code in there to try to show the warning only to visitors from the European Union, but due to the inherent inaccuracy of GeoIP-type lookups, it may fail either way for some users.

aprs.fi has used cookies from the beginning, just like roughly all the other web sites in the world. Nothing in the use of cookies has changed (they're used everywhere and by everyone), but new regulations set by the European Union, and requirements set by Google, require aprs.fi to display this warning.

Once you click the '✖' button in the corner of the warning, a cookie will be set, so that your selection is remembered, preventing the warning box from appearing again. It'd be annoying if the box appeared every time. If you have disabled cookies, the warning will appear every time, since the web site cannot know if you have dismissed the box before or not, since that information would be stored in a cookie.

More information:

Sunday, June 21, 2015

DKIM, SPF and assorted tricks to get through spam filters

Today I've set up DKIM (DomainKeys Identified Mail) on the aprs.fi servers and within the aprs.fi DNS zone. A week back I already set up the SPF (Sender Policy Framework) records in the DNS, and fixed the reverse DNS information for the IPv6 addresses used by aprs.fi to send out email.

Mike Mozart / Creative Commons / Via Flickr: jeepersmedia
In non-technical terms, this should help GMail and other services to figure out cases of others sending email (spam?) on behalf of aprs.fi, and correctly classify those as junk. It also might help GMail figure out that the registration confirmation and password reset emails sent out by aprs.fi are actually not spam.

It has been a rather persistent problem - GMail has consistently labeled the registration emails as spam, and people have been asking why they're not getting the emails. In all cases the mails have been found in the spam folder. We'll see if this helps!

Thank you to postfix and opendkim for making this a rather easy thing to get going.

Thursday, April 2, 2015

Google Maps disabled on aprs.fi temporarily due to a TOS accident

I woke up today to find out that Google Maps are disabled on aprs.fi, and the site is slightly crippled.

Google had noticed that aprs.fi has integrated support for OpenStreetMap map tiles *and* Street View. I hadn't noticed that the combination of Street View on top of anything else than Google Maps is forbidden according to the Maps API Terms Of Service document. Or I had just forgotten about it when implementing OSM or Street View (there was probably a year or two of time between the two, and probably another year or two since I actually read the lengthy TOS).

Google had nicely sent me a few emails to warn about the unapproved combination, first one already on 5th of March. I had moved away from reading emails regularly in Gmail a long while back, but forgotten to arrange for these emails from Google APIs to go to my primary email address. Silly me. Since I didn't respond or do anything about the situation, for almost a full month, they disabled Maps API, which certainly got my attention.

Oh well. OSM support is disabled for now, and I notified Google that aprs.fi is again compliant with the terms. Should be back soonish.

I might put OSM support back later, and remove Street View instead, as OSM is probably more useful in practice at some areas (although admittedly less visual and "cool"). Or maybe make some sort of arrangement where Street View only works if you have Google Maps selected instead of OSM. Have to check with the Google folks if that's OK, there are a few variations we can try. The first thing is to get the main functionality back up, no matter which maps they are.

I'll post updates on Twitter (https://twitter.com/aprsfi) and Facebook (https://www.facebook.com/aprs.fi). Stay tuned.

My mistake entirely, although quite a human one. When did you last read the full text of the license agreement of your new application or mobile phone? :)

Friday, March 13, 2015

Device identification database updated, available for other apps

Most APRS devices and applications transmit an unique AX.25 destination callsign in all their packets, so that receiving stations can figure out which application or device is transmitting each packet. Bob Bruninga maintains a tocalls.txt index file, which lists all the assigned destination callsigns.

Those devices which use the Mic-E encoding to transmit position packets encode the latitude and a little bit of the longitude within the destination callsign, in which case something else has to be done for device identification. Mic-E device IDs are encoded around the comment text, with one character in the beginning of the comment text and zero to two characters in the end of the comment. The Mic-E type codes are indexed in mic-e-types.txt.

Now, when an application such as aprs.fi wishes to automatically decode the destination callsigns and type codes to readable application names, as seen on the aprs.fi station information page, the author of that application needs to collect all the device identifiers from those two files, and somehow convert them to application source code, or a configuration file that can be read by the application. The master files are written with human interpretation in mind, and it's rather hard to make an application automatically parse out the identifiers from them. It needs to be done manually, and whenever new devices or applications are published, all the applications wishing to detect the new ones need to get an update. All softwares authors need to get notified that there are some new devices, and then somehow add the devices in their respective configs. That's quite a lot of extra work that would be better spent writing some fancy new features instead.

OH7LZB-7, correctly identified as a Kenwood TH-D72,
at Mikkeli International yesterday afternoon.
Aircraft and training provided by MIK at Helsinki-Malmi.

For aprs.fi, I initially made a Perl module, Ham::APRS::DeviceID, and published it as open source, so that other software authors could use the index too, and skip the manual labour-intensive part of parsing the master text files. For some obscure reasons a number of programmers have chosen to use other programming languages than Perl, and the module was of limited usefulness for them.

To improve the situation, I converted the device index in YAML (Yet Another Markup Language), which is easy to read and edit by humans, and also easy to read and edit by computer programs. I also wrote a little converter program which parses the tocalls.yaml file and outputs the same data in JSON and XML formats, which are popular file formats for passing data between computer systems.

I then updated the original Perl module to read the YAML file, and removed the old database that was embedded in the code. DeviceID.pm 2.00 and later use tocalls.yaml. I also updated aprs.fi to use the new version of the Perl module, and tocalls.yaml. The update brought a number of new devices to aprs.fi, including the newer Yaesu radios.

Other programmers who wish to do device identification are welcome to download the YAML, JSON or XML files and use those. It should be straightforward to automatically update the device index during an application build, or even automatically update the index directly in the application.

New devices should still be first added to Bob's files, and only then to tocalls.yaml. If you have a new device which has already been added to Bob's index, please make a pull request or an issue ticket to update tocalls.yaml accordingly.

Monday, February 17, 2014

Mic-E mangled packet parsing improvements

One of my APRS transmitters
using the Mic-E packet format.
Fresh photo for the blog.
Mic-E encoded packets, as commonly transmitted by Kenwood & Yaesu radios, Byonics trackers and others, are often corrupted by broken APRS igates. The speed/course bytes in these packets,  which often have binary values in the 0x1C - 0x1F range (unprintable control codes in the ASCII table), or the 0x7F DEL "character", are commonly removed or replaced with spaces (0x20, space bar) by certain iGate setups.

Mangling packets in transit is wrong, and by default it would make these position packets, which are completely valid according the protocol specification, fail to decode, and fail to plot on aprs.fi. To get around this aprs.fi and many other decoders try to detect these broken packets and extract whatever information is left in them after the broken igate.

Yesterday evening I fixed a few bugs in the Mic-E packet decoder used by aprs.fi. This improves the handling of mangled packets a bit further.
  • The correct symbol table identifier (primary or secondary) is now successfully decoded from mangled Mic-E packets. The table identifier byte was incorrectly extracted before the demangling trick - from the wrong offset due to the mangling. Thanks to KD0KZE for the bug report.
  • Speed and course information is no longer decoded from Mic-E packet if demangling was applied. That information has been lost in transit when the packet got mangled by the igate.
  • Mangled Mic-E packets are now indicated with a mice_mangled flag in the decoded raw packets. Switch from Normal to Decoded mode in the raw packets list to see it in action. Expect to see broken igates highlighted on aprs.fi later on.
Today I released version 1.20 of the open source APRS packet parser, which contains the above improvements. There are a few other fixes in there, which were already installed on aprs.fi earlier last year.