Parser updates and new delayed packet filtering feature

I've updated the APRS device identification module Ham::APRS::DeviceID to include detections for new APRS devices (KissOZ, anyfrog, unknown mic-e, SARTrack, Altus Metrum, SM2APRS, aprsc, NW Digital Radio UDR56K). The new version, 1.06, has been published on CPAN for your open source pleasure, and installed on aprs.fi.

I also published version 1.19 of the Ham::APRS::FAP packet parser used by aprs.fi. It only includes a small fix to the binary value telemetry bit order in Base91 comment telemetry. But at least that keeps me in the regular 1 release per year schedule! Just in time!

The larger feature in the aprs.fi upgrade is that it now utilizes the sequence number present in Base91 comment telemetry for detecting delayed packets.

APRS packets notably lack any sort of sequence number that could be used to detect old duplicate packets arriving late, or to place them in the correct order. Newer Byonics trackers can transmit telemetry (battery voltage, temperature, etc) within position packets using the new Base91 comment telemetry format, which is quite tightly packed, and also includes a sequence number in a range of 0...8280. This makes it really easy to detect a packet having an older sequence number than the previously received packet. There are already a lot of those devices in use.

aprs.fi has now been updated to make use of the sequence number when available. It's more reliable and faster than the old methods of detecting too high speed or duplicate packet content. Due to the amount of broken igates and digipeaters delaying packets for minutes or tens of minutes (often due to a buggy Kantronics KPC3+ in KISS mode) it can make sense for a tracker to transmit the sequence number alone without any actual telemetry! Packets dropped due to this will be shown with this error message:

Delayed or out-of-order packet (sequence number)

In practice it will usually be shown together with the error message about dropped telemetry, since the telemetry content is also ignored due to the duplicate sequence number: 

2012-12-31 00:24:51 EET: N0CALL-9>SY0UWY,WIDE1-1,WIDE2*,qAR,N00CALL:
`pOnqgd>/'"KQ}MT-RTG|!D&='a|!w;a!|3
[Duplicate telemetry sequence, Delayed or out-of-order packet (sequence number)]

You can use the Decoded mode of the raw packets view to see sequence numbers and telemetry values decoded from each packet.

Time zone selection on aprs.fi

Most of yesterday I spent on implementing proper time zone selection on aprs.fi. You can now select whichever local time zone you wish using a (hopefully) user friendly map tool.

Click on Preferences (on the right side of the real-time map), select the Units and time tab, and click the Change button next to the currently selected time zone. That'll bring up the new time zone selection tool.

Point the mouse roughly at your location (the state or country which uses your preferred time zone), click, click Save to return to the main Preferences view, and click Save again.

You can also select either the time zone or country from the drop-down menus. Or, if browser geolocation works for you, click the I'm Feeling Lucky button. It's not very accurate (I'm being placed in Moscow for example), but it probably works for more than 90% of the population.

I didn't do all of this from scratch - I copy with pride. Thanks go to these two excellent open source projects: timezonepicker  and TimezoneJS.Date

aprs.fi upgraded: track tail length adjustable

Kukkatarhan Cissus sings:
"We wish you a merry Christmas!"

I upgraded the aprs.fi service yesterday evening. I rebooted both main servers after upgrading operating system and database components. After that I upgraded the aprs.fi software itself.

It's been a long while since the last significant upgrade. There have been small patches and quiet bug fixes here and there. My excuse is the time I've put into completing and deploying aprsc, a new open-source APRS-IS software written in C together with OH2MQK. It's now in use on about 1/3rd of all APRS-IS servers and appears to work fine.

There is now a separate track tail length drop-down selector on the right side of the map. It's now possible to select "show last positions of all stations in the are for the past 24 hours" but still only show a track tail for the past 30 minutes, for example. Drawing tracks for 24 hours tends to clutter the map a lot and it can also be very slow on many devices.

The old wildcards-on-initial-navigation bug is finally fixed. If you go to http://aprs.fi/OH7* (use a wildcard character, or otherwise multiple stations come up from the initial search) clicking on the callsigns within the popup did not work. Fixed!

Fixed exporting of stations which have transmitted telemetry.

Lots of improvements on Tetra protocol support and closed enterprise service model (not visible on aprs.fi).

Improvements in command line tools for service maintenance.

Small performance improvements here and there.

I also upgraded the operating system and database (the usual security fixes and some small performance improvements).

We've got new kittens too. Been taking a lot of kitten photos!

Power meter monitoring using Munin & an Arduino

It's been a bit quiet on the blog lately, but that's just because I've been a bit busy with both aprs.fi and other related projects. I've been writing the new aprsc APRS-IS server software together with OH2MQK, went to DCC 2012 in Atlanta to do presentations about both aprsc and aprs.fi, aprsc has been deployed to about 30% of APRS-IS servers and I've had to fix some bugs too. I'll tell you more about those things later – this blog post is about one of the "other projects", a one-nighter to monitor the electric power consumption of your home using an Arduino, a phototransistor, and Munin. I have to admit it took another day to package and document it for you, and write this blog post, and that it can take another night to adjust the physical installation of the sensors.

The Arduino C code, the two Perl scripts used on the Linux box to get the numbers to Munin, and a README text file with more instructions is available on Github. Click on the ZIP button to download it all.

Here's an example diagram drawn by Munin, with annotations for some events we triggered. Munin might not be the best way to collect and report measurements like this, but it's certainly one of the easiest methods to get data graphed on the web using a Linux box. apt-get install munin munin-node and you're ready to go!

The local power company remotely measures power usage in 1-hour steps (using a GSM/3G connection), lets me view that data on their web site (after logging in, of course), and advertises that as being "very accurate" and "high resolution". I don't think it's much to brag about! Munin polls every 5 minutes which makes it really easy to see how much each device or action consumed energy.

Most modern power meters installed by the utility companies have a red LED which blinks according to the power consumption. The meters installed in our small apartment building blink 1000 times per kWh (kilowatthour) – it reads "1000 imp/kWh" next to the led.

To count the blinks you'll need a photodiode or phototransistor. I used an Osram SFH 300 sold by the local electronics store (Partco). It's sensitive to visible light (not just infrared, like some models).

I'm measuring consumption of 4 apartments (with the consent and for the benefit of my neighbors) and the building's consumption (heating system, outdoor & basement lights, etc), so I attached 5 phototransistors.

I used an Arduino Duemilanove, but it's current replacement model Arduino Uno or just about any other model will work just fine.

The Atmel microcontroller has internal pull-up resistors, so the phototransistors can be connected directly to the board without any other components. The Uno board costs about 20€ or $25 USD, has USB and a good set of analog and digital I/O pins. It's powered by USB – no wall wart required.

It's very easy to program in C using the Arduino IDE, which comes with plenty of simple example code. If you're a programmer with no electronics experience, or an electronics guy with no programming experience, or something in between, you'll love the Arduino.

Here's the box attached to the wall using 3M Dual Lock tape.

Since we measure the blinks using optical coupling – there is no electrical connection between the meter and the photodiodes – installing the instrumentation is completely safe and legal. It's about as safe as taking photos of the meter, which is very safe indeed.

With the small bias voltage given by Arduino's internal pull-ups, the photodiodes are a bit insensitive, so they need to be aligned very closely and sharply at the blinking red led of the meter.

Remember that it might not be wise to publish real-time measurement data on the Internet, since everyone could see when you're home. But that applies to APRS on your car, too.

My meter actually has two blinking leds. The one on the left is for kiloVoltAmpsReactivehours (kvarh), which indicates phase-lagged power consumed by an inductive load. My power company isn't yet charging me for that separately, so I'm not measuring it.

Some aprs.fi email lost during the past weeks

Oops! For a couple weeks about 50% of the emails sent by aprs.fi have been going to the bit bucket due to a broken configuration on one of the servers. These included sign-up email confirmation and password recovery emails. Sorry about that!

The configuration is now fixed, and I've manually triggered a retransmit of the signup emails for the users who've signed up during the time but haven't yet clicked on the confirmation link.

I didn't want to be an engineer.

From the what-else-keeps-me-busy & off-topic department: It was a bit windy in Finland after christmas. So, this is what I found waiting at our summer cottage:

I didn't want to be an engineer. I wanted to be... a lumberjack!

I'm a lumberjack, and I'm okay, I sleep all night and I work all day.

I cut down trees, I eat my lunch, I go to the lavatory.

On Wednesdays I go shoppin'

and have buttered scones for tea.

Luckily it didn't fall the other way, across the road and over the power line.

I couldn't help it - the lumberjack song played on repeat in my ears most of the day.

Next task: manually converting the wood to smaller bits and pieces to make them fit in the stove.

Upgrade: New server, Dead Reckoning, and some smaller stuff

One of aprs.fi's servers getting new disks

First I'd like to thank everyone who showed up in Ham Radio 2012, Friedrichshafen! It was really nice to meet you all.

aprs.fi's second server migration reached completion today. It's now running on two blade servers, each having two quad-core Xeon processors with 12MB cache each, for a total of 8 CPU cores at 3 GHz and 32G of RAM per server. That's a total of 16 CPU cores and 64G of RAM for aprs.fi alone! The memory really helps, as I can now fit more and more stuff in memory and avoid slow hard disk seeks. We found the blade server in a dumpster, so it's not the latest and greatest in the market, but certainly useful for a few more years.

I also upgraded the aprs.fi server software with a few visible changes and several important features which are not directly visible to the end users.

The most visible change is that Dead Reckoning was enabled for all stations which have moved recently. If the station has moved within the past 30 minutes, a blue line will be displayed, indicating where the station would be right now, assuming it has continued on the same course and speed.

For stations which transmit quite often in relation to their speed, or do not turn quickly (ships, airplanes and high-altitude balloons, for example), the DR'ed position will be surprisingly accurate. For cars driving a very curved road (in the city) it will be less accurate, but the DR line still provides an indication on the relative age, speed and usefulness of the displayed position. Your internal non-artificial algorithm can easily figure that the car probably turned along the road, even if the blue line ends in a forest.

The blue line becomes gradually more translucent during the first 10 minutes after the reception of the position report. If it's almost completely translucent, the position is more than 10 minutes old and both the DR'ed position and the displayed old position are pretty outdated. This information can be very useful, too.

It can take a small while until everyone gets used to the DR lines. After having them for a couple months on my development server I can assure you that they really improve the usefulness of the real-time map view!

Ham::APRS::DeviceID module was upgraded to version 1.05, adding detection for a number of new APRS devices: TrackPoint, BPQ32, ircDDB Gateway, DIXPRS, dsDIGI, dsTracker, DireWolf, MiniGate, YAAC, and MotoTRBO. New version of the module will appear on the CPAN soon.


Fixed calendar date selection in the data export tool.


Implemented a nice user and team management web UI (for administrator use only). Especially useful when running the software in a closed "intranet" mode.


Gave a face lift for the service management command line tools.


Web server software was upgraded (as usual).


I've also implemented TETRA LIP position packet decoding and support for Google Maps Enterprise licensing. More on that later!

aprs.fi presentation in Friedrichshafen, Ham Radio 2012

I'll be giving a lecture titled Advanced tricks and new features of aprs.fi at Ham Radio 2012, Friedrichshafen.

In this session I will demonstrate usage of advanced aprs.fi features beyond basic callsign lookup. New features and plans for the future will be announced.

The session will end with a Questions & Answers section and discussion. There should be plenty of time for that.

Time: Saturday, 23 June 2012, 14:00, about 1 hour + discussion
Place: Room/Raum Paris

Beware, it's a 15-minute walk from the center of the fun. The place is huge, so distances are noticeable.

It's not listed in the official program yet, since I got the idea of doing a presentation a bit late and the program had already been sent to be printed, but the organizers managed to find a room for us. Thank you Thilo! Since it's not in the official program, please let other APRS folks know about it when you bump into them on Friday. Thanks!

aprs.fi connected directly to aprs2.net hubs

I'm happy to announce that aprs.fi has tonight been directly connected to all of the APRS Tier 2 Network hubs. Two frontend servers, APRSFI-C1 and APRSFI-C2, are now connected with read-only connections to the five T2 hubs (T2HUB1 to T2HUB5), which act as the backbone of the aprs2.net network. This should provide a very stable and trouble-free connection between aprs.fi and the APRS-IS network.

Before this change aprs.fi was connected to a single APRS-IS server (usually T2FINLAND) and collected packets from that server alone. That server, in turn, is connected to a single T2 hub. Sometimes that connection could have some trouble and be disconnected for a few minutes, causing some packets to be lost. Recently a misconfiguration within the T2 network caused intermittent but severe packet loss for a few users for a long time. Having redundant, parallel connections to all of the servers should provide aprs.fi with copies of all packets even if some parts of the network have issues.

The fact that aprs.fi was usually connected to T2FINLAND also caused many users to prefer that server, which in turn caused a high load to the single server. Also, if that server would have had a hardware failure, all of those users would have lost connectivity to the APRS-IS. aprs.fi would have automatically switched to another server. Right now, T2FINLAND has 210 clients connected, while most other servers only have 30 to 100 clients. The Tier 2 network currently has a total of 4005 clients connected to the 86 servers in 31 countries.

If you're using T2FINLAND (finland.aprs2.net), or if you have configured your server to connect to any other single server, please reconfigure your system to use one of these Regional Rotate Addresses:

Europe and Africa: euro.aprs2.net
Asia: asia.aprs2.net
North America: noam.aprs2.net
South America: soam.aprs2.net
Oceania: aunz.aprs2.net

Check out the map of T2 servers and the rotate address distribution on the aprs2.net home page!

All of the regional rotate addresses will make your client connect to one of the nearly servers which have recently been automatically tested to be available and working. When that server fails, your client will automatically connect to some other server. All of the servers will provide equally good connectivity to aprs.fi. Starting tonight, T2FINLAND is not better in that respect than any of the others.

I repeat: Do not connect to finland.aprs2.net. If you're in europe, use euro.aprs2.net instead. T2FINLAND's server hardware will eventually break (we found it in the dumpster), and your igate or client software might be disconnected for a long time until someone gets to fix the server. Unless, of course, you use a regional rotate address, in which case you'll be automatically rerouted to a working server.

Thanks to all the Tier 2 operators for making this possible!

Linking parameters fixed and proper "Show all" button

The "others" and "timerange" parameters described on the Linking page should now work again. They were broken since the anchor navigation upgrade. Thanks to Oba JA7UDE for the bug report!

To properly fix how the "others" parameter  works I also added a new "Show all" button which is now quite a bit more  sensible than the old "Show all" link:

• The button is disabled and greyed out when it really doesn't do anything (like, when you're not tracking a station and all stations are displayed anyway)
• "Show all" can now be toggled between on and off, not just on
• The button visually indicates whether it's on or off (depressed or not)

Real-time telemetry and graph value lookup

I just finished installing an upgrade on aprs.fi. I had forgotten a configuration change that needed to be done with this upgrade, and the web service stopped working at 19:16 UTC. I had some trouble finding the problem, and managed to fix it at 19:38 UTC. That was completely unnecessary, sorry for the trouble. It should have been a routine upgrade requiring only a minute of downtime or so. Data collection was not interrupted.

The upgrade made the telemetry graph page update itself automatically as new values come in from the station, just like the weather page does. Just look up a telemetry station and leave the graph page open and the contents will be magically updated!

The grapher engine got a little upgrade which enables graph value lookups on all graphs. Just hover the mouse cursor above a graph and the labels will display the values reported at that time. While the pointer is between reported values it will display an interpolated value.

The real-time map's data refreshing algorithm got some updates and fixes.

I did a few updates on strings too, and the translations need to be updated. At least on the moving stations page I had to split a very long string to two short ones.

Telemetry display improvements

After attaching some 1wire temperature sensors to the USB port of my D-Link DIR-825 WLAN router running OpenWRT, and making aprx transmit the temperatures in the new base91 comment telemetry format, I felt like improving the telemetry display on aprs.fi a bit.

Current telemetry values are now shown in the info balloon of a station on the real-time map.

I always thought that many telemetry transmitters are only utilizing a few channels, and the other channels should not be shown. So here goes, now you can configure which channels are shown. If any channel names have been sent using the "PARM." message, only those channels (and bits) are displayed in the current telemetry values summary. If no names have been sent, all 5 analog channels and 8 bits are shown.

The telemetry coefficients transmitted with the EQNS. message now stores 5 decimal digits instead of 3. Base91 comment telemetry supports 13 bits of resolution (values 0 to 8280), which often needs to be scaled down to a range such as -55...+125C for temperature. More decimals in the multiplier and exponent parts help in making it accurate.

Dead reckoning and radio horizon circles

This large passenger boat is actually
making some turns between the islands.

Today aprs.fi got a simple implementation of dead reckoning. Moving stations which have announced their position, speed and course within the past 30 minutes will show a blue line extending from the latest announced position. The line indicates where the station would be right now if it proceeded on that course at exactly the same speed.

To limit the performance impact of this feature the DR indicator will be shown for a single station at a time. Click on a station to enable it. It's also enabled automatically for a tracked station.

Naturally it'll show some rather bogus estimations for vehicles which have stopped moving but have not transmitted a new packet with a speed of 0. Most trackers installed in cars loose their power input before they have a chance of transmitting a packet at the parking spot.

A plane flying at 2648 feet, and the digipeaters
and igates which have received it.

Flying stations having a "flying" symbol such as airplane or balloon will have two circles drawn around them. A larger blue circle indicates the sea-level horizon from the current altitude of the station. A smaller green circle indicates the area where the station can be seen at an elevation larger than 5 degrees.

The circles are useful for estimating the distance at which the station can be heard, which can be of use for tracking teams of high-altitude balloons. It also gives pilots an idea of the large amount of digipeaters and igates in range.

The circles are drawn for stations having a suitable symbol when the altitude is over 100 meters. Other stations will also get the circles when their altitude reaches 1000 meters.

aprs.fi upgrade on Monday 2012-03-19

aprs.fi will be upgraded on Monday 2012-03-19. Some downtime should be expected, starting around 9 AM local time (0700 UTC). If you can find some bugs in beta.aprs.fi, this is the time to report and get them fixed before they make their way to aprs.fi! Here's the long list of changes included in the upgrade. It's so long that, as usual, some new bugs will probably pop up. Please report them on the discussion group and I'll try to get them fixed right away.

Anchor navigation and more AJAX

Achilles (left) and Ajax (right) play a board game with
knucklebones on this late 6th-century lekythos, a type of
oil-storing vessel associated with funeral rites. Photo taken
in Musée du Louvre by Marie-Lan Nguyen, 2011
(CC by attribution, Wikipedia/Wikimedia).

The technical implementation of the navigation in the real-time map has been completely changed. In the new model the map page is not reloaded from the servers and initialized again from scratch every time an user changes the view by searching for a new callsign or an address. This AJAX magic will make callsign searches and other view modifications considerably faster, especially on slow computers and slow connections such as mobile devices. It'll also reduce load on my and Google's servers. And make the browser's bookmark functions and the back button work better than before.

All of the links used by aprs.fi to refer to specific map views have changed, but the traditional ones documented on the linking instructions page will keep working, and I encourage you to use them as before - they're not going anywhere.

Try searching for OH2K and OH2TI one by one again and again, and you'll notice both the improved loading speed, and a nice panning effect. It'll pan whenever the newly searched station is close enough to the current view.

Improved address search

Address search has been improved to make a better use of the data returned by Google's API. A marker for the result is now only shown if the result is accurate (such as a street, or a house number on a specific street).

Address search will also adjust the zoom level - "Finland" or "Pohjois-Karjala" or "USA" should actually fit the specified region in view. The zoom levels come from Google and I can not make it any smarter than that (yes, "USA" will zoom out a bit too much). But it's certainly better than it was (fixed zoom level after address search).

Improved response times

When you make changes to the real-time map view, for example by zooming out or selecting a time range, aprs.fi will now react much more quickly.

New time range back/forward buttons

When a station is tracked, there are now two new handy additional arrow buttons to jump the time range back and forward. If you have selected an arbitrary range, it will jump the same amount back (select a week, and it'll jump to the previous week). Otherwise it'll select the current whole day (UTC 00:00:00 to 23:59:59 - sorry, no local time support yet) on the first click, and the following clicks will jump by 24 hours.

Sharper map graphics on iPad/iPhone

The scaling of the web page was tuned to switch automatically so that map graphics are not blurred on the iOS devices, especially when the device switches between landscape and portrait orientation.

Other small things

Searching for "OH2RDK" will now give a proper "there are other SSIDs available although this one doesn't exist" response.

aprs.fi now uses a new version of the Google Maps API, bringing in some visual updates and speedups from Google.

An old bug which makes stations disappear after a quick zoom-in-and-back-out operation (and some other cases) has been fixed.

The "street view is off by 200 meters" bug is fixed. Also, the map should stay centered when Street View is enabled or disabled.

The ruler tool now displays distances shorter than 1 km in meters to allow measuring short distances.

When a station is tracked on a mobile device, the info balloon does not automatically pop up and block the whole view.

Raw packets view in decoded mode displays the position packet's type (compressed, mic-e...).

Tuned digipeater/igate "heard" map to collect more data and leave less gaps in the map.

Support for new major version of the database server (SQL syntax changes).

CSRF security fixes were implemented in many places.

Some rough corners have been rounded up (literally), and a few shadows have been cast (again, literally).

Does aprs.fi feel slow?

Be sure to try it with a modern, quick browser such as Google Chrome, Safari or Internet Explorer 9. If you're upgrading from Firefox or an older Internet Explorer, you'll be surprised by the difference it makes.

Data collection outage on Sunday, 2011-01-29

aprs.fi data collection was down today (2011-01-29) between 14.20 and 14.55 UTC due to a hardware failure. After finishing my dinner I moved the master service to another server and things appear to be working properly now.


Sorry for the trouble!

aprs.fi closed in U.S. on Wednesday

aprs.fi will join Wikipedia and Reddit, and protest the proposed U.S. SOPA/PROTECT-IP legislation by closing down on Wednesday. (News about Reddit blackout, and Wikipedia joining it.)

The aprs.fi outage will only affect clients in the United States (or those with IP address mapping an U.S. network operator - the targeting is not fully accurate).

Although the law is being made in the U.S., it will break the Internet on a global scale by making sites such as aprs.fi liable for links and content posted by the users of the site. Sites like aprs.fi are commonly run by individual developers or small volunteer teams. Due to the huge volume of automatically published user-generated content (50 packets per second currently!) it would be impossible for me to go through it all before publishing. If some APRS user would post links to copyright-infringing material, even when that material would reside somewhere else than aprs.fi itself, aprs.fi could be shut down in the U.S. and there would not be much that I could do about it. See how difficult it was for a falsely censored music blog to get unblocked under the current legislation.

The law claims to be targeted at pirate web sites, but it won't have any practical effect on criminal file sharing, since those networks can very easily switch to new domain names and IP addresses, or bypass the DNS altogether with modern peer-to-peer technologies. Instead, it will force web site administrators such as myself to pre-censor data (by, for example, removing user-posted links completely). In practice: no home page link shown with your APRS station on aprs.fi. This is just silly – on other sites which depend more on linking out it could ruin the whole show.

If you're an U.S. citizen, you can probably do something about this that would actually make a difference. I'm not there, so I can't (but I voted yesterday evening in Finland's presidential election!). Please read through the material on the Reddit blackout page, there are good "Learn More" and "Get Involved" sections in the end!

Data collection will be running as usual, so Wednesday's data will be available on Thursday.

Now I'm really happy that aprs.fi is aprs.fi instead of aprs.com or aprs.net. And that I'm not living in the UK. The really bad news is that similar laws are being pushed in Europe.

PS. You can get your APRS location from http://www.findu.com/cgi-bin/find.cgi?YOURCALL or DB0ANF. If a life-threatening disaster would happen in the U.S. let me know and I'll open up the site.