Thursday, August 15, 2019

RX-only igates considered beneficial to the network

As probably most of you know by now, besides running the aprs.fi web site, I'm also one of the two main authors of the aprsc server software. aprsc runs on most of the APRS-IS servers where iGates usually connect.

There have been recent and strong claims saying that receive-only (rx-only) iGates destroy the two-way messaging feature of APRS. This has been claimed in blog posts and facebook threads. Some people ask me if this is true.

No, it is not true. Receive-only iGates do not break messaging if there are transmit-capable igates nearby, and those transmit-capable iGates are connected to an APRS-IS server which has a full APRS feed. All aprs.net and aprs2.net servers, where iGates normally connect, do have a full feed. No problem!

Messaging would work from 1650m / 5400 ft above Vihti, even in the presence of
RX-only iGates, as long as there is at least one TX-capable iGate.

If there are no transmit-capable (TX) iGates around, two-way messaging will not work, of course. Having a transmit-capable iGate would therefore be better, but receive-only iGates are easier to set up technically, they are cheaper (receivers are practically free now), and licensing for automatic transmitters is difficult in many countries. Where transmit-capable (TX) iGates are present, receive-only iGates do not break the TX iGates, they just improve reception coverage. For messages, too.

Just to make it perfectly clear: If there are TX iGates present, additional RX-only iGates improve messaging performance. For the RF-to-IS direction (and ACKs for the IS-to-RF messages).

The common incorrect claim is that the APRS-IS server sends the message only to the latest iGate which heard a station. In fact, the APRS-IS servers (both javaprssrvr and aprsc) send the messages to all iGates which heard the station recently. In aprsc, "recently" means "within 3 hours", and I believe javaprssrvr uses something similar.

The server maintains a list of recently heard callsigns independently for each iGate client. There's a separate list of heard stations for each iGate client. When the server has a packet to pass on, it will look at all connected clients, and for each client, if the recipient of the message is found from the list of recently heard stations, the message will be sent. The scanning will not stop; the message will be given to all clients which heard the station.

I can confirm that we have written the software to do it like this, and since it is open source, you can see the code yourself. The automatic test case also validates that the server keeps working that way, so that there won't be a bug creeping in the future to break it accidentally. I've also tested that javaprssrvr behaves like this – I've run the test cases against it to confirm compatibility.

There may be problems when there is a server with a filtered feed involved (possibly a server software running at the "client" without having a full feed), but those are rare and known to be problematic. All Tier2 servers have a full feed (aprs2.net ones), and so do the core servers.

The real problem is that a user, seeing one-way beaconing to the APRS-IS works, may well expect two-way messaging to work too. And when it doesn't work, there's only a timeout, no immediate feedback saying "sorry, this won't work now", which is what a sensible system would do today.

Even in that case, an rx-only igate is better than nothing! I wouldn't be so harsh against them, since the step from RX-only to TX-capable may be a bit difficult for many.

Bottom line:

In each area, there should be one TX igate, maybe two. More may create QRM as the same messages will be transmitted from APRS-IS to RF many times. RX igates will not break messaging if there are TX igates around – they just improve RX coverage.

This picture may at first seem irrelevant, but it does show me working VHF in AM (122.825 MHz),
and Flarm on 868 MHz to OGN (which runs aprsc). Flarm antenna visible in top left corner.

Sunday, April 21, 2019

Server upgrades, and raw packets UI abuse

This month I'm upgrading the aprs.fi servers quite a bit. The old blade server hardware chassis is getting shut down and replaced with a brand new pre-owned and dumpster-dived blade server chassis. The new second-hand blades have 8 CPU cores and 192 GB memory each, which is quite sweet! I suspect the web site & database will be doing very little disk reads with this much cache around.

The downside is that we're not going to run the fibre channel SAN network, and the associated disk racks, since they're a bit power-hungry. So, internal disks only it is. To get relatively good disk IO performance I installed 1 TB SSD disks in the blades – two SSDs in each for RAID1 (~170€ with 24% VAT each, 3 blades, total 6 SSDs). The blades have CERC 6/i raid controllers, which are SATA2 and quite slow. The Samsung SSDs are capable of some 500+ Mbytes/sec reads, but the hardware controller in RAID1 config only gives out 130 Mbytes/sec. With Linux software RAID1 it can balance reads across the SSDs and give out 2*140 Mbytes/sec, which is alright.

aprs.fi being a database application, especially with the huge cache memory, it'll be mostly random-access write heavy, so the streaming read benchmark above is not too relevant. The random-access speed of the SSD gives a nice boost, and the memory will help an awful lot.

The new servers are already running a live replica of the database, but the web service is still running on the old ones. Hoping to get the move done soon. The operating system is also getting a bump to the next major release, and some adjustments are needed to support that.

In less happy news: A few smart folks have, again, figured that it would be a great idea to download raw APRS-IS packets programmatically (using software they've written) from the aprs.fi user interface, the raw packets view. This is forbidden by the TOS (https://aprs.fi/page/tos) and a longer reasoning can be found in the FAQ (https://aprs.fi/page/faq): 
Downloading data for application use using the user interface (for example, fetching the /info/ page just to get the current coordinates of a station) wastes both human and computer resources.  
First, you need to write a parser to extract the data from the HTML (and then fix your parser every time I change my HTML layout). Second, aprs.fi needs to do the user interface magic (language and timezone selection, user-friendly template formatting, session set-up) for every request, all of which is unnecessary.  
[...] All of this extra overhead consumes CPU time, which in turn heats up the computer room, consumes electricity, destroys tropical rain forests, accelerates global warming, and kills kittens. And baby seals.
The correct way to obtain a raw APRS-IS feed is to connect to the APRS-IS  (http://aprs-is.net/Connecting.aspx) - you'll get a lightweight TCP  session with all the packets in real time. As clean CRLF-delimited rows with none of my HTML encoding to mess it up. For free, from the very same place where I get them from, and much faster! 

If one chooses to fetch the packets by hammering on my user interface, as opposed to an API, it'll create unnecessary extra load on my servers, and make me a little bit upset. Especially when someone fetches 1000 packets every time, via the Tor network, repeatedly every few seconds, and ends up fetching the same packets over and over again. That is not a great way to make use of this free service.

To make this sort of abuse harder, you'll now need to log in to view the raw packets. It's a little bit more clumsy if you're not logged in already, but the login cookies have a long lifetime so you don't need to do it that often.

Monday, December 10, 2018

Subscription for extra features within aprs.fi iOS app

This little warning has been in the aprs.fi iOS help text from the first version of the app - my plan from the beginning has been that some functions will require an additional paid subscription. The time for this is coming soon, and I've lately been spending some time implementing those features. This is likely going to upset some people ("what, I already paid for this app, I need to pay more?!?") so it's probably good to go through the reasons once more.

There is a fairly small number of amateur radio APRS users which are using iPhones or iPads. Androids are quite a bit more popular amongst hams. It will only take a short while until everyone, who ever will buy the app, has done it. After that nobody will pay for the application any more.

The work will not stop there: the application and the backend servers supporting it need to be maintained, upgraded and improved over time – if the application does not get some love and care from its creators at regular intervals, it will soon decay and eventually stop working. This has happened to a few ham and APRS apps already, and I'm not at all surprised about that. Even if the application does not get any new features, someone must keep it going, add support for new iOS versions and iPhone models, fix backend servers, and so on.

This maintenance burden may be difficult to appreciate if you haven't tried doing it yourself. When there's a new iOS version, you need new Xcode to build the app for it, new Xcode requires new OS X version on the laptop, and the app's code will need modifications to compile on the new Xcode due to changes in the programming language and APIs (programming interfaces provided by the operating system). And then there are the backend servers (don't get me started about "serverless").

"What? We need to pay more?!? Greedy bastards!"
Photo of Finnish native "kyyttö" breed of cows (c) myself
2016-06-16 
at Sappion luomutila
I do quite some volunteer work as a hobby to support the APRS system (in the form of running aprs.fi itself, contributing free open source software such as aprsc and the aprs2.net backend software, open source APRS symbol graphics, machine-readable APRS device identification database, and a few other bits). I'm already quite busy with everything, and the family, so I decided to make a little bit of money from the iOS app. Due to the relatively small amount of users, it'll probably never pay for all the effort needed (pay-off not really comparable to the daytime job), but at least something. And, to keep the motivation there in the future, it would be nice to have something that is called a sustainable business model - in other words, a little bit of money keeps flowing in the future, too. That's where a subscription payment model comes in handy!

How will it work?


The model will be very simple: There's only a single subscription option, which will enable a set of extra features. Whenever a new feature is added, it may go to the "base" set of features, which are available to all users which have purchased the app. Or, it may go to the "extra" set of features, which are available with the "extra" subscription. If you have the "extra" subscription, you'll have all the features added in the future – there won't be another "pro", "enterprise", or "super" subscription required for those. No per-feature pricing, just the "base" set and the full "extra" set.

Some features going to the "extra" set will seem fairly basic to some, and they will be features that most users hopefully want to have: Messaging, beaconing to APRS-IS, iGate, long and arbitrary time range lookups, soundcard modem, etc. On the other hand, the pricing will be fairly cheap, in my opinion, when compared to the price of an iPhone or any piece of APRS hardware: less than the app purchase price per year, resulting in roughly 50 cents / month price ($0.50 USD / € 0.50). About the same price as a Panini at Starbucks, a bit more than large Caffe Mocha! Subscription period will be 6 or 12 months, and there will be a free trial period in the beginning so that you can try before you pay. I may adjust the price later to one direction, or the other, depending on how it sells.

I hope most users will want the extra set, so that even with this low subscription price I'll still get something. Apple grabs 30% of that for the first year of subscription (like it does for app purchases), and then the taxman grabs another 30-40% of what's left. You pay $6, I get less than $3, and I can buy a couple litres of milk for example. After two years I'd have enough to buy a beer in the pub. Yay.

Some folks will likely ask for a lump sum "lifetime" option for the extra features. It's not good for the sustainable business model – the purchases would end soon, at which point there is no income. Also, it's difficult for anyone to promise that it will actually work for a "lifetime", due to the need for regular maintenance and support from the developer.

When comparing prices of applications one should remember the size of the target audience. Creating and maintaining an application takes roughly as much time for 10, 1000 or 100000 users. This is why some really nice and big apps for the general public can be really cheap, or financed by in-app advertisements: 1€ per user for 100000 users would already make 100000€. This makes apps for smaller audiences (hams with iPhones and interest in APRS) relatively more expensive.

To re-iterate:

  • The subscription price will be small (roughly around 50 cents / month), paid through in-app payment.
  • There will be a free trial period in the beginning of the subscription, so you'll know what you'll get. Just remember to cancel it before the end of the trial; this is how Apple's subscription trials work.
  • It will enable all new features in the future.
  • Current features will NOT stop working or require subscription payment – the subscription is optional and you get to keep what you paid for earlier.
  • Some future new features will also be included the base feature set, not requiring subscription.

Tuesday, May 9, 2017

Important Hints to iGate developers

By connorgoodwolf - Own work, CC BY-SA 4.0
There are a lot of APRS iGate implementations out there, and new ones are popping up every few months these days. A lot of the new ones are made for various microcontroller chips and development boards (Arduino, ESP8266 and friends). Some are made for SDR sticks, some are built in mobile phone applications. Many of them reduce the amount of hardware needed to run an iGate by fully removing the need for a TNC, a full-blown computer, or an actual amateur radio transceiver.

Very cool stuff!


It would seem like writing an iGate would be very easy: receive packets off the air, put them in TNC2 format, and pass them to the APRS-IS servers over TCP (specifications: connecting, design, details). But it turns out that there are quite a few things that can go wrong in the implementation, and in fact, some things seem to be going wrong the same way fairly often. Some bugs are fairly common and some seem to pop up in multiple different iGates, old and new.

Currently the most popular ones are due to the false assumption that all packets are either plain 7-bit English ASCII text with no NUL bytes or other binary stuff, or some specific Unicode encoding such as UTF-8. These assumptions lead to the packets getting truncated at the first NUL (binary 0) byte, or otherwise modified in a way that scändic and øthér ïnternátiønal characters get mutilated if they don't happen to use the same encoding. Sometimes a truncation of a packet at the first NUL may lead to the packet being concatenated together with the next packet.

I get to decode the corrupted packets on aprs.fi, and I also get to process the corrupted ones in the aprsc APRS-IS server software which is fairly common these days. If the bugged packets are not detected and handled, there usually are some folks complaining to me about the odd results, since they are often received from aprsc and viewed on aprs.fi.

I can't blame the authors of the new applications – I have to admit that APRS documentation is not great, and it's easy to stumble across the same bugs if you haven't fixed them once before.

Every now and then I spend an evening trying to identify some of these bugs in the APRS-IS client applications and file bug tickets to the software authors. Last night I filed 3 packet corruption bug reports on Github for 3 different iGate applications. These are not the first ones, I've probably filed 30 in the past. It's not hard to spot these once you know what to look for.

I'd like to take this chance to thank to all the authors who fix the bugs in a timely manner and release the fixed versions!

Based on this experience I wrote a little document listing common iGate bugs: IGATE-HINTS.md. It'd be great if all APRS software authors could glance through it and make sure these bugs are not present in their own software. If you spot someone writing a new iGate, please pass on the link to the new author – hopefully these bugs will then be avoided right from the beginning.

Unfortunately some buggy iGate applications are orphaned and no longer developed. A few of those applications are fairly popular, too. It'd be great for the network if iGate operators could gradually upgrade to applications which still get updates, and don't exhibit any of the packet-corrupting bugs.

A lot of iGates are also not getting updated, even though they run software which would have updates available. Some iGate operators feel that the iGates work, sort of, since most packets go through them just fine, most of the time. I think I'll try to make aprs.fi highlight some of the more common issues on the station info page in the future.

If you know a little Python or C, and you've got a little extra time, let me know – there are some bugs out there in the Open Source Land which could use a helping hand.

Saturday, March 25, 2017

New version of aprs.fi's APRS packet parser released: Ham::APRS::FAP v1.21

Yesterday I uploaded a new version of the APRS packet parser used by aprs.fi, Ham::APRS::FAP, to the CPAN. It's open source, so if you're a little bit weird like me and enjoy programming in Perl, this might be useful. Here's a full list of changes:
  • Improve make_position() to support HMS UTC timestamp. make_position() now returns the packet type character so that it can signal the presence of a timestamp.
  • Improve make_position() to support comment string, !DAO! extension, altitude encoding. Fix rounding errors in lat/lon/speed. Support generating packets with no speed or course. Take optional parameters in a hash parameter. Implement unit tests for make_position().
  • Set up Travis automatic running of the (existing) unit test runs.
  • Additional character escaping in regular expressions to deal with deprecated functionality in Perl 5.22.
The packet parser is one of the more complete APRS decoders, although I'm not sure if any of the parsers deal with all of the APRS features and packet variants yet.

Monday, December 12, 2016

aprs.fi moving to TLS

In an effort to increase security on the web at large scale, web browser vendors and other organisations such as Google are making changes which encourage web sites to move to TLS/SSL encryption. Even web sites which previously did not seem to need it – ones with static content only, and ones without any login / password functionality. This is good and fine – even if it's not a banking web site, it's good that third parties along the network can not observe or modify the content being downloaded. The Chrome web browser has started to label non-encrypted sites with an informative '(i)' symbol which warns the user that "Your connection to this site is not private", and will eventually make those warnings stronger. Google gives better ranking in the search results for https sites.

A real, practical issue right now is that the geolocation Javascript API is no longer available on non-HTTPS sites in recent Android and Chrome versions. This actually broke map center and tracking functionality on the aprs.fi web site.

I wholeheartedly support this movement, it will make the Internet a better place!

These days, with performance-improving developments such as ECDHE, GCM mode AES and hardware accelerated AES, running TLS on a web server is not much of a performance issue any more. Most of the CPU time will be spent on application logic, anyway.

The fun part is that HTTP/2, a new protocol used by modern web browser to access web sites, is only used over TLS/HTTPS – it is not available over plaintext connections. HTTP/2 is faster than older HTTP versions, and a surprising side effect is that a web site may well open up faster over HTTP/2 + TLS than over HTTP 1.1 without the encryption!

Picture not related. I just took it last summer. Kyyttö cows © Sappion luomu.
Before now, aprs.fi has only used TLS/HTTPS for its login and user account management pages. Fairly soon I will have a maintenance break on the aprs.fi servers, upgrade the operating system to the next major release, and install a new version of the aprs.fi software which supports access over both HTTP and HTTPS. To reduce duplicate content (same stuff being available over both HTTP and HTTPS) it will prefer HTTPS and nudge clients that way every now and then, but initially plaintext access should be possible, too. Later on, if there are no surprises, the nudges will gradually become stronger.

There are a few issues which need to be addressed. There are possibly a few Amprnet users accessing this site over amateur radio frequencies. On the other hand, they're then practically surfing the Internet over radio, and probably doing a few requests to other encrypted sites now and then, too, so maybe it's not a big problem for them.

Another thing is that apparently users in China can't access the Google Maps API over HTTPS, so those users would still need the plaintext access for now. I might make the zh.aprs.fi site plaintext only, and bump those users that way, or something along that way. Maybe the Amprnet users can use that, too?

Sunday, October 9, 2016

aprs.fi iPhone/iPad app update: v1.6.2

Version 1.6.2 of the aprs.fi iPhone/iPad app went out yesterday evening. I've been adding a few features and fixes here and there on the weekends, but most of the larger changes in the code are actually under the hood and not yet visible for the users. I've also spent a good amount of time upgrading the aprs.fi web site backend and fixing a few bugs here and there.

Here are the visible changes in 1.6.2, all of them were recently requested by users of the app:

  • Tapping a station on the map multiple times switches the track line colour for that station.
  • Track line width can be adjusted in Settings. The default size is slightly thicker than the previous default.
  • The maximum tracked station tail/track length is now 6 hours.
  • Previously selected stations and addresses can be deleted by swiping the respective table row to the left.
  • Previously selected address search results are retained even if the application is killed manually by the user.
  • Some small visual adjustments (more space between "Beacon now!" and "Delete station" buttons, etc).

Some folks are also asking for a feature to track multiple stations at the same time, but that can already be done, since version 1.0 – just tap the '+' button on the additional stations info view to add them in tracking. This video demonstrates tracking many stations in the iOS app.

Here are some of the new features added in previous versions this summer:

  • Setting to hide/show station callsign labels on the map
  • Setting to disable (accidental) map pinch rotation
  • Latest packets of an APRS station can be viewed by clicking a new button in the station information view. Tap a packet to decode it using the aprs.fi packet decoder.
  • Filtering feature to control what is shown on the map, for hiding weather and AIS stations, for example. Complex custom filtering will be available in the future.
  • A feature to display road traffic information (traffic jams). The information can be hidden in Settings.

I'm also currently working on APRS-IS beaconing and messaging features, and arbitrary date/time range selection (with long time ranges), but they're not complete yet. Larger features take more time.

The app is already getting quite happy reviews, but it still needs a few features to really make it complete. The 5-star average rating and these reviews are for version 1.6.1, from users from the USA: