Wednesday, May 13, 2020

aprs.fi iPhone / iPad 2.0 with Messaging, APRS-IS TX and DSP modem

Version 2.0 of the aprs.fi iPhone & iPad app is now out! A new major version is well deserved, since it comes with the following new major features:
  • APRS text messaging
  • APRS-IS position beaconing
  • A high-performance software DSP modem/TNC (1200 bit/s only at this time)
  • Longer time ranges in map and graph views, arbitrary date/time ranges on map
  • Up to 10 station profiles in Beacon
The features listed above are enabled by purchasing the Extra Features subscription with an in-app payment.

There's a 1-week free trial included in the1-year subscription, but remember to cancel at least 24 hours before the end of the trial, as Apple will charge the next period up to 24 hours before each subscription term ends! They do this to ensure continuous service when the period ends and switches to the next one.

This is a big release for me, since I have literally worked on these features for a few years. It's been a bit slow, but I finally managed to put together a set of major features which should be well worth the subscription price. The story and reasoning behind the subscription model is throughly described in a previous blog post so I won't dive into it now.

In addition to the Extra Features, there are a few new things which are available to all users:
  • Beaconing now has a slider control to adjust the maximum interval between beacon transmissions between 5 minutes and 60 minutes. Previous versions had this fixed to 25 minutes. If the minimum and maximum sliders are set to the same value, it will beacon with quite exactly that interval. If they're set to different intervals, the app will use an algorithm similar to smart beaconing, as before - move faster, it will transmit more often. If you don't move, the maximum interval is used.
  • The map view now has a button to adjust the time range.
And, of course, a few bug fixes here and there.



Messaging is implemented in a very iPhone-like way. Received ACKs are shown as a delivery status for each sent message. Transmitting works either via a TNC (Mobilinkd TNC3 or the new built-in DSP modem), or via the Internet (aprs.fi / APRS-IS). Received messages come in via both paths transparently, and it indicates whether each message got in via TNC or Net, or both. Messages received from the APRS-IS are delivered to the iPhone using a Push Notification, so you'll get a notification just like you'd get from an SMS or iMessage, even if the app would not be running at the time. If you have an iWatch it'll also show the notification with message contents!


The demodulator of the TNC is based on the excellent Direwolf modem by John Langner, who kindly permitted me to redistribute his code under the BSD license. I've just reorganised the code quite a lot, and optimised it for less CPU consumption when compiled for the iPhone. It decodes really well - I've benchmarked it with the WA8LMF TNC Test CD and it's in the top category by the number of decoded packets, when compared to the benchmark results published for other TNCs by John in the Direwolf documentation. I'll leave comparative tests and public benchmarking to other neutral parties!

The modulator of the TNC is partially derived from javAX25 by Sivan Toledo, and its C port by Alejandro Santos, who have also kindly permitted me to redistribute under the BSD license.

I still do recommend the Mobilinkd TNC3, since it has an actual PTT circuit to key the transmitter, and it talks with the iDevice over BLE - no wires needed to attach the iPhone. With an audio / DSP modem you'll have to use VOX to key the transmitter, and the transmitter will start up slowly (requiring a large TX delay setting) and will turn off slowly as well (eating precious channel time).

Here's a little video showing packets being decoded right off the speaker. Once I get an iGate implemented I'll have to run the iGate for a few days like this, just for the silliness value.

Sunday, March 22, 2020

aprs.fi supports Kenneth's Proposed Telemetry Format

Telemetry is the collection of measurements or other data at remote or inaccessible points and their automatic transmission to receiving equipment for monitoring. The word is derived from the Greek roots tele, "remote", and metron, "measure". (Wikipedia)
APRS telemetry can be used to transmit things like voltages, temperatures, and channel traffic status from anywhere to everyone on the APRS network.

Today I've changed how aprs.fi deals with telemetry values. The change is backwards compatible, but if you start to play with this on the transmitting side, strict classic implementations may not accept your data any more. Please read the whole article if you're going to make use of this.

The APRS protocol is documented in APRS101.PDF. Chapter 13 specifies how telemetry can be transmitted. Analog values are defined to be 8-bit integers:
There are five 8-bit unsigned analog data values (expressed as 3-digit decimal numbers in the range 000–255), followed by a single 8-bit digital data value (expressed as 8 bytes, each containing 1 or 0).
In year 2020 you may find this a little bit restrictive, as a lot of measurements are done at slightly higher resolution these days. Even the popular DS18B20 temperature sensor does 12 bits. Oh, and it can do negative temperatures too.

Values that are negative, or larger than 255, or decimal values which fit within 8 bits, are supposed to be scaled to fit within the 0...255 range. A separate EQNS Equation Coefficients packet should be sent to provide coefficients a, b and c, which are used to scale the transmitted value v back to the original measured value X using the formula:
X = a * v2 + b * v + c
So, you could scale temperatures between -40 and +60 Celsius to a range of 0..100 by adding 40, and then transmit an EQNS packet saying a = 0, b = 1, c = -40, and the recipient could then figure that a transmitted value of 60 would be 0*60^2 + 1*60 + -40 = 20°C.

But if you follow the spec, there's no way to transmit a 80-degree temperature range (-40°C to +40°C) with a 0.1°C resolution. It could be scaled to 0..800 range, but it'd overflow the 8-bit upper limit of 255.

One of my in-house temperature & humidity sensors.
ESP8266 on wifi + DHT22 compatible AM2302 sensor.
Not APRS though, MQTT to Graphite & Grafana.
Needless to say, a lot of people found this unnecessarily complicated and limited, and figured they could just ignore the slightly archaic specification and simply transmit and accept values larger than 255, decimal values, and negative values. When I originally implemented telemetry decoding for aprs.fi in 2008, I already found a lot of this happening.

I figured there must be other implementations that are more relaxed than APRS101.PDF and just went along, picking a backwards compatible but slightly more relaxed schema using the Stetson-Harrison method, without really documenting the situation. So, until today, aprs.fi has been storing an unsigned fixed-point decimal number which can represent values between 0 and 999999,99. Two decimals. No negative values. It has been stored as a 32-bit data unit in the database. I admit it's been a bit silly - I didn't go all in and accept negative values, and the fixed point is a bit... fixed.

Kenneth Finnegan (MS in APRS, the highest academic degree anyone has on the subject) has written a some good notes on how APRS telemetry is actually formatted in the wild here:

https://github.com/PhirePhly/aprs_notes/blob/master/telemetry_format.md

Kenneth also outlines a Proposed Telemetry Format, where a telemetry packet can be made shorter if there's less than 5 values to be transmitted, and where large, negative or floating-point values can be transmitted as-is without scaling them to the classic nintendo 8-bit 0...255 range specified in APRS101.PDF. I think the proposal is good, and it also matches the reality of what is already being done widely by many telemetry transmitters.

The proposed format is backwards compatible in a way that allows decoders accepting the new format to also accept the old format without special handling. Old and pedantic decoders can not accept the new format when either the shorter format (with less values) or decimal/negative/large values are sent. Luckily it's easy to upgrade the old decoders to accept the new format, and even the smallest microcontrollers today are easily programmed to handle floating point values like this.

As of today, aprs.fi implements Kenneth's Proposed Telemetry Format.


  • Analog values are stored in signed IEEE single precision (32-bit) base-2 floating point format. Kenneth does not specify a resolution or maximum value, so I just picked the single precision format, since it doesn't actually increase my disk storage requirements. As the wikipedia article says, the single precision float can store a maximum value of 3.4028235 × 1038. "All integers with 7 or fewer decimal digits, and any 2n for a whole number −149 ≤ n ≤ 127, can be converted exactly into an IEEE 754 single-precision floating-point value."
  • The decoder only accepts values between -2147483648 and 2147483647.
  • Since it is a floating point value, if you send very large values, the resolution starts to show up in the less significant digits. But it is still exactly precise in the -9999999 to 9999999 range, and certainly in the 0...255 range too.
  • Only three decimals are printed on the telemetry pages for now - if you transmit 0.0005 it'll be rounded up to 0.001. This is because the values calculated with EQNS coefficients often have a lot of decimals, and for most data it does not make sense to print it out with more than 1, 2 or 3 decimals. But I'm guessing there isn't much data that would have meaningful 4 decimals.

A few examples of packets that can now be accepted

Short telemetry, just one value of 42:
T#001,42
Slightly longer, with three values, first one being negative, second one being large, third one having decimals:
T#042,-1,10000000,142.4242424
Just four bits of digital telemetry:
T#999,,,,,,1101

Caveats

Older decoders which read APRS101.PDF to the letter will not accept these packets. I'm hoping they will be relaxed to accept the proposed format in the future. At this point you'll need to test if your intended recipients parse them.

javAPRSSrvr will pass these packets through, but if a client sets up a "t/t" filter to request all packets of the Telemetry type, it will currently only accept packets which strictly match the APRS101.PDF format. If there is a decimal point in a value, a negative sign, or an analog integer value exceeding 255, the client will not get the packet. I'm hoping Pete will relax this requirement in the future.

aprsc packet type filters (like "t/t") only look at the packet type (first few bytes usually). It does not care if the remaining packet matches some format or not, and it allows upgrading the format without changes in server code.

Final note to people writing APRS decoders

If you implement the Proposed Format, please let me or Kenneth know, or make a github pull request to Kenneth's document, so that we can have a list of software supporting the new format. Thanks!

Saturday, February 1, 2020

How APRS paths work

The APRS packet path is used to control the distribution and retransmission of a packet in the APRS network.

Every now and then someone asks how paths actually work. Having spent quite some time staring at packets that have been digipeated, and decoding them at my little aprs.fi web site, I suspect I might have begun to understand it, so here goes. I'll explain it the long way and begin with a packet without a path, continue with the traditional AX.25 digipeating path, and then go to the APRS world.

No path


Let's assume my call is N0CALL, and I have a generic APRS device which uses the generic APRS tocall (destination callsign) of APRS. The packet, without any digipeaters or path, in the usual text format, as used on the APRS-IS, would look like:

N0CALL>APRS:!1234.56ND01037.50E&

Some of my APRS gear on top of my old bass gear, from left:
Mobilinkd TNC3 for iPhone (prototype unit), Kenwood
TH-D72 (digipeats), an unassembled TNC-X (can digipeat with
a daughtercard added), Argent Data Systems Tracker2 OT2m
(digipeats), Coastal Chipworks TNC-pi on top of a Raspberry
Pi (digipeats with aprx), Byonics TinTrak4 (digipeats),
Kenwood TH-D74 (can digipeat). Moomin for size reference
only. Hartke Kickback 15" 120W. If your APRS hardware
is not shown here yet, my address is on qrz.com. :)
There are two separating characters: the '>' separates the source and destination callsigns, and the ':' separates the packet header from the actual data being transmitted (the APRS formatted position, in this case). A packet like this would not be repeated through any digipeaters, but it could still be directly heard, picked up by an iGate and passed to the APRS-IS on the Internet. Sometimes you might want to do this and configure your transmitter with an empty path.

Classic AX.25 digipeating


The digipeater path may then optionally appear after the destination callsign. Here's a packet that I might transmit, which requests digipeating via two specific digipeaters, in a particular, specified order:

N0CALL>APRS,OH7RDA,OH7RDB:!1234.56ND01037.50E&

Here, the digipeater path is OH7RDA,OH7RDB - the packet should be digipeated by OH7RDA and then OH7RDB. This is how classic AX.25 packet radio digipeating works. APRS is transmitted using AX.25 packet radio packets, so they will follow the old AX.25 rules too. You can certainly do this and most APRS digipeaters will digipeat your packet if you just put the callsign in the path.

When OH7RDA retransmits the packet, it will flip the "has been repeated" bit on for that callsign. It really is a single bit called the "H bit". The packet, in text format, will look like this – note the '*' which indicates the "has been repeated by this digipeater" bit:

N0CALL>APRS,OH7RDA*,OH7RDB:!1234.56ND01037.50E&

Each digipeater will look for the first digipeater callsign in the path which has not been used yet (i.e. the "has been repeated" bit is not on - there's no "*" in there), and then figure out based on that if it should be retransmitted. So, after OH7RDA has retransmitted the packet, and OH7RDB has heard the above packet with OH7RDA*,OH7RDB in it, it will go "oh that's my callsign, I'll retransmit". The resulting packet transmitted by OH7RDB will have OH7RDA*,OH7RDB* as the path! Note that OH7RDB will not retransmit the original packet before OH7RDA has done his part.

To add confusion, and to save a few bytes, usually only the last "*" is usually printed, although the "has been repeated" bit is set on all previous digipeater calls. OH7RDA,OH7RDB,OH7RDC* actually means OH7RDA*,OH7RDB*,OH7RDC* !

Aliases of digipeaters, classic AX.25 packet approach


Digipeaters can be also configured to respond to alias callsigns in addition to their own call. For example, OH7RDA could be configured to repeat a packet which requests digipeating by ALIAS. It may then do one of two things: It can either substitute ALIAS with it's own callsign, or prepend it's own callsign before ALIAS. This varies by digipeater software and its configuration.

N0CALL>APRS,ALIAS:data

When retransmitted by OH7RDA, will become one of:

N0CALL>APRS,OH7RDA*:data
N0CALL>APRS,OH7RDA,ALIAS*:data

The benefit of the second format (digi call prepended) is that, upon receiving such a packet, we know that the originator used ALIAS as the path.

In the classic/old APRS digipeating world, people used aliases like WIDE to ask for any digipeaters to digipeat. Currently, this method is used to digipeat through satellites and the ISS - just use a simple alias of ARISS as the path and it will be digipeated through the RS0ISS digipeater on the ISS or any of the other APRS-supporting satellites on 145.825 MHz! Yes, this means the PATH setting should be just "ARISS". Do not put any of the WIDE or RS0ISS stuff in there, it won't help.

WIDEn-N paths on APRS


This is where it gets interesting and more specific to APRS, as these paths only work on APRS digipeaters. A WIDEn-N path has two integers, n and N. WIDE3-1 would have n of 3, and N of 1. The first integer theoretically means "I'd like to have this packet to be digipeated by this many digipeater hops" (3 in the example of WIDE3-1). The second integer means "there's this many hops left before digipeating stops". When it becomes 0, it won't be digipeated any more, and the "has been repeated" bit will be set (a "*" will appear).

A packet with a path of WIDE3-3, after first retransmission will become WIDE3-2, and then WIDE3-1, and after the third retransmission it'll be WIDE3*. The SSID of 0 is not printed – it's really WIDE3-0 under the hood of the packet but the -0 will not be shown.

The digipeaters supporting this kind of alias usually also prepend their callsigns (all the good ones do, the bad ones don't). And the packet will be often picked up by many digipeaters! So you may see something like this, when both OH7RDA and OH7RDB hear N0CALL, and OH7RDC hears OH7RDB:

N0CALL>APRS,WIDE2-2:data
N0CALL>APRS,OH7RDA*,WIDE2-1:data
N0CALL>APRS,OH7RDB*,WIDE2-1:data
N0CALL>APRS,OH7RDB,OH7RDC,WIDE2*:data

You can also send a packet which originally has a path of WIDE2-1, and it'll be digipeated by digipeaters which are configured to respond to the WIDE2 alias - but only once, since the "remaining hops" number is 1 to begin with. If three digipeaters can hear you, this would result in:

N0CALL>APRS,WIDE2-1:data
N0CALL>APRS,OH7RDA,WIDE2*:data
N0CALL>APRS,OH7RDB,WIDE2*:data
N0CALL>APRS,OH7RDC,WIDE2*:data

A packet of WIDE2-2 is often retransmitted by much more than than 2 digipeaters, since it will be digipeated twice to each direction. Each high-level digipeater in a tower will be often heard by a few digipeaters which are actually quite far away. This is how a WIDE3-3 path might be distributed through a whole country, and a bit to the neighbouring countries as well (rough example but not far from the truth):


This is why we don't usually use paths with more than 2 hops on APRS, and practically never more than 3 (except in some specific cases in rural areas with quiet channels having little traffic). A packet like this can light up the whole APRS network in almost the whole country (or a few states, for you in the larger nations). If a lot of people did this, the channel would become too congested and APRS would stop working as everyone would be transmitting on top of each other. It has happened and we've learned it the hard way.

Fill-in digipeaters


In some areas people set up low-level "fill-in" digipeaters at their homes. They are only configured to respond to the WIDE1 alias, but not WIDE2. Digipeaters higher up at towers and such, serving a wide area, are configured to respond to both WIDE1 and WIDE2 aliases.

The idea is that the fill-in WIDE1 digipeater may well hear packets from nearby transmitters, which might be missed by the higher-up digipeater – due to distance, or because it hears a lot of stations and gets a lot of collisions, or because there are "urban canyon" obstructions. The wide-area-coverage WIDE2 digipeater will be heard by everyone, and there's no need for the fill-in WIDE1 digipeaters to retransmit packets which have already been transmitted by it!

This is accomplished by transmitting a packet with a WIDE1-1,WIDE2-1 path. This is the usual recommended default APRS path, as you might have noticed. It would be first digipeated by any digipeater (fill-in or wide-area), and then by a wide-area WIDE2 digipeater. But only for a total range of 2 hops! This is how it might look, if you could receive the original packet and all the digipeated packets - N1FILL is the fill-in home digipeater, OH7RDA and OH7RDB are higher-up digipeaters which can hear N1FILL but not N0CALL:

N0CALL>APRS,WIDE1-1,WIDE2-1:data
N0CALL>APRS,N1FILL,WIDE1*,WIDE2-1:data
N0CALL>APRS,N1FILL,WIDE1,OH7RDA,WIDE2*:data
N0CALL>APRS,N1FILL,WIDE1,OH7RDB,WIDE2*:data

In reality, many of these fill-in digipeaters are old and dumb TNCs, and can not do callsign prepending. Instead, they just do callsign substitution and replace WIDE1-1 with their own callsign, like this:

N0CALL>APRS,WIDE1-1,WIDE2-1:data
N0CALL>APRS,N1FILL*,WIDE2-1:data

Alternatively, if N1FILL would not receive this packet, but just the higher-up digipeaters directly, then it might go a bit further. OH7RDA heard N0CALL, and subsequently OH6RDA and OH8RDA hear OH7RDA:

N0CALL>APRS,WIDE1-1,WIDE2-1:data
N0CALL>APRS,OH7RDA,WIDE1*,WIDE2-1:data
N0CALL>APRS,OH7RDB,WIDE1,OH6RDA,WIDE2*:data
N0CALL>APRS,OH7RDB,WIDE1,OH8RDA,WIDE2*:data

To send a packet for one hop of digipeating you can choose to use either WIDE1-1 or WIDE2-1. With WIDE1-1 you may have it digipeated through fill-ins and wide-area digis, a WIDE2-1 should only trigger the wide-area ones.

There are also some buggy digipeaters out there which do not add their callsign at all - they just retransmit, and decrement the hop counter of a path element (WIDE2-2 to WIDE2-1 and so on). These are particularly annoying when trying to figure out what happened to a packet.

What about the ,qAR,IGATECALL stuff happening on the APRS-IS?


The APRS-IS servers on the Internet, or alternatively, the iGate itself, append something called the q construct to the packet. After the q construct you'll find the callsign of the iGate which received the packet from a radio and passed it to the Internet:

N0CALL>APRS,WIDE1-1,WIDE2-1,qAR,IGATECALL:data

The q construct does not appear on the radio side - it is only used for tagging additional information to it on the Internet.

Limits and policies


Digipeaters often have policies or filters configured, to prevent digipeating of outright silly paths like WIDE6-6, or paths with a big combined hop length ("WIDE1-1,WIDE2-2,WIDE3-3,WIDE3-3" for a total of 9 hops), or paths where the second integer is larger than the first one (WIDE1-7). Paths like this are considered abusive and cause excess congestion to the APRS network.

Having said that, I think it's alright to have some fun sometimes!

One fun trick is to send a packet, manually, by hand, towards the west, have it travel around the country and then come back to you from the east. You can accomplish this by drawing a map of live digipeaters (aprs.fi can do this for you), and then putting in the callsigns of the digipeaters in the path (up to 8 digis may work): DIGI1,DIGI2,DIGI3,DIGI4,DIGI5. Even though they are many elements in the path, it won't consume much more radio channel time than what a WIDE1-1,WIDE2-2 packet would. Just don't do this with the regularly-beaconing APRS tracker in your car, it'd be silly.

Thursday, August 15, 2019

RX-only igates considered beneficial to the network

As probably most of you know by now, besides running the aprs.fi web site, I'm also one of the two main authors of the aprsc server software. aprsc runs on most of the APRS-IS servers where iGates usually connect.

There have been recent and strong claims saying that receive-only (rx-only) iGates destroy the two-way messaging feature of APRS. This has been claimed in blog posts and facebook threads. Some people ask me if this is true.

No, it is not true. Receive-only iGates do not break messaging if there are transmit-capable igates nearby, and those transmit-capable iGates are connected to an APRS-IS server which has a full APRS feed. All aprs.net and aprs2.net servers, where iGates normally connect, do have a full feed. No problem!

Messaging would work from 1650m / 5400 ft above Vihti, even in the presence of
RX-only iGates, as long as there is at least one TX-capable iGate.

If there are no transmit-capable (TX) iGates around, two-way messaging will not work, of course. Having a transmit-capable iGate would therefore be better, but receive-only iGates are easier to set up technically, they are cheaper (receivers are practically free now), and licensing for automatic transmitters is difficult in many countries. Where transmit-capable (TX) iGates are present, receive-only iGates do not break the TX iGates, they just improve reception coverage. For messages, too.

Just to make it perfectly clear: If there are TX iGates present, additional RX-only iGates improve messaging performance. For the RF-to-IS direction (and ACKs for the IS-to-RF messages).

The common incorrect claim is that the APRS-IS server sends the message only to the latest iGate which heard a station. In fact, the APRS-IS servers (both javaprssrvr and aprsc) send the messages to all iGates which heard the station recently. In aprsc, "recently" means "within 3 hours", and I believe javaprssrvr uses something similar.

The server maintains a list of recently heard callsigns independently for each iGate client. There's a separate list of heard stations for each iGate client. When the server has a packet to pass on, it will look at all connected clients, and for each client, if the recipient of the message is found from the list of recently heard stations, the message will be sent. The scanning will not stop; the message will be given to all clients which heard the station.

I can confirm that we have written the software to do it like this, and since it is open source, you can see the code yourself. The automatic test case also validates that the server keeps working that way, so that there won't be a bug creeping in the future to break it accidentally. I've also tested that javaprssrvr behaves like this – I've run the test cases against it to confirm compatibility.

There may be problems when there is a server with a filtered feed involved (possibly a server software running at the "client" without having a full feed), but those are rare and known to be problematic. All Tier2 servers have a full feed (aprs2.net ones), and so do the core servers.

The real problem is that a user, seeing one-way beaconing to the APRS-IS works, may well expect two-way messaging to work too. And when it doesn't work, there's only a timeout, no immediate feedback saying "sorry, this won't work now", which is what a sensible system would do today.

Even in that case, an rx-only igate is better than nothing! I wouldn't be so harsh against them, since the step from RX-only to TX-capable may be a bit difficult for many.

Bottom line:

In each area, there should be one TX igate, maybe two. More may create QRM as the same messages will be transmitted from APRS-IS to RF many times. RX igates will not break messaging if there are TX igates around – they just improve RX coverage.

This picture may at first seem irrelevant, but it does show me working VHF in AM (122.825 MHz),
and Flarm on 868 MHz to OGN (which runs aprsc). Flarm antenna visible in top left corner.

Sunday, April 21, 2019

Server upgrades, and raw packets UI abuse

This month I'm upgrading the aprs.fi servers quite a bit. The old blade server hardware chassis is getting shut down and replaced with a brand new pre-owned and dumpster-dived blade server chassis. The new second-hand blades have 8 CPU cores and 192 GB memory each, which is quite sweet! I suspect the web site & database will be doing very little disk reads with this much cache around.

The downside is that we're not going to run the fibre channel SAN network, and the associated disk racks, since they're a bit power-hungry. So, internal disks only it is. To get relatively good disk IO performance I installed 1 TB SSD disks in the blades – two SSDs in each for RAID1 (~170€ with 24% VAT each, 3 blades, total 6 SSDs). The blades have CERC 6/i raid controllers, which are SATA2 and quite slow. The Samsung SSDs are capable of some 500+ Mbytes/sec reads, but the hardware controller in RAID1 config only gives out 130 Mbytes/sec. With Linux software RAID1 it can balance reads across the SSDs and give out 2*140 Mbytes/sec, which is alright.

aprs.fi being a database application, especially with the huge cache memory, it'll be mostly random-access write heavy, so the streaming read benchmark above is not too relevant. The random-access speed of the SSD gives a nice boost, and the memory will help an awful lot.

The new servers are already running a live replica of the database, but the web service is still running on the old ones. Hoping to get the move done soon. The operating system is also getting a bump to the next major release, and some adjustments are needed to support that.

In less happy news: A few smart folks have, again, figured that it would be a great idea to download raw APRS-IS packets programmatically (using software they've written) from the aprs.fi user interface, the raw packets view. This is forbidden by the TOS (https://aprs.fi/page/tos) and a longer reasoning can be found in the FAQ (https://aprs.fi/page/faq): 
Downloading data for application use using the user interface (for example, fetching the /info/ page just to get the current coordinates of a station) wastes both human and computer resources.  
First, you need to write a parser to extract the data from the HTML (and then fix your parser every time I change my HTML layout). Second, aprs.fi needs to do the user interface magic (language and timezone selection, user-friendly template formatting, session set-up) for every request, all of which is unnecessary.  
[...] All of this extra overhead consumes CPU time, which in turn heats up the computer room, consumes electricity, destroys tropical rain forests, accelerates global warming, and kills kittens. And baby seals.
The correct way to obtain a raw APRS-IS feed is to connect to the APRS-IS  (http://aprs-is.net/Connecting.aspx) - you'll get a lightweight TCP  session with all the packets in real time. As clean CRLF-delimited rows with none of my HTML encoding to mess it up. For free, from the very same place where I get them from, and much faster! 

If one chooses to fetch the packets by hammering on my user interface, as opposed to an API, it'll create unnecessary extra load on my servers, and make me a little bit upset. Especially when someone fetches 1000 packets every time, via the Tor network, repeatedly every few seconds, and ends up fetching the same packets over and over again. That is not a great way to make use of this free service.

To make this sort of abuse harder, you'll now need to log in to view the raw packets. It's a little bit more clumsy if you're not logged in already, but the login cookies have a long lifetime so you don't need to do it that often.

Monday, December 10, 2018

Subscription for extra features within aprs.fi iOS app

This little warning has been in the aprs.fi iOS help text from the first version of the app - my plan from the beginning has been that some functions will require an additional paid subscription. The time for this is coming soon, and I've lately been spending some time implementing those features. This is likely going to upset some people ("what, I already paid for this app, I need to pay more?!?") so it's probably good to go through the reasons once more.

There is a fairly small number of amateur radio APRS users which are using iPhones or iPads. Androids are quite a bit more popular amongst hams. It will only take a short while until everyone, who ever will buy the app, has done it. After that nobody will pay for the application any more.

The work will not stop there: the application and the backend servers supporting it need to be maintained, upgraded and improved over time – if the application does not get some love and care from its creators at regular intervals, it will soon decay and eventually stop working. This has happened to a few ham and APRS apps already, and I'm not at all surprised about that. Even if the application does not get any new features, someone must keep it going, add support for new iOS versions and iPhone models, fix backend servers, and so on.

This maintenance burden may be difficult to appreciate if you haven't tried doing it yourself. When there's a new iOS version, you need new Xcode to build the app for it, new Xcode requires new OS X version on the laptop, and the app's code will need modifications to compile on the new Xcode due to changes in the programming language and APIs (programming interfaces provided by the operating system). And then there are the backend servers (don't get me started about "serverless").

"What? We need to pay more?!? Greedy bastards!"
Photo of Finnish native "kyyttö" breed of cows (c) myself
2016-06-16 
at Sappion luomutila
I do quite some volunteer work as a hobby to support the APRS system (in the form of running aprs.fi itself, contributing free open source software such as aprsc and the aprs2.net backend software, open source APRS symbol graphics, machine-readable APRS device identification database, and a few other bits). I'm already quite busy with everything, and the family, so I decided to make a little bit of money from the iOS app. Due to the relatively small amount of users, it'll probably never pay for all the effort needed (pay-off not really comparable to the daytime job), but at least something. And, to keep the motivation there in the future, it would be nice to have something that is called a sustainable business model - in other words, a little bit of money keeps flowing in the future, too. That's where a subscription payment model comes in handy!

How will it work?


The model will be very simple: There's only a single subscription option, which will enable a set of extra features. Whenever a new feature is added, it may go to the "base" set of features, which are available to all users which have purchased the app. Or, it may go to the "extra" set of features, which are available with the "extra" subscription. If you have the "extra" subscription, you'll have all the features added in the future – there won't be another "pro", "enterprise", or "super" subscription required for those. No per-feature pricing, just the "base" set and the full "extra" set.

Some features going to the "extra" set will seem fairly basic to some, and they will be features that most users hopefully want to have: Messaging, beaconing to APRS-IS, iGate, long and arbitrary time range lookups, soundcard modem, etc. On the other hand, the pricing will be fairly cheap, in my opinion, when compared to the price of an iPhone or any piece of APRS hardware: less than the app purchase price per year, resulting in roughly 50 cents / month price ($0.50 USD / € 0.50). About the same price as a Panini at Starbucks, a bit more than large Caffe Mocha! Subscription period will be 6 or 12 months, and there will be a free trial period in the beginning so that you can try before you pay. I may adjust the price later to one direction, or the other, depending on how it sells.

I hope most users will want the extra set, so that even with this low subscription price I'll still get something. Apple grabs 30% of that for the first year of subscription (like it does for app purchases), and then the taxman grabs another 30-40% of what's left. You pay $6, I get less than $3, and I can buy a couple litres of milk for example. After two years I'd have enough to buy a beer in the pub. Yay.

Some folks will likely ask for a lump sum "lifetime" option for the extra features. It's not good for the sustainable business model – the purchases would end soon, at which point there is no income. Also, it's difficult for anyone to promise that it will actually work for a "lifetime", due to the need for regular maintenance and support from the developer.

When comparing prices of applications one should remember the size of the target audience. Creating and maintaining an application takes roughly as much time for 10, 1000 or 100000 users. This is why some really nice and big apps for the general public can be really cheap, or financed by in-app advertisements: 1€ per user for 100000 users would already make 100000€. This makes apps for smaller audiences (hams with iPhones and interest in APRS) relatively more expensive.

To re-iterate:

  • The subscription price will be small (roughly around 50 cents / month), paid through in-app payment.
  • There will be a free trial period in the beginning of the subscription, so you'll know what you'll get. Just remember to cancel it before the end of the trial; this is how Apple's subscription trials work.
  • It will enable all new features in the future.
  • Current features will NOT stop working or require subscription payment – the subscription is optional and you get to keep what you paid for earlier.
  • Some future new features will also be included the base feature set, not requiring subscription.

Tuesday, May 9, 2017

Important Hints to iGate developers

By connorgoodwolf - Own work, CC BY-SA 4.0
There are a lot of APRS iGate implementations out there, and new ones are popping up every few months these days. A lot of the new ones are made for various microcontroller chips and development boards (Arduino, ESP8266 and friends). Some are made for SDR sticks, some are built in mobile phone applications. Many of them reduce the amount of hardware needed to run an iGate by fully removing the need for a TNC, a full-blown computer, or an actual amateur radio transceiver.

Very cool stuff!


It would seem like writing an iGate would be very easy: receive packets off the air, put them in TNC2 format, and pass them to the APRS-IS servers over TCP (specifications: connecting, design, details). But it turns out that there are quite a few things that can go wrong in the implementation, and in fact, some things seem to be going wrong the same way fairly often. Some bugs are fairly common and some seem to pop up in multiple different iGates, old and new.

Currently the most popular ones are due to the false assumption that all packets are either plain 7-bit English ASCII text with no NUL bytes or other binary stuff, or some specific Unicode encoding such as UTF-8. These assumptions lead to the packets getting truncated at the first NUL (binary 0) byte, or otherwise modified in a way that scändic and øthér ïnternátiønal characters get mutilated if they don't happen to use the same encoding. Sometimes a truncation of a packet at the first NUL may lead to the packet being concatenated together with the next packet.

I get to decode the corrupted packets on aprs.fi, and I also get to process the corrupted ones in the aprsc APRS-IS server software which is fairly common these days. If the bugged packets are not detected and handled, there usually are some folks complaining to me about the odd results, since they are often received from aprsc and viewed on aprs.fi.

I can't blame the authors of the new applications – I have to admit that APRS documentation is not great, and it's easy to stumble across the same bugs if you haven't fixed them once before.

Every now and then I spend an evening trying to identify some of these bugs in the APRS-IS client applications and file bug tickets to the software authors. Last night I filed 3 packet corruption bug reports on Github for 3 different iGate applications. These are not the first ones, I've probably filed 30 in the past. It's not hard to spot these once you know what to look for.

I'd like to take this chance to thank to all the authors who fix the bugs in a timely manner and release the fixed versions!

Based on this experience I wrote a little document listing common iGate bugs: IGATE-HINTS.md. It'd be great if all APRS software authors could glance through it and make sure these bugs are not present in their own software. If you spot someone writing a new iGate, please pass on the link to the new author – hopefully these bugs will then be avoided right from the beginning.

Unfortunately some buggy iGate applications are orphaned and no longer developed. A few of those applications are fairly popular, too. It'd be great for the network if iGate operators could gradually upgrade to applications which still get updates, and don't exhibit any of the packet-corrupting bugs.

A lot of iGates are also not getting updated, even though they run software which would have updates available. Some iGate operators feel that the iGates work, sort of, since most packets go through them just fine, most of the time. I think I'll try to make aprs.fi highlight some of the more common issues on the station info page in the future.

If you know a little Python or C, and you've got a little extra time, let me know – there are some bugs out there in the Open Source Land which could use a helping hand.