Saturday, April 2, 2011

Map loading problems last night

A memory table containing all of the last position reports for the past 30 hours or so, used for quickly loading the real-time map view, filled up last night on one of the two front-end web servers at around 2010-04-01 23:35 UTC (2010-04-02 2:35 local time). I woke up, noticed the SMS alert and fixed the problem at 2010-04-02 6:50 UTC (9:50 local time).

While the table was full, replication was stopped, and about 50% of real-time map loads (without a station tracked, just an area view) resulted in no stations displayed. Also, 50% of page loads on the site last night returned a bit old data (the stopped frontend continued to display the 23:35 UTC state!).

No data was lost during the incident, APRS-IS data collection continued normally.