Train Departure Board

You can find the code for this article on my github:

Having found myself time-wealthy for a couple of weeks I’ve been playing around with some open data sets. One of which is the National Rail SOAP API. It’s not a new dataset, I think it’s been around for a decade or so, but it seemed like a sensible thing for me to play with as I’ll be on trains a lot more when I start selling my time to a new employer next month!

I live about a mile away from the local station (Thatcham) so it only takes a few minutes to get there. If a train is delayed or cancelled I’d like to know so I can have another coffee. So what I need is a live departures board, for my local station, in my house. Something like this:


The UI is web based – using AngularJS again. Sadly though, the cross origin nonsense means I can’t make the soap calls directly from the web client, I need a back-end to gather and store the data for use on the UI. I used Python for this because that gives me the option to (easily) run it on a Raspberry Pi, reducing power and space costs as well as noise. Python’s library support is stunning, and this is another great reason to use it for small “hacks” like this one.

SOAPing Up

SOAP is horrible. It’s old, it’s heavy, it’s complex and worst of all it’s XML based. This isn’t a huge handicap though, as we can hit a SOAP service using the requests HTTP library – simply sending a POST with some XML like so:

import requests
import xmltodict

xml_payload = """<?xml version="1.0"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="" xmlns:ns1="" xmlns:ns2="">

# url: The URL of the service
# key: Your National Rail API key
# crs: Station code (e.g. THA or PAD)
def fetch_trains(url, key, crs):
    headers = {'content-type': 'text/xml'}
    payload = xml_payload.replace("{KEY}", key).replace("{CRS}", crs)
    response =, data=payload, headers=headers)

    data = xmltodict.parse(response.content)
    services = data["soap:Envelope"]["soap:Body"]["GetDepBoardWithDetailsResponse"]["GetStationBoardResult"]["lt5:trainServices"]["lt5:service"]

    for service in services:
        raw_points = service["lt5:subsequentCallingPoints"]["lt4:callingPointList"]["lt4:callingPoint"]

        calling_points = map(lambda point: {
            "crs": point["lt4:crs"],
            "name": point["lt4:locationName"],
            "st": point.get("lt4:st", "-"),
            "et": point.get("lt4:et", "-")
        }, raw_points)

        cp_string = "|".join(
                map(lambda p: "{0},{1},{2},{3}".format(p["crs"], p["name"], p["st"], p["et"]), calling_points)

        yield {
            "crs": crs,
            "origin": service["lt5:origin"]["lt4:location"]["lt4:locationName"],
            "destination": service["lt5:destination"]["lt4:location"]["lt4:locationName"],
            "std": service.get("lt4:std"),
            "etd": service.get("lt4:etd"),
            "platform": service.get("lt4:platform", "-"),
            "calling_points": cp_string

So, I take a pre-formatted XML request, add the key and station code then POST it to the API URL. Easy. The result comes back in XML which can be parsed very easily using the xmltodict library. I used Postman to test the calls before translating to Python (and if you’re lazy, Postman will even write the code for you!).

The Python script takes the data its gathered and stores it in an SQLite database. I’m not going to show the code because it’s all in github anyway.

Having a REST

So the data is all in a DB, now it needs to be made available to the Javascript client somehow. To do this I created a simple REST service using the excellent Flask library for Python. It’s in the same genre as Sinatra, Nancy, Scalatra and all the other microservice frameworks I love to use. Here’s all the code you need to serve up data via REST:

import argparse
import sqlite3
from flask import Flask, jsonify

app = Flask(__name__)

def departures():
    return jsonify(fetch_departures("departures"))

def departures_for(crs):
    return jsonify(fetch_departures("departures", crs))

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='National Rail Data REST Server')
    parser.add_argument('--db', help='SQLite DB Name', default="../data/trains.db")
    args = parser.parse_args()
    db = args.db

Front End

The front end is a very simple Angular JS app. Not much point showing the code here (see github) it’s about as simple as it gets – using a dot matrix font I found via Google and a theme from Bootswatch.

The design is based on a real life station departures board like this:

All in all the project took me a little over a day. A leisurely day with many interruptions from my daughters! Feel free to pull the code down and play with it – let me know what you do.

2016-06-14 11.32.01

Quick TeamCity Build Status with AngularJS

So, this isn’t supposed to be the ultimate guide to AngularJS or anything like that – I’m not even using the latest version – this is just some notes on my return to The World of the View Model after a couple of years away from WPF. Yeah, that’s right, I just said WPF while talking about Javascript development. They may be different technologies from different eras: one may be the last hurrah of bloated fat-client development and the other may be the latest and greatest addition to the achingly-cool, tie dyed hemp tool belt of the Single Page App hipster, but under the hood they’re very very similar. Put that in your e-pipe and vape it, designer-bearded UX developers!


Anyway, when I started, I knew nothing about SPA development. I’d last done JavaScript several years ago and never really used it as a real language. I still contend that JavaScript isn’t a real language (give me Scala or C# any day of the week) but you can’t ignore the fact that this is how user interfaces are developed these days… so, yeah, I started with a tutorial on YouTube.

I decided to do an Information Radiator to show build status from TeamCity on the web. Information Radiators are my passion – at least they’re one of the few passions I’m allowed to pursue at work – and we use Team City for all our continuous integration, release builds, automated tests and so on. Our old radiators are coded in WPF, which looks awesome on the big TVs dotted around the office, but doesn’t translate well for remote workers.

There is no sunshine and there are no rainbows in this article. I found javascript to be a hateful language, filled with boilerplate and confusion. Likewise, though TeamCity is doubtless the best enterprise CI platform on planet earth, the REST APIs are pretty painful to consume. With that in mind, let’s get into the weeds and see how this thing works…

Enable cross-site scripting (CORS) on your Team City server

You can’t hit a server from a web page unless that server is the server that served the web page you’re hitting the server with… unless of course you tell the server you want to hit that the web page you want to hit it with, served from a different server, is allowed to hit it. Got that? Thought so. This is all because of a really logical thing called “Cross Origin Resource Sharing”, which you can enable pretty easily in TeamCity as long as you have admin permissions.

Check out Administration -> Server Administration -> Diagnostics -> Internal Properties. From there you should be able to edit, or at least get the location of the file. Weirdly, if the file doesn’t exist, there is no option to edit, so you have to go and create the file. Since my TeamCity server is running on a Windows box, I created the new file here:


and added the following:*

You might want to be a little more selective on who you allow to access the server this way – I guess it depends on how secure your network is, how many clients access the dashboard and so on.

Tool Chain

This article is about AngularJS and it’s about TeamCity. It’s not about NPM or Bower or any of that nonsense. I’m not going to minify my code or use to crazy new-fangled pseudo-cosmic CSS. So setting up the build environment for me was pretty easy: create a folder, add a file called “index.html”, fire up the fantastic Fenix Web Server and configure it to serve up the folder we just created. Awesome.

If you’re already confused, or if you just want to play with the code, you can download the lot from GitHib:

I promise to do my best

Hopefully you’ve watched the video I linked above, so you know the basics of an AngularJS app. If not, do so now. Then maybe Google around the subject of promises and http requests in AngularJS. Done that? OK, good.

Web requests take a while to run. In a normal app you might fetch them on another thread but not in JavaScript. JavaScript is all about callbacks. A Promise is basically a callback that promises to get called some time in the future. They are actually pretty cool, and they form the spinal column of the build status app. This is because the TeamCity API is so annoying. Let me explain why. In order to find out the status (OK or broken) and state (running, finished) of each build configuration you need to make roughly six trillion HTTP requests as follows:

  1. Fetch a list of the build configurations in the system. These are called “Build Types” in the API and have properties like “name”, “project” and “id”
  2. For each Build Type, make a REST request to get information on the latest running Build with a matching type ID. This will give you the “name”, “id” and “status” of the last finished build for the given Build Type.
  3. Fetch a list of the currently running builds.
  4. Use the list of finished builds and the list of running builds to create a set of status tiles (more on this later)
  5. Add the tiles to the angular $scope and let the UI render them

Here’s how that looks in code. Hopefully not too much more complicated than above!

	.then(function(responses) {
		$scope.buildResponses = responses
			.filter(function(r) { return (r.status == 200 && > 0)})
			.map(function(r){ return[0] })
	.then(function(data) {
		$scope.runningBuilds = { return row.buildTypeId })
	.then(function() {
		$scope.builds = $ { return buildFactory.decodeBuild(b, $scope.runningBuilds); });
	.then(function() {
		$scope.tiles = buildFactory.generateTiles($scope.builds)
	.then(function() {
		$scope.statusVisible = false;

Most of the REST access has been squirrelled away into a factory. And yes, our build server is called “tc” and guest access is allowed to the REST APIs and I have enabled CORS too… because sometimes productivity is more important than security!

angular.module('buildApp').factory('buildFactory', function($http) {
	var factory = {};
	var getBuildTypes = function() {
		return $http.get('http://tc/guestAuth/app/rest/buildTypes?locator=start:0,count:100');
	var getBuildStatus = function(id) {
		return $http.get('http://tc/guestAuth/app/rest/builds?locator=buildType:' + id + ',start:0,count:1&fields=build(id,status,state,buildType(name,id,projectName))');
	factory.getRunningBuilds = function() {
		return $http.get('http://tc/guestAuth/app/rest/builds?locator=running:true');

// etc

Grouping and Tiles

We have over 100 builds. Good teams have lots of builds. Not too many, just lots. Every product (basically every team) has CI builds, release/packaging builds, continuous deployment builds, continuous test builds, metrics builds… we have a lot of builds. Builds are good.

But a screen with 100+ builds on it means very little. This is an information radiator, not a formal report. So, I use a simple (but messy) algorithm to convert a big list of Builds into a smaller list of Tiles:

  1. Take the broken builds (hopefully not many) and turn each one into a Tile
  2. Take the successful builds and group them by “project” (basically the category, which is basically the team or product name)
  3. Turn each group of successful builds into a Tile, using the “project” as the tile name
  4. Mark any “running” build with a flag so we can give feedback in the UI


Displaying It

Not much very exciting here. I used Bootstrap, well, a derivative of Bootstrap to make the UI look nice. I bound some content to the View Model and that’s about it. Download the code and have a look if you like.

Here’s my index.html (which shows all the libraries I used):

<html ng-app="buildApp">
  <title>Build Status</title>
  <link href="" rel="stylesheet">
  <!--link href="" rel="stylesheet"-->

  <div ng-view>

  <script src=""></script>
  <script src=""></script>
  <script src=""></script>
  <script src=""></script>
  <script src="utils.js"></script>
  <script src="app.js"></script>
  <script src="build-factory.js"></script>

Here’s the “view” HTML for the list (in “templates/list.html”). I love the Angular way of specifying Views and Controllers by the way. Note the cool animated CSS for the “in progress” icon.

	.glyphicon-refresh-animate {
		-animation: spin 1s infinite linear;
		-webkit-animation: spin2 1s infinite linear;

	@-webkit-keyframes spin2 {
		from { -webkit-transform: rotate(0deg);}
		to { -webkit-transform: rotate(360deg);}

	@keyframes spin {
		from { transform: scale(1) rotate(0deg);}
		to { transform: scale(1) rotate(360deg);}
	<div class="page-header">
		<h1>Build Status <small>from TeamCity</small></h1>
    <div class="container-fluid">
		<div class="row">
    		<div class="col-md-3" ng-repeat="tile in tiles | orderBy:'status' | filter:nameFilter">
        		<div ng-class="getPanelClass(tile)">
               <h5><span ng-class="getGlyphClass(tile)" aria-hidden="true"></span>   {{ | limitTo:32 }}{{ > 32 ? '...' : ''}}   {{ tile.buildCount > 0 ? '(' + tile.buildCount + ')' : ''}} </h5>
               <p class="panel-body">{{ tile.project }}</p>
  <nav class="navbar navbar-default navbar-fixed-bottom">
  <div class="container-fluid">
    <p class="navbar-text navbar-left">
		<input type="text" ng-model="nameFilter"/>  <span class="glyphicon glyphicon-filter" aria-hidden="true"></span>  
		<span class="glyphicon glyphicon-refresh glyphicon-refresh-animate" ng-hide="!statusVisible"></span>

That’s about it!

I think I summarized how I feel about this project in the introduction. It looks cool and the MVC MVVM ViewModel vibe is a good one. The data binding is simple and works very well. All my gripes are with JavaScript as a language really. I want Linq-style methods and I want classes and objects with sensible scope. I want less syntactic nonsense, maybe the odd => every now and again. I think some or all of that is possible with libraries and new language specs… but I want it without any effort!

One thing I will say: that whole page is less than 300 lines of code. That’s pretty darned cool.

Feel free to download and use the app however you like – just bung in a link to this page!


Mapserver and Leaflet


Leaflet is a very simple but incredibly powerful Javascript mapping library that lets you add interactive maps to your website very easily. Try scrolling and zooming around this one:

For example, to add that map to this page, all I did was add the following code (after reading the Quick-Start Guide):

<div id="danMap" style="height: 200px;"></div>

<link rel="stylesheet" href="" />
<script type="text/javascript" src=""></script>

<script type="text/javascript">
var map ='danMap').setView([51.4, -1.25], 13);	
  attribution: 'Map data © <a href="">OpenStreetMap</a> contributors, <a href="">CC-BY-SA</a>, Imagery © <a href="">Mapbox</a>',
  maxZoom: 18,

You can add points, polygons and visualise all kinds of live data using a simple web service that returns some GeoJson data. It works like a charm on mobile devices too!

Why combine Leaflet with Mapserver?

I have a couple of use-cases that meant I needed to look at combining Leaflet with Mapserver. This turns out to be easy enough as Leaflet can hook up to any tile provider and Mapserver can be set up to serve images as a Web Map Service (WMS).

The first thing I wanted to do is serve up some map data when not connected to the internet. Imagine I am in the middle of nowhere, connected to the Raspberry Pi in the back of the Land Rover via WiFi to my phone or tablet. I have a GPS signal but I don’t have any connection to a map imagery server as there’s no mobile coverage. I need to use mapserver to render some local map data so I can see where I am. This use case has a boring work-related benefit too – it enables you to serve up maps in a web-based mapping application behind a strict corporate firewall.

The other use case is simple: raster data. Lots of the data we deal with where I work is served up as raster data by Mapserver. Imagine it as heat-maps of some KPI value layered on top of a street map.

Setting up Mapserver

There are a couple of things you need to do to get Mapserver to act as a WMS. The first is to add a projection and web metadata element to the root of the map file (as below). After a large amount of head-scratching, wailing and gnashing of teeth I found that the de-facto standard projection for all “internet” map data is EPSG 3857. Make sure you use that EPSG at the root of your map file.

    "wms_title" "Dans Layers and Stuff"
    "wms_onlineresource" ""
    "wms_enable_request" "*"
    "wms_srs" "EPSG:3857"
    "wms_feature_info_mime_type" "text/html"
    "wms_format" "image/png"

The next thing to do is add some extra stuff to every layer in your map. You need to set the STATUS field to ‘on’; add a METADATA element and set the ‘wms_title’ to something sensible; and finally add a projection, specifying the projection the layer data is stored in. As I am using the OS VectorMap District dataset, which is on the OSGB projection I used EPSG 27700.

  NAME         Woodland
  DATA         Woodland
    "wms_title" "Woodland"
  STATUS       on
  TYPE         POLYGON
      COLOR 20 40 20

Connecting it Together

You can then add a new layer to the Leaflet map, connected to your Mapserver. Here I’m using ms4w, the Windows version of Mapserver and hooking it up to a map file in my Dropbox folder. The map file I am using is the one I created for a previous post on mapserver.

L.tileLayer.wms("http://localhost:8001/cgi-bin/mapserv.exe?map=D:\\Dropbox\\Data\\Mapfiles\\", {
			layers: 'Roads,MotorwayJunctions',
			format: 'image/png',
			transparent: true,
			attribution: "Dan's Amazing Roads",
			maxZoom: 18,
			minZoom: 12,

Sadly I don’t have a mapserver instance on the internet, so all I can show here is a couple of screenshots. You’ll just have to take my word for it – it works brilliantly!



Shape Files and SQL Server

Over the last couple of weeks I have been doing a lot of work importing polygons into an SQL server database, using them for some data processing tasks and then exporting the results as KML for display. I thought it’d be worth a post to record how I did it.

Inserting polygons (or any other geometry type) from a shape file to the database can be done with the ogr2ogr tool which ships with the gdal libraries (and with Mapserver for Windows). I knocked up a little batch file to do it:

SET InputShapeFile="D:\Dropbox\Data\SingleView\Brazillian Polygons\BRA_adm3.shp"

SET SqlConnectionString=";Database=danTest;;Pwd=yourpassword;"

SET TEMPFILE="D:\Dropbox\Data\Temp.shp"
SET OGR2OGR="C:\ms4w\tools\gdal-ogr\ogr2ogr.exe"
SET TABLENAME="TestPolygons"

%OGR2OGR% -overwrite -simplify 0.01 %TEMPFILE% %InputShapeFile% -progress

%OGR2OGR% -lco "SHPT=POLYGON" -f "MSSQLSpatial" %SqlConnectionString% %TEMPFILE% -nln %TABLENAME% -progress

The first ogr2ogr call is used to simplify the polygons. The value 0.01 is the minimum length of an edge (in degrees in this case) to be stored. Results of this command are pushed to a temporary shape file set. The second call to ogr2ogr pushes the polygons from the temp file up to a database in Windows Azure. The same code would work for a local SQL Server, you just need to tweak the connection string.

You can use SQL Server Management Studio to show the spatial results of your query, which is nice! Here I just did a “select * from testPolygons” to see the first 5000 polygons from my file.


Sql Server contains all sorts of interesting data processing options, which I’ll look at another time. Here I’ll just skip to the final step – exporting the polygon data from the database to a local KML file.


SET KmlFile="D:\Dropbox\Data\Brazil.kml"

SET SqlConnectionString=";Database=danTest;;Pwd=yourpassword;"

SET TEMPFILE="D:\Dropbox\Data\Temp.shp"
SET OGR2OGR="C:\ms4w\tools\gdal-ogr\ogr2ogr.exe"
SET SQL="select * from TestPolygons"

%OGR2OGR% -lco "SHPT=POLYGON" -f "KML" %KmlFile% -sql %SQL% %SqlConnectionString%  -progress

Obviously you can make the SQL in that command as complex as you like.

Polygons here are from this site which allows you to download various polygon datasets for various countries.

Combining Shape Files

This is one of those things that’s easy when you know how. Just so I don’t forget, here’s how to combine shape files using ogr2ogr.

I wrote it as a batch file to combine all the OSGB grid squares from the OS VectorMap District dataset into a single large data file for use with MapServer.

echo off

set OGR2OGR="C:\ms4w\tools\gdal-ogr\ogr2ogr"
set inputdir="D:\Dropbox\Data\OS VectorMap"
set outputdir="D:\Dropbox\Data\OS VectorMap Big"


set layers=(Airport AdministrativeBoundary Building ElectricityTransmissionLine Foreshore GlassHouse HeritageSite Land MotorwayJunction NamedPlace PublicAmenity RailwayStation RailwayTrack Road RoadTunnel SpotHeight SurfaceWater_Area SurfaceWater_Line TidalBoundary TidalWater Woodland)

del /Q %outputdir%\*.*

FOR %%L IN %layers% DO (
%OGR2OGR% %outputdir%\%%L.shp %inputdir%\SU_%%L.shp

FOR %%T IN %tiles% DO FOR %%L IN %layers% DO (
%OGR2OGR% -update -append %outputdir%\%%L.shp %inputdir%\%%T_%%L.shp -nln %%L

After a little map file jiggery-pokery I can now render a huge map of the UK or tiles with smaller maps without the many layer definitions needed to use ~20 shape file sets.




Serial on Raspberry Pi Arch Linux

So the new version of Arch Linux doesn’t have runlevels, rc.d or any of that nonsense any more. It just has systemd. Super simple if you know how to use it, but a right pain in the backside if you don’t.

I have a little serial GPS module hooked up to my Raspberry Pi via the hardware serial port (ttyAMA0). My old instructions for getting this to work aren’t much use any more. Here’s the new procedure for getting serial data with the minimum of fuss:

1. Disable serial output during boot

Edit /boot/cmdline.txt using your favourite editor. I like nano these days.

sudo nano /boot/cmdline.txt

Remove all chunks of text that mention ttyAMA0 but leave the rest of the line intact. Bits to remove look like:

console=ttyAMA0,115200 kgdboc=ttyAMA0,115200

2. Disable the console on the serial port

This was the new bit for me. The process used to involve commenting out a line in /etc/innitab but that file is long gone.

Systemd uses links in /etc to decide what to start up, so once you find the right one, removing it is easy. You can find the files associated with consoles by doing:

ls /etc/systemd/system/

One of the entries clearly refers to ttyAMA0. It can be removed using the following command:

sudo systemd disable serial-getty@ttyAMA0.service

3. Check you’re getting data

I used minicom for this as it’s very simple to use. First of all, make sure you plug in your device (with the power off, if you’re as clumsy as me!).

sudo pacman -S minicom
minicom -b 4800 -o -D /dev/ttyAMA0

You should see a lovely stream of data. I my case it was a screen full of NMEA sentences. Great stuff!

WiFi, Raspberry Pi, My oh my

Downloaded and installed the latest Arch Linux image for the Raspberry Pi today, only to find that the whole world has changed since I last looked at wireless connectivity!

There’s no rc.d any more, there’s a totally new way to manage networks (wireless and otherwise) and everything I thought I knew is wrong. I think I’m getting old!

After hours of faffing about, it turns out I could have had this nailed with two very simple commands.

Assuming you have a recent Arch Linux, I think you just need to:

1. Use the GUI tool to connect to your WiFi

sudo wifi-menu -o

Select the right network and enter the password when prompted. When you exit the application you should be connected to your router and the tool will have saved a config file to /etc/netctl/wlan0-YourNetwork

2. Change to a static IP address

Weirdly, the above step fails first time with a dhcp error. It saves a valid config file but doesn’t manage to connect. It seems to want to bring up dhcp against eth0 not wlan0 as I’d expect/prefer. Didn’t manage to find out why this is or how to fix it (yet) so just swapped to a static IP!

Use your favorite editor (I like nano) to change the config file just generated removing…


…and adding this in it’s place…


After you’ve done that, repeat step 1 and proceed to step 3. Note that when you run wifi-menu the second time it will show that there is an existing config for your network. You might also find that step 1 works and you don’t need to do step 2 at all, because you might have downloaded a fixed Arch image. In which case, I salute your fortitude!

3. Install the config

Since you probably want to reconnect the wireless after reboots, you need to install the config using the funky new “netctl” tool.

sudo netctl enable wlan0-YourNetwork

Now you should have a working wireless connection after reboots and a raspberry pi that works like your old one did!

Rant: Who moved my cheese?

People always moan about Windows versions that are different to old Windows versions – why Windows 8 took away the start button, why Windows Vista did security totally differently, blah blah blah…

Today I spent several hours digging around looking for something that should be easy, only to find it is indeed very easy… when you know how.

So today I’m quite happy to argue that Linux does exactly the same thing as Windows. I like Arch Linux a lot, and I accept that it’s not for newbies, but things change as often in Linux world as they do in Windows world. On both sides of the fence, Google is your only hope if you want to keep up to date with the changing software that underpins everything you do.