Stuff I made in 2017

This year has been a year of making – mostly jobs around the house and garden.  It’s been great fun and I thought I’d document some of the projects here for posterity.

Mini-Billy

First make of the year was a mini Ikea Billy bookcase. It only took a couple of hours, but it was very satisfying indeed (and saw me joining the millions of Ikea hackers on the internet).

We’d put three large bookcases across a wall in our living room, but this left a bit of a gap at one side, not quite wide enough for one of the smaller bookcases.  Since I didn’t have anything but the bandsaw at the time, I made a crude fence to ensure a straight and consistent cut and proceeded to chop all the shelves down to size.  Since you can’t see the sides of the bookcase itself, you can’t see the wood screws I used to secure the shelves!

The Lawn

Undoubtedly the hardest thing I’ve ever made (physically at least), this project saw me carting several tons of soil and hardcore backwards and forwards, and laying turf in rainy and miserable weather.  The results are pretty awesome though.  Our garden now has plenty of space for the kids to practice cartwheels and generally go bananas.

Laying the brick edging round the lawn was incredibly difficult because I foolishly decided to mix all the cement by hand. For future projects I’ll buy or hire a cement mixer!

The Barbecue

Having been given a welder (thanks Dad!) and an oil drum (thanks Mother in Law!) for Christmas, and with the lawn out of action, the next project on the list was a new barbecue.  I used some heavy gauge, 100-year old steel fence posts rescued from a friend’s garden to make a solid base, then chopped up the oil drum before adding hinges, a chimney and a grill.

The chimney is made of old welding gas bottles, chopped and welded into a tube and the side tables and bottom shelf are made of pallet wood from the turf delivery for the lawn so the total materials cost was no more than some hinges and a couple of oven shelves from eBay!

The Playhouse

With a newly cleared gap in the bottom corner of the garden, it seemed only fitting to build a playhouse for the girls!  This was pretty expensive, with most of the timber, cladding and OSB coming from the timber merchants.  Not to mention over 800 wood screws!

I built the house in pre-fab sections, since there was very little room on the “site”, I had to complete each section then lift it into place and secure it with screws on the inside.  I deliberately left everything as “wonky” as possible, aiming for a gingerbread house sort of look.

The windows are made of 10mm perspex (found in a skip!) which worked really well as it’s the same thickness as the OSB skin on the walls.  This meant I could attach the windows and OSB together in square sections then screw the window frames and cladding over the top in whatever shape I wanted.

The roof is OSB too, with 250+ wooden tiles made from pallet wood nailed on.  I should have left bigger gaps between the tiles though, a few months on they have expanded in the rain and lifted in places.

The Mood Bot

I haven’t done many software projects this year, but the Mood Bot is one I’m quite happy with.  It’s a Slack integration which tests a team’s mood and draws some pretty graphs.  You can read more about it here.

The Coffee Table

Well, actually it’s more of a TV stand for the loft conversion in our house, which we converted into a study/second living room this year.  The table top is a slab of oak and the legs are 25x50mm steel box section.  I’m pretty happy with the results, but I need to find a tougher, child-resistant finish!

The Map

While looking for an unusual picture for the top room, we thought about buying some kind of world map.  This led me to start looking at Mapserver again, and ultimately the creation of this map of Thatcham.  We got it printed on a canvas by one of those online photo companies and I matched the colours to our garish new sofa!

The Skull Mask

For one of this year’s work events we were supposed to dress for a “masked ball”.  I couldn’t find any masks I liked on the internet and I left it too late to go to a fancy dress shop, so the weekend before the party I headed out to the garage and started snipping and hammering some scrap steel from an old office letter tray.  I was very pleased with the results, but sadly proceeded to get so drunk I left the mask itself behind!  I might have to make another…

So all in all it’s been a make-tastic year.  I’ll have to start thinking about what projects to do in 2018 – though I suspect a new desk for the top room, a snazzy interior for the playhouse and maybe an electric rotisserie for the barbecue are on the cards at some stage…

Controlling a TP-Link Smart Bulb with Python and Requests

We recently added a new build status indicator in the office, using an excellent TP-Link LB130 Smart Bulb. Though these bulbs are pretty expensive, they are super-simple to set up and control via a simple REST API.

It took quite a lot of googling to find the correct commands to send through the API to control the bulb, but once we’d found the answer, it was incredibly simple.

There are many Python libraries out there for controlling these bulbs directly via the local network, but the benefit of using the REST API is that you can control the bulb from anywhere. You are also able to “discover” the bulbs associated with your TP-Link Kasa account, so you don’t need to know the IP address or MAC of your bulbs.

Here’s the code…

import unittest
import requests
import uuid
import json
import random

USERNAME = 'your.email@address.com'
PASSWORD = 'YourPassword123'


class TpLinkApiTests(unittest.TestCase):
    def test_change_bulb_colour(self):
        # First step is to get a token by authenticating with your username (email) and password
        payload = {
            "method": "login",
            "params":
                {
                    "appType": "Kasa_Android",
                    "cloudUserName": USERNAME,
                    "cloudPassword": PASSWORD,
                    "terminalUUID": str(uuid.uuid4())
                }
        }
        response = requests.post("https://wap.tplinkcloud.com/", json=payload)
        self.assertEqual(200, response.status_code)
        obj = json.loads(response.content)
        token = obj["result"]["token"]

        # Find the bulb we want to change
        payload = {"method": "getDeviceList"}
        response = requests.post("https://wap.tplinkcloud.com?token={0}".format(token), json=payload)
        self.assertEqual(200, response.status_code)

        # The JSON returned contains a list of devices. You could filter by name etc, but here we'll just use the first
        obj = json.loads(response.content)
        bulb = obj["result"]["deviceList"][0]

        # The bulb object contains a 'regional' address for control commands
        app_server_url = bulb["appServerUrl"]
        # Also grab the bulbs ID
        device_id = bulb["deviceId"]

        # Send a command through to the bulb to change it's colour
        # This is the command for the bulb itself...
        bulb_command = {
            "smartlife.iot.smartbulb.lightingservice": {
                "transition_light_state": {
                    "on_off": 1,
                    "brightness": 100,
                    "hue": random.randint(1, 360), # Random colour
                    "saturation": 100
                }
            }
        }
        # ...which is escaped and passed within the JSON payload which we post to the API
        payload = {
            "method": "passthrough",
            "params": {
                "deviceId": device_id,
                "requestData": json.dumps(bulb_command)  # Request data needs to be escaped, it's a string!
            }
        }
        # Remember to use the app server URL, not the root one we authenticated with
        response = requests.post("{0}?token={1}".format(app_server_url, token), json=payload)
        self.assertEqual(200, response.status_code)

        # Hopefully the bulb just changed colour!
        print response.content

Mood Bot – a Serverless Slack Integration

Pull me on GitHub!

Mood Bot

So it’s been a tradition in my office to use Slack to gauge the team’s mood once a week. Previously our PM would post a message asking for feedback and people would add a reaction to show how they were feeling. This worked fine, though there were a couple of issues: firstly it was pretty hard to interpret the weird collection of party parrots and doges, and secondly people tend to follow the herd when they can see how others have reacted.

Here’s my idea for the new, automated workflow…

Screenshot 2017-05-03 12.20.48

By far the cheapest and maybe the simplest way to host all of the code to do this is to “go serverless” using many of the cool features available on AWS to host code, databases and APIs on a pay-per-use basis.  Here’t the technical architecture…

MoodBot

Based on the above there are three broad areas for development: send the webhook to Slack; deal with responses when users click the buttons and serve up a chart showing the results for the week.

Sending the Web Hook

Slack allows you to post Interactive Messages using an Incoming Webhook. In order to do this you’ll need to add a new slack bot integration using their very friendly web UI. I called mine “MoodBot”. Once you have a bot set up, you need to enable “Incoming Webhooks” and add the target URL to an environment variable (see here or here for more details).

The format of the message you send needs to be something like the following.  Note that the “interactive” part of the message is included as an attachment.

const message = {
  "text": ":thermometer: @channel *Time for a Team Temp Check!* @channel :thermometer: \n _Click as many times as you like, only your last vote will be counted._",
  "channel": "@laurence.hubbard",
  "attachments": [
    {
      "text": "How are you feeling this week?",
      "fallback": "I am unable to understand your feelings. Too deep maybe?",
      "callback_id": "mood_survey",
      "color": "#3AA3E3",
      "actions": [
        {
          "name": "mood",
          "text": "Good :+1:",
          "type": "button",
          "value": "good"
        },
        {
          "name": "mood",
          "text": "Meh :neutral_face:",
          "type": "button",
          "value": "meh"
        },
        {
          "name": "mood",
          "text": "Bad :-1:",
          "type": "button",
          "value": "bad"
        },
        {
          "name": "mood",
          "text": "Terrible :rage:",
          "type": "button",
          "value": "terrible"
        },
        {
          "name": "mood",
          "text": "AWESOME!!!   :doge:",
          "type": "button",
          "value": "awesome"
        }
      ]
    }
  ]
}

This gives you a slack message looking like this:

The webhook is sent by a Lambda function, which is triggered crontab-style by a CloudWatch event rule.  The Lambda looks like this:

const AWS = require('aws-sdk');
const url = require('url');
const https = require('https');

const kmsEncryptedHookUrl = process.env.kmsEncryptedHookUrl;
let hookUrl;

function postMessage(inputData, callback) {
    const body = JSON.stringify(message);
    const options = url.parse(hookUrl);
    options.method = 'POST';
    options.headers = {
        'Content-Type': 'application/json',
        'Content-Length': Buffer.byteLength(body),
    };

    const postReq = https.request(options, (res) => {
        const chunks = [];
        res.setEncoding('utf8');
        res.on('data', (chunk) => chunks.push(chunk));
        res.on('end', () => {
            if (callback) {
                callback({
                    body: chunks.join(''),
                    statusCode: res.statusCode,
                    statusMessage: res.statusMessage,
                });
            }
        });
        return res;
    });

    postReq.write(body);
    postReq.end();
}

function processEvent(slackMessage, callback) {
    slackMessage.channel = slackChannel;
    
    postMessage(slackMessage, (response) => {
        if (response.statusCode < 400) {
            console.info('Message posted successfully');
            callback(null);
        } else if (response.statusCode < 500) { console.error(`Error posting message to Slack API: ${response.statusCode} - ${response.statusMessage}`); callback(null); // Don't retry because the error is due to a problem with the request } else { // Let Lambda retry callback(`Server error when processing message: ${response.statusCode} - ${response.statusMessage}`); } }); } exports.handler = (event, context, callback) => {
    console.log("Sending a temp check request")
    
    if (hookUrl) {
        // Container reuse, simply process with the key in memory
        processEvent(event, callback);
    } else if (kmsEncryptedHookUrl && kmsEncryptedHookUrl !== '<kmsEncryptedHookUrl>') {
        const encryptedBuf = new Buffer(kmsEncryptedHookUrl, 'base64');
        const cipherText = { CiphertextBlob: encryptedBuf };

        const kms = new AWS.KMS();
        kms.decrypt(cipherText, (err, data) => {
            if (err) {
                console.log('Decrypt error:', err);
                return callback(err);
            }
            hookUrl = `https://${data.Plaintext.toString('ascii')}`;
            processEvent(event, callback);
        });
    } else {
        callback('Hook URL has not been set.');
    }
};

Setting up the rule to trigger the event is pretty simple. Log into the AWS console, select CloudWatch and choose Events -> Rules from the menu on the left. You can specify when the rule will run using a crontab line.  I used…

0 09 ? * WED *

Which will run at 9am (GMT) every Wednesday.  All this is set up via a reasonably clunky web interface!

Collating Responses

This is the most complicated bit (and there’s an extra tricky bit to deal with too). To handle the responses when users click buttons on the interactive Slack message you need four things: 1. A lambda function to handle the POST request and push data to a database, 2. an API Gateway resource to provide an HTTP end-point, translate the request and forward it to the Lambda function, 3. a database to store the data and finally 4. a config setting in Slack to tell it where to send the POST.

Here’s the code for my Lambda function. It’s simple enough – it just takes the JSON in the incoming request, grabs the bits it wants and adds a few dates and times to create another JSON object to post to DynamoDB. The response sent back to slack is a replacement message, which will overwrite the one already in the channel. Here I add a list of users who have clicked so far (a better man would have pulled this list from the DB!).

var AWS = require('aws-sdk');

var dynamo = new AWS.DynamoDB.DocumentClient();
const table = "MoodResponses";

function updateVoters(original, voter) {
    var updated = original;
    
    var msg = "\nVoted so far: ";
    var comma = true;
    if(!updated.includes(msg)) {
        updated = updated + msg;
        comma = false;
    }
    
    if(!updated.includes(voter)) {
        if(comma) {
            updated = updated + ", ";
        }
        
        updated = updated + "<@" + voter + ">";
    }

    return updated;
}

Date.prototype.getWeek = function() {
        var onejan = new Date(this.getFullYear(), 0, 1);
        return Math.ceil((((this - onejan) / 86400000) + onejan.getDay() + 1) / 7);
    }

exports.handler = function(event, context, callback) {
    
    console.log('Received Slack Message: ', JSON.stringify(event, null, 2));
    
    var mood = event.actions[0].value;
    var date = new Date(Number(event.message_ts) * 1000);
    var key = event.user.id + "@" + date.getFullYear() + "-" + date.getWeek();
    var record = {
        TableName: table,
        Item: {
            key: key,
            message_ts: Number(event.message_ts),
            username: event.user.name,
            user_id: event.user.id,
            mood: mood,
            date_string: date.toISOString(),
            day: date.getDate(),
            month: (date.getMonth() + 1),
            week: date.getWeek(),
            year: date.getFullYear()
        }
    };
    
    console.log("Created mood record: " + JSON.stringify(record, null, 2));

    dynamo.put(record, function(err, data) {
        if (err) {
            console.error("Unable to add item. Error JSON:", JSON.stringify(err, null, 2));
                
            callback(null, {
                  text: "An error occurred inserting to DynamoDB. Error attached.",
                  attachments: [{text: JSON.stringify(err, null, 2)}],
                  replace_original: false
                });
        } else {
            console.log("Added item:", JSON.stringify(record, null, 2));
            
            callback(null, {
                  text: updateVoters(event.original_message.text, event.user.id),
                  attachments: event.original_message.attachments,
                  replace_original: true
                });
        }
    });
};

Setting up the API Gateway (The Extra Tricky Bit)

Setting up the API Gateway should be simple enough – you add a new API then a new resource then a new POST method. Then configure the method to forward requests to the Lambda function you just created. However, there are a couple of issues.

Firstly, you need to enable cross site access (CORS) which is easy enough – you just select “Enable CORS” from the “Actions” dropdown. This will open your method up to calls from other sites.

The second and far more infuriating issue is that Slack’s Interactive Buttons send the data in a funky way, encoding it weirdly in the message body rather than just posting JSON as all the other calls do.  After a couple of days of intermittent head-scratching I finally found this Gist, which contains the code to fix the problem:

This code needs to be placed into a Body Mapping Template for your POST method within the AWS API Gateway UI. The following screenshot hopefully give you enough of a clue on how to set this up.  Now, when Slack sends the malformed (IMHO) POST, the API gateway will reformat it and pass it through to your lambda function as if it were a normal JSON payload.

Screenshot 2017-04-24 18.45.47

Database Setup

I decided to use DynamoDB – Amazon’s “Document Database as a Service” (DDaaS?). I’m not sure it’s the perfect choice for this work, since querying is pretty limited, but it is very cheap and incredibly simple to use.

For this step, just use the web UI to create a new table called “MoodResponses”. I used an “id” field as the index.  The lambda creates “id” by concatenating the user ID and current week. This means you automatically limit each user to a single vote per week, which is exactly the functionality I was looking for – more or less for free!

Slack Request URL

Final step is very simple – use the Slack admin UI for your bot to add the address of your API resource as the target for interactive message callbacks.  Go to the admin page and select Features -> Interactive Messages from the panel on the left and paste in the URL of your API Gateway method.

Displaying Results

Though there are more boxes on the diagram below, this is actually the easiest step by far. We serve up a simple D3js “single page app” direct from S3 as static content. This SPA page calls a GET method on the REST service we created above which in turn calls a Lambda function. The Lambda hits out database, pulls out the results and sends them back as a JSON payload.

There’s not much more to explain, so I’ll just link to a Fiddle which includes the code for my front end – this one actually hits my production database, so you’ll be able to see how my team feel!

Serving this code up as a static HTML file is very easy: Create an index.html document and add the javascript, HTML and CSS from the fiddle; create a new S3 bucket and, in the properties for the bucket, enable “Static Website Hosting”; upload your index.html file to the bucket, select it and select “Make Public” from the “Actions” dropdown.

Here’s the code for the Lambda function which is servicing the GET request:

var AWS = require("aws-sdk");

var docClient = new AWS.DynamoDB.DocumentClient();

Array.prototype.countBy = function(key) {
  return this.reduce(function(rv, x) {
    rv[x[key]] = (rv[x[key]] || 0) + 1;
    return rv;
  }, {});
};

Date.prototype.getWeek = function() {
    var onejan = new Date(this.getFullYear(), 0, 1);
    return Math.ceil((((this - onejan) / 86400000) + onejan.getDay() + 1) / 7);
};

Date.prototype.previousWeek = function() {
    return new Date(this.getTime() - (7 * 24 * 60 * 60 * 1000));
};

function forWeek(week) {
    return {
        TableName : "MoodResponses",
        IndexName: 'week-user_id-index',
        KeyConditionExpression: "#wk = :week",
        ExpressionAttributeNames:{
            "#wk": "week"
        },
        ExpressionAttributeValues: {
            ":week":week
        }
    };
}

function handleError(err, callback) {
    console.error("Unable to query. Error:", JSON.stringify(err, null, 2));
    callback(null, {"error": JSON.stringify(err, null, 2)});
}

function handleData(data, callback, week) {
    console.log("Query succeeded.");
            
    var results = {
        week: week,
        moods: data.Items.countBy("mood")
    };

    callback(null, results);
}

function runFor(date, tries, callback) {
    var week = date.getWeek();    
    console.log('Fetching mood results for week: ' + week);
    
    docClient.query(forWeek(week), function(err, data) {
        if (err) {
            handleError(err, callback);
        } else if(data.Items.length > 0 || tries <= 0) {
            handleData(data, callback, week);
        } else {
            runFor(date.previousWeek(), tries - 1, callback);
        }
    });
}

exports.handler = function(event, context, callback) {
    runFor(new Date(), 1, callback);
};

One Last Thing!

Dynamo can only query against fields which are part of an index. Here we need to query by week number, so I added a new index to my Dynamo table by week. This took 5 minutes to update (even though the table only had 5 records in it at the time!) but was simple enough to do.  If you look at the code above, you can see where I specify the index in the query parameters.

Wrap Up

So this all works really well.  There’s lots left to do: making the results look prettier and looking at how the sourcecode is deployed and managed being two things at the top of my list.

Slack is the industry standard tool for team collaboration these days, and bots and integrations amp up the power and productivity at your disposal. Build status, Jira tickets, team morale, coffee orders and whatever else you fancy can all be brought together with conversational APIs, simplifying just about everything.

On the AWS side, there’s still a lot of googling required to build this sort of thing, and sometimes information is scarce. Those who enjoy building “proper applications” using IDEs like IntelliJ or Visual Studio are going to find this frustrating – the pieces feel disjoint and uncontrolled sometimes.  However, all in all it’s pretty cool what you can do without a single server in the mix.

It’s hard to deny that this development model is going to be the de-facto standard within the next couple of years, as it’s just so damned quick and simple. So get out and get serverless!

Evolutionary Algorithm: Playable Demo

Here I’m combining a bit of visualisation with my other favourite subject – the Evolutionary Algorithm (or Genetic Algorithm if you prefer).  I’m not going to write anything about the properties of the algorithm – you can just play with the controls below the chart and see how the different settings effect its ability to find a good solution, adapt to changes and explore the problem space.

The problem: Find a value of x which maximises the value of y. The function is a set of sinusoidal waves of varying frequency and amplitude. The blue line shows the “fitness” for each value of x.

Basically, a population of different solutions is maintained – in this case, each solution is simply a value for x. Every individual has a fitness which can be calculated based on it’s value. Each iteration (100ms here) a solution is removed from the population – killed by selective pressure. Fitter individuals have a greater chance at surviving, less fit individuals have a less of a chance.

A replacement solution is “bred” each iteration, to replace the solution killed-off by selective pressure. This new individual is generated by combining the “genetic material” of one or more parents. In this case, just by taking the x value of a single parent. Importantly, a mutation is applied to the new solution – this is key to exploring the problem space effectively.

And that’s all there is to an Evolutionary Algorithm – it’s just a way of finding the right combination of input variables to maximise some arbitrarily complex fitness function. It does this through a guided random search.

screenshot-2016-11-11-17-25-56

Conway’s Game of Life

For some reason, I have started playing with D3.js a lot. Not sure why – maybe it’s just because I managed to get it integrated with WordPress. Anyway, I recently knocked up this version of Conway’s Game of Life. Most of the clever stuff is stolen from this snippet on bl.ocks.org but I made a few subtle changes to the code and the visual style.

I don’t take credit for much here – I just wanted to record that I’d spent a couple of train journeys building something cool!

screenshot-2016-11-09-18-39-39