Using a BufferBlock to Read and process in Parallel

Wrote an app this week – top secret of course – to load data from a database and process the contents. The reading from the database is the slow part and the processing takes slightly less time. I decided it might help if I could read a batch of results into memory and process it while loading the next batch.

Batching was dead easy, I found an excellent extension method on the internet that batches up an enumerable and yields you a sequence of arrays. The code looks like this, in case you can’t be bothered to click the link:

public static IEnumerable<T[]> Batch<T>(this IEnumerable<T> sequence, int batchSize)
    var batch = new List<T>(batchSize);

    foreach (var item in sequence)

        if (batch.Count >= batchSize)
            yield return batch.ToArray();

    if (batch.Count > 0)
        yield return batch.ToArray();

That works really well, but it doesn’t give me the parallel read and process I’m looking for. After a large amount of research, some help from an esteemed colleague and quite a bit of inappropriate language, I ended up with the following. It uses the BufferBlock class which is a new thing from Microsoft’s new Dataflow Pipeline libraries (which provide all sorts of very useful stuff which I may well write an article on at a later date). The BufferBlock marshals data over thread boundaries in a very clean and simple way.

public static IEnumerable<T[]> BatchAsync<T>(this IEnumerable<T> sequence, int batchSize)
    BufferBlock<T[]> buffer = new BufferBlock<T[]>();

    var reader = new Thread(() =>
            foreach (var batch in sequence.Batch(batchSize))
        }) { Name = "Batch Reader Async" };

    T[] blocktoProcess;
    while ((blocktoProcess = buffer.Receive()) != null)
        yield return blocktoProcess;

The database read is done on a new thread and data is pulled back to the calling thread in batches. This makes for nice clean code on the consumer side!

Bulk Inserts to SQL Server Azure

This has been driving me mad all day, so I’ll document it here if only so I don’t forget!

SQL Server Azure doesn’t support the “traditional” batch insert stuff and you can’t just send an SQL file with 50,000+ “insert into…” statements either as the query processor will run out of space.

What you can do is run a tool called BCP.  This tool is specially designed for loading large datasets into the cloud and is perfect for all your dimension table needs.  The tool takes a tab delimited file as input as well as a huge list of command line parameters.

C:\>bcp [myDatabase].[dbo].[TableName] in C:\Users\Dan.Taylor\Dropbox\Stuff\DansDataFile.txt -c -U dansUsername
@dansAzureServerName -P <password> -S -e c:\errors.txt

Piping the errored records to a text file is very helpful!


The first column in my table is an auto-generated ID, so I make sure that every line in my file starts with a tab. This basically nulls the first column, letting the database generate an ID as normal.

Also, BCP can not parse dates, times or datetimes from strings. I haven’t found a nice way around this yet – in the end I changed the data type of the column because I don’t really care about the date in my dataset very much anyway! Others have said they created a temporary table and then select/inserted the data over to the real table with a date conversion.

Tracking Kanban with TFS

Kanban is a great way to manage your bug backlog.  It’s much better than Scrum simply because of the nature of bugs as compared to user stories. Scrum is all about making firm commitments based on estimates but bugs are very hard to estimate up-front. Generally when you’ve looked hard enough into the code to find the problem, you are in a position to fix it very quickly. Bug fixing is essentially a research task – like a spike – so time-boxing the work makes much more sense.

Set up a prioritised backlog and blast off the top as many bugs as possible in the time you’ve set aside – Kanban Style.  This works very well but, as with most agile approaches, it leaves old fashioned managers a bit grumpy.  They want to track your productivity and it’s fair to say that you should too because that’s how you spot impediments (plus it’s always good to show off).

Scrum-style burn downs don’t work with Kanban because they track progress against some committed target.  The answer is the Cumulative Flow Diagram:


So I did some tweaking to my Information Radiator to add a page showing the CFD for the last 60 days of one of our projects.  The data comes out of TFS via the C# API and a WIQL query – which has a very nice historical query feature which I’ll explain below.

Cumulative Flow Diagrams Explained

Cumulative flow diagrams couldn’t be simpler.  Like a burn-up chart they show a running total of the bugs fixed over time.  Since bugs aren’t estimated, the Y axis shows the bug count.  In the chart above the X axis is in days but I guess you could do weeks or even hours if you like.  In addition to the “fixed bugs” series, there are also stacked series for other states: “committed”, “in development” and “in QA”.

The benefit of showing the other issue states is that it gives you a readout on how the process is working.  The QA and development series should generally be the same thickness.  If the QA area gets fatter than the development area then you have a bottleneck in QA.  If the development series gets too fat then you’re spread too thinly – you have an impediment in development or need to think about your Kanban limit.

Note how there are a couple of “steps” on the left of my graph.  Those correspond to the first couple of sprints in which we used TFS. The team weren’t familiar with it, so work item states were generally changed at the end of the sprint.  As time went on we got better at updating the system and the steps turned into a nice looking slope.

Historical Queries in TFS 2012

It’s not every day that I openly applaud Microsoft for doing something brilliant and until now I’ve never been that cheerful about TFS.  But… the historical querying in WIQL (work item query language) is bloody brilliant!

Drawing a CFD chart depends on an ability to get the historical state of any issue in the system at a specified point in time.  In WIQL this is done using the “AsOf” keyword:

Select [ID], [Title], [Effort - Microsoft Visual Studio Scrum 2_0], [Assigned To]
From WorkItems
  [Team Project] = 'Project'
  [Work Item Type] = 'Bug'
  [Iteration Path] under 'Project\Release'
AsOf '21/01/2013'

So the algorithm for drawing the CFD is pretty simple:

  • Grab the sprints for the project in question and use them to get the start and end dates for your chart
  • For each day on the X axis
    • Run a WIQL statement to get the state of all the bugs in the project on that date
    • Use linq to count issues in the various states you’re showing on the graph series
    • Populate a list of view model/data objects (one for each X value)
  • Throw the values at the chart

The only complications were the fact that the WPF Toolkit chart doesn’t support stacked area series (so I had to do it myself in the view model) and that getting data on group membership from TFS is very hard and very slow (so I build a cache of dev and QA group members up front and do comparisons on the display name).

TFS Dashboard Fun

It turns out it’s really quite simple to get data out of Team Foundation Server.  You can do it using the built-in reporting tools, but that seems a bit boring to me.  After the Big Cheese agreed to buy us a super-sized telly to get some project propaganda into the office I decided to whack out a WPF app.

Three useful bits of information worth recording here.  How to get information out of TFS, how to draw nice looking graphs and how to make a borderless, click-anywhere-drag style app without the windows title bar.

Getting Information From TFS

I’m not going to go into huge depth on how I structured by queries as everyone uses TFS slightly differently.  If you’ve got the smarts to knock up some TFS queries – and if you don’t you might need to learn anyway – then you should be good to go with these code snippets.

Get a list of sprints (iterations). This one was a bit tricky and took some time to work out!

        public void CanGetListOfSprints()
            var tpc = new TfsTeamProjectCollection(tfsUri);
            var workItemStore = (WorkItemStore)tpc.GetService(typeof(WorkItemStore));

            IEnumerable<string> names =
                from Node node in workItemStore.Projects[projectName].IterationRootNodes[releaseName].ChildNodes
                select node.Name;

            Assert.That(names, Is.Not.Empty);

Get a list of stories waiting for signoff by product managers.

        public void CanGetStoriesAwaitingProductManagerSignoff()
            var tpc = new TfsTeamProjectCollection(tfsUri);
            var workItemStore = (WorkItemStore)tpc.GetService(typeof(WorkItemStore));

            StringBuilder queryText = new StringBuilder();
            queryText.AppendLine(@"Select [ID], [Title]");
            queryText.AppendLine(@"From WorkItems");
            queryText.AppendFormat(" [Team Project] = '{0}'\n", projectName);
            queryText.AppendLine(@" [Work Item Type] = 'Product Backlog Item'");
            queryText.AppendFormat(" [Iteration Path] under '{0}\\{1}'\n", projectName, releaseName);
            queryText.AppendLine(@" [State] = 'Committed'");
            queryText.AppendFormat(@" [Assigned To] in ( {0} )", productManagers);

            IEnumerable<string> workItemNames = from WorkItem wi in workItemStore.Query(queryText.ToString())
                                                select wi.Title;

            Assert.That(workItemNames, Is.Not.Empty);

WPF Line Charts

For the charts I used the WPF toolkit. It’s not seen much development in recent years but it’s still an easy and quick way to get a chart on screen. Here are some XAML snippets.

Here’s the chart itself. Two line series here. You select the ItemsSource: list of objects corresponding to points on the line; DependentValuePath: name of a property on the object to use as the value for the Y axis; IndependentValuePath: value/label for the X axis.

            <chart:Chart x:Name="TheChart" Width="1024" Height="728" Foreground="White" FontWeight="Bold" FontSize="18"
                         Template="{StaticResource ChartTemplate}"
                         LegendStyle="{StaticResource InvisibleStyle}" >
                <chart:LineSeries DependentValuePath="ProjectedBurndown" Foreground="White" IndependentValuePath="SprintName" ItemsSource="{Binding Sprints}" IsSelectionEnabled="False"
                               PolylineStyle="{StaticResource ProjectedLineStyle}" DataPointStyle="{StaticResource ProjectedDataPointStyle}"/>
                <chart:LineSeries DependentValuePath="ActualBurndown" IndependentValuePath="SprintName" ItemsSource="{Binding Sprints}" IsSelectionEnabled="False"
                               PolylineStyle="{StaticResource ActualLineStyle}" DataPointStyle="{StaticResource ActualDataPointStyle}"/>

This one is used to hide the legend:

        <Style x:Key="InvisibleStyle" TargetType="Control">
            <Setter Property="Width" Value="0" />

Control Template to give better control over the look of the chart area:

        <ControlTemplate TargetType="chart:Chart" x:Key="ChartTemplate">
            <Border Background="{TemplateBinding Background}" BorderBrush="{TemplateBinding BorderBrush}" 
                    BorderThickness="{TemplateBinding BorderThickness}" Padding="{TemplateBinding Padding}">
                    <primitives:EdgePanel x:Name="ChartArea" Style="{TemplateBinding ChartAreaStyle}">
                        <Grid Canvas.ZIndex="-1" Style="{StaticResource PlotAreaStyle}" />

Two styles needed to render the lines with no data points and custom colour etc:

        <Style x:Key="ProjectedDataPointStyle" TargetType="Control">
            <Setter Property="Width" Value="0" />
            <Setter Property="Background" Value="LightGray"/>
        <Style TargetType="Polyline" x:Key="ProjectedLineStyle">
            <Setter Property="StrokeThickness" Value="4"/>
            <Setter Property="StrokeDashArray" Value="2,1"/>

Borderless Click-Drag

This last one was insanely simple but I don’t want to forget, so here it is. All code is in MainWindow.xaml. Adding these few lines gives you an app with no border that can be dragged by clicking and dragging anywhere. I also added a double-click-to-toggle-maximised-state feature. Use Viewboxes libreally in your XAML to make it look nice!

public partial class MainWindow 
        public MainWindow()
            DataContext = new BurndownViewModel(new Uri("http://whatnot/tfs/thingy"));


            MouseLeftButtonDown += StartDragMove;
            MouseDoubleClick += ToggleWindowState;

        private void ToggleWindowState(object sender, MouseButtonEventArgs e)
            WindowState = (WindowState == WindowState.Maximized) ? WindowState.Normal : WindowState.Maximized;

        private void StartDragMove(object sender, MouseButtonEventArgs e)

        private void CloseButtonClicked(object sender, RoutedEventArgs e)

Changing Project Name in TFS 2012

Renaming a project in Team Foundation Server 2012. Seems like a sensible thing to do and not something that would take you very much time. Except for the fact that…

You can’t do it!  It can’t be done.  Simple as that.

I tried that answer with my boss but he didn’t seem happy.  It’s true though; Microsoft are very clear about it.  That doesn’t change the fact that we have a project with a confusing name and a very clear need to change that name.  Nor does it change the fact that muggins got lumbered with the job of sorting this mess out.

After some googling and much swearing we finally managed to get it to work.  The solution isn’t to rename the project but to create a new one and move all your work items over. Here’s how I did it.  Note that this post only covers work items, not source control.

  • Create a new team project with the correct name
  • Make sure the work item templates match and all your customisations have been applied to the new project
  • Use the TFS Integration Tool to migrate all your work items over

Creating a New Project

I always forget where to click, but the next image makes it clear. Make sure you set the name and choose the correct project template.

Moving Over Custom Work Item Types

If you haven’t customised the project template at all then you can skip this step. If you have made any changes you need to make sure you copy them to the new project before you migrate. If you don’t want to move your changes over there are clever things you can do with an XML mapping file but I’m not going to cover that today.

  • Export work item types from your old project
    • Tools → Process Editor → Work Item Types → Export WIT
    • Choose the item type – make sure you export all the types you’ve changed. You’ll have to do them one at a time.
    • Save it to a file somewhere sensible
  • Restart Visual Studio or it won’t pick up the new project! This annoying but true. If you’ve not restarted VS since you added the new project, do it now.
  • Import the work item types into the new project
    • Tools → Process Editor → Work Item Types → Import WIT
    • Select the file you saved
    • Select the new project
    • Do this for every file you saved
  • Check that the work item templates now look as you’d expect them to

Migrating Work Items

Work items can be moved between projects using the TFS Integration Tools from Microsoft.  You can download them here, though I’d suggest you search for the latest version, just in case! Install the tool on your TFS server and run it from the start menu.

Run the tool and select “Create New” from the menu on the left.

Choose the template: C:\Program Files (x86)\Microsoft Team Foundation Server Integration Tools\Configurations\Team Foundation Server\WorkItemTracking.xml

In the configuration you need to select a project on the left and on the right. For a one way migration the left is the Source and the right is the Target. Make sure you give the migration a sensible name and choose “One Way Migration”. Click the “Configure” button on the left and select your old project. Click the “Configure” button on the right and select your new project.

Click “Save to Database” then click the “Start” button on the menu bar on the left.

Off it goes…

Looking good so far…

Almost there…


Possible Issue With Permissions

When I first tried to run a migration I hit a permissions error.  Three warnings popped up as soon as I started the migration, the full text of which is below:

Microsoft.TeamFoundation.Migration.Tfs2010WitAdapter.PermissionException: TFS WIT bypass-rule submission is enabled. However, the migration service account 'Administrator' is not in the Service Accounts Group on server 'http://zx81:8080/tfs/geo'.
   at Microsoft.TeamFoundation.Migration.Tfs2010WitAdapter.VersionSpecificUtils.CheckBypassRulePermission(TfsTeamProjectCollection tfs)
   at Microsoft.TeamFoundation.Migration.Tfs2010WitAdapter.TfsCore.CheckBypassRulePermission()
   at Microsoft.TeamFoundation.Migration.Tfs2010WitAdapter.TfsWITMigrationProvider.InitializeTfsClient()
   at Microsoft.TeamFoundation.Migration.Tfs2010WitAdapter.TfsWITMigrationProvider.InitializeClient()
   at Microsoft.TeamFoundation.Migration.Toolkit.MigrationEngine.Initialize(Int32 sessionRunId)

“…the migration service account ‘Administrator’ is not in the Service Accounts Group” being the important thing here. The annoying thing is that the group in question turns out to be hidden. Luckily, somebody has posted a solution. Basically you need to run the following command on your TFS server:

c:\Program Files\Microsoft Team Foundation Server 11.0\Tools\tfssecurity /g+ "Team Foundation Service Accounts" n:YourMachine\YourUser ALLOW /server:http://yourserver:8080/tfs

What’s not copied?

The big thing for me is queries. I have 20+ custom queries that all relate to different parts of the process. All these will need to be moved over by hand as the tool doesn’t do it for you.

If you use the reporting services and so on you’ll find that they are not dealt with properly either.

Running it Again

If you have a period of handover between the two projects you might want to run them in parallel or have a rolling handover. The great thing about the migration you set up is that you can run it again and copy any changed made in the old project since your initial copy. This means you can get the new project set up, sort out all your documentation and stuff and have a structured migration plan for your team.

The ASCII Speedometer

Once I’d got Mono up and running, the first little project I did with the Raspberry Pi was to hook up an old GPS module and use it to create a text based speedometer for the car.  It was the first step I took towards making The Duke sentient building an on-board computer for my Land Rover.  I was fun to do and raised a smile with the people who I told about it, so I thought I’d bung the details online.

The Hardware Bit

First step is to get the serial port working.  If you have a USB GPS module then it’s just a case of plugging it in, but mine is a 3.3v logic-level serial module that I bought about eight years ago to use with my Gumstix in a linux-powered walk logger for a couple of big charity walks we did back in 2004 and 2006.  It’s safe to say that hooking it up to the Raspberry Pi was a much simpler affair!

There are some great guides out there about soldering up the right pins (here and here).  I hooked up 3.3v power, ground and connected TX on the GPS to RX on the Raspberry Pi.  I used a little bit of veroboard, a 0.1″ header, some ribbon cable and made the GPS pluggable.

Out of the box the Raspberry has a console running on the serial port.  You need to disable this before you can do anything with the port.  Very easy to do: Edit /etc/inittab and remove the line that refers to /dev/ttyAMA0;  Edit /boot/cmdline.txt and remove the chunks of text that refer to /dev/ttyAMA0.  After a reboot the terminal will be gone.

Reading from the Serial Port in Mono

Right, now we’re ready to write some code.  First off I wrote this simple bit of C# to test that I was getting messages. The ReadData method reads text from the serial port one character at a time, detecting end of line characters to return a string for each line. The main method loops forever reading these lines and printing them to the console if they start with the NMEA “Recommended Minimum Content” message $GPRMC.

using System;
using System.IO.Ports;

public class Serial
   public static void Main()
      SerialPort serial = new SerialPort("/dev/ttyAMA0", 4800);
      serial.ReadTimeout = 1000;

         string data = ReadData(serial);
         if(!string.IsNullOrEmpty(data) && data.StartsWith("$GPRMC"))

   public static string ReadData(SerialPort serial)
      byte tmpByte;
      string rxString = "";

         tmpByte = (byte) serial.ReadByte();
         rxString += ((char) tmpByte);
         tmpByte = (byte) serial.ReadByte();
      }while (tmpByte != 13 && tmpByte != 10);

      return rxString.Trim();

Parsing the NMEA Data

I then did a bit of a Test Driven Development exercise to write a proper parser for NMEA messages. To do this in a test driven way I got my hands on some data files containing raw NMEA data and used that to create a Mock serial port reader. I could then pass these messages through my parser and test that I had managed to extract the right data.

Unit testing and using mocks was a great way to develop this part of the application. I could use recorded routes with real movement to test the parsing of speed data – since coding in a moving car seemed silly. I could also do all the coding work in Visual Studio on my Windows machine. This meant I could make the most of a nice big screen, code completion, resharper’s excellent testing interface and so on, then just push the code onto the Pi when it was done; I didn’t have to worry that “/dev/ttyAMA0” is “COM3” in Windows land, because I wasn’t using a real serial port to do 99% of the development.

A typical test for parsing of individual messages (hand typed!):

[TestCase("$GPRMC,005959,V,4807.038,N,11130.00,E,022.4,084.4,010101,003.1,W*4E", 48.1173, 111.5)]
public void CanParseLatLongFromRmcMessage(string input, double expectedLat, double expectedLong)
    NmeaParser parser = new NmeaParser();
    GpsMeasurement measurement = parser.Parse(input);

    Assert.That(measurement.Latitude, Is.EqualTo(expectedLat));
    Assert.That(measurement.Longitude, Is.EqualTo(expectedLong));

The mock serial reader class:

public class MockSerialPortReader : IPortReader
    private readonly string filename;
    private readonly int sleep;

    public MockSerialPortReader(string filename, int sleep)
        this.filename = filename;
        this.sleep = sleep;

    public IEnumerable Lines
            foreach (string line in File.ReadAllLines(filename))
                yield return line;

A typical unit test using the mock reader:

public void CanGetMeasurementsFromMockReaderDataSet1()
    NmeaParser parser = new NmeaParser();

    IEnumerable measurements = parser.ParseFrom(new MockSerialPortReader(dataSet1, 0));
    Assert.That(measurements, Is.Not.Null);
    CollectionAssert.AllItemsAreInstancesOfType(measurements, typeof(GpsMeasurement));

    Console.WriteLine("{0}, {1}", measurements.Last().Latitude, measurements.Last().Longitude);

Displaying the Speedo Text

The final stage of this little mini-project was to knock up a user interface. I spent a while looking at how to get something working under X Windows, then decided to go back to the Old School and just use ASCII art.

First thing was to find a quick and dirty way to define how each big number would look. Each number is made up of a grid of characters, defined in a class:

public class TextConstants
    public static readonly string[] Zero = new[]
            "   000000",
            "  00000000",
            " 000    000",
            " 000    000",
            " 000    000",
            " 000    000",
            "  00000000",
            "   000000"

   ...etc etc...

Writing this out then becomes an exercise in text placement:

public void DrawSpeed(int speed)


    char[] asciiSpeed = speed.ToString("00").ToCharArray();

    int xOffset = 20;
    foreach (char c in asciiSpeed)
        DrawNumber(TextConstants.For(c), xOffset, 8);
        xOffset += 15;

private void DrawNumber(IEnumerable lines, int xOffset, int yOffset)
    int lineNo = 0;
    foreach (string line in lines)
        WriteAt(line, xOffset, yOffset + lineNo);

Did it Work?

Well, yes! The main issue was the update speed. This is because the old GPS module outputs data very very slowly and has quite a slow refresh rate. As a speedometer it wasn’t much good – it generally showed the speed I was doing about 15 seconds ago. As a project it worked brilliantly though.

I extended the code slightly to add some logging. This saved the location data to a set of simple, size-limited, CSV files on the Pi’s flash card. I then knocked up some more code to turn the measurements into a “Trip Report” using the Google Maps API. Top Notch!

A Trip To Work

I don’t drive The Duke that often because it’d cost a fortune and make me deaf. So to test his new GPS powered brain I have been collecting test data in Vinny the Vectra. I can then use this data to write some unit tests and develop cool stuff while stationary.

In a slack moment at work today I knocked up a couple of functions to detect traffic jams and visualise my speed on my way to work. Here’s a typical trip. Click on the pic or here to explore the interactive “report”.


I quite like these static pages for output. They are very simple to create with some code and can be pushed up onto the web easily. No faffing about with databases, just some HTML and Javascript. On the map above the placemarks show places where I stopped for more than 30 seconds (there was traffic at junction 12 that morning!) and the colour of the line shows my speed. In the end I’d like to detect more events – when we go off road, when I brake suddenly or go round a corner too fast, when we get stuck in a traffic jam and when we visit places we know. I also got the webcam working on the raspberry pi, along with a wireless internet dongle so I can embed photos and videos then upload live and post on twitter.

Here’s some code snippets. I’m just playing, so please don’t think of me as somebody who’d ever return an “Enumerable Of Enumerables” in production code!  First is the function which splits the list of speed measurements based on bands of 10MPH…

        private IEnumerable<IEnumerable<GpsMeasurement>> SplitRouteBySpeed(IEnumerable route)
            var bandedMeasurements = (from measurement in route
                                      select new { Band = (int) (measurement.GroundSpeedMph/10), Measurement = measurement }).ToList();

            int currentBand = int.MaxValue;
            List currentSection = new List();

            foreach (var bm in bandedMeasurements)
                if(bm.Band != currentBand)
                    currentBand = bm.Band;
                    if (currentSection.Count > 0)
                        yield return currentSection;

                    currentSection = new List();


Second is based on somebody else’s hard work really, but I changed it enough to make it worth posting here. I’m basically using speed as a percentage, squishing to a value between -1 and 1 then using four colour “axis” as beautifully described in the comment-linked blog.

        private string SpeedToColour(double groundSpeedMph)
            // Based on this post - which has a very cool image to show what we're doing.

            double fraction = (groundSpeedMph             
            double red, green, blue;

            if ( fraction < -0.5 )
                red = 0.0;
                green = 2*(fraction + 1);
                blue = 1.0;
            else if ( fraction < 0 )
                red = 0.0;
                green = 1.0;
                blue = 1.0 - 2.0*(fraction + 0.5);
            else if ( fraction < 0.5 )
                red = 2.0*fraction;
                green = 1.0;
                blue = 0.0;
                red = 1.0;
                green = 1.0 - 2.0*(fraction - 0.5);
                blue = 0.0;

            byte redByte = (byte) (255 * red);
            byte greenByte = (byte) (255 * green);
            byte blueByte = (byte) (255 * blue);

            return string.Format("#{0}{1}{2}", redByte.ToString("x2"), greenByte.ToString("x2"), blueByte.ToString("x2"));