Mapping the UK and navigating the post code maze
If you have read some of my blogs, you would already know that I love maps and visualising information geographically. That is why in my current job I have pushed for data visualisation tools that incorporate geographic analsysis at their core. In this post I will share some of the challenges and the solutions that I have found when plotting maps of the United Kingdom. I choose UK for this blog because it has a large set of possible different adiministrative divisions and thus might require several methods to achieve the perfect visualisation.
The problem with Excel:
When Microsoft released the 2016 version of Excel, they included a built-in mapping capability powered by Bing/TomTom that will take geographical input (countries/regions, states, counties or postal codes) and attempt to map it versus the best match.
According to the official documentation, where there might be more than one similar location in the world, map charts can’t necessarily tell the difference without more guidance and adding more higher level columns could help disambiguate. However this solution might not always work and most of the time you will end up with a yellow warning telling you that not all your dataset was plotted. I think Excel is a quick and dirty way to achieve a decent looking chart and might work for some uses. When used alongside data slicers, it creates a great interactive solution, that you can create in a copule of minutes. I found a very useful blog entry that contains the suggested names and disambiguations for the UK if you want to use Excel for your projects.
However, most of the UK datasets you will encounter in real life come with different types of geographical units, such as electoral divisions, postcode areas and, if you are dealing with pan-european data, NUTS codes (Nomenclature of Territorial Units for Statistics). The nature of the data will make Excel unusable or it will force you to perform several data mapping attempts before reaching to the label that suits Excel.
In order to produce more effective, scalable and informative maps, it is better to use other open-source tools. The big insight is that in order to get true flexibility, you need to work with tools that can take custom parameters (like custome boundaries and layers). In the next section I will describe 2 python libraries that will give you such flexibility in your mapping efforts.
Undesrtanding boundary hierarchy in the UK
The Open Geography Portal of the Office for National Statistics is the one place you can go to start your mapping quest. As you can see in the picture below, there are plenty of ways of dividing the UK: some of them would be country specificic and will not encompass the entirety of the United Kingdom. Those that are all-encompassing are the “european electoral regions”, “NUTS areas”, “local planning authorities”and “Postcode Areas”. I will deep-dive into these just to see how to map all countries that are included in UK. However, you can apply the same logic and method to the boundaries that are specific to one of the countries and achieve the same result.
I will begin with a NUTS boundary example (which in essence can be exactly replicated to the local planning authorities and the european election areas) . You will need three things to achieve your visualisation:
- A geopgrahic layer with boundaries (I will use a GeoJson file, but it could also be TopoJson or ShapeFile provided that you can converted to GeoJson)
- A mapping library that takes the geographic layer file type you have selected (I will show examples with Plotly and Folium)
- Data that has a key that matches the geographic layer.
The open geography portal has a very comprehensive directory of “boundaries”. See the gif below to see how to navigate the page to get a url of the GeoJson file we will use in the example.
Now, we can either use Plotly or Folium to create beautiful choropleth maps that will use the GeoJson layer and add a data dimension as the color scale. The data used is a dataframe that contains a common key (in this case, the name is “nuts218cd” and some random values I assigned for the visualisation.
Which will produce the following result (I have hidden plotly’s base layer so what you see are the actual GeoJson boundaries):
Line 20 of the gist above is crucial for plotly to work (there are some issues with the current GeoJson format, thus it needs to be slighty transformed). Personally, I like the Plotly end result because it is very easy to add/customize the hover info and map controls and also because it can be easily integrated into a dash application that allows interactivity. However, I am aware that Folium has some other capabilities like overlaying several layers and adding markers. Also no need to perform any corrections on the GeoJson file (compatible with current GeoJson formats). So, if you prefer to use Folium, the code and result would be as follows.
What about Post Codes?
So far we have dealt with pretty straight forward location types, which boundaries are readily available in places such as the Open geography Portal. So, what about postcodes?
Royal Mail is the institution in charge of maintaining the grid of postcodes and its use is so prevalent that most people and businesses know their postcode and it is a familiar and natural choice for collecting and recording data. For example, it is normal for an ecommerce to analyse its geogrpahical performance based on the delivery address of its customers.
As noted in the relevant documentation “postcodes form a compact geographic reference that the public and businesses are familiar with. However, linking postal geographies to other geographic units is not always straightforward” because postcode boundaries do not align with geographical boundaries. The Geoportal provides the “postcode centroids” for the Unit Postcodes, which you could use for very specific and granular analysis. Most of the time you would like to aggregate things to a “district” or “area” level (which Geoportal does not unfortunately provide.
This is where the 3-step method decsribe before shows its value. We can input anytype of GeoJson to Plotly or Folium, regardless of its source. So, if I wanted to see area codes, I could get them from an alternative source (I found the files I was looking for in this site) and create the map. I used an online service to easily transform a different file type (they provided a TopoJson instead of a GeoJson) and managed to create the map with Postcode Areas:
This same approach can be used if the boundary type has any other format (see this useful website to convert ShapeFile to GeoJson). You just need to convert it first to a GeoJson and then use the mapping library as decribed above. Combining the possibility of customizing the layers with the real-world datasets will enable you to create stunning and beautiful visualisations.
Happy mapping!
If you want to connect and have a conversation about this or any other topic, please feel free to find me on LinkedIn:
