Archiving old geodata and images

The first map warper application lived on and for a long while it used to run in parallel to the New! Improved! and much faster warper at

Here’s 27K control points that people added around the world on  (There’s over 107K GCPs on – maybe I’ll do another map for that)

Screenshot - 050716 - 15:45:52

I’ve turned off the Rails application and have archived:

  • 5,100 maps metadata in CSV format
  • 27,409 Ground Control Points in CSV format
  • All the clipping masks in GML
  • All the maps source images converted to TIFF format
  • All the maps georectified images in TIFF and PNG formats

Simply pop over to if you want to check them out.

Mapwarper featured in A Digital Humanities Primer for English Students

Jenna Herdman has written an excellent free e-book about Digital Humanities for English Students which has an entire chapter titled: Digital Mapping Tool Tutorial which features the Mapwarper. It’s been published using gitbook and is available in pdf, html, epub formats.

The tutorial covers adding a map to to chart the movements of David in Charles Dickens’s David Copperfield.


The map is then loaded into Palladio which is a new tool for me. it “is a web-based platform for the visualization of complex, multi-dimensional data”.


Do check out this great resource. The book has seven chapters in total and all of them are interesting and worthwhile to read!

Screenshot after this read more tag:

Continue reading

Colliding The Mental Maps of Edinburgh with

Last autumn I popped up to Edinburgh from the North of England for State of the Map Scotland conference. Together with Edinburgh College of Art in Evolution House participants took part in series of workshops “Map.Makars”

I took part in a memory map of the city. The rules were: no looking at other maps, the map should include the venue, the castle, the train station. We drew, from memory the city on large pieces of paper. Gregory scanned/photographed these and put these on to stretch them to fit. he then combined these together with an interactive and animated transparency control to create the Hand Drawn Map Collider “No-map Map Give it a whirl!

Screenshot from 2016-04-22 11:26:11.png

My map, in case you were wondering was possibly the least accurate of them, coming from furthest away!

Screenshot from 2016-04-22 11:28:00.png


A “Frightful Number!” – Mapping Daniel Defoe’s A Journal of the Plague Year

Two wonderful historical maps using by Vimala C. Pasupathi @Exhaust_Fumes showing Melville in Rome and in London during the Great Plaque

Screenshot from 2016-03-24 00:05:44

“A Frightful Number!” Mapping Daniel Defoe’s A Journal of the Plague Year

“A Frightful Number” uses data culled from Daniel Defoe’s “creative non-fiction” Journal of the Plague Year and from historical sources to track the spread of the 1665 London plague. Created with Hofstra DRC’s innovative mapping tool Itinerary, the site annotates the epidemic’s progress month by month, parish by parish.

As a supplement to the novel, “‘A Frightful Number’” represents and engages with Defoe’s epidemiological mapping of the plague as it makes its way across London and its suburbs, enabling readers to gain a sense of the geography of London to which Defoe refers and a better grasp of the plague’s progression through and impact on the city. “‘A Frightful Number’” also seeks to dissect and analyze the relationship between the novel as a literary form and cartography and how their respective histories intersect throughout the early modern period. Presenting A Journal of the Plague Year in cartographic form helps underscore for readers the difficulties and obstacles that the narrator, H.F., faces as he attempts to “map” and, ultimately, survive the plague, an invisible and elusive entity whose presence is only manifest in its gruesome effects on London’s population. H.F.’s map of the plague is, in this sense, a useful and necessary technology that can make the invisible visible, giving the disease a shape and form in order to better understand its origins and causes, as well as its potential consequences for London and the nation. That said, such a task proves to be immensely difficult, if not impossible, for Defoe’s narrator, who survives the plague but doesn’t seem to come much closer to grasping it in all of its deadly complexity. “‘A Frightful Number’” thus seeks to reflect this sense of ambivalence in Defoe’s novel: to offer a map of the Great Plague that, at the same time, exposes the very real problems and limitations in fully realizing such a project.

Melville in Rome

Screenshot from 2016-03-24 00:07:48

Wikimaps on Maps Mania: Maps in the Age of Cholera (Leeds) & The Vintage Maps of Berlin

Cross posted from The Wikimaps Blog

The wonderful, prolific and very popular Maps Mania blog  featured the Wikimaps Warper a few times recently, do check them out!

The first interactive map: The Vintage Maps of Berlin uses the Wikimaps Warper.


Keir writes:

This collection of old historical maps of Berlin centers around what is now Museum Island in Berlin.

In the oldest maps you can clearly see the two towns of Cölln and Altberlin, on opposite banks of the River Spree. As you progress through the maps you can see how this area of Berlin’s has changed and developed over the centuries.

Do check out the 11 maps of Berlin from 1652 to today  here:

The second post, and interactive map entitled Maps in the Age of Cholera based on an epidemiological map of Leeds (co-incidentally my home!).

logo (1)

This was also created by Keir and he writes:

Twenty Years before John Snow famously mapped the locations of cholera victims in Broad Street, London, Robert Baker plotted the deaths of cholera victims in Leeds.

Maps in the Age of Cholera is a story map based around Robert Baker’s ‘Sanitary Map of the Town of Leeds’ exploring the 1832 cholera epidemic in the Yorkshire town. Baker never made the link between cholera and contaminated water. However, in his map and in the accompanying report to the Leeds Board of Health, Baker noted that “the disease was worst in those parts of the town where there is often an entire want of sewage, drainage and paving”.

The map itself uses this Leaflet Story Map plug-in. The Leaflet Story Map library uses jQuery to create a scroll driven story map. The map tiles scheme for Robert Baker’s 1832 ‘Sanitary Map of the Town of Leeds’ comes from Wikimaps Warper.

do go check out the interactive story map here

strace and ruby-prof to identify slow rails startup on wikimaps

The Wikimaps map warper – a free georeferencing georectification tool for historical maps and images hosted on Wikimedia Commons had some issues recently. Here’s a short document describing some diagnostics to identify the problems.

The wikimaps warper runs on the Wikimedia Labs infrastructure  – essentially an OpenStack cluster of servers. The Labs is run by some great admins and dev ops and they are on the ball when it comes to rolling out needed updates to al the servers and instances. However each instance is generally admin’ed by people within individual projects. It was after they applied a kernel update due to a security patch and restarted all the machines that the warper was seen to be offline. The ultimate issue was that there was something wrong with the NFS – as many other instances and tools were having problems – I’m not going to blame the NFS because it showed up where the warper application was encountering issues.

The problem

The application is a ruby on rails application which runs using Apache and Passenger. It was timing out, it wouldn’t start. I ssh’d into the server. The passenger version was an old one (via a package, rather than a gem) and didn’t support increasing the timeout version. So I removed that, and installed a new version. The library would also be running using a faster version of Ruby to get running. Did all that, but the warper wouldn’t start.

I ssh’ed into the server and couldn’t see any errors anywhere. But running a console command, or a “rake -T” which would load up the Rails environment, took 11 minutes. ELEVEN MINUTES!

Eleven Minutes vs 22 Seconds  – Diagnostics

I use a little old atom cpu, 2gig memory netbook, and it takes less than 22 seconds to load up there.  Enter Strace and ruby-prof. (Note that the screenshots and reports following are based on me getting the loading time down to something workable, but there still work to do)

strace is a linux command line tool which traces system calls and signals. It’s amazing for debugging things. I’ve hardly ever used it. I ran strace

strace rake -T

and watched for around 5 minutes the Rails loading process. I stopped it as it wasn’t telling me much and I wasn’t prepared to wait 11 minutes! But what it did was show me something like the following – the inefficient way Rails loads up all the libraries, the gems:

strace -e trace=open,stat,read,write bin/rake -T

to trace only certain file related calls: here is an example of what it looks like on my netbook


It shows, firstly that its doing lots of file calls, and secondly that its getting lots of misses. Not very efficient but also, totally normal and not revealing.

strace with summary report is what I wanted.

 strace -o strace.txt -c bin/rake -T

gave something better:


You can see in this case that it’s taking around 5 seconds with open calls (the errors are the expected misses, I think). Compare this from my netbook


5 seconds vs 0.04 seconds. But what files was it opening, what was Ruby up to? Enter ruby-prof


ruby-prof -s total -p flat bin/rake -- -T > ruby_prof_flat.txt

ruby prof is more of a profiling tool. I’m using the command line utility, but its most often used in code, for example around a block or class. Check it out.

The ruby-prof can output its data in a few different formats or profiles. I tried the more colourful ones, which didn’t tell me much:


this is somewhat useful, but the document was massive  and hard to compare or see where the bottlenecks were. You can see that “require”  -the loading of the libraries was troublesome though.


colourful stack – not sure what this told me more but it looked pretty.


Before finally choosing the default option – flat:

here it is on the server


and compare that with my crappy netbook


So it gave a much better diagnostic. It’s slowing right down reading in the libraries, the gems. Now, on both machines, I am using rvm and both are loading from a home directory. But on the server, the home directory was on NFS, not a more local partition!

I copied the rvm directory to a local partition and symlinked things and now the application can start much quicker.

There’s probably more I can do to optimise this – I believe it’s still hitting the NFS in a few places, but it’s given me a good pointer in the right direction to making the wikimaps warper run better and stronger for the future.











Wikimedia IEG Grant funds Wikimaps Warper for 2016

The Wikimaps Wikimedia Commons Warper project has recieved an Individual Engagement Grant for 2016. Go read this post by Susanna Annas, project lead for the Wikimaps Warper introducing the work for the Warper 2.0 in the Wikimaps Blog. It’s one of fourteen projects that Wikimedia has funded.

Here is what we are working towards – a complete redesign of the interface.


This grant is to lay the ground before this work can happen. We need to decouple the application, rewire it. To do this I am gutting out the internals of the Mapwarper Rails application and implementing an API so that third party applications can find, query maps and mosaics, warp and crop maps and upload and add maps. We will also be improving the connections to Wikimedia Commons to enabled easier additions of maps. We’re probably going to be adding built in caching of the WMS and Tiles to improve performance, towards a production ready performant system ready for the loads of millions of users. We’ve half a year to do the work.

We are developing on GitHub, using the issues and milestones for issue tracking and I’ll be logging the development on the Wikimaps page on the Commons Wiki.