Doing Digital History 2016 – Map Warper Tutorial

Doing Digital History 2016 is a two week Summer institute sponsored by National Endowment for the Humanities and was held at George Mason university in DC/Arlington earlier in the year. It was attended by “mid-career” historians and the focus was on digital humanities and history – making visualizations, mapping, sound and vision, and more.

Screen-Shot-2016-07-07-at-4.11.16-PM.png

I was happy to see that the participants had a workshop Georectifying maps with Map Warper.

Here’s the screenshot of the tutorial:

 

 

 

 

 

The New Cloud Atlas – Mapping the Physical Infrastructure of the Internet

Introduction

The New Cloud Atlas, (newcloudatlas.org) is a global effort to map each data place that makes up the cloud in an open and accountable way. It’s a project to find and map each warehouse data centre, each internet exchange, each connecting cable and switch. Anything of any physical significance in the operation of the cloud should be observed in some way, and recorded for everyone to see and use. Data is stored in OpenStreetMap and users can map things using the on site iD editor with custom telecoms presets for the first time. Map tiles with two styles have been produced and have now made visible this hidden infrastructure. http://newcloudatlas.org

The New Cloud Atlas, named after the nineteenth collaborative scientific data collection project, is about understanding and making visible the hidden “Cloud”. Although most of these telecoms features are in the open and in plain sight, many are missing from open datasets or may be considered sensitive. Telecoms infrastructure has immense importance in connectivity and power in our connected world  – the more connected a place is the more benefits it has. Indeed the lines of fibre optic backbone have become the new ley lines of the 21st Century powering the forces behind a new Psychogeography of places.

A bit about the name: The First Cloud Atlas was published in 1896 by the Permanent Committee of the first International Meterological Congress. Cloud weather observatories around the world were able to share consistent observations of the clouds and observe weather systems whose scale stretched over national boundaries. The publication of the International Cloud Atlas represented a move beyond national concerns and boundaries to an international perspective.

In addition to its important role in the predicting the weather, the vision is a surprisingly early call for infrastructural globalism and worldwide collaboration:

“If there be any branch of science in which work on a uniform system can be especially useful and advantageous, that branch is the inquiry into laws of weather, which, from its very nature, can only be prosecuted with a hope of success by means of very extensive observations embracing large areas, in fact, we might almost say, extending over the whole surface of the globe”

Site

The site shows frequently updated tiles generated from OpenStreetMap(OSM) data, details about the project and a custom OSM editor for making it easier to add map features. Here are some screenshots.

Screen Shot 2016-08-12 at 21.12.33

Map, Transparent Tiles, Markers, Legend

 

Screen Shot 2016-08-12 at 21.16.25

Cloud X-Ray Style, with scale independent(ish) building polygons

The Cloud X-Ray style, shown above was partially inspired by Kosmtik’s data inspector style, and it shows polygons that are enlarged at low zooms. Polygons should appear to be the same size on the screen as you zoom in. It gives a sci-fi cartography, but I find it very useful finding clusters of mapped features, as all features are shown at all zoom levels.

 

Screen Shot 2016-08-12 at 21.21.00.png

Custom iD Editor with Telecoms Presets

Note: that you can also edit in JOSM or Vespucci OSM Editors using these presets here: https://simonpoole.github.io/new-cloud-atlas-preset/

Background

The New Cloud Atlas is a project initiated by experimental media technologist, artist and designer Ben Dalton with the design and research studio of Amber Frid-Jimenez and Joe Dahmen, and myself. Ben writes about the project – with the main idea that it’s about understanding what the Internet actually is in physical terms, rather than as something that remains clouded and mysterious:

The first appearance of the internet cloud was in network diagrams. The cloud symbol was used to stand in for complexity. The cloud embodied something of the way that the internet functions. The internet was designed to be ‘end-to-end’, so computers are meant to be able to connect to each other without interference as the message passes through a network of interconnections. Only the end points are meant to matter. The clouds here represent ‘something in the middle that is too complex to draw here’, a kind of neutral space through which information passes. It is an act of simplification, but it also contains an implicit statement that ‘the cloud will look after itself’ that this thing is going to carry on being there.

Beclouding is deliberately making something more confusing, in order to obfuscate or conceal its meaning. The use of the cloud has shifted in digital systems. The idea that ‘this is too complicated to think about’ has been moved front and centre and converted into a business model, shedding its innocence along the way. Through a sleight of hand, the cloud sometimes appears as a platform, and sometimes a material. This narrative rests on the idea that the services are to be trusted, and they can take care of themselves on your behalf. We trust them with our emails and our childhood photographs and our meeting plans and whatever else we use the cloud for. In this new definition of the cloud, there is a statement that ‘this is too complex to deconstruct or critique’. You shouldn’t try to look in to the cloud and see what’s there. It’s made up of vapour, and it’s not to be interrogated. Better to simply observe it from a distance and admire it at sunset.

Once the domain of national governments, information infrastructure is increasingly constructed, operated, and maintained by major multinational corporations. These corporations, which include Google, Facebook, Amazon, Apple, and Microsoft, have a similar vested interest in maintaining control over of the flow of goods and information once exercised by national governments, but a reach at once more extensive and less transparent.

Psychogeography

Regular readers may know of my interest in Psychogeography. The British Psychogeography of the 90s employed Ley Lines and “Magico-Marxism” using the language of the occult to explain the unknown forces of power at work in space and in places. I’m developing the idea that the new lines of power in the 21st Century are of Information – and the actual lines of light that transmit these bits of data, and the buildings that house them. More about that in walk or talk later on this year!

Another more obvious connection with psychogeography is the hidden in plain sight angle. These passageways of the internet are often marked, on manhole covers, in mobile phone masts, in big buildings in light industrial estates, but they are utterly overlooked. They may travel along the margins, along canals or train tracks. They are also sited in classic psychogeographical “liminal” spaces – beaches, margins of rural and urban, wasteland, on top of tower buildings etc.

OpenStreetMap and the Telecoms WikiProject

OpenStreetMap allows anything that exists and can be verified to be mapped. There is no notability rule that Wikipedia has, for example. So it allows manhole covers to be mapped in detail, it allows telephone lines and the assorted street cabinet boxes that crowd our pavements. You might get feedback as to how to map these features and you might get funny comments about why these features are being mapped (and indeed, mapping with OpenStreetMap is voluntary!) but pretty much all OSM mappers will agree that these features shouldn’t be excluded.

Telecoms features in OpenStreetMap haven’t been well mapped before. This is both good and bad in that the taxonomy (or folksonomy to be more accurate)  – the tags that describe these features -have not been standardized. We have the opportunity to define the tags, or at least standardize some of them to be more consistent across similar telecoms infrastructure features.

I started WikiProject Telecoms on the OSM Wiki, so please go there to see how to map and tag features – and if you are a telecoms, mapping or tagging specialist please suggest better ways to map these features! https://wiki.openstreetmap.org/wiki/WikiProject_Telecoms

The current features being rendered in the New Cloud Atlas map are:

  • Data Centres
  • Telephone Exchanges
  • Manhole covers
  • Telephone poles and wires
  • Submarine cables etc
  • Telecoms towers, masts, and antennae
  • Street Cabinets

Underground features may be more difficult to map – so we are relying on manhole covers which often show what its use and who operates the cable underneath (in the UK at least) – and those markings sprayed by utility companies, and some data imports. If you don’t know where an underground cable goes its probably best to leave it out.

You might have noticed that many of the options include sound and heat for the street cabinets. One of the side effects of today’s modern fibre optic street cabinets is that they are often installed with more power needs than copper wire ones – and so they need a fan. Often the cabinets are warm to the touch and sometimes they make a quite loud drone sound. This type of data can be useful, I have heard, to people who are vision impaired. Sound and touch can help orientate people in space.

Update: There is a JOSM Preset and a Vespucci Preset that Simon Poole developed

Open Data / Secrecy etc

It’s probably worth talking a little bit about the privacy and secrecy issues. Although the project isn’t about getting releases of data from companies and governments, and it’s not about uncovering the secret installations, it is about collaboratively mapping the world. Almost all of the information that will make up the New Cloud Atlas will be found in the field or in public information sources.

You may be reminded of a story in 2003 (2 years after 9/11) of Sean Gorman‘s PhD dissertation that the US Government wanted to make classified as although it contained only publicly available information (about the Internet connections in the US) he analyzed the data to identify the weak links – weaknesses that, for example, a disaster could take out or a terrorist could exploit.  Officials in the US Govt said that his dissertation should be burnt! Sean successfully graduated and started a mapping company with the DHS as clients. (I actually ended up working there at FortiusOne / GeoIQ several years after that for a bit). Now of course open data and open analysis is encouraged and promoted by governments (and following this trend check out Sean’s new startup Timbr.io).

You may also recall stories about how many national mapping agencies removed military bases (such as Aldermaston, or Greenham Common in the UK) from their paper maps – even when these bases were signed from the motorways and major roads and had nice big clear signs outside the fences.  A relic from the Cold War, perhaps. It appears to me that even in this current year the Ordnance Survey mislabels the Menwith Hill USAF/RAF Listening Base in North Yorkshire as just “Menwith Camp” with no indication of it’s real name, activity nor landuse (as compared to OpenStreetMap for example).

At this point, if you are curious, we should evoke the classic 1996 Wired Article by Neal Stephenson: Mother Earth Mother Board http://www.wired.com/1996/12/ffglass/ It’s essential if you are interested in reading more about the geo political and technology of international internet cable laying. It’s also a great read in general.

 

Liverpool Walk / Workshop

Ben and I ran a series of walks and workshops at FACT in June 2016. Cloud Dowsing Hunting for the Hidden Internet and Mapping the New Cloud Atlas

We used FieldPapers to give to participants and mappers and went around the streets of Liverpool.

Here we are near the main telephone exchange and data centre looking for manhole covers, cabinets and antennae, that’s me pointing.

13391350_1053402384736248_1279453524_n

You can view the photos I took on the Flickr Album https://www.flickr.com/photos/chippee/sets/72157671540933095

 

Development Notes

Code for the site is on github: https://github.com/timwaters/new_cloud_atlas

Mapnik / Kosmtik Style file and processing notes also on github: https://github.com/timwaters/cloud_mapping

Mapnik X-Ray Style

Of possible interest to mapnik style geeks could be the use of the scale denominator and PostGIS ST_Scale commands to scale up building polygons so that they appear to be the same size regardless of the zoom. If anyone wants to fix this to make it work better, please let me know!

select st_translate(st_scale(way, (!scale_denominator! * 0.00028) - (5 - z(!scale_denominator!)) ,
 (!scale_denominator! * 0.00028) - (5 - z(!scale_denominator!)) ), 
st_x(st_centroid(way))*(1-( (!scale_denominator! * 0.00028) - (5 - z(!scale_denominator!)) )), 
st_y(st_centroid(way))*(1-( (!scale_denominator! * 0.00028) - (5 - z(!scale_denominator!)) ))) as way,
 building AS type FROM planet_osm_polygon WHERE (building='data_center' ) AS data",

OSM Tile Generation

Tiles are kept up to date at around 15 minutes with the central OSM database. Occasionally a full planet import is done. I think I could use Lua scripting to ensure that the database remains lean. The system uses TileStache to enable the UTFGrids for the popups. Essentially we filter out a lot of stuff from the OSM database:

  1. Convert an osm.pbf file to an o5m file
    ./osmconvert  planet-latest.osm.pbf -o=planet.o5m
  2.  Filter the o5m file to an .osm file
    ./osmfilter planet.o5m --parameter-file=cloud_mapping/osmfilter_params.txt > planet.filtered.osm
  3. Import the .osm file into the database using the custom osm2pgsql Style
     osm2pgsql --slim -d gis planet.filtered.osm -S cloud_mapping/default.style
  4. Set up replication using Osmosis and osm2pgsql to get changes from OSM db
    osmosis --read-replication-interval  --simplify-change --write-xml-change changes.osc.gz
    osm2pgsql -d gis -S default.style -s -C 800 -a changes.osc.gz -e10-19 -o expire_e10-19.list

http://newcloudatlas.org/

Leodis Collections

What is Leodis Collections?

Leodis Collections is prototype site for the main Leodis.net website – The Leeds City Council Library’s historical photo website, and the most heavily trafficked site on the councils network. It’s also one of the oldest ones and is desperately in need of a reworking to make it up to date. I worked closely with the libraries team to re-imagine how the Leodis archive, which has over 60,000 images, can work. Collections are curated groups of photos around a theme.

front-page

The project was produced as part of an “Innovation Pathway” project with the Council and the Urban Sustainable Development Lab which I’m a team of. The idea is to quickly develop small, cheap, projects to inspire and to encourage future, better solutions. The design was meant to be clean and simple, with a horizontal grid, responsive support and most importantly, not bloated and quick loading. It is live at http://leodiscollections.net/, with links to it from the main Leodis.net site and the councils libraries homepage.

Photos can be georeferenced by users in the application, and shown on a map together for the first time. Collections are groups of photos, and initially we had five such groups: Shopping, Arcades, Parks, Pubs and Cinemas. Each photo can also have associated memories with them which are selected comments from Leodis. Collections can be promoted to appear higher up the list of photos and the photos with a collection can also be promoted to the top. It also has some simple social media sharing buttons, and notably, the images are bigger and have better resolution than on Leodis.net!

The application is written in python and Django and has it’s source code on my GitHub.

There are five main sections of the site:

Front Page: Viewing all Collections

Here all collections are shown

front-page

Collection Page

Here all the photos within a collection are shown, with the locations of the subjects in the photo shown on the map. Map markers, when clicked show a clickable popup. Hovering over the photos in the list shows the image title, and clicking on them leads you to the photo page.

collections-page

Screen Shot 2016-08-07 at 16.25.56

Photo Page

Here the large photo is shown, with some metadata, and a small inset map. Additionally “memories” can be shown underneath. There are links to the photo page on Leodis.net and social sharing buttons.

photo-page

We can compare with the old Leodis.net page (click for bigger):

Screenshot from 2016-08-07 16:10:36

 

Admin Page

Admin page, (django admin pages) it’s here that photos can be georeferenced.

admin-pages

Leodis Collections admin page

Labs, process, prototype, future

The process of this project was interesting. The Urban Sustainable Development Lab had a small hackday where a few technologist and council staff came together. During the morning, we talked in a group about requirements etc, and in the afternoon the technologists worked alone on producing mockups, live sites, wireframes etc before presenting the work.

Here’s a screenshot of one of the images of half a dozen in the mockup I produced in that afternoon. You can see that the memories were brought out and placed prominently in the grid design. The grid design was pretty much taken from existing sites like Flickr and Google Photos. I think its a nice concept for images with varying dimensions.

asc2

In following weeks my ideas were commissioned and myself and the Library’s staff had a few meetings to go over the design and requirements. Here’s an image from one of the wireframe documents I produced:

propt

The site was then built and iterated upon quickly. For the future, now it’s running we want to add pagination to collections, search for photos, multiple relationships, user corrections of locations, and some more detailed responsive, usability and design improvements.

Visit LeodisCollections.net

Code: https://github.com/timwaters/leodis-collections

Leeds Riot Map 1643 – 2002

From varied causes and peoples including Chartists to Suffragettes in many places including Holbeck to Chapeltown the City of Leeds has seen many riots over the years. I created two things – a static map of some Leeds Riots for a print publication (RIOT!) in conjunction with Leeds based Fictions Of Every Kind. And an interactive map website LeedsRiotMap.co.uk

Screenshot - 180716 - 12:38:59

Made for James (Jimmy) Cauty’s ADP Riot Tour visit to The Chemic Tavern, Leeds, 14-21 July 2016, the Leeds Riot Map shows 23 riots from 1643 to 2002.

 

chemic_poster

leeds_riots_smaller

http://leedsriotmap.co.uk/  was made using the JQuery + Leaflet StoryMap plugin. As you scroll through the riots in chronological order, the map zooms and pans to the location of the riot. The basemap is Stamen’s black and white OSM basemap. Locations are approximate (interestingly, it was the more modern riots which were harder to place) and the details and reporting isnt guaranteed to be accurate either!

Should you spot errors or omissions with the site, please contact me or add an issue on github.

CnufXL4XgAAn2bS.jpg:largeCnufXLzWAAE3TDy.jpg:large

 

 

Archiving old warper.geothings.net geodata and images

The first map warper application lived on http://warper.geothings.net and for a long while it used to run in parallel to the New! Improved! and much faster warper at mapwarper.net

Here’s 27K control points that people added around the world on warper.geothings.net  (There’s over 107K GCPs on mapwarper.net – maybe I’ll do another map for that)

Screenshot - 050716 - 15:45:52

I’ve turned off the Rails application and have archived:

  • 5,100 maps metadata in CSV format
  • 27,409 Ground Control Points in CSV format
  • All the clipping masks in GML
  • All the maps source images converted to TIFF format
  • All the maps georectified images in TIFF and PNG formats

Simply pop over to http://warper.geothings.net if you want to check them out.

Mapwarper featured in A Digital Humanities Primer for English Students

Jenna Herdman has written an excellent free e-book about Digital Humanities for English Students which has an entire chapter titled: Digital Mapping Tool Tutorial which features the Mapwarper. It’s been published using gitbook and is available in pdf, html, epub formats.

The tutorial covers adding a map to mapwarper.net to chart the movements of David in Charles Dickens’s David Copperfield.

unspecified-2

The map is then loaded into Palladio which is a new tool for me. it “is a web-based platform for the visualization of complex, multi-dimensional data”.

unspecified-1

Do check out this great resource. The book has seven chapters in total and all of them are interesting and worthwhile to read! https://www.gitbook.com/book/jennaherdman/a-digital-humanities-primer-for-english-students/details

Colliding The Mental Maps of Edinburgh with Mapwarper.net

Last autumn I popped up to Edinburgh from the North of England for State of the Map Scotland conference. Together with Edinburgh College of Art in Evolution House participants took part in series of workshops “Map.Makars”

I took part in a memory map of the city. The rules were: no looking at other maps, the map should include the venue, the castle, the train station. We drew, from memory the city on large pieces of paper. Gregory scanned/photographed these and put these on mapwarper.net to stretch them to fit. he then combined these together with an interactive and animated transparency control to create the Hand Drawn Map Collider “No-map Map Give it a whirl! http://www.livingwithdragons.com/maps/nomap-map/

Screenshot from 2016-04-22 11:26:11.png

My map, in case you were wondering was possibly the least accurate of them, coming from furthest away! http://mapwarper.net/maps/10907

Screenshot from 2016-04-22 11:28:00.png