The New Cloud Atlas – Mapping the Physical Infrastructure of the Internet

Introduction

The New Cloud Atlas, (newcloudatlas.org) is a global effort to map each data place that makes up the cloud in an open and accountable way. It’s a project to find and map each warehouse data centre, each internet exchange, each connecting cable and switch. Anything of any physical significance in the operation of the cloud should be observed in some way, and recorded for everyone to see and use. Data is stored in OpenStreetMap and users can map things using the on site iD editor with custom telecoms presets for the first time. Map tiles with two styles have been produced and have now made visible this hidden infrastructure. http://newcloudatlas.org

The New Cloud Atlas, named after the nineteenth collaborative scientific data collection project, is about understanding and making visible the hidden “Cloud”. Although most of these telecoms features are in the open and in plain sight, many are missing from open datasets or may be considered sensitive. Telecoms infrastructure has immense importance in connectivity and power in our connected world  – the more connected a place is the more benefits it has. Indeed the lines of fibre optic backbone have become the new ley lines of the 21st Century powering the forces behind a new Psychogeography of places.

A bit about the name: The First Cloud Atlas was published in 1896 by the Permanent Committee of the first International Meterological Congress. Cloud weather observatories around the world were able to share consistent observations of the clouds and observe weather systems whose scale stretched over national boundaries. The publication of the International Cloud Atlas represented a move beyond national concerns and boundaries to an international perspective.

In addition to its important role in the predicting the weather, the vision is a surprisingly early call for infrastructural globalism and worldwide collaboration:

“If there be any branch of science in which work on a uniform system can be especially useful and advantageous, that branch is the inquiry into laws of weather, which, from its very nature, can only be prosecuted with a hope of success by means of very extensive observations embracing large areas, in fact, we might almost say, extending over the whole surface of the globe”

Site

The site shows frequently updated tiles generated from OpenStreetMap(OSM) data, details about the project and a custom OSM editor for making it easier to add map features. Here are some screenshots.

Screen Shot 2016-08-12 at 21.12.33

Map, Transparent Tiles, Markers, Legend

 

Screen Shot 2016-08-12 at 21.16.25

Cloud X-Ray Style, with scale independent(ish) building polygons

The Cloud X-Ray style, shown above was partially inspired by Kosmtik’s data inspector style, and it shows polygons that are enlarged at low zooms. Polygons should appear to be the same size on the screen as you zoom in. It gives a sci-fi cartography, but I find it very useful finding clusters of mapped features, as all features are shown at all zoom levels.

 

Screen Shot 2016-08-12 at 21.21.00.png

Custom iD Editor with Telecoms Presets

Note: that you can also edit in JOSM or Vespucci OSM Editors using these presets here: https://simonpoole.github.io/new-cloud-atlas-preset/

Background

The New Cloud Atlas is a project initiated by experimental media technologist, artist and designer Ben Dalton with the design and research studio of Amber Frid-Jimenez and Joe Dahmen, and myself. Ben writes about the project – with the main idea that it’s about understanding what the Internet actually is in physical terms, rather than as something that remains clouded and mysterious:

The first appearance of the internet cloud was in network diagrams. The cloud symbol was used to stand in for complexity. The cloud embodied something of the way that the internet functions. The internet was designed to be ‘end-to-end’, so computers are meant to be able to connect to each other without interference as the message passes through a network of interconnections. Only the end points are meant to matter. The clouds here represent ‘something in the middle that is too complex to draw here’, a kind of neutral space through which information passes. It is an act of simplification, but it also contains an implicit statement that ‘the cloud will look after itself’ that this thing is going to carry on being there.

Beclouding is deliberately making something more confusing, in order to obfuscate or conceal its meaning. The use of the cloud has shifted in digital systems. The idea that ‘this is too complicated to think about’ has been moved front and centre and converted into a business model, shedding its innocence along the way. Through a sleight of hand, the cloud sometimes appears as a platform, and sometimes a material. This narrative rests on the idea that the services are to be trusted, and they can take care of themselves on your behalf. We trust them with our emails and our childhood photographs and our meeting plans and whatever else we use the cloud for. In this new definition of the cloud, there is a statement that ‘this is too complex to deconstruct or critique’. You shouldn’t try to look in to the cloud and see what’s there. It’s made up of vapour, and it’s not to be interrogated. Better to simply observe it from a distance and admire it at sunset.

Once the domain of national governments, information infrastructure is increasingly constructed, operated, and maintained by major multinational corporations. These corporations, which include Google, Facebook, Amazon, Apple, and Microsoft, have a similar vested interest in maintaining control over of the flow of goods and information once exercised by national governments, but a reach at once more extensive and less transparent.

Psychogeography

Regular readers may know of my interest in Psychogeography. The British Psychogeography of the 90s employed Ley Lines and “Magico-Marxism” using the language of the occult to explain the unknown forces of power at work in space and in places. I’m developing the idea that the new lines of power in the 21st Century are of Information – and the actual lines of light that transmit these bits of data, and the buildings that house them. More about that in walk or talk later on this year!

Another more obvious connection with psychogeography is the hidden in plain sight angle. These passageways of the internet are often marked, on manhole covers, in mobile phone masts, in big buildings in light industrial estates, but they are utterly overlooked. They may travel along the margins, along canals or train tracks. They are also sited in classic psychogeographical “liminal” spaces – beaches, margins of rural and urban, wasteland, on top of tower buildings etc.

OpenStreetMap and the Telecoms WikiProject

OpenStreetMap allows anything that exists and can be verified to be mapped. There is no notability rule that Wikipedia has, for example. So it allows manhole covers to be mapped in detail, it allows telephone lines and the assorted street cabinet boxes that crowd our pavements. You might get feedback as to how to map these features and you might get funny comments about why these features are being mapped (and indeed, mapping with OpenStreetMap is voluntary!) but pretty much all OSM mappers will agree that these features shouldn’t be excluded.

Telecoms features in OpenStreetMap haven’t been well mapped before. This is both good and bad in that the taxonomy (or folksonomy to be more accurate)  – the tags that describe these features -have not been standardized. We have the opportunity to define the tags, or at least standardize some of them to be more consistent across similar telecoms infrastructure features.

I started WikiProject Telecoms on the OSM Wiki, so please go there to see how to map and tag features – and if you are a telecoms, mapping or tagging specialist please suggest better ways to map these features! https://wiki.openstreetmap.org/wiki/WikiProject_Telecoms

The current features being rendered in the New Cloud Atlas map are:

  • Data Centres
  • Telephone Exchanges
  • Manhole covers
  • Telephone poles and wires
  • Submarine cables etc
  • Telecoms towers, masts, and antennae
  • Street Cabinets

Underground features may be more difficult to map – so we are relying on manhole covers which often show what its use and who operates the cable underneath (in the UK at least) – and those markings sprayed by utility companies, and some data imports. If you don’t know where an underground cable goes its probably best to leave it out.

You might have noticed that many of the options include sound and heat for the street cabinets. One of the side effects of today’s modern fibre optic street cabinets is that they are often installed with more power needs than copper wire ones – and so they need a fan. Often the cabinets are warm to the touch and sometimes they make a quite loud drone sound. This type of data can be useful, I have heard, to people who are vision impaired. Sound and touch can help orientate people in space.

Update: There is a JOSM Preset and a Vespucci Preset that Simon Poole developed

Open Data / Secrecy etc

It’s probably worth talking a little bit about the privacy and secrecy issues. Although the project isn’t about getting releases of data from companies and governments, and it’s not about uncovering the secret installations, it is about collaboratively mapping the world. Almost all of the information that will make up the New Cloud Atlas will be found in the field or in public information sources.

You may be reminded of a story in 2003 (2 years after 9/11) of Sean Gorman‘s PhD dissertation that the US Government wanted to make classified as although it contained only publicly available information (about the Internet connections in the US) he analyzed the data to identify the weak links – weaknesses that, for example, a disaster could take out or a terrorist could exploit.  Officials in the US Govt said that his dissertation should be burnt! Sean successfully graduated and started a mapping company with the DHS as clients. (I actually ended up working there at FortiusOne / GeoIQ several years after that for a bit). Now of course open data and open analysis is encouraged and promoted by governments (and following this trend check out Sean’s new startup Timbr.io).

You may also recall stories about how many national mapping agencies removed military bases (such as Aldermaston, or Greenham Common in the UK) from their paper maps – even when these bases were signed from the motorways and major roads and had nice big clear signs outside the fences.  A relic from the Cold War, perhaps. It appears to me that even in this current year the Ordnance Survey mislabels the Menwith Hill USAF/RAF Listening Base in North Yorkshire as just “Menwith Camp” with no indication of it’s real name, activity nor landuse (as compared to OpenStreetMap for example).

At this point, if you are curious, we should evoke the classic 1996 Wired Article by Neal Stephenson: Mother Earth Mother Board http://www.wired.com/1996/12/ffglass/ It’s essential if you are interested in reading more about the geo political and technology of international internet cable laying. It’s also a great read in general.

 

Liverpool Walk / Workshop

Ben and I ran a series of walks and workshops at FACT in June 2016. Cloud Dowsing Hunting for the Hidden Internet and Mapping the New Cloud Atlas

We used FieldPapers to give to participants and mappers and went around the streets of Liverpool.

Here we are near the main telephone exchange and data centre looking for manhole covers, cabinets and antennae, that’s me pointing.

13391350_1053402384736248_1279453524_n

You can view the photos I took on the Flickr Album https://www.flickr.com/photos/chippee/sets/72157671540933095

 

Development Notes

Code for the site is on github: https://github.com/timwaters/new_cloud_atlas

Mapnik / Kosmtik Style file and processing notes also on github: https://github.com/timwaters/cloud_mapping

Mapnik X-Ray Style

Of possible interest to mapnik style geeks could be the use of the scale denominator and PostGIS ST_Scale commands to scale up building polygons so that they appear to be the same size regardless of the zoom. If anyone wants to fix this to make it work better, please let me know!

select st_translate(st_scale(way, (!scale_denominator! * 0.00028) - (5 - z(!scale_denominator!)) ,
 (!scale_denominator! * 0.00028) - (5 - z(!scale_denominator!)) ), 
st_x(st_centroid(way))*(1-( (!scale_denominator! * 0.00028) - (5 - z(!scale_denominator!)) )), 
st_y(st_centroid(way))*(1-( (!scale_denominator! * 0.00028) - (5 - z(!scale_denominator!)) ))) as way,
 building AS type FROM planet_osm_polygon WHERE (building='data_center' ) AS data",

OSM Tile Generation

Tiles are kept up to date at around 15 minutes with the central OSM database. Occasionally a full planet import is done. I think I could use Lua scripting to ensure that the database remains lean. The system uses TileStache to enable the UTFGrids for the popups. Essentially we filter out a lot of stuff from the OSM database:

  1. Convert an osm.pbf file to an o5m file
    ./osmconvert  planet-latest.osm.pbf -o=planet.o5m
  2.  Filter the o5m file to an .osm file
    ./osmfilter planet.o5m --parameter-file=cloud_mapping/osmfilter_params.txt > planet.filtered.osm
  3. Import the .osm file into the database using the custom osm2pgsql Style
     osm2pgsql --slim -d gis planet.filtered.osm -S cloud_mapping/default.style
  4. Set up replication using Osmosis and osm2pgsql to get changes from OSM db
    osmosis --read-replication-interval  --simplify-change --write-xml-change changes.osc.gz
    osm2pgsql -d gis -S default.style -s -C 800 -a changes.osc.gz -e10-19 -o expire_e10-19.list

http://newcloudatlas.org/

Colliding The Mental Maps of Edinburgh with Mapwarper.net

Last autumn I popped up to Edinburgh from the North of England for State of the Map Scotland conference. Together with Edinburgh College of Art in Evolution House participants took part in series of workshops “Map.Makars”

I took part in a memory map of the city. The rules were: no looking at other maps, the map should include the venue, the castle, the train station. We drew, from memory the city on large pieces of paper. Gregory scanned/photographed these and put these on mapwarper.net to stretch them to fit. he then combined these together with an interactive and animated transparency control to create the Hand Drawn Map Collider “No-map Map Give it a whirl! http://www.livingwithdragons.com/maps/nomap-map/

Screenshot from 2016-04-22 11:26:11.png

My map, in case you were wondering was possibly the least accurate of them, coming from furthest away! http://mapwarper.net/maps/10907

Screenshot from 2016-04-22 11:28:00.png

 

Markov Chains, Twitter and Radical Texts

The next few posts will cover some pet projects that I did whilst not being able to work due to recent civic duty.  They cover things from the role of familiar strangers on the internet and anti-social networks, through to meteorological hacks, funny memes to twitter bots. The first in this series is about what happens when you use markov chains and radical texts with twitter.

Detournement is a technique now considered to the father of remixes or mashups, but with a satirical political nature. Have a look at the wikipedia entry for detournement if you want to know more about it. Basically you do something to something which twists or re routes it so that it makes new meanings. It was the Situationists, led by Debord who really adopted and ran with this as a practice.

guy_debord1

Debord would often frequently plagiarise other radical texts in his own work. (The Situationists were also the ones behind original notion of psychogeography – something that you may have caught me talking about before.)

So what would happen if we could detourn, or mashup, or plagiarise Debord’s own writings? And how about if we could publish it periodically, and how about if we had a 140 character limit? Yeah so this is my experiments with these ideas.

Bruna Rizzi; it is from this disastrous exaggeration. The peasant class could not recognize the practical change of products

The proletariat is objectively reinforced by the progressive disappearance of the globe as the bureaucracy can

Markov chains basically work like take a couple of sentences: “A lazy dog likes cheese” and “My house likes to be clean” then look at groups of two or three words together. Then if one of these groups share the same word (“likes”), make a new sentence using that word to chain together. “My house likes cheese” or “A lazy dog likes to be clean”. Markov chains result in sentences that look human readable. The more sentences you feed the population sample, the better or more varied the same of generated sentences.

Some radical texts are complete nonsense and really hard to read, so perhaps applying Markov chains to them can help reveal what truths the obscure language hide.

@markov = MarkyMarkov::TemporaryDictionary.new
@markov.parse_file "debord.txt"
raw_text = @markov.generate_23_words

My solution uses Ruby, the Twitter gem and the marky_markov gem.

https://github.com/timwaters/rattoo  is the work in progress twitter bot – it works currently on Heroku using the scheduler to periodically tweet a sentence, see if any other users have asked it questions and reply back to them.

John Gray on maps and cities.

A map can represent the physical structures of which a city is at any one time composed, but the city itself remains uncharted. This is not only because the city will have changed materially by the time the map appears. A map cannot contain the infinite places that the city contains, which come and go along with the people who pass through them. The map is an abstraction, simplifying experiences that are incomparably more variegated.

From The Silence of Animals. John Gray

Leeds Data Thing, Maps and Hackdays

Leeds Data Thing is a new group started in Leeds  (not to be confused with Leeds Ruby Thing!).

I spoke at the first event (read the write up from Rebecca) about Geospatial visualisations and  OpenStreetMap: Here are the slides:

Since then there has been a few other events as part of Big Data Week – including a load of great short talks.

This weekend there was a data hackday at the UK’s NHS Information Centre for Health and Social Care in the centre of Leeds.

hipster photo

There’s a wealth of data on their website , but it was given to us as a mysql database, and we were able to enter remotely. On the first day I poked around the data and had a thought.

Hackdays

I often spend the first part of any hackday wondering what to do, and twiddling thumbs. I find that hackdays become for me a type of busman’s holiday – and this hackday was particularly geographical in nature. Most of the entries had some kind of data on map component. I think that these types of analyses, whilst being very smart and interesting – and may be exactly what the judges are looking for, may not exactly stretch the unexpected or “the hack” in the data.

Fortunately there was plenty of latitude for exploring things laterally. The most interesting dataset was listing the chemicals and drugs each practice spent money on – but I couldn’t find much to do with it.   What caught my eye was the dataset listing the names of the doctors surgeries, practices, medical centres. If I think about my neighbourhood I can pass about half a dozen doctors in a very small area. Leeds is well covered (or perhaps just my area is!) . I was reminded of James Joyce’s quote about being unable to cross Dublin without passing a pub. Perhaps the same can be said for Leeds and doctors!  The names of the surgeries were also interesting. Names such as:

Chapeloak Surgery
The Avenue Surgery
Dr Ca Hicks’ Practice
The Dekeyser Group Practice
The Highfield Medical Centre
Chapeltown Family Surgery

Wonder if the more “leafy” the name, the more “leafy” the neighbourhood it was in? Perhaps the more grandiose sounding practices had more patients? Perhaps the smaller sounding ones had better patient satisfaction reviews?

At the venue, it appeared that I was the only one to be using Linux on the desktop and so the wifi did not work – so I had a bit over one hour to put something together. Decided to go with the concept of “Leeds is covered” and wanted something showing the labels of the practices over the areas where they were. Filling out the map, so to speak.  The hack was called “Tim’s One Hour Data Challenge” and here is the end result:

Leeds is covered

WhereCampEU 2011 recap & berlin psychogeography

Last week in Berlin I was lucky enough to go to WhereCampEU – thanks to Gary and Chris for organising this wonderful unconference. The conference was held in a trendy hipster ish part of the city, but which had also, I heard, the highest number of young families and births. It was also in the former Eastern part of the city. It gave the area a nice appeal, overall.

photo by Chris Fleming

I did a couple of sessions, one on a preview of GeoCommons2.0  talked about in a previous post and the other a psychogeography session. For the psychogeography session I sent four teams out to explore the environs around the campus.

One team followed people around. They said “I’m amazed by how slowly some people moved” and “Well, often we followed someone and then they would wander into a book shop”  – revealing the nature of the people and the type of area, bohem style cafes and shops, lazily people.

Another group were sent to ask people to point to were the centre of Berlin was. I asked some people where they thought was the centre, and most of them scratched their chins, and pointed to the Mitte area of the city, usually on the map, or waved southwards. Part of a consequence of being a split city, really. The western bit, someone said, “looks and feels more like a CBD” – that is, big shops, tall towers etc. I did venture to the former western CBD centre, and came across a mile long car show. This area was where the money was.

The other group was sent to walk around the area according to the Game of Life algorithm, Left left Right where you walk and take the first left, then the second left, then the next right, and so on. It’s impossible to predict where you will end up. I joined this group. We had a good explore over a small area, really, but encountering a lot of different environments. Shared (private) gardens / courtyards in the middle of apartment blocks, churches, cafes, and shops.

The fourth team were given a secret mission, and so I cannot reveal to you what they did. However, they are all in good health, and saw the city in a new light.

Photo by Chris Fleming

Back to the unconference, and some of the highlights were:

* Playing the Skobbler game, treasure hunting for addresses in the neighbourhood.

* Seeing offmaps evolve over the year. I’ve not got an iPhone, but that app looked very nice.

* Spatial databases, and in particular CouchDb – and their spatial bits

* CASA did a few talks – I’m getting more and more fond of their work – if anything they really seem to love the stuff they are doing – they share the same vision as me as giving GI tools and benefits to as many people as possible.

* Peter Batty wore an ipad t-shirt – and gave a great presentation about essentially putting utilities information onto a Gmaps like interface and mobile map.

* Gary Gale gave a compelling reason for standardizing place. And it makes sense.

* Meeting the NomadLabs guys for the first time, and being able to say “Thank You” for their work on Ruby on Rails GIS Hacks that I found very useful 4 years ago!

* Corridor talk, beer and food

Henry Miller on Seeing a Place

From “The Eye of Paris” in The Wisdom of the Heart – Henry Miller, 1941, New Directions:

Now and then, in wandering through the streets, suddenly one comes awake, perceives with a strange exultation that he is moving through an absolutely fresh sliver of reality. Everything has the quality of the marvelous – the murky windows, the rain-sodden vegetables, the contours of the houses, the bill-posters, the slumping figures of men and women, the tin soldiers in the stationery shops,  the colors of the walls – everything written down in an unfamiliar script. After the moment of ecstasy has passed what is one’s amazement but to discover that the street through which he is walking with eyes popping is the street on which he lives. He has simply come upon it unaware, from the wrong end perhaps. Or, moving out of the confines of an unknown region, the sense of wonder and mystery prolonged itself in defiance of reality. It is as if the eye itself had been freshened, as if it had forgotten all that it had been taught. In this condition it happens that one really does see things he had never seen before –  not the fantastic, harrowing, hallucinating objects of dream or drug, but the most banal, the most commonplace things, seen as it were for the first time.

Sticks & Booze – Beating the Bounds in Headingley

About 25 people turned up for a Beating the Bounds psychogeographical walk around the Headingley (Leeds, UK) DPPO Boundary.  It was run by me and the Leeds Psychgeography Group. Tina runs and blogs about the group, and there’s a Leeds Psychogeography Group Facebook Page to boot.  All photos here were taken by Mark Jaffé, cheers!

A DPPO stands for a Designated Public Place Order. Essentially within that area, if you are causing a nuisance or annoyance, a police constable can stop you drinking, confiscate booze, up end cans etc. If you don’t comply then that is when you may be breaking the law. It’s a law to stop street drinkers mainly. In the Hyde Park and Woodhouse DPPO area, well over 80%  (over 300 in a few months) of people stopped, were students. Phil Kirby went on the first of the Beating the Bounds walks, and has blogged about it on The Culture Vulture.

We were to talk widdershins, anti-clockwise around the boundary – a magical act, designed to disperse any energies or what have you. I brought along some sticks (loop cane, chopped in half) for people to beat the ground at certain points. Oh, and we drank as we went – essentially beating the restrictions.

We passed through some nice areas – here on the ridge, was the Wassailing tree. We also encountered the strangest plot of land in Headingley, fish and chips and had assorted adventutres.

Games developed, whereby when a DPPO warning sign was found, a drink had to be taken.

It took about 2 hours. I was quite tipsy at the end. in the above picture, we can see a stick being used in it’s traditional role of beating a boy.

I’ll be doing some more psychgeography posts in time. The next beating the bounds walk will go around the city centre – it’s a huge area!

Settle Mapping Party, Sat 15th May

Here follows a blog post that’s written like a press release, sorry.

A group of volunteers from around the North of England on Saturday 15th May 2010, will attempt to map the entire North Yorkshire town, from every street, bridge, footpath and chip shop – in order to create a free and open map of the town. All welcome, no experience or technology required!

The Association for Geographic Informations Northern Group and the OpenStreetMap Foundation are running a mapping party – a cross between an informal fieldtrip and a hands on workshop. OpenStreetMap is the wikipedia of maps – it’s open, free and anyone can edit and contribute.

Organiser Tim Waters said: “OSM aims to create free geographic data, like street maps, that can be used by anyone, anywhere, and over the Saturday we aim to have a complete map of the streets of Settle and many other features in the town.”

With the announcement of the Ordnance Survey releasing a lot of mid scale mapping data for free, the chances of having a top notch detailed map is greater than ever. By making a free and open map, anyone can edit and correct details, making sure the map stays up to date and relevant. It’s also free to copy and change and distribute, which is impossible to do with almost every other map.

Anyone and everyone is welcome to attend, families and children are also welcome! No previous experience needed, and no GPS units needed either. GPS units will be available for people to borrow, but people can contribute a lot by using a pen and paper. It’s an open organisation with no membership requirements.

People will start assembling at 10 – 10:30 a.m at Ye Olde Naked Man cafe in Settle’s central Market Place, and spend the morning mapping the area. Then they will come back and have some lunch, meeting at 1pm at Thirteen Cafe Bar and either head out again to fill in the gaps, or start editing their notes into the map system. The day comes to an end around half 3pm to 5pm, where volunteers recap on the days mapping, and have a natter over a pint of beer

More information to sign up:

http://wiki.openstreetmap.org/wiki/Settle_Mapping_Party

http://www.agi.org.uk/north/