Rain Prediction for the immediate future using Met Office DataPoint – rain graph

Part of a series of posts to cover some small projects that I did whilst not being able to work. They cover things from the role of familiar strangers on the internet and anti-social networks, through to meteorological hacks, funny memes to twitter bots. This post is about a meteorological hack.

The Met Office in the UK have in this last year published an API for their range of services – all part of the open data movement I think.

DataPoint is a way of accessing freely available Met Office data feeds in a format that is suitable for application developers. It is aimed at professionals, the scientific community and student or amateur developers, in fact anyone looking to re-use Met Office data within their own innovative applications.

The year before this, in Denver, USA, I was shown by a couple of awesome mapping and weather geeks a mobile app that showed when it was going to rain, and more importantly when it wasn’t going to rain in very high temporal resolution. You can use the app and know whether to get a swift half and then leave to get the bus, or whether to stay in for an hour until the showers end. It was very detailed and highly useful. This app Dark Sky App

was freaking awesome. And I want here in the UK, so when the Met Office announced their API I was interested.

You cannot do what DarkSkyApp does with the Met Office DataPoint API though – what you can do is do some interpolations though. The API for precipitation forecasts only gives access to a 3 hourly map tile.

http://www.metoffice.gov.uk/datapoint/product/precipitation-forecast-map-layer

Although further poking around shows that they do have an undocumented 1 hourly image.

Screenshot - 061213 - 16:05:15

These map tiles then could be used. http://rain-graph.herokuapp.com  is the resulting application with the code here: https://github.com/timwaters/rain_graph

It’s a Ruby Sinatra application which for a location, grabs the precipitation tile for a defined location for each hour from now. It looks at the pixel value for the given location and determines the amount predicted. It shows when the heaviest rain is predicted and when it would stop. Interpolation is given by the graph engine itself – no fancy meteorological modelling is done (at this stage). It uses Chunky_png to get the pixel values.

687474703a2f2f692e696d6775722e636f6d2f4343306f5868772e706e67

All requests are cached to avoid hitting the MetOffice API and because an image won’t change for an hour. Additionally it uses another API method to get a human readable upcoming forecast text for that location, and displays it under the graph. Contrary to popular global belief it’s not always raining in the UK, and so most of the time you will never see a a graph showing something!

Considerations:

Pixels to Lat Lon:
Since a lat/lon location is quite specific, it could map to one pixel in a tile, and that pixel could have a lower or higher value than the ones surrounding it. I could use a kernel average – do a 6×6 pass over the pixel and get the average value. But since there are tiles are lower zoom levels, by zooming out, the spatial extent of the pixel would equal that larger area – it would do the work for us.

Interpolation between forecasts:
It wasn’t clear if the forecast images showed the predicted situation over the whole hour, or whether it showed the situation at that moment. Should a user look at an animation to see how rain cloud moves across from A->B and guess that in between that there would be rain, or should they think that that there would be no rain if there is no rain shown?

User Interface:
It looks a bit bland – we should show the image tiles underneath  - perhaps shown when hovering over a point.

Accuracy:
I haven’t tested the accuracy of this.

Location hard coding:
The text forecasts are hardcoded to a set number of regions, but we could do a closest point and get the correct forecast for the given lat and lon.

Use Yr.No API

Yr.no has detailed hour by hour forecasts API for a place giving the amount of precipitation.

http://www.yr.no/place/United_Kingdom/England/Leeds/hour_by_hour_detailed.html

<time from="2013-12-06T19:00:00" to="2013-12-06T20:00:00">
<!-- Valid from 2013-12-06T19:00:00 to 2013-12-06T20:00:00 -->
<symbol number="3" name="Partly cloudy" var="mf/03n.11" />
<precipitation value="0" /><!-- Valid at 2013-12-06T19:00:00 -->
<windDirection deg="294.2" code="WNW" name="West-northwest" />
<windSpeed mps="4.3" name="Gentle breeze" />
<temperature unit="celsius" value="1" />
<pressure unit="hPa" value="1004.9" />
</time>

Mapwarper.net running on a faster, newer server!

Mapwarper.net has finished maintenance work – I really should get rid of the beta sign now. Anyhow its running on a much faster site, courtesy of Topomancy.com. So performance should be better now. All user accounts and maps and points should have been transferred seamlessly.

mapwarper.net georectify maps georeferencing fun!

I’m still configuring the mail server, so if new users when signing up are seeing mail in spam boxes, let me know!

Cheers!

GeoCommons GeoJSON in OpenLayers

So, with GeoCommons you can export all the features from a dataset in GeoJSON format. This is very useful. Then can it be displayed in Openlayers? Why yes it can!

I use the following strategy, wrapping the response with a little bit extra so that the GeoJSON format can read it properly.

var url = "http://geocommons.com/overlays/128725/features.json?limit=100";

var p = new OpenLayers.Format.GeoJSON();

OpenLayers.loadURL(url, {}, null, function (response) {
 var gformat = new OpenLayers.Format.GeoJSON();
 gg = '{"type":"FeatureCollection", "features":' +
      response.responseText + '}';
 var feats = gformat.read(gg);

 vector_layer.addFeatures(feats);
});

Have a look a the live demo of this example of loading points from a Geocommons dataset straight into an OpenLayers map
(Note that I am using an OpenLayers.ProxyHost proxy to make it work in FF)

It is of course very basic in terms of styling, but it’s a start!

Incidentally the points are from “CARMA, India Power Plant Emissions, India, 2000/ 2007/Future” (carbon monitoring)

WhereCampEU 2011 recap & berlin psychogeography

Last week in Berlin I was lucky enough to go to WhereCampEU – thanks to Gary and Chris for organising this wonderful unconference. The conference was held in a trendy hipster ish part of the city, but which had also, I heard, the highest number of young families and births. It was also in the former Eastern part of the city. It gave the area a nice appeal, overall.

photo by Chris Fleming

I did a couple of sessions, one on a preview of GeoCommons2.0  talked about in a previous post and the other a psychogeography session. For the psychogeography session I sent four teams out to explore the environs around the campus.

One team followed people around. They said “I’m amazed by how slowly some people moved” and “Well, often we followed someone and then they would wander into a book shop”  - revealing the nature of the people and the type of area, bohem style cafes and shops, lazily people.

Another group were sent to ask people to point to were the centre of Berlin was. I asked some people where they thought was the centre, and most of them scratched their chins, and pointed to the Mitte area of the city, usually on the map, or waved southwards. Part of a consequence of being a split city, really. The western bit, someone said, “looks and feels more like a CBD” – that is, big shops, tall towers etc. I did venture to the former western CBD centre, and came across a mile long car show. This area was where the money was.

The other group was sent to walk around the area according to the Game of Life algorithm, Left left Right where you walk and take the first left, then the second left, then the next right, and so on. It’s impossible to predict where you will end up. I joined this group. We had a good explore over a small area, really, but encountering a lot of different environments. Shared (private) gardens / courtyards in the middle of apartment blocks, churches, cafes, and shops.

The fourth team were given a secret mission, and so I cannot reveal to you what they did. However, they are all in good health, and saw the city in a new light.

Photo by Chris Fleming

Back to the unconference, and some of the highlights were:

* Playing the Skobbler game, treasure hunting for addresses in the neighbourhood.

* Seeing offmaps evolve over the year. I’ve not got an iPhone, but that app looked very nice.

* Spatial databases, and in particular CouchDb – and their spatial bits

* CASA did a few talks – I’m getting more and more fond of their work – if anything they really seem to love the stuff they are doing – they share the same vision as me as giving GI tools and benefits to as many people as possible.

* Peter Batty wore an ipad t-shirt – and gave a great presentation about essentially putting utilities information onto a Gmaps like interface and mobile map.

* Gary Gale gave a compelling reason for standardizing place. And it makes sense.

* Meeting the NomadLabs guys for the first time, and being able to say “Thank You” for their work on Ruby on Rails GIS Hacks that I found very useful 4 years ago!

* Corridor talk, beer and food

Some stuff I’ve been working on with new GeoCommons 2.0

Last weekend, if you were at WhereCampEU in Berlin (blog post to follow) , you may have caught my sneak peak into the new GeoCommons 2.0, which has been revealed just the other day. Here are some of the highlights of the new GeoCommons

  • The flash map has been overhauled and re-written, mainly by Andrei – and it can handle hundreds of thousands numbers of points quite happily
  • Analytics library is completed, but not currently accessible to normal users of GeoCommons – hopefully it will be soon, if people want it.
  • Behind the scenes, the system uses a number of distributed workers and tasks to offload processing intensive or long processing tasks
  • Datasets and Maps get given nice overview images, and the attributes of datasets have histograms generated for them
  • Data can be edited in the system, and filtered, and saved either to replace itself or as a new dataset
  • Animation of temporal data is much nicer now
  • Polymaps for HTML5 non-flash map support
  • Filters can be applied to the map, so that attributes can be filtered out.
  • Thematic maps can be made with categories now
  • Acetate is used as standard
  • Custom markers can be added to a map, and even animated ones work too!
The GeoIQ developer blog has a developer orientated review of wha’ts new and there is a good overview of GeoCommons on the main GeoIQ blog too.
Keep your eyes peeled on the GeoIQ Developer Blog over the next few days as the team adds some more posts about some of the technology behind it.

WherecampUK 2010 Recap

Last week I journeyed down on the train to Nottingham to go to WhereCampUK – an unconference for all things “geo” – but it was only a few months since the similarly named WhereCampEU of (which I never actually wrote something about) down in London. Before I share some of the best bits, here’s some of the similarities and differences.

* Less international folks

* Less big geo personalities and keynotes

* More OSM

* No T-Shirts

* More beer – we drank a large pub dry, literally. The next day, the landlord swore at me for pissing off their regulars.

* More cake

* Cheaper and quicker to run, setup and organise.

For more pointers in how to run an unconference, check out Steve Coast‘s latest post, where he writes about what he did for Wherecamp in Denver.  How I ran a successful unconference in 6 hours and you can too.

Overall, the event was great.

I ran two sessions. The main one was “What is Psychogeography“. The best part of this was sending all participants out with directions in twos and threes, for 10 mins before lunch. They had directions such as “left left right”, “follow someone”, “ask where the centre is, follow that direction, ask again”, “find hidden portals”, “find fairies”, “hear something, take a photo”.

I also quickly slotted in the NYPL Warper presentation, and included this slide. You get 20 points if you know what this refers to!

I also mentioned the words “neogeography” for the first time in the conference, and that was at 3:30pm, which says quite a bit about the use of the term.

Talks I liked were:

* Vernacular Geography & Informal Placenames

* Geo Games

* Education and mobile maps

* Augmented Reality roundup

* How streets get names

* Peoples Collection Wales

* Haptic Navigation

* OSM Talks including – Potlatch 2

* Gregory Mahler’s – I’m a Psycho Mapper!

* OSGEO



AGI GeoCommunity – NYPL & Mapwarper

Yesterday I presented at the AGI Annual conference on the NYPL Map Rectifier. I was also able to launch MapWarper.net – to showcase the user submitted version of the code base, which you can get on github, should you like! I’ll put the slides up to slideshare asasp

I also showed the video of the newest Stamen Design work on the redesign of the interface. This interface will be applied to the application, and will be featuring at the British Library Growing Knowledge exhibition in a couple of weeks

The whole conference was okay, well organised, and well attended by fellow AGI Northern Group members. The soapbox – an evening even over beers – short 20:20′s but with the emphasis on ranting and comedy proved very popular, and was very amusing – keep an eye out for the videos when they come out!

The free W3Geo unconference which was on the day before the main conference, and as Ed writes, had the more interesting talks and energy here. w3g had a number of fixed keynotes and speakers, which I think was a good idea – it allowed people to attend knowing that certain people would be speaking. Alas, I didn’t really believe it was promoted that well beforehand.

The main conference had a number of interesting talks. The guys from CASA really do seem to be enjoying themselves. Survey Mapper was highlighted – essentially it’s an onlne survey application, but with, yes geography! It would be great to be able to allow the export of the results from a survey, and use this as a data layer in GeoCommons – to compare against any number of other datasets. They also showcased their work with Twitter – exploratory – one thing with pretty much all twitter map experiements is that people are so overwhelmed with the dataset, that the analysis, the task of generating information from this data tends to be overlooked. Most of the mapping done tends to show that, for the very first time, that people tend to live and do things in urban areas.

ESRI were keynoting, but with a somewhat downbeat “why doesn’t anyone understand and use GIS” cry, with a call to promote the usefullness of GIS to decision makers. Nothing new here, but in these economic times, seemed to imply that GIS budgets were being hit hard.

Nigel Shadbolt’s closing talk convinced me of the importance of linked data. However, the talk was more about the benefits and importance of Open Data as a whole, and he was speaking at such a higher level than most of the other talks were operating at. They were talking about “how to share” data, whilst with open data at a national level operates at a much more fundamental level

The Ordnance Survey have now escaped much of the fights and shouting what with the OS Open Data, and today have announced better, clearer, more generous terms for Derivative data (see here and here for more info on that

Platial – some lucky pennies

So one of the original poster childs of neogeography has folded – Platial have bit the dust! Geocommons have rescued some of the data (but I still couldn’t find my pennies there yet).

Way back in 2007, I used Platial for recording where I found lucky pennies on the streets. (Nowadays I use Flickr to record lucky pennies)

A year later, and 2008 saw Platial awarding me a StoryTelling Award - a t-shirt which I’m currently wearing whilst writing this post.

You can get the Lucky Pennies Map as a KMZ, or maybe from the page on Platial itself, but that page won’t be around forever.

WhooMS – a tiny public geotiff WMS server

WhooMS is a tiny public WMS server for those people who have a GeoTIFF and need someplace to serve it as WMS. I wrote it a while ago, but now its running for all to use.

It’s written in Ruby, using the Sinatra Web Framework, which basically means it can all fit neatly on one file.  It uses Ruby Mapscript to read the uploaded GeoTiff and serve it out to the world.

Nice and simple and basic. Got a GeoTiff handy? (EPSG:4326) give it a go. whooms.mapwarper.net

The code for it can be found on github: http://github.com/timwaters/whooms

Main caveat: when the disc space gets full, the older files will be deleted to fill up space.