Geocoding to a shape instead of a point

The project I'm currently working on is Find-A-Record. We are geocoding genealogical record collections and storing them in a spatial index (browse our blog if you want to know more). Searches are based on a shape. We return collections which intersect or are contained within the search area.

During the early states of development, we used geonames to geocode collections to a point. This works well for collections which are associated with lower administrative levels such as cities, towns, and villages. However it really breaks down when you get the county, state, and country level.

The 1940 US Census is associated with the United States and would be assigned a point in Northern Kansas. Any queries within the US that aren't near that point won't return the 1940 US Census.

To solve this we need to geocode collections with a shape instead of a point.

OSM

OpenStreetMaps has the data we need, but it's extremely difficult to extract. The administrative hierarchy is not explicitly stored. Nominatim is used to solve this problem for OSM a Nominatim search only returns features. So a query for Knighton on Teme returns two bus stops but not the administrative boundary relation.

The Overpass API looked promising but it can't do fuzzy string matches. Overpass can only do exact or regex matches. We could use Overpass if there was an easy way to standardize place names. In other words, if OSM provided a way for us to standardize "Knighton on Teme, Worcestershire, England" to "Knighton on Teme CP, Malvern Hills, Worcestershire, West Midlands, England, United Kingdom" according to the OSM hierarchy then fuzzy string matching wouldn't be necessary.

Summary

What we need is a service which allows us to perform fuzzy string searches for a place (or administrative level) and retrieve it's boundaries.

We recognize that it will be difficult to obtain boundary data for the entire world. Thankfully we probably won't need to anytime soon. We only need data for areas of the world where genealogical records exist and genealogists do research.

It's looking like we will need to build our own service which indexes OSM in such a way that enables us to query for administrative boundaries. But we would really prefer not to. Is there any other way we can retrieve this data with existing services?

Cool project! You might take a look at MapIt: Global:

MapIt is a service that maps geographical points to administrative areas. This edition is based on source data from the totally amazing OpenStreetMap project, so add your boundaries there if they’re missing. If you’re in the UK our MapIt UK with open Ordnance Survey data will probably be more useful.

MapIt is useful for anyone who has the co-ordinates of a point on Earth, and who needs to find out what country, region, city, constituency, or state it lies within. It’s also great for looking up the shapes of all those boundaries.

Charitable, low volume use of this service is free – read more.

You can download the source on Github.

Need a licence? Read more or get in touch (commercial@mysociety.org).

  • That's pretty close. I wish they would allow for string searches though. We can pretty much to the same thing with Overpass. – user23321 Dec 5 '13 at 0:25
  • Installing the entire system on your own servers, which is the only way to use the results commercially, looks to be quite a complex process as documented at code.mapit.mysociety.org (they recommend 500GB free space if using OSM). Be aware that one of the sources they use, GADM, has a non commercial restriction as well. – Rob Hoare Dec 5 '13 at 0:39
  • They do have commercial licensing according to global.mapit.mysociety.org/licensing – blah238 Dec 5 '13 at 1:29
  • 1
    @JustinY: Hi, I'm one of the developers who worked on MapIt: Global. I guess you've seen that you can search by string prefixes (example) - if you'd like fuzzy searching could I suggest you request that in the issue tracker? It's an open source project, so even if we don't have time to work on that, perhaps a volunteer will. – Mark Longair Dec 19 '13 at 14:32
  • 1
    @MarkLongair Great suggestion. github.com/mysociety/mapit/issues/101 – user23321 Dec 19 '13 at 15:24

I'm also working with global placenames and boundaries data (including historical) at present so I understand what you're going through!

However, I think there's really two steps to your problem: 1. use a fuzzy name search to turn the name of an admin area into something standard, then 2. look up the boundaries of that place in databases of boundaries. Rather than trying to find data which also has a fuzzy search.

WeRelate Places might help with step 1 for old place names, and many other geocoders like you already use can standardise modern names.

Step 2 is really difficult at present, as much of the suitable boundary data is under a non-commercial license (or worse), especially for historical data. A starting point for countries and admin 1 levels is Natural Earth.

Then there are national-based sources for lower admin divisions, which are extracted from open government data, such as USGeoJSON and uk-atlas. Nothing I'm aware of is really global in scope.

Do be careful to trace the provenance of data and the license, a non-commercial restriction is very common, for example for GADM, Vision of Britain, and (US) Atlas of Historical County Boundaries.

  • Thanks for pointing out that there could be two steps. If there was an easy way to use OSM for standardizing names then we could easily solve our problem (with the boundaries that OSM has available). – user23321 Dec 5 '13 at 4:02
  • At this point we're not terribly concerned with obtaining quality data. We mostly need the large admin areas such as states and countries. Quality county data would be nice but it's not necessary. – user23321 Dec 5 '13 at 4:12
  • At that level, down to adm1 in each country, Natural Earth is probably your best option to start with (the ADM0 and ADM1 boundaries). The SHP files can be converted to GeoJSON or various other formats with ogr2ogr and other similar utilities. – Rob Hoare Dec 5 '13 at 4:20
  • Wow, somehow I missed that in your answer. Thanks for pointing it out again. – user23321 Dec 5 '13 at 4:28

Your Nominatim query includes a place_id field which has the value 4828147 in your example. This field serves as an internal identifier for Nominatim (so it is not globally unique). But it can be used to view Nominatim's internal address hierarchy for this place. This hierarchy also contains boundary relations, for example relation 1875245 and several other.

By walking through this list and looking at the admin_level key it should be possible to find the relation you are looking for. Unfortunately this special page doesn't seem to be available through an API yet.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.