Geocoding to shape instead of coordinate

Question

Geocoding to a shape instead of a point

The project I'm currently working on is Find-A-Record. We are geocoding genealogical record collections and storing them in a spatial index (browse our blog if you want to know more). Searches are based on a shape. We return collections which intersect or are contained within the search area.

During the early states of development, we used geonames to geocode collections to a point. This works well for collections which are associated with lower administrative levels such as cities, towns, and villages. However it really breaks down when you get the county, state, and country level.

The 1940 US Census is associated with the United States and would be assigned a point in Northern Kansas. Any queries within the US that aren't near that point won't return the 1940 US Census.

To solve this we need to geocode collections with a shape instead of a point.

OSM

OpenStreetMap has the data we need, but it's extremely difficult to extract. The administrative hierarchy is not explicitly stored. Nominatim is used to solve this problem for OSM a Nominatim search only returns features. So a query for Knighton on Teme returns two bus stops but not the administrative boundary relation.

The Overpass API looked promising but it can't do fuzzy string matches. Overpass can only do exact or regex matches. We could use Overpass if there was an easy way to standardize place names. In other words, if OSM provided a way for us to standardize "Knighton on Teme, Worcestershire, England" to "Knighton on Teme CP, Malvern Hills, Worcestershire, West Midlands, England, United Kingdom" according to the OSM hierarchy then fuzzy string matching wouldn't be necessary.

Summary

What we need is a service which allows us to perform fuzzy string searches for a place (or administrative level) and retrieve it's boundaries.

We recognize that it will be difficult to obtain boundary data for the entire world. We only need data for areas of the world where genealogical records exist and genealogists do research.

It's looking like we will need to build our own service which indexes OSM in such a way that enables us to query for administrative boundaries but we would really prefer not to.

Is there any other way we can retrieve this data with existing services?

Answer 1

Cool project! You might take a look at MapIt: Global:

MapIt is a service that maps geographical points to administrative areas. This edition is based on source data from the totally amazing OpenStreetMap project, so add your boundaries there if they’re missing. If you’re in the UK our MapIt UK with open Ordnance Survey data will probably be more useful.

MapIt is useful for anyone who has the co-ordinates of a point on Earth, and who needs to find out what country, region, city, constituency, or state it lies within. It’s also great for looking up the shapes of all those boundaries.

Charitable, low volume use of this service is free – read more.

You can download the source on Github.

Need a licence? Read more or get in touch ([email protected]).

Answer 2

I'm also working with global placenames and boundaries data (including historical) at present so I understand what you're going through!

However, I think there's really two steps to your problem: 1. use a fuzzy name search to turn the name of an admin area into something standard, then 2. look up the boundaries of that place in databases of boundaries. Rather than trying to find data which also has a fuzzy search.

WeRelate Places might help with step 1 for old place names, and many other geocoders like you already use can standardise modern names.

Step 2 is really difficult at present, as much of the suitable boundary data is under a non-commercial license (or worse), especially for historical data. A starting point for countries and admin 1 levels is Natural Earth.

Then there are national-based sources for lower admin divisions, which are extracted from open government data, such as USGeoJSON and uk-atlas. Nothing I'm aware of is really global in scope.

Do be careful to trace the provenance of data and the license, a non-commercial restriction is very common, for example for GADM, Vision of Britain, and (US) Atlas of Historical County Boundaries.

Answer 3

Your Nominatim query includes a place_id field which has the value 4828147 in your example. This field serves as an internal identifier for Nominatim (so it is not globally unique). But it can be used to view Nominatim's internal address hierarchy for this place. This hierarchy also contains boundary relations, for example relation 1875245 and several other.

By walking through this list and looking at the admin_level key it should be possible to find the relation you are looking for. Unfortunately this special page doesn't seem to be available through an API yet.