4

Starting with the full OSM Planet data, I'm trying to create smaller, individual osm/pbf files, or eventually Spatia/SQLite or GeoJSON files, that are split on each UTM grid zone. I'm using osmfilter to trim down the data, and I imagine with a UTM zone shapefile, you could create a filter for each UTM zone in a batch process, perhaps using osmosis or mapsplit (but I don't know how, nor do I know where to get the shapefiles or which format to use with osmfilter).

The reason is that I want my individual osm/pbf files to be somewhat evenly sized, and not proportional to the size of the country (for example, the US might be 10 GB and Luxembourg 10 KB).

Is there anyway to do this using the OSM tools like osmfilter?

Or could I do this after I've converted to Spatia/SQLite with ogr2ogr?

update UTM zones shapefile is available here, although I still don't know how to create individual files from planet.osm based on this shapefile

Here's what Europe looks like in UTM - so each of these zones would be one file, derived from planet.osm with osmfilter applied to keep only certain tags.

enter image description here

  • 2
    I played with Osmosis, a simple Java-based tool for querying OSM files. Check out this page: wiki.openstreetmap.org/wiki/Osmosis and do a ctrl+f for "Extract an area". – Jon Feb 23 '18 at 16:35
  • @Jon that should work - can you post it as an answer? – philshem Feb 24 '18 at 18:34
3
+25

I use Python predominately, so here's a skeleton workflow I'd use for this. There are probably better ways to go about it, but this one will get you what you want.

1) Load your UTM shapefile using shapely or geopandas (which uses shapely).

2) I haven't looked at the UTM shapefile, but I assume each cell is defined by four corner lon,lat points. If so, move to next step. If not, you can use the shapely geometry method .box to return the bounding box.

3) Use Osmosis to do the heavy lifting, see here for examples on extracting by area/bounding box. I would loop through each polygon, grab the corner coordinates, the make a subprocess call to osmosis. The relevant options for osmosis are:

--bounding-box top=49.5138 left=10.9351 bottom=49.3866 right=11.201

You can loop through each feature of the shapefile, grab coordinates, create subprocess string, call subprocess. It's easy parallelizable as this is probably a bigger job.

I don't know if there's any value in buffering your bounding boxes a little or if that would cause issues with whatever application you're building. shapely (and geopandas) has easy functions for that. There could be rounding errors or something strange which leave tiny gaps between cells; the safest thing to do is run a couple before cranking the whole planet file. Also, when I did a global extraction on the planet file, I found it faster to unzip the planet file first rather than do it on the fly. But that takes ~1TB of space.

  • thanks for your answer. I plan to roll out the planet.osm process on a cloud virtual machine so I can pay for a powerful processor and enough disk space, but then only turn it on when I need it. there are also high memory cloud servers which could handle the entire ~1TB. – philshem Feb 26 '18 at 7:32
1

Using Mapsplit is efficient for this. You should convert the UTM zones into poly files then you can batch process it in a loop in command line using each file as the --polygon=file parameter in Mapsplit.

ogr2poly.py can help to convert shapefiles to poly files. More options are available at Converting_to.2Ffrom_POLY_format

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.