Convert large OSM files to GeoJSON

Question

I have been trying to convert a large OSM file (5G) to the GeoJSON format. All the tools I have been using, use extensive memory and throw an out of memory exception.

What is the best way to convert a large OSM file to the GeoJSON format? Do I have to split the OSM file?

Answer 1

Standalone tools

You can try it with ogr2ogr using the OpenStreetMap driver.

 ogr2ogr -f GeoJSON output_points.json input.osm.pbf points
 ogr2ogr -f GeoJSON output_lines.json input.osm.pbf lines
 ...

Note ogr2ogr reads OSM-files with 5 layers: points, lines, multilinestrings, multipolygons and other_relations. So you might need to repeat the upper command with each layer. You will obtain 5 JSON layers finally.

An alternative can be osmtogeojson. It is written in JavaScript and has a commandline interface.

However, not sure if it works with large OSM files like yours.

PostgreSQL

Using PostgreSQL works for sure (like mentioned below).

You can import your OSM file using ogr2ogr, osm2pgsql or imposm. For the latter both you need to define which tags of OSM should actually be imported.

Once you have all your OSM data in the PostgreSQL database (probably in many tables) you can export them using ogr2ogr (like in this post):

ogr2ogr -f GeoJSON output.json PG:dbname="my_database" "my_table"

Answer 2

You might want to try out osmium export, which is based on the libosmium library, a proven and widely used lib for high performance OSM processing:

https://docs.osmcode.org/osmium/latest/osmium-export.html

Installation instructions are available here: https://osmcode.org/osmium-tool/

Install

brew install osmium-tool

Extract

extract .pbf to .geojson

osmium export xxx.pbf -o xxx.geojson

Answer 3

If you know how to use PostGIS you can use the following query to construct a valid geojson:

COPY (
SELECT jsonb_build_object(
  'type',     'FeatureCollection',
  'features', jsonb_agg(feature)
)
FROM (
  SELECT jsonb_build_object(
    'type',       'Feature',
    'id',         id,
    'geometry',   ST_AsGeoJSON(geom)::jsonb,
    'properties', to_jsonb(inputs) - 'id' - 'geom'
  ) AS feature
  FROM (
    SELECT * FROM input_table
  ) inputs
) features
) to 'C:\temp\output_geojson.json';

Answer 4

The following Python script can stream through the data, requiring no memory. It converted my 3.1G OSM file from XML into GeoJSON in 81 seconds.

In my case, I had to convert a bunch of ways into either LineString's or Polygons, so I used the existence of highway=* to differentiate between the two. However, you may need to tweak the logic to fit your use case.

import json
import sys

sys.stdout.write('{"type":"FeatureCollection","features":[')

with open(sys.argv[1]) as f:
    # Skip irrelevant metadata lines
    while "<way" not in next(f):
        pass

    coords = ""  # Coordinates of each geometry
    is_highway = False  # Tracked for each way
    is_first_way = True  # Tracked for printing commas

    for line in f:
        # If we close a way, flush it and reset
        if "</way" in line:
            coords = "[" + coords[:-1] + "]"
            if not is_highway:
                coords = "[" + coords + "]"
            sys.stdout.write(
                [",", ""][is_first_way]
                + '{"type":"Feature","properties":null,"geometry":{"type":'
                + ['"Polygon"', '"LineString"'][is_highway]
                + ',"coordinates":'
                + coords
                + "}}"
            )
            is_first_way = False
            coords = ""
            is_highway = False

        # Add coordinates from node
        elif "<nd" in line:
            split = line.split('"')
            coords += "[" + split[-2] + "," + split[-4] + "],"

        # Check if highway, if not already
        elif not is_highway and '<tag k="highway"' in line:
            is_highway = True

    sys.stdout.write("]}")

Sample Input

My input OSM looked like…

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.56.9 76e5016d">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2021-04-05T15:06:01Z"/>

<bounds minlat="33.4393033" minlon="-86.9466304" maxlat="33.6223902" maxlon="-86.6768410"/>

<way id="926253889">
  <bounds minlat="33.4951768" minlon="-86.8805626" maxlat="33.4956986" maxlon="-86.8801360"/>
  <nd ref="8595476500" lat="33.4956986" lon="-86.8805626"/>
  <nd ref="8595476501" lat="33.4951768" lon="-86.8801360"/>
  <tag k="highway" v="service"/>
</way>

...

</osm>

And the output is…

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": null,
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [
            -86.8805626,
            33.4956986
          ],
          [
            -86.880136,
            33.4951768
          ]
        ]
      }
    },

    ...

  ]
}