5

I have been trying to convert a large OSM file (5G) to the GeoJSON format. All the tools I have been using, use extensive memory and throw an out of memory exception.

What is the best way to convert a large OSM file to the GeoJSON format? Do I have to split the OSM file?

5
  • 2
    JSON increases data size significantly, GeoJSON particularly so. It's possible that your output could exceed 500GB.
    – Vince
    Oct 17, 2019 at 21:15
  • I got a subset of my dataset (just San Fransisco) and the size remained almost the same after the conversion.
    – ANN
    Oct 17, 2019 at 21:51
  • How big was that?
    – Hornbydd
    Oct 17, 2019 at 22:41
  • it's just a 6mb osm file
    – ANN
    Oct 18, 2019 at 3:42
  • Because the file size was not increased I suppose your OSM data are in OSM XML format, not as .pbf but that should not really matter. Mention some of the "all tools I have been using".
    – user30184
    Oct 18, 2019 at 5:46

4 Answers 4

7

Standalone tools

You can try it with ogr2ogr using the OpenStreetMap driver.

 ogr2ogr -f GeoJSON output_points.json input.osm.pbf points
 ogr2ogr -f GeoJSON output_lines.json input.osm.pbf lines
 ...

Note ogr2ogr reads OSM-files with 5 layers: points, lines, multilinestrings, multipolygons and other_relations. So you might need to repeat the upper command with each layer. You will obtain 5 JSON layers finally.

An alternative can be osmtogeojson. It is written in JavaScript and has a commandline interface.

However, not sure if it works with large OSM files like yours.

PostgreSQL

Using PostgreSQL works for sure (like mentioned below).

You can import your OSM file using ogr2ogr, osm2pgsql or imposm. For the latter both you need to define which tags of OSM should actually be imported.

Once you have all your OSM data in the PostgreSQL database (probably in many tables) you can export them using ogr2ogr (like in this post):

ogr2ogr -f GeoJSON output.json PG:dbname="my_database" "my_table"
1

If you know how to use PostGIS you can use the following query to construct a valid geojson:

COPY (
SELECT jsonb_build_object(
  'type',     'FeatureCollection',
  'features', jsonb_agg(feature)
)
FROM (
  SELECT jsonb_build_object(
    'type',       'Feature',
    'id',         id,
    'geometry',   ST_AsGeoJSON(geom)::jsonb,
    'properties', to_jsonb(inputs) - 'id' - 'geom'
  ) AS feature
  FROM (
    SELECT * FROM input_table
  ) inputs
) features
) to 'C:\temp\output_geojson.json';
2
  • What are the memory requirements of PostGIS?
    – ANN
    Oct 18, 2019 at 3:43
  • Google it
    – jbalk
    Oct 28, 2019 at 18:19
1

You might want to try out osmium export, which is based on the libosmium library, a proven and widely used lib for high performance OSM processing:

https://docs.osmcode.org/osmium/latest/osmium-export.html

Installation instructions are available here: https://osmcode.org/osmium-tool/

0

The following Python script can stream through the data, requiring no memory. It converted my 3.1G OSM file from XML into GeoJSON in 81 seconds.

In my case, I had to convert a bunch of ways into either LineString's or Polygons, so I used the existence of highway=* to differentiate between the two. However, you may need to tweak the logic to fit your use case.

import json
import sys

sys.stdout.write('{"type":"FeatureCollection","features":[')

with open(sys.argv[1]) as f:
    # Skip irrelevant metadata lines
    while "<way" not in next(f):
        pass

    coords = ""  # Coordinates of each geometry
    is_highway = False  # Tracked for each way
    is_first_way = True  # Tracked for printing commas

    for line in f:
        # If we close a way, flush it and reset
        if "</way" in line:
            coords = "[" + coords[:-1] + "]"
            if not is_highway:
                coords = "[" + coords + "]"
            sys.stdout.write(
                [",", ""][is_first_way]
                + '{"type":"Feature","properties":null,"geometry":{"type":'
                + ['"Polygon"', '"LineString"'][is_highway]
                + ',"coordinates":'
                + coords
                + "}}"
            )
            is_first_way = False
            coords = ""
            is_highway = False

        # Add coordinates from node
        elif "<nd" in line:
            split = line.split('"')
            coords += "[" + split[-2] + "," + split[-4] + "],"

        # Check if highway, if not already
        elif not is_highway and '<tag k="highway"' in line:
            is_highway = True

    sys.stdout.write("]}")

Sample Input

My input OSM looked like…

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.56.9 76e5016d">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2021-04-05T15:06:01Z"/>

<bounds minlat="33.4393033" minlon="-86.9466304" maxlat="33.6223902" maxlon="-86.6768410"/>

<way id="926253889">
  <bounds minlat="33.4951768" minlon="-86.8805626" maxlat="33.4956986" maxlon="-86.8801360"/>
  <nd ref="8595476500" lat="33.4956986" lon="-86.8805626"/>
  <nd ref="8595476501" lat="33.4951768" lon="-86.8801360"/>
  <tag k="highway" v="service"/>
</way>

...

</osm>

And the output is…

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": null,
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [
            -86.8805626,
            33.4956986
          ],
          [
            -86.880136,
            33.4951768
          ]
        ]
      }
    },

    ...

  ]
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.