1

I downloaded the 72 GB OpenStreetMap PBF file (planet-230619.osm.pbf) at this link https://planet.osm-hr.org/pbf/ and I am trying to import it into PostgreSQL + PostGIS with the library osm2pgsql version 1.8.1.

The application crashed in this situation:

Processing: Node(4539260k 1044.7k/s) Way(0k 0.00k/s) Relation(0 0.0/s)

ERROR: bad allocation

System info:

  1. OS Name: Microsoft Windows Server 2012 R2 Standard
  2. Processor: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 2401 Mhz, 6 Core(s), 12 Logical Processor(s)
  3. RAM: 16 GB
  4. SSD: 160 GB (free space)

How can I make it work?

1
  • You haven't provided enough information to determine what you're doing. You need to provide your osm2pgsql command line. However, I can tell that no matter what you do, it's not going to work because you don't have enough disk space. Reading the osm2pgsql manual is a good start. Commented Jul 30, 2023 at 23:23

1 Answer 1

1

Your system resources are not sufficient for a full planet import; the recommendation for in-memory caching is to have at least as much RAM available (i.e. excluding OS and the database cluster if running on the same machine) than the size of the PBF file to be imported. Note that the final DB import will be several hundred GB in size!


That being said, you can run the import in slim mode with the --slim flag added to the command: the necessary OSM node caching is moved to temporary in-DB storage; this allows for smaller systems to import large OSM files, however, the required temp storage for a full planet import will be around 300GB in PostgreSQL, and the overall import speed will drop drastically.

You can somewhat improve on both temp storage size and performance by also using the flat-node mode via the --flat-nodes=<FILE_NAME> flag, where you specify a temp file-based DB location to be used instead of PostgreSQL.

3
  • with --slim is extremely (extremely) slow, with --flat-nodes instead it crashes me anyway
    – Koba
    Commented Jul 7, 2023 at 11:22
  • 1
    @Koba it is absolutely expected to be slow with --slim - I would assume a full planet import would take anything up to 10 days! And if there's no syntax error in the way you set the --flat-nodes flag then that is likely a storage space issue. You have to realize that a full planet OSM file resolves into >1TB of typed data! Large parts of that needs to be extracted and cross referenced in order to generate ways and relations. Simply put: your system is undersized by an order of magnitude on all resources!
    – geozelot
    Commented Jul 7, 2023 at 11:39
  • A planet import should not take 10 days on SSDs, even if the machine only has 16GB of RAM. Commented Jul 30, 2023 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.