2

I've got a PostGIS database that's been loaded with a couple of the continent downloads from geofabrik. One of these continents is Antartica - the pbf download is 29 MB but the size in the database is around 1.5 GB (50 times larger) - at this rate the whole world would be over 2 TB. Is this normal? Is there any way to reduce the size?

One thing got me wondering - when I look at the nodes table, there are over 4.5 million nodes, which includes not just the, approx 1600, node features but also every vertex of every way in the continent too. Is this normal?

All the tags are being extracted into an hstore.

The process was done using osmosis.

2 Answers 2

4

Well, depending on your needs and the schema you use, you could indeed face 2TB storage requirements for the whole planet (but not necessarily...). And OSM data is growing steadily each year, so factor that in as well.

I currently run a complex multi-scale topographic map style database for the whole of Europe (about 1/3 of planet) based on OSM data using a custom schema including materialized views and dedicated indexing, and that database is surely over 500 GB for Europe alone (I run it of a 2TB Samsung EVO SSD, for planet, I would need a 4TB one with this custom schema).

However, as to the "nodes" table: this is only needed if you intend to run (minutely) updates, as the nodes table is needed as the source for creating the new Point, Line and Polygon features contained in updates. And at least for osm2pgsql, you also have the option to store the nodes table outside your database (--flat-nodes option), but I don't know about osmosis, I have never used it.

If you only intend to import once, or replace the whole database by a fresh import now and then, then you can drop the nodes (and possibly ways and relations if present) table.

For your reference, I have found these links the most insightful about OSM in general:

https://learnosm.org/en/osm-data/

https://osm2pgsql.org/doc/manual.html

https://wiki.openstreetmap.org/wiki/Component_overview

2

I imported planet-220103.osm.pbf, using --flat-nodes option. Size:

  • 70GB for flat_nodes.bin
  • 1.1TB for db (on my BRTFS it actually compressed to about half that size)

The import took little over 7days on my Xeon® D-1527 4-core 64GB system, 2TB SSD, running in Docker

1
  • Thanks for this. I have a similar laptop specification and after seeing this decided to do it all in the cloud. Commented Apr 21, 2023 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.