2

I'm currently importing a subsection of North America into PostGIS to render my own basemap tiles using osm2pgsql. Is there any way to get an accurate count of the number of nodes, ways, and relations contained within my PBF file that I am generating to determine how long this will take to process on my computer?

  • osm2pgsql generates the nodes and ways, so you can definitely count those... I can't remember how to count the relations, though, so will add the pgRouting tag to this question just in case someone can answer... – DPSSpatial May 3 '18 at 19:58
1

I'm not sure if it solves your problem but you can use PyOsmium Python package. -Documentation-

It can be time consuming, because to count all elements it scans all the data.

import osmium

class Counter(osmium.SimpleHandler):
    def __init__(self):
        osmium.SimpleHandler.__init__(self)
        self.num_nodes = 0
        self.num_ways = 0
        self.num_relations = 0

    def node(self, n):
        self.num_nodes += 1

    def way(self, w):
        self.num_ways += 1

    def relation(self, r):
        self.num_relations += 1    

counter = Counter()
counter.apply_file("path_to_pbf_file") 

print("Number of nodes : %d"    % counter.num_nodes)
print("Number of ways : %d"     % counter.num_ways)
print("Number of relations: %d" % counter.num_relations)
0

You can run osm2pgsql with null output, no RAM cache, and no slim cache. For a 2.5GB PBF extract of Canada, it took me about 37 seconds to count the nodes, on an NVMe SSD.

$ osm2pgsql --output null --cache 0 canada-latest.osm.pbf
osm2pgsql version 0.96.0 (64 bit id space)

WARNING: ram cache is disabled. This will likely slow down processing a lot.

Using projection SRS 3857 (Spherical Mercator)
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=0MB, maxblocks=0*65536, allocation method=3

Reading in file: canada-latest.osm.pbf
Using PBF parser.
Processing: Node(327502k 32750.3k/s) Way(18801k 895.29k/s) Relation(0 0.00/s)  parse time: 31s
Node stats: total(327502800), max(6338602157) in 10s
Way stats: total(19016827), max(676910855) in 21s
Relation stats: total(321360), max(9397389) in 0s

Going over pending ways...
        0 ways are pending

Using 12 helper-processes
Finished processing 0 ways in 0 s


Going over pending relations...
        0 relations are pending

Using 12 helper-processes
Finished processing 0 relations in 0 s

node cache: stored: 0(0.00%), storage efficiency: nan% (dense blocks: 0, sparse nodes: 0), hit rate: nan%

Osm2pgsql took 37s overall

Note that there are two numbers, the total for this file, and the max for the OSM database.

The node count can then be multiplied by 8 bytes to get a good estimate for the cache size when you import into PostgreSQL. In this case, 327502800 * 8 bytes = 2.44 GB. The osm2pgsql manual recommends adding 10 to 30% overhead on top of that cache value.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.