0

I am trying to install Nominatim based off of a dockerized version of the tool found here: https://github.com/mediagis/nominatim-docker. I was successful with installing dockerised Nominatim with a smaller country locally. And the process ran pretty fast.

Then, I got an EC2 instance with, 64GB memory, 1TB SSD, and 6 core processors to do an install with the entire planet (48GB). The most of the planet file got processed fast, in about 6hours and when processing the relations gave an error and failed. When I ran the planet file based on the same docker image, the object parsing (specially ways) were extremely slow. About 100 times slower than the initial run. This is the error I got when the planet ran faster:

Processing: Node(5640116k 412.0k/s) Way(625947k 118.89k/s) Relation(628790 712.11/s)2019-12-18 22:48:23.270 UTC [210] LOG:  incomplete message from client
2019-12-18 22:48:23.270 UTC [210] CONTEXT:  COPY place, line 233940695
2019-12-18 22:48:23.270 UTC [210] STATEMENT:  COPY place FROM STDIN
2019-12-18 22:48:23.270 UTC [207] LOG:  incomplete message from client
2019-12-18 22:48:23.270 UTC [207] CONTEXT:  COPY planet_osm_rels, line 598784
2019-12-18 22:48:23.270 UTC [207] STATEMENT:  COPY planet_osm_rels FROM STDIN
Killed
2019-12-18 22:48:23.275 UTC [210] ERROR:  unexpected EOF on client connection with an open transaction
2019-12-18 22:48:23.275 UTC [210] CONTEXT:  COPY place, line 233940695
2019-12-18 22:48:23.275 UTC [210] STATEMENT:  COPY place FROM STDIN
2019-12-18 22:48:23.275 UTC [207] ERROR:  unexpected EOF on client connection with an open transaction
2019-12-18 22:48:23.275 UTC [207] CONTEXT:  COPY planet_osm_rels, line 598784
2019-12-18 22:48:23.275 UTC [207] STATEMENT:  COPY planet_osm_rels FROM STDIN
2019-12-18 22:48:23.275 UTC [207] LOG:  could not send data to client: Broken pipe
2019-12-18 22:48:23.275 UTC [207] STATEMENT:  COPY planet_osm_rels FROM STDIN
2019-12-18 22:48:23.275 UTC [210] LOG:  could not send data to client: Broken pipe
2019-12-18 22:48:23.275 UTC [210] STATEMENT:  COPY place FROM STDIN
2019-12-18 22:48:23.275 UTC [207] FATAL:  terminating connection because protocol synchronization was lost
2019-12-18 22:48:23.325 UTC [210] FATAL:  terminating connection because protocol synchronization was lost
ERROR: No Data
string(7) "No Data"

Finally, I just took a pbf of North America (8GB) and tried the install. Object parsing related to North America went relatively fast, but it is still working on rankings and indexes. The install is going on for over 16 hours.

I didn't do any PostgreSQL optimizations before running the install. When the object parsing is slow, I see the container is only using about 1% of the available memory.

In Nominatim install instructions (here: https://nominatim.org/release-docs/latest/admin/Installation/) I see that with less config (32 GB RAM) a planet install is taking only 2 days. What am I doing wrong and how can I optimize the process?

  • Have you maxed out the IOPS for your EC2? At some point the lower-cost EC2 instances are throttled back on I/O. – Vince Dec 20 '19 at 18:08
  • It has not maxed out the IO – c00der Dec 20 '19 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.