I'm trying to render tiles for the entire planet using 8 threads. It is taking weeks to render the whole planet is there any way to optimise the tile pre-rendering. Hardware in use is m4.2xlarge (32.0 GB, 8 cores, ssd disk). The command is render_list -a -n 8 -z 2 -Z 15. osm2pgsql cache size is 18G. I'm using modified OSMBright stylesheet.
How can I speedup the pre rendering of the tiles?
Firstly, that is a LOT of data/tiles you are rendering, so you should expect it to take a long time.
You need to reconsider exactly what you are rendering. If you are using OSM_Bright and OSM data for the entire world at all zoom levels then consider more scale dependent rendering so that at the lower zoom levels, you are attempting to render less data. At Zoom Levels 2 to 5 (or so) you could use layers containing data from (say) Natural Earth, especially less complex coastal outlines. Reserve the full OSM data for those scales where you will actually see it. On the subject of large polygons, consider subdividing them.
Also consider subdividing your data by feature-type and, if you have it in a database, create tables of simplified data for use at lower zoom levels (again splitting log line features and large polygons). When using a database, ensure that you have spatial indexes and appropriate indexes on attributes referenced in the style.
All these suggestions will require re-working your style to use the pre-simplified data and alternative (simpler) scale-dependent layers at lower zoom levels. This is how the 'Official' OSM style works (used by OSM for the entire globe). Understanding how this works will give you plenty of clues how to optimise your rendering/styles. I believe OSM Bright was written with smaller areas in mind and at fewer, higher zoom levels. So, I'd recommend only transitioning OSM Bright at levels 7+ (or thereabouts) and create simpler styles that use less data for levels below that.
Your performance bottleneck will not just be the actual rendering but also your database backend. I've already mentioned spatial indexing, which is essential but you must optimise your database. few databases are optimised for GIS out of the box and Postgresql/PostGIS is not exception (other databases are available). There are many thread here with links on that topic.
You have no any performance bottlenecks, so do we.
We are using 4 cores, 32GB and big SSD with 90k IOPS. We are using OSMBrigh too. We've lost about 2 hours of this expensive machine time. The problem is:
FILE *loadavg = fopen("/proc/loadavg", "r");
fscanf(loadavg, "%lf", &avg);
return avg;
while (avg >= maxLoad) {
/* printf("Load average %d, sleeping\n", avg); */
sleep(5);
avg = get_load_avg();
}
Please uncomment this printf and see whether it is a problem for you too. Renderer used a lot of CPU time and than ... he want to sleep because of itself. This is funny.
I have no time to fix this "feature". If you have a free time please do it. I've just added render_list ... --max-load=99999 and resolved it.
PS do not forget to set all ModTileMaxLoadXXX 99999 in apache2/vhosts.d/mod_tile.conf.
PS The issue is here.
