12

I know that question is rather vague but please bear w/me. I'm trying to get an idea of what sort of product performance - specifically timing - people have seen for various methodologies they have used to create google/bing map tiles. There are a slew of methods for how to do this (e.g. gdal2tiles, FME, maptiler, etc). An initial attempt at simply taking a large PNG and creating tiles using imagemagick, on a pretty decent linux server, yielded some pretty long processing times and so I wanted to see what other people are using in production. New tiles would need to be generated at least daily and so turnaround time on this is pretty critical.

The only real requirement is that it can run on a linux server. Obviously, free is better but I don't want to restrict myself to that. The input can be raw gridded/raster data or a large image. The output needs to be image tiles capable of being used as-is in google or bing maps.

Just for the sake of comparison, I'll say that the timings should be for google map's zoom level 7.

I appreciate everyone's help and again I want to apologize for how vague this question probably seems.

UPDATE: As far as the inputs, I currently have multiple (raw) data sources in various formats: netCDF, GRIB, GRIB2. In addition to the raw data itself, I also have the ability to generate really large images of that data which could then be sliced/tiled.

Ideally, I would just be chopping the image up but I am willing to try whatever will get me the fastest results.

Improve this question
8
  • Recommend you use Adobe fireworks for highly optimizing the final images you are using - adobe.com/products/fireworks - even exported from Photoshop and then optimized in Fireworks reduced file sizes up to 75% (png)
    – Mapperz ♦
    Commented Mar 25, 2011 at 18:27
  • @Mapperz- elaborate on "optimized in Fireworks"?
    – Derek Swingley
    Commented Mar 25, 2011 at 18:48
  • I think you need to expand on your input(s) and if more processing is needed or if you are just chopping them up.
    – Ian Turton
    Commented Mar 25, 2011 at 20:16
  • 4
    @Mapperz: The free equivalent is pngcrush and pngnq for quantization. - I currently work on a similar task and have an automatic chain gdal2tiles > pngnq > pngcrush > pregenerating thumbnails using imagemagick for every file that is fed into the system - I cannot claim it to be fast, but the automation takes a lot of the burden. And in my case there are no updates, it's fire and forget.
    – relet
    Commented Mar 28, 2011 at 14:46
  • 1
    @relet - Any timings you can pass along? What is your hardware setup for this? Thanks
    – malonso
    Commented Mar 29, 2011 at 11:37

4 Answers 4

Reset to default
6

I was having issues with gdal2tiles taking quite a while to process a fairly large (380MB, 39K x 10K pixels) tiff into Google tiles for zoom ranges 0-12. On Ubuntu 12.04 64bit without multiprocessing it took just about all day (8 hours) to process the tiff into 1.99 million tiles @ 3.3GB. Like @Stephan Talpalaru mentions above, making gdal2tiles run in parallel is the key. Make a backup of your original gdal2tiles.py, then install the patch from within the directory that houses gdal2tiles.py (mine was /usr/local/bin):

$ sudo patch -p0 -i gdal2tiles_parallelize_base_and_overview_tiles.patch

Now run gdal2tiles like you normally do. I got an incredible increase in performance, with all 4 of my cores (Intel Core i7 3.4GHz) pegged:

$ time gdal2tiles.py -p raster -z 0-12 -w none ds1105-2235df023_23_b.tif gdal-tiles12
Generating Base Tiles:
0...10...20...30...40...50...60...70...80...90...100 - done.
Generating Overview Tiles:
0...10...20...30...40...50...60...70...80...90...100 - done.

real    39m8.242s
user    104m6.808s
sys 9m16.036s

So from ~8 hours to 39 MINUTES. Game changer.

Improve this answer
5

Try the parallel version of gdal2tiles.py: http://trac.osgeo.org/gdal/ticket/4379

Improve this answer
1
  • 1
    Did you have any success running the parallel version on Windows? I get file access errors on the temp files generated.
    – Michalis Avraam
    Commented Jun 13, 2013 at 1:01
3

Here are some of my results for the following raster file:

JPEG 14456x14490 14456x14490+0+0 DirectClass 62mb

$ time gdal2tiles [...]

Generating Base Tiles:
0...10...20...30...40...50...60...70...80...90...100 - done.
Generating Overview Tiles:
0...10...20...30...40...50...60...70...80...90...100 - done.

real    5m7.675s
user    5m5.070s
sys  0m2.060s

$ time [pngnq && pngcrush for every tile, 4500 in total]

real    9m32.827s
user    18m34.190s
sys  0m27.230s

Yup, that's in minutes - I optimized for output size, not speed. The machine is a virtual Intel Xeon 2x3GHz, 4G of memory. (And clearly, gdal2tiles could make use of some parallelization.)

Improve this answer
2
  • Is the sample file available for download. I would be keen to compare the performance with maptiler.com
    – MapTiler
    Commented Apr 19, 2016 at 14:01
  • Sorry, I changed jobs in the meantime. I could probably find out where the tiles are published, but not the original file.
    – relet
    Commented Apr 20, 2016 at 6:02
2

You mentioned FME and there are some numbers on creating map tiles on FMEpedia

It's a long article, so I pulled out the relevant parts:

Level             Tiles           Minutes (hours)
    8            24,500           18 (0.3)
   10           245,000          105 (1.75)
   11         1,000,000          384 (6.4)

This is using a multi-machine process with FME Server. You could also check out this post by Paul Bissett on the WeoGeo blog: http://www.weogeo.com/blog/Scaling_FME_Engines_on_WeoGeo.html

It has a great movie showing how to process data like this in the cloud - basically firing up a bunch of Amazon virtual machines to spread the processing load and get it done very quickly.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.