thousands of large files (from 1mb to 5gb) containing the coordinates of the points (x, y, z) in a specific coordinate system. Points can intersects

It is required to make an online service where on top of the map there will be a colored surface (like a heatmap depending on the Z coordinate) with some interactive functions. More precisely 3 surfaces with different color maps and you can switch between them

Now GeoServer (with GeoWebCache), GDAL utilities, and some web-stack with leaflet+OSM at the front are used.

How it works now:

  1. Preparing files using the GDAL utilities.

    • 'og2ogr' to Shapefile from CSV (with srs changing)

    • 'gdal_grid -zfield Z' from Shapefile to Tiff

    • 3 times 'gdaldem color-relief ... map.txt ... -alpha' for coloring Tiff and 'gdal_translate -of PNG -scale -co worldfile = yes ...' for creating PNG Worldfile. Output 3 PNG heatmaps in different colors.
  2. Upload PNG to GeoServer. We have 3 groups of layers by color. Adding a new layer to a group of layers and recalculating the boundaries of the group.

  3. On the front by 'L.tileLayer.wms' we show the desired group of layers from GeoServer. how it looks now

Problems and question:

The main problem is the performance. For the quickness on the client, I run Seed in the GWC of each group, but it works very slowly. (for example, 1 group, 300 layers in a group, zoom up to 17 can go about a day). But after the cache is ready, everything works quickly on the client. But as boundaries grow, productivity gets worse.

This is my first GIS project and I do not have enough knowledge. It seems to me that my decision is not good and not optimal. What optimal approaches and tools can be used to solve such a problem, and what parameters/options can speed it up?

  • why not one tiff and three slds to do the mapping - make sure tiff is tiled and compressed with overviews – Ian Turton Apr 9 at 9:31
  • How to merge files into one tiff? gdal_merge freezes on so many files – Vladimir Sukhov Apr 9 at 10:40
  • you have a single tif as the output to step 2 - also mosaic the files rather than use a layergroup – Ian Turton Apr 9 at 10:52
  • probable duplicate of gis.stackexchange.com/questions/349668/… – Ian Turton Apr 9 at 10:53
  • My question is more about the approach, not only about performance. Before creating a single tiff, I have other problems. For example, what is the best way to create a tiff with a surface, instead of a set of points using gdal from XYZ csv file. – Vladimir Sukhov Apr 12 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.