thousands of large files (from 1mb to 5gb) containing the coordinates of the points (x, y, z) in a specific coordinate system. Points can intersects
It is required to make an online service where on top of the map there will be a colored surface (like a heatmap depending on the Z coordinate) with some interactive functions. More precisely 3 surfaces with different color maps and you can switch between them
Now GeoServer (with GeoWebCache), GDAL utilities, and some web-stack with leaflet+OSM at the front are used.
How it works now:
Preparing files using the GDAL utilities.
'og2ogr' to Shapefile from CSV (with srs changing)
'gdal_grid -zfield Z' from Shapefile to Tiff
- 3 times 'gdaldem color-relief ... map.txt ... -alpha' for coloring Tiff and 'gdal_translate -of PNG -scale -co worldfile = yes ...' for creating PNG Worldfile. Output 3 PNG heatmaps in different colors.
Upload PNG to GeoServer. We have 3 groups of layers by color. Adding a new layer to a group of layers and recalculating the boundaries of the group.
Problems and question:
The main problem is the performance. For the quickness on the client, I run Seed in the GWC of each group, but it works very slowly. (for example, 1 group, 300 layers in a group, zoom up to 17 can go about a day). But after the cache is ready, everything works quickly on the client. But as boundaries grow, productivity gets worse.
This is my first GIS project and I do not have enough knowledge. It seems to me that my decision is not good and not optimal. What optimal approaches and tools can be used to solve such a problem, and what parameters/options can speed it up?