Clustering 40k+ points from shapefile and populating Google Maps/Webapp?

Question

I just started a new Python/Django webapp job related to GIS (and i'm totally new to GIS). Here is my task and what i have found so far, and i am stuck as to finding out what's the best way or the global standard way of doing things.

• We are building a webapp where a user can upload a shapefile, and take the records and points from the file then populate them on a map.

  1. Upload a shapefile
  2. Parsed the shapefile using pyshp library - this seems OK
  3. I'm able to read the shapes (coordinates) and the records (value for each field on each shape)
  4. Now say for instance i've got about 40,000+ coordinates, each of them with field records in it
  5. I'll need to populate those 40k points onto Google Maps - those points should be represented with a marker, and the field records can be represented with a heatmap (depending on which field the user choose to filter it as)
  6. Certainly loading 40k points onto Google Maps wouldn't make sense as it will slow down the client side of things.

What can i do to make this fast or make this work?

Thinking of clustering nearby coordinates in a grid, and not only that, each coordinate has a field record and we have to somehow cluster them(?) so that the clustered coordinate has an accurate representation of the values in it. A few options that i have looked at are :-

1. Use http://google-maps-utility-library-v3.googlecode.com/svn/trunk/markerclusterer/docs/reference.html, this would speed things up.. but the server will still be sending 40k points, which may not work at a point where we need to load more than 40k points for each different section on the map
2. Was looking at how we can achieve http://www.gdal.org/grid_tutorial.html by using http://www.gdal.org/gdal_grid.html - I've not been able to get the .tiff generation to work yet
3. Also to achieve grid data, i found http://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.griddata.html - though i don't think this actually does it
4. I just happen to learn about GeoDjango and https://github.com/biodiv/anycluster, and am looking into it to see if this should be something i should use
5. and i'm still continuing to research

I don't know if there is any global standard, it looks like a common problem that plenty of people may have come across. I am probably unaware of the lingo terms to search for.

Answer 1

The two GIS web mapping terms you are describing are indeed clustering and then spatial indexing. Clustering is usually done client side while indexing is done server side, usually using a geodatabase, with a server-side map rendering engine in the middle.

In KML, there is a relatively new feature called "Regions" which tries to handle the display side of things. It was originally designed for GoogleEarth but I think it may work with GoogleMaps as well: https://developers.google.com/kml/documentation/regions?csw=1

If you search around for people displaying AIS marine data you'll find examples of client-side clustering in GoogleMaps as well. Here's good one - zoom out and then drill down into different regions. You can look at the page source: http://www.shinemicro.com/marine_traffic.asp

Here's a related topic with a strategy using OpenLayers and GoogleMaps: How to display several points of a KML with the same coordinates in OpenLayers?

All of the above strategies will still force your users to download probably about 2MB of point data depending on the record size.

You mentioned GeoDjango which would drive you to use a real geodatabase and map server. There would be a steep learning curve and far more application building but ultimately it will work better than any other approach.

I would test the 40,000 points in a KML first and just see how bad it is. That way you have a benchmark to compare other solutions. Also try loading it in GoogleEarth just to compare how GE handles it compared to GMaps.

Part of your decision should be based upon how central this function is to your application. If this is some smaller piece of a larger application, then it may not be worth developing the best possible solution for this part. If everything is built around this function then you're probably going to end up using GeoDjango or similar.