30,000 data points... can this be reasonably served using openlayers?

Question

I'm trying to tackle a web GIS problem and need to figure out the best strategy. I have a map with 30,000 cells on it forming a grid for the province of Alberta. I have a list of data points for each cell - about 50 indicator fields and values for every cell. For example, one value might be human population, so each cell would have a value for human population for that cell. I want to make an interactive map that can show those values visually either with squares for the cells, or a blended heat-map type of look. Beyond that, I'd like to be able to make compound calculations on all of the data points and use those calculations to make additional maps. Taking into consideration the calculations, there are roughly 300,000 permutations to the map. To top it off, it is also temporal; there are 16 different time slices. That means 16 time slices x 300,000 permutations.

My strategy so far:

1) On the server side, calculate the cell values dynamically at run time and send the values to the client (all 30,000 of them). This translates to roughly a value for every 3x3 pixel square in a 525px x 525px image on the screen.

2) On the client side with open layers, use a 1px png that can be resized and have the alpha value adjusted as needed to show the different color gradient corresponding to the data value for each cell. The png would be resized as the map zooms in or out. Theoretically they would be positioned precisely to form a blanket coverage of the map as squares or else using the heat map approach they would overlap each other enough to form a blanket coverage.

The Question:

Is it reasonable to calculate those cell values dynamically on the server and then send to an OpenLayers interface in real time? I'm confident the server can perform the calculations, but can OpenLayers reasonably display 30,000 data points at a time on a map? Is it reasonable to expect to be able to create a blanket coverage using point data? I guess I'm trying to create the equivalent of a vector map that has 30,000 shapes giving 100% blanket coverage of the map, but using point data to simplify it.

The alternative strategies in my mind are:

a) to generate the maps in real time on the server and send as rasterized layers to the client

b) to generate all of the permutations ahead of time and store them on the server as rasterized maps

Any thoughts? Is my approach way off and barking up the wrong tree? Any advice on a better method?

I'd sure appreciate any thoughts on the issue! If you're interested, I might even be looking to contract a developer to help with it.

Thanks so much!

Noah

Answer 1

This questions has been asked a number of times. 30k points, straight up, will not work on an OL map. Or even on a Flash/Silverlight map.

Rough (rough!) order of magnitude numbers to remember - 100 points on a JS map (openlayers), 1,000 points in a Flash Map (e.g ArcGIS Flash or Silverlight), 10,000 points on a desktop app (ArcGIS Desktop) are your fine performance levels. These are "actual" points, not hidden or faked as outlined below.

Not only that, it's not very nice to interact with 30k points. How do I access "that one" marker behind another marker?

You have 2 options:

1) Create raster, like you mentioned, on server. Serve up a WMS service and when your user clicks/hovers over a point, go back to server for the data

2) Reduce # of points you display. Primarily via SERVER-SIDE clustering and also via bounding box trimming. Client-side clustering is "ok" and "cute" but won't help you at 30k points. It'll be slow. So you setup your layer with the bbox strategy and OL will make calls to server on pan/zoom for you, and you return back a new set of data that's trimmed to the point where you return less than 100-300 points.

Answer 2

From a UI and UX point of view, 30,000 individual points on a map is not exactly the best representation of your data.

You may also choose to use UTF8 grids on top of your rasterized point data. However, the resolutions of your grid and the amount of points will definetely make selecting data very unpredictable and not the best UX. http://mapbox.com/developers/utfgrid/

A combination of OL Clusters with UTF8 Grids would be interesting. This can be done by some server size clustering of data which returns raster clusters than can then be selected through the use of UTF8 Grids. Your choice of manipulation past this point might be to zoom in further (as in the classical client side vector clustering methods such as in OL). You may also want to process some data such as calculate the average value inside that cluster and present that to the user.

Answer 3

As already mentioned 30,000 features through OL is a really bad idea. I've managed 10,000 points using canvas rendering in Chrome and that performed OK, but switching to a graphic (.png) marker caused it to crawl.

I just wanted to provide a suggestion for your server-side approach. Obviously you can't pre-generate all 300,000 permutations in advance, and your scope is bound to grow eventually. If you want to use a WMS with SLD you can use the following approach to generate any map.

Client (e.g. OpenLayers) makes WMS image request to server. URL for request includes a parameter ...&SLD=http://app-server/sld?time=2010&measure=population... - here the SLD parameter is a URL (which would have to be URL-encoded) for generating an SLD on the fly. When the WMS server gets your request it queries your application server for a unique SLD and gets an XML response. It then generates whichever permutation your user requested.

I've done this with GeoServer for WMS and PHP generating SLDs and performance was fine (but then I had no reason it shouldn't be, your situation might be different).