Strategies for serving large datasets over WFS

Question

We are implementing WFS using geoserver for our customers to consume vector data from a hosted spatial database to their desktop GIS. They want to be able to just connect and browse the data in the same was that they would to for example an SDE connection. Some of the datasets they want to consume are very large, and contain way more features than the feature limits we want to set on the server. They may therefore not get all features downloaded, and if features are missing the user isn’t aware that they have hit the feature limit and don’t have all the features.

If we just increase feature limits, performance degrades, and the client still may not get all features, unless the limit is set larger than feature count in the data.

Basically we are experiencing performance and usability issues around guaranteeing that all features are downloaded for large datasets to desktop clients.

We are testing with Quantum, Arc9 and 10 with Data Interoperability connection.

So far we have been investigating how the caching options on the client can help. We are coming to the conclusion that caching can help, but this still doesn't guarantee all features are always downloaded.

In quantum turning caching off and using scale thresholds helps, as features are requested again when the client zooms/pans if, but only if, the new envelope is outside the initial envelope.

In Arc 9 Data Interopeability, there is no ability to turn caching off, and no ability to restrict to an envelope, so the client gets the features on the first request for the whole dataset, limited by geoserver feature limit and never re-requests during the life of the cache(24hrs), so if features are missing, they will stay missing.

In Arc10 Data Interopeability, it's possible to use a live connection, which re-requests features on every pan/zoom, but as we increase feature limit on server, performance degrades. Also, for certain tasks (view attribute table, select), arc10 requests all features one by one, which is pretty much un-usable.

In all cases, there are scenarios where the feature limit has been hit, features are missing, but the user isn’t aware because the client doesn’t flag this.

I would like some idea how people have approached serving large datasets over WFS to desktops in a way that guarantees the client gets all the features for their view extent. Do we need to improve performance on the server and up the feature limit? What are people doing to tackle this?

I would also like to try and get to a better understanding of what WFS is for. Is WFS even appropriate for looking at large datasets in a desktop session as users are used to doing with shapefiles or a geo-database connections?

I realise this question is wide open, so any views much appreciated.

Answer 1

I think you're nearly there by questioning what the WFS is for. In reality a user browsing the vector data is only interested in examining a few features at a time, right? So the trick is how to identify those feature before WFS request. Clearly this is somewhat dependant on the data we're talking about, but I don't see much value in a desktop client user seeing all the results in one huge feature set.

Obviously one way is to educate the users on how to filter the WFS calls to limit the response size, either by feature attribution or spatially (e.g. Envelope based on their view window^[1]). However, you'll clearly need some way to assist the user in the filtering process, and one of these ways is demonstrated quite nicely in the WFS GetFeature Example (GeoServer) on the openlayers demo site. This demo demonstrates how you can initially represent your features WMS image to the user, allowing them to get an overall "picture" of the features available which can be easily cached using GWC, then requiring user interaction - through clicking, toggling, hovering and dragging in this example - to get the feature required.

In summary, my principle is that you only need set filter restrictions to the point where your results drop below acceptable feature limits, defined by response times from GeoServer and not seeing performance degradation on your desktop software.

I'm afraid this doesn't really tackle the issue of a missing "flag" on the client to notify when you reach the feature limit, but that's fairly client specific. You could utilise the maxFeatures parameter in the WFS request and see if the returned feature count is equal, but that would require a client specific implementation which is probably too in depth for this answer. Worth raising a question specific to the desktop client if you're unable to find a way to do this, I could probably whip up an OpenLayers example.

1. Unfortunately there seem to be issues with restricting by window envelope in some clients, as you point out with Arc 9

Answer 2

All desktop GIS apps have very bad support for large datasets and I haven't found a way to configure them to improve the performance. If you really require access to the vector data in your GIS then I believe the potential solution is to force users to use the WFS 2.0 paging feature. We have some developer tips here: http://www.linz.govt.nz/about-linz/linz-data-service/help/advanced-user-guides/timeouts

In Mapserver and Geoserver this paging feature has been backported to WFS 1.1 and WFS 1.0. In our testing paging is not too taxing on the server or client. The problem here is the WFS paging client support is almost non-existent. GDAL/OGR has support, but FME/ArcGIS/QGIS do not. I've raised a request with the FME guys to support it. In terms of QGIS I believe the issue could be resolved by implementing WFS paging, advanced caching and incremental rendering.

To get around the current issues we are going to publish a python script (uses GDAL/OGR) that will enable users to replicate data into a local database such as MSSQL, postgresql, SpatiaLite or Esri FileGDB. The user will then be able to load this local database into their application. Gets the benefits of speed and greater functionality, with the downside of having to replicate and regularly run the update process. This software should be released in the next few weeks. The script will leverage our change-set API service which is running as an extension over WFS. This service enables the users to keep their dataset up to date with minimal data transfer and processing. See here for more info:

http://www.linz.govt.nz/about-linz/linz-data-service/features/change-set-service

Hope this helps