3

We are writing a application in which a part of the job is to fetch a relatively large (>10000 features) and often updated dataset from a WFS server, store a local copy, and do geospatial analyses on it. We are stuck on the downloading part, as we hit the default 1000 features limit on the server side, which we do not control.

We see that QGIS for example can access this dataset completely, from which we can in principle manually save it in a suitable format for our application, but this is not the desirable way to go, as we want a completely automatic procedure.

We are using OGR (ogr2ogr) and can get local copies of up to 1000 features. We wonder if there is an easy way to get the entire dataset using this approach. Hints to other approaches are also welcome.

  • 2
    Does the WFS server support paging with "count" and "startindex"? – user30184 May 17 '15 at 12:18
  • 1
    More a clue than an answer: do network sniffing when you use QGIS case (Wireshark is my choice because cross platform but see Fiddler QGIS related answer gis.stackexchange.com/questions/25907/…). You will be able to understand web services requests and then automate them outside QGIS. – ThomasG77 May 25 '15 at 16:27
  • 4
    WFS is a published standard and all you need to do is to sniff the documentation about how it works. If server does not support paging the client must make BBOX queries in a row or use some attribute for limiting request size under 1000 by putting a reasonable value range in WFS filter. Duplicate features can be deleted on client side by keeping only distinct FIDs. – user30184 May 25 '15 at 16:35
  • 1
    You are right but as he told us that QGIS is working the expected way, he could directly inspect the real request and not only read a spec. However, as you stated, he should also read the specs ;) – ThomasG77 May 25 '15 at 16:59
  • 1
    I gave up on getting this working. I suggest you talk to the data provider and ask how they provide bulk data extracts. WFS is not great for downloading large datasets, it's nasty to the server-side hardware. – Alex Leith May 26 '15 at 1:13
4

If you are using Linux, you should be able to connect to a WFS server, and fetch a layer without any problem with wget. I was able to fetch a dataset consisting of 10209 features, which was 33.3 MB in size. My test command was the following:

wget -O output.gml "http://demo.opengeo.org/geoserver/wfs?request=GetFeature&version=1.1.0&typeName=osm:admin_5678&srsname=EPSG:3857"

Be sure to endorse the URL in quotes, as they contain more than one parameters, and otherwise it would be interpreted by the shell incorrectly.

The -O flag writes the received data into a user specified file, but it has to be included before the URL.

output

  • 3
    The server in question has "maxfeatures" limit set to 1000 features on the server side. It can't be overridden from the client side and wget would receive only 1000 features. Some sort of paging is needed for collecting the whole dataset. Another thing is that if the limit is so low the service provider may not really want people to gather all the data. – user30184 May 25 '15 at 19:23
  • 1
    I see, then paging is indeed the only reasonable workaround. – Gabor Farkas May 25 '15 at 20:09
3

The easiest way to do this, is to use GDAL's ogr2ogr utility.

Firstly, you need to create an XML file with the URL to the WFS endpoint.

<OGRWFSDataSource>
  <URL>http://demo.opengeo.org/geoserver/ows?service=wfs&amp;typeName=osm:roads</URL>
  <PagingAllowed>ON</PagingAllowed>
</OGRWFSDataSource>  

If you want to add any other parameter, like a filter, you can add it to the url as well. Just remember to make sure that the URL is XML escaped

Then you can just run ogr2ogr, with a command like this:

ogr2ogr -f "SQLite" roads.db road.xml

If you want to write to a shapefile, you can use a command like:

ogr2ogr roads road.xml

You can get more details about the optional parameters here: WFS driver

  • I get the error: Cannot do natural order without a primary key, please add it or specify a manual sort over existing attributes. How to fix this? – Revo Aug 17 '17 at 10:26
2
+25

You could try the OGR WFS option "-dsco OGR_WFS_PAGING_ALLOWED=ON" as described in the docs http://www.gdal.org/drv_wfs.html

  • Thanks! Tried this one solution, but to no avail. – user21313 May 25 '15 at 11:32
  • Can you describe the command and eventually disclose the WFS weblink? – Stefan May 25 '15 at 16:05
  • Do you mean that WFS server does not support paging? Like in this demo.opengeo.org/geoserver/… – user30184 May 25 '15 at 16:27
  • Interesting enough SecludedPath says "We see that QGIS for example can access this dataset completely". – Stefan May 25 '15 at 16:57
  • 1
    QGIS can be set to no-cache mode when it makes a new request every time when user is panning or zooming. If the area is small enough requests stay under the 1000 feature limit and all the data of the WFS layer can be accessed. There are troubles with this reasonable strategy sometimes, though. See hub.qgis.org/issues/9444 and hub.qgis.org/issues/11968. – user30184 May 25 '15 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy