3

The problem is I have a lot of data to download from a WFS. I've tried this method with different errors on the way. First I tried PyQGIS with this script:

uri = "http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map"
layer = QgsVectorLayer( uri, "my wfs layer", "WFS" )
crs=QgsCoordinateReferenceSystem(4326)
res = QgsVectorFileWriter.writeAsVectorFormat( layer,
                                                r'G..path...\edifita.shp',
                                               'System', # encoding
                                               crs, #crs
                                              'ESRI Shapefile'
       )

if res != QgsVectorFileWriter.NoError:
    print ('Error number:', res)
else:
    print ("WFS saved!")

I extracted from here: Saving a WFS layer directly to the disk using QGIS or PyQGIS?

This method gives me the following error:

exec(open('C:/Users/JORDI~1.PAG/AppData/Local/Temp/tmpac2ti1a7.py'.encode('utf-8')).read())
Error number: (8, '')

Also, I tried this other method from the same source:

ogr2ogr -f "ESRI Shapefile" "...path..." WFS:"http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map" 

However, it gives me these 2 errors:

ERROR 1: HTTP error code : 504                                                                                                                     ERROR 1: Error returned by server : HTTP error code : 504 (0) 

I am not sure if the errors are caused by the large amount of data or something else.

What do you think?

5
  • 504 is a gateway timeout. Your best approach would be to look at OFFSET and LIMIT to request parts of the data
    – Ian Turton
    Feb 1, 2022 at 15:13
  • I've tried this, but I still get the same error: ogr2ogr -f "ESRI Shapefile" "...path..." WFS:"http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map" If I download 0-100, how do I resume the download in the 101-x if I downloaded 0-100?
    – J.Patordi
    Feb 1, 2022 at 15:23
  • Have you already read about paging gdal.org/drivers/vector/wfs.html?
    – user30184
    Feb 12, 2022 at 12:01
  • I get a timeout requesting just one feature. Points to some misconfiguration on the server; possibly the data source is not available
    – nmtoken
    Feb 14, 2022 at 11:09
  • Requesting one feature is fast for me http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map&service=WFS&version=1.1.0&request=GetFeature&typename=ED.EDIFICATO.CAPOLUOGHI.&maxfeatures=1.
    – user30184
    Feb 15, 2022 at 9:36

4 Answers 4

7

The problem is with the WFS server itself and not with your approach. To explain that lets look at it step by step.

WFS url is : http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map

Before downloading the data lets see the description of this WFS service and see what all layers are part of this source using a read only ogrinfo call.

ogrinfo  WFS:"http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map"

This command returns

INFO: Open of `WFS:http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map'
      using driver `WFS' successful.
Metadata:
  ABSTRACT=Edificato dei capoluoghi di provincia. I capoluoghi di provincia rappresentati sono quelli relativi all'anno 2003. Scala 1:10.000
  PROVIDER_NAME=Geoportale Nazionale - Ministero dell'Ambiente e della Tutela del Territorio e del Mare
  TITLE=Edificato dei capoluoghi di provincia
1: ED.EDIFICATO.CAPOLUOGHI. (title: Edificato) (Multi Surface)

which mentions that there is 1 layer with a name "ED.EDIFICATO.CAPOLUOGHI." and it has a Multi Surface geometry.

Next, we try to load metadata of this layer, to see what are the extents, coordinate system etc, information for the layer "ED.EDIFICATO.CAPOLUOGHI." using the same ogrinfo command and passing the layer name in the end.

ogrinfo  WFS:"http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map" ED.EDIFICATO.CAPOLUOGHI.

This gives the following output

INFO: Open of `WFS:http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map'
      using driver `WFS' successful.
Metadata:
  ABSTRACT=Edificato dei capoluoghi di provincia. I capoluoghi di provincia rappresentati sono quelli relativi all'anno 2003. Scala 1:10.000
  PROVIDER_NAME=Geoportale Nazionale - Ministero dell'Ambiente e della Tutela del Territorio e del Mare
  TITLE=Edificato dei capoluoghi di provincia

Layer name: ED.EDIFICATO.CAPOLUOGHI.
Metadata:
  ABSTRACT=La tabella associata contiene le seguenti informazioni principali: id_edifici, identificativo unico dell'edificio; quota_suolo, quota del suolo espressa in metri sul livello del mare; quota_gronda, quota della gronda dell'edificio espressa in metri sul livello del mare; altezza, altezza dell'edificio espressa in metri sul livello del mare; tipologia, tipologia di edificio espressa da un valore numerico variabile tra 1 e 4; area_, area di base dell'edificio espressa in metri quadri;perimetro, perimetro di base dell'edificio espresso in metri.
  KEYWORD_1=Edifici
  KEYWORD_2=Edilizia
  KEYWORD_3=Edificio
  TITLE=Edificato
Geometry: Multi Surface
ERROR 1: HTTP error code : 504
ERROR 1: Error returned by server : HTTP error code : 504 (0)

You see the WFS server is getting timed out while downloading the Metadata of the layer itself and hence the ogr2ogr is also failing while downloading the actual data irrespective of whatever pagination options you provide.

This is a WFS server issue, I can only make a guess that the data is hosted on a Mapserver but it is proxied with a web server like NGINX or Apache and the connection between the webserver and geoserver is probably timing out. As far as an alternative is concerned, I don't see a solution unless this is fixed by the server itself.

4
  • 1
    With map= it is almost certainly MapServer not GeoServer, possibly QGIS server. but the DescribeFeatureType response for http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map&service=WFS&request=DescribeFeatureType&version=1.1.0 points to a MapServer schema
    – nmtoken
    Feb 14, 2022 at 10:54
  • Yes correct, its a mapserver and not geoserver. Updated that part.
    – Abhilshit
    Feb 14, 2022 at 10:59
  • The problem is not in the metadata from DescribeFeatureType but ogrinfo wants to know the total number of features and therefore is sends this request WFS: http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=ED.EDIFICATO.CAPOLUOGHI.&RESULTTYPE=hits
    – user30184
    Feb 14, 2022 at 18:25
  • Server announces in GetCapabilities that &resulttype=hits is supported but what it hits is timeout. I suggest to inform the service provider.
    – user30184
    Feb 14, 2022 at 19:18
5
+25

I am having trouble loading the Edifici service you have linked, both through QGIS, Python and browser. The GetCapabilities document is loading, but the content is not.

I have tried with another service from the same site that works fine.

Alluvioni_Estensione from http://www.pcn.minambiente.it/mattm/en/wfs-service/

I have had success saving large WFS requests from other services using code similar to this:

from requests import Request
import geopandas as gpd

#url = 'http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map&service=wfs&request=getFeature&version=1.0.0&typename=ED.EDIFICATO.CAPOLUOGHI.'
url = 'http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Alluvioni_Estensione.map&service=wfs&request=getFeature&version=1.0.0&typename=ITH2018_Estensione_HPH'

q = Request('GET', url).prepare().url
df = gpd.read_file(q, format='GML')
df.crs = 'EPSG:4326'

df.to_file('output.shp')
1
  • Note the request parameter values are case sensitive, so it should be WFS not wfs for your service value, and GetFeature not getFeature
    – nmtoken
    Feb 14, 2022 at 11:01
2

That server does not behave quite as GDAL believes it does. If you really need to get the data you can do it with manual paging. Even if the server supports only WFS 1.1.0 and paging was introduced in WFS 2.0.0 that server (it is MapServer) supports &StartIndex as a vendor option. That is documented in GDAL documentation https://gdal.org/drivers/vector/wfs.html

Some servers (such as MapServer >= 6.0) support the use of STARTINDEX that allows doing the requests per “page”, and thus to avoid downloading the whole content of the layer in a single request. Paging was introduced in WFS 2.0.0 but servers may support it as an vendor specific option also with WFS 1.0.0 and 1.1.0.

  1. First request that starts from the beginning

http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map&service=WFS&version=1.1.0&request=GetFeature&typename=ED.EDIFICATO.CAPOLUOGHI.&maxfeatures=1000

Save response into file

  1. Second request

http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map&service=WFS&version=1.1.0&request=GetFeature&typename=ED.EDIFICATO.CAPOLUOGHI.&MaxFeatures=1000&StartIndex=1000

Save response into file

  1. Following requests:

Increase the StartIndex value

When you have collected all batches combine the results, for example

ogr2ogr -f gpkg wfs.gpkg batch1.gml -nln data_from_wfs
ogr2ogr -f gpkg -update -append wfs.gpkg batch2.gml -nln data_from_wfs
...
2

It is possible to fetch data from that service by using the alternative way to make a connection to WFS server with a GDAL WFS-XML file as documented in https://gdal.org/drivers/vector/wfs.html

It is also possible to specify the name of an XML file whose content matches the following syntax (the element must be the first bytes of the file): ...

The service description file has the following additional elements as immediate children of the OGRWFSDataSource element that may be optionally set. ...

PagingAllowed: Set to ON if paging must be enabled. See “Request paging” section. PageSize: Page size when paging is enabled. See “Request paging” section.

Copy and paste the following text and save it as "wfs.xml" on your computer.

<OGRWFSDataSource>
  <URL>http://wms.pcn.minambiente.it/ogc?map=/ms_ogc/wfs/Edifici.map</URL>
  <PagingAllowed>ON</PagingAllowed>
  <PageSize>1000</PageSize>
  <WFS_Capabilities xmlns="http://www.opengis.net/wfs" 
</OGRWFSDataSource>

Next run command

ogrinfo wfs.xml

Open the wfs.xml file, it should have now much more data. Then you can convert all the data from the WFS service into GeoPackage with command

ogr2ogr -f gpkg wfs.gpkg wfs.xml

If you want to follow what GDAL is doing use command

ogr2ogr -f gpkg wfs.gpkg wfs.xml --debug on

By the documentation it should also be possible to give the PagingAllowed and PageSize options from the command line as config options. I tried that without success but maybe I used somehow wrong syntax.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.