0

I am extracting data from an ArcGIS Server into QGIS, using the Extract/Clip by Extent processing tool in the Processing Toolbox. The ArcGIS data has several fields that are Coded Value domains (code/name pairs).

An example layer is below, though the question is generic.

https://maps.six.nsw.gov.au/arcgis/rest/services/sixmaps/LPIMap/MapServer/38/queryDomains

When viewing a preview of the data via Layer Properties, the names are shown eg:

enter image description here

However, when the data is extracted into QGIS, it is imported with the codes.

enter image description here

While it is possible to extract both the names and codes from the JSON properties of the layer:

   "name": "fuzzylinefeaturetype",
   "type": "esriFieldTypeSmallInteger",
   "alias": "fuzzylinefeaturetype",
   "domain": {
    "type": "codedValue",
    "name": "WS4_DmT_FuzzyFeatureTypeLine",
    "codedValues": [
     {
      "name": "Beach",
      "code": 2
     },
     {
      "name": "DuneLike",
      "code": 3
     },
     {
      "name": "GeneralLocality",
      "code": 5
     },
     {
      "name": "RangeLike",
      "code": 13
     },
     {
      "name": "ValleyLike",
      "code": 18
     }

it seems like there's going to be a fair bit of overhead to then process this and map the extracted data from the codes to the names: https://maps.six.nsw.gov.au/arcgis/rest/services/sixmaps/LPIMap/MapServer/38/queryDomains?f=pjson

Is there an easy way of extracting the data into QGIS with the names rather than the codes?

Solutions involving PyQGIS as well as straight QGIS are welcome.

1 Answer 1

0

For anyone interested, I ended up writing a fairly involved function to do the job, so not exactly an "easy way". Broadly speaking, the function:

  1. Creates the ArcGIS query url
  2. Retrieves the data and stores it in a GeoPackage (GPKG)
  3. Edits the GPKG fields to replace codes with the name, for specified fields, by
    • creating a new field
    • doing the mapping
    • deleting the old field
    • renaming the new field to the old field

It hasn't been extensively tested so probably has some edge cases that haven't been teased out. There's also little to no error handling. However, it should be possible to run the two code sections below via the Python Console and have QGIS generate a layer.

Function code:

def addArcGisLayer (url, layer, crs, params):
# PARAMETERS
# url (str): URL of the FeatureServer or MapServer
# layer (str): ArcGIS name of the layer
# outfile (str): full filepath of the output layer (GPKG only)
# crs (QgsCoordinateReferenceSystem): spatial reference for the output layer
#
# params - a dict which can contain any of:
# where (str): a WHERE clause for the query filter - default is none
# extent (dict): spatial filter - in {"xmin":<xmin>,"ymin":<ymin>,"xmax":<xmax>,"ymax":<ymax>} format - default is none
# filterCrs (str): spatial reference for the spatial filter (in WKT format) - default is none (if extent is specfied, the extent is assumed to be in the SR of the ArcGIS source layer)
# outputFields (str) - comma-separated string of output fields - default is "*" (ALL)
# outputCodeValueFields (str) - comma-separated string of output fields for which code/value mapping translation will be applied - default is none
# outputFormat (str) - output format - defaults to "json" (not sure if anything else will actually work for PyQGIS)
# NOTE: no checking is carried out on the parameters passed

# RETURNS
# outfile - location of the output file (GeoPackage)
    import urllib.request, json, subprocess
    
    arcGisParams = {
        "where": "1=1",
        "outfields":"*",
        "f":"json"
    }
    
    # Set defaults
    if 'where' in params:
        arcGisParams['where'] = params['where']
    
    if 'extent' in params:
        arcGisParams['geometry'] = params['extent']
    
    if 'filterCrs' in params:
        arcGisParams['inSR'] = {"wkt": params['filterCrs'] }
    
    if 'outputFields' in params:
        arcGisParams['outfields'] = params['outputFields']
    
    if 'outputFormat' in params:
        arcGisParams['f'] = params['outputFormat']
    
    outfile = QgsProcessingUtils.generateTempFilename(layer + '.gpkg')
    
    # Get layers from service
    layersUrl = url + "/layers?f=pjson"
    
    # Download the layers json
    with urllib.request.urlopen(layersUrl) as u:
        data = json.loads(u.read().decode())
    
    layersArcGis = data['layers']
    
    availableLayers = dict()
    for l in layersArcGis:
        availableLayers[l['name']] = l['id']
    
    layerUrl = url + '/' + str(availableLayers[layer])
    
    # Build query string component of URL
    queryStr='?'
    for key, value in arcGisParams.items():
        queryStr += key
        queryStr += '='
        if isinstance(value,str):
             queryStr += urllib.parse.quote(value)
        if isinstance(value,list):
             queryStr += urllib.parse.quote(','.join(value))
        if isinstance(value,dict):
             queryStr += urllib.parse.quote(json.dumps(value))
        queryStr += '&'
    
    queryStr = queryStr.rstrip('&')
    
    # Extract data from ArcGIS Server
    cmd = 'ogr2ogr -f GPKG '+outfile+' "'+layerUrl+'/query'+queryStr+'" -t_srs '+crs.authid()
    print(cmd)
    subprocess.run (cmd)
    
    # Replace code value fields
    if 'outputCodeValueFields' in params:
        layerJsonUrl = url + "/" + str(availableLayers[layer]) + "?f=pjson"
        with urllib.request.urlopen(layerJsonUrl) as u:
            dataLayer = json.loads(u.read().decode())
        
        fields = dataLayer['fields']
        for f in fields:
            if f['name'] in params['outputCodeValueFields']:
                if f['domain'] is not None:
                    if f['domain']['type'] == 'codedValue':
                        mapping = {}
                        for cv in f['domain']['codedValues']:
                            mapping[cv['code']] = cv['name']
                            
                        vl = QgsVectorLayer(outfile, 'temp', 'ogr') 
                        pr = vl.dataProvider()
                        fieldName = f['name']
                        
                        # Create new field
                        tempFieldName = fieldName + '_placeholder'
                        pr.addAttributes([QgsField(tempFieldName, QVariant.String)])
                        vl.updateFields()
                        
                        # Update new field with mapping (code translation)
                        tempFieldIndex = pr.fieldNameIndex(tempFieldName)
                        with edit(vl):
                            for ft in vl.getFeatures():
                                vl.changeAttributeValue(ft.id(), tempFieldIndex, mapping[ft[fieldName]])
                        
                        # Remove original field (with code values)
                        fieldIndex = pr.fieldNameIndex(fieldName)
                        pr.deleteAttributes([fieldIndex])
                        
                        # Rename new field (with code translation) to original
                        tempFieldIndex = pr.fieldNameIndex(tempFieldName)
                        pr.renameAttributes({tempFieldIndex: fieldName})
                        vl.updateFields()
    return outfile

The function is then called by passing a series of parameters, as follows:

extent = QgsRectangle(231224, 6235012, 236480, 6239368)
crs = QgsCoordinateReferenceSystem("EPSG:28356")
layerParams = {
  "layerName": "geographic-line-labels",
  "url": "https://maps.six.nsw.gov.au/arcgis/rest/services/sixmaps/LPIMap/MapServer",
  "layer": "Ridge_Beach",
  "where": "generalname <> 'GREAT DIVIDING RANGE'",
  "outputFields": ["fuzzylinefeaturetype", "generalname"],
  "outputCodeValueFields": ["fuzzylinefeaturetype"],
}
arcGisExtent = {
    "xmin":extent.xMinimum(),
    "ymin":extent.yMinimum(),
    "xmax":extent.xMaximum(),
    "ymax":extent.yMaximum()
}
arcGisParams = {
    "where": layerParams['where'],
    "filterCrs": crs.toWkt(),
    "extent": arcGisExtent,
    "outputFields": layerParams['outputFields'],
    "outputCodeValueFields": layerParams['outputCodeValueFields'],
}
result = addArcGisLayer (layerParams['url'], layerParams['layer'], crs, arcGisParams)
layer = QgsVectorLayer(result, layerParams['layerName'], 'ogr')
QgsProject.instance().addMapLayer(layer)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.