Arcpy way to access underlying feature classes contained in a web map

Question

I am building a stand alone arcpy script, and need to connect to the specific feature classes within a published web map, query/copy from those feature classes, then after some processing update those specific feature classes with new or modified data.

I have a GUID value to the specific web map I need to interact with (this is my only point of entry to the underlying feature classes), but I need to be able to find the database connections to the database and feature classes that make up the web map. The web map was constructed from 3 published web feature layers/feature services, each containing 1-many feature classes, and I ultimately need to obtain the database connection to a specific point feature and line feature, run my script (which makes copies of those feature classes then runs arcpy geo-processing steps-near_analysis, snap, SplitLineAtPoint, AddGeometryAttributes) then need to update/add features back to the respective database and feature classes.

My querying/geo-processing script is already working on hardcoded database connections, but now I need to loop through the various GUIDs I have to different web maps and dynamically connect to variable database connections within the web map (that I have a GUID ID for).

I have found documentation showing how to access webmaps:

from arcgis.gis import GIS
gis = GIS()
web_map = gis.content.get('<GUID value>')#where GUID value is the GUID I have in a list

and then accessing the web feature layers that make up the web map:

web_map_layers = webMap(web_map)
layers = web_map_layers.layers

but printing that out just seems to give me a list of dictionaries, not something that I can assign to a variable and treat as a real feature class within a database or even temporarily in memory.

Answer 1

I was finally able to get this to work, by taking the WebMap guid ID supplied, and:

1. find out the Feature Service of the feature I wanted to copy
2. hit the REST end point of that feature service
3. copy that feature set down to a feature class.

This then allowed me to run my geo-processing tools, and report my findings. The only thing I have yet to figure out is how to push my updates/additions back to the underlying feature classes in the feature services...

Here is the code that worked for me (with some data/code redacted/changed):

import arcpy
from arcpy import env
import timeit
import uuid
from arcgis.gis import GIS
from arcgis import features
from arcgis.mapping import WebMap
from collections import OrderedDict
import json
import urllib
import http.client

start_time = timeit.default_timer()

#Constant values
SURVEY_DB_WORKSPACE = r'C:\Users\survey_gdb.sde'
#survey table server instance and owner prefix values
SURVEY_DB_INSTANCE_AND_OWNER = 'mydb.dbo.'
GEO_PROC_WORKSPACE = r"C:\Users\geo_proc_gdb.sde"
#geo-proc server instance and owner prefix values
GEO_PROC_DB_INSTANCE_AND_OWNER = 'mydb_gdb.DBO.'
#rest services credentials
USERNAME = 'user'
PASSWORD = 'password'
PORTAL_URL = 'https://mywebsite/portal'
SERVERNAME = "mywebsite"
PORT = ''#if using web adaptor, leave empty (modify tokenURL)
TOKEN_URL = "https://mywebsite/server/admin/generateToken"

def main():

    #set environment to the Survey database, loop through surveys table, grab WebMap GUID
    workspace = SURVEY_DB_WORKSPACE
    arcpy.env.workspace = workspace
    arcpy.env.overwriteOutput = True

    surveys = SURVEY_DB_INSTANCE_AND_OWNER + 'Surveys'
    where_clause = "active = 1"
    fields = ['OID@', 'Name', 'Id', 'Progress']
    web_map_ids = OrderedDict()

    with arcpy.da.SearchCursor(surveys, fields) as cursor:
        for row in cursor:
            web_map_ids[row[0]] = [row[1], row[2], row[3]]

    #change environment to the geo-processing database
    workspace = GEO_PROC_WORKSPACE

    #connect to db and set point and line features
    field = 'state'
    lines_backup = GEO_PROC_DB_INSTANCE_AND_OWNER + 'lines_backup'
    points_backup = GEO_PROC_DB_INSTANCE_AND_OWNER + 'points_backup'
    tmp_lines_split = GEO_PROC_DB_INSTANCE_AND_OWNER + 'tmp_lines_split'
    lines_split = GEO_PROC_DB_INSTANCE_AND_OWNER + 'lines_split'
    arcpy.env.workspace = workspace
    arcpy.env.overwriteOutput = True

    #Define how to acquire a rest services token
    #from https://community.esri.com/thread/83654

    def getToken(username, password, serverName, serverPort):
        params = urllib.parse.urlencode({'username': USERNAME, 'password': PASSWORD,'client': 'requestip', 'f': 'json'})
        headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}

        # Connect to URL and post parameters
        httpConn = http.client.HTTPSConnection(serverName, serverPort)
        httpConn.request("POST", TOKEN_URL, params, headers)

        # Read response
        response = httpConn.getresponse()
        print(response.status)
        if (response.status != 200):
            httpConn.close()
            print("Error while fetching tokens from admin URL. Please check the URL and try again.")
            return
        else:
            data = response.read()
            httpConn.close()

        # Check that data returned is not an error object
        if not assertJsonSuccess(data): 
            return

        # Extract the token from it
        token = json.loads(data) 
        return token['token']

        ###End getToken function

# A function that checks that the input JSON object is not an error object. 
def assertJsonSuccess(data):
    obj = json.loads(data)
    if 'status' in obj and obj['status'] == "error":
        print("Error: JSON object returns an error. " + str(obj))
        return False
    else:
        return True

    ### End jsonSuccess function
    """
        This is the Main loop through WebMap values, processing and reporting back to survey, then continuing
    """
    #loop through the survey web map guids for processing-this is where the GUID values come from
    current_survey = ''
    for k,v in web_map_ids.items():
        if v[1] is None:
            continue
        current_survey = v[0]

        #get WebMap
        gis = GIS(PORTAL_URL, username=USERNAME, password=PASSWORD)
        web_map = gis.content.get(v[1])
        web_Map = WebMap(web_map)
        web_map_layers = web_Map.layers

        #extract the feature service name that contains the Point feature class
        num = 0
        for item in web_map_layers:
            for tag in item:
                if tag =='title':
                    if item[tag] == 'Point':
                        num = web_map_layers.index(item)
                    else:
                        continue
                else:
                    continue

        point_url = web_map_layers[num]['url']

        #from https://www.esri.com/arcgis-blog/products/arcgis-desktop/analytics/quick-tips-consuming-feature-services-with-geoprocessing/
        #Extract Points from feature service to feature class
        baseURL = point_url
        where = '1=1'
        fields = '*'
        token = 'getToken(USERNAME, PASSWORD, SERVERNAME, PORT)'
        query = "?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, fields, token)
        # See http://services1.arcgis.com/help/index.html?fsQuery.html for more info on FS-Query
        fsURL = baseURL + query
        fs = arcpy.FeatureSet()
        fs.load(fsURL)

        #clean up prior to re-creating backups
        if arcpy.Exists(lines_backup):
            arcpy.Delete_management(lines_backup)
        if arcpy.Exists(points_backup):
            arcpy.Delete_management(points_backup)

        #copy Points from feature service to feature class, for geo-processing
        arcpy.CopyFeatures_management(fs, points_backup)

Once I was able to copy the feature set from the feature service to a feature class, I was able to then run the rest of my geo-processing script. I have not yet been able to push updates/additions back up to those feature classes underlying the feature services though. tackling that now...