Location Analytics - ArcGIS Web Services (Feature & Raster)

Question

I have been attempting to develop some methodology or approach in order to monitor or track where users of our web map services are looking. Ideally the coordinates of the window.

Mapbox has created a very powerful approach to monitor statistics and locate areas that users are viewing.

I am looking to implement something similar on my ArcGIS Server in order to see where users are viewing the most. This way I can better densify the data in areas that are of the most interest.

So far I have utilized the Map Index Toolbox (I have lost the original link to the toolbox - but the python code for it is below)

"""     
Script Name: Map_Index.py

    https://bitbucket.org/npeihl/mapindextool

    Description: ArcGIS Tool written in Python.  The Map Index Tool creates a geographic index of all ArcMap documents (MXDs) in a folder supplied by the user. For each MXD, the tool finds the largest dataframe on the Layout view and sends the extent of that dataframe to a shapefile in the folder (map_index.shp). Tested on ArcGIS Desktop version 10. Not tested and likely not working on ArcGIS Desktop versions 9.3.1 and below.

    WARNING: This tool will destroy and overwrite any shapefile named map_index within the supplied folder. Please use with caution.

    Author: Nicholas Peihl
    Email: [email protected]

    Last Modified: 04/20/2011
    """
    import arcpy
    import os, sys
    from datetime import datetime
    from time import strftime


    wp = arcpy.GetParameterAsText(0)
    spatialRef = arcpy.GetParameterAsText(1)
    fcName = "map_index.shp"
    scriptPath = sys.path[0]
    arcpy.AddMessage("Script path is %s" % scriptPath)
    toolDataPath = os.path.join(scriptPath, "ToolData")
    fcTemplate = os.path.join(toolDataPath, "map_index_template.shp")

    arcpy.env.workspace = wp
    files = arcpy.ListFiles("*.mxd")
    if len(files) == 0:
        arcpy.AddError("No ArcMap Documents were found in this folder. Please try another folder.")
    else:
        if arcpy.Exists(os.path.join(wp, fcName)):
            arcpy.AddWarning("Existing Map Index shapefile found. Deleting the old index file.")
            arcpy.Delete_management(fcName)

        arcpy.AddMessage("Creating the new Map Index shapefile")
        index = arcpy.CreateFeatureclass_management(wp, fcName, "POLYGON", fcTemplate,"","", spatialRef)

        for mxd in files:
            mxd = wp + "\\" + mxd
            arcpy.AddMessage("Opening map document %s" % mxd)
            mapDoc = arcpy.mapping.MapDocument(mxd)
            dataframes = arcpy.mapping.ListDataFrames(mapDoc)
            if len(dataframes) == 1:
                pFrame = dataframes[0]
            else:
                max = 0
                for frame in dataframes:
                    size = frame.elementWidth * frame.elementHeight
                    if size > max:
                        pFrame = frame
                        max = size

            xmin = pFrame.extent.XMin
            ymin = pFrame.extent.YMin
            xmax = pFrame.extent.XMax
            ymax = pFrame.extent.YMax
            framespatialRef = pFrame.spatialReference

            coords = [xmin,ymin],[xmax,ymin],[xmax,ymax],[xmin, ymax]
            point = arcpy.Point()
            array = arcpy.Array()

            for coord in coords:
                point.X = coord[0]
                point.Y = coord[1]
                array.append(point)

            polygon = arcpy.Polygon(array, framespatialRef)

            arcpy.AddMessage("Adding the extent of %s to the index." % mxd)
            rows = arcpy.InsertCursor(index, spatialRef)
            row = rows.newRow()
            row.PATH = mapDoc.filePath
            row.TITLE = mapDoc.title
            row.AUTHOR = mapDoc.author
            row.LASTSAVED = mapDoc.dateSaved
            row.LASTUPDATE = datetime.strftime(datetime.now(),"%Y-%m-%d")
            row.SHAPE = polygon
            rows.insertRow(row)

            del row
            del rows

The result is a max/min window of all mxd projects created for the year. This result is excellent for all map driven requests. However, it does not deal with online mapping or demand.

---

What I am looking for: any toolset or methodology that will be able to monitor my Map Services for both Raster Types and Feature Types.

MY CURRENT STACK:

Back-end

• ArcGIS For Server 10.3.1
• ArcGIS Desktop 10.3.1

Front-end

• WebAppBuilder
• Developer WebAppBuilder
• ESRI Leaflet API (if needed)

UPDATE I have begun to tweak some basic ESRI JS code to add to WebAppBuilder

I have a fiddle that I am working with as of right now that simply builds out the extent of the viewing window. I am wondering if anyone knows a way to capture the XMin,YMin,XMax,YMax and write that data to some database (preferably a shapefile or SQL Database)

The code to capture the XMin,YMin,XMax,YMax is below:

  dojo.require("esri.map");

  function init() {
    var initialExtent = new esri.geometry.Extent({"xmin":244598,"ymin":6241389,"xmax":278995,"ymax":6264320,"spatialReference":{"wkid":102100}});
    var map = new esri.Map("map", {
      basemap: "streets",
      extent: initialExtent
    });
    dojo.connect(map, "onExtentChange", showExtent);
  }

  function showExtent(extent) {
    var s = "";
    s = "XMin: "+ extent.xmin.toFixed(2) + " "
       +"YMin: " + extent.ymin.toFixed(2) + " "
       +"XMax: " + extent.xmax.toFixed(2) + " "
       +"YMax: " + extent.ymax.toFixed(2);
    dojo.byId("info").innerHTML = s;
  }

  dojo.ready(init);

Answer 1

For those interested:

I was able to locate an ESRI example Write requested map extents to a feature class that works directly with REST services on an ArcGIS Server.

The is also a video on Working with Logs in ArcGIS 10.1 for Server that shows a step by step.

# Queries the logs to find the map extents requested for a given map service 

# For output Featureclass creation
print "\nImporting ArcPy..."
import arcpy

# Set Script arguments
arcpy.env.overwriteOutput = True

# For Http calls
import httplib, urllib, json

# For system tools
import sys, datetime, os

# For reading passwords without echoing
import getpass

#Defines the entry point into the script
def main(argv=None):
    # Print some info
    print
    print "This tool is a sample script that queries the ArcGIS Server logs."
    print

    # Ask for admin/publisher user name and password
    username = raw_input("Enter user name: ")
    password = getpass.getpass("Enter password: ")

    # Ask for server name
    serverName = raw_input("Enter Server name: ")
    serverPort = 6080

    # Ask for map service name
    mapService = raw_input("Enter map service name, using a forward slash / to denote a folder: ")

    if mapService.endswith(".MapServer"):
        pass
    else:
        mapService += ".MapServer"

    # Ask for output workspace
    outputWorkspace = raw_input("Enter output Workspace (Geodatabase location): ")

    # Ask for output featureclass name
    outputFeatureclass = raw_input("Enter output Featureclass name: ")

    # Construct REST service URL
    serviceURL = "/arcgis/rest/services/{0}".format( mapService.replace( ".", "/"))

    # Get a token
    print "Requesting Token..."
    token = getToken(username, password, serverName, serverPort)
    if token == "":
        print "Could not generate a token with the username and password provided."
        return

    # Get Extent detail for service
    print "\nLooking up Service details..."

    serviceURL = serviceURL + "/?Token=" + token
    fullExtent = getFullExtent( serverName, serverPort, serviceURL)

    if not fullExtent:
        return 

    # Construct URL to query the logs
    logQueryURL = "/arcgis/admin/logs/query"
    logFilter = "{'services': ['" + mapService + "']}"

    # Supply the log level, filter, token, and return format
    params = urllib.urlencode({'level': 'FINE', 'filter': logFilter, 'token': token, 'f': 'json'})

    headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}

    # Connect to URL and post parameters
    print "Accessing Logs..."
    httpConn = httplib.HTTPConnection(serverName, serverPort)
    httpConn.request("POST", logQueryURL, params, headers)

    # Read response
    response = httpConn.getresponse()
    if (response.status != 200):
        httpConn.close()
        print "  Error while querying logs."
        return
    else:
        data = response.read()

        # Check that data returned is not an error object
        if not assertJsonSuccess(data):
            print "  Error returned by operation. " + data
        else:
            print "  Operation completed successfully!"

        # Deserialize response into Python object
        dataObj = json.loads(data)
        httpConn.close()

        # Open Insert Cursor on output
        output = openCursor( outputWorkspace, outputFeatureclass, fullExtent[ "spatialReference"][ "wkid"])

        if not output:
            return

        # Need this variable to track number of events found for ExportMapImage call
        logEvents = 0

        # Need Array to hold Shape
        shapeArray = arcpy.Array()

        # Iterate over messages
        for item in dataObj[ "logMessages"]:
            eventDateTime = datetime.datetime.fromtimestamp( float( item[ "time"]) / 1000)

            if item[ "message"].startswith( "Extent:"):
                eventScale = None        # Scale
                eventInvScale = None    # Inverse-Scale
                eventWidth = None        # Width
                eventHeight = None    # Height

                # Cycle through message details
                for pair in item[ "message"].replace(" ", "").split( ";"):
                    if pair.count( ":") == 1:
                        key, val = pair.split( ":")

                        # Pick out Extent
                        if key == "Extent" and val.count( ",") == 3:
                            # Split into ordinate values
                            MinX, MinY, MaxX, MaxY = val.split( ",")
                            MinX = float( MinX)
                            MinY = float( MinY)
                            MaxX = float( MaxX)
                            MaxY = float( MaxY)

                            # Make sure extent is within range
                            if MinX > fullExtent[ "xmin"] and MaxX < fullExtent[ "xmax"] and MinY > fullExtent[ "ymin"] and MaxY < fullExtent[ "ymax"]:
                                shapeArray.add( arcpy.Point( MinX, MinY))
                                shapeArray.add( arcpy.Point( MinX, MaxY))
                                shapeArray.add( arcpy.Point( MaxX, MaxY))
                                shapeArray.add( arcpy.Point( MaxX, MinY))
                                shapeArray.add( arcpy.Point( MinX, MinY))

                        # Pick out Size
                        if key == "Size" and val.count( ",") == 1:
                            eventWidth, eventHeight = val.split( ",")
                            eventWidth = float( eventWidth)
                            eventHeight = float( eventHeight)

                        # Pick out Scale
                        if key == "Scale":
                            eventScale = float( val)
                            eventInvScale = 1 / eventScale

                # Save if Shape created
                if shapeArray.count > 0:
                    # Create new row
                    newRow = output.newRow()

                    # Add Shape and Event Date
                    newRow.setValue( "Shape", shapeArray)
                    newRow.setValue( "EventDate", eventDateTime)
                    newRow.setValue( "Scale", eventScale)
                    newRow.setValue( "InvScale", eventInvScale)
                    newRow.setValue( "Width", eventWidth)
                    newRow.setValue( "Height", eventHeight)

                    output.insertRow( newRow)

                    # Clear out Array points
                    shapeArray.removeAll()

                    logEvents += 1

        print "\nDone!\n\nTotal number of events found in logs: {0}".format( logEvents)

        return

#A function to query service for Extent and Spatial Reference details
def getFullExtent( serverName, serverPort, serviceURL):
    # Supply the return format
    params = urllib.urlencode({'f': 'json'})

    headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}

    # Connect to URL and post parameters
    httpConn = httplib.HTTPConnection(serverName, serverPort)
    httpConn.request("POST", serviceURL, params, headers)

    # Read response
    response = httpConn.getresponse()
    if (response.status != 200):
        httpConn.close()
        print "Error while querying Service details."
        return
    else:
        data = response.read()

        # Check that data returned is not an error object
        if not assertJsonSuccess(data):
            print "Error returned by Service Query operation. " + data

        # Deserialize response into Python object
        dataObj = json.loads(data)
        httpConn.close()

        if not 'fullExtent' in dataObj:
            print "Unable to find Extent detail for '{0}'!".format( serviceURL)
            print dataObj
        elif not 'spatialReference' in dataObj[ 'fullExtent']:
            print "Unable to find Spatial Reference for '{0}'!".format( serviceURL)
            print "dataObj"

        else:
            return dataObj[ 'fullExtent']

    return

#A function to create new Featureclass and return an Insert Cursor, used to store Map Query Extents.
def openCursor( workspace, featureclassName, srid):
    if not arcpy.Exists( workspace):
        print "Unable to find Workspace '{0}'...".format( workspace)
        return

    print "Creating output Featureclass..."
    arcpy.CreateFeatureclass_management( workspace, featureclassName, "POLYGON", None, None, None, srid)

    Featureclass = workspace + os.sep + featureclassName

    print "  Adding field(s)..."
    arcpy.AddField_management( Featureclass, "EventDate", "DATE", None, None, None, None, "NULLABLE", "NON_REQUIRED")
    arcpy.AddField_management( Featureclass, "Scale", "DOUBLE", 19, 2, None, None, "NULLABLE", "NON_REQUIRED")
    arcpy.AddField_management( Featureclass, "InvScale", "DOUBLE", 19, 12, None, None, "NULLABLE", "NON_REQUIRED")
    arcpy.AddField_management( Featureclass, "Width", "LONG", 9, None, None, None, "NULLABLE", "NON_REQUIRED")
    arcpy.AddField_management( Featureclass, "Height", "LONG", 9, None, None, None, "NULLABLE", "NON_REQUIRED")

    print "  Opening Insert Cursor..."
    return arcpy.InsertCursor( Featureclass)

#A function to generate a token given username, password and the adminURL.
def getToken(username, password, serverName, serverPort):
    # Token URL is typically http://server[:port]/arcgis/admin/generateToken
    tokenURL = "/arcgis/admin/generateToken"

    # URL-encode the token parameters
    params = urllib.urlencode({'username': username, 'password': password, 'client': 'requestip', 'f': 'json'})

    headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}

    # Connect to URL and post parameters
    httpConn = httplib.HTTPConnection(serverName, serverPort)
    httpConn.request("POST", tokenURL, params, headers)

    # Read response
    response = httpConn.getresponse()
    if (response.status != 200):
        httpConn.close()
        print "Error while fetching tokens from admin URL. Please check the URL and try again."
        return
    else:
        data = response.read()
        httpConn.close()

        # Check that data returned is not an error object
        if not assertJsonSuccess(data):
            return

        # Extract the token from it
        token = json.loads(data)
        return token['token']


#A function that checks that the input JSON object
# is not an error object.
def assertJsonSuccess(data):
    obj = json.loads(data)
    if 'status' in obj and obj['status'] == "error":
        print "Error: JSON object returns an error. " + str(obj)
        return False
    else:
        return True

# Script start
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

The End Result:

Answer 2

There are a number of ways to solve this depending on your exact needs and existing architecture. Some suggestions.

Publish another ArcGIS Server feature service used to capture this usage data and write to it when the map extent changes. Create a new "add" geometry from your map extent and write directly to the feature service with applyEdits. See https://developers.arcgis.com/javascript/jsapi/featurelayer-amd.html#applyedits.

Use an Amazon Web Services S3 bucket. Here's an example of writing to it in Javascript: http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-examples.html#Uploading_data_into_an_object