1

I have access to Arcgis for Server and am trying to overwrite a couple feature service layers I have hosted on arcgis online. I usually overwrite these manually by uploading zipped shapefiles from a folder on my desktop. I'd like to simply overwrite these feature service layers using python possibly with arcrest but don't know how to do this. I've found a lot of mxd to sd file conversion workflows with python but I want to use zipped shapefiles to upload to arcgis online not mxd's. I also am willing to use arcgis for Server if this is more efficient or secure. So far, my upload attempt has failed using the code in the answer below. with a
KeyError: 'id' Hoping to eventually upload/publish/overwrite several zipped shapefiles with for loops but for now I'm just looking to figure out how to do one at a time.

  • All the pieces you need are here. You'd have to modify the MXD parts. But it has upload and publish. – KHibma Aug 29 '16 at 19:00
  • @KHibma I changed the variable MXD to SHPand added the path to the individual shapefile. couple questions: how do I check the MAXRECORDS currently in my hosted feature layer as well as the sharing settings so as to not change/overwrite the current settings? – StargazingFish Aug 29 '16 at 19:41
  • 1
    Sorry it would take me quite sometime to hack that out. In short, you'll need to go look at the REST end point of the service and get that value, once you have that value you can add in 'maxRecordCount' : 123 to the query parameters in the publish function. – KHibma Aug 29 '16 at 20:17
  • @KHibma No problem. I already got it but had another question about the python code itself. I added a snippet in the main question for reference. – StargazingFish Aug 29 '16 at 20:58
  • Get rid of the code... thats specific to getting the MXD in proper shape. Since you're publishing SHPs inside a ZIP, just delete this code. Upload the ZIP and call Publish on the ItemID you get back from that upload. – KHibma Aug 29 '16 at 21:00
3

The following code (380 lines) will overwrite a ZIP file on arcgis.com and call PUBLISH on that. The zip file is expected to have shapefiles inside. You'll need to start modifying items to match your existing service, credentials, zip file location, etc starting at line 350.

The one thing you may need to massage is if you have multiple layers inside your ZIP. If so, you'll need to look at the layerInfo settings from the REST SPEC, specific to shapefiles, feature collections, and file geodatabases and insert the appropriate key:values into the publishParameters at line 203.

Note - code below only supported with Python 2.7.x

import os, sys, time
import urllib, urllib2
import json, mimetypes
import gzip
from io import BytesIO
import string
import random

class AGOLHandler(object):

    def __init__(self, username, password, serviceName, folderName):

        self.headers = {
            'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
            'User-Agent': ('updatehostedfeatureservice')
        }
        self.username = username
        self.password = password
        self.base_url = "https://www.arcgis.com/sharing/rest"
        self.serviceName = serviceName
        self.token = self.getToken(username, password)
        self.itemID = self.findItem("Feature Service")
        self.SDitemID = self.findItem("Shapefile")
        self.folderName = folderName
        self.folderID = self.findFolder()

    def getToken(self, username, password, exp=60):

        referer = "http://www.arcgis.com/"
        query_dict = {'username': username,
                      'password': password,
                      'expiration': str(exp),
                      'client': 'referer',
                      'referer': referer,
                      'f': 'json'}

        token_url = '{}/generateToken'.format(self.base_url)

        token_response = self.url_request(token_url, query_dict, 'POST')

        if "token" not in token_response:
            print(token_response['error'])
            sys.exit()
        else:
            return token_response['token']

    def findItem(self, findType):
        """ Find the itemID of whats being updated
        """

        searchURL = self.base_url + "/search"

        query_dict = {'f': 'json',
                      'token': self.token,
                      'q': "title:\"" + self.serviceName + "\"AND owner:\"" +
                      self.username + "\" AND type:\"" + findType + "\""}

        jsonResponse = self.url_request(searchURL, query_dict, 'POST')

        if jsonResponse['total'] == 0:
            print("\nCould not find a service to update. Check the service name in the settings.ini")
            sys.exit()
        else:
            resultList = jsonResponse['results']
            for it in resultList:
                if it["title"] == self.serviceName:
                    print("found {} : {}").format(findType, it["id"])
                    return it["id"]

    def findFolder(self, folderName=None):
        """ Find the ID of the folder containing the service
        """

        if self.folderName == "None" or folderName is None:
            return ""

        findURL = "{}/content/users/{}".format(self.base_url, self.username)

        query_dict = {'f': 'json',
                      'num': 1,
                      'token': self.token}

        jsonResponse = self.url_request(findURL, query_dict, 'POST')

        for folder in jsonResponse['folders']:
            if folder['title'] == self.folderName:
                return folder['id']

        print("\nCould not find the specified folder name provided in the settings.ini")
        print("-- If your content is in the root folder, change the folder name to 'None'")
        sys.exit()

    def upload(self, fileName, tags, description):
        """
         Overwrite the SD on AGOL with the new SD.
         This method uses 3rd party module: requests
        """

        updateURL = '{}/content/users/{}/{}/items/{}/update'.format(self.base_url, self.username,
                                                                    self.folderID, self.SDitemID)

        query_dict = {"filename": fileName,
                      "type": "Shapefile",
                      "title": self.serviceName,
                      "tags": tags,
                      "description": description,
                      "f": "json",
                      'multipart': 'true',
                      "token": self.token}

        details = {'filename': fileName}
        add_item_res = self.url_request(updateURL, query_dict, "POST", "", details)

        itemPartJSON = self._add_part(fileName, add_item_res['id'], "Shapefile")

        if "success" in itemPartJSON:
            itemPartID = itemPartJSON['id']

            commit_response = self.commit(itemPartID)

            # valid states: partial | processing | failed | completed
            status = 'processing'
            while status == 'processing' or status == 'partial':
                status = self.item_status(itemPartID)['status']
                time.sleep(1.5)

            print("updated SD:   {}".format(itemPartID))
            return True

        else:
            print("\n.sd file not uploaded. Check the errors and try again.\n")
            print(itemPartJSON)
            sys.exit()

    def _add_part(self, file_to_upload, item_id, upload_type=None):
        """ Add the item to the portal in chunks.
        """

        def read_in_chunks(file_object, chunk_size=10000000):
            """Generate file chunks of 10MB"""
            while True:
                data = file_object.read(chunk_size)
                if not data:
                    break
                yield data

        url = '{}/content/users/{}/items/{}/addPart'.format(self.base_url, self.username, item_id)

        with open(file_to_upload, 'rb') as f:
            for part_num, piece in enumerate(read_in_chunks(f), start=1):
                title = os.path.basename(file_to_upload)
                files = {"file": {"filename": file_to_upload, "content": piece}}
                params = {
                    'f': "json",
                    'token': self.token,
                    'partNum': part_num,
                    'title': title,
                    'itemType': 'file',
                    'type': upload_type
                }

                request_data, request_headers = self.multipart_request(params, files)
                resp = self.url_request(url, request_data, "MULTIPART", request_headers)

        return resp

    def item_status(self, item_id, jobId=None):
        """ Gets the status of an item.
        Returns:
            The item's status. (partial | processing | failed | completed)
        """

        url = '{}/content/users/{}/items/{}/status'.format(self.base_url, self.username, item_id)
        parameters = {'token': self.token,
                      'f': 'json'}

        if jobId:
            parameters['jobId'] = jobId

        return self.url_request(url, parameters)

    def commit(self, item_id):
        """ Commits an item that was uploaded as multipart
        """

        url = '{}/content/users/{}/items/{}/commit'.format(self.base_url, self.username, item_id)
        parameters = {'token': self.token,
                      'f': 'json'}

        return self.url_request(url, parameters)

    def publish(self, summary, maxRecords):
        """ Publish the existing SD on AGOL (it will be turned into a Feature Service)
        """

        publishURL = '{}/content/users/{}/publish'.format(self.base_url, self.username)

        query_dict = {'itemID': self.SDitemID,
                      'filetype': 'shapefile',
                      'overwrite': 'true',
                      'f': 'json',
                      'token': self.token,
                      'publishParameters' : {"name":self.serviceName,
                                             'maxRecordCount':maxRecords,
                                             'description':summary}
                      }

        jsonResponse = self.url_request(publishURL, query_dict, 'POST')
        try:
            if 'jobId' in jsonResponse['services'][0]:
                jobID = jsonResponse['services'][0]['jobId']

                # valid states: partial | processing | failed | completed
                status = 'processing'
                print("Checking the status of publish..")
                while status == 'processing' or status == 'partial':
                    status = self.item_status(self.SDitemID, jobID)['status']
                    print("  {}".format(status))
                    time.sleep(2)

                if status == 'completed':
                    print("item finished published")
                    return jsonResponse['services'][0]['serviceItemId']
                if status == 'failed':
                    raise("Status of publishing returned FAILED.")

        except Exception as e:
            print("Problem trying to check publish status. Might be further errors.")
            print("Returned error Python:\n   {}".format(e))
            print("Message from publish call:\n  {}".format(jsonResponse))
            print(" -- quit --")
            sys.exit()


    def enableSharing(self, newItemID, everyone, orgs, groups):
        """ Share an item with everyone, the organization and/or groups
        """

        shareURL = '{}/content/users/{}/{}/items/{}/share'.format(self.base_url, self.username,
                                                                  self.folderID, newItemID)

        if groups is None:
            groups = ''

        query_dict = {'f': 'json',
                      'everyone': everyone,
                      'org': orgs,
                      'groups': groups,
                      'token': self.token}

        jsonResponse = self.url_request(shareURL, query_dict, 'POST')

        print("successfully shared...{}...".format(jsonResponse['itemId']))

    def url_request(self, in_url, request_parameters, request_type='GET',
                    additional_headers=None, files=None, repeat=0):

        if request_type == 'GET':
            req = urllib2.Request('?'.join((in_url, urllib.urlencode(request_parameters))))
        elif request_type == 'MULTIPART':
            req = urllib2.Request(in_url, request_parameters)
        else:
            req = urllib2.Request(
                in_url, urllib.urlencode(request_parameters), self.headers)

        if additional_headers:
            for key, value in list(additional_headers.items()):
                req.add_header(key, value)
        req.add_header('Accept-encoding', 'gzip')

        response = urllib2.urlopen(req)

        if response.info().get('Content-Encoding') == 'gzip':
            buf = BytesIO(response.read())
            with gzip.GzipFile(fileobj=buf) as gzip_file:
                response_bytes = gzip_file.read()
        else:
            response_bytes = response.read()

        response_text = response_bytes.decode('UTF-8')
        response_json = json.loads(response_text)

        if not response_json or "error" in response_json:
            rerun = False
            if repeat > 0:
                repeat -= 1
                rerun = True

            if rerun:
                time.sleep(2)
                response_json = self.url_request(
                    in_url, request_parameters, request_type,
                    additional_headers, files, repeat)

        return response_json

    def multipart_request(self, params, files):
        """ Uploads files as multipart/form-data. files is a dict and must
            contain the required keys "filename" and "content". The "mimetype"
            value is optional and if not specified will use mimetypes.guess_type
            to determine the type or use type application/octet-stream. params
            is a dict containing the parameters to be passed in the HTTP
            POST request.

            content = open(file_path, "rb").read()
            files = {"file": {"filename": "some_file.sd", "content": content}}
            params = {"f": "json", "token": token, "type": item_type,
                      "title": title, "tags": tags, "description": description}
            data, headers = multipart_request(params, files)
            """
        # Get mix of letters and digits to form boundary.
        letters_digits = "".join(string.digits + string.ascii_letters)
        boundary = "----WebKitFormBoundary{}".format("".join(random.choice(letters_digits) for i in range(16)))
        file_lines = []
        # Parse the params and files dicts to build the multipart request.
        for name, value in params.iteritems():
            file_lines.extend(("--{}".format(boundary),
                               'Content-Disposition: form-data; name="{}"'.format(name),
                               "", str(value)))
        for name, value in files.items():
            if "filename" in value:
                filename = value.get("filename")
            else:
                raise Exception("The filename key is required.")
            if "mimetype" in value:
                mimetype = value.get("mimetype")
            else:
                mimetype = mimetypes.guess_type(filename)[0] or "application/octet-stream"
            if "content" in value:
                file_lines.extend(("--{}".format(boundary),
                                   'Content-Disposition: form-data; name="{}"; filename="{}"'.format(name, filename),
                                   "Content-Type: {}".format(mimetype), "",
                                   (value.get("content"))))
            else:
                raise Exception("The content key is required.")
        # Create the end of the form boundary.
        file_lines.extend(("--{}--".format(boundary), ""))

        request_data = "\r\n".join(file_lines)
        request_headers = {"Content-Type": "multipart/form-data; boundary={}".format(boundary),
                           "Content-Length": str(len(request_data))}
        return request_data, request_headers


if __name__ == "__main__":

    print("Starting Feature Service publish process from zip (shp files)")

    # AGOL Credentials
    inputUsername = 'USERNAME'
    inputPswd = 'PASSWORD'

    # FS values
    ZIPFILE = "C:\\temp\\shps\\mydata.zip"
    serviceName = "mydata"
    folderName = None
    tags = "tag1,tag2"
    summary = "my summary"
    maxRecords = 1000

    # Share FS to: everyone, org, groups
    shared = True
    everyone = 'true'
    orgs = 'true'
    groups = None # Groups are by ID. Multiple groups comma separated

    # initialize AGOLHandler class
    agol = AGOLHandler(inputUsername, inputPswd, serviceName, folderName)

    # overwrite the existing .SD on arcgis.com
    if agol.upload(ZIPFILE, tags, summary):

        # publish the sd which was just uploaded
        fsID = agol.publish(summary, maxRecords)

        # share the item
        if shared:
            agol.enableSharing(fsID, everyone, orgs, groups)

        print("\nfinished.")
  • did you write all of this code? – ziggy Aug 30 '16 at 15:48
  • Vast majority of it yes. I've mashed it together from various projects. Only part that is borrowed is the multipart upload from stackoverflow. Why? – KHibma Aug 30 '16 at 15:50
  • good stuff, very impressive – ziggy Aug 30 '16 at 16:44
  • Rather than having a zip containing a collection of layers I have multiple zips, each is a shapefile exported from a layer. I plan to loop through this process for each zipped shapefile if this works. getting a few errors though like KeyError: 'id' on line 115 also, shouldn't I need to type in the server name and password somewhere? – StargazingFish Aug 30 '16 at 19:16
  • That means the upload failed. As for server name - you said in your original question you're posting to arcgis online. You dont need to set the server for that. Your requirements seem to keep changing. You need to go back and edit your question, clearly defining what you have and what you're trying to accomplish and what you've tried. – KHibma Aug 30 '16 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.