1

Situation: I have setup a python script that writes fire equipment features from AGOL to a FGDB. These features are updated on a weekly basis using Collector.

Script:

============================================================================== ##
## function to update a field - basically converts longs to dates for date fields ##
## since json has dates as a long (milliseconds since unix epoch) and geodb wants ##
## a proper date, not a long.
## ============================================================================== ##
def updateValue(row,field_to_update,value):
    outputfield=next((f for f in fields if f.name ==field_to_update),None)  #find the output field
    if outputfield == None or value == None:        #exit if no field found or empty (null) value passed in
        return
    if outputfield.type == 'Date':
        if value > 0 :                                            # filter "zero" dates
            value = datetime.datetime.fromtimestamp(value/1000)   # convert to date - this is local time, to use utc time
            row.setValue(field_to_update,value)                   # change "fromtimestamp" to "utcfromtimestamp"
    else:
        row.setValue(field_to_update,value)
    return
## ============================================================================== ##

### Generate Token ###
gtUrl = 'https://www.arcgis.com/sharing/rest/generateToken'

#username and password
gtValues = {'username' : 'username',
'password' : 'password',
'referer' : 'http://services1.arcgis.com',
'f' : 'json' }
gtData = urllib.urlencode(gtValues)
gtRequest = urllib2.Request(gtUrl, gtData)
gtResponse = urllib2.urlopen(gtRequest)
gtJson = json.load(gtResponse)
token = gtJson['token']

### Create Replica ###
# Update service url HERE ###
crUrl = 'https://services5.arcgis.com/F5ZjLuZyIJIChBgy/arcgis/rest/services/FIRE_FIGHTING_EQUIPMENT_2/FeatureServer/createReplica'
crValues = {'f' : 'json',
'layers' : '1',
'transportType': 'esriTransportTypeUrl',
'returnAttachments': 'true',
'returnAttachmentsDataByUrl': 'true',
'async': 'false',
'syncModel': 'perLayer',
'targetType': 'client',
'syncDirection': 'download',
'attachmentsSyncDirection': 'bidirectional',
'token' : token }
crData = urllib.urlencode(crValues)
crRequest = urllib2.Request(crUrl, crData)
crResponse = urllib2.urlopen(crRequest)
crJson = json.load(crResponse)
replicaUrl = crJson['URL']

myURLHTTP = str(replicaUrl+ '?token='+token)

myURLHTTPS = string.replace(myURLHTTP,"http", "https")


print myURLHTTPS

urllib.urlretrieve(myURLHTTPS, 'myLayer2.json')
print ("I passed my checkpoint!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")


#urllib.urlretrieve(replicaUrl+ '?token='+token, 'myLayer.json')

### Get Attachment ###
#Change the directory to the FGDB D:\Projects\Jwaneng\Progress\Sprint01\Babatunde\DownloadFGDB
#cwd = os.getcwd()
cwd = r'\\debs-djw-gis-12\Data\Production\DataNonVersioned'

arcpy.DeleteRows_management(cwd + '/GISDB_NonVersioned.gdb/MatchTable')
arcpy.DeleteFeatures_management(cwd + '/GISDB_NonVersioned.gdb/EcohsFireAlarm')

with open('myLayer2.json') as data_file:
    data = json.load(data_file)

for x in data['layers'][0]['attachments']:
    gaUrl = x['url']
    gaFolder = cwd + '\\photos\\' + x['parentGlobalId']
    if not os.path.exists(gaFolder):
        os.makedirs(gaFolder)
    gaName = x['name']
    gaValues = {'token' : token }
    gaData = urllib.urlencode(gaValues)
    urllib.urlretrieve(url=gaUrl + '/' + gaName, filename=os.path.join(gaFolder, gaName),data=gaData)

### Create Features ###
rows = arcpy.InsertCursor(cwd + '/GISDB_NonVersioned.gdb/EcohsFireAlarm')
fields = arcpy.ListFields(cwd + '/GISDB_NonVersioned.gdb/EcohsFireAlarm')

#print (fields)



for cfX in data['layers'][0]['features']:
    pnt = arcpy.Point()
    pnt.X = cfX['geometry']['x']
    pnt.Y = cfX['geometry']['y']
    row = rows.newRow()
    row.shape = pnt

    ### Set Attribute columns HERE ###
    ## makes use of the "updatevalue function to deal with dates ##


    updateValue(row,'EQUIPMENT_NUMBER', cfX['attributes']['EQUIPMENT_NUMBER'])
    updateValue(row,'INSPECTION_DATE', cfX['attributes']['INSPECTION_DATE'])
    updateValue(row,'NEXT_INSPECTION', cfX['attributes']['NEXT_INSPECTION'])
    updateValue(row,'COMMENTS', cfX['attributes']['COMMENTS'])
    updateValue(row,'STATUS', cfX['attributes']['STATUS'])
    updateValue(row,'EQUIPMENT_CONDITION', cfX['attributes']['EQUIPMENT_CONDITION'])
    updateValue(row,'LOCATION', cfX['attributes']['LOCATION'])
    # leave GlobalID out - you cannot edit this field in the destination geodb

    #comment out below fields if you don't have them in your online or destination geodb (editor tracking)
    updateValue(row,'CreationDate', cfX['attributes']['CreationDate'])
    updateValue(row,'Creator', cfX['attributes']['Creator'])
    updateValue(row,'EditDate', cfX['attributes']['EditDate'])
    updateValue(row,'Editor', cfX['attributes']['Editor'])

    updateValue(row,'GlobalID_Str', cfX['attributes']['GlobalID'])

    rows.insertRow(row)

del row
del rows

### Add Attachments ###
### Create Match Table ###
row = arcpy.InsertCursor(cwd + '/GISDB_NonVersioned.gdb/MatchTable')
rows = arcpy.InsertCursor(cwd + '/GISDB_NonVersioned.gdb/MatchTable')

for cmtX in data['layers'][0]['attachments']:
    row = rows.newRow()

    row.setValue('GlobalID_Str', cmtX['parentGlobalId'])
    row.setValue('PhotoPath', cwd + '\\photos\\' + cmtX['parentGlobalId'] + '\\' + cmtX['name'])

    rows.insertRow(row)

del row
del rows

### Add Attachments ###
arcpy.AddAttachments_management(cwd + '/GISDB_NonVersioned.gdb/EcohsFireAlarm', 'GlobalID_Str', cwd + '/GISDB_NonVersioned.gdb/MatchTable', 'GlobalID_Str', 'PhotoPath')

Is there a way I can update my script to keep history of records by looking into last edit date field so that updates on a field are are added as new records on my feature class instead of overwriting whenever the script runs?

  • I would think you would want to add a field to each table or feature class to track Update Dates. You'll need to add a section where you compare the AGOL run against the existing FGDB. I'd assume this would be a for loop in which you go row by row in the AGOL run to check if the feature already exists in the FGDB (maybe verifying using a check using EQUIPMENT_NUMBER). If it already exists, and if every field and the geometry is identical between the two, then you don't append the record to the FGDB. If it does not already exist, you append. If it exists but is different, append. – Zachary Ordo - GISP Apr 24 at 14:23
  • @ZacharyOrdo-GISP how would this handle deleting local features whose remote feature has been deleted? – Kirk Kuykendall Apr 24 at 16:42
  • You could add two additional fields to the tables/feature classes in the FGDB for FromDate and ToDate attributes. In that case, you'd want FromDate to be populated with the original date the record is appended to the FGDB, and ToDate as null. When a change is detected, the outdated record would then have ToDate populated. To detect deletions in AGOL, you'd probably need to have a secondary loop go row by row through the FGDB to compare against AGOL and if a common record is not found in AGOL, the FGDB record would have ToDate populated. – Zachary Ordo - GISP Apr 24 at 18:11
  • The FromDate and ToDate schema design is something that many data models use, particularly UPDM, PODS, and APDM pipeline data models. – Zachary Ordo - GISP Apr 24 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.