I currently work as a GIS Coordinator for a small scale utility that uses AGOL. That being said, many of our feature layers are hosted in AGOL and are regularly edited or modified in the AGOL environment. After some research, I didn't really find anything code shared publicly that would do exactly what I wanted. So I decided to take a stab at writing my first program! I made some simply modifications to ESRI's script (https://support.esri.com/en/technical-article/000022524) to reduce the amount of user input required and then wrote new code from there...
My first issue with the ESRI provided script was that it only addressed downloaded each hosted feature layer as an individual FGDB. This isn't really a huge issue, but when you are talking about working with regular back ups - 40+ individual FGDBs becomes cumbersome. So the first addition you will see is a process to unpack each of the individually compressed FGDBs and deposit the individual feature classes into a single new FGDB.
My second addition aims to take the previously deposited feature classes and export them to shapefiles to be included in the same back up directory as the FGDB. This isn't extremely important for most, but helps with my organization as we often share shapefiles with third parties, like engineering groups, when we are working on capital projects.
All-in-all this code does what I intended it to do. It downloads all hosted feature layers from AGOL, and creates shapefiles of each layer. However, this is my first complete program I have written, so I was interested in getting some input from more experienced coders.
For reference, this code was developed in a Jupyter notebook within ArcGIS Pro. I have provided as much markdown as possible to help anyone else who may be interested in using this code to automate their AGOL backups.
# this cell access your AGOL account and prompts you to enter a password to continue - modify line 5
from arcgis.gis import GIS
import datetime as dt
username = 'USERNAME' # MODIFY - enter username for AGOL as str
gis = GIS("https://arcgis.com", username)
# you will be prompted to enter your AGOL password before the remaining cells can run
print('AGOL Connection Established')
# this cell will print what layers were found in your AGOL account - modify lines 3 and 4
download_path = r"\Where\You\Want\Your\Files\To\Download" # MODIFY - define the path where the AGOL layers will be downloaded to
num_items = int(WHOLE NUMBER) # MODIFY - this is the max number of features that will be downloaded - select a number greater than the total number of feature layers being downloaded
query_string = "type:Feature Service, owner:{}".format(username)
items = gis.content.search(query=query_string, max_items=num_items, sort_field='modifed', sort_order='desc')
print(str(len(items)) + " items will be backed up to " + download_path +". See the list below:")
items
#When this cell runs, all feature layers owned by your AGOL acccount will be downloaded to the previously specified path - no modification needed
def download_as_fgdb(item_list, backup_location):
for item in item_list:
try:
if 'View Service' in item.typeKeywords:
print(item.title + " is view, not downloading")
else:
print("Downloading " + item.title)
version = dt.datetime.now().strftime("%d_%b_%Y")
result = item.export(item.title + "_" + version, "File Geodatabase")
result.download(backup_location)
result.delete()
print("Successfully downloaded " + item.title)
except:
print("An error occurred downloading " + item.title)
print("The function has completed")
download_as_fgdb(items, download_path)
#this cell will unpack all of the zipped geodatabases that were downloaded - no modification needed
import os, zipfile
dir_name = download_path # defined to access the directory where your gdbs were downloaded
extension = ".zip"
os.chdir(dir_name)
for item in os.listdir(dir_name): # loop through items in dir
if item.endswith(extension): # check for ".zip" extension
file_name = os.path.abspath(item) # get full path of files
zip_ref = zipfile.ZipFile(file_name) # create zipfile object
zip_ref.extractall(dir_name) # extract file to dir
zip_ref.close() # close file
os.remove(file_name) # delete zipped file
print('All geodatabases unzipped!')
# this cell will create a new back up directory that is time stamped - modify line 4
current_date = datetime.datetime.today().strftime ('%d_%b_%Y')
path = r"\WHERE\YOU\WANT\YOUR\BACK UP\FOLDER\TO\BE\CREATED" + "\\back_up_" + str(current_date) # MODIFY - alter the first string to reflect the folder you want your new back up directory to appear
os.mkdir( path ) # created directory
print('Path created!')
# this cell defines the variable to be used in the following arcpy function - no modification needed
output_gdb = "back_up_" + str(current_date)
print('output_gdb defined!')
# this cell creates a new geodatabase for your features to be copied to - no modification needed
arcpy.management.CreateFileGDB(path, output_gdb, "CURRENT")
# this cell defines variable as the new geodatabase to be used in the following function - no modification needed
feature_copy_path = path + '\\' + output_gdb + '.gdb'
print('New geodatabase defined!')
# this cell will copy all feature classes from the downloaded geodatabases into your newly created geodatabase - no modification needed
import arcpy, os, sys
from arcpy import env
arcpy.env.workspace = download_path
inWorkspace = arcpy.env.workspace
workspaces = arcpy.ListWorkspaces("*", "FileGDB")
for item in workspaces:
print(item)
env.workspace = item
fcs = arcpy.ListFeatureClasses()
for fc in fcs:
arcpy.CopyFeatures_management(
fc, os.path.join(feature_copy_path,
os.path.splitext(fc)[0]))
print('All data deposited into new geodatabase!')
# this cell defines the variable to be used in the following function - no modification needed
shapefile_path = path + '\\shapefiles_' + str(current_date)
print('Shapefile back up directory defined!')
# this cell creates a new folder in the same directory your geodatabase was created in - no modification needed
os.mkdir( shapefile_path)
# this cell converts all of the featureclasses in your new geodatabase to shapefiles in the previously defined folder - no modification needed
arcpy.env.workspace = feature_copy_path
fc_list = arcpy.ListFeatureClasses()
for fc in fc_list:
out_path = shapefile_path + '\\' + fc + ".shp"
arcpy.FeatureClassToShapefile_conversion(fc, shapefile_path)
print('Back up shapefiles created!')
# this cell will remove all layers from the ArcGIS Pro project and then save - no modification required
aprx = arcpy.mp.ArcGISProject("CURRENT")
maps = aprx.listMaps()
for map in maps:
layers = map.listLayers()
for layer in layers:
if layer.isFeatureLayer:
map.removeLayer(layer)
aprx.save()
print('Back up complete and geodatabase folder disconnected!')
# Your feature class back up is complete!
