0

What I've Tried: I have used the script in this article before. However it does require more input than I would like, and the feature layers are downloaded as individual FGDB instead of into a single, consolidated FGDB. https://support.esri.com/en/technical-article/000018909

What I Want To Do: I am looking for a way to retrieve these files and deposit them into a single FGDB, as opposed to the many. I would also like to see this modified to limit the amount of input required for the script to run. Ideally, I would prefer for to only have to enter my AGOL password when I run the script!

3
  • 1
    Welcome to GIS SE. As a new user, please take the Tour. We use a Focused question/Best answer model. This seems more like a blog post than a Question. Code reviews are off-charter here (opinion-based).
    – Vince
    Jan 6, 2023 at 16:24
  • To prevent deletion you could rewrite your question as a "real" question (ie. what you want to do) and answer it yourself with your script as a solution (self-answer are fine here and it's better to have a question with an accepted answer so people who have the same need will find the solution more easily). If you really want a code review try to highlight the problem with your current script (like to slow or to many steep, or ...) and ask how to solve it.
    – J.R
    Jan 6, 2023 at 16:55
  • For code review there is the Code Review Stack Exchange.
    – PolyGeo
    Jan 6, 2023 at 19:21

1 Answer 1

1

I currently work as a GIS Coordinator for a small scale utility that uses AGOL. That being said, many of our feature layers are hosted in AGOL and are regularly edited or modified in the AGOL environment. After some research, I didn't really find anything code shared publicly that would do exactly what I wanted. So I decided to take a stab at writing my first program! I made some simply modifications to ESRI's script (https://support.esri.com/en/technical-article/000022524) to reduce the amount of user input required and then wrote new code from there...

My first issue with the ESRI provided script was that it only addressed downloaded each hosted feature layer as an individual FGDB. This isn't really a huge issue, but when you are talking about working with regular back ups - 40+ individual FGDBs becomes cumbersome. So the first addition you will see is a process to unpack each of the individually compressed FGDBs and deposit the individual feature classes into a single new FGDB.

My second addition aims to take the previously deposited feature classes and export them to shapefiles to be included in the same back up directory as the FGDB. This isn't extremely important for most, but helps with my organization as we often share shapefiles with third parties, like engineering groups, when we are working on capital projects.

All-in-all this code does what I intended it to do. It downloads all hosted feature layers from AGOL, and creates shapefiles of each layer. However, this is my first complete program I have written, so I was interested in getting some input from more experienced coders.

For reference, this code was developed in a Jupyter notebook within ArcGIS Pro. I have provided as much markdown as possible to help anyone else who may be interested in using this code to automate their AGOL backups.

# this cell access your AGOL account and prompts you to enter a password to continue - modify line 5

from arcgis.gis import GIS
import datetime as dt
username = 'USERNAME' # MODIFY - enter username for AGOL as str
gis = GIS("https://arcgis.com", username) 

# you will be prompted to enter your AGOL password before the remaining cells can run

print('AGOL Connection Established')

# this cell will print what layers were found in your AGOL account - modify lines 3 and 4

download_path = r"\Where\You\Want\Your\Files\To\Download" # MODIFY - define the path where the AGOL layers will be downloaded to
num_items = int(WHOLE NUMBER) # MODIFY - this is the max number of features that will be downloaded - select a number greater than the total number of feature layers being downloaded
query_string = "type:Feature Service, owner:{}".format(username)
items = gis.content.search(query=query_string, max_items=num_items, sort_field='modifed', sort_order='desc')
print(str(len(items)) + " items will be backed up to " + download_path +". See the list below:")
items

#When this cell runs, all feature layers owned by your AGOL acccount will be downloaded to the previously specified path - no modification needed

def download_as_fgdb(item_list, backup_location): 
    for item in item_list:
        try:
            if 'View Service' in item.typeKeywords:
                print(item.title + " is view, not downloading")
            else: 
                print("Downloading " + item.title)
                version = dt.datetime.now().strftime("%d_%b_%Y")
                result = item.export(item.title + "_" + version, "File Geodatabase")
                result.download(backup_location)
                result.delete()
                print("Successfully downloaded " + item.title)
        except:
            print("An error occurred downloading " + item.title)
    print("The function has completed")

download_as_fgdb(items, download_path)

#this cell will unpack all of the zipped geodatabases that were downloaded - no modification needed

import os, zipfile 

dir_name = download_path # defined to access the directory where your gdbs were downloaded
extension = ".zip"

os.chdir(dir_name) 

for item in os.listdir(dir_name): # loop through items in dir
    if item.endswith(extension): # check for ".zip" extension
        file_name = os.path.abspath(item) # get full path of files
        zip_ref = zipfile.ZipFile(file_name) # create zipfile object
        zip_ref.extractall(dir_name) # extract file to dir
        zip_ref.close() # close file
        os.remove(file_name) # delete zipped file
        
print('All geodatabases unzipped!')

# this cell will create a new back up directory that is time stamped - modify line 4

current_date = datetime.datetime.today().strftime ('%d_%b_%Y') 
path = r"\WHERE\YOU\WANT\YOUR\BACK UP\FOLDER\TO\BE\CREATED" + "\\back_up_" + str(current_date) # MODIFY - alter the first string to reflect the folder you want your new back up directory to appear


os.mkdir( path ) # created directory

print('Path created!')

# this cell defines the variable to be used in the following arcpy function - no modification needed

output_gdb = "back_up_" + str(current_date)
print('output_gdb defined!')

# this cell creates a new geodatabase for your features to be copied to - no modification needed

arcpy.management.CreateFileGDB(path, output_gdb, "CURRENT")

# this cell defines variable as the new geodatabase to be used in the following function - no modification needed

feature_copy_path = path + '\\' + output_gdb + '.gdb'
print('New geodatabase defined!')

# this cell will copy all feature classes from the downloaded geodatabases into your newly created geodatabase - no modification needed

import arcpy, os, sys
from arcpy import env

arcpy.env.workspace = download_path
inWorkspace = arcpy.env.workspace

workspaces = arcpy.ListWorkspaces("*", "FileGDB")
for item in workspaces:
    print(item)
    env.workspace = item
    fcs = arcpy.ListFeatureClasses()
    for fc in fcs:
        arcpy.CopyFeatures_management(
        fc, os.path.join(feature_copy_path,
                         os.path.splitext(fc)[0]))
        
print('All data deposited into new geodatabase!')

# this cell defines the variable to be used in the following function - no modification needed

shapefile_path = path + '\\shapefiles_' + str(current_date)
print('Shapefile back up directory defined!')

# this cell creates a new folder in the same directory your geodatabase was created in - no modification needed

os.mkdir( shapefile_path)

# this cell converts all of the featureclasses in your new geodatabase to shapefiles in the previously defined folder - no modification needed

arcpy.env.workspace = feature_copy_path
fc_list = arcpy.ListFeatureClasses()

for fc in fc_list:
    out_path = shapefile_path + '\\' + fc + ".shp"
    arcpy.FeatureClassToShapefile_conversion(fc, shapefile_path)
    
print('Back up shapefiles created!')

# this cell will remove all layers from the ArcGIS Pro project and then save - no modification required

aprx = arcpy.mp.ArcGISProject("CURRENT")
maps = aprx.listMaps()

for map in maps:
    layers = map.listLayers()
    for layer in layers:
        if layer.isFeatureLayer:
            map.removeLayer(layer)

            
aprx.save()


print('Back up complete and geodatabase folder disconnected!')

# Your feature class back up is complete!
3
  • As a note, if you're looking to schedule this to run after hours or have a similar need for it to run without manually inputting the password each time, you could look at incorporating the techniques documented at developers.arcgis.com/python/guide/… . All ease of use comes with some amount of security trade offs, but, this is a pretty reasonable option if you have the need.
    – John
    Jan 9, 2023 at 15:45
  • For my work application, I want to keep the password input to prevent unauthorized access to the content with the script. But this is definitely something I might implement for my personal stuff! Jan 9, 2023 at 15:57
  • Note: This does not save symbology that may have been associated with the AGOL feature layers. Not sure if this could be improved. But this does produce a viable solution for backing up the data. Jan 9, 2023 at 18:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.