0

I'm trying to move a feature class on our network SDE that's made up of polylines as a CSV file to the local machine (or fileGDB but slow speed seems to be the theme regardless of output format)

import arcpy, csv
import datetime
from arcpy import env

def tableToCSV(input_tbl, csv_filepath):
    fld_list = arcpy.ListFields(input_tbl)
    fld_names = [fld.name for fld in fld_list]
    with open(csv_filepath, 'wb') as csv_file:
        writer = csv.writer(csv_file)
        writer.writerow(fld_names)
        with arcpy.da.SearchCursor(input_tbl, fld_names) as cursor:
            for row in cursor:
                writer.writerow(row)
        print csv_filepath + " CREATED"
    csv_file.close()

env.workspace = r"Q:\SDE\Direct Connection to delivery.sde"
desiredFC = arcpy.ListFeatureClasses("","","Topography") 


for fc in desiredFC:     
    if fc == 'Contours':
        print 'Starting to Process/' + fc 
        startTime = datetime.datetime.now()
        out_csv = "C:\Users\CHOK\Desktop\Extract Data Method/" + fc[14:] + '.csv'
        tableToCSV(fc, out_csv)
        endTime = datetime.datetime.now()
        print 'Finished Processing/' + fc
        print 'It took ' + str(endTime - startTime) + ' to process ' + fc 

raw_input('Press Enter to exit')

When I run the above script, I see that transferring 3.6 gb of data from the SDE to a CSV (or file geodatabase using arcpy copy methods) takes more than 3~4 hours to run.

I can see in task manager that the network speed utilised is usually between 0.3 Mbps to 2 Mbps but sometimes the speed jumps to 10~20 Mbps and other feature classes are transferred at around 5~7 Mbps.

Is there something I can try to improve the speed at which SDE stored on the network is copied over to a local drive or is this a lost cause due to the network currently in place?

  • That's probably because your HDD is thrashing writing to CSV from a feature class - open your resource monitor to see how hard your drive is being pushed. I would think a file geodatabase would be better to write to, a shapefile is definitely not an option due to size, you could create a replica perhaps; what is your underlying database for your SDE? What else is running on that server? when was it last restarted? Have you tried running the export on the server instead of on a remote workstation to negate the network speed? – Michael Stimson Apr 3 at 5:23
  • The disk writing is at 1% so I don't think it's a hard drive issue. Attempts to write to a new file gdb also seems to yield similar speeds. Unfortunately don't have authority to the actual SDE (located in different location and manged by different team) – Andy Apr 3 at 5:31
  • That often happens. Have you tried to create replica resources.arcgis.com/en/help/main/10.2/index.html#//…, it should be optimized for this sort of thing but it may also create a version, which you may not have permission to do, otherwise Copy Features resources.arcgis.com/en/help/main/10.2/index.html#//… or Feature Class to Feature Class resources.arcgis.com/en/help/main/10.2/index.html#//… should be faster, if not then it's a server/network issue - contact the team that does administer the database and see what's up – Michael Stimson Apr 3 at 5:37
  • A few years back I needed to use ASCII to export and import several hundred million rows. I used arcpy.da.SearchCursor and csv, and the export to 68Gb took 20 minutes and the import two hours. 4Gb is a drop in that bucket and should only take minutes, which makes me wonder if you're using an IO-limited resource like an RDS instance for the Enterprise geodatabase (there is no longer anything called SDE). – Vince Apr 3 at 11:21
  • I think this is an XY Problem because the root issue is poor performance of the data itself, not the access methodology. If you intersect your contours with a coarse fishnet, then Dissolve or dissolve on the attribute used for rendering, the feature count will be reduced and the spatial index more effective. The fact that the data is in a remote data Center may be a contributing factor, as may fragmentation and uncompressed version tree, but for contours, you have an additional issue. – Vince Apr 3 at 11:54
0

I believe this is happening because you are using cursors for your data fetching operation, I had encountered same behavior in my past. I would suggest you to use plain SQL statements for query. Use this pseudo python code. Once you get result set then you can iterate through result set and save in CSV.

ArcSDESQLExecute.execute(sql_statement)

Now since result set in plain format, you can perform write operation with much faster speed. . ArcSDESQLExecute

  import arcpy

  # Use a connection file to create the connection
  db = r'Database Connections\sDEConnection_pointing_to_your_version.sde'
  egdb_conn = arcpy.ArcSDESQLExecute(db)

  table_name = 'YourFeatureClassName'
  field_name = 'FieldName'

  # don't  select the Shape field until its necessary as Shape stored in BLOB field which take time to process.
  sql = '''
  SELECT * FROM {0}
  '''.format(table_name)

 egdb_return = egdb_conn.execute(sql)
 for i in egdb_return:
   print('{}: {}'.format(*i))
  • A simple database cursor is not going to be version-aware, so your answer should mention this. The Python code is pretty basic, so giving a C# example seems confusing. – Vince Apr 3 at 11:16
  • Agree, I will update my answer. – Bhaskar Singh Apr 3 at 12:10
  • Thanks for the help, I specified my db and have table_name = 'Topography\Contour_Other' and field_name = 'load_date' when I run the rest of the code the sql is shown as '\nSELECT * FROM Topography\\Contour_Other\n' and egdb_return = egdb_conn.execute(sql) returns this error AttributeError: ArcSDESQLExecute: StreamPrepareSQL ArcSDE Extended error -202 [Informix][Informix ODBC Driver][Informix]An illegal character has been found in the statement. – Andy Apr 3 at 21:59
  • Are you storing your user name and passwords in SDE file along with version information. If not then you must have to specify the username and password explicitly. "ArcSDESQLExecute ({server}, {instance}, {database}, {user}, {password})". Also you must be setting "env.workspace". after that you would be executing SQL statement again of your table/feature class. in that case you need to pass feature class or table name only, not full path. – Bhaskar Singh Apr 4 at 7:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.