2

This project came out of my need to build a justification for a faster work computer. Before asking for new equipment I want to be prepared to show the increase in performance by adding RAM or more processors. In order to achieve this I wrote a python script and tested it on different machines in my office. This is the process and results.

Data

Streets - 1,805,375 records

Streets_table - 1,808,798 records

Code

import platform, timeit, arcpy, time, csv, os, io

#get machine specs
name = platform.node()
processor = platform.machine()
oper_sys = platform.platform()
py_ver = platform.python_version()

#test loops
x = 0
start = timeit.timeit()
while x<=10000:
    foo = 1
    bar = 2
    foo+bar
    x=x+1
end = timeit.timeit()
tl = str(end-start)+'seconds to complete test loops'

#test make feature
arcpy.env.workspace = r'C:\GIS\PyTesting\data.gdb'
real_start = time.clock()
comp_start = timeit.timeit()
x=0
while x<=100:
    arcpy.management.MakeFeatureLayer('Streets','Streets_lyr')
    arcpy.management.Delete('Streets_lyr','Layer')
    x=x+1
comp_end = timeit.timeit()
real_end = time.clock()
comp_time = str(comp_end - comp_start)
real_time = (str(real_end - real_start))+' seconds'
mf_real = ('make feature real time: '+real_time)
mf_comp = 'make feature computer time: '+comp_time

#test join and calculate
arcpy.env.workspace = r'C:\GIS\PyTesting\data.gdb'
real_start = time.clock()
comp_start = timeit.timeit()
arcpy.management.MakeFeatureLayer('Streets','Streets_lyr')
arcpy.management.MakeTableView('Streets_County','district_tbl')
arcpy.management.AddJoin('Streets_lyr','ID','district_tbl','ID')
arcpy.CalculateField_management(in_table="Streets_lyr", field="Streets.COUNTY", expression="!Streets_County.COUNTY!", expression_type="PYTHON", code_block="")
comp_end = timeit.timeit()
real_end = time.clock()
comp_time = str(comp_end - comp_start)
real_time = str(real_end - real_start)

jac_real = 'join and calculate real time: '+real_time
jac_comp = 'join and calculate computer time: '+comp_time

#test dissolve
real_start = time.clock()
comp_start = timeit.timeit()
arcpy.management.MakeFeatureLayer('Streets','Streets_lyr_C3', "COUNTY = '03'")
arcpy.Buffer_analysis(in_features="Streets_lyr_C3", out_feature_class="Streets_C3_buffer", buffer_distance_or_field="100 Feet", line_side="FULL", line_end_type="ROUND", dissolve_option="NONE", dissolve_field="", method="PLANAR")
arcpy.management.Dissolve('Streets_d3_buffer','Streets_C3_buffer_diss','ST_NAME')
comp_end = timeit.timeit()
real_end = time.clock()
comp_time = str(comp_end - comp_start)
real_time = str(real_end - real_start)

dis_real = 'buffer and dissolve real time: '+real_time
dis_comp = 'buffer and dissolve computer time: '+comp_time

#write results to network file
location = r'c:\GIS\Pytesting'
note_name = name +'_'+ time.strftime('%m_%d_%y')+'.txt'
note = os.path.join(location,note_name)
f = io.open(note, 'w', encoding = 'utf-8')
f.write(u'Computer Name: '+name)
f.write(u'\r\n')
f.write(u'Processor: '+processor)
f.write(u'\r\n')
f.write('Operating System: '+oper_sys)
f.write(u'\r\n')
f.write('Python Version: '+py_ver)
f.write(u'\r\n')
f.write(time.strftime('%c'))
f.write(u'\r\n')
f.write(tl)
f.write(u'\r\n')
f.write(mf_real+u'\r\n'+mf_comp)
f.write(u'\r\n')
f.write(jac_real+u'\r\n'+jac_comp)
f.write(u'\r\n')
f.write(dis_real+u'\r\n'+dis_comp)
f.close()

#reset environment
delete_list = ['Streets_C3_buffer','Streets_c3_buffer_diss','Streets_lyr','Streets_lyr_c3']
for item in delete_list:
    arcpy.management.Delete(item)
arcpy.management.Compact(arcpy.env.workspace)

Results

Computer 1

Specs

Processor: AMD64

Operating System: Windows-7-6.1.7601-SP1

Python Version: 2.7.10

System Manufacturer Dell Inc.

System Model Precision T7610

System Type x64-based PC

Processor #1 Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz, 3501 Mhz, 4 Core(s), 8 Logical Processor(s)

Processor #2 Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz, 3501 Mhz, 4 Core(s), 8 Logical Processor(s)

Installed Physical Memory (RAM) 16.0 GB

Total Physical Memory 15.9 GB

Available Physical Memory 12.7 GB

Total Virtual Memory 31.9 GB

Available Virtual Memory 28.6 GB

Page File Space 15.9 GB

Times

-0.00763508549659 seconds to complete test loops

make feature real time: 38.0314714857 seconds

make feature computer time: -0.000239311276225

join and calculate real time: 549.478392039

join and calculate computer time: 0.0045615779294

buffer and dissolve real time: 589.134321079

buffer and dissolve computer time: -0.00725412306065

Computer #2

Specs

Processor: AMD64

Operating System: Windows-7-6.1.7601-SP1

Python Version: 2.7.8

System Manufacturer Dell Inc.

System Model Latitude E6540

System Type x64-based PC

Processor Intel(R) Core(TM) i5-4310M CPU @ 2.70GHz, 2701 Mhz, 2 Core(s), 4 Logical Processor(s)

Installed Physical Memory (RAM) 8.00 GB

Total Physical Memory 7.91 GB

Available Physical Memory 5.47 GB

Total Virtual Memory 15.8 GB

Available Virtual Memory 12.9 GB

Page File Space 7.91 GB

Times

-0.00420638742934 seconds to complete test loops

make feature real time: 29.6121126168 seconds

make feature computer time: -0.000301813809763

join and calculate real time: 319.010255588

join and calculate computer time: -2.47076796356e-05

buffer and dissolve real time: 438.06708097

buffer and dissolve computer time: -0.0164959872828

Computer #3

Specs

Processor: AMD64

Operating System: Windows-7-6.1.7601-SP1

Python Version: 2.7.8

System Manufacturer Dell Inc.

System Model OptiPlex 990

System Type x64-based PC

Processor Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz, 3101 Mhz, 4 Core(s), 4 Logical Processor(s)

Installed Physical Memory (RAM) 8.00 GB

Total Physical Memory 7.96 GB

Available Physical Memory 5.18 GB

Total Virtual Memory 15.9 GB

Available Virtual Memory 12.8 GB

Page File Space 7.96 GB

Times

-0.0124842039677 seconds to complete test loops

make feature real time: 49.5458052658 seconds

make feature computer time: 0.000226449800154

join and calculate real time: 616.387228165

join and calculate computer time: 0.000223139130561

buffer and dissolve real time: 894.755151485

buffer and dissolve computer time: 0.0110589607079

Question

Why is Computer #2 the fastest when it has the fewest number of processors and far less memory than Computer #1?

closed as unclear what you're asking by PolyGeo Aug 12 '16 at 6:23

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • I voted to close this as too broad because it is asking three questions when the tour says that there should be just one question asked per question. I recommend that you remove the first sentence that starts "I'm looking for some input and collaboration" and focus on your third question - you may well find answers to your others as "by products" of answers to the third. – PolyGeo Jan 13 '16 at 23:30
  • 1
    Timing information beyond a tenth of a millisecond is random noise. Benchmarking is far more accurate when a thousand things are done several times, and the mean of those passes is reported. Random noise could account multi-millisecond variance when a single thing is done once, rendering the differences moot. I/O benchmarks must "prime the pump" by reading files before measuring access, lest subsequent runs be influenced by file I/O cache. – Vince Jan 14 '16 at 2:01
  • 1
    The only GIS-ish test you have is effectively an I/O benchmark, but nowhere is the disk device mentioned. Disk is probably the most important piece of similarly configured GIS platforms. – Vince Jan 14 '16 at 2:55
  • Any suggestions on a more GIS-ish test to perform? I chose these because they're workflows that time intensive and done with some frequency. Edited post based on PolyGeo comments. – fathom analytics Jan 14 '16 at 12:42
  • 1
    As Vince points out, you really need to run a given workflow N# of times and then average the results. Running 1 time and comparing against other single run is ripe with errors. N being hundreds, if not 1000s of executions. Plus, you Calc example, its just re-calcing the same thing on an already calc'd dataset. Not sure if thats a great test. On top of making sure you've isolated as many variables as possible (what other software is running on the machine, anti-virus? That could have a marginal impact). These are just some of the things to watch for. – KHibma Jan 14 '16 at 15:04