9

My script is intersecting lines with polygons. It's a long process since there are more than 3000 lines and more than 500000 polygons. I executed from PyScripter:

# Import
import arcpy
import time

# Set envvironment
arcpy.env.workspace = r"E:\DensityMaps\DensityMapsTest1.gdb"
arcpy.env.overwriteOutput = True

# Set timer
from datetime import datetime
startTime = datetime.now()

# Set local variables
inFeatures = [r"E:\DensityMaps\DensityMapsTest.gdb\Grid1km_Clip", "JanuaryLines2"]
outFeatures = "JanuaryLinesIntersect"
outType = "LINE"

# Make lines
arcpy.Intersect_analysis(inFeatures, outFeatures, "", "", outType)

#Print end time
print "Finished "+str(datetime.now() - startTime)


My question is: is there a way to make the CPU work at 100%? It's running at 25% all the time. I guess that the script would run faster if the processor was at 100%. Wrong guess?
My machine is:

  • Windows Server 2012 R2 Standard
  • Processor: Intel Xeon CPU E5-2630 0 @2.30 GHz 2.29 GHz
  • Installed memory: 31,6 GB
  • System type: 64-bit Operating System, x64-based processor


enter image description here

  • I would strongly suggest to go for multi-threading. That is non-trivial to set up but will more than compensate for the efforts. – alok jha Sep 15 '15 at 14:55
  • 1
    What sort of spatial index have you applied to your polygons? – Kirk Kuykendall Sep 15 '15 at 15:06
  • 1
    Also, have you tried the same operation with ArcGIS Pro? It's 64 bit and supports multithreaded. I'd be surprised if it's smart enough to break up an Intersect into multiple threads, but worth a try. – Kirk Kuykendall Sep 15 '15 at 15:09
  • The polygon feature class has a spatial index named FDO_Shape. I haven't thought about this. Should I create another one? Isn't this enough? – Manuel Frias Sep 16 '15 at 9:19
  • 1
    Since you've got a lot of RAM ... did you try copying the polygons into an in-memory featureclass and then intersect the lines with that? Or if keeping it on disk, did you try compacting it? Supposedly compacting improves i/o. – Kirk Kuykendall Sep 16 '15 at 13:54
13

Let me guess: Your cpu has 4 cores, so 25% cpu usage, is 100% usage of one core, and 3 idle cores.

So only solution is to make the code multi threaded, but that is no simple task.

  • 4
    The CPU he mentions utilizes 6 cores and 12 threads. – Kersten Sep 15 '15 at 13:27
  • 5
    Hi there, I can't downvote but I'd like to! Python has a GIL unfortunately so you cannot multithread stuff at all (the best you can do is have the GIL unlocked when a thread blocks on a syscall) – Alec Teal Sep 16 '15 at 9:41
  • 2
    @AlecTeal you definitely can, for example with Jython or the multiprocessing module. – rightfold Sep 16 '15 at 20:20
  • @elyse going "Oh yeah, you can totally do that in Python, if by Python you mean Jython" doesn't count. I'd have to look into multiprocessing, would an import have the power to reimplement what makes Python Python? – Alec Teal Sep 17 '15 at 23:16
  • @AlecTeal It spawns processes (which are one way to do parallelism). See the documentation of the multiprocessing module. – rightfold Sep 18 '15 at 7:42
13

I am not so sure that this is a CPU-bound task. I'd think it would be an I/O-bound operation, so I'd be looking to use the fastest disk to which I had access.

If E: is a network drive, then eliminating that would be the first step. If it isn't a high performance disk (<7ms seek), then that would be second. You may achieve some benefit from copying the polygon layer to an in_memory workspace, but the benefit may be dependent on the size of the polygon feature class, and whether you're using 64-bit background processing.

Optimizing I/O throughput is often key to GIS performance, so I'd recommend you pay less attention to the CPU meter and more attention to the network and disk meters.

4

I had similar problems of performance regarding arcpy scripts, the major bottleneck is not CPU it the hard drive, if you are using data from network that's the worst scenario, try to move your data to SSD drive, then launch your script from the command line not from pyscripter , pyscripter is slightly slower may be because it contains some debugging stuff, if you are not satisfied again, think about paralleling your script, because each python thread takes one CPU core, your CPU is having 6 cores, so you can launch 6 scripts simultaneously.

3

As you are using python and as suggested above consider using multiprocessing if your problem can be run in parallel.

I wrote a small article on the geonet website about converting a python script into a python script tool that could be used within modelbuilder. The document lists the code and describes some pitfalls for running it as a script tool. This is just one place to start looking:

https://geonet.esri.com/docs/DOC-3824

  • This seems the way to go! Your script works fine but I don't know how to modify it to make it work with my script. Better, I was thinking of doing a Tabulate Intersection with polygons and lines. Any idea? – Manuel Frias Sep 17 '15 at 13:03
3

As said before you should use multiprocessing or threading. But here comes the caveat: The problem must be divisible! So have a look at https://en.wikipedia.org/wiki/Divide_and_conquer_algorithms.

If your problem is divisible you would proceed like:

  • Create a queue where you store the input data for the processes/thread
  • Create a queue where the results are stored in
  • Create a function or class that can be used as a process/thread that solves our problem

But as geogeek has said, it might not be a CPU limiting problem, but an IO one. If you have enough RAM you can pre-load all the data and then process it, which has the advantage that the data can be read in one go thus does not always interrupt the calculation process.

3

I decided to test it using 21513 lines and 498596 polygons. I tested multiprocessor approach (12 processors on my machine) using this script:

import arcpy,os
import multiprocessing
import time
t0 = time.time()
arcpy.env.overwriteOutput = True
nProcessors=4
folder=r'd:\scratch'

def function(inputs):
        nGroup=inputs[0]
        pGons=inputs[1]
        lines=inputs[2]
        outFeatures = '%s%s%s_%i.shp' %(folder,os.sep,'inters',nGroup)
        fids= tuple([i for i in range(nGroup,500000,nProcessors-1)])
        lyr='layer%s'%nGroup
        query='"FID" in %s' %str(fids)
        arcpy.MakeFeatureLayer_management(pGons,lyr,query)
        arcpy.Intersect_analysis([lines,lyr], outFeatures)
        return outFeatures
if __name__ == "__main__":
        inPgons='%s%s%s' %(folder,os.sep,'parcels.shp')
        inLines='%s%s%s' %(folder,os.sep,'roads.shp')
        m,bList=0,[]
        for i in range(nProcessors):
                bList.append([i,inPgons,inLines])
        pool = multiprocessing.Pool(nProcessors-1)
        listik=pool.map(function, bList)
##      apply merge here
        print listik
        print ('%i seconds' %(time.time()-t0))

Results, seconds:

  • normal local hard drive - 191
  • superfast local drive - 220
  • network drive - 252

The funny thing it took only 87 sec using geoprocessing tool from mxd. Perhaps something wrong with my approach to pool...

As one can see I’ve used rather ugly query FID in (0, 4, 8,12…500000) to make task divisible.

It is possible that query based on pre-calculated field, e.g. CFIELD=0 will reduce time greatly.

I also found that time reported by multiprocessing tools can vary a lot.

  • 1
    Yeah you are using a list, which comes along with locking issues. Try a multiprocessing.queue. Also try not to write out stuff in the worker processes, but create an ouput queue with the data you want to write and let this be done by a writer process. – Benjamin Sep 18 '15 at 12:03
3

I'm not familiar with PyScripter, but if it's backed by CPython, then you should go for multiprocessing and not multi-threading as long as the problem itself is divisible (as others already mentioned it).

CPython has a Global Interpreter Lock, which cancels out any benefits which multiple threads could bring in your case.

For sure in other contexts python threads are useful, but not in cases where you're CPU bound.

1

My question is: is there a way to make the CPU work at 100%

As your CPU has multiple cores you will only max out the core that your process is running on. Depending on how you have your Xeon chip configured it will be running up to 12 cores (6 physical and 6 virtual with hyperthreading on). Even 64bit ArcGIS is not really able to take advantage of this - and that can result in CPU limitations when your single threaded process maxes out the core its running on. You need a multi-threaded application to spread the load across the cores OR (much more simply) you can reduce the number of cores your CPU is running to increase throughput.

The easiest way to stop CPU limitation (and make sure it really is CPU limitation not disk i/o restrictions) is to change the BIOS settings for your Xeon and set it to one massive single core. The performance increase will be substantial. Just remember this also trades off your PCs multi-tasking ability quite considerably so is best if you have dedicated process machine to implement this on. It is much simpler than trying to multi-thread your code - which most ArcGIS Desktop functions (as at 10.3.1) don't support anyway.

  • What setting should you look for to turn your CPU into "one massive single core"? – Alex McVittie Oct 7 '15 at 20:24
  • 1
    Exact menu will depend on your BIOS and chip firmware but it will usually be in the BIOS Menu Setup>Advanced>CPU Configuration. You will want to turn hyper-threading off and then set the number of cores to activate. 0 is usually all - set to 1 if you want one big core. Good idea to take a note of settings before you change things - sounds obvious but easy to overlook should things not work out. – kingmi Oct 12 '15 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.