Python only using one core at a time to work?

Question

I noticed recently that python is only using one core at a time to work. Is there a way, maybe special builds or commands, to make it use all resources (second core)?

Answer 1

You can use subprocesses to take advantage of multiple cores within a Python script, so that several tasks can run in parallel. But you can't split a single task over several cores. See a detailed explanation in this FAQ: Does ArcGIS 10 support multi-core processors and/or 64-bit Operating Systems?

If you're a seasoned Python user, you might be interested in the approach explained in this user presentation from the Dev Summit 2014, called Parallel Geoprocessing Using Python Multiprocessing and Critical Path Methodology, still it won't allow you to use several cores for the same task.

Otherwise you'll have to wait for the release of ArcGIS Pro (fully 64 bits application) later this year to test how multi-threading performs... (or try the Beta right now)

Answer 2

As some mentioned, there is no support for multiprocessing in ArcGIS Desktop. When talking about processing GIS datasets in a desktop environment, I am trying to find out whether I can split a large workflow into smaller chunks which will be calculated at the same time loading multiple cores. Almost every case should be investigated individually since GP tools behaviours can differ significantly.

Think what is faster to do when solving a simple math problem. What is the fastest way to count all numbers from 1 to 100?

1) by summing the results one by one and adding the sum to each other incrementally (1+2=3, 3+3=6,6+4=10 and so on). One core is working on this task.

2) split values beforehand into individual chunks and sum the values there first (1 to 30, 31 to 60 and 60 to 100). Three cores will be working at the same time (last step would be to sum three values received).

Since different GP tools are implementations of different algorithms with different big-O notation, you would probably need to approach them differently in terms of submitting multiples processes.

A good starting point would be to learn how multiprocessing library in Python works. I use quite heavily.

I have also noticed that running Python scripts from a command line by using the 64-bit Python usually results in faster running (comparing to the IDE's run, but this might not be the case on your machine). Background geoprocessing was introduced in 10.1, but try to run the Python scripts with 64 bit Python and see how the performance is being affected.

ArcGIS Pro named in another answer is available in beta 5 for download (keep in mind that you would need to be a participant of the Esri Beta Community to submit any bugs and having access to an ArcGIS Online for Organizations account in order to be able to run the Pro).

Answer 3

Pypy is a compliant version of python that runs 4-5 times faster than CPython (the "standard" python).

If you're brave enough to build it from source, there is a branch that "can run multiple independent CPU-hungry threads in the same process in parallel." This means that you get the benefits of multi-threading without having to re-write any code.

Answer 4

Simple answer is no. Better answer is it depends.

Due to the implementation of CPython (the most commonly used python), it's safe to assume your Python cannot actually take advantage of multithreading. See:

http://en.wikipedia.org/wiki/Global_Interpreter_Lock

https://wiki.python.org/moin/GlobalInterpreterLock

Note that IO is immune to the GIL.

Now you can work about this. As others have pointed out, you can spawn off subprocesses and the multiprocessing library can help you.

Answer 5

I can't address ArcGIS issues, but as far as processing multiple tasks using Python, have you considered a task manager like Celery (celeryproject.org)? This would require that you identify different processing tasks, submit them to a "manager" for distribution, run "workers" that receive tasks from the manager, process them, and report results.

This is non-trivial to implement, but has incredible flexibility, and allows you to take full advantage of processing ability (i.e., use those idle cores).