13

I noticed recently that python is only using one core at a time to work. Is there a way, maybe special builds or commands, to make it use all resources (second core)?

  • 1
    If you are running a batch process for repeated tasks and the script is doing the right thing and utilizes 100% of one core for doing something useful it may not be so dangerous. Start a second process which run the other core also at 100%. Gross effect may be better than with one threaded process. Situation is different if you have only one heavy task to process. – user30184 Sep 10 '14 at 12:07
  • This is already discuss see this thread for more info and solutions gis.stackexchange.com/questions/55048/… – iRfAn Sep 12 '14 at 7:30
  • Would you be able to edit your question to clarify whether it is about Python generally or more specifically ArcPy (as alluded to by your tags), please? If it is the latter can you include some details to demonstrate that this is the case i.e. how did you notice it? – PolyGeo Sep 18 '14 at 0:39
15

You can use subprocesses to take advantage of multiple cores within a Python script, so that several tasks can run in parallel. But you can't split a single task over several cores. See a detailed explanation in this FAQ: Does ArcGIS 10 support multi-core processors and/or 64-bit Operating Systems?

If you're a seasoned Python user, you might be interested in the approach explained in this user presentation from the Dev Summit 2014, called Parallel Geoprocessing Using Python Multiprocessing and Critical Path Methodology, still it won't allow you to use several cores for the same task.

Otherwise you'll have to wait for the release of ArcGIS Pro (fully 64 bits application) later this year to test how multi-threading performs... (or try the Beta right now)

| improve this answer | |
4

As some mentioned, there is no support for multiprocessing in ArcGIS Desktop. When talking about processing GIS datasets in a desktop environment, I am trying to find out whether I can split a large workflow into smaller chunks which will be calculated at the same time loading multiple cores. Almost every case should be investigated individually since GP tools behaviours can differ significantly.

Think what is faster to do when solving a simple math problem. What is the fastest way to count all numbers from 1 to 100?

1) by summing the results one by one and adding the sum to each other incrementally (1+2=3, 3+3=6,6+4=10 and so on). One core is working on this task.

2) split values beforehand into individual chunks and sum the values there first (1 to 30, 31 to 60 and 60 to 100). Three cores will be working at the same time (last step would be to sum three values received).

Since different GP tools are implementations of different algorithms with different big-O notation, you would probably need to approach them differently in terms of submitting multiples processes.

A good starting point would be to learn how multiprocessing library in Python works. I use quite heavily.

I have also noticed that running Python scripts from a command line by using the 64-bit Python usually results in faster running (comparing to the IDE's run, but this might not be the case on your machine). Background geoprocessing was introduced in 10.1, but try to run the Python scripts with 64 bit Python and see how the performance is being affected.

ArcGIS Pro named in another answer is available in beta 5 for download (keep in mind that you would need to be a participant of the Esri Beta Community to submit any bugs and having access to an ArcGIS Online for Organizations account in order to be able to run the Pro).

| improve this answer | |
1

Pypy is a compliant version of python that runs 4-5 times faster than CPython (the "standard" python).

If you're brave enough to build it from source, there is a branch that "can run multiple independent CPU-hungry threads in the same process in parallel." This means that you get the benefits of multi-threading without having to re-write any code.

| improve this answer | |
0

Simple answer is no. Better answer is it depends.

Due to the implementation of CPython (the most commonly used python), it's safe to assume your Python cannot actually take advantage of multithreading. See:

http://en.wikipedia.org/wiki/Global_Interpreter_Lock

https://wiki.python.org/moin/GlobalInterpreterLock

Note that IO is immune to the GIL.

Now you can work about this. As others have pointed out, you can spawn off subprocesses and the multiprocessing library can help you.

| improve this answer | |
  • 1
    I/O is immune, but so is calling most functions implemented in C, including most GP tools and everything in numpy. The GIL is less a limitation in practical GIS environments than one would think. – Jason Scheirer Sep 10 '14 at 22:46
0

I can't address ArcGIS issues, but as far as processing multiple tasks using Python, have you considered a task manager like Celery (celeryproject.org)? This would require that you identify different processing tasks, submit them to a "manager" for distribution, run "workers" that receive tasks from the manager, process them, and report results.

This is non-trivial to implement, but has incredible flexibility, and allows you to take full advantage of processing ability (i.e., use those idle cores).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.