3

I have made a Python toolbox which processes relatively large polyline feature class (millions of features), builds a network graph from it as dictionary of lists, and depending on user selected features in map finds shortest paths between selected features and creates a new selection of these shortest paths. This tool is used repeatedly in one of our workflows. I would love to keep the "network graph" as dictionary of lists in memory. I am aware of in_memory workspace and possibility to convert my dictionary into table and store it like that there.

I wonder if there is some way to save directly Python objects into generally accessible space (variable) in memory in ArcGIS Pro from Python toolbox?

Something like in_memory workspace but directly for Python objects (variables). In my particular case so I can check if this dictionary of lists exists and potentially reuse it if object meets the user settings and other requirements, when the tool is rerun?

3

In VB .net if you want to save an object to file to be used later, e.g. a dictionary you would serialize it. I believe the equivalent in python is pickle.

Be aware not all objects can be "pickled", I would imagine especially anything arcpy related (but I may be wrong).

So if you are extracting network topology as dictionaries then you should be able to pickle them and then reload them back into memory from a well known location at a later date.

My experience with serialization is principally in VB. net so I can't comment on the speed of pickling or the reading back into memory but I would imagine it should be quite fast.

0

Just some thoughts about working with graphs in Python. I believe the base ArcGIS Pro and Desktop both come loaded with Numpy and Scipy, so these should be available to everyone. You could consider using a sparse matrix instead of a dictionary of lists. These will use less memory and be faster computing. You can also perform djikstra's shortest path algorithms using Scipy. You can save the sparse matrix and load it as needed. The matrix is held entirely in memory, which may be a problem for you depending on how large the files are.

Creating graphs (also has links to different shortest path algorithms):

https://docs.scipy.org/doc/scipy-0.16.1/reference/sparse.csgraph.html

Basic shortest path algorithm:

https://docs.scipy.org/doc/scipy-0.16.1/reference/sparse.csgraph.html

To save the sparse matrix:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.save_npz.html

  • Thank you, I do use scipy for some parts and implementation of Dijkstra using heapq heap which works really well with dictionary of lists. – Miro Aug 7 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.