Hot answers tagged

26

General python optimization techniques can save you substantial amounts of time. One really good technique for getting a lowdown of where the hold ups are in your script is using the built-in cProfile module: from cProfile import run run("code") # replace code with your code or function Testing using a small data sample will allow you to pinpoint which ...


20

A couple potential suggestions to help speed up your process are: Select Layer By Attribute can be in a Python-only script, without ever launching ArcGIS Desktop. You need to convert your "buff" reference from a file-based reference to an "ArcGIS layer" reference, which ArcGIS can process selection queries against. Use ...


12

Make sure you are writing to internal drive on the computer. Reaching across the network when it is not necessary can really slow the processing. It can even be faster to copy the data as the first step in the process to keep the subsequent read-writes as quick as possible Running the script completely outside of ArcMap can be much faster. If a Map isn't ...


7

Try running the sp_help_spatial_geography_index stored procedure to get details on how your spatial index is being used. You should be able to use something like: declare @ms_at geography = 'POINT (-95.66 30.04)' set @ms_at = @ms_at.STBuffer(1000).STAsText() exec sp_help_spatial_geography_index 'lidar', 'SPATIAL_lidar', 0, @ms_at; Post the results in ...


7

If all you are looking for are proximity point searches (nearest neighbour queries), then you don't want to use the old ST_DWithin or ST_Distance + ORDER BYs for that. Not anymore. Now that PostGIS 2.0 shipped, you should be using the knngist index support (a native PostgreSQL feature). It will be orders of magnitude faster. An excerpt from this blog ...


6

Spatial queries are definitely the thing to use. With PostGIS I would first try something simplistic like this and tweak the range as needed: SELECT * FROM table AS a WHERE ST_DWithin (mylocation, a.LatLong, 10000) -- 10km ORDER BY ST_Distance (mylocation, a.LatLong) LIMIT 20 This would compare points (actually their bounding boxes) using the spatial ...


5

Try commenting out arcpy.SetProgressorLabel and see how much you speed up. I've found that any screen output, going back to DOS daze, drastically slows processing times. If you really need to see that output, trying showing it every Nth loop.


5

This may not answer your question for running ArcPy tools inside ArcMap but when I need to do some meaty processing with geo-processing tools and Python I tend to run it outside the GIS system using the IDE PyScripter. I have found it runs faster. I have also employed a RAMDISK for small temporary output datasets (a bit like the in_memory workspace) Well ...


5

With PostGIS 2.0 on PostgreSQL 9.1, you can use the KNN indexed nearest neighbour operator, e.g.: SELECT *, geom <-> ST_MakePoint(-90, 40) AS distance FROM table ORDER BY geom <-> ST_MakePoint(-90, 40) LIMIT 20 OFFSET 0; The above should query within a few milliseconds. For the next multiples of 20, modify to OFFSET 20, OFFSET 40, etc ...


4

Something must be wrong with your mysql installation or the .ini settings. Just tested a geospatial index on my old mac (10.6.8 / MySQL 5.2). That configuration is similar to yours and I tested the big geodata dump (9 million records). I did this query: SET @radius = 30; SET @center = GeomFromText('POINT(51.51359 7.465425)'); SET @r = @radius/69.1; SET ...


4

MySQL, like PostGIS, stores it’s spatial index data in an R-tree structure so it can find stuff fast. An R-tree, like a B-tree, is organized in such a manner that it is optimized for retrieving only a small fraction of the total data in the table. It is actually faster to ignore the index for queries that need to read a large section of ...


4

Make sure that you remove any import xxxx lines that aren't being used. (ie. if you're not using any mathematical functions yet you have import Math, this will take some time from the script loading) Although this will not have a great impact on single scripts which run (such as yours), it will effect any scripts that run frequently and repetitively.


4

Very nice problem indeed! I good approach might be to build an explicit cost function returning a cost estimation for a given construction schedule - the 'best' construction schedule might then be obtained using optimisation techniques to minimise this cost function. What you might need are: Criteria to build the cost function. You should list what has an ...


3

There are many approaches to parallelization. Some GRASS GIS modules are parallelized internally using OpenMP or pthreads when GRASS GIS is compiled in the way that these are supported. This applies to modules written in C, the parallelized modules written in Python are using different Python ways for parallelization based on processes. These modules usually ...


3

GRASS hasn't been fully optimised with running in parallel as it doesn't lock files which are being processed. From the GRASS wiki: GRASS doesn't perform any locking on the files within a GRASS database, so the user may end up with one process reading a file while another process is in the middle of writing it. The most problematic case is the WIND file, ...


2

The objective is to find a point which minimizes the distance; which mathematically it means that your first derivation of the distance equations needs to be equal with zero. The order of the equations follow: In the above equation x and y is the coordinates you are looking for, and with the subscript are the coordinates of your known location. My ...


2

Having looked at your code where is row[] being created, I guess we are not seeing the full picture? I've seen this "GPL0" error before. It's indicating that the first parameter is invalid. If you look at the syntax section of the help for this tool it states the input features are of Data Type FeatureLayer. As we are not seeing the full code we have no ...


1

My suggestion is based on a method I apply to group smaller subcatchments into larger. There are no streams, thus 1st step to compute Euclidean minimum spanning tree: I guess it is some sort of optimisation already. Pick your sink and create directed graph. Picture below shows "Flow Accumulation" in links, i.e. count of nodes discharging into it: ...


1

This is the travelling salesman problem, although I'm not sure that you care about returning to the same place as you began. The TSP is NP-Hard: for even a small number of destinations to be visited, there is an incredibly large number of possibilities for an algorithm to consider before it can tell you what the optimal path is. I haven't tried with every ...


1

Check out this resource by Isaac Kunen about using a numbers table to optimize nearest neighbor using a spatial index http://blogs.msdn.com/b/isaac/archive/2008/10/23/nearest-neighbors.aspx


1

Have you thought about breaking it up into two 1D columns instead of a single 2D column? The optimizer could be choking on all the similar data and having two columns with greater variety might help. What you might also check is the order in which items are checked. I had a problem in Oracle Spatial where I was search on Last Name and an IN_REGION filter. ...



Only top voted, non community-wiki answers of a minimum length are eligible