Hot answers tagged

27

General python optimization techniques can save you substantial amounts of time. One really good technique for getting a lowdown of where the hold ups are in your script is using the built-in cProfile module: from cProfile import run run("code") # replace code with your code or function Testing using a small data sample will allow you to pinpoint which ...


21

A couple potential suggestions to help speed up your process are: Select Layer By Attribute can be in a Python-only script, without ever launching ArcGIS Desktop. You need to convert your "buff" reference from a file-based reference to an "ArcGIS layer" reference, which ArcGIS can process selection queries against. Use arcpy.MakeFeatureLayer_management("...


13

Make sure you are writing to internal drive on the computer. Reaching across the network when it is not necessary can really slow the processing. It can even be faster to copy the data as the first step in the process to keep the subsequent read-writes as quick as possible Running the script completely outside of ArcMap can be much faster. If a Map isn't ...


11

First off, yes you will definitely want to make sure your primary and foreign key fields are indexed on both tables. This lets the DBMS plan and execute queries against these fields much more efficiently. Secondly, you are calling SelectLayerByAttribute_management in a tight, nested loop (once per tree per treatment). This is highly inefficient, for several ...


7

Try running the sp_help_spatial_geography_index stored procedure to get details on how your spatial index is being used. You should be able to use something like: declare @ms_at geography = 'POINT (-95.66 30.04)' set @ms_at = @ms_at.STBuffer(1000).STAsText() exec sp_help_spatial_geography_index 'lidar', 'SPATIAL_lidar', 0, @ms_at; Post the results in ...


7

If all you are looking for are proximity point searches (nearest neighbour queries), then you don't want to use the old ST_DWithin or ST_Distance + ORDER BYs for that. Not anymore. Now that PostGIS 2.0 shipped, you should be using the knngist index support (a native PostgreSQL feature). It will be orders of magnitude faster. An excerpt from this blog ...


6

Spatial queries are definitely the thing to use. With PostGIS I would first try something simplistic like this and tweak the range as needed: SELECT * FROM table AS a WHERE ST_DWithin (mylocation, a.LatLong, 10000) -- 10km ORDER BY ST_Distance (mylocation, a.LatLong) LIMIT 20 This would compare points (actually their bounding boxes) using the spatial ...


6

This may not answer your question for running ArcPy tools inside ArcMap but when I need to do some meaty processing with geo-processing tools and Python I tend to run it outside the GIS system using the IDE PyScripter. I have found it runs faster. I have also employed a RAMDISK for small temporary output datasets (a bit like the in_memory workspace) Well ...


5

Try commenting out arcpy.SetProgressorLabel and see how much you speed up. I've found that any screen output, going back to DOS daze, drastically slows processing times. If you really need to see that output, trying showing it every Nth loop.


5

With PostGIS 2.0 on PostgreSQL 9.1, you can use the KNN indexed nearest neighbour operator, e.g.: SELECT *, geom <-> ST_MakePoint(-90, 40) AS distance FROM table ORDER BY geom <-> ST_MakePoint(-90, 40) LIMIT 20 OFFSET 0; The above should query within a few milliseconds. For the next multiples of 20, modify to OFFSET 20, OFFSET 40, etc ...


5

QGIS API provides you with a couple of ways for optimizing features requests. In your case, if you don't need geometry and the rest of attributes in the result, you can: Use flag NoGeometry (see docs). Set subset of attributes you really need using setSubsetOfAttributes() (see docs). That should speed your request up.


4

After a long time of wondering this myself, I looked into it: When first imported 'arcpy' needs to do some black-box validation stuff that checks if your license is active among other things (like loading DLLs). This process takes the longest (I timed ~3.4 seconds using instructions from this blog). This long process can be emulated by the following code: ...


4

Something must be wrong with your mysql installation or the .ini settings. Just tested a geospatial index on my old mac (10.6.8 / MySQL 5.2). That configuration is similar to yours and I tested the big geodata dump (9 million records). I did this query: SET @radius = 30; SET @center = GeomFromText('POINT(51.51359 7.465425)'); SET @r = @radius/69.1; SET @...


4

MySQL, like PostGIS, stores it’s spatial index data in an R-tree structure so it can find stuff fast. An R-tree, like a B-tree, is organized in such a manner that it is optimized for retrieving only a small fraction of the total data in the table. It is actually faster to ignore the index for queries that need to read a large section of the ...


4

Make sure that you remove any import xxxx lines that aren't being used. (ie. if you're not using any mathematical functions yet you have import Math, this will take some time from the script loading) Although this will not have a great impact on single scripts which run (such as yours), it will effect any scripts that run frequently and repetitively.


4

Very nice problem indeed! I good approach might be to build an explicit cost function returning a cost estimation for a given construction schedule - the 'best' construction schedule might then be obtained using optimisation techniques to minimise this cost function. What you might need are: Criteria to build the cost function. You should list what has an ...


3

The above solution works great for me and was very quick. Using the above code and referenced code from the other post this is how I built it: # Local Variables OriginTable = "This must be a Table View or Feature Layer" DestinationTable = "This must be a Table View or Feature Layer" PrimaryKeyField = "Matching Origin Table Field" ForiegnKeyField = "Matching ...


3

Another suggestion which may also help - make sure you have a suitable index created on your field on the Postgres database, and then switch on the option under Settings -> Data Sources -> "Execute expressions on server-side if possible". That should make a huge difference, as all the filtering will be done on the server rather then sending ALL the features ...


3

There are many approaches to parallelization. Some GRASS GIS modules are parallelized internally using OpenMP or pthreads when GRASS GIS is compiled in the way that these are supported. This applies to modules written in C, the parallelized modules written in Python are using different Python ways for parallelization based on processes. These modules usually ...


3

GRASS hasn't been fully optimised with running in parallel as it doesn't lock files which are being processed. From the GRASS wiki: GRASS doesn't perform any locking on the files within a GRASS database, so the user may end up with one process reading a file while another process is in the middle of writing it. The most problematic case is the WIND file, ...


2

You can use the OSM(OpenStreetMaps) road data, take a look at How to extract primary and secondary roads from OSM data? and also look at this question from the OpenStreetMap Help as well.


2

My suggestion is based on a method I apply to group smaller subcatchments into larger. There are no streams, thus 1st step to compute Euclidean minimum spanning tree: I guess it is some sort of optimisation already. Pick your sink and create directed graph. Picture below shows "Flow Accumulation" in links, i.e. count of nodes discharging into it: ...


2

The objective is to find a point which minimizes the distance; which mathematically it means that your first derivation of the distance equations needs to be equal with zero. The order of the equations follow: In the above equation x and y is the coordinates you are looking for, and with the subscript are the coordinates of your known location. My ...


1

This sounds like an OSM issue to me. You can obtain the data for Saudi Arabia (GCC States) here e.g.: http://download.geofabrik.de/asia/gcc-states.html. A close look at the data will be required, and if it fits for your application depends strongly on the mapping quality, but i did some routing application for germany with the related data and for me it ...


1

This is the travelling salesman problem, although I'm not sure that you care about returning to the same place as you began. The TSP is NP-Hard: for even a small number of destinations to be visited, there is an incredibly large number of possibilities for an algorithm to consider before it can tell you what the optimal path is. I haven't tried with every ...


1

Check out this resource by Isaac Kunen about using a numbers table to optimize nearest neighbor using a spatial index http://blogs.msdn.com/b/isaac/archive/2008/10/23/nearest-neighbors.aspx


1

Have you thought about breaking it up into two 1D columns instead of a single 2D column? The optimizer could be choking on all the similar data and having two columns with greater variety might help. What you might also check is the order in which items are checked. I had a problem in Oracle Spatial where I was search on Last Name and an IN_REGION filter. ...



Only top voted, non community-wiki answers of a minimum length are eligible