29

General python optimization techniques can save you substantial amounts of time. One really good technique for getting a lowdown of where the hold ups are in your script is using the built-in cProfile module: from cProfile import run run("code") # replace code with your code or function Testing using a small data sample will allow you to pinpoint which ...


26

A couple potential suggestions to help speed up your process are: Select Layer By Attribute can be in a Python-only script, without ever launching ArcGIS Desktop. You need to convert your "buff" reference from a file-based reference to an "ArcGIS layer" reference, which ArcGIS can process selection queries against. Use arcpy.MakeFeatureLayer_management("...


15

You need to use the beginCluster and endCluster functions of the raster package. See the example below. library(raster) library(snow) # Make test data # RasterStack r <- raster(ncol=36, nrow=18) r[] <- 1:ncell(r) s <- stack(r, sqrt(r), r/r) # SpatialPolygons cds1 <- rbind(c(-180,-20), c(-160,5), c(-60, 0), c(-160,-60), c(-180,-20)) cds2 <- ...


14

Make sure you are writing to internal drive on the computer. Reaching across the network when it is not necessary can really slow the processing. It can even be faster to copy the data as the first step in the process to keep the subsequent read-writes as quick as possible Running the script completely outside of ArcMap can be much faster. If a Map isn't ...


12

First off, yes you will definitely want to make sure your primary and foreign key fields are indexed on both tables. This lets the DBMS plan and execute queries against these fields much more efficiently. Secondly, you are calling SelectLayerByAttribute_management in a tight, nested loop (once per tree per treatment). This is highly inefficient, for several ...


11

Contraction Hierarchy is a very fast algorithm: http://algo2.iti.kit.edu/1087.php This algorithm is RAM friendly while executing a query (to hold a contracted graph some more RAM is necessary as well as massive preprocessing) There are some other algorithms - including the ones that solve public transit routing: http://i11www.iti.uni-karlsruhe.de/members/...


10

The spped of the different approaches will, of course, depend on the complexity of the polygons involved. The algorithms to calculate a centroid, as with ST_Centroid, require you to actually deserialize the polygon as @dbaston has said, and also do the calculation using all the points. ST_Xmax, ST_Xmin and ST_Ymax, ST_Ymin will be faster, as there is less ...


9

If all you are looking for are proximity point searches (nearest neighbour queries), then you don't want to use the old ST_DWithin or ST_Distance + ORDER BYs for that. Not anymore. Now that PostGIS 2.0 shipped, you should be using the knngist index support (a native PostgreSQL feature). It will be orders of magnitude faster. An excerpt from this blog ...


8

With PostGIS 2.0 on PostgreSQL 9.1, you can use the KNN indexed nearest neighbour operator, e.g.: SELECT *, geom <-> ST_MakePoint(-90, 40) AS distance FROM table ORDER BY geom <-> ST_MakePoint(-90, 40) LIMIT 20 OFFSET 0; The above should query within a few milliseconds. For the next multiples of 20, modify to OFFSET 20, OFFSET 40, etc ...


8

In addition to using the new arcpy.da cursor, I would also suggest: You have many different search cursors on the same layer, see if you can eliminate some of those and pull your attributes from one or two Apply an Add Attribute Index on any column that you are querying against See if you can remove the select layer by attribute logic and apply that query ...


7

Filtering (ie. using Layer / Query or the Query Builder in layer properties) in QGIS justs adds a where clause to the query that is executed. So that shouldn't be different to you want to do "in" PostGIS. Although I'm not sure what you intent to do there.


7

Spatial queries are definitely the thing to use. With PostGIS I would first try something simplistic like this and tweak the range as needed: SELECT * FROM table AS a WHERE ST_DWithin (mylocation, a.LatLong, 10000) -- 10km ORDER BY ST_Distance (mylocation, a.LatLong) LIMIT 20 This would compare points (actually their bounding boxes) using the spatial ...


6

This may not answer your question for running ArcPy tools inside ArcMap but when I need to do some meaty processing with geo-processing tools and Python I tend to run it outside the GIS system using the IDE PyScripter. I have found it runs faster. I have also employed a RAMDISK for small temporary output datasets (a bit like the in_memory workspace) Well ...


6

I was having issues with gdal2tiles taking quite a while to process a fairly large (380MB, 39K x 10K pixels) tiff into Google tiles for zoom ranges 0-12. On Ubuntu 12.04 64bit without multiprocessing it took just about all day (8 hours) to process the tiff into 1.99 million tiles @ 3.3GB. Like @Stephan Talpalaru mentions above, making gdal2tiles run in ...


6

QGIS API provides you with a couple of ways for optimizing features requests. In your case, if you don't need geometry and the rest of attributes in the result, you can: Use flag NoGeometry (see docs). Set subset of attributes you really need using setSubsetOfAttributes() (see docs). That should speed your request up.


6

Would you be able to add the result of putting "EXPLAIN ANALYZE" before the query in your question? Then I can update my answer with suggestions. Of course, you will need GIST indexes on your table's geometry fields and vacuum analyze first. CREATE INDEX ON road USING gist (geom); CREATE INDEX ON point USING gist (geom); VACUUM ANALYZE road; VACUUM ANALYZE ...


5

Something must be wrong with your mysql installation or the .ini settings. Just tested a geospatial index on my old mac (10.6.8 / MySQL 5.2). That configuration is similar to yours and I tested the big geodata dump (9 million records). I did this query: SET @radius = 30; SET @center = GeomFromText('POINT(51.51359 7.465425)'); SET @r = @radius/69.1; SET @...


5

This can be done in GIMP http://www.gimp.org/ You need to create a custom colour palette - this will read each image loaded and best match to the pixels forcing the colours to become a near as match. You can also restrict contrast and brightness in a similar way http://gimp.open-source-solution.org/manual/gimp-tool-brightness-contrast.html http://docs....


5

Try commenting out arcpy.SetProgressorLabel and see how much you speed up. I've found that any screen output, going back to DOS daze, drastically slows processing times. If you really need to see that output, trying showing it every Nth loop.


5

The above solution works great for me and was very quick. Using the above code and referenced code from the other post this is how I built it: # Local Variables OriginTable = "This must be a Table View or Feature Layer" DestinationTable = "This must be a Table View or Feature Layer" PrimaryKeyField = "Matching Origin Table Field" ForiegnKeyField = "Matching ...


5

Don't use the distance operation unless you actually need the distance. You can use the ST_DWithin to get geometries within a certain distance. Right now I don't have a PostGres database to test and give you a SQL query for your data, but have a look at the sample query given on the documentation page


5

I was wondering the same thing the other day and found an interesting article experimentally comparing different classification methods. The experiment is about showing the same data with 7 different classification methods, asking questions about the data, and seeing how accurate the answers are for each classification. The particular experiment suggests ...


5

You can use the GeodeticCalculator which should be faster. Something like: package com.envitia.spike; import org.geotools.geometry.DirectPosition2D; import org.geotools.referencing.CRS; import org.geotools.referencing.GeodeticCalculator; import org.opengis.referencing.FactoryException; import org.opengis.referencing.NoSuchAuthorityCodeException; import org....


5

There are many approaches to parallelization. Some GRASS GIS modules are parallelized internally using OpenMP or pthreads when GRASS GIS is compiled in the way that these are supported. This applies to modules written in C, the parallelized modules written in Python are using different Python ways for parallelization based on processes. These modules usually ...


5

After a long time of wondering this myself, I looked into it: When first imported 'arcpy' needs to do some black-box validation stuff that checks if your license is active among other things (like loading DLLs). This process takes the longest (I timed ~3.4 seconds using instructions from this blog). This long process can be emulated by the following code: ...


5

You could try looking int this OS option, Skeletron, it: generalizes collections of lines to a specific spherical mercator zoom level and pixel precision, using a polygon buffer and voronoi diagram It is based off of a 1996 paper by Alnoor Ladak and Roberto B. Martinez, "Automated Derivation of High Accuracy Road Centrelines Thiessen Polygons ...


4

Make sure that you remove any import xxxx lines that aren't being used. (ie. if you're not using any mathematical functions yet you have import Math, this will take some time from the script loading) Although this will not have a great impact on single scripts which run (such as yours), it will effect any scripts that run frequently and repetitively.


4

Here are some links: http://www.geofabrik.de/media/2010-07-10-rendering-toolchain-performance.pdf http://www.geofabrik.de/media/2012-09-08-osm2pgsql-performance.pdf another point: split data to more tables: the less objects to filter, the faster the rendering


4

This is TSP. You just haven't defined a valid distance metric because it does not satisfy the triangle inequality: if there is a route from A to C through B which is shorter than the stated distance from A to C, then the stated distance from A to C is, quite simply, wrong. The solution is to update the distance matrix by setting the length from A to C to ...


4

Very nice problem indeed! I good approach might be to build an explicit cost function returning a cost estimation for a given construction schedule - the 'best' construction schedule might then be obtained using optimisation techniques to minimise this cost function. What you might need are: Criteria to build the cost function. You should list what has an ...


Only top voted, non community-wiki answers of a minimum length are eligible