47

I am using an example with 1 million randomly generated points inside of a filegeodatabase. Attached here. Here is some code to get us started: import time import arcpy arcpy.env.workspace = "C:\CountTest.gdb" time.sleep(5) # Let the cpu/ram calm before proceeding! """Method 1""" StartTime = time.clock() with arcpy.da.SearchCursor("RandomPoints", ["...


31

Assuming the given bounding box limits are in the same spatial reference system as the stored coordinates, and you know which spatial operator (intersects or contained by) you need: SELECT * FROM my_table WHERE coordinates && -- intersects, gets more rows -- CHOOSE ONLY THE @ -- contained by, gets fewer rows -- ONE YOU NEED! ...


27

Ill quote some references from Dave Peters System Design Strategies wiki, which is recommended for a more thorough read to understand the complexity of answering this question. I would also recommend checking the relevant version of web-help on tuning services. I think this is actually a really good question, albeit a little vague, as it is something that ...


25

One of the developers of arcpy.da here. We got the performance where it is because performance was our primary concern: the main gripe with the old cursors were that they were slow, not that they lacked any particular functionality. The code uses the same underlying ArcObjects available in ArcGIS since 8.x (the CPython implementation of the search cursor, ...


25

I have finally gotten around to improving this function. I found that for my purposes, it was fastest to rasterize() the polygon first and use getValues() instead of extract(). The rasterizing isn't much faster than the original code for tabulating raster values in small polygons, but it shines when it came to large polygon areas that had large rasters to be ...


22

My thoughts are: Export your shapefile to a file geodatabase feature class - I think its drawing performance will be better but am not sure by how much If you are using ArcGIS Desktop 10.0 or later move it into a Basemap Layer - this will improve drawing performance dramatically If you like the sound of pyramids for vector data, be sure to vote for this ...


20

If you need to create a second cursor for parcels.shp, do so outside of the loop for your first cursor. As it stands, your script is creating a new cursor object for each row in malls.shp which is what's costing you all that processing time. ... rows = arcpy.UpdateCursor('malls.shp',"","",'ParcelID') polyrows = arcpy.SearchCursor('parcels.shp') for row in ...


19

A couple of general tricks I have found useful in the past in this situation: Run your Python script as stand-alone (e.g. from IDLE, PyWin, Eclipse or preferably CMD) to remove the overhead of ArcMap. Spawning subprocesses is an old trick to solving ArcGIS memory leaks even if you don't want to parallize a process. It works because the memory is released ...


16

The most efficient index for the query expressed in your question is the one on gid as it is the only column that appears in a where expression: CREATE INDEX table_gid ON table (gid); You can safely drop the gist index as it will only consume space and slow inserts/updates/deletes down. Long explanation As I said the most effective index in your case is ...


16

Esri has released ArcGIS Pro, which makes use of the GPU for rendering and some processing: Graphics adapter resources In ArcGIS Pro, the graphics engine limits drawing based on the abilities of your graphics processing unit (GPU). GPU processing with Spatial Analyst Spatial Analyst now offers enhanced performance with the use of Graphics ...


14

PostGIS. Geoserver documentation has the following comment: "Shapefiles are a very common format for geospatial data. But if you are running GeoServer in a production environment, it is better to use a spatial database such as PostGIS. This is essential if doing transactions (WFS-T). Most spatial databases provide shapefile conversion tools. Although there ...


14

I want to take the chance of promoting OGR's virtual file system that writes geometries to a in-memory dataset. Using it is simple as @Luke demonstrated in this post drv = ogr.GetDriverByName( 'ESRI Shapefile' ) ds = drv.CreateDataSource(r'/vsimem/virtual.shp') This works just great. Creating a point shape file with ~300.000 geometries and two attribute ...


13

As unicoletti said, the gist index in the geometry column would only work if you use ST_Contains() in the WHERE expression. For instance, if you would like to know all polygons that contain one another, you could use something like this: SELECT a.gid, b.gid FROM table AS a, table as b WHERE a.gid != b.gid and ST_Contains(a.way, b.way) In this case, ...


13

What if you fed the points into a numpy array and used a scipy cKDTree to look for neighbors. I process LiDAR point clouds with large numbers of points (> 20 million) in several MINUTES using this technique. There is documentation here for kdtree and here for numpy conversion. Basically, you read the x,y into an array, and iterate over each point in the ...


13

I am not so sure that this is a CPU-bound task. I'd think it would be an I/O-bound operation, so I'd be looking to use the fastest disk to which I had access. If E: is a network drive, then eliminating that would be the first step. If it isn't a high performance disk (<7ms seek), then that would be second. You may achieve some benefit from copying the ...


13

Let me guess: Your cpu has 4 cores, so 25% cpu usage, is 100% usage of one core, and 3 idle cores. So only solution is to make the code multi threaded, but that is no simple task.


12

This is an exciting question! How big is the raster you want to query? WKTRaster is stored in the database as a BLOB. In order to find the value at a specific point, from a known (x_0, y_0) corner coordinate row/column indices (i, j) are computed using (dx, dy) steps and rotation. With (i, j) known, the ST_Value() function can access the actual data at the ...


11

You need to use a spatial index. Without an spatial index, you must iterate through all the geometries. With a bounding spatial index, you iterate only through the geometries which have a chance to intersect the other geometries. Popular bounding spatial indexes in Python: R-tree index (Python modules Rtree or pyrtree) Quadtree index (Quadtree module) ...


11

I used to work in that exact same environment (the exact same one!). I have not done any benchmark testing but my sense of this is that number of layers in the project doesn't have much effect by itself. In my experience the labeling and number of features is a much bigger issue than the number of layers (especially if many are turned off). I used to have ...


11

I have to keep posting that I will never use embedded search cursors ever for relating tables to each other. Only dictionaries should be used to do what you are doing. Performance will increase 100 fold over a 3 level set of embedded search cursors, since dictionaries use random access from data stored in memory which is immediate, verses cursor's and sql ...


11

It seems to be a bug in QGIS that has existed for some time. I've tested versions 2.18, 3.4 and 3.10. When using a bigint as a primary key, QGIS will cast the primary key to a text field. This causes the index on the primary key to be skipped causing a huge slow down in large tables. This does not happen with if the primary key is of type: Int serial ...


10

The problem with @Jason's answer (and your original approach) is that it does not take advantage of the spatial index and requires a nested, two-cursor loop which is going to become exponentially slower as the number of points increases. An alternative workflow that may be faster while still letting you update the point feature class in-place (Spatial Join ...


10

Load the data into SpatiaLite in QGIS... best way is to create a new SpatiaLite database via the right-click GUI in the browser window, then simply drag and drop your shapefile onto the SpatiaLite database you just created. From there you have all the power of SpatiaLite at your disposal, including SQL and SQL Spatial functions, which are a way more ...


10

You should use an SQL wrapper instead. plpgsql wrappers tend to be slower since they are not inlined and you are forcing an intermediary step too which can get slow the bigger the geometry. So write your function as follows: CREATE OR REPLACE FUNCTION expand(geom geometry,value double precision) RETURNS geometry AS $$ SELECT st_expand(geom,value); $$ ...


9

FGDB_BULK_LOAD is not a compilation setting, it is a configuration option for the command line tools (can also be done programmatically). ogr2ogr --config FGDB_BULK_LOAD YES -f "FileGDB" MyFileGDB.gdb myKML.kml Would create a filegdb and load the KML vector data to it. Let me know if your performance still sucks. By the way, what platform are you on? ...


9

See this help document for creating multiple connections. This simple code will create multiple connections: EXEC SQL CONNECT TO testdb1 AS con1 USER testuser; EXEC SQL CONNECT TO testdb2 AS con2 USER testuser; EXEC SQL CONNECT TO testdb3 AS con3 USER testuser; You can then choose a connection to use: EXEC SQL AT connection-name SELECT ......


9

I agree with afalciano. You can create a model that combines the two tools, that could look like this: It would give you a interface that offers both selection types: DonĀ“t forget to set the Selection type of the second selection to "SUBSET_SELECTION" This model could then be called from a python script importing the toolbox using arcpy.ImportToolbox() ...


9

If you want to show your geometries as vectors instead of images there are a couple of tricks that you can apply to reduce the load of your page: Use TopoJSON instead of GeoJSON Remove all the attributes that you are not going to use in the applicaation and also the whitspaces. Taking into account your visualization scale, simplify your geometries and ...


9

Turn off snapping (Snapping toolbar, uncheck "Use Snapping"). I've had this problem before when there are many vector layers in a project, the cursor is getting bogged down looking for a vertex (or edge, or whatever) to snap to. You could also try copying the image to be georeferenced to a new, empty ArcGIS project along with the bare minimum of vector ...


9

I would first check out Best Practices Using Citrix XenApp and ArcGIS, a guide put together by ESRI. For a previous client, I went through quite a bit of performance troubleshooting with ESRI and our Citrix environment. Below are the highlights from those conversations: I'm assuming you are going to be making edits in a tight area (zoomed in pretty close)....


Only top voted, non community-wiki answers of a minimum length are eligible