I am using an example with 1 million randomly generated points inside of a filegeodatabase. Attached here.
Here is some code to get us started:
arcpy.env.workspace = "C:\CountTest.gdb"
time.sleep(5) # Let the cpu/ram calm before proceeding!
StartTime = time.clock()
with arcpy.da.SearchCursor("RandomPoints", ["...
Assuming the given bounding box limits are in the same spatial reference system as the stored coordinates, and you know which spatial operator (intersects or contained by) you need:
&& -- intersects, gets more rows -- CHOOSE ONLY THE
@ -- contained by, gets fewer rows -- ONE YOU NEED!
Ill quote some references from Dave Peters System Design Strategies wiki, which is recommended for a more thorough read to understand the complexity of answering this question. I would also recommend checking the relevant version of web-help on tuning services.
I think this is actually a really good question, albeit a little vague, as it is something that ...
One of the developers of arcpy.da here. We got the performance where it is because performance was our primary concern: the main gripe with the old cursors were that they were slow, not that they lacked any particular functionality. The code uses the same underlying ArcObjects available in ArcGIS since 8.x (the CPython implementation of the search cursor, ...
I have finally gotten around to improving this function. I found that for my purposes, it was fastest to rasterize() the polygon first and use getValues() instead of extract(). The rasterizing isn't much faster than the original code for tabulating raster values in small polygons, but it shines when it came to large polygon areas that had large rasters to be ...
My thoughts are:
Export your shapefile to a file geodatabase feature class - I think its drawing performance will be better but am not sure by how much
If you are using ArcGIS Desktop 10.0 or later move it into a Basemap Layer - this will improve drawing performance dramatically
If you like the sound of pyramids for vector data, be sure to vote for this ...
If you need to create a second cursor for parcels.shp, do so outside of the loop for your first cursor. As it stands, your script is creating a new cursor object for each row in malls.shp which is what's costing you all that processing time.
rows = arcpy.UpdateCursor('malls.shp',"","",'ParcelID')
polyrows = arcpy.SearchCursor('parcels.shp')
for row in ...
A couple of general tricks I have found useful in the past in this situation:
Run your Python script as stand-alone (e.g. from IDLE, PyWin, Eclipse or preferably CMD) to remove the overhead of ArcMap.
Spawning subprocesses is an old trick to solving ArcGIS memory leaks even if you don't want to parallize a process. It works because the memory is released ...
The most efficient index for the query expressed in your question is the one on gid as it is the only column that appears in a where expression:
CREATE INDEX table_gid ON table (gid);
You can safely drop the gist index as it will only consume space and slow inserts/updates/deletes down.
As I said the most effective index in your case is ...
Esri has released ArcGIS Pro, which makes use of the GPU for rendering and some processing:
Graphics adapter resources
In ArcGIS Pro, the graphics engine limits drawing based on the
abilities of your graphics processing unit (GPU).
GPU processing with Spatial Analyst
Spatial Analyst now offers enhanced performance with the use of
PostGIS. Geoserver documentation has the following comment:
"Shapefiles are a very common format for geospatial data. But if you are running GeoServer in a production environment, it is better to use a spatial database such as PostGIS. This is essential if doing transactions (WFS-T). Most spatial databases provide shapefile conversion tools. Although there ...
I want to take the chance of promoting OGR's virtual file system that writes geometries to a in-memory dataset.
Using it is simple as @Luke demonstrated in this post
drv = ogr.GetDriverByName( 'ESRI Shapefile' )
ds = drv.CreateDataSource(r'/vsimem/virtual.shp')
This works just great. Creating a point shape file with ~300.000 geometries and two attribute ...
As unicoletti said, the gist index in the geometry column would only work if you use ST_Contains() in the WHERE expression.
For instance, if you would like to know all polygons that contain one another, you could use something like this:
SELECT a.gid, b.gid
FROM table AS a, table as b
WHERE a.gid != b.gid and ST_Contains(a.way, b.way)
In this case, ...
What if you fed the points into a numpy array and used a scipy cKDTree to look for neighbors. I process LiDAR point clouds with large numbers of points (> 20 million) in several MINUTES using this technique. There is documentation here for kdtree and here for numpy conversion. Basically, you read the x,y into an array, and iterate over each point in the ...
I am not so sure that this is a CPU-bound task. I'd think it would be an I/O-bound operation, so I'd be looking to use the fastest disk to which I had access.
If E: is a network drive, then eliminating that would be the first step. If it isn't a high performance disk (<7ms seek), then that would be second. You may achieve some benefit from copying the ...
I'm not that familiar with QGIS, but I wonder how it compares with ArcGIS in terms of extensibility. Unfortunately there seems to be at least some tradeoffs between extensibility and performance. The best way I've found to get a feel for ArcGIS extensibility is to take a look at Esri's COM component categories found in the registry.
Each category ...
This is an exciting question! How big is the raster you want to query? WKTRaster is stored in the database as a BLOB. In order to find the value at a specific point, from a known (x_0, y_0) corner coordinate row/column indices (i, j) are computed using (dx, dy) steps and rotation. With (i, j) known, the ST_Value() function can access the actual data at the ...
You need to use a spatial index. Without an spatial index, you must iterate through all the geometries. With a bounding spatial index, you iterate only through the geometries which have a chance to intersect the other geometries.
Popular bounding spatial indexes in Python:
R-tree index (Python modules Rtree or pyrtree)
Quadtree index (Quadtree module)
I used to work in that exact same environment (the exact same one!). I have not done any benchmark testing but my sense of this is that number of layers in the project doesn't have much effect by itself.
In my experience the labeling and number of features is a much bigger issue than the number of layers (especially if many are turned off). I used to have ...
I have to keep posting that I will never use embedded search cursors ever for relating tables to each other. Only dictionaries should be used to do what you are doing. Performance will increase 100 fold over a 3 level set of embedded search cursors, since dictionaries use random access from data stored in memory which is immediate, verses cursor's and sql ...
It seems to be a bug in QGIS that has existed for some time. I've tested versions 2.18, 3.4 and 3.10.
When using a bigint as a primary key, QGIS will cast the primary key to a text field. This causes the index on the primary key to be skipped causing a huge slow down in large tables.
This does not happen with if the primary key is of type:
ArcGIS seems very bloated. I remember a huge performance hit when migrating from Arcview 3.2 to ArcGIS 8.0, and in a lot of places it still exists. At that time I thought it had a lot to do with ESRI migrating earlier Arc/Info code to Windows and having to cut some corners in performance, but I'm not sure if that's true. I recall seeing some examples on this ...
After contacting the appropriate IT admin, I recieved the following instructions, resulting in load times closer to my expectation: ArcMap now takes ~4s, ArcCatalog ~3s to load.
Install ArcGIS service pack 3 (this did not change boot time on its own)
run "esriremove.reg" to remove changes made by ArcGIS to the registry, the contents of esriremove.reg are:
The problem with @Jason's answer (and your original approach) is that it does not take advantage of the spatial index and requires a nested, two-cursor loop which is going to become exponentially slower as the number of points increases.
An alternative workflow that may be faster while still letting you update the point feature class in-place (Spatial Join ...
Load the data into SpatiaLite in QGIS... best way is to create a new SpatiaLite database via the right-click GUI in the browser window, then simply drag and drop your shapefile onto the SpatiaLite database you just created.
From there you have all the power of SpatiaLite at your disposal, including SQL and SQL Spatial functions, which are a way more ...
You should use an SQL wrapper instead. plpgsql wrappers tend to be slower since they are not inlined and you are forcing an intermediary step too which can get slow the bigger the geometry.
So write your function as follows:
CREATE OR REPLACE FUNCTION expand(geom geometry,value double precision)
RETURNS geometry AS
FGDB_BULK_LOAD is not a compilation setting, it is a configuration option for the command line tools (can also be done programmatically).
ogr2ogr --config FGDB_BULK_LOAD YES -f "FileGDB" MyFileGDB.gdb myKML.kml
Would create a filegdb and load the KML vector data to it. Let me know if your performance still sucks. By the way, what platform are you on?
See this help document for creating multiple connections.
This simple code will create multiple connections:
EXEC SQL CONNECT TO testdb1 AS con1 USER testuser;
EXEC SQL CONNECT TO testdb2 AS con2 USER testuser;
EXEC SQL CONNECT TO testdb3 AS con3 USER testuser;
You can then choose a connection to use:
EXEC SQL AT connection-name SELECT ......
I agree with afalciano.
You can create a model that combines the two tools, that could look like this:
It would give you a interface that offers both selection types:
Don´t forget to set the Selection type of the second selection to "SUBSET_SELECTION"
This model could then be called from a python script importing the toolbox using arcpy.ImportToolbox() ...