56

I have seen this technique used in the past. It was explained to me by Zain Memon (from Trulia) who helped giving some input when Michal Migurski was creating TileStache. Zain went over it while explaining his Trulia demo that uses this technique at one of our older SF GeoMeetup meetings some time back. In fact, if you are in SF next week (this is my lame ...


47

I am using an example with 1 million randomly generated points inside of a filegeodatabase. Attached here. Here is some code to get us started: import time import arcpy arcpy.env.workspace = "C:\CountTest.gdb" time.sleep(5) # Let the cpu/ram calm before proceeding! """Method 1""" StartTime = time.clock() with arcpy.da.SearchCursor("RandomPoints", ["...


27

Ill quote some references from Dave Peters System Design Strategies wiki, which is recommended for a more thorough read to understand the complexity of answering this question. I would also recommend checking the relevant version of web-help on tuning services. I think this is actually a really good question, albeit a little vague, as it is something that ...


25

One of the developers of arcpy.da here. We got the performance where it is because performance was our primary concern: the main gripe with the old cursors were that they were slow, not that they lacked any particular functionality. The code uses the same underlying ArcObjects available in ArcGIS since 8.x (the CPython implementation of the search cursor, ...


24

Assuming the given bounding box limits are in the same spatial reference system as the stored coordinates, and you know which spatial operator (intersects or contained by) you need: SELECT * FROM my_table WHERE coordinates && -- intersects, gets more rows -- CHOOSE ONLY THE @ -- contained by, gets fewer rows -- ONE YOU NEED! ...


23

Direct from the developer Dino Ravnic on a recent mailing list post: It's not a big secret how we did it so I would be happy to share that with you..the key is in two things: removing from a tile all vectors which are to small to be visible i.e. their area when calculated into pixels is less than 1px. so we drop such a vector and instead of ...


23

I have finally gotten around to improving this function. I found that for my purposes, it was fastest to rasterize() the polygon first and use getValues() instead of extract(). The rasterizing isn't much faster than the original code for tabulating raster values in small polygons, but it shines when it came to large polygon areas that had large rasters to be ...


22

My thoughts are: Export your shapefile to a file geodatabase feature class - I think its drawing performance will be better but am not sure by how much If you are using ArcGIS Desktop 10.0 or later move it into a Basemap Layer - this will improve drawing performance dramatically If you like the sound of pyramids for vector data, be sure to vote for this ...


21

Since Postgis is a component of Postgres I would recommend this great book (I own it and I found it extremely valuable) on Postgres performance tuning: http://www.packtpub.com/postgresql-90-high-performance/book It starts from the basics (planning the hardware, os, etc) and then grows into explaining all those misterious configuration params that I never ...


19

A couple of general tricks I have found useful in the past in this situation: Run your Python script as stand-alone (e.g. from IDLE, PyWin, Eclipse or preferably CMD) to remove the overhead of ArcMap. Spawning subprocesses is an old trick to solving ArcGIS memory leaks even if you don't want to parallize a process. It works because the memory is released ...


19

If you need to create a second cursor for parcels.shp, do so outside of the loop for your first cursor. As it stands, your script is creating a new cursor object for each row in malls.shp which is what's costing you all that processing time. ... rows = arcpy.UpdateCursor('malls.shp',"","",'ParcelID') polyrows = arcpy.SearchCursor('parcels.shp') for row in ...


17

As I described on the OSGeo list the key is in delivering data as vector JSON tiles that have pixels for subpixel geometry and generalized geometry for those features that will be actually visible on a certain level. Performance is great because this technique eliminates all unnecessary vector information and leave only those vectors that will actually have ...


16

The most efficient index for the query expressed in your question is the one on gid as it is the only column that appears in a where expression: CREATE INDEX table_gid ON table (gid); You can safely drop the gist index as it will only consume space and slow inserts/updates/deletes down. Long explanation As I said the most effective index in your case is ...


16

Esri has released ArcGIS Pro, which makes use of the GPU for rendering and some processing: Graphics adapter resources In ArcGIS Pro, the graphics engine limits drawing based on the abilities of your graphics processing unit (GPU). GPU processing with Spatial Analyst Spatial Analyst now offers enhanced performance with the use of Graphics ...


14

Looks like a very similar question was recently asked on the OSGeo Open Layers forum, with the GIS Cloud developers describing their approach, which is a interesting mix of GeoJSON geometries and static pixels. They actually generate all vector tiles on the fly instead of using a pre-built cache of GeoJSON files. Esri has implemented a similar approach, ...


14

PostGIS. Geoserver documentation has the following comment: "Shapefiles are a very common format for geospatial data. But if you are running GeoServer in a production environment, it is better to use a spatial database such as PostGIS. This is essential if doing transactions (WFS-T). Most spatial databases provide shapefile conversion tools. Although there ...


14

I want to take the chance of promoting OGR's virtual file system that writes geometries to a in-memory dataset. Using it is simple as @Luke demonstrated in this post drv = ogr.GetDriverByName( 'ESRI Shapefile' ) ds = drv.CreateDataSource(r'/vsimem/virtual.shp') This works just great. Creating a point shape file with ~300.000 geometries and two attribute ...


13

As unicoletti said, the gist index in the geometry column would only work if you use ST_Contains() in the WHERE expression. For instance, if you would like to know all polygons that contain one another, you could use something like this: SELECT a.gid, b.gid FROM table AS a, table as b WHERE a.gid != b.gid and ST_Contains(a.way, b.way) In this case, ...


13

What if you fed the points into a numpy array and used a scipy cKDTree to look for neighbors. I process LiDAR point clouds with large numbers of points (> 20 million) in several MINUTES using this technique. There is documentation here for kdtree and here for numpy conversion. Basically, you read the x,y into an array, and iterate over each point in the ...


13

I am not so sure that this is a CPU-bound task. I'd think it would be an I/O-bound operation, so I'd be looking to use the fastest disk to which I had access. If E: is a network drive, then eliminating that would be the first step. If it isn't a high performance disk (<7ms seek), then that would be second. You may achieve some benefit from copying the ...


13

Let me guess: Your cpu has 4 cores, so 25% cpu usage, is 100% usage of one core, and 3 idle cores. So only solution is to make the code multi threaded, but that is no simple task.


12

I'm not that familiar with QGIS, but I wonder how it compares with ArcGIS in terms of extensibility. Unfortunately there seems to be at least some tradeoffs between extensibility and performance. The best way I've found to get a feel for ArcGIS extensibility is to take a look at Esri's COM component categories found in the registry. Each category ...


12

This is an exciting question! How big is the raster you want to query? WKTRaster is stored in the database as a BLOB. In order to find the value at a specific point, from a known (x_0, y_0) corner coordinate row/column indices (i, j) are computed using (dx, dy) steps and rotation. With (i, j) known, the ST_Value() function can access the actual data at the ...


11

I used to work in that exact same environment (the exact same one!). I have not done any benchmark testing but my sense of this is that number of layers in the project doesn't have much effect by itself. In my experience the labeling and number of features is a much bigger issue than the number of layers (especially if many are turned off). I used to have ...


10

When faced with tasks like this your primary objective is to be rational. Don't change params based on 'gut feeling'. While the gut seems to works for Hollywood it does not for us who live in the real world. Well, at least not my gut ;-). You should: establish a usable and repeatable metric (like the time required by a pgrouting query) save metric results ...


10

Ensure you are storing valid geometries, otherwise unexpected errors can happen (server crashes, erroneous distance and area calculations, etc., see What are the implications of invalid geometries). I highly recommend adding check constraints to production geometry columns to make sure the geometries are valid. It is simple to add (just replace "...


10

OPENGEO has a simple tutorial that will serve as a good starting point: Tuning PostgreSQL for Spatial Although not directly related to PostGIS, I've found these two posts informative when it comes to more 'general' PostgreSQL setups: Tuning PostgreSQL on your Dedicated Server Performance Tuning PostgreSQL This presentation by PGexperts is also very ...


10

According to System Design Strategies - Software Performance, an Esri technical article, file geodatabases are faster than shapefiles in their specific tests. They go into a lot of additional detail regarding what bottlenecks may exist and other performance considerations for various ArcGIS software use cases.


10

ArcGIS seems very bloated. I remember a huge performance hit when migrating from Arcview 3.2 to ArcGIS 8.0, and in a lot of places it still exists. At that time I thought it had a lot to do with ESRI migrating earlier Arc/Info code to Windows and having to cut some corners in performance, but I'm not sure if that's true. I recall seeing some examples on this ...


10

After contacting the appropriate IT admin, I recieved the following instructions, resulting in load times closer to my expectation: ArcMap now takes ~4s, ArcCatalog ~3s to load. Install ArcGIS service pack 3 (this did not change boot time on its own) run "esriremove.reg" to remove changes made by ArcGIS to the registry, the contents of esriremove.reg are: ...


Only top voted, non community-wiki answers of a minimum length are eligible