Tag Info

Hot answers tagged

21

General python optimization techniques can save you substantial amounts of time. One really good technique for getting a lowdown of where the hold ups are in your script is using the built-in cProfile module: from cProfile import run run("code") # replace code with your code or function Testing using a small data sample will allow you to pinpoint which ...


19

The truth is that most people use a custom variation of the A* algorithm. You will see this across the most of the "big guys"(I can't say who they are in a public forum, but I can tell you that you probably use one of them - guaranteed), where the modification of the heuristics is very dependent on the datasets that they use. You mentioned pgrouting ...


19

EDIT Oh, so many typos in one post must be some sort of record. Table names was messed up, I hope it is better now. I also realized on the way home that something is wrong here. ST_DWithin was faster in my test than ST_Intersects. That is surprising, especially since the prepared geometry algorithm is supposed to kick in on cases like this. I think there ...


13

Not sure if it is newer but pgRouting has a Shooting-Star algorithm: Shooting-Star algorithm is the latest of pgRouting shortest path algorithms. Its speciality is that it routes from link to link, not from vertex to vertex as Dijkstra and A-Star algorithms do. This makes it possible to define relations between links for example, and it ...


12

Speed Tests There are some very speed tests of shapefiles versus database (PostGIS) for MapServer in this presentation (from 2007). In summary: For a dataset of 3 million features running requests for 30 features one after another PostGIS was faster than shapefile (although this may have since changed by a fix to reading the shapefile index) For a ...


12

If "fastest" includes the amount of your time that is spent, the solution will depend on what software you are comfortable with and can use expeditiously. The following remarks consequently focus on ideas for achieving the fastest possible computing times. If you use a canned program, almost surely the best you can do is pre-process the polygons to set up ...


12

A couple potential suggestions to help speed up your process are: Select Layer By Attribute can be in a Python-only script, without ever launching ArcGIS Desktop. You need to convert your "buff" reference from a file-based reference to an "ArcGIS layer" reference, which ArcGIS can process selection queries against. Use ...


11

Make sure you are writing to internal drive on the computer. Reaching across the network when it is not necessary can really slow the processing. It can even be faster to copy the data as the first step in the process to keep the subsequent read-writes as quick as possible Running the script completely outside of ArcMap can be much faster. If a Map isn't ...


10

Short answer: No. With this type of UPDATE query, we are updating each row in locations ("Seq Scan"), and the GiST index on the_geom in regions is sufficient in helping limit rows for the ST_Within condition to pair-up the right row from regions. Longer answer: The magic to figuring this out is to compare what you get from explain query. From pgAdmin III, ...


9

Contrary to what dariapra says, my experience in developing Maperitive tells me that the greatest bottleneck is in actual loading of the data before rendering. It all very much depends on how large the overall stored dataset is and how large is the dataset you are trying to render in one go. If you can load it all up into memory, then shapefiles are probably ...


7

As the documentation say you may need more than 256gb of ram to do that. I don't know much about EC2, but you can try the slim (--slim) mode or try Osmosis. There is an interesting post: http://weait.com/content/build-your-own-openstreetmap-server It says, 'you must use slim mode'.


7

Contraction Hierarchy is a very fast algorithm: http://algo2.iti.kit.edu/1087.php This algorithm is RAM friendly while executing a query (to hold a contracted graph some more RAM is necessary as well as massive preprocessing) There are some other algorithms - including the ones that solve public transit routing: ...


7

Filtering (ie. using Layer / Query or the Query Builder in layer properties) in QGIS justs adds a where clause to the query that is executed. So that shouldn't be different to you want to do "in" PostGIS. Although I'm not sure what you intent to do there.


7

First off, yes you will definitely want to make sure your primary and foreign key fields are indexed on both tables. This lets the DBMS plan and execute queries against these fields much more efficiently. Secondly, you are calling SelectLayerByAttribute_management in a tight, nested loop (once per tree per treatment). This is highly inefficient, for several ...


6

For a Python solution, you may want to look at Shapely http://gispython.org/shapely/docs/1.2/ and RTree http://pypi.python.org/pypi/Rtree/ Rtree will help you create spatial indexes.


6

This can be done in GIMP http://www.gimp.org/ You need to create a custom colour palette - this will read each image loaded and best match to the pixels forcing the colours to become a near as match. You can also restrict contrast and brightness in a similar way http://gimp.open-source-solution.org/manual/gimp-tool-brightness-contrast.html ...


5

Peter, What version of PostGIS, GEOS, and PostgreSQL are you using? do a SELECT postgis_full_version(), version(); A lot of enhancements have been made between 1.4 and 1.5 and GEOS 3.2+ for this kind of thing. Also how many vertices do your polygons have? Do a SELECT Max(ST_NPoints(the_geom)) As maxp FROM sometable; To get a sense of your worst case ...


5

One method would be to query for the tags you are interested in and place those records in a new table. Then you will only need to query the new table instead of all 53 million records. If you are trying to keep your database updated, you could have this query run every time you get new data from OSM.


5

what makes you think that it is the python-part that takes time. without more information it would be more likely that the sql-query part could be optimized. do you have an index on the attribute you are querying on for instance. is there joins involved in the layer-definitions? run the script without the python-part and see if that executes faster. if the ...


5

Instead of expansive intersect, you can perform pre-selection of polygons based on comparison of bounding boxes. In other words, find all polygons overlapped / adjacent to MBR of segments of your track. Then perform detailed test on the subset of polygons.


5

Try running the sp_help_spatial_geography_index stored procedure to get details on how your spatial index is being used. You should be able to use something like: declare @ms_at geography = 'POINT (-95.66 30.04)' set @ms_at = @ms_at.STBuffer(1000).STAsText() exec sp_help_spatial_geography_index 'lidar', 'SPATIAL_lidar', 0, @ms_at; Post the results in ...


5

For my part, I would probably load CSV data into a shp file and then write a python script using shapefile and shapely to get the containing polygon id and update the field value. I don't know whether geotools and JTS is faster than shapefile/shapely ... Have no time to test it! edit : By the way, the csv conversion to shapefile format is probably not ...


5

I think you are missing some thing here :-) This is not how WMS servers work, You are also conflating tiles and features. With tiles. If you are using tiles then the hope is that a tile will be drawn once and cached for subsequent viewings. The cache can be at the server (where the cost of drawing a tile can be shared between all users) or in the browser ...


5

From this page: In particular, installing the native JAI is important for all raster processing, which is used heavily in both WMS and WCS to rescale, cut and reproject rasters. Installing the native JAI is also important for all raster reading and writing, which affects both WMS and WCS. Finally, native JAI is very useful even if there is no raster data ...


5

Spatial queries are definitely the thing to use. With PostGIS I would first try something simplistic like this and tweak the range as needed: SELECT * FROM table AS a WHERE ST_DWithin (mylocation, a.LatLong, 10000) -- 10km ORDER BY ST_Distance (mylocation, a.LatLong) LIMIT 20 This would compare points (actually their bounding boxes) using the spatial ...


5

With PostGIS 2.0 on PostgreSQL 9.1, you can use the KNN indexed nearest neighbour operator, e.g.: SELECT *, geom <-> ST_MakePoint(-90, 40) AS distance FROM table ORDER BY geom <-> ST_MakePoint(-90, 40) LIMIT 20 OFFSET 0; The above should query within a few milliseconds. For the next multiples of 20, modify to OFFSET 20, OFFSET 40, etc ...


5

If all you are looking for are proximity point searches (nearest neighbour queries), then you don't want to use the old ST_DWithin or ST_Distance + ORDER BYs for that. Not anymore. Now that PostGIS 2.0 shipped, you should be using the knngist index support (a native PostgreSQL feature). It will be orders of magnitude faster. An excerpt from this blog ...


4

The more rows that are being updated, the longer it will take to run the update. Each time you update it creates a new row. Over time, with lots of updates, things will get slower. Be sure to VACUUM on occasion. This may speed things up a bunch if you've been doing lot of updates with no vacuuming. Is a.needed_value indexed as well? If so, it will take ...


4

Due to the memory constraints I didn't even try to use osm2pgsql to load the planet.osm's routing data. Instead I used osm2po: http://osm2po.de/ Most of the documentation is in German but with a bit of experimentation I managed to get it to work. Takes a few days on a dedicated Core 2 Quad (but it is only using one thread).


4

Something must be wrong with your mysql installation or the .ini settings. Just tested a geospatial index on my old mac (10.6.8 / MySQL 5.2). That configuration is similar to yours and I tested the big geodata dump (9 million records). I did this query: SET @radius = 30; SET @center = GeomFromText('POINT(51.51359 7.465425)'); SET @r = @radius/69.1; SET ...



Only top voted, non community-wiki answers of a minimum length are eligible