After the suggestions of user30184, Paul Ramsey and my own experiments. I decided to answer this question.
I failed to mention in this question that I am importing data to a remote server. (although it is described in the blog post I refer to). Operations such as inserts, over the internet are subject to a network latency. Perhaps it is not irrelevant to ...
Use just the Shape field as the compare fields in the Delete Identical tool which:
Deletes records in a feature class or table which have identical
values in a list of fields. If the field Shape is selected, feature
geometries are compared.
Any of the common GIS systems can handle large rasters (TIFF, JPEG, BMP etc etc). You can choose from paid-for solutions like ArcGIS or free ones like QGIS (SAGA and GRASS are available as stand-alone solutions but are also bundled with QGIS).
The GIS packages are pretty good at creating pyramids (down-sampled preview versions) which help with zooming and ...
Yes, the CF 1.6 Conventions for NetCDF include the specification of collections of time series and it seems your data is similar to example H.2.1 "Orthogonal multidimensional array representation of time series":
If you store your data this way, IDV should be able to recognize this as "point data". Hopefully more applications in the future will take ...
Yes, although this seems not to be mentioned in the documentation.
If you include num_threads=8 or num_threads='all_cpus' as an argument to rasterio.open then multithreading will be enabled (for writing of compressed data).
import numpy as np, rasterio
size = 16384
chunk = 512
with rasterio.open('test.tiff', 'w', driver='GTiff', nodata=0,
Keep in mind what you are doing -- You are asking the database to return all features within the display envelope AND whose ids are in the first 1000. And you are likely doing so with a spatial constraint first option. Yes, it is very normal to see poor performance under such circumstances.
Best practice for rendering very large tables is not to. This ...
You can cut it directly with gdal's tool gdal_translate if you know the coordinates of your Area of Interest, if its georeferenced:
gdal_translate -projwin [ulx uly lrx lry] infile outfile
If not use the -srswin flag like this:
gdal_translate -srcwin [xoff yoff xsize ysize] infile outfile.
Another option is to build a 'virtual' raster (of a few ...
Other ideas you could try:
gdal_translate with the -srcwin switch
gdalwarp with -cutline and -crop_to_cutline and -wm switches. The last one specifies memory for caching and may get you over the issues you had using clipper in QGIS (as this is essentially the same function)
QGIS raster calculator setting the extent to the area you want (simpler than ...
You might be able to do your processing with the gdal_calc.py utility, which has block-level processing (not whole file processing). For example, this command should do the same thing your program does, using Numpy's where command:
gdal_calc.py -A input1.tif --outfile=result.tif --calc="where(A < -9000, -9999, A)" \
Field calculator can be activated from Layer Properties window. It is on Fields tab (2.18) or Source Fields tab (3.0/3.2), which calculation is much faster than the one activated from the attribute table.
There are a couple of techniques you can use to speed up this query. you can use either one, or both together.
test overlap of the building centroids rather than polygons. this may be faster, it also stops any buildings being double-counted if they fall into two areas - which may be good or bad, depending on your requirements.
use st_subdivide on your flood ...
ST_intersects is generally considered a fast option but see PostGIS docs on getting intersections faster for example queries. ST_within is another option to try as well as per the above link especially in conjunction with using building centroids per Steven Kay's suggestion (less geometry to consider).
However, all this said, there are a number of other ...
Instead of playing with workarounds try if GDAL can do the job for you directly. If command
ogrinfo -al -so your_big_shapefile.shp
seems successful you have good chance to have luck with ogr2ogr as well. Read
http://www.gdal.org/ogr2ogr.html and http://www.gdal.org/drv_pg.html and try
ogr2ogr -f PG PG:"dbname='databasename' host='addr' port='5432' user='...
Yes, the NetCDF CF Metadata Conventions version 1.6 specifies how to store point and station time series data in chapter 9 "Discrete Sampling Geometries".
Since your data has the same sample times for all stations, I agree with Rich that you can base your netCDF structure on the example in section H.2.1 "Orthogonal multidimensional array representation of ...
It is common to aggregate by rounded up coordinates. I.e. call ST_SnapToGrid with some precision, that will return identical points for close-enough points, then group by the resulting point.
Another way to do the same, but with less control, is to compute geohash - length of geohash determines the snap distance in this case.
This is, generally, a very bad idea and defeats the point of a raster format. There is very little in the way of software that can deal with a 60M x N dataset. With 20 parameters the number of observations grows to n=1,200,000,000. In statistical terms, you would be better served taking a sub-sampling approach or just leverage raster functions for your ...
I think the crux of the question here is which tasks in your workflow are not really ArcGIS dependent? Obvious candidates include tabular and raster operations. If the data must start and end within a gdb or some other ESRI format, then you need to figure out how to minimize the cost of this reformat (i.e., minimize the number of round trips) or even justify ...
One approach would be to split your line feature layer into multiple chunks, calculate the length for lines in each, then merge the split layer.
For example, you can clip your roads to individual polygons in an a shapefile of administrative boundaries (counties in the example below). Select from the menu Vector -> Geoprocessing tools -> Clip, and enable ...
There were two solutions people described using in BQGIS group,
One is import plugin for QGIS,
Another is foreign data wrapper for PostgreSQL, then using Postgre layers in QGIS:
There is no streaming equivalent of ReturnNumber <= NumberOfReturns I can see some options:
I'm pretty sure that the warnings comes from points that have a NumberOfReturns = 0. Thus I would try filter = "-drop_number_of_returns 0".
Go to the github repo of the rlas package and open an issue with a feature request. This is not hard to add such filter.
WARNING! "Find Identical" and "Delete Identical" require an "Advanced" license.
I was burned by this yesterday because I delivered a python tool to a client who only has a Basic license. Fortunately I was able to use a simple point comparison instead (it took less code than calling the tool!) and the resulting tool is about 3x faster now.
You can iterate ...
I agree with some of the others that a web-style solution (either via internet and log-in site or via intranet) would be a good one and would be especially effective if you have a distributed user base that need to access the data. However, if you are working more within an office environment where all your potential users are on the same network and could ...
Web-based maps are a common way to allow read-only access to spatial data, as they have a low learning curve, don't require special software on end-user machines, and are usually responsive and easy to use. There are many methods of exposing a web map.
ArcGIS Server and Flex viewer or ESRI jsapi
Since you already have ESRI products and are probably ...
I am going to suggest that you use a few of the profiling tools in Python to give you some insight as to what is going on with your script. You may have an undetected memory leak or you may be running out of memory as suggested.
I would encourage you to seek visibility into how your script's performance and memory management or else you are just ...
The approach I would do in this situation is.
In ArcMap load the large shapefile and convert this into a file-geodatabase.
Why? The structure of the file geodatabase has been created by ESRI not by choice but because they had to - larger and larger files impacted on the shapefile (2GB limit at the time) and also the access/personal geodatabase (.mdb) has ...
Unfortunately it looks like a fault in FME. It is related to content size, but not in a way I think you can workaround.
I've filed a problem report with the developers and it does appear a relatively simple fix. If you're able to supply a copy of the data then please report the issue via safe.com/support, referencing PR#48195 and making the data available (...
It is probably some quite large datasets you have to handle, and therefore i would perhaps not suggest a WPS solution, since you would be transferring data with the process request.
WPS 1.0 has limited capabilites for asyc. requests - which will be enhanced in version 2.0 - making the solution with WPS a little more feasible i guess - but for now and in ...
Take a look on pyWPS, an OGC Web Processing Service implementation. Its easy to install on a python environment. Most of the examples use GRASS GIS as pyWPS only implements the interface for remote handling, but it is possible to use any GIS backend to do the actual processing work.
So check out the gallery first, to get an idea what is possible and ...
Loading 20GB of data, either from disk or over the web, will kill almost every desktop application.
You have to restrict the data volume to the rendered canvas extent, e.g. using virtual index files or a postgis server with spatial indices, and use overviews (pyramids) for lower zoom levels to get reasonable fast results on the screen.