Hot answers tagged

14

Aaaand I found it. Use processing.runandload, which loads the output layer into the table of contents after running the algorithm. processing.runandload("qgis:intersection", layer1, layer2, "memory:myLayerName") layer = QgsMapLayerRegistry.instance().mapLayersByName("memory:myLayerName")[0] # Should do error checking as well, but this works!!


13

I would guess it has something to do with recording the geoprocessing results info.


11

As it turns out, this works fine as long as you add the memory layer to the table of contents before using it. It seems the dataobjects.getObjectFromUri function in the QGIS source can't handle it otherwise. So the following works very well: QgsMapLayerRegistry.instance().addMapLayer(mem_layer) processing.runalg("qgis:clip", layer, mem_layer, output) Also ...


10

The first thing to do once you upgrade to 10.1 on your new 64 bit computer is to install ArcGIS 10.1 SP1 because it enables 64 bit background processing, thus increasing your memory usage abilities. Whether or not ArcGIS will use it, I cannot say - I'm running 10.0. The next thing to do would be to learn about using the In-Memory workspace, which ...


10

Although your Windows installation is 64-bit, ArcGIS Desktop is still 32-bit software and can't use more than 4GB of RAM in a single process. However, you can install the ArcGIS 64-bit Background Geoprocessing addon, that will let you run most geoprocessing tools in 64-bit mode and they can use more than 4GB RAM in that case. There are some exceptions, but ...


10

The following code works for me from both the Python Console and plugin. It takes the features from the source input layer and copies the attributes to a memory layer (in this case, a polygon layer but you can change it to LineString or Point depending on layer type): layer = QgsVectorLayer("path/to/layer", "polygon", "ogr") feats = [feat for feat in layer....


9

As @PolyGeo said, you won't be able to use the in_memory workspace for Geodatabases. If you want a "temporary" geodatabase that is saved on disk, you can use the Scratch GDB. #Set the path to GDB, if desired. #Consult help if you don't want to set this and want to know its location. arcpy.env.scratchWorkspace = r"D:\GIS\data" #Path to newly created gdb. ...


9

This works for me: arcpy.Clip_management("C12.TIF","481919 5456830 482895 5456851","in_memory/raster2","#","256","NONE") When I first tried this, I just wrote right into the python window, I got an error. So I ran the GUI tool, and copy/pasted the 'python snippet' from that, it worked. I also tried just 'in_memory', this writes a raster named 'in_memory' ...


9

You need to read the rasters in in chunks instead of all at once. See the documentation for the raster package, in particular - Writing functions for large raster files.


8

To test out your question, I wrote up a quick script that I ran two copies of simultaneously- one as a script tool in ArcMap and one in Pythonwin. Somewhat to my surprise, I was unable to run them at the same time because the "in_memory" workspace was shared. There is a way around this, however. You can add in an output check to determine if the file in ...


7

Geoprocessing is set to run in two ways: foreground and background. Someone who has more ezperience then I do can comment on the specific nuances of either setting in relation to memory leakage. However, when running geoprocessing tools, many of which store temporary data in memory and it could be the tool itself that is causing the memory to reach capacity ...


7

An in_memory workspace is one of the three broad categories of workspace available to you - the others are folder and geodatabase. However, you can think of the in_memory workspace being already analogous in structure to a file geodatabase held in memory, but there are limitations that are mostly documented. What you are describing sounds like it is ...


5

I realized what was wrong. See the correct code here: FC = arcpy.CreateFeatureclass_management("in_memory", "FC", "POLYGON", "", "DISABLED", "DISABLED", Coordinate_System, "", "0", "0", "0")


5

You seem to be trying to union all the shapes without any use of attributes to group, which is odd, but taking that as a given... you want to try and union things that are close together first. So CREATE SEQUENCE bseq; WITH ordered AS ( SELECT ST_Buffer(geom, 10) AS geom FROM points ORDER BY ST_GeoHash(geom) ), grouped AS ( SELECT nextval('bseq') / ...


5

I did a lot of raster processing about 2 years ago and to speed up the processing I wrote the grids (which were temporary and fairly small) to a RAM drive. Just do a search for ram drive and your operating system, you will find lots of information on how to set one up. Once you've set it up and given it a drive letter lets say Z:\ then your output path ...


5

Loading in the first place depends a lot on the HDD/SSD and not the RAM. Drawing depends a lot on your CPU or Graphic-Card, which is done after loading or manipulating something in the map (like zoom or pan). Qgis can be set to use multiple cores to draw features. This you can find in the Options. Maybe its faster then. Then you can just open the task-...


5

In QGIS 3 you can make a copy of a layer without saving any reference to the parent layer in this way: layer.selectAll() clone_layer = processing.run("native:saveselectedfeatures", {'INPUT': layer, 'OUTPUT': 'memory:'})['OUTPUT'] The QgsVectorLayer class has a clone() function that allows you to clone the layer in a new layer, the problem is that if you ...


4

Depending on how big your file geodatabase, you might be able to use a RAM disk (e.g. ImDisk: http://www.ltr-data.se/opencode.html/#ImDisk ). It might speed things up if your hard disks are slower. Here's a question (mine) about RAM Disks: Does using RAM Disk improve ArcGIS Desktop performance appreciably? The RAM Disk option is low-hanging fruit. And ...


4

If anyone is still interested in this, it was fixed at Version 10.1. ESRI Technical Support Number: NIM070156 and NIM062420 http://support.esri.com/en/bugs/nimbus/TklNMDcwMTU2 http://support.esri.com/en/bugs/nimbus/TklNMDYyNDIw


4

Just an assumption: osm2po builds blocks of 25 data rows per insert for performance reasons and to avoid memory issues. Do you import it in one single Transaction (parameter -1 or --single-transaction)?


4

it is the correct way, it's explained in the documentation http://docs.qgis.org/2.14/es/docs/user_manual/processing/console.html the next code work with in memory all except the last that it is load MDT=path/mdt.tif drain=processing.runalg("grass:r.drain",MDT,"",(pun),False,False,False,"%f,%f,%f,%f"% (xmin, xmax, ymin, ymax),0,-1,0.00100,None) vect=...


4

You might be able to pull this off by doing some tuning on your ArcGIS for Server without increasing the RAM. 1. Process Low Isolation Using process low isolation instead of high isolation might be an efficient solution. This will make you run your image services with much less RAM. This will basically squeeze multiple instances into a single process. To ...


4

I'm sorry, but you are mistaken. 32-bits can address 2^32 (4 billion) values, while 64-bits can address 2^64 (18 quintillion) values. However, the underlying architecture of application memory partitions code into code, data, stack, and heap, which greatly lowers the amount of RAM which can be dynamically allocated. There's also issues with memory use ...


4

There's a useful debugging tool, from memory it came in QGIS 2.18 Go to settings and under 'Rendering', look for 'Debugging' (you may need to scroll down to see it) and check the box. Now, the rendering tab on the Log messages panel shows per-layer rendering times.


4

I am able to reproduce the memory issue; I am on ArcGIS 10.4.1. I have also tried to delete the map layers that are created after each conversion call, but was not able to release the memory. result = arcpy.MakeNetCDFRasterLayer_md(in_file, variable, x_dimension, y_dimension, layer,'#', '#', valueSelectionMethod) del(write_file) arcpy.Delete_management(...


3

I'm using my W7 box as a virtual terminal (Outlook + Firefox + 2 PuTTY sessions) and it's using 2.5Gb of 4Gb (4200Gb swap). Squeezing the last bit of RAM out of any Windows box will always be a challenge. Even if you run up the swap to (physical + 4Gb) you're unlikely to get much more out of this system (if you actually start utilizing swap, performance ...


3

I suggest you do it in smaller steps, with let's say 100k points per iteration. I wouldn't store the result of ST_Collect for these, but their convex hull directly, since it will have a much simpler representation and will be more efficient to work with. Once you have a new table of convex hulls, just repeat the same process over it, since the desired convex ...


3

This question has been answered in the comments (by mdsummer). This is just a way to put those ideas in order and get this question out of unanswered queue. Here you can download world wide jpg's NVDI from nasa. Here you have the code and a raster file to tryout. As shown in question, loading the raster into R with raster() function does not load the ...


3

Very likely your computational region is not matching the elevation map, i.e. too large. Check it with g.region -p I just used the EU DEM 25m to make a test on my tiny ASUS laptop (4GB RAM, Intel i3), using Fedora 22, 64bit: 13407 * 11050 = 148147350 <<--- your DEM 12880 * 16370 = 210845600 <<-- my DEM (I just had this DEM ready ...


3

A possible workaround would be to add an empty memory layer in your project which acts as a placeholder for your large point dataset. This memory layer is saved, alongside any styles etc. that you typically applied, and as it is empty, will load immediately when you open your project. You can use the following in the Python Console to create a simple point ...


Only top voted, non community-wiki answers of a minimum length are eligible