0

If I'm processing a lot of rasters, so far I've been creating files. Which I like doing in the sense it gives me confirmation that the intermediary steps are working correctly. However, after I'm done confirming, this seems inefficient.

I want to attempt to pass the references to the object rather than writing to file. Namely, some of these concepts confuse me:

  1. If I have a list where I append a number of references to gdal raster datasets. I am then holding all of that in my memory. Does that mean 200 mb of rasters would be equivalent to 200 mb of space in my RAM?

  2. You set gdal datasets to none after finishing with them in order to close them. When you are holding references to datasets in a list, and transferring them to another function, do you then set the list to None? Do you have to loop through the list at some point to set the dataset reference to None?

        list_of_datasets = []
        for file in files: 
            raster_data = gdal.Open(file)
            cut_data = gdal.Translate("", raster_data, format="MEM", projWin=[minx,maxy,maxx,miny], outputSRS=out_srs)
            list_of_datasets.append(cut_data)
        another_function(list_of_datasets)
2

Regarding 1: if you store the data in Memory Drivers, then yes, you will hold all of the data in RAM (it might be a little more than on disk, if e.g. the files on disk have some compression scheme).

The GDAL documentation mentions:

Native object gets destroyed when Python object goes out of scope, or when they are assigned to None. So replace foo.Destroy() by foo = None if you really want to control when the underlying C++ object is destroyed.

Note that Python is garbage collected. So generally if an object goes out of scope, garbage collection free up the memory. A convenient way of doing this, is by e.g. defining a function that does the work. As the function finishes, the variables go out of scope, and the memory is freed.

In general, however, I would greatly recommend you use rasterio rather than the gdal package if you have the option: https://rasterio.readthedocs.io/en/latest/intro.html

Rasterio provides a much more "Pythonic" API to GDAL methods.

2
  • Is setting to None equivalent to a variable going out of scope? For example in this video, youtube.com/watch?v=p_BsFdV_LUk&t=630s (from 10:30 to 10:50), the raster does not appear correct initially. She corrected this by setting the dataset to None at the end of the script, which theoretically at the end of the script the variable should be getting removed by garbage collection anyways so I'm not sure why setting to None is necessary? On another note, thanks, I will look into using Rasterio instead. – mbase May 6 at 8:33
  • 1
    If you're okay with the dataset living in memory until the end of the script, there is no need to set it to None. In the youtube example you refer to, she's running the script interactively, so the script does not "end" -- Python keeps on running with the variables in scope. Setting to None is equivalent to going out of scope. Every variable in Python is a pointer. So first the variable points to the memory where the dataset is stored, and then it points to None. This dereferences the dataset part of memory, and garbage collection deletes it. Out-of-scope also dereferences. – Huite Bootsma May 6 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.