Moving gdal raster references through functions

Question

If I'm processing a lot of rasters, so far I've been creating files. Which I like doing in the sense it gives me confirmation that the intermediary steps are working correctly. However, after I'm done confirming, this seems inefficient.

I want to attempt to pass the references to the object rather than writing to file. Namely, some of these concepts confuse me:

1. If I have a list where I append a number of references to gdal raster datasets. I am then holding all of that in my memory. Does that mean 200 mb of rasters would be equivalent to 200 mb of space in my RAM?

2. You set gdal datasets to none after finishing with them in order to close them. When you are holding references to datasets in a list, and transferring them to another function, do you then set the list to None? Do you have to loop through the list at some point to set the dataset reference to None?

        list_of_datasets = []
        for file in files: 
            raster_data = gdal.Open(file)
            cut_data = gdal.Translate("", raster_data, format="MEM", projWin=[minx,maxy,maxx,miny], outputSRS=out_srs)
            list_of_datasets.append(cut_data)
        another_function(list_of_datasets)

Answer 1

Regarding 1: if you store the data in Memory Drivers, then yes, you will hold all of the data in RAM (it might be a little more than on disk, if e.g. the files on disk have some compression scheme).

The GDAL documentation mentions:

Native object gets destroyed when Python object goes out of scope, or when they are assigned to None. So replace foo.Destroy() by foo = None if you really want to control when the underlying C++ object is destroyed.

Note that Python is garbage collected. So generally if an object goes out of scope, garbage collection free up the memory. A convenient way of doing this, is by e.g. defining a function that does the work. As the function finishes, the variables go out of scope, and the memory is freed.

In general, however, I would greatly recommend you use rasterio rather than the gdal package if you have the option: https://rasterio.readthedocs.io/en/latest/intro.html

Rasterio provides a much more "Pythonic" API to GDAL methods.