It's possible to use gdal_calc.py with an entire path of rasters?

Question

I try to use gdal_calc.py to scale and transform a time-series of MODIS images. I've done that when i worked with a single image in the -A variable and a mask in -B, but i need to work with a directory of 350 .tif images and convert to ENVI .bin raster. My problem is how to assign the -A variable to realize the operation with all the images and obtain 1 output file per image

To work with a single image i use:

gdal_calc.py -A image.tif -B MASK10000 --calc="A/B" --NoDataValue=0 --format=ENVI --outfile=image.bin

I use the linux bash, but maybe a python scrip in linux runs better

Answer 1

I use a simple way to solve this using the UNIX bash.

I made a script in the same path of images and ran it ("script.sh").

#!/bin/bash
for i in *.tif
do
gdal_calc.py -A $i -B MASK.dat --calc="A/B" --NoDataValue=0 --format=ENVI --outfile=directory/$i.bin
done

• MASK.dat is a binary image with only 10,000 values to be scaled within the NDVI range.
• "i" are all the MODIS images that I need to convert. "directory" is the path where all the outputs will be created.

This was to then use the image files in TIMESAT software.

Answer 2

There are two python ways to do this simply, use os.walk() to find the files and then either of create a batch file or use subprocess.Popen() to execute the command.

This code contains both.. default is Batch File which can be switched by setting UsingBat = False:

import os, sys, subprocess

BaseInFolder  = sys.argv[1]
BaseOutFolder = sys.argv[2]
MaskImage     = sys.argv[3]
UsingBat      = True

if UsingBat:
    FirstPartOfBat  = "GDAL_CALC.py -A "
    SecondPartOfBat = " -B " + MaskImage + " --calc=\"A/B\" --NoDataValue=0 --format=ENVI "#--outfile=image.bin
    TempDir = os.environ.get("Temp") # get the 'temp' directory
    BatFile = os.path.join(TempDir,"RunMe.bat")# use your own batch file if you like

    with open(BatFile,'w') as BatchWriter:
        for (path, dirs, files) in os.walk(BaseInFolder):
            for file in files:
                fName,fExt = os.path.splitext(file)
                # you might want to think of something here to limit the file types
                # like if len( fExt ) == 0: I don't have experience with MODIS so I can't say
                OutFileName = fName + fExt
                # if that name is already in use try another name
                if os.path.exists(os.path.join(BaseOutFolder,OutFileName)):
                    Try = 1
                    while os.path.exists(os.path.join(BaseOutFolder,OutFileName)):
                        OutFileName = fName + str(Try) + fExt
                        Try += 1

                # assemble the command line and write to batch file
                pBat = FirstPartOfBat + os.path.join(BaseOutFolder,file) + SecondPartOfBat + "--outfile=" +os.path.join(BaseOutFolder,OutFileName)
                BatchWriter.write(pBat + "\n") # can't forget newline.. Linux/Mac may be different
else:
    FirstPartOfCommand = ["GDAL_CALC.py","-A"]
    SecondPartOfCommand = ["-B",MaskImage,"--calc=\"A/B\"","--NoDataValue=0","--format=ENVI"]#--outfile=image.bin
    for (path, dirs, files) in os.walk(BaseInFolder):
        for file in files:
            fName,fExt = os.path.splitext(file)
            # you might want to think of something here to limit the file types
            # like if len( fExt ) == 0: I don't have experience with MODIS so I can't say
            OutFileName = fName + fExt
            # if that name is already in use try another name
            if os.path.exists(os.path.join(BaseOutFolder,OutFileName)):
                Try = 1
                while os.path.exists(os.path.join(BaseOutFolder,OutFileName)):
                    OutFileName = fName + str(Try) + fExt
                    Try += 1
            # got to use a new list, otherwise direct assingment modifies the existing list
            # see http://stackoverflow.com/questions/2612802/how-to-clone-or-copy-a-list-in-python
            pOpenCommand = list(FirstPartOfCommand)
            pOpenCommand.append(os.path.join(path,file))
            pOpenCommand.extend(SecondPartOfCommand)
            pOpenCommand.append("--outfile=" +os.path.join(BaseOutFolder,OutFileName))

            ThisProc = subprocess.Popen(pOpenCommand)
            ThisProc.wait() # wait until finished, you don't want 350 of them running at the same time

Using a batch file you need to compile strings and insert values where appropriate, write to file and then run the file (double click); Using Popen you need to create a list of each element of the command - and not forgetting to use .wait() on the process otherwise the code will spawn a process and move on to spawn another process before the previous one has exited - you will end up filling up your memory in a few seconds.

I haven't had anything to do with MODIS data so I can't help limiting the input files so you should think about how you are going to limit the files.. os.walk() returns every file in the current folder and all subfolders - you will need a way to detect if the file is suitable before trying to convert it.