Merging a large number of GeoTIFF files via gdal_merge.py

Question

I've got 136 GeoTIFF files that I'd like to merge using a Python program and gdal_merge.py on Windows but the command line max character limit is 8191 and concatenating all the file names goes over that limit. I could rename all the files to something much shorter as a potential work-around but when I attempt to merge the files interactively in QGIS it appears to be reading all the file names from an input/config file of sorts. I'm wondering if there's a way I can manage to do something similar with gdal_merge.py

Trying to use gdal_merge.bat vs. gdal_merge.py errors out - likely because I have more than one Python environment on my system, not just the Python environment that is included with QGIS/ISGeo4W.

My code:

import os
import sys
import subprocess
import glob

print('Merging the DEMM files...')

# Find all files with a .tif extension in the subdirectory geotif_source_files
input_files = glob.glob("./geotif_source_files/*.tif")

# Define the output file
output_file = "./geotif_source_files/geotiff.tiff"

input_string = " ".join(input_files)

# Build the command to run gdal_merge with -co NUM_THREADS=1 option
cmd = f"gdal_merge.py -o {output_file} -co {input_string}"
# Run the command
subprocess.run(cmd, shell=True, check=True)

The error I get:

Merging the DEMM files... The command line is too long.

The GDAL command from interactive use in QGIS:

gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/MattReidy/AppData/Local/Temp/processing_LEZXJS/ddc6540bf65e4dca9b5df2ed74b81c93/OUTPUT.tif --optfile C:/Users/MattReidy/AppData/Local/Temp/processing_LEZXJS/24dd8e3280d140eb9adb885b37cb197d/mergeInputFiles.txt {SNIP}

Answer 1

This does the trick, using gdal.BuildVRT

import os
import sys
import subprocess
import glob
import string
from osgeo import gdal

print('Merging the DEMM files...')

# Find all files with a .tif extension
input_files = glob.glob("*.tif")

# Define the output file
output_file = "geotiff.tiff"

# Create a dictionary to store the original file names as keys and new short file names as values
original_names = {}

# Create list of characters (numbers and letters) to use for unique file names
characters = string.digits + string.ascii_letters
j = 0
# Rename all the files in the current directory to have unique short names of 2 characters
for i, file in enumerate(input_files):
    new_name = ''.join(characters[i%len(characters)] + characters[i//len(characters)] + ".t")
    # Check if the new name already exists, ignoring the case
    while os.path.exists(new_name) or new_name.lower() in [f.lower() for f in os.listdir()]:
        i += 1
        new_name = ''.join(characters[i%len(characters)] + characters[i//len(characters)] + ".t")
    os.rename(file, new_name)
    original_names[new_name] = file
    input_files[j] = new_name
    j += 1

input_string = " ".join(input_files)

#Build VRT from input files 
vrt_file = "merged.vrt"
gdal.BuildVRT(vrt_file,input_files)

# Translate VRT to TIFF
gdal.Translate(output_file,vrt_file)

# Rename the files back to their original names
for new_name, original_name in original_names.items():
    os.rename(new_name, original_name)

# remove the vrt file
os.remove(vrt_file)