0

I've got 136 GeoTIFF files that I'd like to merge using a Python program and gdal_merge.py on Windows but the command line max character limit is 8191 and concatenating all the file names goes over that limit. I could rename all the files to something much shorter as a potential work-around but when I attempt to merge the files interactively in QGIS it appears to be reading all the file names from an input/config file of sorts. I'm wondering if there's a way I can manage to do something similar with gdal_merge.py

Trying to use gdal_merge.bat vs. gdal_merge.py errors out - likely because I have more than one Python environment on my system, not just the Python environment that is included with QGIS/ISGeo4W.

My code:

import os
import sys
import subprocess
import glob

print('Merging the DEMM files...')

# Find all files with a .tif extension in the subdirectory geotif_source_files
input_files = glob.glob("./geotif_source_files/*.tif")

# Define the output file
output_file = "./geotif_source_files/geotiff.tiff"

input_string = " ".join(input_files)

# Build the command to run gdal_merge with -co NUM_THREADS=1 option
cmd = f"gdal_merge.py -o {output_file} -co {input_string}"
# Run the command
subprocess.run(cmd, shell=True, check=True)

The error I get:

Merging the DEMM files... The command line is too long.

The GDAL command from interactive use in QGIS:

gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/MattReidy/AppData/Local/Temp/processing_LEZXJS/ddc6540bf65e4dca9b5df2ed74b81c93/OUTPUT.tif --optfile C:/Users/MattReidy/AppData/Local/Temp/processing_LEZXJS/24dd8e3280d140eb9adb885b37cb197d/mergeInputFiles.txt {SNIP}
4
  • I have tinkered with merging the first two files, then merging that merged file with the next, etc - looping until they've all been added. There's got to be a better way...
    – Matt R
    Jan 13, 2023 at 20:44
  • one option which might save you characters is to cd to your working directory so you don't have to use full path names
    – GeoMonkey
    Jan 13, 2023 at 21:11
  • GeoMonkey - Yes, that is an option, along with renaming all the files to something very short.
    – Matt R
    Jan 13, 2023 at 22:07
  • 1
    For most use cases gdalbuildvrt gdal.org/programs/gdalbuildvrt.html is better than gdal_merge. But if you use the --optfile from the command line why not to try the same with Python?
    – user30184
    Jan 14, 2023 at 13:49

1 Answer 1

0

This does the trick, using gdal.BuildVRT

import os
import sys
import subprocess
import glob
import string
from osgeo import gdal

print('Merging the DEMM files...')

# Find all files with a .tif extension
input_files = glob.glob("*.tif")

# Define the output file
output_file = "geotiff.tiff"

# Create a dictionary to store the original file names as keys and new short file names as values
original_names = {}

# Create list of characters (numbers and letters) to use for unique file names
characters = string.digits + string.ascii_letters
j = 0
# Rename all the files in the current directory to have unique short names of 2 characters
for i, file in enumerate(input_files):
    new_name = ''.join(characters[i%len(characters)] + characters[i//len(characters)] + ".t")
    # Check if the new name already exists, ignoring the case
    while os.path.exists(new_name) or new_name.lower() in [f.lower() for f in os.listdir()]:
        i += 1
        new_name = ''.join(characters[i%len(characters)] + characters[i//len(characters)] + ".t")
    os.rename(file, new_name)
    original_names[new_name] = file
    input_files[j] = new_name
    j += 1

input_string = " ".join(input_files)

#Build VRT from input files 
vrt_file = "merged.vrt"
gdal.BuildVRT(vrt_file,input_files)

# Translate VRT to TIFF
gdal.Translate(output_file,vrt_file)

# Rename the files back to their original names
for new_name, original_name in original_names.items():
    os.rename(new_name, original_name)

# remove the vrt file
os.remove(vrt_file)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.