2

I have a Python loop, the loop has to run commands on 200 plus files. But it randomly stops and does not continue producing the new output files.

here is the code:

from subprocess import Popen, PIPE
import os,glob

os.chdir(r'C:\Program Files\QGIS 3.0\bin')
grassbat = r'C:\Program Files\QGIS 3.0\bin\grass74.bat'  #not needed as the `os dir is changed`,

osgeobat = str(r'C:\Program Files\QGIS 3.0\OSGeo4W.bat')
input_las = r'E:\Rohan\Heimdall\data\LAZ\heimdall_final_demo_lidar'
output_directory = r'E:\Rohan\Heimdall\data\Raster\GRASS'
grass_location = r'E:\Rohan\Heimdall\grassdata\lidar\PERMANENT' # r'  reads a path as a raw string
print(grass_location)

# Loop to transform LAS to raster tif.
def las_to_GTIFF_lidardata(input_las, grass_location, output_directory):
    las_file_list = glob.glob(input_las + '\\*.las')  # lists all files in folder that (*.las)
    print(las_file_list)
    for file in las_file_list:
        tif_file_name = os.path.basename(file)[:-4] + '.tif'
        out_tif = os.path.join(output_directory, tif_file_name)

        firstcommand = str(osgeobat + " grass74 -text -f E:\Rohan\Heimdall\grassdata\lidar\PERMANENT --exec r.in.lidar -e -n -o --overwrite --verbose input=" + file +" output=lidar_in resolution=2")
        secondcommand = str(osgeobat + " grass74 -text -f E:\Rohan\Heimdall\grassdata\lidar\PERMANENT --exec r.out.gdal --overwrite --verbose -m input=lidar_in@PERMANENT output=" + out_tif + " format=GTiff type=Float32 nodata=-9999")
    firstcommandlst = firstcommand.split(" ")
    secondcommandlst = secondcommand.split(" ")

    las2raster_commands = []
    las2raster_commands.append(firstcommandlst)
    las2raster_commands.append(secondcommandlst)

    for command in las2raster_commands:
        print("command: ")
        print(command)
        process = Popen(command, stdout=PIPE, stderr=PIPE)
        stdout = process.communicate()
        print(stdout)
return True


las_to_GTIFF_lidardata(input_las=input_las, grass_location=grass_location, output_directory=output_directory)    

closed as unclear what you're asking by Andre Silva, BERA, aldo_tapia, Jochen Schwarze, whyzar May 4 '18 at 13:04

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    Try placing print statements throughout to get a better idea of where it is stopping. Also google debugging, idk what IDE you are using but you can use debugging tools to tell you what the variables are holding when it fails. – atxgis May 2 '18 at 14:08
  • 1
    My guess would be that this has something to do with the fact that you're calling system processes within the loop. Perhaps there's a maximum it can handle? If this were the case, you could break your commands up into chunks and complete each chunk before running the next one. This also seems like a good job for the multiprocessing module. – Jon May 2 '18 at 14:21
  • 1
    As a side note, the way you print the stdout I'm assuming that you see some encoding issues and some \n all over the place. You can use this to increase readability : stdout = process.communicate()[0].decode('utf-8').strip() – GHRF May 3 '18 at 8:23
  • 1
    Another thing you could do is to make sure you have a successful stdout message before moving to the next iteration of the loop. – Jon May 3 '18 at 14:18
  • 1
    @RK4gis Check out the comments on your answer, also, try calling your string with this: proc = subprocess.Popen(callstring, stdout=subprocess.PIPE,stderr=subprocess.PIPE) and then stdout,stderr=proc.communicate()--the variable stderr will contain any error messages. It seems likely that you're not calling your processes correctly. – Jon May 3 '18 at 16:58
0

I actually think it was just an indentation issue with your FOR loop - as it is presented here, it iterates through all the LAS files, and only the last one gets the two commands added. Indent the lines as shown below. Also, it's probably fine the way you have it, but if there is a synchronization/PIPE issue, you can simplify your subprocess command with the wait() function, to ensure processes are run one at a time:

from subprocess import Popen, PIPE
import os,glob

os.chdir(r'C:\Program Files\QGIS 3.0\bin')
grassbat = r'C:\Program Files\QGIS 3.0\bin\grass74.bat'  #not needed as the `os dir is changed`,

osgeobat = str(r'C:\Program Files\QGIS 3.0\OSGeo4W.bat')
input_las = r'E:\Rohan\Heimdall\data\LAZ\heimdall_final_demo_lidar'
output_directory = r'E:\Rohan\Heimdall\data\Raster\GRASS'
grass_location = r'E:\Rohan\Heimdall\grassdata\lidar\PERMANENT' # r'  reads a path as a raw string
print(grass_location)

# Loop to transform LAS to raster tif.
def las_to_GTIFF_lidardata(input_las, grass_location, output_directory):
    las_file_list = glob.glob(input_las + '\\*.las')  # lists all files in folder that (*.las)
    print(las_file_list)
    las2raster_commands = []   ###List created once, populated with ALL commands
    for file in las_file_list:
        tif_file_name = os.path.basename(file)[:-4] + '.tif'
        out_tif = os.path.join(output_directory, tif_file_name)

        firstcommand = ['"' + osgeobat + '"',"grass74","-text","-f",'"E:\Rohan\Heimdall\grassdata\lidar\PERMANENT"',"--exec","r.in.lidar","-e","-n","-o","--overwrite","--verbose","input=" + file,"output=lidar_in", "resolution=2"]
        secondcommand = ['"' + osgeobat + '"',"grass74","-text","-f",'"E:\Rohan\Heimdall\grassdata\lidar\PERMANENT"', "--exec", "r.out.gdal", "--overwrite", "--verbose", "-m", "input=lidar_in@PERMANENT", "output=" + out_tif, "format=GTiff", "type=Float32", "nodata=-9999"]



        las2raster_commands.append(firstcommand)  ###INDENT THESE TWO LINES - for each las file, add two commands in sequence 
        las2raster_commands.append(secondcommand)  ###Include in FOR loop, indent

    for command in las2raster_commands:
        print("command: ")
        print(command)
        process = Popen(command)  ###Simplify, just in case this is a PIPE read/write issue
        process.wait() ###Simplify, just in case this is a PIPE read/write issue




las_to_GTIFF_lidardata(input_las=input_las, grass_location=grass_location, output_directory=output_directory) 
  • Can't be this for two reasons: 1) he has run the same code multiple times and it stops at different points each time, and 2) his indentation is correct. The split() breaks a single large string into individual list elements after the string has been created, which is what he wants. – Jon May 3 '18 at 14:15
  • Jon your first point makes sense - however, for the second, he's splitting an os command (variables firstcommand, secondcommand), into a list that can be input into subprocess Popen(), looped from a list of listed commands. – AlecZ May 3 '18 at 16:20
  • 2
    You're right--there's definitely a problem with the creation of the commands. Even with your proposed fix, using split(" ") breaks the "Program Files" into two list elements, as well as "QGIS 3.0", which would not behave well with subprocess. – Jon May 3 '18 at 16:48
  • 1
    Wow that's true I hadn't noticed that aspect of it - it should fail then, odd that it's even working for the ones it's going through. I'll update this to make sure. – AlecZ May 3 '18 at 16:51
  • 1
    Yeah, after testing the original code, it seems impossible that any of the commands would run successfully, so I'm wondering if @RK4gis actually checked the output--he says "it stops" but doesn't mention if it was successful at all. – Jon May 3 '18 at 16:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.