0

I'm attempting to merge thousands of rasters into a smaller more manageable number using the Mosaic to NewRaster_management tool in ArcPy. The code I've written works just fine when I run it the normal way without using multiprocessing. However this is far too slow for 30000 rasters so I've tried to use Multiprocessing and have 3 pool workers creating rasters at the same time. Problem is I get this error when I try run it. Its not the most descriptive so I cant figure out what the issue is. I've avoided using geodatabase and each worker should ideally be getting a number of rasters and creating a new .tif file so theres no reason for any issues that I can see. Any ideas?

arcgisscripting.ExecuteError: ERROR 000582: Error occurred during execution

arcpy.env.workspace = "in_memory"
arcpy.env.overwriteOutput = True
#arcpy.parallelProcessingFactor = "90%"

##Variables
#outGDB = "M:/Working/DataProcessing/FishMonitoring.gdb"
outGDB = "M:\\DATA RECEIVED - DO NOT EDIT\\2019-04-04 DTM\\MergedDTM"
outCord = "PROJCS['British_National_Grid',GEOGCS['GCS_OSGB_1936',DATUM['D_OSGB_1936',SPHEROID['Airy_1830',6377563.396,299.3249646]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Transverse_Mercator'],PARAMETER['False_Easting',400000.0],PARAMETER['False_Northing',-100000.0],PARAMETER['Central_Meridian',-2.0],PARAMETER['Scale_Factor',0.9996012717],PARAMETER['Latitude_Of_Origin',49.0],UNIT['Meter',1.0]]"

#List Rasters pure Python
directory = 'M:/DATA RECEIVED - DO NOT EDIT/2019-04-04 DTM/DSM_2M'
extension_list = ".asc"
Rasterlist = []
for file in os.listdir(directory):
    for extension in extension_list:
        if file.endswith(extension):
            newfile = os.path.join('M:/DATA RECEIVED - DO NOT EDIT/2019-04-04 DTM/DSM_2M/',file)
            Rasterlist.append(newfile)
Rasters = Rasterlist[:60]

##Function
def process(raster):
    # Set Outraster name
    rastername = Rasters[:1]
    rastername = ''.join(map(str,rastername))
    rastername = rastername[53:]
    rastername = rastername[:-4]
    rastername = rastername + '.tif'

    arcpy.MosaicToNewRaster_management(input_rasters=raster, output_location=outGDB,
                                        raster_dataset_name_with_extension=rastername,
                                        coordinate_system_for_the_raster= outCord,
                                         pixel_type="8_BIT_UNSIGNED", cellsize="",
                                         number_of_bands="1", mosaic_method="LAST", mosaic_colormap_mode="FIRST")

def main():
    startTime = time.clock()
    pool = Pool.Pool(3)
    pool.map(process,Rasters)
    pool.close()
    endTime  = time.clock()
    elapTime = endTime - startTime
    print(elapTime)

#Parellize function
if __name__ == "__main__":
    main()
  • First thing I would do and is good programming practise is to get rid of that insane folder path you have. You have spaces and hyphens and folders starting with numbers a real no no in scripting.Also you are not actually multi-processing you appear to create a list of ASC files and then send the first 60 only, nothing is running in parallel in that respect. – Hornbydd Apr 10 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.