2

I have an indexed dictionary containing files and want to "mosaic" (arcpy.MosaicToNewRaster_management) all files with the same index. The number of files per index range from 1 to about 30.

dic_of_files = {"index1":["file11", "file12","file13"], "index2":["file21", "file22"]}

for key in dic_of_files.iterkeys():
    arcpy.MosaicToNewRaster_management(dic_of_files[key], otherarcpyoptions)

Mosaicing 30 files in one go doesnt seem to work, I guess number is just to high. I had the idea of writing a script where python counts the number of files per index via and the loops through processing first file 1 and 2, then processing the result with file 3 and so on till it reaches the last file.

for key in dic_of_files.iterkeys():
    key_len = len(dic_of_files[key])
    if key_len < 2:
        arcpy.MosaicToNewRaster_management(dic_of_files[key], otherarcpyoptions)
    else:
        yes.. else what?

Can anybody tell me if this is a reasonable way to solve the problem? And maybe give me a hint how to script this solution?

EDIT: ok I've gotten one step farther. I've nested a couple of loops so that python pairs up the first two items in the list, then the next two and so on (in case of an uneven item number it pairs up the last three).

The code now looks as follows (I'm omitting the arcpy function to keep it as simple as possible):

dic_of_files = {"index1": ["file10", "file11", "file12", "file13", "file14", "file15", "file16", "file17"], "index2": ["file20", "file21", "file22"]}
dic_of_pairs = {}

def addstuff(i="", v="", mydict=""):
    if i not in mydict:
        mydict[i] = []
    mydict[i].append(v)

for key, val in dic_of_files.items():
    key_len = len(val)
    if key_len < 2:
        print "do nothing"
    else:
        if key_len % 2 == 0:
            for j in range(0, (key_len - 1), 2):
                # Even number of files, process pairs all through
                output = str(str(j) + str(j + 1))
                addstuff(key, output, dic_of_pairs)
        else:
            for k in range(0, (key_len - 2), 2):
                if k < (key_len - 3):
                    # Uneven number of files. Process pairs at first..
                    output = str(str(k) + str(k + 1))
                    addstuff(key, output, dic_of_pairs)
                else:
                    # ... and process the last three files as triplets
                    output = str(str(k) + str(k + 1) + str(k + 2))
                    addstuff(key, output, dic_of_pairs)

Sadly, one round doesn't do the trick, I have to repeat this block of code on the new dictionary (here: "dic_of_pairs"). It would be very elegant if I could use this same piece of code for the second (and all succeeding) rounds, until the every index contains just one file.

Is there a way to do this? Is overwriting the original dictionary ("dic_of_files") an option, or better to pack this code into a function?

3

as for the first part of the question - there might be some guidance here since it is suggested on the mosaicToNewRaster page that "When working with a large number of raster datasets, the Raster Catalog To Raster Dataset tool performs more efficiently".

However, to look at the scripting option - you may be close (i can't test this at the moment, so you'll likely have to tweak it a bit)

for key,val in dic_of_files.items(): #items() should work in python 2.7
    key_len = len(val) #no need to call the dict - val is the value of the key/value pair
    if key_len < 2:
        arcpy.MosaicToNewRaster_management(val, otherarcpyoptions) #val will give you the array of raster names
    else:
        pass #some other process
        #yes.. else what?

EDIT: not sure this would be any closer, but perhaps an elegant way to handle groups of name is using a generator such as this example.

def chunks(l, n):
    """ Yield successive n-sized chunks from l.
    """
    for i in xrange(0, len(l), n):
        yield l[i:i+n]

dic_of_files = {"index1": ["file10", "file11", "file12", "file13", "file14", "file15", "file16", "file17"], "index2": ["file20", "file21", "file22"]}

rnum=2
for key, val in dic_of_files.items():
    print val
    while rnum>1:
        rasters=list(chunks(dic_of_files[key],2))
        rnum=len(rasters)
        dic_of_files[key]=[]
        for i,r in enumerate(rasters):
            #mosaic list r to new raster - maybe new raster name of 'rast{}'.format(i)
            print 'mosaic',r
            dic_of_files[key].append('rast{}'.format(i))
            print 'dic',dic_of_files[key]
    rnum=2

still not tested with arc, but it returns groups of names until it gradually reduces each dict value to a single name.

  • Thanks for the improvments! It seems though that you do need to cal the dict while assessing the number of items per index. Using 'key_len = len(key)' seems to return the number of characters of the index itself, not the number of values assigned to it. – Ratnanil Dec 5 '14 at 22:11
  • oh sorry - i failed to catch that part. len(val) should do it. I'll edit accordingly. – fluidmotion Dec 5 '14 at 23:40
2

this is not an ArcGIS solution, but in this case it could help you a lot.

install gdal and use gdalbuildvrt to create your mosaics. This will create a single virtual file (which you can write in a file if you want, but this is not necessary(ArcGIS does read the vrt).

import subprocess

for key,val in dic_of_files.items():
    subprocess.call(["gdalbuildvrt", key + ".vrt"] + val)
    subprocess.call(["gdal_translate", "-co", "COMPRESS=LZW", "-co", "BIGTIFF=YES", key + ".vrt", key + ".tif" ])
  • Thanks for the tipp! I'll definitly look into it. At the moment I'm pretty fixed on solving the problem with python/arcpy tools. – Ratnanil Dec 6 '14 at 1:00
0

I've really gotten myself hoocked on solving the problem by pairing up the files and then pairing up the pairs and so on. I would've never thought i'd have to write such a long code, but this is what I ended up with.

I'm new to python and programming in general. Please comment if things are solved in a rather unusual/inelegant manner.

I have only sparsely commented it. Please ask if something is unclear.

dic = {"index1": ["file10", "file11", "file12"], "index2": ["file20"]}

# a list container for the maximum number of values per key
maxi = []


# defining the paths
mosaic_final = "Rasterdaten/Mosaic/final/"
mosaic_temp = "Rasterdaten/Mosaic/temp/"

def pairup_mosaic(inputdict=""):
    # start a counter q to enabel individual namegiving
    q = 0
    # reset the counter in case something was left in it
    del maxi[:]
    # loop to find the number of values per key
    for key, val in inputdict.items():
        vallen = len(val)
        maxi.append(vallen)
    # run code only if at least one key has more than one value
    if max(maxi) != 1:
        while max(maxi) != 1:
            for key, val in dict(inputdict).items():
                inputdict.clear()
                vallen = len(val)
                if vallen < 2:
                    if not os.path.isfile(mosaic_final + str(val)):
                        #arcpy.CopyRaster_management(val,mosaic_final+key)
                        print "copy raster " + str(val) + " to " + mosaic_final + str(key)
                else:
                    if vallen % 2 == 0:
                        for j in range(0, (vallen - 1), 2):
                            q = q + 1
                            del pair[:]
                            pair.append(val[j])
                            pair.append(val[j+1])
                            output_name = q
                            addstuff(key, str(output_name), inputdict)
                            if vallen > 2:
                                #arcpy.MosaicToNewRaster_management(pair, mosaic_temp, str(output_name), "", "8_BIT_UNSIGNED", "", "1", "LAST", "FIRST")
                                print "mosaic" + str(pair) + " to " + mosaic_temp + "with name " + str(output_name)
                            else:
                                #arcpy.MosaicToNewRaster_management(pair, mosaic_final, str(output_name), "", "8_BIT_UNSIGNED", "", "1", "LAST", "FIRST")
                                print "mosaic" + str(pair) + " to " + mosaic_temp + "with name " + str(output_name)
                    else:
                        for k in range(0, (vallen - 2), 2):
                            if k < (vallen - 3):
                                q = q + 1
                                del pair[:]
                                pair.append(val[k])
                                pair.append(val[k+1])
                                output_name = q
                                addstuff(key, str(output_name), inputdict)
                                if vallen > 3:
                                    #arcpy.MosaicToNewRaster_management(pair, mosaic_temp, str(output_name), "", "8_BIT_UNSIGNED", "", "1", "LAST", "FIRST")
                                    print "mosaic" + str(pair) + " to " + mosaic_temp + "with name " + str(output_name)
                                else:
                                    #arcpy.MosaicToNewRaster_management(pair, mosaic_final, str(output_name), "", "8_BIT_UNSIGNED", "", "1", "LAST", "FIRST")
                                    print "mosaic" + str(pair) + " to " + mosaic_final + "with name " + str(output_name)
                            else:
                                q = q + 1
                                del pair[:]
                                pair.append(val[k])
                                pair.append(val[k+1])
                                pair.append(val[k+2])
                                output_name = q
                                addstuff(key, str(output_name), inputdict)
                                #arcpy.MosaicToNewRaster_management(pair, mosaic_temp, str(output_name), "", "8_BIT_UNSIGNED", "", "1", "LAST", "FIRST")
                                print "mosaic" + str(pair) + " to " + mosaic_temp + "with name " + str(output_name)
            del maxi[:]
            for key, val in inputdict.items():
                vallen = len(val)
                maxi.append(vallen)
        print "done"
    else:
        print "nothing done since all indices contain maximum 1 file"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.