Process every 100 records in list

Question

An arcpy merge process is failing due to the size of the datasets and I need to just merge every 100 tiles before it goes on to the next 100 tiles.

I wrote the following snippet but it is just putting the same tif into the list. It's probably something simple that I am doing wrong but would appreciate your advice....

arcpy.env.workspace=root+"\\Stage1\\"+name+"\\GeoTif"
tifList=arcpy.ListRasters('*.tif')
tifListCount=len(tifList)

tempTifList=[]
x=0
oldx=0
while x <= tifListCount:
    for tif in tifList:
        while x<(oldx+100):
            tempTifList.append(tif)
            x=x+1

        print "List "+str(oldx)+" is"+ str(tempTifList)
        # Add merge process on tempTifList
        tempTifList=[]
        oldx=x

Gives

, u'PFTI_Roads2_Z31.tif', u'PFTI_Roads2_Z31.tif', u'PFTI_Roads2_Z31.tif', u'PFTI_Roads2_Z31.tif', u'PFTI_Roads2_Z31.tif', u'PFTI_Roads2_Z31.tif', u'PFTI_Roads2_Z31.tif', u'PFTI_Roads2_Z31.tif', u'PFTI_Roads2_Z31.tif', u'PFTI_Roads2_Z31.tif'] List 75900 is[u'PFTI_Roads2_Z32.tif', u'PFTI_Roads2_Z32.tif', u'PFTI_Roads2_Z32.tif', u'PFTI_Roads2_Z32.tif', u'PFTI_Roads2_Z32.tif', u'PFTI_Roads2_Z32.tif', u'PFTI_Roads2_Z32.tif', u'PFTI_Roads2_Z32.tif', u'PFTI_Roads2_Z32.tif', u'PFTI_Roads2_Z32.tif

Answer 1

You have erroneously nested a loop that is iterating over the same tif - i.e., basically on the innermost you have: while x is less than 100, loop on the tif list, increment x by 1

So the while loop executes inside your 'for' loop, starting with the 1st tif. The intended mechanism of the 'for' loop doesn't hand off successive list tifs because x has to run until reaching its limit of 100.

Then, probably do something like this instead, since ListRasters returns a Python list-- EDIT: The 1st code I posted was untested and didn't function as I expected. Properly working code is posted below (and there is likely a more straightforward way, but at any rate it works):

x = 1
for i in range(0, tifListCount, 100):
    for j in range(i, 100*x):
        if j + 1 <= tifListCount:
            tempTifList.append(tifList[j])
    print tempTifList
    # do your processing here
    tempTifList = []
    x += 1

Answer 2

This StackOverflow answer has a more concise, Pythonic suggestion:

>>> print list(izip_longest(*(iter(range(10)),) * 3))
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, None, None)]

You could incorporate it like this:

from itertools import izip_longest
for i, group in enumerate(izip_longest(*(iter(tifList),) * 100)):
    group = filter(None, group) # Remove None objects from group
    #Merge group here