TypeError: tuple indices must be integers or slices, not str after ingested data into the open data cube

Question

I've ingested some landsat 5 images into a database using the open data cube. The problem is that when I try to run the continous change detection function following the jupyter notebook I get a TypeError: tuple indices must be integers or slices, not str.

%time ccd_product = ccd.process_xarray(landsat_dataset, distributed = True) #Run process xarray on large dataset

---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\envs\Toronto_ODC\lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
  File "C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py", line 363, in _ccd_product_from_pixel
ccd_product = _convert_ccd_results_into_dataset(results=ccd_results, model_dataset=pixel)
  File "C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py", line 110, in _convert_ccd_results_into_dataset
start_times = [datetime.fromordinal(model['start_day']) for model in results['change_models']]
  File "C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py", line 110, in <listcomp>
start_times = [datetime.fromordinal(model['start_day']) for model in results['change_models']]
TypeError: tuple indices must be integers or slices, not str
"""

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
<timed exec> in <module>

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in process_xarray(ds, distributed, process)
    459     }
    460 
--> 461     return processing_options[process]()
    462 
    463 

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in change_count()
    447         return _generate_change_matrix(ds, distributed = distributed)
    448     def change_count():
--> 449         return (generate_matrix().sum(dim='time') - 1).rename('change_volume')
    450     def first_change():
    451         return _nth_occurence_in_ccd_matrix(generate_matrix(),

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in generate_matrix()
    445     ### Declare several processing outputs.
    446     def generate_matrix():
--> 447         return _generate_change_matrix(ds, distributed = distributed)
    448     def change_count():
    449         return (generate_matrix().sum(dim='time') - 1).rename('change_volume')

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in _func(*params, **kwargs)
    273         logging.getLogger("ccd").setLevel(logging.WARNING)
    274         logging.getLogger("lcmap-pyccd").setLevel(logging.WARNING)
--> 275         result = function(*params, **kwargs)
    276         return result
    277 

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in _generate_change_matrix(ds, distributed)
    435     ccd_products = _ccd_product_iterator_from_pixels(pixels, distributed=distributed)
    436     ccd_products = filter(partial(is_not, None), ccd_products)
--> 437     ccd_change_count_xarray = _rebuild_xarray_from_pixels(ccd_products) # Change matrix
    438     return ccd_change_count_xarray
    439 

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in _rebuild_xarray_from_pixels(pixels)
    408 
    409     """
--> 410     return reduce(lambda x, y: x.combine_first(y), pixels)
    411 
    412 

C:\envs\Toronto_ODC\lib\multiprocessing\pool.py in next(self, timeout)
    729         if success:
    730             return value
--> 731         raise value
    732 
    733     __next__ = next                    # XXX

TypeError: tuple indices must be integers or slices, not str

I thought it might be one of my package issue within python but I've gone through and checked them.

I can load the images and makes rgb from my landsat_dataset so I know that I'm at least on the right track.

Any suggestions on how I can fix this error?

Answer 1

I'm not certain what the error is just yet. I have this (the CEOS wrapper for pyCCD) working on my machine and can relay details I think are relevant to pyccd.

Data

• Landsat data should be Landsat collection-1 SR.
• My functioning version of this algorithm uses data acquired from USGS earth explorer
  These bands should be included: [red,green,blue,nir,swir1,swir2,pixel_qa]
• My functioning version has only been validated on Landsat 7 and Landsat 8, but should work with Landsat 5

lcmap-ccd dependency
The CCD I'm running uses lcmap-ccd version 2017.6.8. To re-install this version you can run:

pip install --upgrade lcmap-pyccd==2017.6.8

The Algorithm

Here is a rough description:

• for each lat,lon pixel in the raster stack

  • for each band in the pixel

    • curve fit a 6 frequency signal to the first 12 or so observations in the pixel.

    • keep expanding the size of the window forward in your time series adding more observations to the model.

    • a change is characterized by new observations deviating from a stable model on one of the bands [red,green,blue,nir,swir1,swir1]. A rolling RMSE is the metric used to determine whether or not new ovbservations have deviated or broken the model

    • if a change occurs, ccd starts building a new model and needs a certain amount of new observations until it can be used to detect new changes.

Restrictions Imposed by PYCCD

• Since a rolling RMSE is used detect change, pyCCD doesn't pick up on change immediately.

• The minimum observation count for a model to be considered stable is around 12.

• The model is used/validated on the scale of 150 - 300 observations/acquisitions. 5 scenes will not be enough for pyccd.

Performance
The issues with speed are a known thing. Newer versions of lcmap-pyccd might help, but aren't natively compatible with xarrays.

• Processing times include about 180ms per pixel for a time series of over 150 observations.

• There is also overhead incurred when distributing the workload to multiple cores.

• Even a 400px * 400px can take a while.

Relevant Links

• CEOS xarrays wrapper for pyccd

• lcmap-pyccd