2

I've ingested some landsat 5 images into a database using the open data cube. The problem is that when I try to run the continous change detection function following the jupyter notebook I get a TypeError: tuple indices must be integers or slices, not str.

%time ccd_product = ccd.process_xarray(landsat_dataset, distributed = True) #Run process xarray on large dataset

---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\envs\Toronto_ODC\lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
  File "C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py", line 363, in _ccd_product_from_pixel
ccd_product = _convert_ccd_results_into_dataset(results=ccd_results, model_dataset=pixel)
  File "C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py", line 110, in _convert_ccd_results_into_dataset
start_times = [datetime.fromordinal(model['start_day']) for model in results['change_models']]
  File "C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py", line 110, in <listcomp>
start_times = [datetime.fromordinal(model['start_day']) for model in results['change_models']]
TypeError: tuple indices must be integers or slices, not str
"""

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
<timed exec> in <module>

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in process_xarray(ds, distributed, process)
    459     }
    460 
--> 461     return processing_options[process]()
    462 
    463 

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in change_count()
    447         return _generate_change_matrix(ds, distributed = distributed)
    448     def change_count():
--> 449         return (generate_matrix().sum(dim='time') - 1).rename('change_volume')
    450     def first_change():
    451         return _nth_occurence_in_ccd_matrix(generate_matrix(),

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in generate_matrix()
    445     ### Declare several processing outputs.
    446     def generate_matrix():
--> 447         return _generate_change_matrix(ds, distributed = distributed)
    448     def change_count():
    449         return (generate_matrix().sum(dim='time') - 1).rename('change_volume')

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in _func(*params, **kwargs)
    273         logging.getLogger("ccd").setLevel(logging.WARNING)
    274         logging.getLogger("lcmap-pyccd").setLevel(logging.WARNING)
--> 275         result = function(*params, **kwargs)
    276         return result
    277 

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in _generate_change_matrix(ds, distributed)
    435     ccd_products = _ccd_product_iterator_from_pixels(pixels, distributed=distributed)
    436     ccd_products = filter(partial(is_not, None), ccd_products)
--> 437     ccd_change_count_xarray = _rebuild_xarray_from_pixels(ccd_products) # Change matrix
    438     return ccd_change_count_xarray
    439 

C:\envs\Toronto_ODC\data_cube_notebooks\utils\data_cube_utilities\dc_ccd.py in _rebuild_xarray_from_pixels(pixels)
    408 
    409     """
--> 410     return reduce(lambda x, y: x.combine_first(y), pixels)
    411 
    412 

C:\envs\Toronto_ODC\lib\multiprocessing\pool.py in next(self, timeout)
    729         if success:
    730             return value
--> 731         raise value
    732 
    733     __next__ = next                    # XXX

TypeError: tuple indices must be integers or slices, not str

I thought it might be one of my package issue within python but I've gone through and checked them.

I can load the images and makes rgb from my landsat_dataset so I know that I'm at least on the right track.

Any suggestions on how I can fix this error?

4

I'm not certain what the error is just yet. I have this (the CEOS wrapper for pyCCD) working on my machine and can relay details I think are relevant to pyccd.

Data

  • Landsat data should be Landsat collection-1 SR.
  • My functioning version of this algorithm uses data acquired from USGS earth explorer
    These bands should be included: [red,green,blue,nir,swir1,swir2,pixel_qa]
  • My functioning version has only been validated on Landsat 7 and Landsat 8, but should work with Landsat 5

lcmap-ccd dependency
The CCD I'm running uses lcmap-ccd version 2017.6.8. To re-install this version you can run:

pip install --upgrade lcmap-pyccd==2017.6.8

The Algorithm

Here is a rough description:

  • for each lat,lon pixel in the raster stack

    • for each band in the pixel

      • curve fit a 6 frequency signal to the first 12 or so observations in the pixel.

      • keep expanding the size of the window forward in your time series adding more observations to the model.

      • a change is characterized by new observations deviating from a stable model on one of the bands [red,green,blue,nir,swir1,swir1]. A rolling RMSE is the metric used to determine whether or not new ovbservations have deviated or broken the model

      • if a change occurs, ccd starts building a new model and needs a certain amount of new observations until it can be used to detect new changes.

Restrictions Imposed by PYCCD

  • Since a rolling RMSE is used detect change, pyCCD doesn't pick up on change immediately.

  • The minimum observation count for a model to be considered stable is around 12.

  • The model is used/validated on the scale of 150 - 300 observations/acquisitions. 5 scenes will not be enough for pyccd.

Performance
The issues with speed are a known thing. Newer versions of lcmap-pyccd might help, but aren't natively compatible with xarrays.

  • Processing times include about 180ms per pixel for a time series of over 150 observations.

  • There is also overhead incurred when distributing the workload to multiple cores.

  • Even a 400px * 400px can take a while.

Relevant Links

  • I actually had to update it to the latest version of the lcmap-pyccd==2018.10.17 to resolve the issue. Thanks your answer definitely helped me out! – Cam Apr 2 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.