Having pretty much followed this Guide:
we can get large amounts of data indexed and ingested into our ODC. These data comprise 3 main products, locally generated aerial RGB, Sentinel and Landsat.
In all cases we have our data sets downloaded and prepared, have created our product definitions, generated our data set yamls, indexed these in the ODC and then run the ingest to build out the NetCDF's. However we are currently stuck at the 'Ingest new data Sets' stage...we can only achieve this cleanly by re-ingesting the whole data archive again for a particular product, there is some step we don't understand where we can't seem to just ingest the latest data set thats been indexed.. Its seems that when we receive a new data set and index this that a new record is written into the agdc.dataset table....which already has many records from previous index and ingest operations... However when we run the ingest command expecting it to pick up only the latest indexed data set it seems to pick up all previous indexed data set records and set off trying to ingest them all.