My problem: I am unable to use Open Data Cube's Python API on a client machine to load and operate on centrally stored products/data files (i.e. which reside on a remote computer).
What I have tried: As a first step, I was able use ODC to index/ingest local data files, then access the corresponding products using the Python API (from the same machine where the data is stored). This works as expected:
dc = datacube.Datacube(app='example-1') prod = dc.load( latitude=latitude, longitude=longitude, time=date_range, product=product, measurements=measurements) print(prod)
<xarray.Dataset> Dimensions: ... Coordinates: ... Data variables: ... Attributes: ...
As a next step, I installed ODC on a VM and indexed/ingested some data files that were stored on that VM, then remotely accessed the corresponding Postgres database by using the Python API and setting the following parameters in the
[testdev] db_database: datacube db_hostname: testdev.org db_username: postgres db_password: *****
I then tried the following:
dc = datacube.Datacube(app='example-2', env='testdev') prod = dc.load( latitude=latitude, longitude=longitude, time=date_range, product=product, measurements=measurements) print(prod)
... which fails with:
Error opening source dataset: /vm/path/to/datafile.TIF
RasterioIOError: /vm/path/to/datafile.TIF: No such file or directory
However, I know that I am connected to the database on the VM because I can view ODC products and measurements with
datacube.list_measurements(), respectively, and can execute searches:
prods = dc.find_datasets(product=product) for p in prods: print(p)
Dataset <id=... type=... location=file:///vm/path/to/metadata-1.yaml> Dataset <id=... type=... location=file:///vm/path/to/metadata-2.yaml> Dataset <id=... type=... location=file:///vm/path/to/metadata-3.yaml> Dataset <id=... type=... location=file:///vm/path/to/metadata-4.yaml> Dataset <id=... type=... location=file:///vm/path/to/metadata-5.yaml> Dataset <id=... type=... location=file:///vm/path/to/metadata-6.yaml> Dataset <id=... type=... location=file:///vm/path/to/metadata-7.yaml>
While I can query the remote database for information about the data that's been indexed, I can't load the products themselves because the API is looking locally for data files that are stored on a different computer.
How do I configure ODC's Python API such that I can load and operate on remotely stored products?
Perhaps there is something special that I have to do when indexing the data..
It seems like a common use case, but scouring the documentation and trying different ways of indexing data have left me empty handed. Data replication and the datacube UI aren't quite what I'm looking for. I understand that there is a REST API in development that may be relevant, but in the meantime, I am wondering if the existing Python API can do what I want.