1

My problem: I am unable to use Open Data Cube's Python API on a client machine to load and operate on centrally stored products/data files (i.e. which reside on a remote computer).

What I have tried: As a first step, I was able use ODC to index/ingest local data files, then access the corresponding products using the Python API (from the same machine where the data is stored). This works as expected:

dc = datacube.Datacube(app='example-1')
prod = dc.load( latitude=latitude, 
                longitude=longitude,
                time=date_range,
                product=product,
                measurements=measurements)
print(prod)

Output:

<xarray.Dataset>
Dimensions: ...
Coordinates:
  ...
Data variables:
    ...
Attributes:
    ...

As a next step, I installed ODC on a VM and indexed/ingested some data files that were stored on that VM, then remotely accessed the corresponding Postgres database by using the Python API and setting the following parameters in the .datacube.conf file:

[testdev]
db_database: datacube
db_hostname: testdev.org
db_username: postgres
db_password: *****

I then tried the following:

dc = datacube.Datacube(app='example-2', env='testdev')
prod = dc.load( latitude=latitude, 
                longitude=longitude,
                time=date_range,
                product=product,
                measurements=measurements)
print(prod)

... which fails with:

Error opening source dataset: /vm/path/to/datafile.TIF

and

RasterioIOError: /vm/path/to/datafile.TIF: No such file or directory

However, I know that I am connected to the database on the VM because I can view ODC products and measurements with datacube.list_products() and datacube.list_measurements(), respectively, and can execute searches:

prods = dc.find_datasets(product=product) 
for p in prods:
    print(p)

Output:

Dataset <id=... type=... location=file:///vm/path/to/metadata-1.yaml>
Dataset <id=... type=... location=file:///vm/path/to/metadata-2.yaml>
Dataset <id=... type=... location=file:///vm/path/to/metadata-3.yaml>
Dataset <id=... type=... location=file:///vm/path/to/metadata-4.yaml>
Dataset <id=... type=... location=file:///vm/path/to/metadata-5.yaml>
Dataset <id=... type=... location=file:///vm/path/to/metadata-6.yaml>
Dataset <id=... type=... location=file:///vm/path/to/metadata-7.yaml>

While I can query the remote database for information about the data that's been indexed, I can't load the products themselves because the API is looking locally for data files that are stored on a different computer.

My question:

How do I configure ODC's Python API such that I can load and operate on remotely stored products?

Perhaps there is something special that I have to do when indexing the data..

It seems like a common use case, but scouring the documentation and trying different ways of indexing data have left me empty handed. Data replication and the datacube UI aren't quite what I'm looking for. I understand that there is a REST API in development that may be relevant, but in the meantime, I am wondering if the existing Python API can do what I want.

  • Is it possible to mount the remote drive on the virtual machine you'd like to access the data on? As far as I'm aware, it's not possible to load in the data if the files are not connected in some sense to the machine you are using. – JackLidge Aug 7 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.