3

Are there any Open Data Cube (ODC) examples/documentation that cover indexing data residing on Google Cloud.

I know there is documentation on indexing data on AWS S3 at https://datacube-core.readthedocs.io/en/latest/ops/indexing.html#indexing-data-on-amazon-aws-s3

I am mainly exploring Google Cloud for Open Data Cube (ODC), not AWS.

3

Google Cloud Storage support was only added to datacube-core recently and hasn't been officially released yet, so the documentation for this capability needs some work. In the mean time I'll add the examples here.

In the DEA Proto repository there are some python scripts that allow you to index from google storage.

#!/bin/bash

# Google Storage support
gs-to-tar --bucket data.deadev.com --prefix mangrove_cover
dc-index-from-tar --env mangroves --ignore-lineage metadata.tar.gz

There are some dependency issues with installing datacube-core and DEA Proto, So I've put together the datacube-index repository that wraps dea-proto in a docker image for easier use. This docker compose file shows how it can be used to index from a Google Storage bucket:

version: '3'

services:
  postgres:
    image: postgres:10.3
    environment:
      - POSTGRES_DB=opendatacube
      - POSTGRES_PASSWORD=opendatacubepassword
      - POSTGRES_USER=opendatacube
    restart: always

  index:
    image: opendatacube/index:google
    environment:
      - WMS_CONFIG_URL=https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/dev/services/wms/ows/wms_cfg.py
      - PRODUCT_URLS=https://raw.githubusercontent.com/GeoscienceAustralia/dea-config/master/dev/products/mangrove/product-definition.yaml
      - DC_S3_INDEX_BUCKET=data.deadev.com
      - DC_S3_INDEX_PREFIX=mangrove_cover/-10_-18/
      - DC_S3_INDEX_SUFFIX=.yaml
      - DC_INDEX_PROTOCOL=gs
      - DC_RANGES_PRODUCT=mangrove_cover
      - DB_HOSTNAME=postgres
      - DB_USERNAME=opendatacube
      - DB_PASSWORD=opendatacubepassword
      - DB_DATABASE=opendatacube
      - DB_PORT=5432
    restart: on-failure
    command: 
      - /bin/bash
      - -c 
      - "cd index; ./create-index.sh"

When I put together some documentation on indexing from Google Cloud Storage I'll update this answer to reference it.

N.B. The docker image was built with the aim of indexing for datacube-ows, so it'll also create some additional tables in your database and store the extents of your data there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.