2

I have been trying to make an ODC index database state (products and their corresponding datasets) available (1) as a Docker image - probably kept locally due to its size - or (2) as a Docker volume. I've been restoring databases with compressed SQL files from pg_dump and I want to avoid pushing images that are 10s of GiB in size.

I tried using docker commit to implement option #1, and when running a container based on the resulting image, the size of the /var/lib/postgresql/data directory is the same as in the origin container and datacube system check perceives the connection to the container to be a valid one - so (1) the connection to the origin container and the container from the image from docker commit both appear to be valid to an ODC client - with datacube system check not requesting datacube system init to be run in either case and (2) have the same amount of data in Postgres (same data directory size).

However, for connections to the origin container, datacube product list shows the products and ODC queries in Python return the expected results. In contrast, for connections to the derived container (from docker commit), datacube product list shows no products and ODC queries perceive there to be no products as well.

I also tried option 2 - creating a Docker volume and using the same Postgres image for both - terminating the origin before launching the other container to use the data in the volume. The results are the same.

The ODC client for the origin and later Postgres containers have the same execution environments - same image, same launch settings - but they are in different docker-compose environments (that is, the first docker-compose environment sets up the container with the data, then shuts down, then the second docker-compose environment starts another Postgres container using the same image).

How should this be done? How can databases be restored and then reused elsewhere as Docker containers?

Improve this question

1 Answer 1

Reset to default
2

I don't think you should store application state in a Docker container. And DB data kind of fits this. There's a post here, which says it's not best practice.

using a Dockerfile allows you to formally modeling the image content and ensure you can rebuild/reproduce its initial state.

So, the right way to do this would be to fill the DB as part of your Dockerfile.

But that's not really appropriate either.

My view is that Docker is not a way to transfer data around. Some options are to use a volume to store your DB's content, and to move that as a folder between systems. But really, the best way to use Postgres is as a service from a cloud provider and the best way to move the Open Data Cube DB content around is to be able to index from scratch in a repeatable way.

Final comment is you're asking about how to do a technical piece of work and haven't defined your problem. Perhaps let us know what you're trying to achieve (i.e., are you wanting to replicate a DB between two servers? Whatever...). Let's think about what you're doing and why not how you're doing it!

Improve this answer
3
  • Thank you for your answer, Alex. We want to make an index database with a starting state available to users as part of a training environment. The users may index data in this environment at some point during training, so it needs to be isolated to them. Is it possible to use a single Dockerfile to do this? Would this require a multi-stage build, with the first part indexing with the indexing container - with postgres running in it - and then copying the Postgres data to a postgres container (i.e. FROM postgres:10-alpine for example) as multi-stage builds do?
    – John Rattz
    Commented May 22, 2021 at 4:53
  • 1
    You can do FROM postgres, like you say, then you need to do some crazy stuff, start postgres in the background, load data, all as docker run commands. It's not fun. I have an example I can dig up somewhere if you're really desperate!
    – Alex Leith
    Commented May 23, 2021 at 2:42
  • 1
    This is a better way to do it, @JohnRattz: github.com/opendatacube/cube-in-a-box
    – Alex Leith
    Commented May 23, 2021 at 2:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.