0

I broke Google Colab with GDAL and Anaconda, how do I repair it?

So hear my story: In a project I need to use the RSGISLib Python library (link) developed for object-based analysis of georeferenced images.

I also should work on google colaboratory for the sake of ease of access for all project members.

RSGISLib is only available on conda repository...

I managed to install Conda on Colab and get it running, then my notebook correctly finds the RSGISLib functions and executes them. So far so good. RSGISLib relies on GDAL for creation of spatial datasets. One of necessary things to get when creating such dataset is spatial projection, provided in GDAL by the PROJ library.

Once my notebook reaches a RSGISLib command, which needs PROJ at some point the entire Kernel/Linux/runtime (not sure really) of the Colab spectacularly crashes and I have to factory reset the runtime. The logs look like this, that's why I know it's a PROJ problem:

Jun 9, 2020, 12:09:38 PM    WARNING WARNING:root:kernel 11aacb5e-4fc8-4d62-8178-841f9a287a02 restarted
Jun 9, 2020, 12:09:38 PM    INFO    KernelRestarter: restarting kernel (1/5), keep random ports
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: proj_create_from_database: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: proj_create_from_wkt: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: proj_as_wkt: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: proj_as_wkt: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: pj_obj_create: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: proj_create_from_wkt: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: proj_create_from_wkt: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: proj_as_wkt: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: pj_obj_create: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: createGeodeticReferenceFrame: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: proj_as_wkt: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: createGeodeticReferenceFrame: Open of /usr/condaFolder/share/proj failed
Jun 9, 2020, 12:09:37 PM    WARNING ERROR 1: PROJ: pj_obj_create: Open of /usr/condaFolder/share/proj failed

Nothing nice, as you can guess. As I googled, I learned, that the reason is that PROJ when run from Anaconda requires an environment variable to be passed to it somehow for some reason. I thought "fine, I'll just:"

!conda activate base

Oh no, I wont:

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

!conda init bash doesn't work, it just doesn't change a thing. Funnily enough, when asked

!/usr/condaFolder/bin/conda info --envs

my on-line Jupyter gladly responds, that:

# conda environments:
#
base                  *  /usr/condaFolder

so technically I'm working in an active environment and there shouldn't be a problem... Yet there is. Currently I gave up on Colab and moved my notebook to a Conda environment on the local drive, but if you know how to get these things running please let me know.

2

After some stressful fight I managed to patch up a solution. I post the answer below because I believe a lot of people will hit the same problem trying to do geospatial work with Conda in Google Colab:

The PROJ library requires an environment variable, named PROJ_LIB, which defines its location for all other modules. Usually in Conda this is set when conda activate my-env-name is called. When Conda is run in Colab, though, an environment is active, but is not activated by a command call. It the activation event does not happen the PROJ variable doesn't get set and the whole system crashes. Therefore we need to define PROJ_LIB manually.

Place the following line at the beginning of your script, after you set up Conda:

![path to conda]/conda env config vars set PROJ_LIB = [path to proj library folder]

Example from my own implementation:

!/usr/condaFolder/bin/conda env config vars set PROJ_LIB=/usr/condaFolder/pkgs/proj-6.2.1-hc80f0dc_0

This is not everything though. If you build a Conda with RSGISLib (or any other geospatial library) in Colab you run into a conflict with the Google Colab's natively installed GDAL. There is one "default" that comes with the Colab runtime and one you shipped with whatever GIS library you smuggled with Conda. Therefore certain functions, including those of RSGISLib, sometimes get distracted as to which GDAL should be used - and the wrong one might not have a driver which RSGISLib needs for its KEA image format. Therefore to make sure things run smoothly you have to remove the native Colab's GDAL. It's enough to delete all files from the folder /usr/lib/python3/dist-packages/osgeo with !rm. I have done it and ran a few different GIS libs in the same notebook and everything was working fine (because my Conda build brings OSGeo libs, on which anything Colab has can rely).

| improve this answer | |
  • This helped me a lot solving a similar issue with Conda and PROJ in JupyterLab. The solution is to execute import os; os.environ['PROJ_LIB'] = '/path/to/env/share/proj' at the top of the cell. – Stefan Oct 6 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.