3

Problem

For several days I've been trying to build an Ubuntu based GDAL distribution including the OCI driver (Oracle Spatial). The goal is to create a GDAL image including PostgreSQL and Oracle spatial drivers which is as small as possible. To my understanding, Oracle Spatial drivers are not included in the provided GDAL docker files from osgeo: https://github.com/OSGeo/gdal/tree/master/gdal/docker. Therefore I started to try to create an image myself, to no avail.

The desired output should be that the following command:

ogrinfo --formats

Includes all the usual drivers from ubuntu-small in addition to OCI. How do I enable the OCI driver on a Ubuntu Docker image based on the official GDAL docker repository? This issue has been taunting me to the point that it is becoming personal.

Resources

The only docker resource I found on this specific topic was this Github repo: https://github.com/fegyi001/docker-gdal-oci which creates a huge 5gb+ image using ubuntu:20.04. However, it did provide me with some pointers to how I should include OCI in GDAL.

Other resources on this topic are:

What I have done so far

I copied the files from this repo: https://github.com/OSGeo/gdal/tree/master/gdal/docker/ubuntu-small + the util.sh in the directory above. I edited the Dockerfile:

I suspect that this has something to do with wrong / absent file links to the Oracle Instant Client. It could also have something to do with the LDFLAGS during the GDAL build proces. However, I have no clue how to debug this as I can't find any log files and I am unsure what files GDAL exactly expects in the ./configure step.

My edited Dockerfile looks like this:

##
# osgeo/gdal:ubuntu-small

# Build instruction usage: TARGET_IMAGE="dockername/tracebepaling" docker_build/build.sh --release --gdal v3.2.0 --proj master

# This file is available at the option of the licensee under:
# Public domain
# or licensed under X/MIT (LICENSE.TXT) Copyright 2019 Even Rouault <[email protected]>

ARG PROJ_INSTALL_PREFIX=/usr/local
ARG BASE_IMAGE=ubuntu:20.04

FROM $BASE_IMAGE as builder

# Derived from osgeo/proj by Howard Butler <[email protected]>
LABEL maintainer="Even Rouault <[email protected]>"

# Setup build env for PROJ
USER root
RUN apt-get update -y \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y --fix-missing --no-install-recommends \
            software-properties-common build-essential ca-certificates \
            git make cmake wget unzip libtool automake \
            zlib1g-dev libsqlite3-dev pkg-config sqlite3 libcurl4-gnutls-dev \
            libtiff5-dev

# Setup build env for GDAL
RUN apt-get update -y \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y --fix-missing --no-install-recommends \
       python3-dev python3-numpy \
       libjpeg-dev libgeos-dev \
       libexpat-dev libxerces-c-dev \
       libwebp-dev \
       libzstd-dev bash zip curl \
       libpq-dev libssl-dev libopenjp2-7-dev \
       autoconf automake sqlite3 bash-completion

# Build openjpeg
ARG OPENJPEG_VERSION=
RUN if test "${OPENJPEG_VERSION}" != ""; then ( \
    wget -q https://github.com/uclouvain/openjpeg/archive/v${OPENJPEG_VERSION}.tar.gz \
    && tar xzf v${OPENJPEG_VERSION}.tar.gz \
    && rm -f v${OPENJPEG_VERSION}.tar.gz \
    && cd openjpeg-${OPENJPEG_VERSION} \
    && cmake . -DBUILD_SHARED_LIBS=ON  -DBUILD_STATIC_LIBS=OFF -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_INSTALL_PREFIX=/usr \
    && make -j$(nproc) \
    && make install \
    && mkdir -p /build_thirdparty/usr/lib \
    && cp -P /usr/lib/libopenjp2*.so* /build_thirdparty/usr/lib \
    && for i in /build_thirdparty/usr/lib/*; do strip -s $i 2>/dev/null || /bin/true; done \
    && cd .. \
    && rm -rf openjpeg-${OPENJPEG_VERSION} \
    ); fi

ARG PROJ_INSTALL_PREFIX
ARG PROJ_DATUMGRID_LATEST_LAST_MODIFIED
RUN \
    mkdir -p /build_projgrids/${PROJ_INSTALL_PREFIX}/share/proj \
    && curl -LOs http://download.osgeo.org/proj/proj-datumgrid-latest.zip \
    && unzip -q -j -u -o proj-datumgrid-latest.zip  -d /build_projgrids/${PROJ_INSTALL_PREFIX}/share/proj \
    && rm -f *.zip

RUN apt-get update -y \
    && apt-get install -y --fix-missing --no-install-recommends rsync ccache
ARG RSYNC_REMOTE

# Build PROJ
ARG PROJ_VERSION=master
RUN mkdir proj \
    && wget -q https://github.com/OSGeo/PROJ/archive/${PROJ_VERSION}.tar.gz -O - \
        | tar xz -C proj --strip-components=1 \
    && cd proj \
    && ./autogen.sh \
    && if test "${RSYNC_REMOTE}" != ""; then \
        echo "Downloading cache..."; \
        rsync -ra ${RSYNC_REMOTE}/proj/ $HOME/; \
        echo "Finished"; \
        export CC="ccache gcc"; \
        export CXX="ccache g++"; \
        export PROJ_DB_CACHE_DIR="$HOME/.ccache"; \
        ccache -M 100M; \
    fi \
    && CFLAGS='-DPROJ_RENAME_SYMBOLS -O2' CXXFLAGS='-DPROJ_RENAME_SYMBOLS -DPROJ_INTERNAL_CPP_NAMESPACE -O2' \
        ./configure --prefix=${PROJ_INSTALL_PREFIX} --disable-static \
    && make -j$(nproc) \
    && make install DESTDIR="/build" \
    && if test "${RSYNC_REMOTE}" != ""; then \
        ccache -s; \
        echo "Uploading cache..."; \
        rsync -ra --delete $HOME/.ccache ${RSYNC_REMOTE}/proj/; \
        echo "Finished"; \
        rm -rf $HOME/.ccache; \
        unset CC; \
        unset CXX; \
    fi \
    && cd .. \
    && rm -rf proj \
    && PROJ_SO=$(readlink /build${PROJ_INSTALL_PREFIX}/lib/libproj.so | sed "s/libproj\.so\.//") \
    && PROJ_SO_FIRST=$(echo $PROJ_SO | awk 'BEGIN {FS="."} {print $1}') \
    && mv /build${PROJ_INSTALL_PREFIX}/lib/libproj.so.${PROJ_SO} /build${PROJ_INSTALL_PREFIX}/lib/libinternalproj.so.${PROJ_SO} \
    && ln -s libinternalproj.so.${PROJ_SO} /build${PROJ_INSTALL_PREFIX}/lib/libinternalproj.so.${PROJ_SO_FIRST} \
    && ln -s libinternalproj.so.${PROJ_SO} /build${PROJ_INSTALL_PREFIX}/lib/libinternalproj.so \
    && rm /build${PROJ_INSTALL_PREFIX}/lib/libproj.*  \
    && ln -s libinternalproj.so.${PROJ_SO} /build${PROJ_INSTALL_PREFIX}/lib/libproj.so.${PROJ_SO_FIRST} \
    && strip -s /build${PROJ_INSTALL_PREFIX}/lib/libinternalproj.so.${PROJ_SO} \
    && for i in /build${PROJ_INSTALL_PREFIX}/bin/*; do strip -s $i 2>/dev/null || /bin/true; done

## install oracle
RUN apt-get -y install wget libaio1
RUN wget https://download.oracle.com/otn_software/linux/instantclient/211000/instantclient-basic-linux.x64-21.1.0.0.0.zip && \
  wget https://download.oracle.com/otn_software/linux/instantclient/211000/instantclient-sqlplus-linux.x64-21.1.0.0.0.zip && \
  wget https://download.oracle.com/otn_software/linux/instantclient/211000/instantclient-sdk-linux.x64-21.1.0.0.0.zip
RUN unzip instantclient-basic-linux.x64-21.1.0.0.0.zip -d /usr && \
    unzip instantclient-sqlplus-linux.x64-21.1.0.0.0.zip -d /usr && \
    unzip instantclient-sdk-linux.x64-21.1.0.0.0.zip -d /usr
ENV ORACLE_HOME=/usr/instantclient_21_1
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/instantclient_21_1
ENV PATH=$PATH:$ORACLE_HOME
# Create symbolic links?
#RUN ln -s /usr/instantclient_21_1/libclntsh.so.21.1 libclntsh.so && \
#    ln -s /usr/instantclient_21_1/libocci.so.21.1 libocci.so
RUN ldconfig

# Build GDAL
ARG GDAL_VERSION=master
ARG GDAL_RELEASE_DATE
ARG GDAL_BUILD_IS_RELEASE
ARG GDAL_REPOSITORY=OSGeo/gdal

RUN if test "${GDAL_VERSION}" = "master"; then \
        export GDAL_VERSION=$(curl -Ls https://api.github.com/repos/${GDAL_REPOSITORY}/commits/HEAD -H "Accept: application/vnd.github.VERSION.sha"); \
        export GDAL_RELEASE_DATE=$(date "+%Y%m%d"); \
    fi \
    && if test "x${GDAL_BUILD_IS_RELEASE}" = "x"; then \
        export GDAL_SHA1SUM=${GDAL_VERSION}; \
    fi \
    && mkdir gdal \
    && wget -q https://github.com/${GDAL_REPOSITORY}/archive/${GDAL_VERSION}.tar.gz -O - \
        | tar xz -C gdal --strip-components=1 \
    && cd gdal/gdal \
    && if test "${RSYNC_REMOTE}" != ""; then \
        echo "Downloading cache..."; \
        rsync -ra ${RSYNC_REMOTE}/gdal/ $HOME/; \
        echo "Finished"; \
        # Little trick to avoid issues with Python bindings
        printf "#!/bin/sh\nccache gcc \$*" > ccache_gcc.sh; \
        chmod +x ccache_gcc.sh; \
        printf "#!/bin/sh\nccache g++ \$*" > ccache_g++.sh; \
        chmod +x ccache_g++.sh; \
        export CC=$PWD/ccache_gcc.sh; \
        export CXX=$PWD/ccache_g++.sh; \
        ccache -M 1G; \
    fi \
    && LDFLAGS="-L/build${PROJ_INSTALL_PREFIX}/lib -linternalproj" ./configure --prefix=/usr --without-libtool \
    --with-hide-internal-symbols \
    --with-jpeg12 \
    --with-python \
    --with-webp --with-proj=/build${PROJ_INSTALL_PREFIX} \
    --with-libtiff=internal --with-rename-internal-libtiff-symbols \
    --with-geotiff=internal --with-rename-internal-libgeotiff-symbols \

    ### enable OCI
    --with-oci=yes \
    --with-oci-lib=/usr/instantclient_21_1 \
    --with-oci-include=/usr/instantclient_21_1/sdk/include \

    && make -j$(nproc) \
    && make install DESTDIR="/build" \
    && if test "${RSYNC_REMOTE}" != ""; then \
        ccache -s; \
        echo "Uploading cache..."; \
        rsync -ra --delete $HOME/.ccache ${RSYNC_REMOTE}/gdal/; \
        echo "Finished"; \
        rm -rf $HOME/.ccache; \
        unset CC; \
        unset CXX; \
    fi \
    && cd ../.. \
    && rm -rf gdal \
    && mkdir -p /build_gdal_python/usr/lib \
    && mkdir -p /build_gdal_python/usr/bin \
    && mkdir -p /build_gdal_version_changing/usr/include \
    && mv /build/usr/lib/python*            /build_gdal_python/usr/lib \
    && mv /build/usr/lib                    /build_gdal_version_changing/usr \
    && mv /build/usr/include/gdal_version.h /build_gdal_version_changing/usr/include \
    && mv /build/usr/bin/*.py               /build_gdal_python/usr/bin \
    && mv /build/usr/bin                    /build_gdal_version_changing/usr \
    && for i in /build_gdal_version_changing/usr/lib/*; do strip -s $i 2>/dev/null || /bin/true; done \
    && for i in /build_gdal_python/usr/lib/python3/dist-packages/osgeo/*.so; do strip -s $i 2>/dev/null || /bin/true; done \
    && for i in /build_gdal_version_changing/usr/bin/*; do strip -s $i 2>/dev/null || /bin/true; done

### Edited, this part should enable ogrinfo during the build stage.

# PROJ dependencies
RUN apt-get update; \
    DEBIAN_FRONTEND=noninteractive apt-get install -y  --no-install-recommends \
        libsqlite3-0 libtiff5 libcurl4 \
        curl unzip ca-certificates

 GDAL dependencies
RUN apt-get update -y; \
    DEBIAN_FRONTEND=noninteractive apt-get install -y  --no-install-recommends \
        python3-numpy libpython3.8 \
        libjpeg-turbo8 libgeos-3.8.0 libgeos-c1v5 \
        libexpat1 \
        libxerces-c3.2 \
        libwebp6 \
        libzstd1 bash libpq5 libssl1.1 libopenjp2-7

RUN cp -R /build_projgrids/usr/ /usr/
RUN cp -R /build/usr/local/share/proj/ /usr/local/share/proj/
RUN cp -R /build/usr/local/include/* /usr/local/include/
RUN cp -R /build/usr/local/bin/* /usr/local/bin/
RUN cp -R /build/usr/local/lib/* /usr/local/lib/
RUN cp -R /build/usr/share/gdal/ /usr/share/gdal/
RUN cp -R /build/usr/include/ /usr/include/
RUN cp -R /build_gdal_python/usr/bin/* /usr/bin/
RUN cp -R /build_gdal_python/usr/lib/python3/dist-packages/* /usr/lib/python3/dist-packages/
RUN cp -R /build_gdal_version_changing/usr/bin/* /usr/bin/
RUN cp -R /build_gdal_version_changing/usr/include/* /usr/include/
RUN cp -R /build_gdal_version_changing/usr/lib/* /usr/lib/

RUN ldconfig

### End of edit

# Build final image
FROM $BASE_IMAGE as runner

USER root
RUN date

# PROJ dependencies
RUN apt-get update; \
    DEBIAN_FRONTEND=noninteractive apt-get install -y  --no-install-recommends \
        libsqlite3-0 libtiff5 libcurl4 \
        curl unzip ca-certificates

# GDAL dependencies
RUN apt-get update -y; \
    DEBIAN_FRONTEND=noninteractive apt-get install -y  --no-install-recommends \
        python3-numpy libpython3.8 \
        libjpeg-turbo8 libgeos-3.8.0 libgeos-c1v5 \
        libexpat1 \
        libxerces-c3.2 \
        libwebp6 \
        libzstd1 bash libpq5 libssl1.1 libopenjp2-7

RUN apt-get update; \
    DEBIAN_FRONTEND=noninteractive apt-get install -y python-is-python3

# Order layers starting with less frequently varying ones
# Only used for custom libopenjp2
# COPY --from=builder  /build_thirdparty/usr/ /usr/

COPY --from=builder  /build_projgrids/usr/ /usr/

ARG PROJ_INSTALL_PREFIX
COPY --from=builder  /build${PROJ_INSTALL_PREFIX}/share/proj/ ${PROJ_INSTALL_PREFIX}/share/proj/
COPY --from=builder  /build${PROJ_INSTALL_PREFIX}/include/ ${PROJ_INSTALL_PREFIX}/include/
COPY --from=builder  /build${PROJ_INSTALL_PREFIX}/bin/ ${PROJ_INSTALL_PREFIX}/bin/
COPY --from=builder  /build${PROJ_INSTALL_PREFIX}/lib/ ${PROJ_INSTALL_PREFIX}/lib/

COPY --from=builder  /build/usr/share/gdal/ /usr/share/gdal/
COPY --from=builder  /build/usr/include/ /usr/include/
COPY --from=builder  /build_gdal_python/usr/ /usr/
COPY --from=builder  /build_gdal_version_changing/usr/ /usr/
COPY --from=builder  /usr/instantclient_21_1 /usr/instantclient_21_1

RUN ldconfig
6
  • What happens during the build ? Do you see any errors when trying to build the OCI driver ? I also saw this: with-oci-lib=usr/instantclient_21_1. I don't think using a relative path is a good idea. I'd rather use a fixed path. Jun 28, 2021 at 7:50
  • Check also what packages of the Instant Client you downloaded. You need the Basic and SDK packages. And I would also include the SQL*Plus and Tools packages - they may come handy (and the whole thing is very small anyway). I would also make sure the Instant Client is correctly installed and works, for example by using sql plus to test connection to one of your databases. Jun 28, 2021 at 7:54
  • Thanks for replying. I see I made a type regarding the paths to Oracle and apparently you need to install the "libaio1" package or sqlplus will not work. I edited my question to reflect the current Dockerfile. Sqlplus is now working in the build stage. However, OCI is not enabled. I do not see any errors from GDAL side. I downloaded the basic/sdk/sqplplus modules. Is there any way I can check some sort of build log file?
    – Web_GIS
    Jun 28, 2021 at 20:14
  • When you say "OCI is not enabled" you mean it does not appear in the ogrinfo --formats output ? Jul 1, 2021 at 10:03
  • Yes unfortunately.
    – Web_GIS
    Jul 1, 2021 at 15:10

2 Answers 2

0

I have done GDAL builds in the past (long time ago) and it has always been a rocky path. One thing I remember is that the "optional" plugins needed to go to a specific location ($GDAL_HOME/lib/gdalplugins).

I'm using the GDAL build that comes with the Oracle database installation since 19c. If you have a 19c installation handy, you will find it here:

# ls $ORACLE_HOME/md/gdal
bin  data  gdal24_linux64.zip  gdal24_win64.zip  include  lib  LICENSE.TXT  README.TXT  setup_gdal.conf  setup_hdfs_gdal.conf

Oracle has been doing this build for many years, and it was distributed via Oracle Technical Resources (software downloads). Since 19c it is also distributed inside the Oracle kits. Or you will also find it here: https://www.oracle.com/downloads/samplecode/spatial-samplecode-downloads.html (download file GeoRasterTool.zip). There are two builds (one for Linux64, one for Win64), and both contain the Oracle OCI and the Oracle Georaster plugins ...

... but no PostGIS plugin.

That does not help you much, other than maybe giving indications as to how to construct your build:

  1. The "optional" plugins (so also the OCI plugin) are in the $GDAL_HOME/lib/gdalplugins directory. Maybe that is what you miss in your build ?

  2. The environment must be setup to include this directory location (GDAL_DRIVER_PATH). Here is the setup I use for that:

# grep GDAL ~/.bash_profile
# GDAL setup
# Use the GDAL version of the database
export GDAL_HOME=$ORACLE_HOME/md/gdal
export PATH=$GDAL_HOME/bin:$PATH
export GDAL_DATA=$GDAL_HOME/data
export LD_LIBRARY_PATH=$GDAL_HOME/lib:$LD_LIBRARY_PATH
export GDAL_DRIVER_PATH=$GDAL_HOME/lib/gdalplugins

Maybe that will help you with your build. I don't think you can just grab the ogr_OCI.solibrary from the Oracle build and inject it into an existing build because the Oracle build is on GDAL 2.4.0, which is a bit old.

EDIT

After looking closer at our Oracle builds, I just noticed that we actually do include a build of the PostGIS driver:

# ls -l $GDAL_HOME/lib/gdalplugins
total 1068
-rw-r--r--. 1 oracle oinstall  62094 Jan 25  2019 gdal_ECW_JP2ECW.zip
-rw-r--r--. 1 oracle oinstall 217800 Jan 25  2019 gdal_GEOR.so
-rw-r--r--. 1 oracle oinstall 239970 Jan 25  2019 gdal_HDF4.zip
-rw-r--r--. 1 oracle oinstall 128000 Jan 25  2019 gdal_HDF5.zip
-rw-r--r--. 1 oracle oinstall  94056 Jan 25  2019 gdal_JP2OpenJPEG.so
-rw-r--r--. 1 oracle oinstall  82601 Jan 25  2019 ogr_FileGDB.zip
-rw-r--r--. 1 oracle oinstall 166408 Jan 25  2019 ogr_OCI.so
-rw-r--r--. 1 oracle oinstall  78236 Jan 25  2019 ogr_PG.zip
-rw-r--r--. 1 oracle oinstall    191 Jan 25  2019 README

See the ogr_PG.zip file ? All you need is to unzip it in the same place. This will give ogr_PG.so, the PostGIS plugin. I have not tried using it - I do not have any PostGIS installation, so I can't tell if it works as expected. But it is worth trying out.

3
  • Thanks for your elaborate answer. However I'm unsure how this will help me in proces of creating a Docker build with OCI. Something I did notice from the 19c build was that something changed in the GDAL >3 which changes folder structure and arguments for building OCI. Less path variables are necessary. Also, I'm starting to think that GDAL does not accept instantclient 21, only <19.3.
    – Web_GIS
    Jul 6, 2021 at 11:45
  • My suggestion was that you could just use the oracle provided build that includes both OCI and PostGIS plugins - i.e. should be ready for use. It saves you building the entire GDAL from scratch. I was not aware that the directory structure changed for GDAL 3.x. Jul 6, 2021 at 17:39
  • As for instant client 21 ... I can't tell. But yes, you could try using instant client 19.x Jul 6, 2021 at 17:40
0

Found the answer in this question and the respective Dockerfile.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.