GDAL loads in Python, but not IPython (Linux)

Question

I am trying my hardest to utilize the work of the good folks at the GeoDa center, but I cannot seem to get all of the components to load. GDAL, in particular, has been giving me some difficulty. The biggest issue arises because I am operating within the IPython Notebook. If you are unfamiliar with the Notebook, to say it's worth a look is the understatement of the century.

In any event, after some difficulty and some assistance, I was able to install the GDAL module. It was even importable at the Python prompt:

Python 2.7.5 |Anaconda 1.6.1 (64-bit)| (default, Jun 28 2013, 22:10:09) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gdal
>>> 

However, after my difficulties in the Notebook, I went back to the IPython prompt to discover the following difficulty:

Python 2.7.5 |Anaconda 1.6.1 (64-bit)| (default, Jun 28 2013, 22:10:09) 
Type "copyright", "credits" or "license" for more information.

IPython 1.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import gdal
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-27bf4694dd2b> in <module>()
----> 1 import gdal

/home/choct155/analysis/Anaconda/lib/python2.7/site-packages/gdal.py in <module>()
      1 # import osgeo.gdal as a convenience
----> 2 from osgeo.gdal import deprecation_warn
      3 deprecation_warn('gdal')
      4 
      5 from osgeo.gdal import *

/home/choct155/analysis/Anaconda/lib/python2.7/site-packages/osgeo/__init__.py in <module>()
     19                 fp.close()
     20             return _mod
---> 21     _gdal = swig_import_helper()
     22     del swig_import_helper
     23 else:

/home/choct155/analysis/Anaconda/lib/python2.7/site-packages/osgeo/__init__.py in swig_import_helper()
     15         if fp is not None:
     16             try:
---> 17                 _mod = imp.load_module('_gdal', fp, pathname, description)
     18             finally:
     19                 fp.close()

ImportError: /usr/lib/libgdal.so.1: undefined symbol: sqlite3_column_table_name

Has anyone else encountered this issue?

I know that sqlite3 is an installed module already, so if I can't see it being a dependency issue. That being said, I am not sure where to start beyond poking around the osgeo source.

I am using Ubuntu 13.04.

I first discovered this problem when having trouble with the R package, rgdal from within Notebook (but curiously not from within RStudio). Upon discovering deficiencies on the Python side as well, I thought it might be useful to talk to a more spatially-focused crowd.

---

With respect to the first comment (SQLite), how would I verify that one was using a different version from the other?

I checked the paths in os.environment, and they are the same.

With respect to the latter, sys.executable was identical (and I initiated both from the same working directory):

>>> import sys, os
>>> sys.executable
'/home/choct155/analysis/Anaconda/bin/python'

---

In [2]: import sys, os

In [3]: sys.executable
Out[3]: '/home/choct155/analysis/Anaconda/bin/python'

However, I did find differences in os.environ:

print set(python_environ.items())-set(ipython_environ.items())

print set(ipython_environ.items())-set(python_environ.items())
*****************************
set([('OLDPWD', '/home/choct155/analysis/fwtools/FWTools-linux-x86_64-3.0.6'), ('WINDOWID', '54529262'), ('_', '/home/choct155/analysis/Anaconda/bin/python'), ('C_INCLUDE_PATH', '/usr/include/gdal'), ('CPLUS_INCLUDE_PATH', '/usr/include/gdal')])
set([('WINDOWID', '54526233'), ('_', '/home/choct155/analysis/Anaconda/bin/ipython'), ('OLDPWD', '/tmp/pip-build-root/GDAL')])

To my eye, the environmental variables that I set in .bashrc (the include paths) are the elephant in the room. In fact, without these set, I couldn't get GDAL to install at all. What I am not clear on is why they would apply only to Python when IPython is using the same Python executable as the back end.

Sidenote: To those that are working with these tools, in general it's probably a good idea to check out minrk's github page and GeospatialPython.org.

---

Admittedly, I have very little theory behind this attempted fix, but I tried to harmonize the two environments by explicitly adding the missing environmental variables with a startup script in the default_profile for IPython. This, probably predictably, did not work. I am guessing that these variables are utilized only during installation. Since I already installed gdal...

!sudo pip install gdal
*********************
Requirement already satisfied (use --upgrade to upgrade): gdal in /home/choct155/analysis/Anaconda/lib/python2.7/site-packages
Cleaning up...

… I am still not sure how to remedy this. Further, I am still unclear on how and why my Python and IPython platforms would be looking at different versions of SQLite when they have the same PATH.

Answer 1

So it turned out to be a SQLite version conflict (hat tip GP.com). It seems as though IPython loaded a version of SQLite upon initializing the session, and GDAL requires another version. The solution was to interrupt the IPython default importation (thank you IPython help chat). This was done by creating a new script titled sqlite3.py, and populating it with raise ImportError. The script was placed in the first directory in the IPython sys.path.

Caveat: SQLite is used under the hood in IPython for various actions, and thus a version switch may cause some issues. At the current time, this is the cost of doing business for using these (awesome) tools together.