Compiling QGIS 3.x (master) on Ubuntu 16.04 with Python3.6 and the latest Qt 5 and PyQt5 familiy modules (PyQt5, sip, qscintilla, pyqtchart)

Question

The problem:

Now that the compilation issue is over (Unable to compile QGIS 3.x (master) on Ubuntu 16.04: PyQt5.pyqtconfig module does not exist), QGIS 3 (master) refuses to load Python at startup.
I am working on Ubuntu 16.04 (4.15.0-30-generic x86_64 GNU/Linux).
- Python3 version is 3.5.2
- PyQt5 is 5.10.1
- SIP is 4.19.12 (manually installed, detected version during the compilation is 4.19.12, but sip.SIP_VERSION_STR when imported directly in a python3.5 console is 4.19.8. I cannot understand why)
- QScintilla is 2.10.4
These packages have been dowgraded as explained here: Unable to compile QGIS 3.x (master) on Ubuntu 16.04: PyQt5.pyqtconfig module does not exist

Here is the error message I am facing in a popup window during QGIS 3 startup:

Couldn't load PyQt.
Python support will be disabled.


Traceback (most recent call last):
  File "", line 1, in 
ImportError: /usr/lib/x86_64-linux-gnu/libQt5Core.so.5: version `Qt_5.10' not found (required by /usr/local/lib/python3.5/dist-packages/PyQt5/QtCore.so)


Python version:
3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', ad4ddb1

Followed by a second popup window:

An error occurred during execution of following code:
qgis.utils.uninstallErrorHook()


Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'qgis' is not defined


Python version:
3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', ad4ddb1

Then QGIS is ready to use, but there is no Python console (which I really need).

Second try; updating Python3 from 3.5.2 to 3.6.5:

Then I though it was a Python related issue, so I decided to install Python 3.6.5 (from this PPA: http://ppa.launchpad.net/jonathonf/python-3.6/ubuntu) in parallel to the default system Python 3.5.2 in Ubuntu 16.04.
So I changed the PYTHON_EXECUTABLE, PYTHON_INCLUDE_DIR and PYTHON_LIBRARY to version 3.6 instead of 3.5.
I also made a pip3.6 command in order to install packages for this version of Python and I installed, this time the latest version of the PyQt5 serie:
- PyQt5 version: 5.11.2
- SIP version: 4.19.12
- QScintilla2 PyQt module: 2.10.7

At QGIS startup, the same two popup windows appeared:

Couldn't load PyQt.
Python support will be disabled.

Traceback (most recent call last):
  File "", line 1, in 
ImportError: /usr/lib/x86_64-linux-gnu/libQt5Core.so.5: version `Qt_5.11' not found (required by /usr/local/lib/python3.6/dist-packages/PyQt5/QtCore.so)

Python version:
3.6.5 (default, May  3 2018, 10:08:28) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', ad4ddb1

Followed by:

An error occurred during execution of following code:
qgis.utils.uninstallErrorHook()


Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'qgis' is not defined


Python version:
3.6.5 (default, May  3 2018, 10:08:28) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', ad4ddb1

What I realized at this point:

is that I was using the Qt5 I installed from the Ubuntu repositories, which version is Qt 5.5.1.
That's probably why QGIS cannot find the Qt_5.10' string the first try and theQt_5.11' string in the second try. It probably wants the system Qt version to be the same than the PyQt module, in the first case PyQt was 5.10.1 and in the second 5.11.2. So I assume they should be like a family and share the same version numbers (at least up to the minor digit). I looked after 'how to install Qt 5.10 or Qt 5.11 on Ubuntu 16.04 and foud these PPA:
- http://ppa.launchpad.net/beineri/opt-qt-5.10.1-xenial/ubuntu
- http://ppa.launchpad.net/beineri/opt-qt-5.11.1-xenial/ubuntu

I added them both, installed Qt5.10 and Qt5.11 in /opt as described on the respective web pages from the PPAs and changed all the Qt5 variable path in cmake to the ones in '/opt/qt51x/lib/cmake/' + the QT_LRELEASE_EXECUTABLE variable to '/opt/qt51x/bin/lrelease' with x in [0 when building QGIS with Python3.5 and x=1 when building with Python3.6].

But then...

New issue:

The configuration and generation is all OK, but the build outputs a lot (!) of C++ errors. I don't want to print them all but it looks like that:

https://pastebin.com/vZ20XPxv

So, to summarize:

Either QGIS compile well with a PyQt 5.11.2 and system Qt 5.5.1 but refuses to load the Python console at startup telling that it cannot find `Qt_5.11' in '/usr/lib/x86_64-linux-gnu/libQt5Core.so
or QGIS doesn't compile when PyQt and Qt version are the same (up to the minor digit).

Answer 1

So, based on AndreJ comment, I uninstall all PyQt5 related packages with pip3.5 (namely pyqt5, sip, qscintilla and pyqtchart) and used the official ones from my Ubuntu distribution.

Here is the cmake log which looks fine:

QGIS version: 3.3.0 Master (30300)
Could not find GRASS 7
Found Proj: /usr/local/lib/libproj.so (5.2.0)
Found GEOS: /usr/lib/x86_64-linux-gnu/libgeos_c.so (3.5.0)
Found GDAL: /usr/local/lib/libgdal.so (2.3.1)
Found Expat: /usr/lib/x86_64-linux-gnu/libexpat.so
Found Spatialindex: /usr/lib/x86_64-linux-gnu/libspatialindex.so
Found Qwt: /usr/lib/libqwt-qt5.so (6.1.2)
Found libzip: /usr/lib/x86_64-linux-gnu/libzip.so
Found Sqlite3: /usr/lib/x86_64-linux-gnu/libsqlite3.so
Found PostgreSQL: /usr/lib/x86_64-linux-gnu/libpq.so
Found SpatiaLite: /usr/lib/x86_64-linux-gnu/libspatialite.so
Qt WebKit support DISABLED.
Found Qt version: 5.5.1
Found QScintilla2: /usr/lib/libqt5scintilla2.so (2.9.1)
Found QtKeychain: /usr/lib/x86_64-linux-gnu/libqt5keychain.so
Found QCA: /usr/lib/x86_64-linux-gnu/libqca-qt5.so (2.1.1)
Found QCA OpenSSL plugin
Pedantic compiler settings enabled
Found Python executable: /usr/bin/python3
Found Python version: 3.5.2
Found Python library: /usr/lib/x86_64-linux-gnu/libpython3.5m.so
Found Python site-packages: /usr/lib/python3/dist-packages
Found PyQt5 version: 5.5.1
Found SIP version: 4.17
Found QScintilla2 PyQt module: 2.9.1
Found GSL: -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas -lm
Ctest Binary Directory set to: /opt/QGIS/build_master/output/bin
Configuring done
Generating done

(Note: one has to deactivate QtWebKit because it was deprecated since Qt>=5.5)

And here is an error during the compilation:

[ 30%] Generating core/sip_corepart0.cpp, core/sip_corepart1.cpp, core/sip_corepart2.cpp, core/sip_corepart3.cpp
/bin/sh: 1: /usr/bin/sip: not found
python/CMakeFiles/python_module_qgis__core.dir/build.make:515: recipe for target 'python/core/sip_corepart0.cpp' failed
make[2]: *** [python/core/sip_corepart0.cpp] Error 127
make[2]: *** Deleting file 'python/core/sip_corepart0.cpp'
CMakeFiles/Makefile2:4078: recipe for target 'python/CMakeFiles/python_module_qgis__core.dir/all' failed
make[1]: *** [python/CMakeFiles/python_module_qgis__core.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2

It seems that cmake can find sip during the configuration step but the compilation step with make does not. And I do not have any /usr/bin/sip on my system even after reinstalling python3-sip and python3-sip-dev packages with apt.

Parenthesis:

If you run into a sipconfig error or a:

CMake Error at cmake/FindSIP.cmake:54 (MESSAGE):
  Could not find SIP
Call Stack (most recent call first):
  CMakeLists.txt:753 (FIND_PACKAGE)

after having pip3.5 uninstalled a latest sip package and if you already have the system sip (python3-sip python3-sip-dev) just force the reinstallation of the later:

$ apt-get install python3-sip python3-sip-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-sip is already the newest version (4.17+dfsg-1build1).
python3-sip-dev is already the newest version (4.17+dfsg-1build1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

$ apt-get install --reinstall python3-sip python3-sip-dev

Then cmake will find it.

Identified problem:

There is two files in the QGIS sources cmake/ folder:
- FindSIP.cmake
- FindSIP.py

the later beeing called by the former.

In the python file, one can read at line:39:

print("sip_bin:%s" % sipcfg.sip_bin)

this line prints:

sip_bin:/usr/bin/sip

But on the system, there is no such sip file:

ll /usr/bin | grep -i sip
-rwxr-xr-x 1 947 oct 24  2015 dh_sip3*
-rwxr-xr-x 1 868 jan 18  2016 dh_sip*
-rwxr-xr-x 1 59K mai 16 17:00 lsipc*

There is a sip folder /usr/share but no executable.

By trying to run sip:

$ sip -V
The program 'sip' is currently not installed. You can install it by typing: 
sudo apt install sip-dev

So.... install it:

$ apt-get install sip-dev

Check it's there:

$ sip -V
4.17
$ ll /usr/bin/ | grep sip
-rwxr-xr-x 1 947 oct 24  2015 dh_sip3*
-rwxr-xr-x 1 868 jan 18  2016 dh_sip*
-rwxr-xr-x 1 357K jan 18  2016 sip*
-rwxr-xr-x 1 59K mai 16 17:00 lsipc*

Yepee, sip is here now!

And retry the QGIS compilation...:

[...]
[ 99%] Building CXX object python/CMakeFiles/python_module_qgis__gui.dir/gui/sip_guipart0.cpp.o
[ 99%] Building CXX object python/CMakeFiles/python_module_qgis__gui.dir/gui/sip_guipart1.cpp.o
[ 99%] Built target translations
[ 99%] Building CXX object python/CMakeFiles/python_module_qgis__gui.dir/gui/sip_guipart2.cpp.o
[ 99%] Building CXX object python/CMakeFiles/python_module_qgis__gui.dir/gui/sip_guipart3.cpp.o
[100%] Built target staged-plugins-on-build

\o/ YES!

The new issues (2x):

Couldn't load PyQGIS.
Python support will be disabled.


Traceback (most recent call last):
  File "", line 1, in 
  File "/opt/QGIS/build_master/output/python/qgis/core/__init__.py", line 27, in 
    from qgis._core import *
ImportError: No module named 'PyQt5.QtSql'


Python version:
3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', a82de83

To fix this one, one has to install pyqt5.qtsql manually (it does not seem to come along python3-pyqt5):

$ apt-get install python3-pyqt5.qtsql

(...) Same thing for 'PyQt5.QtSvg':

$ apt-get install python3-pyqt5.qtsvg

Then, QGIS finally starts normally with a nice Python console:

The two last issues were clearly due to a lack in the dependencies installation from my side.