Activating a conda environment from within QGIS -- cannot be done due to inherited environmental variables?

Question

Background

I've been working on a small QGIS Python plugin. Part of the functionality requires a number of packages that have complex dependencies. In my regular Python use, I use conda to manage it. QGIS can be installed via conda, but I quickly run into issues: numba-JIT doesn't work, and I think there's issues with the GIS stack (geopandas, rasterio) versus QGIS (both requiring GDAL for example).

Anyway, I have a working solution, which is to run two Python interpreters (conda & QGIS) and have them communicate via socketserver (this idea: https://stackoverflow.com/questions/43334302/communication-between-two-separate-python-engines).

The problem

This works fine, I'd just like to be able to start the conda interpreter from within the QGIS GUI. My idea was basically to:

• Figure out the conda installation (like e.g. VSCode does: https://github.com/microsoft/vscode-python/blob/abfb8c31714ae4252dc1be88e00b365a0137c701/src/client/pythonEnvironments/discovery/locators/services/conda.ts#L36)
• Call the conda activate script to set the appropriate environmental variables for the conda environment via a subprocess call.
• Start the interpreter of that conda environment.
• Start my server on localhost with a specified port, and connect.

However, without exception, I get this error when calling activate.bat {name of environment} (source activate on *nix):

"c:\Miniconda3\Scripts\conda-script.py", line 11, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

All of the options I've tried (direct subprocess call, subprocess to intermediate Windows Batch/Powershell scripts) work fine outside of QGIS, but result in this error when tried from inside of QGIS.

My current hypothesis is that it's the environmental variables being set when starting QGIS are somehow interfering, effectively crippling the conda environment activation.

So I've tried stuff like the ignore switch /i for cmd, or -UseNewEnvironment for Powershell, but to no avail (from: https://stackoverflow.com/questions/8261156/start-new-cmd-exe-and-not-inherit-environment).

The question

Anybody know a workaround? I feel like there ought to be a relatively easy way to "escape" the QGIS-set environmental variables (which I'm blaming so far). Or is this a fool's errand?

I've currently resorted to asking the user to start an interpreter, and enter the port number in a field in the plugin, which are kinda ugly steps.

Answer 1

Alright, I think I've figured something out. It is isn't especially pretty but it does work.

Basically, the conda activate call sets environment variables. This can be achieved in other ways as well, of course, namely: Python's os.environ.

(It could also be done with a command line script, basically an alternative version of the conda activate script, I guess.)

Step one

How do I get the environmental variables? Well I can just start the conda environment once, and dump them in a file:

import json
import os

with open("env-vars.json", "w") as f:
    f.write(json.dumps(dict(os.environ)))

Step two

I want to replace the QGIS environmental variables by the ones I defined before. Any process that is started next (via e.g. subprocess) will inherit those. However, to keep my QGIS interpreter running properly, I don't want to screw around with its environmental variables.

So we start up a new Python interpreter. Given how it's configured, it's best to start another instance of the QGIS Python interpreter. Otherwise, you'll end up importing QGIS Python packages with the other interpreter (which may give issues if they're not the same version).

Step three

We get rid of the existing environmental variables, and set the environmental variables as the conda environment requires it:

import json
import os

with open("env-vars.json", "r") as f:
    d = json.loads(f.read())

for key in os.environ:
    os.environ.pop(key)

for key, value in d.items():
    os.environ[key] = value

Step four

Finally, with all the environmental variables in place, we start the conda interpreter.

import subprocess

subprocess.call(path_conda_interpreter)

Somewhat to my surprise, this appears to work and I can import my conda based packages without errors!

Convenience

I can define a configure function in my Python module which writes the environmental variables the an appropriate place e.g. %home% during setup, from which the environment definition is read by the plugin.