How to process multiple GRASS GIS jobs with different regions in parallel?

Question

I want to send requests from a web frontend to GRASS GIS. I am currently using the Python Web Framework FastAPI for communication between the frontend and GRASS GIS.

I built the API as follows:

1. First I start GRASS GIS with the grass command, because I want to use the PyGRASS library
2. Then I'm using the GRASS GIS Python distribution to start the FastAPI
3. If the API receives a request, a new GRASS session will be created with the following command: gsetup.init('grassdata', 'location01', 'PERMANENT'). That means in case of multiple requests at once, the API will create multiple sessions
4. Each request has different region parameters. These parameters are passed individually to GRASS GIS with the g.Region module
5. GRASS should now start with the actual geoprocessing and return the result to the API

Current behavior:

• If there are two parallel requests, the first request is not processed correctly
• Request 1 mostly contains elements from the geoprocessing of request 2

Unfortunately, the guide Parallel GRASS jobs did not help me. I don't understand how the environment variable WIND_OVERRIDE can help to use different GRASS GIS regions at the same time. In the end, the environment variable is getting overwritten with every request. So if request 2 is started a few milliseconds after request 1, request 2 will overwrite request 1's region.

Actinia should have had the same problem. If I understand the code and documentation correctly, the actinia developers are using a Redis queue as a workaround. The queue ensures that each actinia node only processes one request at a time. Is that right?

Answer 1

From the described behavior, I think you are running everything in one process with shared set of environment variables (hard to sure without code, but likely).

Things like WIND_OVERRIDE are environment variables, so if you are only using os.environ (explicitly or implicitly), you can have only one WIND_OVERRIDE (or GRASS_REGION). The solution is to use more than one environment. You need this not only for the computational region, but for using multiple locations (or mapsets) from one process. I'm not sure if that's what you are doing, but it may be worth considering.

Another thing to consider in context of GRASS GIS is that the environment variables will apply mostly for the subprocesses. All the tools (modules) called are subprocesses, but using C API through ctypes (grass.lib) or its grass.pygrass wrappers is operating in the same process.

What you can do is to create an initial session with init, then copy os.environ for each of your requests, and then do the modifications of region and any calculations there. grass.script.create_environment is usually helpful in these cases. A higher level API for mapsets is in a PR as of now (Aug 10, 2022).

Another approach is to separate each calculation into a subprocess using the grass command (this may be more straightforward because you are using grass already anyway). Write your GRASS code separately and execute it with parameters for each request using one or more grass ... --exec calls. This approach allows you to use grass.pygrass freely and you get a great separation between each request. This assumes you use separate mapsets (or possibly locations) for each request. With separate mapsets, you may not even need WIND_OVERRIDE or GRASS_REGION.