DLL error listing domains in Python script

Question

There is a Python script intended to create a UPDM geodatabase, it receives several parameters, among them, database admin user, database dataowner user, and their passwords.

The script runs using the ArcGIS Pro 2.8 Python, it starts its process, creates the database, as well as connection files (.sde). In some point of the script, it has been created different structures: tables, relationships, domains, etc. but then it fails while calling:

arcpy.da.ListDomains(self.database)

Here you can see the function:

def create_domains(self, domains=None):
    # Function to create domains within the geodatabase based on the JSON file
    if domains is None:
        domains = self.parser.parse_key(self.json_config, self.DOMAINS)
    for dm in domains:
        d_name = self.parser.parse_key(dm, self._NAME)
        d_desc = self.parser.parse_key(dm, self._DESCRIPTION)
        d_data_type = self.parser.parse_key(dm, self.DATA_TYPE)
        d_type = self.parser.parse_key(dm, self.DOMAIN_TYPE)
        d_values = self.parser.parse_key(dm, self.DOMAIN_VALUES)
        existing_domains = arcpy.da.ListDomains(self.database)
        domain_exist = any(check_domain_exists(d_name, d) for d in existing_domains)
        if domain_exist:
            self.logger.add("Domain {0} already exists".format(d_name))
            continue
        arcpy.CreateDomain_management(
            self.database,
            d_name,
            d_desc,
            d_data_type,
            d_type)
        self.logger.add("Domain {0} is created".format(d_name))
        for c_v in d_values:
            code = self.parser.parse_key(c_v, self._CODE)
            value = self.parser.parse_key(c_v, self.VALUE)
            arcpy.AddCodedValueToDomain_management(
                self.database,
                d_name,
                code,
                value)
        self.logger.add("Values for domain {0} is created".format(d_name))
        if d_type == "RANGE":
            d_max = self.parser.parse_key(dm, "MaxValue")
            d_min = self.parser.parse_key(dm, "MinValue")
            arcpy.SetValueForRangeDomain_management(self.database, d_name, d_min, d_max)

The error is quite weird (at least for me):

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

ERROR 1: Can't load requested DLL: C:\ArcGISPro\bin\gdalplugins\gdal_MG4Lidar.dll
1114: A dynamic link library (DLL) initialization routine failed.

11:51.29.583441 ERROR C:\temp\Installation\Data\DataOwner.sde
11:51.29.589433 ERROR ['Traceback (most recent call last):\n', '  File ".\\Arcpy\\InstallDataModel.py", line 95, in main\n', '  File "C:\\temp\\Installation\\Arcpy\\Classes\\JSONGeoDataModelClass.py", line 598, in create_domains\n    existing_domains = arcpy.da.ListDomains(self.database)\n', 'OSError: C:\\temp\\Installation\\Data\\DataOwner.sde\n']

Furthermore, the ArcGIS Pro Crash Window appear 4 times (probably one time for each crash in the request of the gdal_MG4Lidar.dll).

In this point of the execution, this line has been called several times, the script has been using the En terprise geodatabase connection file for several actions, inside the script there is no direct call to GDAL libraries (I assume it is an internal call from the ArcGIS Pro libraries/modules) and despite the message, I can verify the existence of the connection file and even use it in a new ArcGIS Pro instance.

It would be expected that domains were created and no error appears.

UPDATE: After digging a little bit in the behavior of the memory in the computer while the script run, then I implemented the use of the garbage collector at the end of the functions, this does not remove the error.

I decided to split the execution in several parts, each part called independently. The script is still failing, but now, the error is :

CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!
CPLGetTLSList(): TlsAlloc() failed!

According t this GDAL github page the error now is generated in that point, but the reason is still unknown.

Answer 1

After digging in the behavior of the computer, and with the latest error messages I think, the issue is related to the thread management implemented underneath by arcpy library.

I split the script in several parts, and call every part from a new python process, in this way the number of threads does not exceed the allowed amount of threads. Example:

python.exe script_part1.py
python.exe script_part2.py
python.exe script_part3.py
python.exe script_part4.py