I have a issue about memory management in PyQGIS during loop.
I'm writing a code where there is a while loop.
During the operatione inside the loop that are almost the same,I noticed that the execution time increases and also the RAM occupation too (from 200MB to 128GB). So I tried to write a simplified test script that always executes the same instructions 10.000 times but I notice once again that the execution time increases and the RAM too.
The test script:
- input1: dots shape (23.000 points)
- input2: lines shape (115.000 lines)
- execution: extracts a point from the points shape (always the
same)-->buffer of 10.000m around the point-->extraction of the lines in the buffer. Cycle 10.000 times.
Below you can find the code and a graph of the times laps for each cycle.
Can you explain me why time increases even if the instructions for each cycle are always the same?
I assume that the memory is not freed every cycle.
Is there a way to fix and keep the time the same every loop?
#1A - spatial index on lines
alg_params = {
'INPUT': parameters['inputline']#input lines
}
outputs['ASpatialIndexOnLines'] = processing.run('native:createspatialindex', alg_params, context=context, is_child_algorithm=True)
feedback.setCurrentStep(1)
n_loop=0
while n_loop<10000: #loop 10.000 times
start_time=time.time()
# 1 - extract random point
alg_params = {
'FIELD': 'k',
'INPUT': parameters['inputpoints'],#input points
'OPERATOR': 0, # =
'VALUE': 100, #extract always point with k=100
'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT
}
outputs['ExtractRandomPoint'] = processing.run('native:extractbyattribute', alg_params, context=context, is_child_algorithm=True)
feedback.setCurrentStep(2)
# 2 - Buffer 10000m on random point
alg_params = {
'DISSOLVE': False,
'DISTANCE': 10000,
'END_CAP_STYLE': 0, # Arrotondato
'INPUT': outputs['ExtractRandomPoint']['OUTPUT'],
'JOIN_STYLE': 0, # Arrotondato
'MITER_LIMIT': 2,
'SEGMENTS': 5,
'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT
}
outputs['Buffer10000mOnRandomPoint'] = processing.run('native:buffer', alg_params, context=context, is_child_algorithm=True)
feedback.setCurrentStep(3)
# 3 - spatial index on buffer
alg_params = {
'INPUT': outputs['Buffer10000mOnRandomPoint']['OUTPUT']
}
outputs['SpatialIndexOnBuffer'] = processing.run('native:createspatialindex', alg_params, context=context, is_child_algorithm=True)
feedback.setCurrentStep(4)
# 4 - extract lines in buffer
alg_params = {
'INPUT': outputs['ASpatialIndexOnLines']['OUTPUT'],
'INTERSECT': outputs['SpatialIndexOnBuffer']['OUTPUT'],
'PREDICATE': [0], # interseca
'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT
}
outputs['ExtractLinesInBuffer'] = processing.run('native:extractbylocation', alg_params, context=context, is_child_algorithm=True)
end_time=time.time()
delta_time=round(end_time-start_time,3)
feedback.pushInfo('{},{}'.format(delta_time,n_loop)) #write time and loop
n_loop+=1
if feedback.isCanceled():
return {}
# when loop finish-->out_test, last extraction
alg_params = {
'INPUT': outputs['ExtractLinesInBuffer']['OUTPUT'],
'OUTPUT': parameters['out_test']
}
outputs['fixGeometries'] = processing.run('native:fixgeometries', alg_params, context=context, is_child_algorithm=True)
return {}