3

I am currently trying to determine the 2 nearest points for a database of 8 million people against 425 locations.

I have 2 csv files uploaded into QGIS one with a unique number for 8 million people with a latitude and longitude. The second csv has 425 locations with a latitude and longitude. I then start the distance matrix analysis under the QGIS Geoalgorithms section of the processing tools feature.

Once I set it up and tell the analysis to run it begins. However, it takes about 4 hours to process which is not bad when considering the amount of calculations I am asking, but when I open the task manager QGIS is only using 20% of CPU and 150 MB of RAM.

To test it further I opened up to instances of QGIS running the same distance matrix and both are running at about 150 MB.

Is there a known limit of RAM use for vector analysis where it is reading from a CSV and writing to a new csv output?

I am currently using QGIS 2.14.15-Essen on Windows 8 with an i7 and 32 GB of RAM.

  • 2
    Why should an app use more RAM than it needs to? You'll see a much better return on investment replacing ASCII files with real (indexable) data tables. The query would finish in seconds if you structured it to do 425 searches on 8m indexed locations. – Vince Aug 8 '17 at 20:18
  • 2
    You're definitely trying to do the work of a database in a 'GIS GUI' (no offence to QGIS) - if you think of the work that is trying to happen, QGIS is converting the LAT/LON to geometry on the fly to figure this out, which I'm sure is taking up a lot of memory - I don't know if there is a limit, but this is definitely not the best way to approach this problem... – DPSSpatial Aug 8 '17 at 21:21
  • I don't think this question is 'unclear', comments beneath question seems to indicate otherwise. I don't think it is off-topic either, see, we have a performance tag with more than 400 questions. Perhaps, it is broad, but I'd be interested in an answer explaining details what would be alternatives to OP. – Andre Silva Aug 9 '17 at 22:04
  • I think it is a valid question, but very hard to answer unless you wrote the tool. It uses as much RAM as it needs to, but could it be written to use more RAM and be quicker? I don't know. – HeikkiVesanto Aug 11 '17 at 12:17
2

Disclaimer: I did not check the tool how it works in detail. But its likely that it uses quite common principles.

First is reading both .csv. They are written into memory. As .csv always have to reside at full in memory for processing, nothing changes here.

The task you run needs to be run by the program point by point. 8mio*425 times. The outcome is written to disk. Just have a look at your .csv file it is growing line by line. In the windows explorer you see the file increasing a few kilobytes each second. No further memory usage needed here, after the alghorithm is intialized. Usually you build an algorithm this way that your memory will not overflow which means you control the maximum working buffer until it writes to disk. But this buffer is flushed each time so it is not increasing.

The CPU is used at 100% for a single core only resulting in a small overall usage. Thats the problem when the calculation is not run in parallel. Not many tools do that at all. Its difficult for the programmer on the one side, or the task can´t be parallelized at all. That is why you can get specialized workstations with processors trimmed for single-core performance and supercomputing and parallel-computing being an own field of research.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.