2

I know there are several questions related to speed issues in QGIS, but I am having problems when running Ftools.

I have a vector with ~70,000 points in it. When I run the generate mean coordinates operation 99% of the time QGIS will crash. When QGIS chooses not to crash, it will take ~ 2 hours to generate the mean coordinates required.

Is there a way to speed up the process/prevent crashing?

Or would the best option simply be to leave the program running until it decides to respond again?

Thanks for your help.

e/ @James s shapefile format, 1.8.0 Lisboa

  • What format is your data in? And which version of QGIS are you using? – James S Sep 17 '13 at 12:52
  • If you're on a 64 bit Windows machine, try the experimental build of 64bit OSGEO4W (trac.osgeo.org/osgeo4w) I've had good luck with it being faster/less 'crashy' on big tasks. – Darren Cope Sep 17 '13 at 14:45
  • This sounds like a bug. This calculation is I/O bound: the actual computation requires less than a millisecond on any computer built in the last decade. Resorting to any other platform (Python and R are well-known and free) will provide quick, efficient solutions. – whuber Sep 17 '13 at 20:10
5

I'd suggest using the cross-platform, open source GIS Whitebox GAT (http://www.uoguelph.ca/~hydrogeo/Whitebox/). I had a contract last year that involved extracting complex shape characteristics of 168,707 lakes in the arctic, with a total of 1,786,312 nodes within the features. It performed the analysis without crashing a single time and the processing time was certainly reasonable (as in minutes not hours).

enter image description here

EDIT: Okay, here's a script that you can use to find the centroid based on a grouping ID field in a shapefile's attribute table. You simply need to open the Whitebox Scripter (scroll icon on toolbar), change the scripting language to Groovy, paste the code, save it, and run it. I tested it on my lakes coverage, which has 1,786,312 node points, using the Lake ID as the grouping field. It took 165.6 seconds to complete the operation on my i7 quad core machine. If that isn't fast enough for you, I can look into parallelizing the script. I was in a bit of a rush when I wrote it, so I hard coded it instead of making a dialog that would allow you to specify your own files. As such, you need to change the commented lines at the very bottom of the script to reflect your input file name, the field used for grouping, and the output file name. Good luck and let me know how it goes for you.

/*
 * Copyright (C) 2013 Dr. John Lindsay <jlindsay@uoguelph.ca>
 *
 * This program is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */

import java.io.File
import whitebox.geospatialfiles.ShapeFile
import whitebox.geospatialfiles.shapefile.ShapeType
import whitebox.geospatialfiles.shapefile.*
import whitebox.geospatialfiles.shapefile.Point
import whitebox.geospatialfiles.shapefile.attributes.AttributeTable
import whitebox.geospatialfiles.shapefile.attributes.DBFField
import whitebox.geospatialfiles.WhiteboxRasterBase
import whitebox.interfaces.WhiteboxPluginHost
import whitebox.stats.TwoSampleKSTest;
import groovy.transform.CompileStatic

@CompileStatic
class CentroidByGrouping {
private WhiteboxPluginHost pluginHost

CentroidByGrouping(WhiteboxPluginHost pluginHost) {
    this.pluginHost = pluginHost
}

public void run(String[] args) {
    def timeStart = System.nanoTime()

    try {
        // declare variables
        def inputFile = args[0]
        def input = new ShapeFile(inputFile)
        ShapeType st = input.getShapeType()
        def fieldName = args[1]
        def outputFile = args[2]

        int i
        double x, y
        int progress
        int parentID

        DBFField[] fields = input.attributeTable.getAllFields()
        def fieldNum = -1;
        for (i = 0; i < fields.length; i++) {
            if (fields[i].getName().equals(fieldName)) {
                fieldNum = i
                break
            }
        }

        if (fieldNum < 0) {
            pluginHost.showFeedback("Could not find the field of interest.")
            return
        }

        DBFField[] outputFields = new DBFField[1];
        outputFields[0] = new DBFField()
        outputFields[0].setName("GROUP_ID")
        outputFields[0].setDataType(DBFField.DBFDataType.NUMERIC)
        outputFields[0].setFieldLength(10)
        outputFields[0].setDecimalCount(0)

        ShapeFile output = new ShapeFile(outputFile, ShapeType.POINT, outputFields);

        int numRecords = input.getNumberOfRecords()

        // find the minimum, maximum and the number of groups based on the Grouping ID
        int minVal = 9999999
        int maxVal = -9999999

        int oldProgressValue = -1
        int recNum = 0;
        for (ShapeFileRecord record : input.records) {
            parentID = record.getRecordNumber()
            Object[] attributes = input.attributeTable.getRecord(parentID - 1)
            i = (int)attributes[fieldNum]
            if (i < minVal) { minVal = i }
            if (i > maxVal) { maxVal = i }

            // update the progress
            recNum++
            progress = (int) (100f * recNum / numRecords)
            if (progress != oldProgressValue) {
                pluginHost.updateProgress("Loop 1 of 3", progress)
                oldProgressValue = progress
            }
            // check to see if the user has requested a cancellation
            if (pluginHost.isRequestForOperationCancelSet()) {
                pluginHost.showFeedback("Operation cancelled")
                return
            }
        }
        int numGroups = maxVal - minVal + 1
        println minVal
        println maxVal
        println numGroups

        // find the centroid of each group
        int[] numPointsInGroup = new int[numGroups]
        double[] totalXInGroup = new double[numGroups]
        double[] totalYInGroup = new double[numGroups]
        double[][] points
        oldProgressValue = -1
        recNum = 0;
        for (ShapeFileRecord record : input.records) {
            parentID = record.getRecordNumber()
            points = record.getGeometry().getPoints()
            Object[] attributes = input.attributeTable.getRecord(parentID - 1)
            i = ((int)attributes[fieldNum]) - minVal
            numPointsInGroup[i]++
            totalXInGroup[i] += points[0][0]
            totalYInGroup[i] += points[0][1]

            // update the progress
            recNum++
            progress = (int) (100f * recNum / numRecords)
            if (progress != oldProgressValue) {
                pluginHost.updateProgress("Loop 2 of 3", progress)
                oldProgressValue = progress
            }
            // check to see if the user has requested a cancellation
            if (pluginHost.isRequestForOperationCancelSet()) {
                pluginHost.showFeedback("Operation cancelled")
                return
            }
        }

        // now output the centroid points
        oldProgressValue = -1
        for (i = 0; i < numGroups; i++) {
            if (numPointsInGroup[i] > 0) {
                x = totalXInGroup[i] / numPointsInGroup[i]
                y = totalYInGroup[i] / numPointsInGroup[i]
                whitebox.geospatialfiles.shapefile.Point wbGeometry = new whitebox.geospatialfiles.shapefile.Point(x, y);
                Object[] rowData = new Object[1];
                rowData[0] = new Double(i + minVal);
                output.addRecord(wbGeometry, rowData);
            }
            progress = (int)(100f * i / (numGroups - 1))
            if (progress != oldProgressValue) {
                pluginHost.updateProgress("Loop 3 of 3", progress)
                oldProgressValue = progress
            }
            // check to see if the user has requested a cancellation
            if (pluginHost.isRequestForOperationCancelSet()) {
                pluginHost.showFeedback("Operation cancelled")
                return
            }
        }

        output.write()

        // displays the final shapefile
        pluginHost.returnData(outputFile)

    } catch (Exception e) {
        pluginHost.showFeedback(e.getMessage())
    } finally {
        def timeStop = System.nanoTime()
        pluginHost.showFeedback("Operation completed in ${((timeStop - timeStart)  / 1000000000.0)} seconds")
    }
}

}


String[] myArgs = new String[3]
// the input shapefile
myArgs[0] = "/Users/johnlindsay/Documents/Research/Contracts/NRCan 2012/Data/LakeFilePnts/lakepoint.shp"
// the name of the field used for grouping
myArgs[1] = "PARENT_ID"
// the name of the output centroid points shapefile
myArgs[2] = "/Users/johnlindsay/Documents/Research/Contracts/NRCan 2012/Data/LakeFilePnts/points centroid.shp"
def centroid = new CentroidByGrouping(pluginHost)
centroid.run(myArgs)
  • Hi John. I've downloaded whitebox. Could you talk me through the process of generating mean coordinates please? My data is split up based on IDs, and i need a new coordinate per ID group - not just one coordinate for the whole 70k group of points. Thanks – Rob Lodge Sep 17 '13 at 13:02
  • Hi Rob, Okay can you give me a description of your data first? Is it in a shapefile format? Are they all points? Is this ID group a numerical attribute stored in an associated dbf? – user21951 Sep 17 '13 at 13:07
  • The format is in shapefile. I have an assocaited dbf,prj,qpj and shx format of the file. The shapefile contains 76347 points. These points are grouped into IDs. Each ID has from 1 up to 1000 nodes assigned to it. I wish to create a mean coordinate for each ID, based on the spatial distribution of the points within each ID class. I've had no problems running the operation in QGIS (minus the processing time) - so all the necessary files are there to work with! – Rob Lodge Sep 17 '13 at 13:15
  • Is the group ID numerical and an integer or is it floating point? – user21951 Sep 17 '13 at 13:26
  • 1
    This worked like a charm, John. Created mean coordinates for 70k nodes in 16.2 seconds. Cannot stress my appreciation, thank you! – Rob Lodge Sep 17 '13 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.