4

My data set contains radar plots for flights on approach to an airport. For each unique flight I have hundreds of points as they are recorded every four seconds. Using Points2One I can join the points to form individual flight lines for each unique flight. In using Points2One by its nature it only creates a single line and a single record entry in the attribute table hence I'm loosing the data granularity of altitude and speed at each four second segment of the flight. Is there a way that I can create a single flight path that is composed of individual line segments, where each individual line segement would have the same record identifier (Operation No) and attrbutes as either the starting or ending point node of that line segement.

Example radar plot data Example radar plot data

So with the data sample I'd like to end up with eight line segments that all have the Operation No. 16173263 and each segment would have the time delay, Latitude, Lon, Altitude & Velocity attributes of the starting point.

The radar plots for a single day contain approximately 1200 different flights and over 100,000 data points.

I have no python or coding experience and so far all QGIS experience consists purely of using menu driven functions.

Operation No, Time Delay (seconds), Latitude, Longitude, Altitude (ft), Velocity (knots)    
16173263,0,42.005191,-81.987368,12444,0
16173263,1,42.003938,-81.986167,12418,328.5
16173263,5,41.999006,-81.981324,12303,326.6
16173263,9,41.994118,-81.976394,12182,326.6
16173263,13,41.989301,-81.971414,12054,324.6
16173263,17,41.98452,-81.966409,11919,322.7
16173263,21,41.979784,-81.961404,11778,320.7
16173263,25,41.975057,-81.956437,11637,320.7
16173263,29,41.970348,-81.95152,11490,318.8
16173264,0,41.512822,-82.082972,5331,0
16173264,3,41.513811,-82.078243,5266,258.5
16173264,7,41.515101,-82.071902,5180,258.5
16173264,11,41.516373,-82.065547,5105,258.5
16173264,15,41.517608,-82.059206,5036,258.5
16173264,19,41.518844,-82.052913,4970,256.6
16173264,23,41.520053,-82.04672,4905,252.7
16173264,27,41.521254,-82.040638,4846,246.9
  • I would use python or sql for this. I'll see if I can help. So ultimately each point will become a line between it and the next point but retaining all data from the attribute table? And the proper order of them is time delay low -> high? – Jesse McMillan Feb 1 '17 at 21:37
  • Thank you for posting sample data, but perhaps it would be more helpful to post it as text instead of an image so that it can be copy-pasted into a file for testing. Would you mind doing that? – Jesse McMillan Feb 1 '17 at 21:46
  • Have added data as CSV text that contains records for 2 different flights although I've missed half of the last line. – N Bod Feb 1 '17 at 22:04
2

Here's a python script to get the job done.

Before you use this you'll need to know a few things:

  • This script creates a memory layer which means the layer is not saved until you right-click it in the layers panel and select save-as.
  • In order for this script to work, you'll need to make sure that the layer name and attribute names match yours perfectly. I've put variables containing those up at the top so you can adjust. They are capital and whitespace sensitive.

To run this script, open the python window in Qgis:

python window

Then open the editor:

editor

And finally, paste in the script (or open it from a saved location) then hit run:

run script

New Approach: The old script ran through the sample points in 0.51s, this new one runs through the sample points in 0.017s. It relies on getting the points from QGIS in the correct order to begin with rather then sorting them all into a dictionary. At any one time it only has to keep two points in memory instead of literally all of them.

# To avoid 'QVariant' is not defined error
from PyQt4.QtCore import *
import time

starttime = time.time()
# update these:
lyrName = 'test'
attributeList = [QgsField("Operation No", QVariant.String),
                        QgsField("Time Delay (seconds)", QVariant.Int),
                        QgsField("Latitude", QVariant.Double),
                        QgsField("Longitude", QVariant.Double),
                        QgsField("Altitude(ft)",QVariant.Double),
                        QgsField("Velocity(knots)",QVariant.Double)]
# get old layer
lyr = QgsMapLayerRegistry.instance().mapLayersByName(lyrName)[0]
# create new layer in memory to hold created lines
newLyr = QgsVectorLayer("LineString?crs=epsg:4326", "testPaths", "memory")
newProvider = newLyr.dataProvider()
# make sure field names are identical
newProvider.addAttributes(attributeList)
newLyr.updateFields()

# set up feature request so we get the features in the correct order
req = QgsFeatureRequest()
req.addOrderBy('"Operation No"')
req.addOrderBy('"Time Delay (seconds)"')

currentFeat = False
for nextFeat in lyr.getFeatures(req):
    if currentFeat:
        # we now have both current and next feature. Are they same operation?
        if nextFeat['Operation No'] == currentFeat['Operation No']:
            # if so create a line between them
            pLine = QgsGeometry.fromPolyline([currentFeat.geometry().asPoint(),nextFeat.geometry().asPoint()])
        else:
            # if not, both points in line from current Feature
            pLine = QgsGeometry.fromPolyline([currentFeat.geometry().asPoint(),currentFeat.geometry().asPoint()])
        newFeat = QgsFeature()
        newFeat.setGeometry(pLine)
        newFeat.setAttributes(currentFeat.attributes())
        newProvider.addFeatures([newFeat])
    currentFeat = nextFeat

# because we're always working with previous feature to iteratable
# we will miss the last feature so manually process it:
pLine = QgsGeometry.fromPolyline([currentFeat.geometry().asPoint(),currentFeat.geometry().asPoint()])
newFeat = QgsFeature()
newFeat.setGeometry(pLine)
newFeat.setAttributes(currentFeat.attributes())
newProvider.addFeatures([newFeat])

newLyr.updateExtents()
QgsMapLayerRegistry.instance().addMapLayer(newLyr)

endtime = time.time()
print 'done in {} seconds'.format(endtime - starttime)

The old approach:

# To avoid 'QVariant' is not defined error
from PyQt4.QtCore import *

# update these:
lyrName = 'test'
attributeList = [QgsField("Operation No", QVariant.String),
                        QgsField("Time Delay (seconds)", QVariant.Int),
                        QgsField("Latitude", QVariant.Double),
                        QgsField("Longitude", QVariant.Double),
                        QgsField("Altitude(ft)",QVariant.Double),
                        QgsField("Velocity(knots)",QVariant.Double)]

# get layer
lyr = QgsMapLayerRegistry.instance().mapLayersByName(lyrName)[0]

# load into dictionary:
# points {operation : { timedelay : point }}
operationDict = {}
for feat in lyr.getFeatures():
    if feat['Operation No'] in operationDict:
        operationDict[feat['Operation No']][feat['Time Delay (seconds)']] = feat
    else:
        operationDict[feat['Operation No']] = {feat['Time Delay (seconds)'] : feat}
# create new layer in memory to hold created lines
newLyr = QgsVectorLayer("LineString?crs=epsg:4326", "testPaths", "memory")
newProvider = newLyr.dataProvider()
# make sure field names are identical
newProvider.addAttributes(attributeList)
newLyr.updateFields()

# iterate over dictionary and add features to new layer.
for key in operationDict:
    op = operationDict[key]
    sortedTimes = sorted(op.iterkeys())
    print 'sortedTimes:'
    print sortedTimes
    for tstamp in sortedTimes:
        oldFeat = op[tstamp]
        tIndex = sortedTimes.index(tstamp)
        print 'time: {} index: {}'.format(tstamp,tIndex)
        # if not on the last one
        if tIndex + 1 < len(sortedTimes):
            nextFeat = op[sortedTimes[tIndex + 1]]
        else:
            # last feature, next geometry will just be itself
            nextFeat = oldFeat
        # build new feature
        print oldFeat.geometry().asPoint()
        print nextFeat.geometry().asPoint()
        pLine = QgsGeometry.fromPolyline([oldFeat.geometry().asPoint(),nextFeat.geometry().asPoint()])
        newFeat = QgsFeature()
        newFeat.setGeometry(pLine)
        newFeat.setAttributes(oldFeat.attributes())
        newProvider.addFeatures([newFeat])

newLyr.updateExtents()
QgsMapLayerRegistry.instance().addMapLayer(newLyr)

Some info on how the script works: First it sorts the points into a nested dictionary. A python dictionary is a list of key:value pairs. Like this: {'color:'red','model':'corvette'}. In this case we have {'operation': {time: point, time: point}}

That allows us to sort through them easily. Next, the script iterates over the operations (so each op is kept separate) and within the operation iterates over a sorted list of the points. For each point it checks if there's a 'next' point and if there is creates a line to it, saving to a new layer.

  • 1
    Jesse - The code does exactly what I need it to, thanks so much. Please don't think me ungrateful but is there a way the code could be more efficient as it is taking nearly 1.5hrs to process 2 days of data and I need to process a couple of months. I have already commented out the Print commands. 2 days has radar 370,000 points, which implies a month will have over 5.5m points and from the few test runs I have done the processing time is growing exponentially with the increase in points. – N Bod Feb 2 '17 at 20:30
  • @Nbod hope this helps. I'm curious to see how much faster it is for you. If you don't mind, let me know how much time it takes compared to the old method (for my own knowledge). Theoretically processing time should increase in a linear relationship to number of points rather then exponentially. – Jesse McMillan Feb 2 '17 at 23:22
  • Thanks a lot for having a second go at this. The test data, which has 18 points and 2 "Operation No" values, with the old code took 0.1930 secs with the new code 0.0160 secs. Running two days of data with the new code took "5382.57299 secs" which is 89mins and it has 370265 points with 2774 "Operation No" values. The 2 day data set is running approx 16 times slower per point than the test data. I may just have to leave code running over night or over a weekend. I'm using an average PC: Intel i5 3.2Ghz 4 core processor and 8Gb DDR3 RAM at 1600Mhz, Windows 10 and QGIS 2.18.3. – N Bod Feb 3 '17 at 10:16
  • 1
    I'm sorry the new code isn't substantially improving processing time. It seems like the bottleneck is somewhere else. Possibly qgis is hitting it's upper limit for ram utilization and is having to swap the data to and from the harddisk. One option could perhaps be to split the data into multiple files for processing, but you'd still have to merge them later so I'm not sure if that would really improve things... I bet a postgis based solution would perform somewhat better, but that's a little beyond me at the moment. – Jesse McMillan Feb 3 '17 at 14:28
  • A possible tweak if ram is the problem here: – Jesse McMillan Feb 8 '17 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.