3

I am working with a dataset of thousands of datapoints in QGIS. The datapoints are satellite pings that are usually taken each 2 hours. But for some reason there are some datapoints that are taken 1 hour or even 15 minutes after each other. I want/need an equally distributed dataset, so I have to filter this data out. There should be a minimum distance of 2 hours between each point.

I have already tried to measure the difference between following datapoints. However, if I would filter on that, I would lose a lot of valuable data because it would filter out any point that is less than 2 hours away from the next point. And there are some parts of the data where hundreds of points just have an interval of 15 minutes. I would lose all of them in this way.

For example: Point A: 10:15 Point B: 10:45 Point C: 11:15 Point D: 11:45 Point E: 12:15

If I would filter out data based on distance between points <2 hours, I would lose all data. What I want is to create some kind of a pattern that measures =>2 hours ahead from Point A. So it will detect Point E and delete point B, C and D from the dataset.

So does anyone know how to create some kind of a pattern-based filter in the dataset so that there is a minimum time difference of 2 hours between each point? Is anything like that possible in QGIS?

Btw. The data is ordered in this format: YYYY-MM-DD hours:minutes

  • Are there timestamps larger than a 2 hour interval? And are the timestamps precisely taken every X hours? – Konan Pruiksma Nov 24 '17 at 14:25
  • I don't believe this is possible. An option is delete all records between the two times and go to the next interval. This can be done with, for example, a pythonscript. – B.Termeer Nov 24 '17 at 14:27
  • The timestamps are all around 2 hours but the timestamps are not very precisely. Sometimes 1:58, sometimes 2:05. – Ruben van Beek Nov 27 '17 at 14:19
1

Although this type of filter does not exist in QGIS (as for as I know), I've made a small python script that performs the algorithm that you may be after. The only thing left for you to do is reference your own dataset instead of the PingRecords dataset that I used in the script.

Note that this script doesn't account for the imprecise readings of your data and may not be the most elegant solution for your project as we don't know what file format or schema your data is in.

As a work around, you could export your spatial dataset to a csv, run the csv through the script below, write out the results to another csv, join the csv back to the spatial dataset in QGIS.

#-------------------------------------------------------------------------------
# Use the script at your own risk
#-------------------------------------------------------------------------------

from datetime import datetime
from datetime import timedelta
from operator import itemgetter

def main():
    time_interval = timedelta(hours=2)  # The desired time gap of 2 hours

    # This is your dataset with 2 columns "ID" and "datetime"
    PingRecords = [
            ["A", datetime.strptime("20-12-2017-10:15", "%d-%m-%Y-%H:%M")], # keep record
            ["B", datetime.strptime("20-12-2017-10:45", "%d-%m-%Y-%H:%M")],
            ["C", datetime.strptime("20-12-2017-11:15", "%d-%m-%Y-%H:%M")],
            ["D", datetime.strptime("20-12-2017-11:45", "%d-%m-%Y-%H:%M")],
            ["E", datetime.strptime("20-12-2017-12:15", "%d-%m-%Y-%H:%M")], # keep record
            ["F", datetime.strptime("20-12-2017-14:15", "%d-%m-%Y-%H:%M")], # keep record
            ["G", datetime.strptime("20-12-2017-16:15", "%d-%m-%Y-%H:%M")], # keep record
            ["H", datetime.strptime("20-12-2017-17:15", "%d-%m-%Y-%H:%M")],
            ["I", datetime.strptime("20-12-2017-18:15", "%d-%m-%Y-%H:%M")], # keep record
            ["J", datetime.strptime("20-12-2017-20:15", "%d-%m-%Y-%H:%M")], # keep record
            ["K", datetime.strptime("20-12-2017-22:15", "%d-%m-%Y-%H:%M")]] # keep record

    # Sort the dataset in chronological order using the date column
    sorted(PingRecords, key=itemgetter(1))

    # Set initial parameters
    start_rec = 0
    record_to_compare = len(PingRecords) - 1

    # Set the initial record locations within the dataset list
    i = 0
    n = i

    PingRecordsNew = [] # This will hold the new records with 2 hour intervals

    # Loop through the ecords within the supplied dataset
    total_records = len(PingRecords)
    while total_records > 1:
        calc_time_interval = timedelta() # Set initial time difference calculation to 0

        # Start traversing through the entire dataset in chronological order.
        while calc_time_interval < time_interval:
            StartRecord = PingRecords[i][1] # This gets the datetime value
            n = calculateComparisonRecord(n, record_to_compare) # Set the next record in the list to compare with the initial point
            NextRecord = PingRecords[n][1]

            # Find the time interval between the 2 records
            calc_time_interval = NextRecord - StartRecord
            total_records = total_records - 1

        PingRecordsNew.append(PingRecords[i])

        i = n
        next_rec = calculateInitialRecord(n)
    PingRecordsNew.append(PingRecords[n])

    print PingRecordsNew # This list hold the data that has been filtered

def calculateInitialRecord(start_rec):
    StartRecord = start_rec
    return StartRecord

def calculateComparisonRecord(newRec, record_to_compare):
    if newRec < record_to_compare:
        NextRecord = newRec + 1
        return NextRecord

if __name__ == '__main__':
    main()

Output

[['A', datetime.datetime(2017, 12, 20, 10, 15)], 
['E', datetime.datetime(2017, 12, 20, 12, 15)], 
['F', datetime.datetime(2017, 12, 20, 14, 15)], 
['G', datetime.datetime(2017, 12, 20, 16, 15)], 
['I', datetime.datetime(2017, 12, 20, 18, 15)], 
['J', datetime.datetime(2017, 12, 20, 20, 15)], 
['K', datetime.datetime(2017, 12, 20, 22, 15)]]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.