I'm trying to map routes from an O-D survey with about 450,000 individual trips with different start and end points defined by lat, long. I want to estimate total VMT for the area by extrapolating from that. I'd been hoping to use the (really great) QNEAT3 plug-in to calculate the network cost of each route. But I've run into a problem. QNEAT3 generates a complete O-D matrix, measuring the network distance between each and every origin point and every other destination point. That's not a problem at all when dealing with a smaller data set. Since I've assigned a shared trip ID to each origin and destination pair, I'm able to just select the shortest path cost calculations I'm interested in by selecting rows where the route ID is the same for both. Then I can just ignore the "superfluous" data. The real problem is when the data set is so much bigger. Even parceling out 20,000 pairs at a time - something I can't feasibly do with so much data to get through - the number of extraneous calculations is so big that the operation inevitably fails after about 5 or 6 hours of processing.
I've spent a ton of time on this and I'm willing to learn any new skill that I might need to do this calculation. I'm just not sure what that might be. I'm very familiar with QGIS. I could probably struggle through something in R if that works. And maybe with step by step support could try to pull something off in PostgreSQL or Python. But I'm feeling pretty stuck.
Does anyone have any suggestions of how to go about this?