I have a large dataset of lines (ca 400 000). Each line represents a calculated route on the same road network.

My goal is to summarise a total number of calculated trips that pass each part of the road network, ie. a line on line overlay resulting in an overlap count on each segment.

I've tried running it in the line on line overlayer in FME, but I keep running into memory issues. I've also tried intersecting in Spatialite, but have memory issues there as well. I'm stuck on a 32 bit computer at work, so if the answer is "that dataset should be fine on 64 bit with enough RAM" that could be a viable answer. But something also tells me there should be a more elegant way to go about it...

So, in short, what's an efficient way of counting overlaps/intersecting in a large line dataset? I have access to and mainly work with FME, QGIS and R.

  • 32bit will be restricted to the OS (Windows is 4GB). If you can get on a 64-bit Windows you get to use 8GB/16GB/32GB/64GB/128GB RAM (get as much RAM as you can afford). some of the transformers in FME also can do parallel processing that will cut the time down. We just went from 8GB to 64bit 32GB and benchmarked a 63% improvement speedwise. – Mapperz Dec 16 '16 at 14:49
  • 1
    btw FME best practice - when trying different methods - would be to extract an area of data, say around 25,000 features and try each method on that. Find out which is quickest before scaling up to the full dataset. – Mark Ireland Dec 16 '16 at 16:00

With FME... If it's the exact same piece of road network for each route - and it sounds like it is - then instead of comparing the entire piece of line, why not replace them with a bounding box and compare them? That should give you the same result with the bonus that there are way fewer vertices to compare.

In fact, if you just extract the min/max coordinates you could do the test as a text comparison, rather than geometry.

For example, try running the BoundsExtractor on all your data. Then use an Aggregator with a group-by set to the xmin,xmax,ymin,ymax values. The count attribute will tell you how many features there are in each group.

In fact, I would also use the GeometryRemover before the Aggregator. That's because you don't really need the geometry and it would only slow down the aggregation.

As far as I can see it would only fail if two road segments had exactly the same bounding box, which I find very unlikely. But if this doesn't help then let us know and maybe we can find a different method.

  • btw if your original data isn't split up at intersections, then use an Intersector transformer first to do that. – Mark Ireland Dec 16 '16 at 15:56
  • Thanks! I'm close to figuring it out with this approach. However, there's no original road network since the data is output from a model in a generalised form. As such, I'm having problem with cutting the lines at intersections at the start (since I effectively have to run all the data through the intersector, which is what I actually originally tried as it calculates overlaps). I'm trying to use a chopper set to two nodes, but it obviously creates a copious amount of features along the way. Is there a clever way to find intersections without the intersector? – hexamon Dec 19 '16 at 8:50
  • 1
    In the end, the chopper method actually worked fine. Since I'm not interested in routes that are covered by less than 10 trips anyway, the final geometry could be cut down considerably. Again, thanks! – hexamon Dec 19 '16 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.