1

I have asked a similar question about millions of points a few days ago:Display millions of points in the browser.

It focused on how to display these data in the client(browser), however I think it is still a big challenge to save the in the server side.

For example for a device like a truck, which will send its location to the server every 60 seconds(not short for trucks), and suppose the truck just work 15 hours one day, so there will be 15 * 60 = 900 records one day.

And if there are 100 trucks in the system, there will be 900 * 100 * 30 = 2700000 = 2.7 million points one month. Though we can reduce the points by removing the repeat points, but this can not take a explicit effect.

Now we use mongodb to save these points, however the storage size increase rapidly.

I wonder how do you handle this kind of use case?

  • 1
    Have you benchmarked how much disk space is using for each record? 2.7 million records a month isn't really that much for a modern database system on modern hardware, assuming even a very large disk footprint of 1024 bytes per record (though at the bare minimum as X:float,Y:float,timestamp:float,truck_id:uint32 it fits in 28 bytes) it amounts to about 2.6 gigabytes. That's still small enough to comfortably fit an entire month of records in memory on any modern computer. – Jason Scheirer Nov 15 '14 at 17:47
  • I have checked that,our another case:i.imgur.com/4z3lPqf.png, start a few days ago,only 10k+ records, but the file size are almost 200M. – giser Nov 16 '14 at 7:12
  • Are you able to tell how much of that is just baseline data? That is, graph the fileSize over the course of adding thousands of records? – Jason Scheirer Nov 16 '14 at 16:10
2

Assuming you are going to be doing some very geospatially analysis on this data. You should seriously consider moving to a GIS focused DB solution such as PostGIS. With that focus the plugins and geoanalysis tools are extensive and will prevent you from rebuilding the wheel. Most, if not all of the analysis you will undoubtedly be doing on this data has been done before and has been made into a module or plugin of postgis, and that has been done by people with extensive GIS backgrounds. Most of this analysis is also relational by nature, in your scenario.

Secondly, you may want to look into a method other than single record. You could, for example, store daily data as single records and then simplify on the server before moving into historical tables. Or you could store that on the remote device, if within specifications, and then then use something like TopoJSON to simplify into arcs to only return the most necessary point to the server for storage at the end of the day or at some reporting threshold.

http://postgis.net/

https://github.com/mbostock/topojson

  • For our case, we will not do very geospatially analysis, we just need to fetch the location traces for a certain time range. – giser Nov 16 '14 at 7:14
  • @giser you definitely should take Frank's advice and investigate a better-suited DBMS like PostGIS. If you truly don't need any advanced querying you might even get away with storing the data as flat files on the filesystem assuming good backups. This isn't a large dataset at all and simple logging/reporting like this is a long-ago solved problem. – Jason Scheirer Nov 16 '14 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.