4

Bit of a theoretical one this, but I was wondering if anyone had tried to use CartoDB and Torque with any big(ish) data. What I am talking about is having the location of 75,000 people at 1 minute resolutions for a 24 hour period. Would torque 'work' or would it just crash? I've done it successfully with 200 people, but am now getting to the stage where I'll want to test this with much larger datasets.

1

We have done torque visualizations of 3,000,000+ points a few times* with no problem. We are also researching specialized indexes for tables used in torque visualizations that would make it even faster. That said, 100 million is a huge dataset. On the server side, torque can be a memory intensive, so I imagine you will hit the limit before this.

*http://carbon-tool.cartodb.com/viz/3a10a8bc-4ae9-11e3-9028-1359048250e5/embed_map?title=false&description=false&search=false&shareable=false&cartodb_logo=true&layer_selector=false&legends=false&scrollwheel=false&sublayer_options=1&sql=select%20the_geom,sightingdate,the_geom_webmercator,cartodb_id%20%20from%20vms_ecvesreg2%20where%20the_geom%20is%20not%20null&sw_lat=51.944264879028765&sw_lon=-13.53515625&ne_lat=58.10110549730587&ne_lon=7.55859375

  • Hi Andrew. Thanks for the response. Well I'll have a crack then and see what happens! ha. The data is pretty much ready to go, I just need to try and sort out a local installation (or AWS installation) of CartoDB so that I have enough space to put the data into to try. I guess I could always split the data up into age groups or something like that. In an ideal world I'll look to put some sort of filter that the user can play around with to show different people I think. Cheers. – TheRealJimShady Jan 21 '14 at 17:23
  • as a side note, we have a binary transfer file format that we used in some projects that save around 30% trasfer size and does not use extra memory in the client (it uses transfer buffer only) – javisantana Jan 22 '14 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.