2

I'm about to start yet another project which will consume and produce several to tens of GB of data. For efficiency, performance, and source code control* I want to do everything on the local machine. For backup, sharing with colleagues, and future proofing everything needs to be on the server (in our organization only network resources are backed up).

Windows has a built in feature for scenarios like this called Offline Files which purports to be more or less fire and forget. Has anyone used it for large data sets like we work with? Is it worth attempting/exploring? (And is there any control over where the data is synced on the local machine?)

It's not that difficult to setup up a batch file and a scheduled task to robocopy or whatever from the local machine to the server, but I'd rather not write, debug and maintain that in addition to the actual project (times N projects). If Offline Folders work better, great! If not, well... ;-)

make folder "Always available offline"

* A portion of the project involves code and version control (e.g. git/mecurial), and it is ill-advised to run VC within remote folders.

Improve this question
3
  • Will you need to sync files daily? Maybe a Virtual Machine?
    – Mapperz ♦
    Commented Jan 21, 2015 at 17:57
  • @Mapperz, 2-4 times a day would be ideal, but it's okay if only nightly. I don't see how a virtual machine could be applied here, can you elaborate?
    – matt wilkie
    Commented Jan 21, 2015 at 21:40
  • You can build snapsnots in VM and replace the exsiting VM (on a users PC)overnight with up to date data. We do it with about 9GB of data nightly after some FME data vaildation processes.
    – Mapperz ♦
    Commented Jan 21, 2015 at 21:52

1 Answer 1

Reset to default
1

From a sample size of 1, my answer is DON'T.

With offline files enabled, and the relevant folder in "Work Offline" mode, a simple arcpy.ListFeatureClasses() balloons from the already-far-too-long 2min25s to 35min(!) For reference, same call to gdb on local SSD volume is 27.5s.

I haven't ruled out other problems, the two tests were conducted a week apart and at different times of day. We may have other completely unrelated network issues in the mix. If someone else would like to test Offline Folders my test script is at https://github.com/maphew/arcplus/blob/master/arcplus/profile-arcpy-listFC.py

Procedure:

  1. edit script, set input gdb accordingly (my test gdb is 385mb, 424 feature classes, nothing else)
  2. run, capture results
  3. enable Offline Files, set input gdb folder to Work Offline mode
  4. run, capture results
  5. repeat with other inputs, off and on network, diff times of day, etc.

(#5 is where I got off the bus and went back to my real project!)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.