Data loss with Collector for ArcGIS

Question

Last week we used Collector for ArcGIS (version 10.3 on an iPad Air, AT&T) with our ArcGIS Online organization account for field data collection. We experienced loss of data, sometimes everything including the point feature itself, other times photo attachments. Since the collection was at or near urban areas and cellular signal strength was good in general, we didn't use offline mode and it was all done real time.

On day 2 of the collection there were a few instances with multi-photo points when the “submit” button was tapped it would take the App a while to process then a submission failed message would appear, offering two options – “Cancel” and “Retry”. Initially “Retry” was selected but the same failure message would appear again and again, so eventually the field crew clicked on “Cancel”. At that point the collected feature as well as attributes and photos were visible on the App, so they moved on to the next site.

When they were back at the hotel (on Wi-Fi) they realized 2 points they collected during the day were completely gone, and 3 other points were missing all the photo attachments. They also noticed 2 points from day 1 were completely gone (they couldn't recall with certainty whether the “failed to submit” message was encountered with them). Not knowing how to avoid more data loss, they went with paper and pen for the remaining of the collection and re-collection of the lost points.

Once they were back in the office we did more testing. While the cellular signal strength was at least 3 bars out of 5, it was pretty easy to duplicate the issue by taking more than 3 or 4 photos and submit.

Here are our questions:

1. Is our instinct correct that our issue could be caused by the data volume submitted each time?

2. We like the convenience and ease of set up with this solution. However based on our first project the reliability is too low for serious application. I can’t imagine we are the only one needing multiple photos for each collected features. Has data loss been a common issue?

3. What is everyone else doing to make this solution work? I assume the reliability will improve if each submission is limited to one photo, but it’s such an interruption to the workflow.

4. When collecting in “online mode”, is there a way to explicitly back up the data collected onto the mobile device?

5. We are thinking maybe offline mode can be more reliable and will do some tests. But from experience, will syncing (Wi-Fi or cellular) be problematic if a collected feature contains a large number of photo attachments?

6. Any recommendation on more reliable solutions that works with ArcGIS Online?

7. How about complete on device Apps?

Update: after talking with an ESRI technical support back and forth, he started being able to duplicate the problem on his end, and has registered this issue as a bug. We did perform more tests of our own. It seems that one photo per submission/update is problem free, so while on cellular only if multi photos are needed for a collection point you'll have to update after each photo is taken. Offline mode seems to be working fine.

Answer 1

Here are some responses based on my experiences working with a team using Collector over the last year or so.

Is our instinct correct that our issue could be caused by the data volume submitted each time?

• we've only used fairly light datasets so far, but see the photo bullet.

We like the convenience and ease of set up with this solution. However based on our first project the reliability is too low for serious application. I can’t imagine we are the only one needing multiple photos for each collected features. Has data loss been a common issue?

• We've never lost data. only had to recover it by downloading on iPad geodatabase when synch'ing fails.

What is everyone else doing to make this solution work? I assume the reliability will improve if each submission is limited to one photo, but it’s such an interruption to the workflow.

• We have scripted the removal of photos from the geodatabase on server side so that Collector is not trying to sync large amounts of data (photos), and yes limited to only a couple of photos. If you're taking a lot of photos, you need a different toolset, in my opion. Collector also appears to remove all the exif data that is of use, so avoiding it for photos is not a bad idea.

When collecting in “online mode”, is there a way to explicitly back up the data collected onto the mobile device?

• good question. i don't know.

We are thinking maybe offline mode can be more reliable and will do some tests. But from experience, will syncing (Wi-Fi or cellular) be problematic if a collected feature contains a large number of photo attachments?

• we generally are working in areas of intermittent cellular coverage so we've usually worked in an offline mode, however we rarely have data submission issues when online, only when synch'ing from offline

Answer 2

Disclaimer: I work for AmigoCloud and we have a data collection solution.

Some answers:

Is our instinct correct that our issue could be caused by the data volume submitted each time?

Doesn't seem like a big volume, so I doubt that was the issue.

We like the convenience and ease of set up with this solution. However based on our first project the reliability is too low for serious application. I can’t imagine we are the only one needing multiple photos for each collected features.

Multiple photos associated with a feature is not out of the ordinary.

Has data loss been a common issue?

This is a bigger question. There are workflows that can cause data loss, for example, if you edit the schema (e.g. add a field) while the devices are out on the field.

What is everyone else doing to make this solution work? I assume the reliability will improve if each submission is limited to one photo, but it’s such an interruption to the workflow.

Your workflow should work with any of the data collection solutions, this doesn't seem to the issue.

When collecting in “online mode”, is there a way to explicitly back up the data collected onto the mobile device?

Most of the data collection solutions keep a copy of the data submitted locally - even in "online mode". That data is usually deleted when the device syncs.

We are thinking maybe offline mode can be more reliable and will do some tests. But from experience, will syncing (Wi-Fi or cellular) be problematic if a collected feature contains a large number of photo attachments?

Should not be a problem.

Now to answer your question about Collector, if I had to guess, it probably has to do with schemas changing or intermittent connection which kills the sync before it can finish submitting the records. This is a typical problem with most data collection apps.

Any recommendation on more reliable solutions that works with ArcGIS Online?

Here is where the shameless plug comes in. You can get that project setup in AmigoCloud in less than 1 minute.

Check out this 60s video: https://youtu.be/lj6OeTZgzM8