Is there a fast way to copy large raster datasets between File Geodatabases?

Question

Why does it take so long to copy a large raster dataset from one file geodatabase to another? I can copy the entire database in windows explorer in minutes but copying a single raster in ArcCatalog takes forever. I always assumed it is not advisable to use the windows explorer to copy data between to File Geodatabases but is there a safe way to do this? If so, how?

Answer 1

I think the key is the difference between how Windows Explorer treats a single file, and how ArcCatalog/ArcGIS treat a raster. There are a few factors here.

1. In the File GDB format, a raster is not represented by a single file. It is made up of the data table, along with probably pyramids and indexes. Why this is important is because when ArcCatalog is copying the raster, it is first combining all of these separate files into what we see as the raster. It is then running it through the ArcGIS programming layer to copy it to a new File GDB. This is much more complicated on the surface than just using the Windows System resources to copy a file because it is an additional layer of programming.

2. The next part is related to something that you see when you copy another feature class in ArcCatalog. If you copy and paste in ArcCatalog, it brings up a progress window showing the original and target featureclasses, and then an iterator that counts the number of features that have been transferred. This shows that it is iterating row by row through the featureclass. This makes sense from a data integrity standpoint as each feature is copied and verified before moving to the next. Where this is relevant for the Raster is that a raster is made up of hundreds, if not thousands of Points. If a raster dataset is copied the same way as a vector featureclass, then ArcCatalog is essentially iterating through each coordinate pair in the Raster, to copy it from one to another.

3. The last factor is the size of the dataset. In your case, the File GDB you are showing has a table that is 20,671,104 KB, which is ~20GB. That is a large Raster, and definitely contains many millions of points. That is a lot of coordinate pairs to iterate through.

When you combine the size of your Raster dataset, with the need to iterate over each row (Coordinate pair), and run that all through the ArcGIS programming layer, you end up with a much more complex, and thus longer operation than copying a single file using the Windows System functions.

-------Edit

To answer your original question of a faster way to copy rasters between file geodatabases, I would say one thing to try would be writing a python script calling arcpy. Since you would then be directly interacting with the API, without the additional ArcCatalog/ArcMap code running, it may be slightly faster. A code sample follows:

import arcpy
from arcpy import env

env.workspace = "C:/data"
arcpy.Copy_management("olddata.gdb/raster1", "newdata.gdb/raster1")

I'm not sure if the speed difference would be notable, but it is an option.
The other option, as @blah238 referred to, is to not store rasters in a file geodatabase, but in a file based format, like "geotiff", or even an ArcINFO Grid, as the file can then be completely manipulated through the Windows interface.

Unfortunately, other than this, your options for working with the file geodatabase outside of the ArcGIS interface are limited. There is an API for the File GDB which allows programmers to interface with it in different software. One example currently is through the GDAL/OGR libraries, and soon, Quantum GIS. The issue with the API is that it Does Not support Raster datasets at this time. If this is something you would like to advocate for, I would add your support to this entry at the ArcGIS Ideas site.