Attempting to bring 2,300 .sid files into one raster

Question

I've tried all of the raster tools within ArcToolbox with no luck having such a large number of files. There are about 7GBs worth of MrSID files totaling about 2,300. Most tools fail or process too slowly when I try to upload all the files. Even more than 10 files seem to slow ArcGIS Desktop down too much. I don't know whether it's my procedure or computer that's causing problems.

Answer 1

A few things to think about.

1. What is the uncompressed size? If you figure 20 to 1 compression, you are looking at 140GB of rasters. Any processing you do will involve expanding them to full size, then merging. Are you planning to maintain them as MrSID or move to a different format?
2. Any logical groupings to batch process these? Set up a script to take 20 at a time and merge them to a raster in a new directory. Work through all of them. Then move to the new directory and process those a few at a time into a larger raster.
3. Incorporate multiprocessing any way you can, again, perhaps with a python script.
4. Since MrSID is a lossy data compression technique for anything above a 2:1 ratio, do you have access to the source files? This will enable you to create a merged raster that contains all of the original data, vs expanding the MrSIDs, then merging and compressing again which will have inherent data loss.
5. Storage space. ArcGIS creates a number of working files when processing raster images. You need to make sure that you have as much space in your workspace directory, as you do in your actual data directory.
6. Consider the option to create a Raster Catalog, which simply references each individual raster you have now, as opposed to creating a new Raster with a tool like: Workspace to Raster Dataset. From the help:
   • UNMANAGED —With an unmanaged raster catalog, the raster catalog only contains links or pointers connecting a row to a raster dataset stored outside the geodatabase. All raster datasets loaded into an unmanaged raster catalog must be a file on disk.

With regard to creating the batch python script, you have the right idea to create a list of all the raster names. The next step is to work through that list, and have a loop that is adding rasters to the merge tool, or whatever you choose, limited to a number. Once you reach that number it repeats, but with a new name, and so on.

For example: Names: Raster0001 through Raster0050

• Merge: Raster0001 to Raster0010 to form Raster0000_001
• Merge: Raster0011 to Raster0020 to form Raster0000_002
• Merge: Raster0021 to Raster0030 to form Raster0000_003
• Merge: Raster0031 to Raster0040 to form Raster0000_004
• Merge: Raster0041 to Raster0050 to form Raster0000_005

I don't have any python syntax off hand, but a search for batch processing with python will definitely yield you some good results.

Answer 2

i dont know what you will do 2,300 .sid files into one raster but if you want to tile or anything with it, you can use gdalbuildvrt.

gdalbuildvrt all_sid_files.vrt sids/*.sid

Answer 3

I believe I have found an answer. After days of work, with nothing to show for it, it seems quite simple now. My computer could not execute any task with regards to these files because it would run out of memory. MrSID (.sid) files are generally way too large to be used in geoprocessing so they must be converted to a more raster-friendly format such as GeoTIFF (.tif). I found a file converter available for free download (Contenta Converter) which took all of the MrSID files and converted them to GeoTIFF.

The 2,300 MrSID files I was using were 7.6GB compressed. No joke, using a trial of the LizardTech software in an attempt to mosaic all the files to verify this, the uncompressed size was 10.7TB!!! An unbelievable compression ratio. This partly explains the advantage of MrSID files for data transfer but a drawback for analysis as during such analysis, files must be decompressed in order to be usable. The decompressed size of the GeoTIFF files after conversion was 27.8GB, albeit still a large number, but much easier for Arc to deal with in terms of geoprocessing.

Once the rasters were of the GeoTIFF file format, mosaicking and building of raster catalogs and datasets was possible. Before the conversion, tools would not even run. Now they run well, while still taking the required hours to complete, and all 2,300 rasters have been mosaicked into one large, continuous image.

Answer 4

Perhaps consider using a Mosaic Dataset. You didn't mention what version of ArcGIS you are using but they were introduced at 10.0.

A mosaic dataset allows you to store, manage, view, and query small to vast collections of raster and image data. It is a data model within the geodatabase used to manage a collection of raster datasets (images) stored as a catalog and viewed as a mosaicked image. Mosaic datasets have advanced raster querying capabilities and processing functions and can also be used as a source for serving image services.