Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I've tried all of the raster tools within ArcToolbox with no luck having such a large number of files. There are about 7GBs worth of MrSID files totaling about 2,300. Most tools fail or process too slowly when I try to upload all the files. Even more than 10 files seems to slow Arc down too much. I don't know whether it's my procedure or computer that's causing problems. Please provide whatever advice you have. Thank you.

share|improve this question
    
Got any other software at your disposal, say GlobalMapper maybe? –  Chad Cooper Nov 1 '12 at 14:54
    
Unfortunately, no. –  Matt Garcia Nov 1 '12 at 15:07
    
A few things to think about. 1. What is the uncompressed size? If you figure 20 to 1 compression, you are looking at 140GB of rasters. This is what the processing is going to have to deal with, is the uncompressed size. 2. Any logical groupings to batch process these? Set up a script to take 20 at a time and merge them to a raster in a new directory. Work through all of them. Then move to the new directory and process those a few at a time into a larger raster. 3. Incorporate multiprocessing any way you can, again, perhaps with a python script. –  Get Spatial Nov 1 '12 at 15:11
    
Thank you. This seems to be taking me in the right direction. I was assuming I would need to use Python at some point but I was avoiding it because I have very limited experience. The basic of the script would be to create a raster list and then mosaic the files in the list? How would I stop at a certain number and have the next script pick up at that point? –  Matt Garcia Nov 1 '12 at 15:20
    
If you want the final output in mrsid, try Erdas. It can handle more tiles while mosaicking. –  mapsir May 21 '13 at 5:00

4 Answers 4

i dont know what you will do 2,300 .sid files into one raster but if you want to tile or anything with it, you can use gdalbuildvrt.

gdalbuildvrt all_sid_files.vrt sids/*.sid

i hope it helps you...

share|improve this answer

A few things to think about.

  1. What is the uncompressed size? If you figure 20 to 1 compression, you are looking at 140GB of rasters. Any processing you do will involve expanding them to full size, then merging. Are you planning to maintain them as MrSID or move to a different format?
  2. Any logical groupings to batch process these? Set up a script to take 20 at a time and merge them to a raster in a new directory. Work through all of them. Then move to the new directory and process those a few at a time into a larger raster.
  3. Incorporate multiprocessing any way you can, again, perhaps with a python script.
  4. Since MrSID is a lossy data compression technique for anything above a 2:1 ratio, do you have access to the source files? This will enable you to create a merged raster that contains all of the original data, vs expanding the MrSIDs, then merging and compressing again which will have inherent data loss.
  5. Storage space. ArcGIS creates a number of working files when processing raster images. You need to make sure that you have as much space in your workspace directory, as you do in your actual data directory.
  6. Consider the option to create a Raster Catalog, which simply references each individual raster you have now, as opposed to creating a new Raster with a tool like: Workspace to Raster Dataset. From the help:
    • UNMANAGED —With an unmanaged raster catalog, the raster catalog only contains links or pointers connecting a row to a raster dataset stored outside the geodatabase. All raster datasets loaded into an unmanaged raster catalog must be a file on disk.

With regard to creating the batch python script, you have the right idea to create a list of all the raster names. The next step is to work through that list, and have a loop that is adding rasters to the merge tool, or whatever you choose, limited to a number. Once you reach that number it repeats, but with a new name, and so on.

For example: Names: Raster0001 through Raster0050

  • Merge: Raster0001 to Raster0010 to form Raster0000_001
  • Merge: Raster0011 to Raster0020 to form Raster0000_002
  • Merge: Raster0021 to Raster0030 to form Raster0000_003
  • Merge: Raster0031 to Raster0040 to form Raster0000_004
  • Merge: Raster0041 to Raster0050 to form Raster0000_005

I don't have any python syntax off hand, but a search for batch processing with python will definitely yield you some good results.

share|improve this answer
    
I'd like to move them to a different format if possible. Is there one that will load faster and not take up as much space? I have found success with the unmanaged raster catalog. All the images display properly and fit together well. What is the next step from here? I'm afraid to do anything else as it always gets hung up during processing. Will this be a viable option for another user if they had the original MrSID files on-hand? This seems to save a lot of space and time. I'd like to avoid using Python. –  Matt Garcia Nov 1 '12 at 20:01
up vote 1 down vote accepted

I believe I have found an answer. After days of work, with nothing to show for it, it seems quite simple now. My computer could not execute any task with regards to these files because it would run out of memory. MrSID (.sid) files are generally way too large to be used in geoprocessing so they must be converted to a more raster-friendly format such as GeoTIFF (.tif). I found a file converter available for free download (Contenta Converter) which took all of the MrSID files and converted them to GeoTIFF.

The 2,300 MrSID files I was using were 7.6GB compressed. No joke, using a trial of the LizardTech software in an attempt to mosaic all the files to verify this, the uncompressed size was 10.7TB!!! An unbelievable compression ratio. This partly explains the advantage of MrSID files for data transfer but a drawback for analysis as during such analysis, files must be decompressed in order to be usable. The decompressed size of the GeoTIFF files after conversion was 27.8GB, albeit still a large number, but much easier for Arc to deal with in terms of geoprocessing.

Once the rasters were of the GeoTIFF file format, mosaicking and building of raster catalogs and datasets was possible. Before the conversion, tools would not even run. Now they run well, while still taking the required hours to complete, and all 2,300 rasters have been mosaicked into one large, continuous image.

Thank you to the users for their answers.

share|improve this answer

Have you considered using a Mosaic Dataset? You didn't mention what version of ArcGIS you are using but they were introduced at 10.0.

A mosaic dataset allows you to store, manage, view, and query small to vast collections of raster and image data. It is a data model within the geodatabase used to manage a collection of raster datasets (images) stored as a catalog and viewed as a mosaicked image. Mosaic datasets have advanced raster querying capabilities and processing functions and can also be used as a source for serving image services.

share|improve this answer
    
I'm using 10.0. I have tried using a Mosaic Dataset but the images are too large to allow for a clean process. The unmanaged raster catalog I have at the moment has all of the images mosaicked so I'd like to know where to go from there. Raster Catalog to Raster Dataset tool? BTW, once the images are mosaicked properly, will it help them look more unified as opposed to clearly showing that they were taken at different times of the day or year? –  Matt Garcia Nov 1 '12 at 20:18
    
I was able to "Create Referenced Mosaic Dataset" from the raster catalog with the 2,300 .sid files. This shows up in ArcMap as its own layer with footprints, a boundary, and images but nothing shows up in the footprints. To display the images in the raster catalog, I had to get rid of the wireframes. It seems as though there is no option like that with the Mosaic Dataset. –  Matt Garcia Nov 1 '12 at 20:25
    
Have you defined and built overviews yet? I don't think they will display above a certain scale until you do. –  blah238 Nov 1 '12 at 21:07
    
They display from any scale but take some time to load. Are overviews required with a raster catalog? Are they only used with mosaic datasets? Because I haven't been able to mosaic the dataset due to the size. –  Matt Garcia Nov 1 '12 at 21:22
    
So since I've come to realize that I need to create a Mosaic Dataset of all these image ,something I tried originally but could not carry out because of the size and number of files, what is the best method for mosaicking the unmanaged raster catalog? –  Matt Garcia Nov 1 '12 at 21:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.