Tell me more ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.
up vote 8 down vote favorite

For "renting purposes", I have to specify the computing resources I need from the computing center at my University. I have very little idea of what to ask for.

Considering the information below may you suggest something?

This is the computational cluster: "A ten blade server with each blade consisting of 2 Quad Core Intel Xeon 2.33 Ghz CPUs with 16 GB of memory for a total of 80 CPUs and 160 GB of memory. Accounts on a 2 TB disk system. Running 64-bit Linux"

With lossless compression, the original data is one 50gb tiff file. At some point in time i shall be working with several (up to ten) files 50gb in size.

I would use GDAL, Python scripting and maybe C++ scripting. If allowed, I would use softwares like Grass GIS and Saga GIS. Also, R language with spatial libraries. I will be deriving the usual terrain parameters, trying to apply object-oriented algorithms for the extraction of specific features (landforms) and using statistical analysis for descriptive and modeling purposes.

Thank you in advance, Marco

share|improve this question
3  
What exactly are you expecting from an answer -- what quantity or quantities are able to be requested? Number of blades, number of cores on 1 blade, etc.? Is there a form you have to fill out that might give any clues? – blah238 Nov 7 '12 at 3:03
1  
Hi blah. There's no form. My supervisor is asking me these questions beforehand (with something in mind I am not aware of). But, later, when accessing the platform, the number of processors should be specified exactly, as well as the expected memory needs and processing time. So, it would be good to have an idea on the no. of processors and the amount of memory that would allow performing simple matrix algebra (A*0.1+B+C/50), each of the matrices 50Gb in size in, for example, less than one hour (considering that the software allows parallel computing). Thank you. – Marco Nov 7 '12 at 22:09
2  
It may help if you determine your strategy for attacking the problem first. I don't believe that 'out of the box' your Python scripts (using GDAL bindings) will be able to take advantage of multiple processors. How do you plan to divide up the data and work for parallel processing. You could then run some tests on a chunk of the data and extrapolate total processing time based on the number of cores that you plan to use, etc. – DavidF Nov 9 '12 at 16:03
Thanks David. I have thought more thoroughly about this. I'll do some tests with Matlab. – Marco Nov 10 '12 at 23:44

1 Answer

up vote 1 down vote accepted

So, it would be good to have an idea on the no. of processors and the amount of memory that would allow performing simple matrix algebra (A*0.1+B+C/50)

As DavidF stated in the comments more important is the stategy, never mind the machine, you can't run (or it's not a good idea to run) a 50GB matrix algebra with the whole matrix at once since conceptually it implies that the whole matrix has to be written to memory.

A good strategy, fast, very easy and efficient is to use gdal_calc, is reads and writes the raster in chunks so it is very memory efficient.

For example: gdal_calc.py -A input.tif -B input2.tif --outfile=result.tif --calc="(A+B)/2"

Try it, it's very likely that you can run the processing in you desktop, and then you may just need a better machine to speed up the process or not.

Obs: You need to spam multiple gdal_calc processes to take advantage of multicore processors.

share|improve this answer
Muito obrigado! Thanks a lot Pablo. This really is of great help! – Marco Mar 19 at 21:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.