1

For MODIS satellite images how to automate the download, unzip, import and process to point and then to Excel for each climate satellite MODIS layer.

There are 3650 days of imagery to process with 8 layers per day across modis aqua and terra, giving LST AM and PM and the corresponding timestamps.

So this is 29,200 layers to be run over a set of point files to give the corresponding satellite data at each point for each layer type needed.

To me it seems like a massive undertaking but my knowledge of coding and automation is limited

I realise the broadness of the questions to further wittle down my queries from the already useful answers,

  • How do I create such a cloud library of modis data ?

  • How is such data then further automated using Python, specifically are there any examples of such automation or any places I need to go to learn this specific parts of Python?

If this is still too basic I apologise, but I am coming at this from a complete Python beginners perspective.

|improve this question

closed as too broad by Vince, whyzar, tinlyx, nmtoken, Midavalo♦ May 27 '17 at 20:05

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    Welcome to GIS SE. As a new user, please take the Tour. Unfortunately, GIS SE uses a "Focused question / Best answer" model, and you haven't asked a question yet ("Any advice?" doesn't count). Automation with Python is not all that difficult to learn; I strongly recommend that you spend a week learning Python and ArcPy, then step through your process, saving the commands issued into a processing script. If you have trouble, first research the problem, and it no answer is evident, then condense it into a reproducible problem, and ask it here. – Vince May 27 '17 at 12:08
  • @Vince it was good enough to get an answer – Stevetech May 27 '17 at 17:14
  • 2
    @Stevetech Just because somebody added an answer doesn't mean the question is necessarily suitable for our format. Questions asking for advice or suggestions don't fit nicely into a Question and Answer format where answers are given to specific problems. There is no specific problem in this question (in fact there is no question asked at all). – Midavalo♦ May 27 '17 at 20:38
  • The poster is specifically asking for a strategy to automate a manual process. – Stevetech May 27 '17 at 20:41
  • For automating the MODIS download you need to do it outside ArcGIS. You may bulk download the data from Earthdata without doing any scripting. If you are familiar with scripting you can use a library available on R. This R library is very easy to use and does not really need any hardcore scripting skills to use ModisDownload.R. Once the data is downloaded you may automate the processing part in ArcGIS very easily using model builder or Arcpy scripting, though Arcpy scripting requires little skills in Python – Mario May 28 '17 at 6:06
3

This task calls for an automated, scripted processing. You need to write a script which will download the data, unzip it, and then process it. Based on my experience with a similar task, I'll suggest the following pointers:

  • Python is a perfect fit for this task. If you are not comfortable with it, you better spend some time and effort learning it, and getting good at it.
  • While you might have experience in ArcGIS, and people would suggest that you use arcpy, it's worth exploring other APIs like GDAL, numpy & R. In my experience, the performance boost from using other libraries more than makes up for the time spent learning them.
  • The Amount of data which has to be downloaded and processed makes it good fit for the processing to happen in the cloud. The resources & cost required for downloading and storing large amounts of data on to a local workstation can get prohibitively expensive quite quickly. The cost savings of doing this in the cloud can be substantial.
|improve this answer
  • ^^^ agree, for beginners with no exp. with coding check out python for non-arcgis tasks and use Arcpy for the ArcGIS analysis (or check out R). the cloud services issue is one you will need to discuss with supervisors depending on budget for this project. – NULL.Dude May 27 '17 at 17:25
  • Thank you for your direction I now realise the issues ahead, what exactly does cloud processing of such data involve how do I get such data to the cloud? And further link this to arc and the processing jobs ? I am coming at this as a complete machine learning beginner. – Richard Hargreaves May 28 '17 at 12:31
  • Generally speaking, you don't usually push data to the cloud. You directly download data from the source into your cloud system. For example, since you are processing MODIS data, you can use something like: pymodis.org or download data from the LP DAAC FTP site on to your cloud system. – Devdatta Tengshe May 28 '17 at 16:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.