3

I'm looking for a programmatic way to download MODIS/MCD19A2 AOD data in R. So far, I've found a pretty solid R package called MODISTools that works nicely with the MODIS/VIIRS RESTful API. However, this web service does not provide access to the AOD product that I need (MCD19A2). Currently, the only way I know of downloading the AOD product is manually via the Nasa EarthData Search tool, or via the NASA Data Pool.

Is there anyone out there that has found a more convenient and programmatic way to work with the MCD19A2 AOD product?

3

This can be easily achieved using the MODIS package, version 1.1.3 or higher (previous versions didn't provide access to this product).

library(MODIS)

# set MODIS options (these can also be passed to '...' in getHdf())
MODISoptions(localArcPath = "OutputDestinationFolder", quiet = FALSE)

# download data
hdf = getHdf("MCD19A2", collection = "006"
             , tileH = 9, tileV = 4
             , begin = "2018.08.28", end = "2018.08.31")
hdf
# $`MCD19A2.006`
# [1] "OutputDestinationFolder/MODIS/MCD19A2.006/2018.08.28/MCD19A2.A2018240.h09v04.006.2018242043359.hdf"
# [2] "OutputDestinationFolder/MODIS/MCD19A2.006/2018.08.29/MCD19A2.A2018241.h09v04.006.2018250043019.hdf"
# [3] "OutputDestinationFolder/MODIS/MCD19A2.006/2018.08.30/MCD19A2.A2018242.h09v04.006.2018250043020.hdf"
# [4] "OutputDestinationFolder/MODIS/MCD19A2.006/2018.08.31/MCD19A2.A2018243.h09v04.006.2018250043021.hdf"

This downloads all .hdf files found for the specified tile indices and time frame and puts them in the desired target folder. You might also use runGdal() instead to download and extract the data in one go, provided that GDAL is available. This renders unnecessary an explicit call to getHdf().

tfs = runGdal("MCD19A2", collection = "006"
              , tileH = 9, tileV = 4
              , begin = "2018.08.28", end = "2018.08.31"
              , job = "MCD19A2", SDSstring = "1010000000000")
tfs
# $`MCD19A2.006`
# $`MCD19A2.006`$`2018-08-28`
# [1] "OutputDestinationFolder/PROCESSED/MCD19A2/MCD19A2.A2018240.Optical_Depth_047.tif"
# [2] "OutputDestinationFolder/PROCESSED/MCD19A2/MCD19A2.A2018240.AOD_Uncertainty.tif"  
# 
# $`MCD19A2.006`$`2018-08-29`
# [1] "OutputDestinationFolder/PROCESSED/MCD19A2/MCD19A2.A2018241.Optical_Depth_047.tif"
# [2] "OutputDestinationFolder/PROCESSED/MCD19A2/MCD19A2.A2018241.AOD_Uncertainty.tif"  
# ...

For further information, see also

  • ?getProduct() for a list of available products;
  • getProduct("MCD19A2") for details on a paricular product;
  • argument 'outDirPath' in ?MODISoptions() to specify a target folder for extracted layers;
  • and arguments 'dlmethod' and 'dataFormat' in ?MODISoptions() for supported download methods and write formats, respectively.
  • Are you using a dev version of the MODIS package? Right now I'm unable to download the MCD19A2 data with getHdf() or runGdal(). Also, getProduct("MCD19A2") produces the following error: No product found with the name MCD19A2.. – spacedSparking Sep 12 '18 at 8:53
  • 1
    MODIS_1.1.3 hit CRAN only yesterday. I am using this version; it includes a set of newly introduced products as well as snow and ice cover datasets from NSIDC, which haven't been available before. – fdetsch Sep 12 '18 at 8:54
  • 1
    Installing MODIS_1.1.3 solved my problem. Thanks for the help! – spacedSparking Sep 12 '18 at 8:59
  • 1
    You're very welcome! I also added the required version to my answer. If you should encounter any difficulties, bugs, etc., please feel free to report back to github.com/MatMatt/MODIS/issues - we're thankful for any additional input. – fdetsch Sep 12 '18 at 9:01
3

Okay after a bit of digging, I've figured out how to download MCD19A2 AOD data using wget and R!

I've created a basic R function that downloads all .hdf files for a given date based on the specified MODIS tile. If you are a linux user and have a NASA EarthData account, then the following code should also work for you!

getAOD <- function(user = 'EarthDataUsername'
               ,pw = 'EarthDataPassword'
               ,dir = 'OutputDestinationFolder'
               ,product = 'MCD19A2' # default product for MAIAC AOD
               ,date = '2018.08.28' # day of interest
               ,tile = 'h09v04' # your tile id (depends on your location)
               ,collection = "006") {

  # parse date and julian date for url building
  Date <- lubridate::ymd(date)
  julianDate <- paste0(lubridate::year(Date)
                   ,lubridate::yday(Date))

  # extracts hdf files for a given tile across all observation times
  pattern <- paste0(product, ".A", julianDate, ".", tile, ".", collection, "*.hdf")

  # base url for wget
  url <- paste0("https://e4ftl01.cr.usgs.gov/MOTA/", product, ".", collection, "/", date, "/")

  # call wget from command line
  system(paste0("wget -r -A", pattern, " -L --user=", user
            ," --password=", pw
            ," --directory-prefix=", dir
            ," --load-cookies ~/.cookies --save-cookies ~/.cookies ", url))

}

I keep my EarthDataPassword in a file called batteries.dppss. I then scan this while calling the getAOD() function.

getAOD(user = 'myUserName'
    ,pw = scan("./batteries.dppss", what = "")
    ,dir = "./test")
2

If you don't have wget installed, or prefer not to rely on system calls, see https://github.com/AustralianAntarcticDivision/blueant/issues/11. It's very similar to your code, but the recursive downloading all happens in R, not wget.

  • Thanks for the link! I'm still trying to get bowerbird installed/configured, but I'm interested in comparing the run times between this and wget. – spacedSparking Sep 8 '18 at 0:04
  • 1
    Run times will almost certainly be faster with wget – Ben Sep 8 '18 at 2:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.