1

I'm working in an undergraduate project and just started using Google Earth Engine. Firstly, I need to filter some Sentinel-2 MSI images that cover my area of interest (a watershed with 55 km²) from 2016 to 2022 (the idea is to get one image per month). So far, I've applied several filters. I noticed that the QA60 band for the 2022 images is not working properly, so I decided to use the Cloud Probability collection for analysing cloud coverage. After applying the function get_cloud_percent(), which I created to analyze the cloud band within my area of interest and add the information to the metadata, I got unable to get the metadata as a list using my_dataset.getInfo(). It works just fine before the application of get_cloud_percent(). But the execution of the last line of the code bellow returns the following error: EEException: computation timed out. Could anyone point out what I'm doing wrong? I've coded in Python, as follows:

# %% SHAPEFILE UPLOAD

# Uploading the shapefile
lc_shape = gpd.read_file('lc_bh/bacia_LC.shp')
print(f'\nShapefile CRS is {lc_shape.crs}')

# Conversion of CRS to WGS84
lc_shape = lc_shape.to_crs(epsg=4326)
print(f'\nShapefile CRS converted to {lc_shape.crs}')

# Selection of the feature of interest
lc_shape = lc_shape.geometry[0][2]

# Extraction of the contours
# Coordinates are generated in pairs
lc_ext_coords = lc_shape.exterior.coords

# A tuple is generated to store a list of latitudes and a list of longitudes
lc_ext_coords = lc_ext_coords.xy

# Coordinates grouped in pairs (each pair latlong forms a list)
lc_ext_coords = np.dstack(lc_ext_coords).tolist()

# Creation of a ee.Feature with the external coordinates
lc_geometry = ee.Geometry.Polygon(lc_ext_coords)

# %% UPLOADING OF SENTINEL-2 DATA

# Initial and final dates
DATE_I, DATE_F = '2016-01-01', '2022-03-31'

# Sentinel 2 Harmonized data
# Filter: by date
# Filter: by bounds (select the images that touch the geometry)
dataset_1 = ee.ImageCollection('COPERNICUS/S2_HARMONIZED')\
    .filterDate(DATE_I, DATE_F).filterBounds(lc_geometry)

# %% FILTERING BY IMAGE EXTENT AND QUALITY

# Filter: images must contain the area of interest
dataset_2 = dataset_1.filter(ee.Filter.contains('.geo', lc_geometry))

# Filter: the images must have a proper general quality
dataset_3 = dataset_2.filter(
    ee.Filter.Or(ee.Filter.eq('GENERAL_QUALITY', 'PASSED'),
                 ee.Filter.eq('GENERAL_QUALITY_FLAG', 'PASSED')))

# %% INCLUSION OF CLOUD PROBABILITY BAND

# Cloud Probability data
cloud_prob_data = ee.ImageCollection('COPERNICUS/S2_CLOUD_PROBABILITY')\
    .filterDate(DATE_I, DATE_F).filterBounds(lc_geometry)

# The cloud probability band is added to the image dataset considering
# metadata matches
dataset_4 = dataset_3.combine(cloud_prob_data)

# %% CLOUD MASK ANALYSIS

# Function used on the processing
# Used because the values are quite low
def scale_dict(key, value):

    """
    This function takes an ee.Dictionary and multiply its values
    by a factor of 1e4 and rounds it to the nearest integer
    """

    return ee.Number(value).multiply(1e4).round()


# Function to be mapped over the collection
def get_cloud_percent(image):

    """
    This function calculates the cloud cover proportion at the geometry and
    adds it to the metadata as a new attribute called cloudiness, whose values
    are multiplied by a factor of 1e4
    """

    # Selection of cloud probability band
    # Clouds assumed as pixels with probability >= 70%
    # It is assigned 1 to the selected pixels and 0 to the others
    cloud_band = image.select(['probability']).gte(70)

    # Generation of cloud proportion
    # Since the values are just 0 e 1, the mean is equal to the proportion
    # A ee.Dictionary is generated with a key renamed to "cloudiness"
    # The proportion (not the percentage) is multiplied by 1e4 and rounded
    cloud_percent = cloud_band.reduceRegion(**{
        'reducer':ee.Reducer.mean(),
        'geometry':lc_geometry,
        'scale':10}).rename(['probability'], ['CLOUDINESS'], True)\
        .map(scale_dict)

    # Information added to metadata
    return image.set(cloud_percent)


# Function mapped over the collection
# Filter: election of images with cloudiness <= 1% (0.01x1e4)
dataset_5 = dataset_4.map(get_cloud_percent)\
    .filter(ee.Filter.lte('CLOUDINESS', 0.01*1e4))

# Get list of metadata
info_list = dataset_5.getInfo()

2 Answers 2

1

When you call getInfo you are using the "interactive" processing environment, which is intended for small requests that finish quickly (learn more here). If you do dataset_5.limit(5).getInfo() it should work (limit to small amount for checking). What do you intend to do with all the metadata? Maybe consider using ee.ImageCollection.reduceColumns and ee.ImageCollection.aggregate_* functions to summarize the metadata.

1
  • I was doing some filtering using the metadata in Pandas Dataframe format. But I reduced a lot of code by filtering it with the proper GEE functions. However, now I need to select a single image per month, and I'm not sure about how to do it without using loops over a dataframe. This is the reason I'd like the metadata (some specific attributes). If I use ee.ImageCollection.reduceColumns, I also can't use getInfo() to get a Python list (i.e., a list I can read). In this case, should I export the metadata or there is something equivalent to getInfo() I could use to save it as a list?
    – Bruno Rech
    Commented Apr 20, 2022 at 14:29
0

The "calculation timed out" is because you are processing many images in addition to calculating the percentage of clouds in each one.

Try to use this code to select only one image per month with COPERNICUS/S2_CLOUD_PROBABILITY. It works for up to 6 months due to the number of images that exist, but it will depend on your area.

BTW the shadows of the clouds are seen in the image.

1
  • Welcome to GIS SE. AS a new user, please take the Tour. Coding Questions without code get closed, and coding Answers without code get deleted (since all links fail over time, link-only answers are flagged for deletion). Please place the relevant code block within the body of your answer.
    – Vince
    Commented Apr 24, 2022 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.