0

I would like to download Sentinel-2 L2A Granules using Open Data Protocol (OData) with new Copernicus Data Space Ecosystem. I have followed some tutorials on how to do it.

https://dataspace.copernicus.eu/news/2023-9-28-accessing-sentinel-mission-data-new-copernicus-data-space-ecosystem-apis You need to create a new account and use your credentials.

In addition, I used the code provided on the following jupyter notebook that includes thorough instructions.

https://github.com/eu-cdse/notebook-samples/blob/c0e0ade601973c5d4e4bf66a13c0b76ebb099805/sentinelhub/migration_from_scihub_guide.ipynb

I want to download many products; therefore, I created a loop. I have selected to download only the first three products from the 'catalog' since this is a test and I do not want it to take a lot of time. The problem is that even though it works for the first product (I get a zip file with the data) but not for the other two products. The zip files (for the 2nd and 3rd product) are created but their size is 1 kb and when I open them, the zip files are corrupted. I introduced a mechanism (try and except) to retry failed downloads but it did not solve the problem.

To reproduce the error, you need create an account to Copernicus Data Space Ecosystem and use the credentials.

import time
import os
import datetime
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import requests

start_date = "2022-06-01"
end_date = "2022-06-10"
data_collection = "SENTINEL-2"
aoi = "POLYGON((4.220581 50.958859,4.521264 50.953236,4.545977 50.906064,4.541858 50.802029,4.489685 50.763825,4.23843 50.767734,4.192435 50.806369,4.189689 50.907363,4.220581 50.958859))'"

json = requests.get(f"https://catalogue.dataspace.copernicus.eu/odata/v1/Products?$filter=Collection/Name eq '{data_collection}' and OData.CSC.Intersects(area=geography'SRID=4326;{aoi}) and ContentDate/Start gt {start_date}T00:00:00.000Z and ContentDate/Start lt {end_date}T00:00:00.000Z").json()
catalog = pd.DataFrame.from_dict(json['value'])
# I have selected only the first three products since this is a test
catalog_sel = catalog.head(3)

def get_keycloak(username: str, password: str) -> str:
    data = {
        "client_id": "cdse-public",
        "username": username,
        "password": password,
        "grant_type": "password",
        }
    try:
        r = requests.post("https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token",
        data=data,
        )
        r.raise_for_status()
    except Exception as e:
        raise Exception(
            f"Keycloak token creation failed. Reponse from the server was: {r.json()}"
            )
    return r.json()["access_token"]
        

keycloak_token = get_keycloak("xxxx@xxxxxxxx", "xxxxxxxxx")


max_attempts = 3  # Maximum number of attempts to download each product

for i in range(catalog_sel.shape[0]):
    product_i = catalog_sel.Id.iloc[i]
    attempts = 0

    while attempts < max_attempts:
        try:
            session = requests.Session()
            session.headers.update({'Authorization': f'Bearer {keycloak_token}'})
            url = f"https://catalogue.dataspace.copernicus.eu/odata/v1/Products({product_i})/$value"
            print(url)

            response = session.get(url, allow_redirects=False)
            while response.status_code in (301, 302, 303, 307):
                url = response.headers['Location']
                response = session.get(url, allow_redirects=False)

            file = session.get(url, verify=False, allow_redirects=True)
            file_name = f"outputs/attempt_3_product_{product_i}.zip"

            with open(file_name, 'wb') as p:
                p.write(file.content)

            # If the download is successful, break out of the loop and proceed to the next product
            break

        except requests.RequestException as e:
            print(f"Failed to download product {product_i}. Retrying...")
            attempts += 1
            time.sleep(1)  # Adding a small delay before retrying to avoid flooding the server

    if attempts == max_attempts:
        print(f"Failed to download product {product_i} after {max_attempts} attempts. Moving to the next product.")

    time.sleep(1)  # Adding 

1 Answer 1

0

The solution is that the keycloak_token should be inside the loop and run each time (for each product). I have provided the code snippet.

....
for i in range(catalog_sel.shape[0]):
    product_i = catalog_sel.Id.iloc[i]
    attempts = 0
    
    keycloak_token = get_keycloak("xxxx@xxxx", "xxxxxxx")

    while attempts < max_attempts:
        try:
        ....

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.