When running code with the DL Tasks API, tasks will only recognize the functions and imported packages that are contained within the main function (argument f
within tasks.create_function
, or that are imported into the tasks.create_function
as modules.
Since get_features
and get_sentinel2_cloudmasked
are functions within your notebook, you will need to put those function definitions within the save_training_samples
function in order for the task image to find them. For the other imports (gpd
, np
, pickle
, etc.), these are likely currently imported at the top of your notebook. These imports also need to be moved into the save_training_samples
function.
If your script is importing other functions, say from a utils file, you can point the task to these files by including them into the modules
argument within tasks.create_function
. For example, you might have an import statement within your save_training_samples
function that says from utils import function_name
, which is importing the function function_name
from a utils.py
file. In order for the task image to find this function, you would include utils
as a module in tasks.create_function
, like so:
# define a new task group
tasks = dl.Tasks()
async_func = tasks.create_function(
f=save_training_samples,
name='get-training-timeseries',
image='us.gcr.io/dl-ci-cd/images/tasks/public/py3.7:v2021.01.27-1-g371bb0bf',
maximum_concurrency=10,
include_modules=['utils'], # include custom modules
)
You will also need an __init__.py
file that lives in the same directory as the utils.py
file, so that the task image will recognize this as a module.