Botocore will throw an error of botocore.exceptions.NoCredentialsError if too many s3 objects are requested simultaneously on an AWS EC2 instance (including Kubernetes.) In my use case this was because we had a foreach with 3000 splits. The following join step failed because metaflow was attempting to load all of the artifacts from the previous step.

This is because the code here creates a new boto session for every s3 url that is being downloaded. The AWS Instance Metadata Service has a throttle limit that causes the above error when it is queried to often.

The current workaround is to pass in boto3 credentials to the DATATOOLS_DEFAULT_CLIENT_PARAMS metaflow config variable. The solution looks something like this:

With the following in a creds.py file:

import boto3
session = boto3.Session()
c = session.get_credentials().get_frozen_credentials()
print('{{ "aws_access_key_id": "{}", "aws_secret_access_key": "{}", "aws_session_token": "{}" }}'.format(c.access_key,c.secret_key,c.token))

The flow can then be ran like so:

METAFLOW_DATATOOLS_DEFAULT_CLIENT_PARAMS=$(python creds.py) python flow_name.py …

My proposed solution is to pull the boto3 session creation outside of the worker function and put it into the start_workers function with some method of recreating the credentials if they expire. I'll work on a PR next week with the proposed solution