You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
They've added the option to specify the retry strategy for the job.
Unfortunately for someone who relies on the default retry strategy, which can be set/customized in the AWS Batch job definition, this value is now ignored
What you think should happen instead?
When the user does not specify a retry_strategy in constructor BatchOperator it should default to None. Then it will automatically fall back to the job definition retry strategy.
It should NOT be set to {"attempt": 1} as it is currently the case.
How to reproduce
Create an aws job definition with a retry strategy, for example:
Apache Airflow version
2.9.1
If "Other Airflow 2 version" selected, which one?
No response
What happened?
There's been 2 recent changes to the AWS
BatchOperator
:They've added the option to specify the retry strategy for the job.
Unfortunately for someone who relies on the default retry strategy, which can be set/customized in the AWS Batch job definition, this value is now ignored
What you think should happen instead?
When the user does not specify a retry_strategy in constructor
BatchOperator
it should default to None. Then it will automatically fall back to the job definition retry strategy.It should NOT be set to
{"attempt": 1}
as it is currently the case.How to reproduce
BatchOperator
with noretry_strategy
orretry_strategy=None
Operating System
MWAA
Versions of Apache Airflow Providers
Using https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt
So:
But it is still happening with the latest version as far as I can tell:
airflow/airflow/providers/amazon/aws/operators/batch.py
Line 209 in 05b6b74
Deployment
Official Apache Airflow Helm Chart
Deployment details
MWAA
Anything else?
Here's a quick workaround that should work even with older version of the
BatchOperator
that don't haveretry_strategy
:Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: