2.10.2

2.10.2

Features

• New configuration option to use same headers as metadata service for argo events webhook calls by @oavdeev in #1560 . Default behavior is the same as before.

• Metaflow CLI now supports list-workflow-templates command to list deployed argo workflows by @saikonen in #1577

Full Changelog: 2.10.0...2.10.2

2.10.0

Coming soon!

Full Changelog: 2.9.15...2.10.0

2.9.15

Improvements

Improve the performance of parallel_map

We now check for processes in the order in which they complete not in the order in which they are launched. This also increases the likelihood of failing fast.

Fix issues with the environment escape mechanism

Deadlocks and errors could occur when using the environment escape mechanism in two cases: (a) GC would occur at an inopportune moment or (b) subprocesses were involved. Both issues were fixed.

What's Changed

• Fix two possible issues with the environment escape communication by @romain-intel in #1555
• Check for the first finished parallel proccess by @maxzheng in #1546

New Contributors

• @maxzheng made their first contribution in #1546

Full Changelog: 2.9.14...2.9.15

2.9.14

Improvements

Fixes merging of log lines

This release fixes an issue with merging broken log lines.

Fix issue with using LD_LIBRARY_PATH with Conda environments

In a Conda environment, it is sometimes necessary to set LD_LIBRARY_PATH to first include the Conda's environment libraries before anything else. Prior to this release, this used to cause issues with the escape hatch.

What's Changed

• fix: GPU scheduling issue by @saikonen in #1520
• vend packaging by @savingoyal in #1506
• fix: vendor older version of packaging for python 3.5 by @saikonen in #1530
• Fix an issue when LD_LIBRARY_PATH is overriden in a Conda environment by @romain-intel in #1540
• Fix the merging of borked lines with MFLOG by @romain-intel in #1533
• Revert "fix: GPU scheduling issue" by @savingoyal in #1542
• Bump version to 2.9.14 by @saikonen in #1543

Full Changelog: 2.9.13...2.9.14

2.9.13

Bug fix

Revert annotations changes to fix a regression

The recent annotations feature introduced an issue where project, flow_name or user annotations are not being populated for Kubernetes. This release reverts the changes.

What's Changed

• Revert "Adds custom annotations via env variables" by @savingoyal in #1516
• Bump to 2.9.13 by @savingoyal in #1517

Full Changelog: 2.9.12...2.9.13

2.9.12

Known issues

The annotations feature introduced in this release has an issue where project, flow_name or user annotations are not being populated for Kubernetes. This has been reverted in the next release.

Features

Custom annotations for K8S and Argo Workflows

This release enables users to add custom annotations to the Kubernetes resources that Flows create. The annotations can be configured much in the same way as custom labels

1. Globally with an environment variable. For example with

export METAFLOW_KUBERNETES_ANNOTATIONS="first=A,second=B"

2. At a step level by passing a dictionary to the Kubernetes decorator.

@kubernetes(annotations={"first": "A", "second": "B"})

What's Changed

• Adds custom annotations via env variables by @tylerpotts in #1442
• Pass the user-defined executable to environment's executable by @romain-intel in #1454
• Remove validate_environment from task lifecycle by @savingoyal in #1507
• Fix/863 - Improve error message in metaflow.S3 class when DATATOOLS_S3ROOT is not configured. by @tfurmston in #1491
• Fix an issue where 0 was not considered False for extension debug opt… by @romain-intel in #1511
• Bump version to 2.9.12 by @saikonen in #1514

Full Changelog: 2.9.11...2.9.12

2.9.11

Bug Fix

Fix regression for @Batch decorator introduced by v2.9.10

This release reverts a validation fix introduced in 2.9.10, which prevented executions of Metaflow tasks on AWS Batch

What's Changed

• Revert "fix: validate required configuration for Batch" by @savingoyal in #1486
• Bump version to 2.9.11 by @savingoyal in #1487

Full Changelog: 2.9.10...2.9.11

2.9.10

Features

Introduce PagerDuty support for workflows running on Argo Workflows

With this release, Metaflow users can get events on PagerDuty when their workflows succeed or fail on Argo Workflows.
Setting up the notifications is similar to the existing Slack notifications support

1. Follow these instructions on PagerDuty to set up an Events API V2 integration for your PagerDuty service
2. You should be able to view the required integration key from the Events API V2 dropdown
3. To enable notifications on PagerDuty when your Metaflow flow running on Argo Workflows succeeds or fails, deploy it using the --notify-on-error or --notify-on-success flags:

python flow.py argo-workflows create --notify-on-error --notify-on-success --notify-pager-duty-integration-key <pager-duty-integration-key>

4. You can also set the following environment variable instead of specifying --notify-slack-webhook-url on the CLI everytime

METAFLOW_ARGO_WORKFLOWS_CREATE_NOTIFY_PAGER_DUTY_INTEGRATION_KEY=<pager-duty-integration-key>

5. Next time the flow fails or succeeds, you should receive a new event on PagerDuty under Incidents (Flow failed) or Changes (Flow succeeded)

What's Changed

• fix: validate required configuration for Batch by @saikonen in #1483
• feature: add PagerDuty support for Argo Workflows by @saikonen in #1478
• Bump version to 2.9.10 by @saikonen in #1484

Full Changelog: 2.9.9...2.9.10

2.9.9

Improvements

Fixes a bug with the S3 operations affecting @conda with some S3 providers

This release fixes a bug with the @conda bootstrapping process. There was an issue with the ServerSideEncryption support, that affected some of the S3 operations when using S3 providers that do not implement the encryption headers (for example MinIO).
Affected operations were all that handle multiple files at once:

get_many / get_all / get_recursive / put_many / info_many

which are used as part of bootstrapping a @conda environment when executing remotely.

What's Changed

• fix: s3 op bug with ServerSideEncryption by @saikonen in #1479
• Bump version to 2.9.9 by @saikonen in #1480

Full Changelog: 2.9.8...2.9.9

2.9.8

Improvements

Fixes bug with Argo events parameters

This release fixes an issue with mapping values with spaces from the Argo events payload to flow parameters.

What's Changed

• sanitize / in secret names by @oavdeev in #1470
• chore: upgrade packages in cards plugin by @saikonen in #1473
• fix: Argo events parameters with spaces by @saikonen in #1475
• allow to customize env var name in @secrets by @oavdeev in #1474
• Bump version to 2.9.8 by @saikonen in #1476

Full Changelog: 2.9.7...2.9.8