Using requirements file in VirtualEnvPythonOperation appears to be broken

[timc]

Apache Airflow version

2.7.3

What happened

When creating a virtual env task and passing in a requirements file like this:

@task.virtualenv( use_dill=True, system_site_packages=False, requirements='requirements.txt')

The result is that the contents of the requirements file using to populate the venv is

requirements.txt

Which is wrong. And you get this:

[2023-12-05, 12:33:06 UTC] {{process_utils.py:181}} INFO - Executing cmd: python3 /usr/local//.local/lib/python3.10/site-packages/virtualenv /tmp/venv3cdlqjlq
[2023-12-05, 12:33:06 UTC] {{process_utils.py:185}} INFO - Output:
[2023-12-05, 12:33:07 UTC] {{process_utils.py:189}} INFO - created virtual environment CPython3.10.9.final.0-64 in 397ms
[2023-12-05, 12:33:07 UTC] {{process_utils.py:189}} INFO - creator CPython3Posix(dest=/tmp/venv3cdlqjlq, clear=False, no_vcs_ignore=False, global=False)
[2023-12-05, 12:33:07 UTC] {{process_utils.py:189}} INFO - seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/usr/local//.local/share/virtualenv)
[2023-12-05, 12:33:07 UTC] {{process_utils.py:189}} INFO - added seed packages: pip==23.3.1, setuptools==69.0.2, wheel==0.42.0
[2023-12-05, 12:33:07 UTC] {{process_utils.py:189}} INFO - activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2023-12-05, 12:33:07 UTC] {{process_utils.py:181}} INFO - Executing cmd: /tmp/venv3cdlqjlq/bin/pip install -r /tmp/venv3cdlqjlq/requirements.txt
[2023-12-05, 12:33:07 UTC] {{process_utils.py:185}} INFO - Output:
[2023-12-05, 12:33:09 UTC] {{process_utils.py:189}} INFO - ERROR: Could not find a version that satisfies the requirement requirements.txt (from versions: none)
[2023-12-05, 12:33:09 UTC] {{process_utils.py:189}} INFO - HINT: You are attempting to install a package literally named "requirements.txt" (which cannot exist). Consider using the '-r' flag to install the packages listed in requirements.txt
[2023-12-05, 12:33:09 UTC] {{process_utils.py:189}} INFO - ERROR: No matching distribution found for requirements.txt
[2023-12-05, 12:33:09 UTC] {{taskinstance.py:1824}} ERROR - Task failed with exception

The issue appears to be that the requirements parameter is added to a list on construction of the operator so the templating never happens.

What you think should happen instead

The provided requirements file should be used in the pip command to set up the venv.

How to reproduce

Create a dag:

from datetime import datetime
from airflow.decorators import dag, task

 @dag(schedule_interval=None, start_date=datetime(2021, 1, 1), catchup=False, tags=['example']) 
def virtualenv_task():
    @task.virtualenv(
        use_dill=True,
        system_site_packages=False,
        requirements='requirements.txt',
    )
    def extract():
        import pandas 
        x = pandas.DataFrame()

    extract()

dag = virtualenv_task()

And a requirements.txt file

pandas

Run AirFlow

Operating System

Ubuntu 23.04

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.2.0
apache-airflow-providers-celery==3.2.1
apache-airflow-providers-common-sql==1.5.2
apache-airflow-providers-ftp==3.4.2
apache-airflow-providers-http==4.4.2
apache-airflow-providers-imap==3.2.2
apache-airflow-providers-postgres==5.5.1
apache-airflow-providers-sqlite==3.4.2

Deployment

Docker-Compose

Deployment details

No response

Anything else

Everytime.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[timc]

A workaround is to define the dag as such:

from some_util_file.py import load_reqs
@task.virtualenv(use_dill=True, system_site_packages=False, requirements=load_reqs())

where load_reqs is as follows:

from pathlib import Path
def load_reqs(file: str = 'requirements.txt'):
    with open(Path(__file__).parent / file) as f:
        return "\n".join(f.readlines())

So the requirements content is effectively copied to another file and use in the venv.

[potiuk]

No:

    :param requirements: Either a list of requirement strings, or a (templated)
        "requirements file" as specified by pip.

This follows the same behaviour as any other templated fields. If you use requirements.txt in your operator, you need to have a "requirements.txt" in the sams package/folder as your operator and then requirements.txt will be replaced with jinja-processed content of the file.

See https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#concepts-jinja-templating and look for template_ext.

Likely you did not have "requirements.txt" placed in either DAG folder or template_searchpath if you specify one:

By default, paths provided in this way should be provided relative to the DAG’s folder (as this is the default Jinja template search path), but additional paths can be added by setting the template_searchpath arg on the DAG.

[timc]

Hi, thanks, here is the zip file.

This is the logging I get:

*** Found local files:
***   * /usr/local/airflow/logs/dag_id=venv-test/run_id=manual__2023-12-06T23:36:07.504592+00:00/task_id=RunInVenv/attempt=1.log
[2023-12-06, 23:36:13 UTC] {{taskinstance.py:1103}} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: venv-test.RunInVenv manual__2023-12-06T23:36:07.504592+00:00 [queued]>
[2023-12-06, 23:36:13 UTC] {{taskinstance.py:1103}} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: venv-test.RunInVenv manual__2023-12-06T23:36:07.504592+00:00 [queued]>
[2023-12-06, 23:36:13 UTC] {{taskinstance.py:1308}} INFO - Starting attempt 1 of 1
[2023-12-06, 23:36:13 UTC] {{taskinstance.py:1327}} INFO - Executing <Task(_PythonVirtualenvDecoratedOperator): RunInVenv> on 2023-12-06 23:36:07.504592+00:00
[2023-12-06, 23:36:13 UTC] {{standard_task_runner.py:57}} INFO - Started process 227 to run task
[2023-12-06, 23:36:13 UTC] {{standard_task_runner.py:84}} INFO - Running: ['***', 'tasks', 'run', 'venv-test', 'RunInVenv', 'manual__2023-12-06T23:36:07.504592+00:00', '--job-id', '399', '--raw', '--subdir', 'DAGS_FOLDER/venv_dag.py', '--cfg-path', '/tmp/tmp71_s1ko0']
[2023-12-06, 23:36:13 UTC] {{standard_task_runner.py:85}} INFO - Job 399: Subtask RunInVenv
[2023-12-06, 23:36:13 UTC] {{logging_mixin.py:150}} WARNING - /usr/local/***/.local/lib/python3.10/site-packages/***/settings.py:195 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
[2023-12-06, 23:36:13 UTC] {{task_command.py:410}} INFO - Running <TaskInstance: venv-test.RunInVenv manual__2023-12-06T23:36:07.504592+00:00 [running]> on host 0d2ea8a03651
[2023-12-06, 23:36:13 UTC] {{taskinstance.py:1545}} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='venv-test' AIRFLOW_CTX_TASK_ID='RunInVenv' AIRFLOW_CTX_EXECUTION_DATE='2023-12-06T23:36:07.504592+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2023-12-06T23:36:07.504592+00:00'
[2023-12-06, 23:36:13 UTC] {{process_utils.py:181}} INFO - Executing cmd: python3 /usr/local/***/.local/lib/python3.10/site-packages/virtualenv /tmp/venvtk7yhxgj
[2023-12-06, 23:36:13 UTC] {{process_utils.py:185}} INFO - Output:
[2023-12-06, 23:36:14 UTC] {{process_utils.py:189}} INFO - created virtual environment CPython3.10.9.final.0-64 in 222ms
[2023-12-06, 23:36:14 UTC] {{process_utils.py:189}} INFO -   creator CPython3Posix(dest=/tmp/venvtk7yhxgj, clear=False, no_vcs_ignore=False, global=False)
[2023-12-06, 23:36:14 UTC] {{process_utils.py:189}} INFO -   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/usr/local/***/.local/share/virtualenv)
[2023-12-06, 23:36:14 UTC] {{process_utils.py:189}} INFO -     added seed packages: pip==23.3.1, setuptools==69.0.2, wheel==0.42.0
[2023-12-06, 23:36:14 UTC] {{process_utils.py:189}} INFO -   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2023-12-06, 23:36:14 UTC] {{process_utils.py:181}} INFO - Executing cmd: /tmp/venvtk7yhxgj/bin/pip install -r /tmp/venvtk7yhxgj/requirements.txt
[2023-12-06, 23:36:14 UTC] {{process_utils.py:185}} INFO - Output:
[2023-12-06, 23:36:15 UTC] {{process_utils.py:189}} INFO - ERROR: Could not find a version that satisfies the requirement requirements.txt (from versions: none)
[2023-12-06, 23:36:15 UTC] {{process_utils.py:189}} INFO - HINT: You are attempting to install a package literally named "requirements.txt" (which cannot exist). Consider using the '-r' flag to install the packages listed in requirements.txt
[2023-12-06, 23:36:15 UTC] {{process_utils.py:189}} INFO - ERROR: No matching distribution found for requirements.txt
[2023-12-06, 23:36:15 UTC] {{taskinstance.py:1824}} ERROR - Task failed with exception

AirFlowVenv.zip

[potiuk]

Confirmed :) .

Thanks for this. Created back an issue from the discussion: #36102

Fix coming shortly

[potiuk]

Fix in #36103 - thanks again for issue and reproducible example (I was quite sure it was some mistake in your env, but it helped to see that indeed - it was our mistake).

[timc]

All good, thanks Jarek. Glad to see it fixed. In the meantime, the workaround posted above gets around this issue if anyone else faces the same problem.

[potiuk]

Scheduled for 2.8.0

[potiuk]

Converted it to a discussion in case more is needed. But you should look there.

[manel00]

hello! sorry didnt see the solution, and i am interested in! thanks!

[timc]

The fix should have been released in the latest versions of AirFlow.

If you are using an older version there is a workaround here: #36076 (comment)