Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow flow parameter schema generation when dependencies are missing #13315

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

desertaxle
Copy link
Member

@desertaxle desertaxle commented May 13, 2024

This PR adds a new method of generating a parameter schema for a function so users and use prefect deploy when not all dependencies are available on the current machine.

Our current method of generating a parameter schema requires us to load the flow function itself and get the signature using the built-in inspect module. This causes issues when not all dependencies are present because loading the flow causes the entire module the flow is located in to be run and can result in Python raising an ImportError.

This PR updates this approach to allow a parameter schema given an entrypoint that points to a function with the new parameter_schema_from_entrypoint function. This new function will generate a inspect.Signature object and docstring dictionary from the source code directly and use it with the existing machinery for generating a parameter schema.

Under the hood, the new functionality will load the source code at the provided entrypoint and parse it using the ast module. A new utility called safe_load_namespace will load each module-scoped import one by one and create a namespace dictionary for downstream use. Custom class definitions will also be included in the namespace. Any import errors will be caught and logged, but will not stop execution. The created namespace is used as context when walking the AST for the given function so that custom typing can be correctly resolved. In the case that typing cannot be resolved, the generated Signature will use no annotation to prevent interrupting execution. This can result in overly broad typing if the annotations rely on external types that aren't installed at deployment time.

The diff is big, but it's mostly tests.

Closes #9512

Example

Given code stored at flow.py:

from prefect import flow
from pandas import DataFrame
from pydantic import BaseModel

class MyModel(BaseModel):
    x: int

@flow
def my_flow(input: MyModel) -> pandas.DataFrame:
    # creates the dataframe in a clever way

The following call will generate the expected parameter schema with or without panadas installed:

parameter_schema_from_entrypoint("flow.py:my_flow")

The dictionary format looks like this:

{
    "title": "Parameters",
    "type": "object",
    "properties": {
        "input": {
            "allOf": [{"$ref": "#/definitions/MyModel"}],
            "position": 0,
            "title": "input",
        }
    },
    "required": ["input"],
    "definitions": {
        "MyModel": {
            "properties": {"x": {"title": "X", "type": "integer"}},
            "required": ["x"],
            "title": "MyModel",
            "type": "object",
        }
    },
}

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • This pull request includes a label categorizing the change e.g. maintenance, fix, feature, enhancement, docs.

For documentation changes:

  • This pull request includes redirect settings in netlify.toml for files that are removed or renamed.

For new functions or classes in the Python SDK:

  • This pull request includes helpful docstrings.
  • If a new Python file was added, this pull request contains a stub page in the Python SDK docs and an entry in mkdocs.yml navigation.

@desertaxle desertaxle added the enhancement An improvement of an existing feature label May 13, 2024
Copy link

netlify bot commented May 13, 2024

Deploy Preview for prefect-docs-preview ready!

Name Link
🔨 Latest commit 4482c51
🔍 Latest deploy log https://app.netlify.com/sites/prefect-docs-preview/deploys/664276a230c16c000827893c
😎 Deploy Preview https://deploy-preview-13315--prefect-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@desertaxle desertaxle force-pushed the fix/flow-flow-with-missing-dependencies branch from a085d62 to d34425f Compare May 13, 2024 17:34
@desertaxle desertaxle marked this pull request as ready for review May 13, 2024 21:28
@desertaxle desertaxle requested a review from a team as a code owner May 13, 2024 21:28
@desertaxle desertaxle force-pushed the fix/flow-flow-with-missing-dependencies branch 2 times, most recently from df1f676 to 52f574a Compare May 17, 2024 17:00
@desertaxle desertaxle force-pushed the fix/flow-flow-with-missing-dependencies branch from d5c70ee to b51640f Compare May 29, 2024 01:28
@desertaxle
Copy link
Member Author

@serinamarie Added the docstrings that we identified as missing during our sync review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

prefect deploy command fails if not all dependencies are present at runtime
1 participant