Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce lazy_tree (super dictionaries) #1733

Open
wants to merge 44 commits into
base: main
Choose a base branch
from

Conversation

braingram
Copy link
Contributor

@braingram braingram commented Jan 12, 2024

Description

This PR adds:

  • lazy_tree option to AsdfConfig
  • lazy_tree argument to asdf.open (defaults to AsdfConfig.lazy_tree)
  • Converter.lazy attribute used to indicate if a converter supports "lazy" objects
  • asdf.lazy_nodes "lazy" container classes for list, dict, ordered dict

By default the "lazy" option is False.

When lazy_tree is True and an ASDF file is opened the tagged nodes in the tree are not immediately converted to custom objects. Instead, the containers in the tree (dicts, lists, OrderedDicts) are replaced with AsdfNode subclasses that act like these containers and convert tagged values to custom objects when they are accessed (See #1705 for discussion of this feature). During conversion, if asdf encounters a Converter that either defines lazy=False or does not define lazy the remainder of the branch will be converted to non-"lazy" objects and passed to the Converter. If instead the Converter defines lazy=True the "lazy" object (ie a AsdfDictNode for a dict) will be passed to the Converter.

Checklist:

  • pre-commit checks ran successfully
  • tests ran successfully
  • for a public change, a changelog entry was added
  • for a public change, documentation was updated
  • for any new features, unit tests were added

@braingram braingram marked this pull request as ready for review January 17, 2024 16:31
@braingram braingram requested a review from a team as a code owner January 17, 2024 16:31
@braingram braingram added this to the 3.1.0 milestone Jan 17, 2024
asdf/_tests/conftest.py Show resolved Hide resolved
asdf/_tests/core/_converters/test_complex.py Show resolved Hide resolved
asdf/_tests/test_lazy_nodes.py Show resolved Hide resolved
asdf/exceptions.py Show resolved Hide resolved
asdf/lazy_nodes.py Outdated Show resolved Hide resolved
asdf/lazy_nodes.py Outdated Show resolved Hide resolved
asdf/lazy_nodes.py Outdated Show resolved Hide resolved
asdf/lazy_nodes.py Outdated Show resolved Hide resolved
asdf/lazy_nodes.py Show resolved Hide resolved
asdf/lazy_nodes.py Outdated Show resolved Hide resolved
@braingram
Copy link
Contributor Author

converted to draft until #1733 (comment) is addressed

@braingram
Copy link
Contributor Author

The following branch of roman_datamodels adds lazy to the node converters (and makes some minor node changes to account for AsdfDictNode not passing isinstance(..., dict) etc) to allow lazy loading of roman trees:
https://github.com/spacetelescope/roman_datamodels/compare/main...braingram:roman_datamodels:lazy?expand=1

@braingram braingram marked this pull request as ready for review May 14, 2024 16:39
@braingram braingram requested a review from eslavich May 14, 2024 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants