Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SCD Type 2-style data sources #4162

Open
tokoko opened this issue Apr 30, 2024 · 0 comments
Open

SCD Type 2-style data sources #4162

tokoko opened this issue Apr 30, 2024 · 0 comments
Labels
kind/feature New feature or request

Comments

@tokoko
Copy link
Collaborator

tokoko commented Apr 30, 2024

Is your feature request related to a problem? Please describe.
Feast only supports specifying a single event_timestamp value for each row in a data source. While feast also supports expiring these values, it's managed with a single feature-view level ttl value. This is useful for a lot of use cases when feature computations might happen sporadically, but doesn't really make sense when feature computations are done as part of a periodic (daily, monthly) ETL processes or when features are already in-place in data warehouse as type 2 scd tables.

Describe the solution you'd like
Add an option to specify a column for row expiration in data sources (something like event_expire_timestamp). user will be able to specify either this column or a ttl value. offline store engines will have to handle both scenarios. in case event_expire_timestamp is provided, I think offline stores should assume that there are no overlaps, as in most scenarios users will have much easier time making sure there are no overlaps with scd type 2 tables themselves. This assumption will simplify offline store point-in-time logic quite a bit as offline store engine won't have to do additional window operations to get rid of duplicates.

@tokoko tokoko added the kind/feature New feature or request label Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant