[Data] Raise warning if user tries to pass large argument to UDF #45288
Labels
data
Ray Data-related issues
enhancement
Request for new feature and/or capability
P2
Important issue, but not time-critical
Description
If you pass a large arg if
fn_args
orfn_kwargs
, Ray stores a copy of that object in heap memory for each task. So, if Ray Data launches many tasks, this can cause heap OOMs. See Anti-pattern: Passing the same large argument by value repeatedly harms performanceTo avoid users shooting themselves in the foot with this, we should raise a warning.
Use case
UX
The text was updated successfully, but these errors were encountered: