-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Maximum number of artifacts exceeded. How to aggregate artifacts from >100 ParallelFor iterations #7513
Comments
hi @bronevet-abc , I'm not sure but this may help but you can try this you can create a custom container image for your ParallelFor loop execution. This image could handle artifact management internally, bypassing the 100-artifact limit. |
Interesting! I'm not sure exactly the mechanics of this. Are you suggesting a custom container for the task running within the ParallelFor or for the task receiving the aggregated data objects? Can you point to any examples for this or the relevant APIs? Thank you! |
/close this is the wrong repository. Use kubeflow/pipelines for pipelines stuff. |
@juliusvonkohout: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
/kind bug
I am running a ParallelFor loop with > 100 iterations and aggregating the resulting artifacts into a dsl.Collection but I'm getting the following error:
com.google.cloud.ai.platform.common.errors.AiPlatformException: code=INVALID_ARGUMENT, message=Maximum number of artifacts exceeded. Maximum allowed: 100, requested: 270, cause=null
Is there a way to raise this maximum so I can run larger ParallelFor iterations?
The text was updated successfully, but these errors were encountered: