[WIP] AI video prototype #2959
Draft
[WIP] AI video prototype #2959
+3,625
−559
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yondonfu
force-pushed
the
ai-video
branch
5 times, most recently
from
feb3f3e
to
563d199
Compare
|
d32b1b5 temporarily disables Linux arm64 builds because they fail due to an error related to not being able to find zlib during ffmpeg compilation. This error doesn't occur for amd64 builds so I suspect the issue is related to the amd64 -> arm64 cross-compilation process. I noticed that we compile an arm64 specific version of x264 before compiling ffmpeg - perhaps we have to do something similar with zlib? zlib is currently required as a dependency as of 133050d#diff-4ae778054809274731b9da0c6a5a869c0bd214e92f954a5c9c39181748c2f175 which enabled the png decoder and image2 muxer which are used to demux + decode a sequence of PNG files so they can be encoded into an mp4 file. Ideally, we would replace the PNG demux/decode component by passing tensors (that represent frames) outputted by a model directly from GPU memory to NVENC using torchaudio.StreamWriter, but torchaudio.StreamWriter doesn't support RGB -> YUV conversion on the GPU yet - it can still encode a larger, less-streaming friendly (my understanding is yuv420p is preferred for streaming) RGB output, but I didn't jump to implement this yet due to current limitations. Until this replacement happens, zlib would be a required dependency to support the temporary PNG demux + decode component. |
|
@yondonfu Weird - on release go-livepeer right now zlib is dynamically linked. I'll have a look. |
This commit fixed the labels that were specified in the Issue Templates to the one found in the repository.
* feat(ai): add pipelines optimization flags This commit adds a new `OptimizationFlags` field to the `aiModels` config so that users can forward optimization environment variables to the [ai-worker](git@github.com:livepeer/ai-worker.git) for more information see livepeer/ai-worker#61. * chore: update ai-worker to latest commit This commit ensures that the https://github.com/livepeer/ai-worker dependency is on the latest commit that includes the new optimization flags feature (see livepeer/ai-worker#61). * refactor: improve OptFlags logging This commit ensures that the `optimzation flag not supported` warning is shown for each model that is not loaded warm.
This commit adds a new section explaining the new `optimization_flags` that were enabled #3013.
This commit temporary changes the push branch of the `build.yml` to the `ai-video` branch since the `ai-video` branch has conflicts with the `master` branch preventing the containers to be build.
This commit temporary changes the push branch of the `docker.yml` to the `ai-video` branch since the `ai-video` branch has conflicts with the `master` branch preventing the containers to be build.
This commit gets rid of the Pull request labeler configuration file warning.
This commit flushes the data in the image writer to ensure that all data gets written to the PNG.
This commit ensures that the labeler action also runs on a 'pull_request_target' to ensure pull requests from forks are correctly labeled.
This commit ensures that all AI related issues and feature requests are assigned to the AI team.
* feat(ai): enable AI orchestrator discovery This commit incorporates the AIServiceRegistry contract address, superseding the conventional ServiceRegistry contract address. This strategic alteration streamlines the discovery process of AI Orchestrators within the AI Subnet, thereby bolstering network accessibility and interaction. While this approach serves as a swift workaround to enable the feature without extensive code modification, it's important to note that it may disrupt the existing transcoding discovery mechanism. We have to fix this if we want to merge the two networks in the future. * docs(ai): improve discovery documentation This commit ensures that people are aware that they have to interact with the `AIServiceRegistry` using their main Orch wallet. * fix: fix 'AIServiceRegistry' devnet and testnet issue This commit ensure that the hardcoded `AIServiceRegistry` contract doesn't break the go-livepeer binary on local devnets or testnets.
This commit adds extra devtool input arguments allowing developers to spin up multiple Os on the ETH devnet.
This commit improves the devtool documentation and adds a helpful script if developers want to create multiple Os at the same time.
This commit logs the advertised capabilities and price on startup if users have their logging verbosity level set to 6 or higher.
This commit ensures that an error is thrown when users don't specify the 'aiModels' flag but have the 'aiWorker' flag set.
* fix(ai): improve selection algorithm This commit modifies the selection algorithm to continue retrying for a duration of one second instead of stopping after four attempts. This change addresses issues encountered with the current algorithm's performance in environments with 15 nodes on the network, ensuring more robust and reliable operation until further optimizations can be implemented. * refactor(ai): enhance selection algorithm retry logic This commit replaces the time-based for-loop in the selection algorithm's retry logic with a more context-aware approach.
This commit refines context handling in the orchestrator selection loop for idiomatic Go and enhanced propagation of parent cancellations.
This commit improves the orchestrator selection retry ctx timeout msg.
* fix(ai): handle insufficient capacity payments This commit enhances the Orchestrator's capacity handling by returning an error prior to processing payments when capacity is insufficient. This prevents that the Gateway overpays for requests. * chore(ai): update ai-worker dependency This commit updates the ai-worker dependency to the latest version.
This commit ensures that the `upload_build.sh` script uploads the latest binary that is deployed to the `ai-video` branch under one url. This is done to simplify binary installation.
This commit removes the tempoary AI subnet docs now that the final docs have been deployed on https://docs.livepeer.ai/ai/introduction.
This commit prevents the orchestrator selection go routine from staying in a infinite loop when no Orchestrators can be found.
) This commit introduces latency consideration into the orchestrator selection process, addressing two key issues. Firstly, it resolves a minor bug where the algorithm consistently selected known orchestrators due to a condition that never evaluated to true (see [this condition](https://github.com/livepeer/go-livepeer/blob/1239b4e56133003fe6a98a863cce6bdd6b5f2532/server/selection.go#L110)). Secondly, this change ensures that, once all orchestrators have been evaluated, the one with the fastest response time for a specific job is chosen. While the current method for calculating latency is somewhat basic, it sets the foundation for more sophisticated enhancements in the future. Co-authored-by: Brad P <0xb79orch@gmail.com>
This commit updates the 'ai-worker' dependency to the latest commit.
This commit adds the `gateway` flag and deprecates the `broadcaster` flag per core team decision (details: https://discord.com/channels/423160867534929930/1051963444598943784/1210356864643109004).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Opening this draft PR to kick off CI build process for the WIP
ai-videobranch. This PR should not be merged as-is and the code on this branch will likely be refactored + cleaned up separately later on.