Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] AI video prototype #2959

Draft
wants to merge 125 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
125 commits
Select commit Hold shift + click to select a range
3ede48f
server: Add unimplemented AI handler
yondonfu Jan 15, 2024
6f29698
multi: Add /text-to-image for O
yondonfu Jan 17, 2024
e2735cd
core+server: Add /image-to-image to O
yondonfu Jan 17, 2024
8883824
core+server: Add /image-to-video for O
yondonfu Jan 17, 2024
92bfa74
multi: Transcode PNG -> mp4 for image-to-video
yondonfu Jan 21, 2024
be72c37
server: Impl B -> O image-to-video
yondonfu Jan 21, 2024
7cd7913
server: Impl B -> O text-to-image
yondonfu Jan 21, 2024
abe1b5a
server: Set Content-Type header on B /image-to-video
yondonfu Jan 21, 2024
7ec59b3
server: Impl B -> O image-to-image
yondonfu Jan 22, 2024
1822db5
mod: Bump go-tools to v0.3.5
yondonfu Jan 22, 2024
52785b2
server: Upload to OS for all AI endpoints
yondonfu Jan 22, 2024
332ecbd
mod: Bump ai-worker + go-tools
yondonfu Jan 23, 2024
14deb6a
cmd: Add -aiModels to load models
yondonfu Jan 23, 2024
289cb49
server: Log oapi validation error
yondonfu Jan 23, 2024
48f560c
temp disable CI tests
yondonfu Jan 24, 2024
24c1623
ci+docker: Use go1.21.5
yondonfu Jan 24, 2024
d60b801
docker: Install zlib
yondonfu Jan 24, 2024
b49d503
temp disable CI arm64 builds
yondonfu Jan 24, 2024
27c8da1
cmd: Use livepeer/ai-runner image ID
yondonfu Jan 24, 2024
0adf0b6
cmd: Create models dir if DNE
yondonfu Jan 24, 2024
d894ee8
cmd: Add -aiModelsDir flag
yondonfu Jan 25, 2024
d1af6e2
server: Log HTTP errors for AI endpoints
yondonfu Jan 25, 2024
1759f35
server: Add AI process logs for O
yondonfu Jan 25, 2024
e089c10
mod+cmd: Bump ai-worker
yondonfu Jan 25, 2024
857ee5f
server: Check JSON200 from O
yondonfu Jan 26, 2024
3fcc300
server: Temp disable oapi validation
yondonfu Jan 26, 2024
d6d4261
mod: Bump ai-worker
yondonfu Jan 26, 2024
19da59d
mod+server: Prefer: respond-async for image-to-video
yondonfu Jan 30, 2024
7d765d6
server: Fix error check in ImageToVideo
yondonfu Jan 31, 2024
9985d5f
server: Fix JSON for ImageToVideoResult.Error
yondonfu Feb 1, 2024
2ecaaab
mod: Bump ai-worker
yondonfu Feb 1, 2024
dafefd1
server: Add error struct for AI endpoints
yondonfu Feb 1, 2024
c35735c
server: Add backoff for AI requests
yondonfu Feb 1, 2024
9521fcb
mod: Bump ai-worker
yondonfu Feb 5, 2024
9f5792a
server: Read errors from O for AI requests
yondonfu Feb 5, 2024
924c712
server: Use multipart writer helpers for AI reqs
yondonfu Feb 5, 2024
209ec48
mod+server: Add seed in AI responses
yondonfu Feb 8, 2024
04c751c
multi: Support specifying external containers
yondonfu Feb 13, 2024
9f47d4e
multi: Pass bearer token to runners
yondonfu Feb 13, 2024
0b31700
mod: Bump livepeer/ai-worker
yondonfu Feb 14, 2024
c5427d7
mod: Bump livepeer/ai-worker
yondonfu Feb 14, 2024
5e73dcb
multi: Model cap constraints + multi-O for text-to-image
yondonfu Feb 16, 2024
c22f398
multi: Multi-O for image-to-image
yondonfu Feb 19, 2024
579a31d
multi: Multi-O for image-to-video
yondonfu Feb 19, 2024
f82c014
mod: Bump livepeer/ai-worker
yondonfu Feb 20, 2024
a61818b
core: Fix resolution for image-to-video
yondonfu Feb 20, 2024
3d77eed
server: Fix service unavailable error image-to-video
yondonfu Feb 20, 2024
f2bc443
core: Use software transcoder for image-to-video
yondonfu Feb 25, 2024
8cc7430
server: Return 503 if no Os have cap
yondonfu Feb 27, 2024
f583d93
server: Add AISessionManager
yondonfu Mar 11, 2024
e122757
cmd: Use abs path for default model dir
yondonfu Mar 13, 2024
a2b194e
server: Remove check for deprecated seg data profiles
yondonfu Mar 13, 2024
13f5637
server: Payments for text-to-image
yondonfu Mar 13, 2024
a77f0a0
server: Payments + session manager for all AI caps
yondonfu Mar 14, 2024
f8ef935
multi: Support price per AI cap
yondonfu Mar 14, 2024
0e4d35a
core: Default transcode max price
yondonfu Mar 14, 2024
f698708
server: Use frame count for out pixels
yondonfu Mar 15, 2024
bd12e99
server: Re-use session logic for AI
yondonfu Mar 20, 2024
a818a61
docs(ai): add AI subnet orch setup guide
rickstaa Mar 25, 2024
b3fdf31
docs(ai): add ai subnet broadcaster instructions
rickstaa Mar 25, 2024
f13b8b0
docs(ai): enhance AI Subnet documentation with binary installation guide
rickstaa Mar 25, 2024
43f4166
docs(ai): improve ai subnet documentation
rickstaa Mar 27, 2024
be3afe8
docs(ai): fix broken huggingface documentation link
rickstaa Mar 27, 2024
c7130b4
docs(ai): fix broken 'dl_checkpoint' command
rickstaa Mar 27, 2024
732bf36
docs(ai): enhance clarity and accuracy of AI subnet docs
rickstaa Mar 28, 2024
6895dca
docs(ai): improve model configuration documentation
rickstaa Mar 28, 2024
a8346e4
docs(ai): add instructions for on-chain configuration of AI Subnet Or…
rickstaa Mar 28, 2024
f5bb8c2
docs(ai): improve AI on-chain instructions
rickstaa Mar 28, 2024
16edc6e
docs(ai): improve ai docs syntax
rickstaa Mar 28, 2024
3a14fb1
Merge pull request #2987 from livepeer/improve_ai_docs_text
rickstaa Mar 28, 2024
07ec545
docs(ai): add promtail metrics sending docs
rickstaa Mar 28, 2024
bced6be
docs(ai): update volume mount config for promtail
rickstaa Mar 28, 2024
5e26f41
docs(ai): name docker containers
rickstaa Mar 29, 2024
77b9a48
docs(ai): add promtail metrics sending docs
rickstaa Apr 5, 2024
ab40d04
docs(ai): improve models config descriptiona and add `ticketEV` param
rickstaa Apr 5, 2024
aa706dc
docs(ai): improve cli description and remove redeemer method
rickstaa Apr 5, 2024
0e6be96
docs(ai): add command outputs
rickstaa Apr 5, 2024
9db7a1b
docs(ai): add AI subnet orch setup guide
rickstaa Apr 8, 2024
0ed61b8
ci(ai): add AI issue templates
rickstaa Apr 13, 2024
f76726d
ci(ai): add AI pull request labeler
rickstaa Apr 13, 2024
065edbc
ci: change issue template order
rickstaa Apr 15, 2024
1019d42
ci(ai): add PR labeler config file
rickstaa Apr 15, 2024
af6186f
ci(ai): fix incorrect labels
rickstaa Apr 15, 2024
51406be
ci: rename labeler and remove trailing whitespace
rickstaa Apr 16, 2024
9502ea0
feat(ai): add pipelines optimization flags (#3013)
rickstaa Apr 16, 2024
cea4e94
docs(ai): add optimization flags to docs (#3014)
rickstaa Apr 16, 2024
bc629b7
ci(ai): temporary change build action branch to ai-video
rickstaa Apr 16, 2024
6aa0b00
ci(ai): temporary change docker action branch to ai-video
rickstaa Apr 16, 2024
cecd3a5
ci(ai): fix pull request config warning (#3018)
rickstaa Apr 18, 2024
2a782ed
fix: flush writer when encoding AI results (fix invalid PNG) (#3020)
stronk-dev Apr 18, 2024
e1db239
ci(ai): add myself as branch CODE OWNER
rickstaa Apr 18, 2024
1643a1e
ci(ai): run labeler also on 'pull_request_target'
rickstaa Apr 18, 2024
5bb92fa
ci(ai): cleanup labeler actions
rickstaa Apr 18, 2024
23fdfcb
ci(ai): auto assign AI issues and feature requests
rickstaa Apr 18, 2024
e9258f3
feat(ai): enable AI orchestrator discovery (#3004)
rickstaa Apr 18, 2024
bfccbc4
refactor(ai): add extra devtool input arguments (#3026)
rickstaa Apr 20, 2024
865314d
chore: improve devtool documentation and add scripts
rickstaa Apr 20, 2024
ea82cde
refactor: log advertised capabilities and price on startup (#3031)
eliteprox Apr 24, 2024
93caa3b
feat(ai): enforce 'aiModels' flag requirement (#3032)
rickstaa Apr 24, 2024
bbda633
fix(ai): improve AI selection algorithm (#3030)
rickstaa May 2, 2024
72dced7
refactor(ai): improve orch select retry ctx logic (#3039)
rickstaa May 3, 2024
6fc1afd
refactor(ai): improve orch retry timeout msg
rickstaa May 3, 2024
a0448d4
fix(ai): prevent insufficient capacity payments (#3035)
rickstaa May 3, 2024
fb9764b
ci(ai): add temporary ai-video latest binary url upload
rickstaa May 4, 2024
40a40a5
chore(ai): remove temporary AI subnet docs
rickstaa May 4, 2024
ebd5045
fix(ai): fix infinite loop when no Os are found (#3042)
rickstaa May 4, 2024
dfb4320
feat(ai): Enhance orchestrator selection by incorporating latency (#3…
rickstaa May 6, 2024
45cf167
chore(ai): update 'ai-worker' dependency
rickstaa May 13, 2024
180041d
feat: add '-gateway' and deprecate '-broadcaster' (#3048)
eliteprox May 13, 2024
c18d325
feat: remove -pricePerUnit requirement for -aiWorker flag (#3047)
eliteprox May 13, 2024
ee787ea
perf(ai): update ai-worker to enable DEEPCACHE optimization
rickstaa May 14, 2024
d5f5e1c
chore: fix Mockgen dependency error
rickstaa May 15, 2024
62acd4a
ci(ai): ensure docker builder is build and pushed
rickstaa May 15, 2024
816920a
feat(ai): enable NSFW safety filter (#3054)
rickstaa May 15, 2024
fc51f04
chore(ai): update ai-worker version
rickstaa May 17, 2024
f9aa0c1
ci(ai): ensure livepeer builder builds on AI version tags
rickstaa May 17, 2024
9584c3c
fix: apply runner nil error fix (#3058)
rickstaa May 17, 2024
add7368
refactor(census): rename Broadcaster metrics to Gateway (#3055)
rickstaa May 16, 2024
35db3f2
refactor: add -pricePerGateway and deprecate -pricePerBroadcaster (#3…
rickstaa May 20, 2024
55185db
ci: Protect Docker 'stable' tag
rickstaa May 20, 2024
2b8478d
ci: fix syntax error in Docker action tags
rickstaa May 20, 2024
03b64fb
fix(ai): fix runtime error in aiWorker when pricePerUnit is unset (#3…
eliteprox May 20, 2024
b7a9fb7
fix(ai): fix cli prices nil error (#3063)
rickstaa May 20, 2024
6843eda
feat: add -aiRunnerImage flag to pin docker image ver (#3064)
eliteprox May 21, 2024
4382fbe
chore(ai): update ai-worker dependency
rickstaa May 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
@@ -0,0 +1,3 @@
# Default reviewers for the `ai-video` branch.
# TODO: Change if merged into `master` branch.
* @rickstaa
File renamed without changes.
78 changes: 78 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report_ai_video.yml
@@ -0,0 +1,78 @@
name: AI Bug report
description: Create a report to help us improve.
labels:
- "type: bug"
- "AI"
assignees:
- rickstaa
body:
- type: markdown
attributes:
value: |
## Bug report
Please fill out the following information to help us understand your issue.

> [!IMPORTANT]
> This repository is only related to the core bugs with the AI branch of the go-livepeer software (i.e. `ai-video`). It does not cover bugs related to running AI pipelines and AI models used on the AI subnet. For these issues, please refer to the [AI-worker repository](https://github.com/livepeer/ai-worker/issues/new/choose)
- type: textarea
attributes:
label: Describe the bug
description: A clear and concise description of what the bug is.
validations:
required: true
- type: textarea
attributes:
label: Reproduction steps
description: "How do you trigger this bug? Please walk us through it step by step."
value: |
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
- type: textarea
attributes:
label: Expected behaviour
description: A clear and concise description of what you expected to happen.
- type: dropdown
id: severity
attributes:
label: Severity
description: "How severe is this bug?"
options:
- Minor
- Major
- Critical
- type: textarea
attributes:
label: Screenshots / Live demo link
description: If applicable, add screenshots to help explain your problem.
placeholder: Paste the image link as markdown image
- type: dropdown
id: os
attributes:
label: OS
description: "What operating system are you using?"
options:
- Windows
- Mac
- Linux
- type: dropdown
id: running_on
attributes:
label: Running on
description: "Where are you running the application?"
options:
- Local
- Docker
- type: input
attributes:
label: AI go-livepeer version
description: "What version of the AI-worker are you using?"
- type: input
attributes:
label: AI go-livepeer commit hash
description: "Could you please provide the commit hash of the `ai-video` branch that you are currently using?"
- type: textarea
attributes:
label: Additional context
description: Add any other context about the problem here.
8 changes: 8 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
@@ -0,0 +1,8 @@
blank_issues_enabled: true
contact_links:
- name: Go-livepeer Question
url: https://github.com/livepeer/go-livepeer/discussions
about: Please ask and answer questions related to the go-livepeer software here.
- name: Livepeer Question
url: https://discord.gg/livepeer
about: "Have a general Livepeer question? Join us in the Livepeer Discord server. We're here to help!"
47 changes: 47 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request_ai.yml
@@ -0,0 +1,47 @@
name: AI Feature request
description: Suggest an idea for this project.
labels:
- "type: feature"
- "AI"
assignees:
- rickstaa
body:
- type: markdown
attributes:
value: |
## Feature Request
Please fill out the following information to help us understand your request.

> [!IMPORTANT]
> This repository is only related to feature requests related to the the AI branch of the go-livepeer software (i.e. `ai-video`). It does not cover feature requests related to the addition of new AI pipelines and AI models used on the AI subnet. For these issues, please refer to the [AI-worker repository](https://github.com/livepeer/ai-worker/issues/new/choose).
- type: textarea
attributes:
label: Is your feature request related to a problem? Please describe.
description:
A clear and concise description of what the problem is. Ex. I'm always
frustrated when [...]
validations:
required: true
- type: textarea
attributes:
label: Describe the solution you'd like
description: A clear and concise description of what you want to happen.
- type: textarea
attributes:
label: Describe alternatives you've considered
description:
A clear and concise description of any alternative solutions or features
you've considered.
- type: textarea
attributes:
label: Use Case
description: "Please describe why you want this feature to be added. This will help us prioritize your request."
- type: textarea
attributes:
label: Expected Outcome
description: "What do you expect to happen once this feature is implemented?"
- type: textarea
attributes:
label: Additional context
description:
Add any other context or screenshots about the feature request here.
2 changes: 2 additions & 0 deletions .github/labeler.yml
@@ -0,0 +1,2 @@
AI:
- base-branch: "ai-video"
29 changes: 15 additions & 14 deletions .github/workflows/build.yaml
Expand Up @@ -4,7 +4,8 @@ on:
pull_request:
push:
branches:
- master
# - master
- ai-video
tags:
- "v*"

Expand All @@ -30,23 +31,23 @@ jobs:
container: ubuntu:20.04
type: cpu

- GOOS: linux
GOARCH: arm64
runner: ubuntu-20.04
container: ubuntu:20.04
type: cpu
# - GOOS: linux
# GOARCH: arm64
# runner: ubuntu-20.04
# container: ubuntu:20.04
# type: cpu

- GOOS: linux
GOARCH: amd64
runner: ubuntu-20.04
container: livepeerci/cuda:12.0.0-cudnn8-devel-ubuntu20.04
type: gpu

- GOOS: linux
GOARCH: arm64
runner: ubuntu-20.04
container: livepeerci/cuda:12.0.0-cudnn8-devel-ubuntu20.04
type: gpu
# - GOOS: linux
# GOARCH: arm64
# runner: ubuntu-20.04
# container: livepeerci/cuda:12.0.0-cudnn8-devel-ubuntu20.04
# type: gpu

- GOOS: windows
GOARCH: amd64
Expand Down Expand Up @@ -74,7 +75,7 @@ jobs:
id: go
uses: actions/setup-go@v5
with:
go-version: 1.20.4
go-version: 1.21.5
cache: true
cache-dependency-path: go.sum

Expand Down Expand Up @@ -176,7 +177,7 @@ jobs:
id: go
uses: actions/setup-go@v5
with:
go-version: 1.20.4
go-version: 1.21.5
cache: true
cache-dependency-path: go.sum

Expand Down Expand Up @@ -216,7 +217,7 @@ jobs:
id: match-tag
with:
text: ${{ github.ref_name }}
regex: '^(master|main|v[0-9]+\.\d+\.\d+)$'
regex: '^(master|main|ai-video|v[0-9]+\.\d+\.\d+)$'

- name: Codesign and notarize binaries
if: steps.match-tag.outputs.match != '' && matrix.target.GOOS == 'darwin'
Expand Down
11 changes: 6 additions & 5 deletions .github/workflows/docker.yaml
Expand Up @@ -4,7 +4,8 @@ on:
pull_request:
push:
branches:
- master
# - master
- ai-video
tags:
- "v*"

Expand Down Expand Up @@ -81,15 +82,15 @@ jobs:
type=semver,pattern={{major}}.{{minor}},prefix=v
type=raw,value=latest,enable={{is_default_branch}}
type=raw,value=${{ github.event.pull_request.head.ref }}
type=raw,value=stable,enable=${{ startsWith(github.event.ref, 'refs/tags/v') }}
type=raw,value=stable,enable=${{ startsWith(github.event.ref, 'refs/tags/v') && github.event.base_ref == 'refs/heads/main' }}

- name: Build and push livepeer docker image
uses: docker/build-push-action@v5
with:
build-args: |
BUILD_TAGS=${{ steps.build-tag.outputs.build-tags }}
context: .
platforms: linux/amd64, linux/arm64
platforms: linux/amd64
push: true
tags: ${{ steps.meta.outputs.tags }}
file: "docker/Dockerfile"
Expand Down Expand Up @@ -126,7 +127,7 @@ jobs:
id: match-tag
with:
text: ${{ github.ref_name }}
regex: '^(main|master|v[0-9]+\.\d+\.\d+)$'
regex: '^(main|master|ai-video|v[0-9]+\.\d+\.\d+|v[0-9]+\.\d+\.\d+-ai-video-\d+)'

- name: Get build tags
id: build-tag
Expand Down Expand Up @@ -177,7 +178,7 @@ jobs:
build-args: |
BUILD_TAGS=${{ steps.build-tag.outputs.build-tags }}
context: .
platforms: linux/amd64, linux/arm64
platforms: linux/amd64
push: true
tags: ${{ steps.meta-builder.outputs.tags }}
file: "docker/Dockerfile"
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/issue-labeler.yml
@@ -1,11 +1,11 @@
name: Label issues
on:
issues:
types:
- reopened
- opened
types: [opened, reopened]

jobs:
label_issues:
if: ${{ github.event_name == 'issues' }}
runs-on: ubuntu-latest
permissions:
issues: write
Expand Down
14 changes: 14 additions & 0 deletions .github/workflows/pr-labeler.yml
@@ -0,0 +1,14 @@
name: Label issues
on:
pull_request_target:
types: [opened, reopened]

jobs:
label_pull_requests:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
- uses: actions/labeler@v5
8 changes: 4 additions & 4 deletions .github/workflows/test.yaml
@@ -1,9 +1,9 @@
name: Trigger test suite

on:
pull_request:
branches:
- master
# pull_request:
# branches:
# - master
push:
branches:
- master
Expand Down Expand Up @@ -43,7 +43,7 @@ jobs:
id: go
uses: actions/setup-go@v5
with:
go-version: 1.20.4
go-version: 1.21.5
cache: true
cache-dependency-path: go.sum

Expand Down
6 changes: 4 additions & 2 deletions Makefile
@@ -1,6 +1,8 @@
SHELL=/bin/bash
GO_BUILD_DIR?="./"

MOCKGEN=go run github.com/golang/mock/mockgen

all: net/lp_rpc.pb.go net/redeemer.pb.go net/redeemer_mock.pb.go core/test_segment.go livepeer livepeer_cli livepeer_router livepeer_bench

net/lp_rpc.pb.go: net/lp_rpc.proto
Expand All @@ -10,8 +12,8 @@ net/redeemer.pb.go: net/redeemer.proto
protoc -I=. --go_out=. --go-grpc_out=. $^

net/redeemer_mock.pb.go net/redeemer_grpc_mock.pb.go: net/redeemer.pb.go net/redeemer_grpc.pb.go
@mockgen -source net/redeemer.pb.go -destination net/redeemer_mock.pb.go -package net
@mockgen -source net/redeemer_grpc.pb.go -destination net/redeemer_grpc_mock.pb.go -package net
@$(MOCKGEN) -source net/redeemer.pb.go -destination net/redeemer_mock.pb.go -package net
@$(MOCKGEN) -source net/redeemer_grpc.pb.go -destination net/redeemer_grpc_mock.pb.go -package net

core/test_segment.go:
core/test_segment.sh core/test_segment.go
Expand Down