Get IPFS CID back from uploader and invoke RECORDING_END trigger

[cyberj0g]

An ugly, but fully functional, hack to get back IPFS CID from catalyst-uploader and fire RECORDING_END trigger through log handling functions. Hope it's a good starting point to spec out a proper implementation of handling uploader output. Worst case it can be merged, if we need IPFS output tomorrow.

Catalyst issue: livepeer/catalyst#154

Important: MP4 "streaming" is not currently supported by Mist, so this is tested with FLV output.

Testing:

1. Add autopush to mist.conf, using Pinata JWT auth token

  "autopushes": [
    [
      "test",
"ipfs://:jwt_token@pinata.cloud/out.flv"
    ]
  ],

2. Add RECORDING_END trigger (https://webhook.site/ is a convenient service for that)

"RECORDING_END": [
  {
    "handler": "https://webhook.site/id",
    "sync": false
  }
]

3. Push:

ffmpeg -re -i bbb/source_000.ts -c:a copy -c:v copy -f flv rtmp://localhost/live/test

4. Get CID from callback and watch (it takes about 5 mins for gateway URL to become available)

ffplay "https://gateway.pinata.cloud/ipfs/CID?filename=out.flv" 

CC: @yondonfu @victorges

[victorges]

Is this saving Livestream recordings on IPFS? We likely don't want that since at least Pinata is very expensive and we don't want every recording to go to IPFS anyway, especially not as individual segment files. In other words, what is the relationship between IPFS and RECORDING_END?

…

On Wed, 28 Sept 2022, 07:07 Ivan Poleshchuk, ***@***.***> wrote: An ugly, but fully functional, hack to get back IPFS CID from catalyst-uploader and fire RECORDING_END trigger through log handling functions. Hope it's a good starting point to spec out a proper implementation of handling uploader output. Worst case it can be merged, if we need IPFS output tomorrow. *Catalyst issue:* livepeer/catalyst#154 <livepeer/catalyst#154> *Important:* MP4 "streaming" is not currently supported by Mist, so this is tested with FLV output. *Testing:* 1. Add autopush to mist.conf, using Pinata JWT auth token "autopushes": [ [ "test", ***@***.***/out.flv" ] ], 1. Add RECORDING_END trigger (https://webhook.site/ is a convenient service for that) "RECORDING_END": [ { "handler": "https://webhook.site/id", "sync": false } ] 1. Push: ffmpeg -re -i bbb/source_000.ts -c:a copy -c:v copy -f flv rtmp://localhost/live/test 1. Get CID from callback and watch (it takes about 5 mins for gateway URL to become available) ffplay "https://gateway.pinata.cloud/ipfs/CID?filename=out.flv" CC: @yondonfu <https://github.com/yondonfu> @victorges <https://github.com/victorges> ------------------------------ You can view, comment on, or merge this pull request online at: #111 Commit Summary - 86c6e4a <86c6e4a> Get IPFS CID back from uploader and invoke RECORDING_END trigger File Changes (6 files <https://github.com/DDVTECH/mistserver/pull/111/files>) - *M* lib/procs.cpp <https://github.com/DDVTECH/mistserver/pull/111/files#diff-945f2384dbc31dd2cee1d0bbbb8046a556b4af3cbd0e70896db70f80c90ac037> (4) - *M* lib/procs.h <https://github.com/DDVTECH/mistserver/pull/111/files#diff-342734eb8936d57061b2f1e79c8ba16995ad2875434e4a6ffbfdff18e79453e1> (2) - *M* lib/util.cpp <https://github.com/DDVTECH/mistserver/pull/111/files#diff-08a4140127ae96e3cd9b623958ac14415670101fd6cb9f7cb88a2c20916e77db> (18) - *M* lib/util.h <https://github.com/DDVTECH/mistserver/pull/111/files#diff-ea93b77f3cebd94adffe126d5b27594935e2625050fe51a27b1aafc8b66866cc> (4) - *M* src/output/output.cpp <https://github.com/DDVTECH/mistserver/pull/111/files#diff-00ebf6c0a68d2ab6673bcd2f5e1b7e79a95c5efe865366eb85eb26a64e819f7b> (27) - *M* src/output/output_flv.cpp <https://github.com/DDVTECH/mistserver/pull/111/files#diff-60e5b7a63e8e8bcd8b9efaa49d6438cd5c184ce70157882a7c63a209142b9ea3> (2) Patch Links: - https://github.com/DDVTECH/mistserver/pull/111.patch - https://github.com/DDVTECH/mistserver/pull/111.diff — Reply to this email directly, view it on GitHub <#111>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMJ4RZUHWAF2EMJKTYSJRTWAQKF7ANCNFSM6AAAAAAQXTTHRQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cyberj0g]

@victorges we need some trigger to catch CID on the caller, when upload finishes, and decided to use RECORDING_END. If autopush is configured, it will push all incoming streams which match the pattern. In other cases, it will upload to ipfs://:jwt_secret@pinata.cloud/out.flv if it set as destination (MP4 not working right now), though, I'm not sure how exactly VOD works with the studio, do you have some sample curl call?

[victorges]

Ah I see! Makes sense! Here is the VOD upload design: https://www.notion.so/livepeer/Technical-Design-Studio-Catalyst-API-for-VOD-Recording-bba2b160efff4ad4963dae2d4e9ff1a2

…

On Wed, 28 Sept 2022, 13:40 Ivan Poleshchuk, ***@***.***> wrote: @victorges <https://github.com/victorges> we need some trigger to catch CID on the caller, when upload finishes, and decided to use RECORDING_END. If autopush is configured, it will push all incoming streams which match the pattern. In other cases, it will upload to ipfs:// if it set as destination, though, I'm not sure how exactly VOD works with the studio, do you have some sample curl call? — Reply to this email directly, view it on GitHub <#111 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMJ4R755F4HJHBHSX6TUI3WARYGNANCNFSM6AAAAAAQXTTHRQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[victorges]

In other cases, it will upload to ipfs://:jwt_secret@pinata.cloud/out.flv

Just a reminder to use a different scheme here not to confuse this with the IPFS protocol itself. WDYT of pinata://, or even ipfs+pinata:// since the driver implementation will also not work for any other provider, only Pinata?

[cyberj0g]

WDYT of pinata://, or even ipfs+pinata://

Sure, missed that, I think ipfs+pinata:// makes most sense.