Transition from v2 ffmpeg read/write to v3 pyav read/write: quality, batching

[tfaehse]

Hi!

I'm currently trying to switch to v3's read and write functionality for videos. For the most part everything works smoothly, I just have a few understanding issues on my side that I'd love to get some help with.

1. My write code looks as follow:

with iio.imopen(temp_output, "w", plugin="pyav") as writer:
    writer.init_video_stream("libx264", fps=fps)
    ...
    writer.write(batch)

This works great, but I can't seem to figure out how to control the quality - in v2's FFmpeg writer there's the quality keyword that allows a quality between 0 and 10. How can I achieve something similar in v3 with pyav?

2. My read works as follows:

with iio.imopen(input_path, "r", plugin="pyav") as reader:
    for batch_index, frame_buffer in enumerate(chunked(reader.iter(thread_type="FRAME"), batch_size)):
        ...

I feel like I'm losing out on performance here, because I don't understand how to do a batch read properly. This snippet absolutely works, but do I understand the batching correctly? My goal is to load a batch of images into memory as fast as possible, process it and move on with the next batch.

I have, of course, tried to read the docs, but for these two questions I can't seem to come up with a solution. Thanks in advance for your feedback, and please tell me if there's any information I'm missing!

System information: python 3.11 on macOS (arm), av 9.2.0 and imageio 2.33.1.

[Pandede]

For the first question, you may modify the quality via crf: https://trac.ffmpeg.org/wiki/Encode/H.264#crf.
For the vanilla pyav, writing the video with specific crf can invoke the av.open like:

import av
container = av.open('video.mp4', mode='w', options={'crf': '18'})

However, imageio still does not support passing parameters to the internal function av.open. This feature #1061 is still under the progress.

[tfaehse]

That's absolutely fantastic, thank you! Even better because imageio v2's quality mapping to crf is known, I can just change that internally :)
I'll leave this open in case someone knows more about how to "properly" read frames in batches, but next weekend I should also be able to look into it a bit more and send an update here.

[FirefoxMetzger]

I just finished and merged #1061 and it should be released on Monday unless there are problems during CD. With it you should be able to do:

with iio.imopen(temp_output, "w", plugin="pyav", options={'crf': '18'}) as writer:
    writer.init_video_stream("libx264", fps=fps)
    ...
    writer.write(batch)

and it should control quality :)

As for how to read batches for frames: Performance-wise your code looks quite good already. You could simplify the syntax by introducing a generator though, which might give you an edge performance-wise depending on how chunked is implemented today.

import imageio.v3 as iio
import numpy as np
from numpy.typing import NDArray


def batches(filename, /, *, batch_size=32):
    frame:NDArray

    with iio.imopen(filename, "r", plugin="pyav") as file:
        props = file.properties()
        batch = np.empty((batch_size, *props.shape[1:]), dtype=props.dtype)
        for frame_number, frame in enumerate(file.iter(thread_type="FRAME")):
            batch_idx = frame_number % batch_size

            if batch_idx == 0 and frame_number > 0:
                yield batch
            
            batch[batch_idx] = frame

        if batch_idx > 0:
            yield batch[:batch_idx+1]
        

for batch in batches("imageio:cockatoo.mp4"):
    # Note: Hashing to show that data is different
    print(batch.shape, hash(batch.data.tobytes()))

The above creates a single ndarray called batch and reuses it when yielding frames. This is slightly better than the alternative of using np.stack(frames) because it allocates a buffer once instead of once per batch (less malloc is faster) and it only requires batch.size + frame.size amount of memory instead of batch.size + batch_size * frame.size.

Unfortunately, it's (to my knowledge) currently not possible to "buffer pushdown" in pyav. I.e., I can't pass batch[batch_idx] into pyav's read function and have it use that to store the result. Instead, we must copy the array in the iterator. If we could the ideal implementation would change and we could get better performance than either your or my variant.

Sidenote: pyav is threaded and these threads run independently of the python interpreter. In other words, while you are consuming the first batch pyav (or rather ffmpeg) will already decode frames for the next batch in the background. Afaik, these frames are queued up in an internal buffer on the ffmpeg side and surfaced to the python layer once you request them (during next(file.iter)). This is why thread_type="FRAME" can give you such a performance boost when reading multiple times.

[ibbbyy]

Whenever I pass in crf as an options kwarg I get an end of file error. Has anyone else had this problem?

[FirefoxMetzger]

@ibbbyy Could you write/share an example snippet I can test? This could be a bug either in the plugin or in PyAV, but it's hard to say without being able to "play around" with it.

[ibbbyy]

@FirefoxMetzger After further experimentation I've realized now that my error came from passing in an integer instead of a string. However, now I am getting Some options were not used: {'crf': '18'}.
The only difference I can really see in my code is that I'm using h264 as my codec, using libx264 just gives me an unknown codec error. (perhaps I need to install it?) I am using version 2.34.0 which as far as I can tell is the most recent version of the library.
Here's a stripped down version of my code that has the issue.
Thank you so much for looking into this!

[CescMessi]

@FirefoxMetzger After further experimentation I've realized now that my error came from passing in an integer instead of a string. However, now I am getting Some options were not used: {'crf': '18'}. The only difference I can really see in my code is that I'm using h264 as my codec, using libx264 just gives me an unknown codec error. (perhaps I need to install it?) I am using version 2.34.0 which as far as I can tell is the most recent version of the library. Here's a stripped down version of my code that has the issue. Thank you so much for looking into this!

I also got Some options were not used: {'crf': '18'} when I add options to av.open, and the result file size didn't changed. I found using stream.options is useful according to that issue. And we don't need to update imageio to pass parameters.

output_file = iio.imopen(output_video_path, 'w', plugin='pyav')
output_file.init_video_stream("libx264", fps=output_video_fps)
output_file._video_stream.options = {'crf': '18'}