Video

A stream of images (like those produced by a camera) can be logged to Rerun in several different ways:

Uncompressed, as many Images
Compressed as many EncodedImages, using e.g. JPEG.
Compressed as a single AssetVideo, using e.g. MP4.

These alternatives range on a scale of "simple, lossless, and big" to "complex, lossy, and small".

If you want lossless encoded images (with no compression artifacts), then you should log each video frame as Image. This will use up a lot of space and bandwidth. You can also encode them as PNG and log them as EncodedImage, though it should be noted that PNG encoding usually does very little for the file size of photographic images.

If you want to reduce bandwidth and storage cost, you can encode each frame as a JPEG and log it using EncodedImage. This can easily reduce the file sizes by almost two orders of magnitude with minimal perceptual loss. This is also very simple to do, and the Python logging SDK has built-in support for it using Image.compress.

Finally, you can encode the images as a video file, and log it using AssetVideo. This gives the best compression ratio, reducing file sizes and bandwidth requirements.

"""Log a video asset using automatically determined frame references."""
# TODO(#7298): ⚠️ Video is currently only supported in the Rerun web viewer.

import sys

import rerun as rr

if len(sys.argv) < 2:
    # TODO(#7354): Only mp4 is supported for now.
    print(f"Usage: {sys.argv[0]} <path_to_video.[mp4]>")
    sys.exit(1)

rr.init("rerun_example_asset_video_auto_frames", spawn=True)

# Log video asset which is referred to by frame references.
video_asset = rr.AssetVideo(path=sys.argv[1])
rr.log("video", video_asset, static=True)

# Send automatically determined video frame timestamps.
frame_timestamps_ns = video_asset.read_frame_timestamps_ns()
rr.send_columns(
    "video",
    # Note timeline values don't have to be the same as the video timestamps.
    times=[rr.TimeNanosColumn("video_time", frame_timestamps_ns)],
    components=[rr.VideoFrameReference.indicator(), rr.components.VideoTimestamp.nanoseconds(frame_timestamps_ns)],
)

Video playback limitations video-playback-limitations

Video support is new in Rerun, and has a few limitations:

#7354: Only the MP4 container format is supported
#7755: No AV1 support on Linux ARM
#5181: There is no audio support
#7594: HDR video is not supported
There is no video encoder in the Rerun SDK, so you need to create the video file yourself

Streaming video streaming-video

Rerun does not yet support streaming video support. For scenarios where you don't need live video, you can work around this limitation by logging many small AssetVideos to the same Entity Path. See #7484 for more.

Codec support codec-support

Overview overview

Codec support varies in the web & native viewer:

	Browser	Native
AV1	✅	✅
H.264/avc	✅	✅
H.265/hevc	🔳	❌
VP9	✅	❌

Details see below.

When choosing a codec, we recommend AV1, as it seems to have the best overall playback support while also having very high compression quality.

Since AV1 can have very long encoding times, it is often not suitable for streaming. In those cases where encoding time matters, we recommend H.264/avc.

Native viewer native-viewer

AV1 is supported out of the box using a software decoder paired with gpu based image conversion
H.264/avc is supported via a system installed FFmpeg binary
- When no binary is found, Rerun offers a download link to a build of FFmpeg for your platform in the selection panel upon selecting the failed video

Web viewer web-viewer

Video playback in the Rerun Web Viewer is done using the browser's own video decoder, so the exact supported codecs depend on your browser.

Overall, we recommend using Chrome or another Chromium-based browser, as it seems to have the best video support as of writing.

For decoding video in the Web Viewer, we use the WebCodecs API. This API enables us to take advantage of the browser's hardware accelerated video decoding capabilities. It is implemented by all modern browsers, but with varying levels of support for different codecs, and varying levels of quality.

When it comes to codecs, we aim to support any codec which the browser supports, but we currently cannot guarantee that all of them will work. For more information about which codecs are supported by which browser, see Video codecs on MDN.

We tested the following codecs in more detail:

	Linux Firefox	Linux Chrome^{^2}	macOS Firefox	macOS Chrome	macOS Safari	Windows Firefox	Windows Chrome^{^3}
AV1	✅	✅	✅	✅	🚧^{^4}	✅	✅
H.264/avc	✅^{^4}	✅	✅	✅	✅	✅	✅
H.265/hevc	❌	❌	❌	✅	🚧^{^5}	❌	🚧^{^6}

1: Firefox on Linux has been observed to stutter when playing back H.264 video.

2: Any Chromium-based browser should work, but we don't test all of them. ^1

3: Chrome on Windows has been observed to stutter on playback. It can be mitigated by using software decoding, but this may lead to high memory usage. See #7595. ^1

4: Safari/WebKit does not support AV1 decoding except on Apple Silicon devices with hardware support. ^1 ^2

5: Safari/WebKit has been observed suttering when playing hvc1 but working fine with hevc1. Despite support being advertised Safari 16.5 has been observed not support H.265 decoding. ^1

6: Only supported if hardware encoding is available. Therefore always affected by Windows stuttering issues, see above. ^1

Beyond this, for best compatibility we recommend:

prefer YUV over RGB & monochrome formats
don't use more than 8bit per color channel
keep resolutions at 8k & lower (see also #3782)

Links links

Web video codec guide, by Mozilla