FFmpeg at the speed of M5

Thanks to VideoToolbox GPU hardware support, encoding 4K HDR videos to HEVC 1080p SDR is extremely fast. My MacBook Air M5 is running the pipeline at 256fps.

For Apple devices Hardware video encoding goes through VideoToolbox, which is part of the OS. FFmpeg has native AAC and AC3 encoders. A zero dependency build covers the audio + video support I needed.

curl -LO https://ffmpeg.org/releases/ffmpeg-8.1.2.tar.xz
tar xf ffmpeg-8.1.2.tar.xz && cd ffmpeg-8.1.2

./configure \
  --prefix=$HOME/.local \
  --enable-videotoolbox \
  --enable-audiotoolbox \
  --disable-doc --disable-debug
make -j$(sysctl -n hw.ncpu) && make install

Building that gives us hevc_videotoolbox, h264_videotoolbox, aac, aac_at (Apple's hardware-assisted AAC), and ac3. For most transcoding jobs, we are done here.

The HDR problem

My source files are HDR10: HEVC Main 10 with BT.2020 colors. My projector is SDR. We cannot just downscale; we need a tonemap, or the output looks gray and washed out.

We could use zscale filter chain from libzimg. But that will slow down our pipeline as process will be done on CPU rather than GPU.

Enter scale_vt

Buried in ffmpeg's VideoToolbox support is scale_vt: a hardware scaler that also converts color properties. As for bt709 and Apple's own HDR-to-SDR conversion runs on the GPU. Combined with hardware decode and encode, frames never touch the CPU:

[GPU decode] → [GPU scale + tonemap] → [GPU encode]

The final command:

ffmpeg \
-hwaccel videotoolbox \
-hwaccel_output_format videotoolbox_vld \
-i input.mkv \
-map 0 \
-vf "scale_vt=w=1920:h=-2:color_matrix=bt709:color_primaries=bt709:color_transfer=bt709" \
-c:v hevc_videotoolbox \
-b:v 19M \
-maxrate 28M \
-bufsize 38M -tag:v hvc1 \
-c:a copy \
-c:s mov_text \
output.mp4

Some details that matter:

h=-2 preserves aspect ratio instead of forcing 1080. -tag:v hvc1 makes the HEVC playable in QuickTime and Safari. -c:s mov_text converts srt subtitles to MP4's native format. VideoToolbox rate control is loose. -b:v 19M alone let bitrate peaks hit 50 Mbps. That may be ok for most video players may result in frame drops in low budget ones. We can use -maxrate and -bufsize to control that.

My MacBook Air runs cool and silent, and a 2.5 hour video gets encoded in around 15 minutes.

The trade-offs

Tonemap quality. With scale_vt you get Apple's tonemap curve. The zscale chain gives you more control and peak colors.

Film grain. Hardware encoders are mediocre at grain. Grain is noise; noise does not compress, and starved hardware encoders smear it into blotches. For grainy films, either keep the bitrate high (19M at 1080p is Blu-ray territory but defensible) or accept that true grain preservation means software x265 with -tune grain at a tenth of the speed.

Audio compatibility. Some of my files carry E-AC3 with Atmos. As the projector's firmware has a HiSilicon E-AC3 decoder I would keep that.

Tonemaping HDR to SDR with zscale

Apple tonemap may result in dark mid tones in some HDR content. To fix that we will need to use zscale at CPU level at a cost of ~40% of the speed.

ffmpeg -hwaccel videotoolbox \
  -i ./input.mkv \
  -map 0:0 -map 0:s -map 0:a \
  -vf "zscale=w=1920:h=-2:f=lanczos,zscale=t=linear:npl=100,tonemap=mobius:desat=0,zscale=p=bt709:t=bt709:m=bt709:r=tv,format=yuv420p" \
  -c:v hevc_videotoolbox -b:v 25M -maxrate 35M -bufsize 40M -tag:v hvc1 \
  -c:a copy \
  -c:s copy \
  ./out.mkv

Consider replacing tonemap=mobius with tonemap=hable (o other tonemap technic) if image gets too washed or some details are lost. See ffmpeg-filters for reference.