# FFmpeg at the speed of M5

Thanks to VideoToolbox GPU hardware support, encoding 4K HDR videos to HEVC 1080p SDR is extremely fast. My MacBook Air M5 is running the pipeline at 256fps.

For Apple devices Hardware video encoding goes through VideoToolbox, which is part of the OS. FFmpeg has native AAC and AC3 encoders. A zero dependency build covers the audio + video support I needed.

```shell
curl -LO https://ffmpeg.org/releases/ffmpeg-8.1.2.tar.xz
tar xf ffmpeg-8.1.2.tar.xz && cd ffmpeg-8.1.2

./configure \
  --prefix=$HOME/.local \
  --enable-videotoolbox \
  --enable-audiotoolbox \
  --disable-doc --disable-debug
make -j$(sysctl -n hw.ncpu) && make install
```

Building that gives us `hevc_videotoolbox`, `h264_videotoolbox`, `aac`, `aac_at` (Apple's hardware-assisted AAC), and `ac3`. For most transcoding jobs, we are done here.

### The HDR problem

My source files are HDR10: HEVC Main 10 with BT.2020 colors. My projector is SDR. We cannot just downscale; we need a tonemap, or the output looks gray and washed out.

We could use `zscale` filter chain from libzimg. But that will slow down our pipeline as process will be done on CPU rather than GPU.

### Enter scale\_vt

Buried in ffmpeg's VideoToolbox support is scale\_vt: a hardware scaler that also converts color properties. As for bt709 and Apple's own HDR-to-SDR conversion runs on the GPU. Combined with hardware decode and encode, frames never touch the CPU:

`[GPU decode] → [GPU scale + tonemap] → [GPU encode]`

The final command:

```shell
ffmpeg \
-hwaccel videotoolbox \
-hwaccel_output_format videotoolbox_vld \
-i input.mkv \
-map 0 \
-vf "scale_vt=w=1920:h=-2:color_matrix=bt709:color_primaries=bt709:color_transfer=bt709" \
-c:v hevc_videotoolbox \
-b:v 19M \
-maxrate 28M \
-bufsize 38M -tag:v hvc1 \
-c:a copy \
-c:s mov_text \
output.mp4
```

Some details that matter:

`h=-2` preserves aspect ratio instead of forcing 1080. `-tag:v hvc1` makes the HEVC playable in QuickTime and Safari. `-c:s mov_text` converts srt subtitles to MP4's native format. VideoToolbox rate control is loose. `-b:v 19M` alone let bitrate peaks hit 50 Mbps. That may be ok for most video players may result in frame drops in low budget ones. We can use `-maxrate` and `-bufsize` to control that.

My MacBook Air runs cool and silent, and a 2.5 hour video gets encoded in around 15 minutes.

## The trade-offs

**Tonemap quality.** With `scale_vt` you get Apple's tonemap curve. The zscale chain gives you more control and peak colors.

**Film grain.** Hardware encoders are mediocre at grain. Grain is noise; noise does not compress, and starved hardware encoders smear it into blotches. For grainy films, either keep the bitrate high (19M at 1080p is Blu-ray territory but defensible) or accept that true grain preservation means software x265 with `-tune grain` at a tenth of the speed.

**Audio compatibility.** Some of my files carry E-AC3 with Atmos. As the projector's firmware has a HiSilicon E-AC3 decoder I would keep that.

### Tonemaping HDR to SDR with zscale

Apple tonemap may result in dark mid tones in some HDR content. To fix that we will need to use zscale at CPU level at a cost of ~40% of the speed.

```shell
ffmpeg -hwaccel videotoolbox \
  -i ./input.mkv \
  -map 0:0 -map 0:s -map 0:a \
  -vf "zscale=w=1920:h=-2:f=lanczos,zscale=t=linear:npl=100,tonemap=mobius:desat=0,zscale=p=bt709:t=bt709:m=bt709:r=tv,format=yuv420p" \
  -c:v hevc_videotoolbox -b:v 25M -maxrate 35M -bufsize 40M -tag:v hvc1 \
  -c:a copy \
  -c:s copy \
  ./out.mkv
```

Consider replacing `tonemap=mobius` with `tonemap=hable` (o other tonemap technic) if image gets too washed or some details are lost. See [ffmpeg-filters](https://ffmpeg.org/ffmpeg-filters.html#Options-3) for reference.