feat(spectrum): capture macOS output with a Core Audio process tap
SYNC mode on macOS needed a virtual loopback driver (BlackHole, Loopback, Soundflower) routed through a Multi-Output Device, and shelled out to 'ffmpeg -f avfoundation' to read it. macOS 14.2 lets a process tap the system mix directly, so do that instead: CATapDescription -> AudioHardwareCreateProcessTap -> private aggregate device -> IOProc, straight through ctypes. No driver, no helper binary, no ffmpeg. The tap hands out native-rate interleaved float32, so the analyser can now retune its window and band edges when the capture rate differs from 44.1 kHz, and mono-downmixes the frames before the FFT. A tap that was never granted System Audio Recording permission is created happily but only ever delivers digital silence: treat that as a failed capture instead of flat bars, explain it in the status bar and fall back to the stream tap. A tap that renders nothing *yet* (playback still starting, or mpv pointed at another device) is not a permanent failure either - the stream tap covers for a minute, then the tap is retried. Drop the avfoundation device enumeration, the loopback-name heuristics and the microphone safeguards' tests, and document the new permission and its diagnostics instead.
This commit is contained in:
1 parent
7042707e84
commit
48e480d58c
8 files changed
+988
-282
No files matched your search
@@ -22,10 +22,10 @@ implemented here from the public SomaFM endpoints and open protocols.
|
||||
spectrum; or a full-height oscilloscope. Three modes (`v`).
|
||||
- **Synced spectrum, not a guess** — by default the analyser records your
|
||||
actual audio output, so the bars follow what you hear (`SYNC`): the
|
||||
PulseAudio/PipeWire monitor via `parec` on Linux, an `ffmpeg`/avfoundation
|
||||
loopback input (BlackHole, Loopback, Soundflower, …) on macOS. Without an
|
||||
output capture it falls back to a muted second `ffmpeg` decode of the stream
|
||||
(`FFT`, can drift), then to a synthesized display (`SIM`).
|
||||
PulseAudio/PipeWire monitor via `parec` on Linux, and a native Core Audio
|
||||
process tap on macOS (no driver, no BlackHole, nothing to install). Without
|
||||
an output capture it falls back to a muted second `ffmpeg` decode of the
|
||||
stream (`FFT`, can drift), then to a synthesized display (`SIM`).
|
||||
- **Real artwork on iTerm2, Kitty and WezTerm** — Kitty uses the kitty
|
||||
graphics protocol; iTerm2 (including on macOS) and WezTerm use their native
|
||||
iTerm2 inline-image protocol. Images occupy exactly the ON AIR art cell box
|
||||
@@ -65,11 +65,14 @@ Python **3.10+** and a few command-line audio tools on `$PATH`.
|
||||
| `python3` | runtime (3.10 or newer; no exact version pin) |
|
||||
| [`uv`](https://docs.astral.sh/uv/) | creates the venv and installs every Python dependency from `uv.lock` |
|
||||
| `mpv` | playback and JSON IPC control |
|
||||
| `ffmpeg` | muted stream decode for the fallback analyser, and the macOS output capture |
|
||||
| `ffmpeg` | muted stream decode for the fallback analyser |
|
||||
| `parec` / `pactl` *(Linux)* | synced monitor capture for the spectrum (`pulseaudio-utils`; works with PipeWire) |
|
||||
| a loopback audio device *(macOS, optional)* | synced output capture for the spectrum: BlackHole, Loopback, Soundflower, … |
|
||||
| D-Bus session bus *(Linux)* | MPRIS desktop media integration (`playerctl` is handy for testing) |
|
||||
|
||||
On macOS the spectrum capture is part of the app: it uses the Core Audio
|
||||
process-tap API (macOS 14.2 or newer), so there is no loopback driver to
|
||||
install.
|
||||
|
||||
On macOS the media integration needs nothing extra: mpv is what the system's
|
||||
Now Playing session (Control Center, media keys) talks to.
|
||||
|
||||
@@ -110,33 +113,20 @@ the synced spectrum needs `parec`/`pactl`.
|
||||
brew install mpv ffmpeg uv # ffplay ships with ffmpeg
|
||||
```
|
||||
|
||||
That is enough to run: playback, artwork in iTerm2/Kitty/WezTerm, the muted
|
||||
`ffmpeg` spectrum tap and the macOS Now Playing integration all work out of the
|
||||
box. The first capture attempt makes macOS ask for **microphone** access for
|
||||
whatever terminal you use — that permission is what lets `ffmpeg` read the
|
||||
loopback device.
|
||||
That is enough to run: playback, artwork in iTerm2/Kitty/WezTerm, the synced
|
||||
Core Audio spectrum tap and the macOS Now Playing integration all work out of
|
||||
the box.
|
||||
|
||||
For a *synced* spectrum (`SYNC` instead of `FFT`) macOS needs an output
|
||||
loopback, because CoreAudio does not let an ordinary process record the
|
||||
speakers directly:
|
||||
The first time the analyser starts it asks macOS for **System Audio
|
||||
Recording** permission — a one-off prompt for your terminal (iTerm2, Kitty,
|
||||
WezTerm, …). Approve it under *System Settings → Privacy & Security → System
|
||||
Audio Recording*; no loopback driver, Multi-Output Device or audio routing is
|
||||
needed. If the permission is denied the tap only ever hands out silence, the
|
||||
analyser notices and falls back to `FFT` instead of showing dead bars.
|
||||
|
||||
```bash
|
||||
brew install blackhole-2ch # or Loopback / Soundflower
|
||||
```
|
||||
|
||||
Then either
|
||||
|
||||
1. create a **Multi-Output Device** in *Audio MIDI Setup* containing both your
|
||||
speakers/headphones and *BlackHole 2ch*, and select it as the system output,
|
||||
or
|
||||
2. set *BlackHole 2ch* as the output and monitor the audio elsewhere.
|
||||
|
||||
The app auto-detects loopback-style inputs (BlackHole, Loopback, Soundflower,
|
||||
“Multi-Output”, “Aggregate”) and never grabs a plain microphone by accident.
|
||||
Pick one explicitly with `SOMAFM_SPECTRUM_DEVICE` (name or avfoundation index,
|
||||
see `ffmpeg -f avfoundation -list_devices true -i ""`). If the loopback turns
|
||||
out to be silent (nothing routed into it), the analyser notices after a few
|
||||
seconds and falls back to `FFT` instead of showing dead bars.
|
||||
The tap records the **default output device**, which is also what mpv uses; if
|
||||
you point mpv elsewhere with `SOMAFM_MPV_ARGS='--audio-device=...'` the
|
||||
analyser falls back to `FFT`.
|
||||
|
||||
Terminal image rendering uses [Pillow](https://python-pillow.org/) and is
|
||||
implemented directly (Kitty TGP + iTerm2 OSC 1337) — no terminal probing.
|
||||
@@ -248,12 +238,16 @@ spectrum and the station list stay in sync. The status bar shows
|
||||
engine, and uses `SIGSTOP`/`SIGCONT` when falling back to `ffplay`. On macOS
|
||||
it also asks mpv to publish Now Playing and maps the media keys onto
|
||||
`script-message`s that come back over the same IPC socket.
|
||||
- `somafm_tui/spectrum.py` — 44.1 kHz mono s16le analyser. Prefers capturing
|
||||
the real output: `parec` (PulseAudio/PipeWire monitor) on Linux, or
|
||||
`ffmpeg -f avfoundation` on a loopback input on macOS (SYNC). Otherwise a
|
||||
- `somafm_tui/spectrum.py` — analyser (44.1 kHz mono s16le from `parec`, or
|
||||
the device's native float32 rate from the macOS tap). Prefers capturing the
|
||||
real output: `parec` (PulseAudio/PipeWire monitor) on Linux, or a native
|
||||
Core Audio process tap on macOS (SYNC, see `macos_tap.py`). Otherwise a
|
||||
muted low-latency `ffmpeg` decode of the stream (FFT). Hann-windowed
|
||||
2048-point FFT, log band grouping, auto-gain, fast attack/decay smoothing,
|
||||
peak hold and waveform downsampling.
|
||||
- `somafm_tui/macos_tap.py` — the macOS output capture, in pure `ctypes`
|
||||
against Core Audio: `CATapDescription` → `AudioHardwareCreateProcessTap` →
|
||||
private aggregate device → IOProc. No driver, no helper binary, no ffmpeg.
|
||||
- `somafm_tui/mediasession.py` — picks the desktop media backend (MPRIS on
|
||||
Linux, Now Playing on macOS) so the app has one platform-neutral interface.
|
||||
- `somafm_tui/mpris.py` — `dbus-next` MPRIS service in its own thread with a
|
||||
@@ -303,11 +297,11 @@ same dotfiles/config work everywhere; `XDG_CONFIG_HOME`, `XDG_CACHE_HOME` and
|
||||
either, the UI still runs and tells you. Extra mpv args can be injected with
|
||||
`SOMAFM_MPV_ARGS` (e.g. `SOMAFM_MPV_ARGS='--audio-device=...'`, which is the
|
||||
macOS way to choose an output device).
|
||||
- **Spectrum says `FFT` instead of `SYNC`?** `parec`/`pactl` are unavailable,
|
||||
the monitor device failed, or you forced the stream tap. The bars are still
|
||||
real audio, but the second stream can drift relative to playback. Use
|
||||
`--spectrum-source monitor` to force sync, or `SOMAFM_SPECTRUM_DEVICE=<source>`
|
||||
to pick a specific monitor/source. `SIM` means the bars are synthesized.
|
||||
- **Spectrum says `FFT` instead of `SYNC`?** The monitor is unavailable or
|
||||
failed, or you forced the stream tap. The bars are still real audio, but the
|
||||
second stream can drift relative to playback. Use `--spectrum-source
|
||||
monitor` to force sync, or `SOMAFM_SPECTRUM_DEVICE=<source>` on Linux to pick
|
||||
a specific monitor. `SIM` means the bars are synthesized.
|
||||
- **Diagnose artwork:** `uv run somafm-tui --diagnose-images` prints the
|
||||
detected protocol, widget class, terminal env, dependency versions and cache
|
||||
path. If `protocol` is `None`, force it with `SOMAFM_IMAGES=tgp` (Kitty) or
|
||||
@@ -315,12 +309,11 @@ same dotfiles/config work everywhere; `XDG_CONFIG_HOME`, `XDG_CACHE_HOME` and
|
||||
- **`SYNC` but the bars are flat?** mpv may be playing to a different sink (or
|
||||
directly to ALSA). Point `SOMAFM_SPECTRUM_DEVICE` at that sink's monitor, or
|
||||
use `--spectrum-source stream`.
|
||||
- **macOS: spectrum stays `FFT`?** Install a loopback (BlackHole/Loopback) and
|
||||
route your speakers through a Multi-Output Device, then check
|
||||
`ffmpeg -f avfoundation -list_devices true -i ""` lists it. The first capture
|
||||
triggers a macOS microphone-permission prompt; grant it to your terminal
|
||||
(System Settings → Privacy & Security → Microphone), otherwise ffmpeg cannot
|
||||
open the device and the analyser stays on `FFT`.
|
||||
- **macOS: spectrum stays `FFT`?** Grant your terminal **System Audio
|
||||
Recording** permission (*System Settings → Privacy & Security → System Audio
|
||||
Recording*), then restart the app; the status bar explains when the tap is
|
||||
silent because of this. Also check that mpv is playing to the default output
|
||||
device, since that is what the tap records.
|
||||
- **macOS: media keys do nothing?** Another app currently owns the Now Playing
|
||||
session, or the app was started with `--no-mpris`, or the audio backend fell
|
||||
back to `ffplay` (mpv is required for Now Playing). The status bar shows
|
||||
|
||||
Reference in new issue
Block a user