I carry two YubiKeys. One lives on my keyring, one lives in a drawer, and both are enrolled everywhere that matters. A hardware token with no backup is a single point of failure you have chosen on purpose.
For months I had a small irritation I never bothered to chase. Some days git push asked for my PIN, I typed it, and it worked. Other days it asked for my PIN, failed, and then asked again, and the second one worked. Same command, same repo, same laptop. The variable turned out to be which of the two keys happened to be in the USB port.
It is a five-second annoyance — and I hit it maybe fifteen times a day.
What was actually happening
Both keys are sk-ssh-ed25519 credentials, and both get loaded into the gnome-keyring agent automatically:
$ ssh-add -l
256 SHA256:5imEVNLX... yk1-12345678 (ED25519-SK)
256 SHA256:ATug2xnP... yk2-12345679 (ED25519-SK)
I had no ~/.ssh/config at all. Without one, ssh offers the agent’s keys in the order the agent hands them over, and the server takes the first one it recognises. GitHub has both of these enrolled, so it accepts the first offer every time. Which means ssh commits to yk1 before anything has checked whether the token in the port can actually satisfy it.
With yk1 in the port, that guess is right and you see one prompt. With yk2 in the port, ssh asks the attached token to sign for a credential it has never heard of. You get a PIN dialog, you type your PIN, it fails, ssh falls through to the second key, and that one works.
The detail that makes this worse than it needs to be is how these keys were generated. Both are resident keys with user verification required. The flags byte in the private key blob is 0x25, which decodes, using the constant names from OpenSSH’s sk-api.h, as:
0x25 = SSH_SK_USER_PRESENCE_REQD | SSH_SK_USER_VERIFICATION_REQD | SSH_SK_RESIDENT_KEY
USER_VERIFICATION_REQD is what turns this from invisible into annoying, and the ordering that causes it sits above the token rather than inside it. Before ssh can request an assertion at all, the client has to obtain a PIN/UV auth token from the authenticator, and that is the step that puts the dialog on screen. Only after you have typed the PIN does the assertion request go down the wire, and only then does the key answer that it has never held this credential. Without the flag, the wrong-key attempt would fail silently and I would never have noticed.
The information was already there
Here is the part I feel slightly silly about. Look at the comments on those keys again:
yk1-12345678
yk2-12345679
Past me, generating these, had already written the serial number of the owning YubiKey into each key’s comment. The mapping from “which token is plugged in” to “which key can possibly work” had been sitting in ssh-add -l output the entire time, and I had spent months not reading it.
OpenSSH gives you exactly the right hook for this. Match exec runs a shell command and applies the block if it exits zero:
Match host github.com,gitlab.com exec "%d/.ssh/yk-plugged 12345678"
IdentityFile ~/.ssh/id_ed25519_sk_yk1
IdentitiesOnly yes
Match host github.com,gitlab.com exec "%d/.ssh/yk-plugged 12345679"
IdentityFile ~/.ssh/id_ed25519_sk_yk2
IdentitiesOnly yes
Note the host criterion, because leaving it off is the mistake that turns a papercut fix into a bad afternoon. Match exec on its own applies to every destination, and the first block that matches pins IdentitiesOnly and a single IdentityFile for whatever you happen to be connecting to. Every other host, with every other key, stops authenticating.
IdentitiesOnly yes is the load-bearing line, and it is the one people leave out. I had assumed the name meant “ignore the agent”. The -v output says otherwise: the agent still returns both keys. What the option actually restricts is which of them are eligible to be offered: only the ones whose public half matches an IdentityFile entry in scope. Two keys in, one considered.
That matching is also why the IdentityFile paths have to exist on disk even though the keys live in the agent. Since these are resident credentials, it is entirely possible to arrive on a fresh machine, pull them off the token with ssh-add -K, and have nothing in ~/.ssh at all. In that state the blocks match nothing and do nothing, which is confusing for about ten minutes.
If neither serial matches, no block applies, and you fall back to the old behaviour of trying everything. That is the correct failure mode: a config that gets worse than no config when it does not understand the situation is a config you will eventually curse. If both keys happen to be attached, both blocks match and both IdentityFile entries land in scope (IdentityFile accumulates across directives, unlike most keywords), so you are back to a coin flip and possibly one doomed prompt. Correct behaviour, but it does mean the fix looks broken if you plug in the drawer key without unplugging the keyring one.
The bit that was actually hard
Match exec runs on every single ssh invocation. Every git fetch, every scp, every ssh -G a tool makes to introspect your config. Whatever goes in there has to be cheap.
My first instinct was to read the serial out of sysfs, the way you would for most USB devices. That does not work:
$ cat /sys/bus/usb/devices/3-4/serial
$ cat /sys/bus/usb/devices/3-4/product
YubiKey OTP+FIDO+CCID
Empty. A YubiKey does not publish its serial in the USB device descriptor. You have to ask the device over one of its application interfaces, which in practice means ykman:
$ ykman list --serials
12345678
That call takes about 350 milliseconds on my laptop, most of it spent bringing the CCID interface up. Two Match exec blocks means two invocations, so seven tenths of a second on every git operation in order to save five seconds fifteen times a day. The arithmetic does not work.
So the helper caches. The question is what to key the cache on, and “a timeout” is the wrong answer. A 60-second TTL buys you either a minute of wrong answers after a swap or a minute of pointless ykman calls, and you get to choose which.
There is a better key available. The Linux USB stack assigns a device number on enumeration, and it increments. Swap tokens, even into the same physical port, and the number changes:
# yk1 attached
/sys/bus/usb/devices/3-4/ busnum=3 devnum=10
# swapped to yk2, same port
/sys/bus/usb/devices/3-4/ busnum=3 devnum=11
# swapped back to yk1
/sys/bus/usb/devices/3-4/ busnum=3 devnum=12
That gives a cache key derived from the thing whose change I care about. Reading it costs a cat of a sysfs file. No timeout, no staleness window, no daemon watching udev.
#!/bin/sh
# Exit 0 if a YubiKey with the given serial number is attached.
set -eu
want=$1
# Fingerprint the attached Yubico (vendor 1050) USB devices.
fp=
for d in /sys/bus/usb/devices/*/; do
[ -r "$d/idVendor" ] || continue
[ "$(cat "$d/idVendor")" = 1050 ] || continue
fp="$fp$(cat "$d/busnum")-$(cat "$d/devnum"),"
done
[ -n "$fp" ] || exit 1 # nothing attached; no point spawning ykman
# Deliberately no /tmp fallback; see below.
cache=${XDG_RUNTIME_DIR:?}/yk-serials
if [ -r "$cache" ] && [ "$(head -n1 "$cache")" = "$fp" ]; then
serials=$(tail -n +2 "$cache")
else
serials=$(ykman list --serials 2>/dev/null) || serials=
# Don't cache a failed lookup (busy pcscd, ykman missing); it would stick
# until the next replug.
if [ -n "$serials" ]; then
tmp=$cache.$$
(umask 077; printf '%s\n%s\n' "$fp" "$serials" >"$tmp") && mv -f "$tmp" "$cache"
fi
fi
printf '%s\n' "$serials" | grep -qx -- "$want"
Two details in there are scar tissue rather than design. The early exit when no Yubico device is present keeps ykman from being spawned at all on a laptop with nothing plugged in, which is the common case when I am working offline. And declining to cache an empty result matters because a transient failure — a busy pcscd, a ykman that is not installed yet — would otherwise be cached against a fingerprint that stays valid until the next replug.
The hard requirement on XDG_RUNTIME_DIR is there for a related reason. It is torn down at logout, so the cache can never outlive a boot. Fall back to /tmp on a system where /tmp survives a reboot and the fingerprint stops meaning anything: device numbers are allocated from the low end again after every boot, so a stale entry has a very good chance of matching one.
Warm, the helper returns in about 9ms. Resolving the whole config with both Match blocks in it (ssh -G github.com) comes to 87ms, the remainder being ssh’s own startup rather than anything I added. Cold, on the first connection after a swap, you pay the 350ms once.
Checking it actually works
ssh -v is useful here because it prints the whole decision path, key by key, rather than just the result. With the backup key in the port:
debug1: get_agent_identities: agent returned 2 keys
debug1: Will attempt key: ~/.ssh/id_ed25519_sk_yk2 ED25519-SK
debug1: Offering public key: ~/.ssh/id_ed25519_sk_yk2 ED25519-SK
debug1: Server accepts key: ~/.ssh/id_ed25519_sk_yk2 ED25519-SK
Authenticated to github.com using "publickey".
Two keys in, one attempt out. agent returned 2 keys alongside a single Offering public key is the whole fix between them: the agent’s inventory is unchanged, and the filtering happened before anything went out on the wire, let alone reached the token.
I tested the other direction too, and I want to be honest about what that proved, because it is less than it looks. With yk1 attached, the run is identical and authentication succeeds on the first attempt. But that direction already worked before any of this, purely because id_ed25519_sk_yk1 sorts before id_ed25519_sk_yk2 and got offered first by luck. The value of testing it was confirming I had not broken the path that was fine. The yk2 case is the one that fixed a real failure.
That asymmetry is also why the bug survived so long. Half my usage was already correct, so it never looked like a broken configuration — it looked like the drawer key being flaky.
What this does not fix
It does not remove the PIN prompt. It removes the doomed one. I still verify and still touch, which is the entire reason for carrying a hardware token at all; a change that removed those would be trading the whole point for a fraction of a second.
There is one theoretical hole. USB device numbers wrap at 128, so a sufficiently unlucky sequence of replugs could land on a device number matching the cached one while holding a different token. The cost would be a single stale prompt, self-correcting on the next swap. I thought about defending against it for about a minute and then decided that engineering around a 1-in-128 chance of reproducing the exact bug I started with was not a good use of an afternoon.
The general shape
Match exec is the most underused thing in ssh_config. Most people know Match host and stop there, and then work around everything else with wrapper scripts or shell aliases or a mess of -i flags in their git remotes. But the predicate can be any command — which means your ssh config can respond to essentially any fact about the machine it is running on: which VPN is up, which network you are on, whether a particular file exists, which hardware token is in the port.
The constraint that makes it interesting is that it sits in the hot path of every connection, so the fact you want has to be cheap to establish. Half the work here was not “how do I select the right key”, which is four lines. It was “how do I answer a 350ms question in under 10ms without ever answering it wrong”. That is a caching problem, and the trick is nearly always the same — find the thing that already changes when your answer changes, and key on that instead of on the clock.
Setup notes: Fedora, OpenSSH 10.2p1, YubiKey 5C NFC on firmware 5.4.3, keys held by the gnome-keyring agent at $XDG_RUNTIME_DIR/gcr/ssh. Serial numbers in the examples are made up; use your own from ykman list --serials.
Reviewed with Claude, which caught the unscoped Match blocks and an error in how I’d explained the PIN/UV ordering. Prose and conclusions are mine.