One evening at the end of May we rolled out a new supervisor for our SMTP relay, and for the rest of that rollout one client's mail went out without a DKIM signature. Every log line said the mail was sent, and it was. It just wasn't signed, and DMARC failed on every message. Nine separate problems lined up to cause it. The most important one was a single Postfix setting that turned "the signer is down" into "send it anyway".
We run a multi-tenant SMTP relay. Devices like scanners and printers submit mail to it, and it signs that mail with DKIM and passes it on. For this client, it delivers into their Microsoft 365 tenant. Signing is done by OpenDKIM running as a milter next to Postfix.
The relay had been running from a shell entrypoint script. We were replacing that with a Go agent that supervises the mail daemons and renders their config. That evening it went to production as seven releases in a row.
Exchange Online's Authentication-Results header on one of the client's messages said:
dkim=none (message not signed)
dmarc=fail action=orejectThe domain publishes a DMARC policy of reject. The mail only reached inboxes because of how the client's tenant handles mail from the relay. Sent to any other receiver, it would have been rejected. Mail for our own domains was fine the whole time, because it goes through an email provider that signs it.
We rolled back to the shell version, and the next message showed dkim=pass.
The agent started OpenDKIM wrong in two ways, and each one alone would have broken signing.
First, it launched opendkim without -x /etc/opendkim.conf. The Socket line lives in that file, so OpenDKIM never bound the port Postfix was trying to reach. Started in the foreground with no config, it just dies with smfi_opensocket() failed.
Second, an earlier version launched it without -f. OpenDKIM then forks into the background, the process the agent started exits and becomes a zombie, and the real daemon runs with nothing watching it. When it died, nothing restarted it.
The fix is one line, plus a supervisor that restarts the signer with a capped backoff:
opendkim -x /etc/opendkim.conf -fThere was also a startup race. Postfix could start accepting mail before OpenDKIM had bound its port. The agent now waits until the milter port accepts a TCP connection before it starts Postfix.
With the signer gone, Postfix still had a choice about what to do with each message. That choice is milter_default_action, and ours was set to accept. So when Postfix couldn't reach the milter, it skipped it and delivered the message unsigned. No error, no deferral.
Postfix's own default for this setting is tempfail. The shell entrypoint had overridden it to accept, which trades signing for availability. Under the shell version it had never mattered, because OpenDKIM stayed up.
With tempfail, Postfix answers the device with a 4xx and the message isn't accepted. The sender queues it and retries. By then the supervisor has restarted the signer, and the retry goes out signed. A short signer outage costs a few minutes of delay instead of a batch of DMARC failures.
# main.cf
milter_default_action = tempfail
smtpd_milters = inet:127.0.0.1:<port>
non_smtpd_milters = inet:127.0.0.1:<port>That covers the signer being unreachable. It doesn't cover the signer being up but failing. OpenDKIM decides those cases itself, so we also set:
# opendkim.conf
On-InternalError tempfailThere's still one case neither setting catches. If OpenDKIM is running but has no key for a domain, because a signing-table entry is missing, it just doesn't sign. Nothing errors. The only way to catch that is to look at the result where the mail lands, in received headers or in DMARC aggregate reports.
Not all nine causes were about signing, but they made the evening longer and the picture harder to read:
| Problem | Effect |
|---|---|
The slim container image had no procps | pkill and pgrep were missing, so the OpenDKIM reload failed and the health check broke |
The health check looked for a process named postfix | Postfix's master process is called master, so health showed "degraded" when Postfix was fine |
| Nothing in the health check looked at the signer | The one real failure didn't show up in health at all |
| SMTP logins were seeded in the wrong format | Devices wouldn't have been able to authenticate |
| A background poller reloaded Postfix at boot | Postfix could be reloaded before it had finished starting |
| Postfix could start before the milter was listening | Early mail had no signer to reach |
The Dockerfile used an older Go than go.mod | The first deploy failed to build |
The health check is worth thinking about. It was wrong both ways: it reported a problem that didn't exist, and it said nothing about the one that did. A health check that only checks processes can't tell you the mail is signed.
The technical causes explain what broke. They don't explain why it broke in production.
There was no test environment. Every change went straight to the one production app.
We were checking the wrong signal. For most of the rollout, success meant status=sent in the Postfix log. That proves delivery. It says nothing about signing. The check that mattered was the Authentication-Results header at the receiver, and it came late.
We were in a fix, deploy, fix loop. Six fixes went out across five releases, each aimed at the last symptom, and none was verified end to end before the next one shipped.
And accept hid the worst outcome. If the signer had been required, the first broken release would have deferred mail instead of sending it unsigned, and scans that don't arrive get noticed. With accept it looked like success.
The next morning we built a separate test app and wrote down how changes get verified there before they reach production, with a received dkim=pass header as the proof, not status=sent. The same morning we wrote and tested the fail-safe: tempfail and On-InternalError tempfail. The health endpoint now reports whether the signer is reachable and how many times it has restarted, and marks itself degraded when the signer is down.
The test app paid for itself within hours. With structured logs on the test app, we could see OpenDKIM dying about a minute after every clean boot. The config poller treated its first check as a change, ran a full reload, and the reload sent pkill -HUP opendkim. That killed the foreground signer and left about a second with no signer. Under accept that second would have meant unsigned mail. The poller now starts from the config version that was applied at boot, so it only reloads when something actually changes.
The rollback was the part that went well. We'd kept the previous image, the shell version's deployment config was in git, and one dkim=pass check confirmed it.
postconf milter_default_action. If it says accept, a dead signer means unsigned mail. Use tempfail unless you've decided availability matters more than DMARC.-f) and is started with its config file (-x). Then check that the milter port is actually listening, not just that the process exists.tempfail to cover the gap.dkim=pass in Authentication-Results is proof. status=sent is not.The fix that mattered most was one word in main.cf. The rest made the failure easier to see, but tempfail is what turns a broken signer into delayed mail instead of failed DMARC.
