Lessons Learned: Evidence Over Assumptions

This isn’t a complaint about any company or service. It’s a record of an engineering process — the one BlueCrow follows when a production decision has to get made and the available information doesn’t agree with itself.

The objective

We needed our website to send real transactional email — signup confirmations, verification links, welcome messages — through our email provider, on our own verified domain, without disrupting the free email forwarding our root domain already depends on for day-to-day contact mail. That second constraint mattered as much as the first: whatever we did to get outbound email working could not be allowed to break inbound mail to the address our own site lists as a way to reach us.

The conflicting evidence

Almost nothing about this went the way the documentation said it would.

An environment variable that should have been present reported as missing — until we discovered the tool we were using to inspect it redacts sensitive values instead of showing them, so we had, briefly, mistaken a redaction placeholder for a real diagnosis and reported the wrong root cause. Support for our DNS registrar told us plainly that a needed configuration mode couldn’t coexist with our existing forwarding setup, and separately pointed us at a specific settings panel to work around it — a panel that, when we actually opened it, didn’t contain the option we’d been told to use. A secondary DNS provider we tried to sign up with never delivered its account-confirmation email, for reasons that turned out to be plausible but undocumented. And a different feature of our own registrar, one that looked from its own knowledge base like it should sidestep the whole problem, could not be confirmed to actually work that way — no one could show us, firsthand, that it did.

Every one of these was a case where something written down — a doc, a support reply, a public wiki article — disagreed with what we could actually observe. None of the sources were acting in bad faith. Documentation goes stale. Support scripts lag behind product changes. Free tiers change their own rules. That’s normal. What isn’t acceptable is building production infrastructure on top of a claim we hadn’t checked.

How we investigated

Every step got the same treatment: state the claim precisely, then find a way to observe it directly instead of accepting it. That meant reading actual server logs instead of guessing at an error from its HTTP status code, querying DNS directly against the authoritative name servers instead of trusting a single resolver’s answer, and running the exact production request path instead of relying on a dashboard’s status word. Several times, an assumption that looked completely reasonable — “the environment variable is set, so the send will work”; “if support says a panel supports this, it supports this” — failed the direct check. Each failure changed the plan immediately, before any DNS record was touched.

The decision criteria

One rule governed every choice along the way: if the evidence for a claim was “someone said so” or “the documentation implies it,” that claim did not get acted on until it was independently confirmed. Confirmation had to be something we could point to afterward — a log line, a DNS response from an authoritative server, a successful end-to-end test — not a paraphrase of what a support article or a settings label suggested.

Why we refused to rely on assumptions

Twice during this investigation we were one click away from an action that looked correct on paper and would have made things measurably worse in practice — either by risking the very forwarding setup we were trying to protect, or by spending the time budget we had on a path whose critical step had never actually been confirmed to exist. Both times, the fix was the same: stop, and get firsthand evidence before continuing. Neither problem was obvious from documentation alone. Both were obvious the moment we looked directly at the system instead of a description of the system.

Why changing direction was correct

Partway through, our first-choice secondary DNS provider stopped being viable for reasons entirely outside our control, and a fallback we had lined up turned out to depend on a product behavior we could not verify firsthand. In both cases we changed course rather than push forward on an unconfirmed assumption to save time. That is the correct trade in engineering: a plan you can prove is slower than a plan you can’t, but it is the only kind of plan you can actually stand behind once it’s carrying production traffic.

The architecture we chose

Phase one of our email infrastructure is deliberately temporary and deliberately narrow: a single subdomain, carved out and delegated to a dedicated DNS host solely to carry the one record our email provider needs, with our primary domain’s forwarding setup left completely untouched throughout. Phase two, planned and documented but not yet started, consolidates everything onto a single, standard business email setup once we’re ready to make that change on our own schedule rather than under time pressure. We wrote both phases down before finishing phase one, specifically so the temporary piece doesn’t quietly become permanent by default.

What we learned

Tooling can mislead you without lying to you — a value can be redacted for good security reasons and still look, to an unfamiliar eye, like a malformed secret. A support answer can be accurate about a rule and wrong about where to apply it. A wiki article can describe a product’s intent without describing its current behavior. None of these are failures of the people or systems involved. They’re the normal gap between “documented” and “true, right now, for this exact configuration” — and that gap is precisely where production incidents come from if nobody checks it.

The principle

We’re writing this down as a permanent standard, not a one-off lesson from a single afternoon.

Engineering Principle #001 Evidence outweighs documentation. When documentation and observable behavior disagree, production decisions must be based on verified evidence.