Running validators in production: keys, MEV-Boost, and uptime

A tale of two machines

In February 2021 a respected staking provider, Staked, watched about 75 of its validators get slashed in a single afternoon. There was no hacker and no exotic bug. They had done the most natural thing in operations — built a redundant, high-availability setup so that if one machine died, a backup would take over. But the backup ran a validator client loaded with the same signing keys, and for a window both machines were alive at once. Both dutifully signed. To the network, one key had signed two conflicting messages — the textbook definition of cheating — and the protocol did what it promises to do: it destroyed stake.

That story is the whole of this guide in miniature. Running a validator in production is not really about squeezing out extra yield; it is a discipline of not losing. You have three jobs, and their priority order never changes: guard the keys so they cannot be stolen, never sign two conflicting things, and stay online. Earlier rungs taught you what a validator does — attest, propose, earn rewards under proof of stake. This one is about how professionals keep that machine running for years without ending up in the slashing logs.

Two keys, and what each one can do

A validator is controlled by two different keys with two very different risk profiles. The signing key is a hot BLS12-381 key (it produces a BLS signature) that must be online every single epoch to attest and occasionally propose. The withdrawal credentials decide where the money can ever go. Modern setups use `0x01` credentials, which point at an ordinary Ethereum execution address that you set once and which is then immutable: all withdrawals — both the periodic skim of rewards above 32 ETH and the full balance when the validator exits — flow automatically to that address and nowhere else.

This split is a deliberate safety property, and it is worth internalizing exactly what an attacker who steals your signing key can and cannot do. They can grief you: they can get you slashed by double-signing, or submit a voluntary exit that forces your validator out of the active set. They cannot steal the funds — the money still leaves only to your `0x01` withdrawal address, which can live in cold storage on a machine that never touches the internet. So the signing key is the key you must keep available; the withdrawal key (or the mnemonic that derives both) is the key you must keep safe.

# Derive the BLS signing key + keystore from a fresh mnemonic
# (EIP-2333/2334 key tree; EIP-2335 password-encrypted keystore JSON)
deposit new-mnemonic \
  --num_validators 1 \
  --chain mainnet \
  --eth1_withdrawal_address 0xYourColdExecutionAddress   # sets 0x01 creds

# Output, per validator:
#   keystore-m_12381_3600_0_0_0-<ts>.json   <- signing key, encrypted
#   deposit_data-<ts>.json                  <- 32 ETH deposit + withdrawal creds
#
# The mnemonic re-derives EVERYTHING. It should NOT live on the signing box;
# back it up offline (paper / steel / Shamir shares) and walk away.

In a serious operation the signing key is rarely left sitting in a file next to the validator client. A common pattern is a remote signer (such as Web3Signer): the validator client asks it to sign each duty over a local network, the keys are encrypted at rest or held in an HSM, and — crucially — the signer keeps its own anti-slashing database. The deeper point is that the key material is derived deterministically from a single mnemonic via a key-derivation function, so whoever holds that seed holds everything.

The slashing-protection database: the conscience that says no

There are only two ways a well-behaved validator gets slashed, and both are violations of a slashing condition. The first is proposing two different blocks for the same slot. The second is a bad attestation: casting two votes for the same target epoch (a double vote), or a vote whose source/target interval surrounds an earlier one (a surround vote). Notice what these have in common — they all require the key to sign something it should never sign given its own history. That history is exactly what the slashing-protection database remembers.

The database is small and conceptually simple: for each public key it stores the highest block slot ever signed and the highest attestation source/target epochs ever signed. Before releasing any signature, the client checks the request against this record and refuses anything that would move backward or conflict. The order matters: it records the intent before it hands out the signature, so a crash mid-sign can only ever make it more conservative, never less.

# A validator client MUST consult its local DB before every signature.
function may_sign_block(pubkey, slot):
    last = db.max_signed_block_slot(pubkey)
    if last is not None and slot <= last:
        refuse("second block at/below an already-signed slot")
    db.record_block(pubkey, slot)         # commit BEFORE releasing signature
    return sign

function may_sign_attestation(pubkey, source, target):
    if target <= db.max_signed_target(pubkey):
        refuse("double vote: target epoch not strictly increasing")
    if source <  db.max_signed_source(pubkey):
        refuse("surround vote: source epoch moved backwards")
    db.record_attestation(pubkey, source, target)
    return sign

Because this safety lives in a local file, moving a validator between machines is the riskiest moment in its life. The clients agree on a standard export format, the EIP-3076 interchange JSON, precisely so that when you migrate you carry the signing history with you and the new client refuses to sign below the slot the old one already reached. Never start a fresh client on a key without importing this — an empty database happily signs a slot you already signed.

{
  "metadata": {
    "interchange_format_version": "5",
    "genesis_validators_root": "0x043db0...c43efb"
  },
  "data": [{
    "pubkey": "0xb845089a1457f811bfc000d4f76e09",
    "signed_blocks":       [{ "slot": "8195201" }],
    "signed_attestations": [{ "source_epoch": "2290", "target_epoch": "3007" }]
  }]
}

It helps to know exactly how much a slip costs, because it explains every paranoid habit below. On Ethereum mainnet a slashing carries an immediate penalty of up to ~1 ETH (1/32 of your effective balance), a forced trip through the exit queue, and — the part that turns a mistake into a catastrophe — a correlation penalty assessed about 18 days later that scales with how much other stake was slashed in the same window. Slash one validator alone and the correlation penalty is small. Slash a large fraction of the network at once and it approaches your entire balance. That single fact is why the rest of professional validator economics is obsessed with de-correlating failures.

Redundancy without double-signing

Here is the central tension of the whole job. Every instinct trained on web servers says: for high availability, run a hot spare and fail over to it. Apply that instinct to a validator client by cloning its keys onto a second box, and you have rebuilt the Staked disaster exactly — because the slashing-protection database is local, and two databases do not talk to each other. Each will happily sign, neither knowing what the other did. The naive failover is the attack.

1. One signer, many eyes. Run a single validator client, but connect it to two or three independent beacon (consensus) nodes. Beacon nodes never sign anything, so duplicating them is free resilience; if one falls behind or crashes, the client keeps attesting via another. Duplicating the signer is what kills you — duplicating its data sources does not.
2. Doppelganger detection. On (re)start, configure the client to listen for 2–3 epochs for attestations coming from its own public keys before it signs anything. If it hears itself on the network, another instance is already live — so it aborts instead of becoming the second signer. This is your seatbelt against a botched failover.
3. True fault tolerance via DVT. Distributed Validator Technology (Obol, SSV) splits one validator key into shares using distributed key generation, and a quorum of operators must cooperate to produce one threshold signature. The validator is a single logical signer that survives any one node dying — redundancy with no second key to collide with. This is the only way to get genuine high availability without courting a slash.

MEV-Boost: outsourcing the block you rarely get to build

With a million-plus validators, any one of yours gets to propose a block only about once every few months. But the value of that rare proposal is wildly uneven: most of the juice is maximal extractable value — arbitrage, liquidations, the occasional sandwich — and a validator naively building a block from its own mempool view captures almost none of it. MEV-Boost is the standard piece of software that closes that gap by implementing proposer-builder separation outside the protocol.

The flow is a careful commit-reveal. Specialized block builders compete to assemble the most valuable block. They submit those blocks, with bids, to a relay, which holds each block body in escrow and exposes only its header and bid. When it is your slot, your MEV-Boost picks the highest bid and your validator signs a blinded block header — committing to propose that block before it has seen the contents. Only then does the relay publish the full body. You never get to peek and run off with someone else's transactions, and if a relay misbehaves it can cost you a missed slot but cannot get you slashed.

# mev-boost: poll several relays; fall back to a LOCAL block if no bid clears min-bid
mev-boost -mainnet -relay-check -min-bid 0.05 \
  -relays \
    https://[email protected],\
    https://[email protected],\
    https://[email protected]

# Consensus + validator client point at the local mev-boost endpoint:
lighthouse bn --builder http://127.0.0.1:18550
lighthouse vc --builder-proposals \
               --suggested-fee-recipient 0xYourFeeAddress

The convenience is real but so are the trade-offs, and a professional names them. You are trusting the relay: it can fail to reveal a block (you eat a missed slot), and some relays are OFAC-compliant, filtering out sanctioned transactions — outsourcing your block to them quietly outsources your censorship policy too. Concentrating proposals through a few relays is also a centralization pressure on the network. The mitigations are baked into the config above: run several relays including censorship-resistant ones, and set a `min-bid` so that whenever the MEV on offer is trivial, your client just builds the block locally and stays neutral.

Monitoring, client diversity, and a runbook that won't bite

You cannot keep what you do not measure. A production validator is wired into Prometheus and Grafana with alerting that pages a human, watching a handful of metrics that turn invisible drift into a visible alarm: attestation inclusion distance and effectiveness (are your votes landing promptly?), missed attestations and missed proposals, the balance trend over time, beacon-node peer count, and whether the execution and consensus clients are synced and talking over the Engine API. A validator that silently stops attesting at 3 a.m. is just slowly bleeding rewards until someone notices.

There is one more defense that is invisible until catastrophe: client diversity. If a single consensus or execution client implementation runs on more than two-thirds of validators and ships a consensus bug, it could cause that supermajority to attest to a bad chain and get correlated-slashed together — the exact scenario the correlation penalty was designed to make ruinous. Serious staking infrastructure therefore deliberately runs minority clients, accepting a little extra operational friction in exchange for not being in the blast radius of a single project's bug.

Finally, write your failover as a runbook and rehearse it, because failover is the one routine operation that can slash you. The whole point is to guarantee that the old signer is dead and its history is carried forward before the new one ever signs:

1. Confirm the old host is genuinely dead, not merely unreachable — a node that looks down but later revives and signs is the classic way to slash yourself. If you cannot prove it is dead, treat it as alive.
2. Stop the validator client on the old host and disable its autostart, so a reboot or an orchestrator cannot bring it back signing behind your back.
3. Export the slashing-protection database (the EIP-3076 interchange) from the old host and import it into the new one, so the new client knows the highest slot and epochs already signed.
4. Start the new validator client with doppelganger detection enabled and let it watch the network for 2–3 epochs before it signs a single thing.
5. Only then promote it to primary in your alerting. Accept the handful of missed attestations during the gap as the cheap, correct price of never double-signing.

Put it all together and the professional validator operator looks less like a trader chasing yield and more like an airline mechanic: obsessive about the one mnemonic, religious about the slashing-protection database, paranoid about ever running two signers, and calm about a little downtime. With the keys guarded and the node humming, you have built the bedrock the next guide assembles into a full dApp stack — wallet, RPC, contract, and frontend — that ordinary users will trust without ever knowing your name.