Troubleshooting

Common Issues

Node Not Syncing

Symptoms:

• catching_up: true persists

• Block height not increasing

• No peers connected

Diagnosis:

# Check sync status
curl -s http://localhost:26657/status | jq '.result.sync_info'

# Check peers
curl -s http://localhost:26657/net_info | jq '.result.peers | length'

# View logs
sudo journalctl -u evmd -f --output cat | grep -i error

Solutions:

# 1. Check persistent peers configuration
grep persistent_peers $HOME/.evmd/config/config.toml

# 2. Add more peers
PEERS="<node_id>@<ip>:26656,<node_id2>@<ip2>:26656"
sed -i.bak "s/^persistent_peers *=.*/persistent_peers = \"$PEERS\"/" $HOME/.evmd/config/config.toml

# 3. Restart node
sudo systemctl restart evmd

# 4. If still not syncing, use state sync (faster)
# Get trust height and hash from a synced node
TRUST_HEIGHT=$(curl -s http://rpc-endpoint:26657/block | jq -r '.result.block.header.height')
TRUST_HEIGHT=$((TRUST_HEIGHT - 1000))  # Use height ~1000 blocks ago for safety
TRUST_HASH=$(curl -s http://rpc-endpoint:26657/block?height=$TRUST_HEIGHT | jq -r '.result.block_id.hash')

echo "Trust Height: $TRUST_HEIGHT"
echo "Trust Hash: $TRUST_HASH"

# Edit config.toml with these values
[statesync]
enable = true
rpc_servers = "http://rpc1.example.com:26657,http://rpc2.example.com:26657"
trust_height = $TRUST_HEIGHT
trust_hash = "$TRUST_HASH"
trust_period = "168h0m0s"  # 7 days (must be less than unbonding period of 21 days)

# Then restart the node
sudo systemctl restart evmd

Validator Jailed

Symptoms:

• Validator status shows jailed: true

• Not signing blocks

• Alerts triggered

Diagnosis:

# Note: Set VALCONS if not already set (see "Checking Validator Status" section)
# Check validator status
evmd query staking validator $(evmd keys show validator --bech val -a --keyring-backend file --home $HOME/.evmd) \
  --node http://localhost:26657 | jq '{jailed, status, tokens}'

# Check signing info
evmd query slashing signing-info $VALCONS \
  --node http://localhost:26657

Solutions:

# 1. Ensure node is fully synced
curl -s http://localhost:26657/status | jq '.result.sync_info.catching_up'
# Should return: false

# 2. Unjail validator
evmd tx slashing unjail \
  --from validator \
  --keyring-backend file \
  --home $HOME/.evmd \
  --node http://localhost:26657 \
  --chain-id 9001 \
  --gas auto \
  --gas-adjustment 1.5 \
  --gas-prices 10000000bplcoin \
  --yes

# 3. Verify unjailing
evmd query staking validator $(evmd keys show validator --bech val -a --keyring-backend file --home $HOME/.evmd) \
  --node http://localhost:26657 | jq '.jailed'

Genesis Compatibility Issues

Symptoms:

• Node fails to start with genesis validation errors

• Errors like invalid CreatorSplit: invalid decimal string

• Genesis parameter format incompatibilities

Diagnosis:

# Validate genesis file
evmd genesis validate-genesis --home $HOME/.evmd

# Check genesis hash
sha256sum $HOME/.evmd/config/genesis.json

# Compare with running nodes
curl -s http://rpc-endpoint:26657/genesis | jq -r '.result.genesis' | sha256sum

Solutions:

# 1. Verify you're using the correct evmd binary version
evmd version

# 2. Re-download genesis from a trusted source
curl -s http://rpc-endpoint:26657/genesis | jq -r '.result.genesis' > $HOME/.evmd/config/genesis.json

# 3. If genesis sync fails, use state sync instead
# See "Node Not Syncing" section above for state sync configuration

# 4. Ensure genesis file matches the network
GENESIS_HASH=$(sha256sum $HOME/.evmd/config/genesis.json | awk '{print $1}')
echo "Your genesis hash: $GENESIS_HASH"
echo "Compare with network's genesis hash"

Consensus Failure Recovery

Double Sign Prevention

If you suspect double signing:

# IMMEDIATELY stop the validator
sudo systemctl stop evmd

# Check if validator key is compromised
# Compare priv_validator_key.json across all your nodes
# THERE SHOULD ONLY BE ONE ACTIVE NODE WITH THIS KEY

# If running multiple nodes by mistake:
# 1. Stop ALL nodes
# 2. Identify the correct, synced node
# 3. Start ONLY that one node
# 4. Delete priv_validator_key.json from all other nodes

# Verify no double signing occurred
# Note: Set VALCONS if not already set (see "Checking Validator Status" section)
evmd query slashing signing-info $VALCONS \
  --node http://localhost:26657

State Corruption

Symptoms:

• Node crashes on startup

• Database errors in logs

• Inconsistent state errors

Recovery:

# 1. Stop node
sudo systemctl stop evmd

# 2. Reset tendermint state (keeps application state)
evmd tendermint unsafe-reset-all --home $HOME/.evmd

# 3. If corruption persists, restore from snapshot
rm -rf $HOME/.evmd/data
tar -xzf evmd-state-backup.tar.gz -C $HOME/.evmd/data

# 4. Restart node
sudo systemctl start evmd

# 5. Monitor sync
sudo journalctl -u evmd -f

Log Analysis

Useful Log Patterns

# View real-time logs
sudo journalctl -u evmd -f --output cat

# Filter for errors only
sudo journalctl -u evmd -f | grep -i error

# Check for specific issues
sudo journalctl -u evmd --since "1 hour ago" | grep -i "connection refused"

# View consensus logs
sudo journalctl -u evmd -f | grep "consensus"

# Check block production
sudo journalctl -u evmd -f | grep "committed state"

Next Steps

• Review setup procedures in Setup & Configuration

• Learn about daily operations in Operations

• Follow step-by-step procedures in Runbooks

• Access quick reference in Additional Resources