|
|
- Corruption
- ==========
-
- Important step
- --------------
-
- Make sure you have a backup of the Tendermint data directory.
-
- Possible causes
- ---------------
-
- Remember that most corruption is caused by hardware issues:
-
- - RAID controllers with faulty / worn out battery backup, and an unexpected power loss
- - Hard disk drives with write-back cache enabled, and an unexpected power loss
- - Cheap SSDs with insufficient power-loss protection, and an unexpected power-loss
- - Defective RAM
- - Defective or overheating CPU(s)
-
- Other causes can be:
-
- - Database systems configured with fsync=off and an OS crash or power loss
- - Filesystems configured to use write barriers plus a storage layer that ignores write barriers. LVM is a particular culprit.
- - Tendermint bugs
- - Operating system bugs
- - Admin error
- - directly modifying Tendermint data-directory contents
-
- (Source: https://wiki.postgresql.org/wiki/Corruption)
-
- WAL Corruption
- --------------
-
- If consensus WAL is corrupted at the lastest height and you are trying to start
- Tendermint, replay will fail with panic.
-
- Recovering from data corruption can be hard and time-consuming. Here are two approaches you can take:
-
- 1) Delete the WAL file and restart Tendermint. It will attempt to sync with other peers.
- 2) Try to repair the WAL file manually:
-
- 1. Create a backup of the corrupted WAL file:
-
- .. code:: bash
-
- cp "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup
-
- 2. Use ./scripts/wal2json to create a human-readable version
-
- .. code:: bash
-
- ./scripts/wal2json/wal2json "$TMHOME/data/cs.wal/wal" > /tmp/corrupted_wal
-
- 3. Search for a "CORRUPTED MESSAGE" line.
- 4. By looking at the previous message and the message after the corrupted one
- and looking at the logs, try to rebuild the message. If the consequent
- messages are marked as corrupted too (this may happen if length header
- got corrupted or some writes did not make it to the WAL ~ truncation),
- then remove all the lines starting from the corrupted one and restart
- Tendermint.
-
- .. code:: bash
-
- $EDITOR /tmp/corrupted_wal
-
- 5. After editing, convert this file back into binary form by running:
-
- .. code:: bash
-
- ./scripts/json2wal/json2wal /tmp/corrupted_wal > "$TMHOME/data/cs.wal/wal"
|
