The Tendermint p2p protocol uses an authenticated encryption scheme based on the Station-to-Station Protocol. The implementation uses golang's nacl box for the actual authenticated encryption algorithm.
Each peer generates an ED25519 key-pair to use as a persistent (long-term) id.
When two peers establish a TCP connection, they first each generate an ephemeral ED25519 key-pair to use for this session, and send each other their respective ephemeral public keys. This happens in the clear.
They then each compute the shared secret. The shared secret is the multiplication of the peer's ephemeral private key by the other peer's ephemeral public key. The result is the same for both peers by the magic of elliptic curves. The shared secret is used as the symmetric key for the encryption algorithm.
The two ephemeral public keys are sorted to establish a canonical order. Then a 24-byte nonce is generated by concatenating the public keys and hashing them with Ripemd160. Note Ripemd160 produces 20byte hashes, so the nonce ends with four 0s.
The nonce is used to seed the encryption - it is critical that the same nonce never be used twice with the same private key. For convenience, the last bit of the nonce is flipped, giving us two nonces: one for encrypting our own messages, one for decrypting our peer's. Which ever peer has the higher public key uses the "bit-flipped" nonce for encryption.
Now, a challenge is generated by concatenating the ephemeral public keys and taking the SHA256 hash.
Each peer signs the challenge with their persistent private key, and sends the other peer an AuthSigMsg, containing their persistent public key and the signature. On receiving an AuthSigMsg, the peer verifies the signature.
The peers are now authenticated.
All future communications can now be encrypted using the shared secret and the generated nonces, where each nonce is incremented by one each time it is used. The communications maintain Perfect Forward Secrecy, as the persistent key pair was not used for generating secrets - only for authenticating.
This system is still vulnerable to a Man-In-The-Middle attack if the persistent public key of the remote node is not known in advance. The only way to mitigate this is with a public key authentication system, such as the Web-of-Trust or Certificate Authorities. In our case, we can use the blockchain itself as a certificate authority to ensure that we are connected to at least one validator.