diff --git a/docs/abci-cli.rst b/docs/abci-cli.rst new file mode 100644 index 000000000..4dc1a7bd7 --- /dev/null +++ b/docs/abci-cli.rst @@ -0,0 +1,257 @@ +Using the abci-cli +================== + +To facilitate testing and debugging of ABCI servers and simple apps, we +built a CLI, the ``abci-cli``, for sending ABCI messages from the +command line. + +Install +------- + +Make sure you `have Go installed `__ and +`put ``$GOPATH/bin`` in your +``$PATH`` `__. + +Next, install the ``abci-cli`` tool and example applications: + +:: + + go get -u github.com/tendermint/abci/cmd/... + +If this fails, you may need to use ``glide`` to get vendored +dependencies: + +:: + + go get github.com/Masterminds/glide + cd $GOPATH/src/github.com/tendermint/abci + glide install + go install ./cmd/... + +Now run ``abci-cli --help`` to see the list of commands: + +:: + + COMMANDS: + batch Run a batch of ABCI commands against an application + console Start an interactive console for multiple commands + echo Have the application echo a message + info Get some info about the application + set_option Set an option on the application + deliver_tx Append a new tx to application + check_tx Validate a tx + commit Get application Merkle root hash + help, h Shows a list of commands or help for one command + + GLOBAL OPTIONS: + --address "tcp://127.0.0.1:46658" address of application socket + --help, -h show help + --version, -v print the version + +First Example - Dummy +--------------------- + +The ``abci-cli`` tool lets us send ABCI messages to our application, to +help build and debug them. + +The most important messages are ``deliver_tx``, ``check_tx``, and +``commit``, but there are others for convenience, configuration, and +information purposes. + +Let's start a dummy application, which was installed at the same time as +``abci-cli`` above. The dummy just stores transactions in a merkle tree: + +:: + + dummy + +In another terminal, run + +:: + + abci-cli echo hello + abci-cli info + +The application should echo ``hello`` and give you some information +about itself. + +An ABCI application must provide two things: + +- a socket server +- a handler for ABCI messages + +When we run the ``abci-cli`` tool we open a new connection to the +application's socket server, send the given ABCI message, and wait for a +response. + +The server may be generic for a particular language, and we provide a +`reference implementation in +Golang `__. See +the `list of other ABCI +implementations `__ for servers in +other languages. + +The handler is specific to the application, and may be arbitrary, so +long as it is deterministic and conforms to the ABCI interface +specification. + +So when we run ``abci-cli info``, we open a new connection to the ABCI +server, which calls the ``Info()`` method on the application, which +tells us the number of transactions in our Merkle tree. + +Now, since every command opens a new connection, we provide the +``abci-cli console`` and ``abci-cli batch`` commands, to allow multiple +ABCI messages to be sent over a single connection. + +Running ``abci-cli console`` should drop you in an interactive console +for speaking ABCI messages to your application. + +Try running these commands: + +:: + + > echo hello + -> data: hello + + > info + -> data: {"size":0} + + > commit + -> data: 0x + + > deliver_tx "abc" + -> code: OK + + > info + -> data: {"size":1} + + > commit + -> data: 0x750502FC7E84BBD788ED589624F06CFA871845D1 + + > query "abc" + -> code: OK + -> data: {"index":0,"value":"abc","exists":true} + + > deliver_tx "def=xyz" + -> code: OK + + > commit + -> data: 0x76393B8A182E450286B0694C629ECB51B286EFD5 + + > query "def" + -> code: OK + -> data: {"index":1,"value":"xyz","exists":true} + +Note that if we do ``deliver_tx "abc"`` it will store ``(abc, abc)``, +but if we do ``deliver_tx "abc=efg"`` it will store ``(abc, efg)``. + +Similarly, you could put the commands in a file and run +``abci-cli --verbose batch < myfile``. + +Another Example - Counter +------------------------- + +Now that we've got the hang of it, let's try another application, the +"counter" app. + +The counter app doesn't use a Merkle tree, it just counts how many times +we've sent a transaction, asked for a hash, or committed the state. The +result of ``commit`` is just the number of transactions sent. + +This application has two modes: ``serial=off`` and ``serial=on``. + +When ``serial=on``, transactions must be a big-endian encoded +incrementing integer, starting at 0. + +If ``serial=off``, there are no restrictions on transactions. + +We can toggle the value of ``serial`` using the ``set_option`` ABCI +message. + +When ``serial=on``, some transactions are invalid. In a live blockchain, +transactions collect in memory before they are committed into blocks. To +avoid wasting resources on invalid transactions, ABCI provides the +``check_tx`` message, which application developers can use to accept or +reject transactions, before they are stored in memory or gossipped to +other peers. + +In this instance of the counter app, ``check_tx`` only allows +transactions whose integer is greater than the last committed one. + +Let's kill the console and the dummy application, and start the counter +app: + +:: + + counter + +In another window, start the ``abci-cli console``: + +:: + + > set_option serial on + -> data: serial=on + + > check_tx 0x00 + -> code: OK + + > check_tx 0xff + -> code: OK + + > deliver_tx 0x00 + -> code: OK + + > check_tx 0x00 + -> code: BadNonce + -> log: Invalid nonce. Expected >= 1, got 0 + + > deliver_tx 0x01 + -> code: OK + + > deliver_tx 0x04 + -> code: BadNonce + -> log: Invalid nonce. Expected 2, got 4 + + > info + -> data: {"hashes":0,"txs":2} + +This is a very simple application, but between ``counter`` and +``dummy``, its easy to see how you can build out arbitrary application +states on top of the ABCI. `Hyperledger's +Burrow `__ also runs atop ABCI, +bringing with it Ethereum-like accounts, the Ethereum virtual-machine, +Monax's permissioning scheme, and native contracts extensions. + +But the ultimate flexibility comes from being able to write the +application easily in any language. + +We have implemented the counter in a number of languages (see the +example directory). + +To run the Node JS version, ``cd`` to ``example/js`` and run + +:: + + node app.js + +(you'll have to kill the other counter application process). In another +window, run the console and those previous ABCI commands. You should get +the same results as for the Go version. + +Want to write the counter app in your favorite language?! We'd be happy +to add you to our `ecosystem `__! +We're also offering `bounties `__ for +implementations in new languages! + +Notes +----- + +The ``abci-cli`` is designed strictly for testing and debugging. In a +real deployment, the role of sending messages is taken by Tendermint, +which connects to the app using three separate connections, each with +its own pattern of messages. + +For more information, see the `application developers +guide `__. For examples of running an ABCI +app with Tendermint, see the `introductory +guide `__. diff --git a/docs/app-architecture.rst b/docs/app-architecture.rst new file mode 100644 index 000000000..fe4fb517f --- /dev/null +++ b/docs/app-architecture.rst @@ -0,0 +1,128 @@ +Application Architecture Guide +============================== + +Overview +-------- + +A blockchain application is more than the consensus engine and the +transaction logic (eg. smart contracts, business logic) as implemented +in the ABCI app. There are also (mobile, web, desktop) clients that will +need to connect and make use of the app. We will assume for now that you +have a well designed transactions and database model, but maybe this +will be the topic of another article. This article is more interested in +various ways of setting up the "plumbing" and connecting these pieces, +and demonstrating some evolving best practices. + +Security +-------- + +A very important aspect when constructing a blockchain is security. The +consensus model can be DoSed (no consensus possible) by corrupting 1/3 +of the validators and exploited (writing arbitrary blocks) by corrupting +2/3 of the validators. So, while the security is not that of the +"weakest link", you should take care that the "average link" is +sufficiently hardened. + +One big attack surface on the validators is the communication between +the ABCI app and the tendermint core. This should be highly protected. +Ideally, the app and the core are running on the same machine, so no +external agent can target the communication channel. You can use unix +sockets (with permissions preventing access from other users), or even +compile the two apps into one binary if the ABCI app is also writen in +go. If you are unable to do that due to language support, then the ABCI +app should bind a TCP connection to localhost (127.0.0.1), which is less +efficient and secure, but still not reachable from outside. If you must +run the ABCI app and tendermint core on separate machines, make sure you +have a secure communication channel (ssh tunnel?) + +Now assuming, you have linked together your app and the core securely, +you must also make sure no one can get on the machine it is hosted on. +At this point it is basic network security. Run on a secure operating +system (SELinux?). Limit who has access to the machine (user accounts, +but also where the physical machine is hosted). Turn off all services +except for ssh, which should only be accessible by some well-guarded +public/private key pairs (no password). And maybe even firewall off +access to the ports used by the validators, so only known validators can +connect. + +There was also a suggestion on slack from @jhon about compiling +everything together with a unikernel for more security, such as +`Mirage `__ or +`UNIK `__. + +Connecting your client to the blockchain +---------------------------------------- + +Tendermint Core RPC +~~~~~~~~~~~~~~~~~~~ + +The concept is that the ABCI app is completely hidden from the outside +world and only communicated through a tested and secured `interface +exposed by the tendermint core `__. This interface +exposes a lot of data on the block header and consensus process, which +is quite useful for externally verifying the system. It also includes +3(!) methods to broadcast a transaction (propose it for the blockchain, +and possibly await a response). And one method to query app-specific +data from the ABCI application. + +Pros: \* Server code already written \* Access to block headers to +validate merkle proofs (nice for light clients) \* Basic read/write +functionality is supported + +Cons: \* Limited interface to app. All queries must be serialized into +[]byte (less expressive than JSON over HTTP) and there is no way to push +data from ABCI app to the client (eg. notify me if account X receives a +transaction) + +Custom ABCI server +~~~~~~~~~~~~~~~~~~ + +This was proposed by @wolfposd on slack and demonstrated by +`TMChat `__, a sample app. The +concept is to write a custom server for your app (with typical REST +API/websockets/etc for easy use by a mobile app). This custom server is +in the same binary as the ABCI app and data store, so can easily react +to complex events there that involve understanding the data format (send +a message if my balance drops below 500). All "writes" sent to this +server are proxied via websocket/JSON-RPC to tendermint core. When they +come back as deliver\_tx over ABCI, they will be written to the data +store. For "reads", we can do any queries we wish that are supported by +our architecture, using any web technology that is useful. The general +architecture is shown in the following diagram: + +Pros: \* Separates application logic from blockchain logic \* Allows +much richer, more flexible client-facing API \* Allows pub-sub, watching +certain fields, etc. + +Cons: \* Access to ABCI app can be dangerous (be VERY careful not to +write unless it comes from the validator node) \* No direct access to +the blockchain headers to verify tx \* You must write your own API (but +maybe that's a pro...) + +Hybrid solutions +~~~~~~~~~~~~~~~~ + +Likely the least secure but most versatile. The client can access both +the tendermint node for all blockchain info, as well as a custom app +server, for complex queries and pub-sub on the abci app. + +Pros: \* All from both above solutions + +Cons: \* Even more complexity \* Even more attack vectors (less +security) + +Scalability +----------- + +Read replica using non-validating nodes? They could forward transactions +to the validators (fewer connections, more security), and locally allow +all queries in any of the above configurations. Thus, while +transaction-processing speed is limited by the speed of the abci app and +the number of validators, one should be able to scale our read +performance to quite an extent (until the replication process drains too +many resources from the validator nodes). + +Example Code +------------ + +- `TMChat `__ diff --git a/docs/app-development.rst b/docs/app-development.rst new file mode 100644 index 000000000..8b41857bc --- /dev/null +++ b/docs/app-development.rst @@ -0,0 +1,299 @@ +Application Development Guide +============================= + +ABCI Design +----------- + +The purpose of ABCI is to provide a clean interface between state +transition machines on one computer and the mechanics of their +replication across multiple computers. The former we call 'application +logic' and the latter the 'consensus engine'. Application logic +validates transactions and optionally executes transactions against some +persistent state. A consensus engine ensures all transactions are +replicated in the same order on every machine. We call each machine in a +consensus engine a 'validator', and each validator runs the same +transactions through the same application logic. In particular, we are +interested in blockchain-style consensus engines, where transactions are +committed in hash-linked blocks. + +The ABCI design has a few distinct components: + +- message protocol + + - pairs of request and response messages + - consensus makes requests, application responds + - defined using protobuf + +- server/client + + - consensus engine runs the client + - application runs the server + - two implementations: + + - async raw bytes + - grpc + +- blockchain protocol + + - abci is connection oriented + - Tendermint Core maintains three connections: + + - `mempool connection <#mempool-connection>`__: for checking if + transactions should be relayed before they are committed; only + uses ``CheckTx`` + - `consensus connection <#consensus-connection>`__: for executing + transactions that have been committed. Message sequence is - + for every block - + ``BeginBlock, [DeliverTx, ...], EndBlock, Commit`` + - `query connection <#query-connection>`__: for querying the + application state; only uses Query and Info + +The mempool and consensus logic act as clients, and each maintains an +open ABCI connection with the application, which hosts an ABCI server. +Shown are the request and response types sent on each connection. + +Message Protocol +---------------- + +The message protocol consists of pairs of requests and responses. Some +messages have no fields, while others may include byte-arrays, strings, +or integers. See the ``message Request`` and ``message Response`` +definitions in `the protobuf definition +file `__, +and the `protobuf +documentation `__ +for more details. + +For each request, a server should respond with the corresponding +response, where order of requests is preserved in the order of +responses. + +Server +------ + +To use ABCI in your programming language of choice, there must be a ABCI +server in that language. Tendermint supports two kinds of implementation +of the server: + +- Asynchronous, raw socket server (Tendermint Socket Protocol, also + known as TSP or Teaspoon) +- GRPC + +Both can be tested using the ``abci-cli`` by setting the ``--abci`` flag +appropriately (ie. to ``socket`` or ``grpc``). + +See examples, in various stages of maintenance, in +`Go `__, +`JavaScript `__, +`Python `__, +`C++ `__, and +`Java `__. + +GRPC +~~~~ + +If GRPC is available in your language, this is the easiest approach, +though it will have significant performance overhead. + +To get started with GRPC, copy in the `protobuf +file `__ +and compile it using the GRPC plugin for your language. For instance, +for golang, the command is +``protoc --go_out=plugins=grpc:. types.proto``. See the `grpc +documentation for more details `__. ``protoc`` +will autogenerate all the necessary code for ABCI client and server in +your language, including whatever interface your application must +satisfy to be used by the ABCI server for handling requests. + +TSP +~~~ + +If GRPC is not available in your language, or you require higher +performance, or otherwise enjoy programming, you may implement your own +ABCI server using the Tendermint Socket Protocol, known affectionaltely +as Teaspoon. The first step is still to auto-generate the relevant data +types and codec in your language using ``protoc``. Messages coming over +the socket are Protobuf3 encoded, but additionally length-prefixed to +facilitate use as a streaming protocol. Protobuf3 doesn't have an +official length-prefix standard, so we use our own. The first byte in +the prefix represents the length of the Big Endian encoded length. The +remaining bytes in the prefix are the Big Endian encoded length. + +For example, if the Protobuf3 encoded ABCI message is 0xDEADBEEF (4 +bytes), the length-prefixed message is 0x0104DEADBEEF. If the Protobuf3 +encoded ABCI message is 65535 bytes long, the length-prefixed message +would be like 0x02FFFF.... + +Note this prefixing does not apply for grpc. + +An ABCI server must also be able to support multiple connections, as +Tendermint uses three connections. + +Client +------ + +There are currently two use-cases for an ABCI client. One is a testing +tool, as in the ``abci-cli``, which allows ABCI requests to be sent via +command line. The other is a consensus engine, such as Tendermint Core, +which makes requests to the application every time a new transaction is +received or a block is committed. + +It is unlikely that you will need to implement a client. For details of +our client, see +`here `__. + +Blockchain Protocol +------------------- + +In ABCI, a transaction is simply an arbitrary length byte-array. It is +the application's responsibility to define the transaction codec as they +please, and to use it for both CheckTx and DeliverTx. + +Note that there are two distinct means for running transactions, +corresponding to stages of 'awareness' of the transaction in the +network. The first stage is when a transaction is received by a +validator from a client into the so-called mempool or transaction pool - +this is where we use CheckTx. The second is when the transaction is +successfully committed on more than 2/3 of validators - where we use +DeliverTx. In the former case, it may not be necessary to run all the +state transitions associated with the transaction, as the transaction +may not ultimately be committed until some much later time, when the +result of its execution will be different. For instance, an Ethereum +ABCI app would check signatures and amounts in CheckTx, but would not +actually execute any contract code until the DeliverTx, so as to avoid +executing state transitions that have not been finalized. + +To formalize the distinction further, two explicit ABCI connections are +made between Tendermint Core and the application: the mempool connection +and the consensus connection. We also make a third connection, the query +connection, to query the local state of the app. + +Mempool Connection +~~~~~~~~~~~~~~~~~~ + +The mempool connection is used *only* for CheckTx requests. Transactions +are run using CheckTx in the same order they were received by the +validator. If the CheckTx returns ``OK``, the transaction is kept in +memory and relayed to other peers in the same order it was received. +Otherwise, it is discarded. + +CheckTx requests run concurrently with block processing; so they should +run against a copy of the main application state which is reset after +every block. This copy is necessary to track transitions made by a +sequence of CheckTx requests before they are included in a block. When a +block is committed, the application must ensure to reset the mempool +state to the latest committed state. Tendermint Core will then filter +through all transactions in the mempool, removing any that were included +in the block, and re-run the rest using CheckTx against the post-Commit +mempool state. + +Consensus Connection +~~~~~~~~~~~~~~~~~~~~ + +The consensus connection is used only when a new block is committed, and +communicates all information from the block in a series of requests: +``BeginBlock, [DeliverTx, ...], EndBlock, Commit``. That is, when a +block is committed in the consensus, we send a list of DeliverTx +requests (one for each transaction) sandwiched by BeginBlock and +EndBlock requests, and followed by a Commit. + +DeliverTx +^^^^^^^^^ + +DeliverTx is the workhorse of the blockchain. Tendermint sends the +DeliverTx requests asynchronously but in order, and relies on the +underlying socket protocol (ie. TCP) to ensure they are received by the +app in order. They have already been ordered in the global consensus by +the Tendermint protocol. + +DeliverTx returns a abci.Result, which includes a Code, Data, and Log. +The code may be non-zero (non-OK), meaning the corresponding transaction +should have been rejected by the mempool, but may have been included in +a block by a Byzantine proposer. + +The block header will be updated (TODO) to include some commitment to +the results of DeliverTx, be it a bitarray of non-OK transactions, or a +merkle root of the data returned by the DeliverTx requests, or both. + +Commit +^^^^^^ + +Once all processing of the block is complete, Tendermint sends the +Commit request and blocks waiting for a response. While the mempool may +run concurrently with block processing (the BeginBlock, DeliverTxs, and +EndBlock), it is locked for the Commit request so that its state can be +safely reset during Commit. This means the app *MUST NOT* do any +blocking communication with the mempool (ie. broadcast\_tx) during +Commit, or there will be deadlock. Note also that all remaining +transactions in the mempool are replayed on the mempool connection +(CheckTx) following a commit. + +The Commit response includes a byte array, which is the deterministic +state root of the application. It is included in the header of the next +block. It can be used to provide easily verified Merkle-proofs of the +state of the application. + +It is expected that the app will persist state to disk on Commit. The +option to have all transactions replayed from some previous block is the +job of the `Handshake <#handshake>`__. + +BeginBlock +^^^^^^^^^^ + +The BeginBlock request can be used to run some code at the beginning of +every block. It also allows Tendermint to send the current block hash +and header to the application, before it sends any of the transactions. + +The app should remember the latest height and header (ie. from which it +has run a successful Commit) so that it can tell Tendermint where to +pick up from when it restarts. See information on the Handshake, below. + +EndBlock +^^^^^^^^ + +The EndBlock request can be used to run some code at the end of every +block. Additionally, the response may contain a list of validators, +which can be used to update the validator set. To add a new validator or +update an existing one, simply include them in the list returned in the +EndBlock response. To remove one, include it in the list with a +``power`` equal to ``0``. Tendermint core will take care of updating the +validator set. Note validator set changes are only available in v0.8.0 +and up. + +Query Connection +~~~~~~~~~~~~~~~~ + +This connection is used to query the application without engaging +consensus. It's exposed over the tendermint core rpc, so clients can +query the app without exposing a server on the app itself, but they must +serialize each query as a single byte array. Additionally, certain +"standardized" queries may be used to inform local decisions, for +instance about which peers to connect to. + +Tendermint Core currently uses the Query connection to filter peers upon +connecting, according to IP address or public key. For instance, +returning non-OK ABCI response to either of the following queries will +cause Tendermint to not connect to the corresponding peer: + +- ``p2p/filter/addr/``, where ```` is an IP address. +- ``p2p/filter/pubkey/``, where ```` is the hex-encoded + ED25519 key of the node (not it's validator key) + +Note: these query formats are subject to change! + +Handshake +~~~~~~~~~ + +When the app or tendermint restarts, they need to sync to a common +height. When an ABCI connection is first established, Tendermint will +call ``Info`` on the Query connection. The response should contain the +LastBlockHeight and LastBlockAppHash - the former is the last block for +the which the app ran Commit successfully, the latter is the response +from that Commit. + +Using this information, Tendermint will determine what needs to be +replayed, if anything, against the app, to ensure both Tendermint and +the app are synced to the latest block height. + +If the app returns a LastBlockHeight of 0, Tendermint will just replay +all blocks. diff --git a/docs/deploy-testnets.rst b/docs/deploy-testnets.rst new file mode 100644 index 000000000..e74889700 --- /dev/null +++ b/docs/deploy-testnets.rst @@ -0,0 +1,57 @@ +Deploy a Testnet +================ + +Now that we've seen how ABCI works, and even played with a few +applications on a single validator node, it's time to deploy a test +network to four validator nodes. For this deployment, we'll use the +``basecoin`` application. + +Manual Deployments +------------------ + +It's relatively easy to setup a Tendermint cluster manually. The only +requirements for a particular Tendermint node are a private key for the +validator, stored as ``priv_validator.json``, and a list of the public +keys of all validators, stored as ``genesis.json``. These files should +be stored in ``~/.tendermint``, or wherever the ``$TMROOT`` variable +might be set to. + +Here are the steps to setting up a testnet manually: + +1) Provision nodes on your cloud provider of choice +2) Install Tendermint and the application of interest on all nodes +3) Generate a private key for each validator using + ``tendermint gen_validator`` +4) Compile a list of public keys for each validator into a + ``genesis.json`` file. +5) Run ``tendermint node --p2p.seeds=< seed addresses >`` on each node, + where ``< seed addresses >`` is a comma separated list of the IP:PORT + combination for each node. The default port for Tendermint is + ``46656``. Thus, if the IP addresses of your nodes were + ``192.168.0.1, 192.168.0.2, 192.168.0.3, 192.168.0.4``, the command + would look like: + ``tendermint node --p2p.seeds=192.168.0.1:46656,192.168.0.2:46656,192.168.0.3:46656,192.168.0.4:46656``. + +After a few seconds, all the nodes should connect to eachother and start +making blocks! For more information, see the Tendermint Networks section +of `the guide to using Tendermint `__. + +Automated Deployments +--------------------- + +While the manual deployment is easy enough, an automated deployment is +always better. For this, we have the `mintnet-kubernetes +tool `__, +which allows us to automate the deployment of a Tendermint network on an +already provisioned kubernetes cluster. + +For more details, see the `mintnet-kubernetes +directory `__, +and check out `Google Cloud Platform `__ for +simple provisioning of kubernetes clusters. + +Next Steps +---------- + +Done trying out the testnet? Continue +`onwards `__. diff --git a/docs/getting-started.rst b/docs/getting-started.rst new file mode 100644 index 000000000..b8d7477e5 --- /dev/null +++ b/docs/getting-started.rst @@ -0,0 +1,291 @@ +First Tendermint App +==================== + +As a general purpose blockchain engine, Tendermint is agnostic to the +application you want to run. So, to run a complete blockchain that does +something useful, you must start two programs: one is Tendermint Core, +the other is your application, which can be written in any programming +language. Recall from `the intro to ABCI `__ that +Tendermint Core handles all the p2p and consensus stuff, and just +forwards transactions to the application when they need to be validated, +or when they're ready to be committed to a block. + +In this guide, we show you some examples of how to run an application +using Tendermint. + +**Note:** It is highly recommended to read the `Using Tendermint +Guide <./using-tendermint>`__ prior to working through this +tutorial. + +Install +------- + +First, make sure you have `installed Tendermint `__. The +first apps we will work with are written in Go. To install them, you +need to `install Go `__ and `put +``$GOPATH/bin`` in your +``$PATH`` `__. + +Then run + +:: + + go get -u github.com/tendermint/abci/cmd/... + +If there is an error, install and run the ``glide`` tool to pin the +dependencies: + +:: + + go get github.com/Masterminds/glide + cd $GOPATH/src/github.com/tendermint/abci + glide install + go install ./cmd/... + +Now you should have the ``abci-cli`` plus two apps installed: + +:: + + dummy --help + counter --help + +These binaries are installed on ``$GOPATH/bin`` and all come from within +the ``./cmd/...`` directory of the abci repository. + +Both of these example applications are in Go. See below for an +application written in Javascript. + +Now, let's run some apps! + +A First Example - Dummy +----------------------- + +The dummy app is a `Merkle +tree `__ that just stores all +transactions. If the transaction contains an ``=``, eg. ``key=value``, +then the ``value`` is stored under the ``key`` in the Merkle tree. +Otherwise, the full transaction bytes are stored as the key and the +value. + +Let's start a dummy application. + +:: + + dummy + +In another terminal, we can start Tendermint. If you have never run +Tendermint before, use: + +:: + + tendermint init + tendermint node + +If you have used Tendermint, you may want to reset the data for a new +blockchain by running ``tendermint unsafe_reset_all``. Then you can run +``tendermint node`` to start Tendermint, and connect to the app. For +more details, see `the guide on using +Tendermint `__. + +You should see Tendermint making blocks! We can get the status of our +Tendermint node as follows: + +:: + + curl -s localhost:46657/status + +The ``-s`` just silences ``curl``. For nicer output, pipe the result +into a tool like `jq `__ or +`jsonpp `__. + +Now let's send some transactions to the dummy. + +:: + + curl -s 'localhost:46657/broadcast_tx_commit?tx="abcd"' + +Note the single quote (``'``) around the url, which ensures that the +double quotes (``"``) are not escaped by bash. This command sent a +transaction with bytes ``abcd``, so ``abcd`` will be stored as both the +key and the value in the Merkle tree. The response should look something +like: + +:: + + {"jsonrpc":"2.0","id":"","result":[98,{"check_tx":{},"deliver_tx":{}}],"error":""} + +The ``98`` is a type-byte, and can be ignored (it's useful for +serializing and deserializing arbitrary json). Otherwise, this result is +empty - there's nothing to report on and everything is OK. + +We can confirm that our transaction worked and the value got stored by +querying the app: + +:: + + curl -s 'localhost:46657/abci_query?data="abcd"&path=""&prove=false' + +The ``path`` and ``prove`` arguments can be ignored for now, and in a +future release can be left out. The result should look like: + +:: + + {"jsonrpc":"2.0","id":"","result":[112,{"response":{"value":"61626364","log":"exists"}}],"error":""} + +Again, the ``112`` is the type-byte. Note the ``value`` in the result +(``61626364``); this is the hex-encoding of the ASCII of ``abcd``. You +can verify this in a python shell by running +``"61626364".decode('hex')``. Stay tuned for a future release that makes +this output more human-readable ;). + +Now let's try setting a different key and value: + +:: + + curl -s 'localhost:46657/broadcast_tx_commit?tx="name=satoshi"' + +Now if we query for ``name``, we should get ``satoshi``, or +``7361746F736869`` in hex: + +:: + + curl -s 'localhost:46657/abci_query?data="name"&path=""&prove=false' + +Try some other transactions and queries to make sure everything is +working! + +Another Example - Counter +------------------------- + +Now that we've got the hang of it, let's try another application, the +"counter" app. + +The counter app doesn't use a Merkle tree, it just counts how many times +we've sent a transaction, or committed the state. + +This application has two modes: ``serial=off`` and ``serial=on``. + +When ``serial=on``, transactions must be a big-endian encoded +incrementing integer, starting at 0. + +If ``serial=off``, there are no restrictions on transactions. + +In a live blockchain, transactions collect in memory before they are +committed into blocks. To avoid wasting resources on invalid +transactions, ABCI provides the ``CheckTx`` message, which application +developers can use to accept or reject transactions, before they are +stored in memory or gossipped to other peers. + +In this instance of the counter app, with ``serial=on``, ``CheckTx`` +only allows transactions whose integer is greater than the last +committed one. + +Let's kill the previous instance of ``tendermint`` and the ``dummy`` +application, and start the counter app. We can enable ``serial=on`` with +a flag: + +:: + + counter --serial + +In another window, reset then start Tendermint: + +:: + + tendermint unsafe_reset_all + tendermint node + +Once again, you can see the blocks streaming by. Let's send some +transactions. Since we have set ``serial=on``, the first transaction +must be the number ``0``: + +:: + + curl localhost:46657/broadcast_tx_commit?tx=0x00 + +Note the empty (hence successful) response. The next transaction must be +the number ``1``. If instead, we try to send a ``5``, we get an error: + +:: + + > curl localhost:46657/broadcast_tx_commit?tx=0x05 + {"jsonrpc":"2.0","id":"","result":[98,{"check_tx":{},"deliver_tx":{"code":3,"log":"Invalid nonce. Expected 1, got 5"}}],"error":""} + +But if we send a ``1``, it works again: + +:: + + > curl localhost:46657/broadcast_tx_commit?tx=0x01 + {"jsonrpc":"2.0","id":"","result":[98,{"check_tx":{},"deliver_tx":{}}],"error":""} + +For more details on the ``broadcast_tx`` API, see `the guide on using +Tendermint `__. + +Example in Another Language - CounterJS +--------------------------------------- + +We also want to run applications in another language - in this case, +we'll run a Javascript version of the ``counter``. To run it, you'll +need to `install node `__. + +You'll also need to fetch the relevant repository, from +https://github.com/tendermint/js-abci then install it. As go devs, we +keep all our code under the ``$GOPATH``, so run: + +:: + + go get github.com/tendermint/js-abci &> /dev/null + cd $GOPATH/src/github.com/tendermint/js-abci/example + npm install + +Kill the previous ``counter`` and ``tendermint`` processes. Now run the +app: + +:: + + node example/app.js + +In another window, reset and start ``tendermint``: + +:: + + tendermint unsafe_reset_all + tendermint node + +Once again, you should see blocks streaming by - but now, our +application is written in javascript! Try sending some transactions, and +like before - the results should be the same: + +:: + + curl localhost:46657/broadcast_tx_commit?tx=0x00 # ok + curl localhost:46657/broadcast_tx_commit?tx=0x05 # invalid nonce + curl localhost:46657/broadcast_tx_commit?tx=0x01 # ok + +Neat, eh? + +A More Interesting Example - Basecoin +------------------------------------- + +Before concluding, we'd like to introduce you to our star application, +`Basecoin `__. Unlike the +``dummy`` and ``counter``, which are strictly for example purposes, +``basecoin`` is designed to be actually useful - it's a general purpose +framework for building cryptocurrencies. + +The default ``basecoin`` application is a multi-asset cryptocurrency +that supports inter-blockchain communication. For more details on how +basecoin works and how to use it, see our `basecoin +guide `__ + +Next Step +--------- + +In this tutorial you learned how to run applications using Tendermint on +a single node. You saw how applications could be written in different +languages, and how to send transactions and query for the latest state. +But the true power of Tendermint comes from its ability to securely and +efficiently run an application across a distributed network of nodes, +while keeping them all in sync using its state-of-the-art consensus +protocol. This is the subject of the next tutorial, where we show you +`how to deploy Tendermint networks `__. diff --git a/docs/getting-started/#02-first-app.md b/docs/getting-started/#02-first-app.md deleted file mode 100644 index fdc05f582..000000000 --- a/docs/getting-started/#02-first-app.md +++ /dev/null @@ -1,246 +0,0 @@ -# First Tendermint App - -As a general purpose blockchain engine, Tendermint is agnostic to the application you want to run. -So, to run a complete blockchain that does something useful, you must start two programs: -one is Tendermint Core, the other is your application, which can be written in any programming language. -Recall from [the intro to ABCI](/intro/abci-overview) that Tendermint Core handles all the p2p and consensus stuff, -and just forwards transactions to the application when they need to be validated, or when they're ready to be committed to a block. - -In this guide, we show you some examples of how to run an application using Tendermint. - -**Note:** It is highly recommended to read the [Using Tendermint Guide](/docs/guides/using-tendermint) prior to working through this tutorial. - -## Install - -First, make sure you have [installed Tendermint](/download). -The first apps we will work with are written in Go. -To install them, you need to [install Go](https://golang.org/doc/install) and -[put `$GOPATH/bin` in your `$PATH`](https://github.com/tendermint/tendermint/wiki/Setting-GOPATH). - -Then run - -``` -go get -u github.com/tendermint/abci/cmd/... -``` - -If there is an error, install and run the `glide` tool to pin the dependencies: - -``` -go get github.com/Masterminds/glide -cd $GOPATH/src/github.com/tendermint/abci -glide install -go install ./cmd/... -``` - -Now you should have the `abci-cli` plus two apps installed: - -``` -dummy --help -counter --help -``` - -These binaries are installed on `$GOPATH/bin` and all come from within the `./cmd/...` directory of the abci repository. - -Both of these example applications are in Go. See below for an application written in Javascript. - -Now, let's run some apps! - -## A First Example - Dummy - -The dummy app is a [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree) that just stores all transactions. -If the transaction contains an `=`, eg. `key=value`, -then the `value` is stored under the `key` in the Merkle tree. -Otherwise, the full transaction bytes are stored as the key and the value. - -Let's start a dummy application. - -``` -dummy -``` - -In another terminal, we can start Tendermint. -If you have never run Tendermint before, use: - -``` -tendermint init -tendermint node -``` - -If you have used Tendermint, you may want to reset the data for a new blockchain by running `tendermint unsafe_reset_all`. -Then you can run `tendermint node` to start Tendermint, and connect to the app. -For more details, see [the guide on using Tendermint](/docs/guides/using-tendermint). - -You should see Tendermint making blocks! -We can get the status of our Tendermint node as follows: - -``` -curl -s localhost:46657/status -``` - -The `-s` just silences `curl`. For nicer output, pipe the result into a tool like [jq](https://stedolan.github.io/jq/) -or [jsonpp](https://github.com/jmhodges/jsonpp). - -Now let's send some transactions to the dummy. - -``` -curl -s 'localhost:46657/broadcast_tx_commit?tx="abcd"' -``` - -Note the single quote (`'`) around the url, which ensures that the double quotes (`"`) are not escaped by bash. -This command sent a transaction with bytes `abcd`, so `abcd` will be stored as both the key and the value in the Merkle tree. -The response should look something like: - -``` -{"jsonrpc":"2.0","id":"","result":[98,{"check_tx":{},"deliver_tx":{}}],"error":""} -``` - -The `98` is a type-byte, and can be ignored (it's useful for serializing and deserializing arbitrary json). -Otherwise, this result is empty - there's nothing to report on and everything is OK. - -We can confirm that our transaction worked and the value got stored by querying the app: - -``` -curl -s 'localhost:46657/abci_query?data="abcd"&path=""&prove=false' -``` - -The `path` and `prove` arguments can be ignored for now, and in a future release can be left out. -The result should look like: - - -``` -{"jsonrpc":"2.0","id":"","result":[112,{"response":{"value":"61626364","log":"exists"}}],"error":""} -``` - -Again, the `112` is the type-byte. Note the `value` in the result (`61626364`); this is the hex-encoding of the ASCII of `abcd`. -You can verify this in a python shell by running `"61626364".decode('hex')`. -Stay tuned for a future release that makes this output more human-readable ;). - -Now let's try setting a different key and value: - -``` -curl -s 'localhost:46657/broadcast_tx_commit?tx="name=satoshi"' -``` - -Now if we query for `name`, we should get `satoshi`, or `7361746F736869` in hex: - -``` -curl -s 'localhost:46657/abci_query?data="name"&path=""&prove=false' -``` - -Try some other transactions and queries to make sure everything is working! - -## Another Example - Counter - -Now that we've got the hang of it, let's try another application, the "counter" app. - -The counter app doesn't use a Merkle tree, it just counts how many times we've sent a transaction, -or committed the state. - -This application has two modes: `serial=off` and `serial=on`. - -When `serial=on`, transactions must be a big-endian encoded incrementing integer, starting at 0. - -If `serial=off`, there are no restrictions on transactions. - -In a live blockchain, transactions collect in memory before they are committed into blocks. -To avoid wasting resources on invalid transactions, -ABCI provides the `CheckTx` message, -which application developers can use to accept or reject transactions, -before they are stored in memory or gossipped to other peers. - -In this instance of the counter app, with `serial=on`, `CheckTx` only allows transactions whose integer is greater than the last committed one. - -Let's kill the previous instance of `tendermint` and the `dummy` application, and start the counter app. -We can enable `serial=on` with a flag: - -``` -counter --serial -``` - -In another window, reset then start Tendermint: - -``` -tendermint unsafe_reset_all -tendermint node -``` - -Once again, you can see the blocks streaming by. Let's send some transactions. -Since we have set `serial=on`, the first transaction must be the number `0`: - -``` -curl localhost:46657/broadcast_tx_commit?tx=0x00 -``` - -Note the empty (hence successful) response. -The next transaction must be the number `1`. If instead, we try to send a `5`, we get an error: - -``` -> curl localhost:46657/broadcast_tx_commit?tx=0x05 -{"jsonrpc":"2.0","id":"","result":[98,{"check_tx":{},"deliver_tx":{"code":3,"log":"Invalid nonce. Expected 1, got 5"}}],"error":""} -``` - -But if we send a `1`, it works again: - -``` -> curl localhost:46657/broadcast_tx_commit?tx=0x01 -{"jsonrpc":"2.0","id":"","result":[98,{"check_tx":{},"deliver_tx":{}}],"error":""} -``` - -For more details on the `broadcast_tx` API, -see [the guide on using Tendermint](/docs/guides/using-tendermint). - -## Example in Another Language - CounterJS - -We also want to run applications in another language - in this case, we'll run a Javascript version of the `counter`. -To run it, you'll need to [install node](https://nodejs.org/en/download/). - -You'll also need to fetch the relevant repository, from https://github.com/tendermint/js-abci then install it. -As go devs, we keep all our code under the `$GOPATH`, so run: - -``` -go get github.com/tendermint/js-abci &> /dev/null -cd $GOPATH/src/github.com/tendermint/js-abci/example -npm install -``` - -Kill the previous `counter` and `tendermint` processes. Now run the app: - -``` -node example/app.js -``` - -In another window, reset and start `tendermint`: - -``` -tendermint unsafe_reset_all -tendermint node -``` - -Once again, you should see blocks streaming by - but now, our application is written in javascript! -Try sending some transactions, and like before - the results should be the same: - -``` -curl localhost:46657/broadcast_tx_commit?tx=0x00 # ok -curl localhost:46657/broadcast_tx_commit?tx=0x05 # invalid nonce -curl localhost:46657/broadcast_tx_commit?tx=0x01 # ok -``` - -Neat, eh? - -## A More Interesting Example - Basecoin - -Before concluding, we'd like to introduce you to our star application, [Basecoin](https://github.com/tendermint/basecoin). -Unlike the `dummy` and `counter`, which are strictly for example purposes, -`basecoin` is designed to be actually useful - it's a general purpose framework for building cryptocurrencies. - -The default `basecoin` application is a multi-asset cryptocurrency that supports inter-blockchain communication. -For more details on how basecoin works and how to use it, see our [basecoin guide](https://github.com/tendermint/basecoin/blob/develop/docs/guide/basecoin-basics.md) - -## Next Step - -In this tutorial you learned how to run applications using Tendermint on a single node. -You saw how applications could be written in different languages, -and how to send transactions and query for the latest state. -But the true power of Tendermint comes from its ability to securely and efficiently run an application -across a distributed network of nodes, while keeping them all in sync using its state-of-the-art consensus protocol. -This is the subject of the next tutorial, where we show you [how to deploy Tendermint networks](/docs/getting-started/deploy-testnet). diff --git a/docs/getting-started/#03-deploy-testnet.md b/docs/getting-started/#03-deploy-testnet.md deleted file mode 100644 index 6f1caa6fa..000000000 --- a/docs/getting-started/#03-deploy-testnet.md +++ /dev/null @@ -1,39 +0,0 @@ -# Deploy a Testnet - -Now that we've seen how ABCI works, and even played with a few applications on a single validator node, -it's time to deploy a test network to four validator nodes. -For this deployment, we'll use the `basecoin` application. - -## Manual Deployments - -It's relatively easy to setup a Tendermint cluster manually. -The only requirements for a particular Tendermint node are a private key for the validator, -stored as `priv_validator.json`, and a list of the public keys of all validators, stored as `genesis.json`. -These files should be stored in `~/.tendermint`, or wherever the `$TMROOT` variable might be set to. - -Here are the steps to setting up a testnet manually: - -1) Provision nodes on your cloud provider of choice -2) Install Tendermint and the application of interest on all nodes -3) Generate a private key for each validator using `tendermint gen_validator` -4) Compile a list of public keys for each validator into a `genesis.json` file. -5) Run `tendermint node --p2p.seeds=< seed addresses >` on each node, where `< seed addresses >` is a -comma separated list of the IP:PORT combination for each node. The default port for Tendermint is `46656`. -Thus, if the IP addresses of your nodes were `192.168.0.1, 192.168.0.2, 192.168.0.3, 192.168.0.4`, -the command would look like: `tendermint node --p2p.seeds=192.168.0.1:46656,192.168.0.2:46656,192.168.0.3:46656,192.168.0.4:46656`. - -After a few seconds, all the nodes should connect to eachother and start making blocks! -For more information, see the Tendermint Networks section of [the guide to using Tendermint](/docs/guides/using-tendermint). - -## Automated Deployments - -While the manual deployment is easy enough, an automated deployment is always better. -For this, we have the [mintnet-kubernetes tool](https://github.com/tendermint/tools/tree/master/mintnet-kubernetes), -which allows us to automate the deployment of a Tendermint network on an already provisioned kubernetes cluster. - -For more details, see the [mintnet-kubernetes directory](https://github.com/tendermint/tools/tree/master/mintnet-kubernetes), -and check out [Google Cloud Platform](https://cloud.google.com/) for simple provisioning of kubernetes clusters. - -## Next Steps - -Done trying out the testnet? Continue [onwards](/docs/getting-started/next-steps). diff --git a/docs/getting-started/#04-next-steps.md b/docs/getting-started/#04-next-steps.md deleted file mode 100644 index e346ceeb4..000000000 --- a/docs/getting-started/#04-next-steps.md +++ /dev/null @@ -1,21 +0,0 @@ -# Next Steps - -By now you've seen how to run a simple example ABCI application on a local Tendermint node -and on a remote Tendermint cluster. - -To learn more about building ABCI applications and integrating with Tendermint, see the [Developer Guides](/docs/guides/app-development). -To learn more about running the Tendermint software, see the [Using Tendermint Guide](/docs/guides/using-tendermint). - -To learn more about Tendermint's various pieces, check out the [Documentation](/docs). -For a deeper dive, see [this thesis](https://atrium.lib.uoguelph.ca/xmlui/handle/10214/9769). -There is also the [original whitepaper](/static/docs/tendermint.pdf), though it is now quite outdated. - -The Tendermint [Software Ecosystem](/ecosystem) contains many example applications and related software built by the Tendermint team and others. Check it out for some inspiration! - -For details on how the software has changed, and what changes are in store, see the [Changelog](/docs/changelog) and the [Roadmap](/docs/roadmap). - -See our [Community](/community) page for more ways to collaborate. - -You can also [get in touch with the team](/contact). - -Most importantly, enjoy! diff --git a/docs/guides/abci-cli.md b/docs/guides/abci-cli.md deleted file mode 100644 index 39b105f51..000000000 --- a/docs/guides/abci-cli.md +++ /dev/null @@ -1,219 +0,0 @@ -# Using the abci-cli - -To facilitate testing and debugging of ABCI servers and simple apps, -we built a CLI, the `abci-cli`, for sending ABCI messages from the command line. - -## Install - -Make sure you [have Go installed](https://golang.org/doc/install) and [put `$GOPATH/bin` in your `$PATH`](https://github.com/tendermint/tendermint/wiki/Setting-GOPATH). - -Next, install the `abci-cli` tool and example applications: - -``` -go get -u github.com/tendermint/abci/cmd/... -``` - -If this fails, you may need to use `glide` to get vendored dependencies: - -``` -go get github.com/Masterminds/glide -cd $GOPATH/src/github.com/tendermint/abci -glide install -go install ./cmd/... -``` - -Now run `abci-cli --help` to see the list of commands: - -``` -COMMANDS: - batch Run a batch of ABCI commands against an application - console Start an interactive console for multiple commands - echo Have the application echo a message - info Get some info about the application - set_option Set an option on the application - deliver_tx Append a new tx to application - check_tx Validate a tx - commit Get application Merkle root hash - help, h Shows a list of commands or help for one command - -GLOBAL OPTIONS: - --address "tcp://127.0.0.1:46658" address of application socket - --help, -h show help - --version, -v print the version -``` - -## First Example - Dummy - -The `abci-cli` tool lets us send ABCI messages to our application, to help build and debug them. - -The most important messages are `deliver_tx`, `check_tx`, and `commit`, -but there are others for convenience, configuration, and information purposes. - -Let's start a dummy application, which was installed at the same time as `abci-cli` above. The dummy just stores transactions in a merkle tree: - -``` -dummy -``` - -In another terminal, run - -``` -abci-cli echo hello -abci-cli info -``` - -The application should echo `hello` and give you some information about itself. - -An ABCI application must provide two things: - - - a socket server - - a handler for ABCI messages - -When we run the `abci-cli` tool we open a new connection to the application's socket server, -send the given ABCI message, and wait for a response. - -The server may be generic for a particular language, and we provide a [reference implementation -in Golang](https://github.com/tendermint/abci/tree/master/server). -See the [list of other ABCI implementations](https://tendermint.com/ecosystem) -for servers in other languages. - -The handler is specific to the application, and may be arbitrary, -so long as it is deterministic and conforms to the ABCI interface specification. - -So when we run `abci-cli info`, we open a new connection to the ABCI server, which calls the `Info()` method on the application, which tells us the number of transactions in our Merkle tree. - -Now, since every command opens a new connection, we provide the `abci-cli console` and `abci-cli batch` commands, -to allow multiple ABCI messages to be sent over a single connection. - -Running `abci-cli console` should drop you in an interactive console for speaking ABCI messages to your application. - -Try running these commands: - -``` -> echo hello --> data: hello - -> info --> data: {"size":0} - -> commit --> data: 0x - -> deliver_tx "abc" --> code: OK - -> info --> data: {"size":1} - -> commit --> data: 0x750502FC7E84BBD788ED589624F06CFA871845D1 - -> query "abc" --> code: OK --> data: {"index":0,"value":"abc","exists":true} - -> deliver_tx "def=xyz" --> code: OK - -> commit --> data: 0x76393B8A182E450286B0694C629ECB51B286EFD5 - -> query "def" --> code: OK --> data: {"index":1,"value":"xyz","exists":true} -``` - -Note that if we do `deliver_tx "abc"` it will store `(abc, abc)`, -but if we do `deliver_tx "abc=efg"` it will store `(abc, efg)`. - -Similarly, you could put the commands in a file and run `abci-cli --verbose batch < myfile`. - - -## Another Example - Counter - -Now that we've got the hang of it, let's try another application, the "counter" app. - -The counter app doesn't use a Merkle tree, it just counts how many times we've sent a transaction, -asked for a hash, or committed the state. The result of `commit` is just the number of transactions sent. - -This application has two modes: `serial=off` and `serial=on`. - -When `serial=on`, transactions must be a big-endian encoded incrementing integer, starting at 0. - -If `serial=off`, there are no restrictions on transactions. - -We can toggle the value of `serial` using the `set_option` ABCI message. - -When `serial=on`, some transactions are invalid. -In a live blockchain, transactions collect in memory before they are committed into blocks. -To avoid wasting resources on invalid transactions, -ABCI provides the `check_tx` message, -which application developers can use to accept or reject transactions, -before they are stored in memory or gossipped to other peers. - -In this instance of the counter app, `check_tx` only allows transactions whose integer is greater than the last committed one. - -Let's kill the console and the dummy application, and start the counter app: - -``` -counter -``` - -In another window, start the `abci-cli console`: - -``` -> set_option serial on --> data: serial=on - -> check_tx 0x00 --> code: OK - -> check_tx 0xff --> code: OK - -> deliver_tx 0x00 --> code: OK - -> check_tx 0x00 --> code: BadNonce --> log: Invalid nonce. Expected >= 1, got 0 - -> deliver_tx 0x01 --> code: OK - -> deliver_tx 0x04 --> code: BadNonce --> log: Invalid nonce. Expected 2, got 4 - -> info --> data: {"hashes":0,"txs":2} -``` - -This is a very simple application, but between `counter` and `dummy`, its easy to see how you can build out arbitrary application states on top of the ABCI. -[Hyperledger's Burrow](https://github.com/hyperledger/burrow) also runs atop ABCI, bringing with it Ethereum-like accounts, the Ethereum virtual-machine, Monax's permissioning scheme, and native contracts extensions. - -But the ultimate flexibility comes from being able to write the application easily in any language. - -We have implemented the counter in a number of languages (see the example directory). - -To run the Node JS version, `cd` to `example/js` and run - -``` -node app.js -``` - -(you'll have to kill the other counter application process). -In another window, run the console and those previous ABCI commands. -You should get the same results as for the Go version. - -Want to write the counter app in your favorite language?! We'd be happy to add you to our [ecosystem](https://tendermint.com/ecosystem)! We're also offering [bounties](https://tendermint.com/bounties) for implementations in new languages! - -## Notes - -The `abci-cli` is designed strictly for testing and debugging. -In a real deployment, the role of sending messages is taken by Tendermint, -which connects to the app using three separate connections, -each with its own pattern of messages. - -For more information, see the [application developers guide](/docs/guides/app-development). -For examples of running an ABCI app with Tendermint, see the [introductory guide](/docs/getting-started/first-abci-app). diff --git a/docs/guides/app-architecture.md b/docs/guides/app-architecture.md deleted file mode 100644 index fddd6e3a1..000000000 --- a/docs/guides/app-architecture.md +++ /dev/null @@ -1,64 +0,0 @@ -# Application Architecture Guide - -## Overview - -A blockchain application is more than the consensus engine and the transaction logic (eg. smart contracts, business logic) as implemented in the ABCI app. There are also (mobile, web, desktop) clients that will need to connect and make use of the app. We will assume for now that you have a well designed transactions and database model, but maybe this will be the topic of another article. This article is more interested in various ways of setting up the "plumbing" and connecting these pieces, and demonstrating some evolving best practices. - -## Security - -A very important aspect when constructing a blockchain is security. The consensus model can be DoSed (no consensus possible) by corrupting 1/3 of the validators and exploited (writing arbitrary blocks) by corrupting 2/3 of the validators. So, while the security is not that of the "weakest link", you should take care that the "average link" is sufficiently hardened. - -One big attack surface on the validators is the communication between the ABCI app and the tendermint core. This should be highly protected. Ideally, the app and the core are running on the same machine, so no external agent can target the communication channel. You can use unix sockets (with permissions preventing access from other users), or even compile the two apps into one binary if the ABCI app is also writen in go. If you are unable to do that due to language support, then the ABCI app should bind a TCP connection to localhost (127.0.0.1), which is less efficient and secure, but still not reachable from outside. If you must run the ABCI app and tendermint core on separate machines, make sure you have a secure communication channel (ssh tunnel?) - -Now assuming, you have linked together your app and the core securely, you must also make sure no one can get on the machine it is hosted on. At this point it is basic network security. Run on a secure operating system (SELinux?). Limit who has access to the machine (user accounts, but also where the physical machine is hosted). Turn off all services except for ssh, which should only be accessible by some well-guarded public/private key pairs (no password). And maybe even firewall off access to the ports used by the validators, so only known validators can connect. - -There was also a suggestion on slack from @jhon about compiling everything together with a unikernel for more security, such as [Mirage](https://mirage.io) or [UNIK](https://github.com/emc-advanced-dev/unik). - -## Connecting your client to the blockchain - -### Tendermint Core RPC - -The concept is that the ABCI app is completely hidden from the outside world and only communicated through a tested and secured [interface exposed by the tendermint core](/docs/specs/rpc). This interface exposes a lot of data on the block header and consensus process, which is quite useful for externally verifying the system. It also includes 3(!) methods to broadcast a transaction (propose it for the blockchain, and possibly await a response). And one method to query app-specific data from the ABCI application. - -Pros: -* Server code already written -* Access to block headers to validate merkle proofs (nice for light clients) -* Basic read/write functionality is supported - -Cons: -* Limited interface to app. All queries must be serialized into []byte (less expressive than JSON over HTTP) and there is no way to push data from ABCI app to the client (eg. notify me if account X receives a transaction) - -### Custom ABCI server - -This was proposed by @wolfposd on slack and demonstrated by [TMChat](https://github.com/wolfposd/TMChat), a sample app. The concept is to write a custom server for your app (with typical REST API/websockets/etc for easy use by a mobile app). This custom server is in the same binary as the ABCI app and data store, so can easily react to complex events there that involve understanding the data format (send a message if my balance drops below 500). All "writes" sent to this server are proxied via websocket/JSON-RPC to tendermint core. When they come back as deliver_tx over ABCI, they will be written to the data store. For "reads", we can do any queries we wish that are supported by our architecture, using any web technology that is useful. The general architecture is shown in the following diagram: - -Application Architecture - -Pros: -* Separates application logic from blockchain logic -* Allows much richer, more flexible client-facing API -* Allows pub-sub, watching certain fields, etc. - -Cons: -* Access to ABCI app can be dangerous (be VERY careful not to write unless it comes from the validator node) -* No direct access to the blockchain headers to verify tx -* You must write your own API (but maybe that's a pro...) - -### Hybrid solutions - -Likely the least secure but most versatile. The client can access both the tendermint node for all blockchain info, as well as a custom app server, for complex queries and pub-sub on the abci app. - -Pros: -* All from both above solutions - -Cons: -* Even more complexity -* Even more attack vectors (less security) - -## Scalability - -Read replica using non-validating nodes? They could forward transactions to the validators (fewer connections, more security), and locally allow all queries in any of the above configurations. Thus, while transaction-processing speed is limited by the speed of the abci app and the number of validators, one should be able to scale our read performance to quite an extent (until the replication process drains too many resources from the validator nodes). - -## Example Code - -* [TMChat](https://github.com/wolfposd/TMChat) diff --git a/docs/guides/app-development.md b/docs/guides/app-development.md deleted file mode 100644 index c8f25df87..000000000 --- a/docs/guides/app-development.md +++ /dev/null @@ -1,155 +0,0 @@ -# Application Development Guide - -## ABCI Design - -The purpose of ABCI is to provide a clean interface between state transition machines on one computer and the mechanics of their replication across multiple computers. The former we call 'application logic' and the latter the 'consensus engine'. Application logic validates transactions and optionally executes transactions against some persistent state. A consensus engine ensures all transactions are replicated in the same order on every machine. We call each machine in a consensus engine a 'validator', and each validator runs the same transactions through the same application logic. In particular, we are interested in blockchain-style consensus engines, where transactions are committed in hash-linked blocks. - -The ABCI design has a few distinct components: - -- message protocol - - pairs of request and response messages - - consensus makes requests, application responds - - defined using protobuf -- server/client - - consensus engine runs the client - - application runs the server - - two implementations: - - async raw bytes - - grpc -- blockchain protocol - - abci is connection oriented - - Tendermint Core maintains three connections: - - [mempool connection](#mempool-connection): for checking if transactions should be relayed before they are committed; only uses `CheckTx` - - [consensus connection](#consensus-connection): for executing transactions that have been committed. Message sequence is - for every block - `BeginBlock, [DeliverTx, ...], EndBlock, Commit` - - [query connection](#query-connection): for querying the application state; only uses Query and Info - - - -The mempool and consensus logic act as clients, and each maintains an open ABCI connection with the application, which hosts an ABCI server. Shown are the request and response types sent on each connection. - -## Message Protocol - -The message protocol consists of pairs of requests and responses. Some messages have no fields, while others may include byte-arrays, strings, or integers. See the `message Request` and `message Response` definitions in [the protobuf definition file](https://github.com/tendermint/abci/blob/master/types/types.proto), and the [protobuf documentation](https://developers.google.com/protocol-buffers/docs/overview) for more details. - -For each request, a server should respond with the corresponding response, where order of requests is preserved in the order of responses. - -## Server - -To use ABCI in your programming language of choice, there must be a ABCI server in that language. -Tendermint supports two kinds of implementation of the server: - -- Asynchronous, raw socket server (Tendermint Socket Protocol, also known as TSP or Teaspoon) -- GRPC - -Both can be tested using the `abci-cli` by setting the `--abci` flag appropriately (ie. to `socket` or `grpc`). - -See examples, in various stages of maintenance, in [Go](https://github.com/tendermint/abci/tree/master/server), [JavaScript](https://github.com/tendermint/js-abci), [Python](https://github.com/tendermint/abci/tree/master/example/python3/abci), [C++](https://github.com/mdyring/cpp-tmsp), and [Java](https://github.com/jTendermint/jabci). - -### GRPC - -If GRPC is available in your language, this is the easiest approach, -though it will have significant performance overhead. - -To get started with GRPC, copy in the [protobuf file](https://github.com/tendermint/abci/blob/master/types/types.proto) and compile it using the GRPC plugin for your language. -For instance, for golang, the command is `protoc --go_out=plugins=grpc:. types.proto`. See the [grpc documentation for more details](http://www.grpc.io/docs/). `protoc` will autogenerate all the necessary code for ABCI client and server in your language, including whatever interface your application must satisfy to be used by the ABCI server for handling requests. - -### TSP - -If GRPC is not available in your language, or you require higher performance, or otherwise enjoy programming, you may implement your own ABCI server -using the Tendermint Socket Protocol, known affectionaltely as Teaspoon. -The first step is still to auto-generate the relevant data types and codec in your language using `protoc`. -Messages coming over the socket are Protobuf3 encoded, but additionally length-prefixed to facilitate use as a streaming protocol. Protobuf3 doesn't have an official length-prefix standard, so we use our own. The first byte in the prefix represents the length of the Big Endian encoded length. The remaining bytes in the prefix are the Big Endian encoded length. - -For example, if the Protobuf3 encoded ABCI message is 0xDEADBEEF (4 bytes), the length-prefixed message is 0x0104DEADBEEF. If the Protobuf3 encoded ABCI message is 65535 bytes long, the length-prefixed message would be like 0x02FFFF.... - -Note this prefixing does not apply for grpc. - -An ABCI server must also be able to support multiple connections, as Tendermint uses three connections. - -## Client - -There are currently two use-cases for an ABCI client. -One is a testing tool, as in the `abci-cli`, which allows ABCI requests to be sent via command line. -The other is a consensus engine, such as Tendermint Core, which makes requests to the application every time a new transaction is received or a block is committed. - -It is unlikely that you will need to implement a client. For details of our client, see [here](https://github.com/tendermint/abci/tree/master/client). - -## Blockchain Protocol - -In ABCI, a transaction is simply an arbitrary length byte-array. -It is the application's responsibility to define the transaction codec as they please, -and to use it for both CheckTx and DeliverTx. - -Note that there are two distinct means for running transactions, corresponding to stages of 'awareness' -of the transaction in the network. The first stage is when a transaction is received by a validator from a client into the so-called mempool or transaction pool - this is where we use CheckTx. The second is when the transaction is successfully committed on more than 2/3 of validators - where we use DeliverTx. In the former case, it may not be necessary to run all the state transitions associated with the transaction, as the transaction may not ultimately be committed until some much later time, when the result of its execution will be different. -For instance, an Ethereum ABCI app would check signatures and amounts in CheckTx, but would not actually execute any contract code until the DeliverTx, so as to avoid executing state transitions that have not been finalized. - -To formalize the distinction further, two explicit ABCI connections are made between Tendermint Core and the application: the mempool connection and the consensus connection. We also make a third connection, the query connection, to query the local state of the app. - -### Mempool Connection - -The mempool connection is used *only* for CheckTx requests. -Transactions are run using CheckTx in the same order they were received by the validator. -If the CheckTx returns `OK`, the transaction is kept in memory and relayed to other peers in the same order it was received. Otherwise, it is discarded. - -CheckTx requests run concurrently with block processing; -so they should run against a copy of the main application state which is reset after every block. -This copy is necessary to track transitions made by a sequence of CheckTx requests before they are included in a block. When a block is committed, the application must ensure to reset the mempool state to the latest committed state. Tendermint Core will then filter through all transactions in the mempool, removing any that were included in the block, and re-run the rest using CheckTx against the post-Commit mempool state. - -### Consensus Connection - -The consensus connection is used only when a new block is committed, and communicates all information from the block in a series of requests: `BeginBlock, [DeliverTx, ...], EndBlock, Commit`. -That is, when a block is committed in the consensus, we send a list of DeliverTx requests (one for each transaction) sandwiched by BeginBlock and EndBlock requests, and followed by a Commit. - -#### DeliverTx - -DeliverTx is the workhorse of the blockchain. Tendermint sends the DeliverTx requests asynchronously but in order, -and relies on the underlying socket protocol (ie. TCP) to ensure they are received by the app in order. They have already been ordered in the global consensus by the Tendermint protocol. - -DeliverTx returns a abci.Result, which includes a Code, Data, and Log. The code may be non-zero (non-OK), meaning the corresponding transaction should have been rejected by the mempool, -but may have been included in a block by a Byzantine proposer. - -The block header will be updated (TODO) to include some commitment to the results of DeliverTx, be it a bitarray of non-OK transactions, or a merkle root of the data returned by the DeliverTx requests, or both. - -#### Commit - -Once all processing of the block is complete, Tendermint sends the Commit request and blocks waiting -for a response. While the mempool may run concurrently with block processing (the BeginBlock, DeliverTxs, and EndBlock), it is locked for the Commit request so that its state can be safely reset during Commit. This means the app *MUST NOT* do any blocking communication with the mempool (ie. broadcast_tx) during Commit, or there will be deadlock. Note also that all remaining transactions in the mempool are replayed on the mempool connection (CheckTx) following a commit. - -The Commit response includes a byte array, which is the deterministic state root of the application. It is included in the header of the next block. It can be used to provide easily verified Merkle-proofs of the state of the application. - -It is expected that the app will persist state to disk on Commit. The option to have all transactions replayed from some previous block is the job of the [Handshake](#handshake). - -#### BeginBlock - -The BeginBlock request can be used to run some code at the beginning of every block. It also allows Tendermint to send the current block hash and header to the application, before it sends any of the transactions. - -The app should remember the latest height and header (ie. from which it has run a successful Commit) so that it can tell Tendermint where to pick up from when it restarts. See information on the Handshake, below. - -#### EndBlock - -The EndBlock request can be used to run some code at the end of every block. Additionally, the response may contain a list of validators, which can be used to update the validator set. To add a new validator or update an existing one, simply include them in the list returned in the EndBlock response. To remove one, include it in the list with a `power` equal to `0`. Tendermint core will take care of updating the validator set. Note validator set changes are only available in v0.8.0 and up. - -### Query Connection - -This connection is used to query the application without engaging consensus. It's exposed over the tendermint core rpc, so clients can query the app without exposing a server on the app itself, but they must serialize each query as a single byte array. Additionally, certain "standardized" queries may be used to inform local decisions, for instance about which peers to connect to. - -Tendermint Core currently uses the Query connection to filter peers upon connecting, according to IP address or public key. For instance, returning non-OK ABCI response to either of the following queries will cause Tendermint to not connect to the corresponding peer: - -- `p2p/filter/addr/`, where `` is an IP address. -- `p2p/filter/pubkey/`, where `` is the hex-encoded ED25519 key of the node (not it's validator key) - -Note: these query formats are subject to change! - -### Handshake - -When the app or tendermint restarts, they need to sync to a common height. -When an ABCI connection is first established, Tendermint will call `Info` on the Query connection. -The response should contain the LastBlockHeight and LastBlockAppHash -- the former is the last block for the which the app ran Commit successfully, -the latter is the response from that Commit. - -Using this information, Tendermint will determine what needs to be replayed, if anything, against the app, -to ensure both Tendermint and the app are synced to the latest block height. - -If the app returns a LastBlockHeight of 0, Tendermint will just replay all blocks. diff --git a/docs/guides/install-from-source.md b/docs/guides/install-from-source.md deleted file mode 100644 index b9b3d87ab..000000000 --- a/docs/guides/install-from-source.md +++ /dev/null @@ -1,100 +0,0 @@ -# Install from Source - -This page provides instructions on installing Tendermint from source. -To download pre-built binaries, see the [Download page](/download). - -## Install Go - -Make sure you have [installed Go](https://golang.org/doc/install) and set the `GOPATH`. - -## Install Tendermint - -You should be able to install the latest with a simple - -``` -go get github.com/tendermint/tendermint/cmd/tendermint -``` - -Run `tendermint --help` for more. - -If the installation failed, a dependency may been updated and become incompatible with the latest Tendermint master branch. -We solve this using the `glide` tool for dependency management. - -Fist, install `glide`: - -``` -go get github.com/Masterminds/glide -``` - -Now we can fetch the correct versions of each dependency by running: - -``` -cd $GOPATH/src/github.com/tendermint/tendermint -glide install -go install ./cmd/tendermint -``` - -Note that even though `go get` originally failed, -the repository was still cloned to the correct location in the `$GOPATH`. - -The latest Tendermint Core version is now installed. - -### Reinstall - -If you already have Tendermint installed, and you make updates, -simply - -``` -cd $GOPATH/src/github.com/tendermint/tendermint -go install ./cmd/tendermint -``` - -To upgrade, there are a few options: - -- set a new `$GOPATH` and run `go get github.com/tendermint/tendermint/cmd/tendermint`. This makes a fresh copy of everything for the new version. -- run `go get -u github.com/tendermint/tendermint/cmd/tendermint`, where the `-u` fetches the latest updates for the repository and its dependencies -- fetch and checkout the latest master branch in `$GOPATH/src/github.com/tendermint/tendermint`, and then run `glide install && go install ./cmd/tendermint` as above. - -Note the first two options should usually work, but may fail. -If they do, use `glide`, as above: - -``` -cd $GOPATH/src/github.com/tendermint/tendermint -glide install -go install ./cmd/tendermint -``` - -Since the third option just uses `glide` right away, it should always work. - - -### Troubleshooting - -If `go get` failing bothers you, fetch the code using `git`: - -``` -mkdir -p $GOPATH/src/github.com/tendermint -git clone https://github.com/tendermint/tendermint $GOPATH/src/github.com/tendermint/tendermint -cd $GOPATH/src/github.com/tendermint/tendermint -glide install -go install ./cmd/tendermint -``` - -### Run - -To start a one-node blockchain with a simple in-process application: - -``` -tendermint init -tendermint node --proxy_app=dummy -``` - -See the -[App Development](/docs/guides/app-development) -guide for more details on building applications, -and the -[Using Tendermint](/docs/guides/using-tendermint) -guide for more details about using the `tendermint` program. - -## Next Step - -Learn how to [create your first ABCI app](/docs/getting-started/first-abci-app). diff --git a/docs/guides/using-tendermint.md b/docs/guides/using-tendermint.md deleted file mode 100644 index d2a0b31a4..000000000 --- a/docs/guides/using-tendermint.md +++ /dev/null @@ -1,306 +0,0 @@ -# Using Tendermint - -This is a guide to using the `tendermint` program from the command line. -It assumes only that you [have tendermint installed](/download) and have some rudimentary idea -of what Tendermint and ABCI are. - -You can see the help menu with `tendermint --help`, and the version number with `tendermint version`. - -## Directory Root - -The default directory for blockchain data is `~/.tendermint`. Override this by setting the `TMROOT` environment variable. - -## Initialize - -Initialize the root directory by running: - -``` -tendermint init -``` - -This will create a new private key (`priv_validator.json`), and a genesis file (`genesis.json`) containing the associated public key. -This is all that's necessary to run a local testnet with one validator. - -For more elaborate initialization, see our [testnet deployment tool](https://github.com/tendermint/tools/tree/master/mintnet-kubernetes). - -## Run - -To run a tendermint node, use - -``` -tendermint node -``` - -By default, Tendermint will try to connect to a abci appliction on [127.0.0.1:46658](127.0.0.1:46658). -If you have the `dummy` ABCI app installed, run it in another window. -If you don't, kill tendermint and run an in-process version with - -``` -tendermint node --proxy_app=dummy -``` - -After a few seconds you should see blocks start streaming in. -Note that blocks are produced regularly, even if there are no transactions. -This changes [with this pull request](https://github.com/tendermint/tendermint/pull/584). - -Tendermint supports in-process versions of the dummy, counter, and nil apps that ship as examples in the [ABCI repository](https://github.com/tendermint/abci). -It's easy to compile your own app in-process with tendermint if it's written in Go. -If your app is not written in Go, simply run it in another process, -and use the `--proxy_app` flag to specify the address of the socket it is listening on, for instance - - -``` -tendermint node --proxy_app=/var/run/abci.sock -``` - -## Transactions - -To send a transaction, use `curl` to make requests to the Tendermint RPC server: - -``` -curl http://localhost:46657/broadcast_tx_commit?tx=\"abcd\" -``` - -For handling responses, we recommend you [install the jsonpp tool](http://jmhodges.github.io/jsonpp/) to pretty print the JSON. - -We can see the chain's status at the `/status` end-point: - -``` -curl http://localhost:46657/status | jsonpp -``` - -and the `latest_app_hash` in particular: - -``` -curl http://localhost:46657/status | jsonpp | grep app_hash -``` - -Visit [http://localhost:46657](http://localhost:46657) in your browser to see the list of other endpoints. -Some take no arguments (like `/status`), while others specify the argument name and use `_` as a placeholder. - -## Reset - -**WARNING: UNSAFE** Only do this in development and only if you can afford to lose all blockchain data! - -To reset a blockchain, stop the node, remove the `~/.tendermint/data` directory and run - -``` -tendermint unsafe_reset_priv_validator -``` - -This final step is necessary to reset the `priv_validator.json`, -which otherwise prevents you from making conflicting votes in the consensus -(something that could get you in trouble if you do it on a real blockchain). -If you don't reset the `priv_validator.json`, your fresh new blockchain will not make any blocks. - -## Configuration - -Tendermint uses a `config.toml` for configutation. For details, see [the documentation](/docs/specs/configuration). - -Notable options include the socket address of the application (`proxy_app`), -the listenting address of the tendermint peer (`p2p.laddr`), -and the listening address of the rpc server (`rpc.laddr`). - -Some fields from the config file can be overwritten with flags. - -## Broadcast API - -Earlier, we used the `broadcast_tx_commit` endpoint to send a transaction. -When a transaction is sent to a tendermint node, -it will run via `CheckTx` against the application. -If it passes `CheckTx`, it will be included in the mempool, -broadcast to other peers, and eventually included in a block. - -Since there are multiple phases to processing a transaction, we offer multiple endpoints to broadcast a transaction: - -``` -/broadcast_tx_async -/broadcast_tx_sync -/broadcast_tx_commit -``` - -These correspond to no-processing, processing through the mempool, and processing through a block, respectively. -That is, `broadcast_tx_async`, will return right away without waiting to hear if the transaction is even valid, -while `broadcast_tx_sync` will return with the result of running the transaction through `CheckTx`. -Using `broadcast_tx_commit` will wait until the transaction is committed in a block or until some timeout is reached, -but will return right away if the transaction does not pass `CheckTx`. -The return value for `broadcast_tx_commit` includes two fields, `check_tx` and `deliver_tx`, pertaining to the result of running -the transaction through those ABCI messages. - -The benefit of using `broadcast_tx_commit` is that the request returns after the transaction is committed (ie. included in a block), but that can take on the order of a second. For a quick result, use `broadcast_tx_sync`, -but the transaction will not be committed until later, and by that point its effect on the state may change. - -## Tendermint Networks - -When `tendermint init` is run, both a `genesis.json` and `priv_validator.json` are created in `~/.tendermint`. -The `genesis.json` might look like: - -``` -{ - "app_hash": "", - "chain_id": "test-chain-HZw6TB", - "genesis_time": "0001-01-01T00:00:00.000Z", - "validators": [ - { - "amount": 10, - "name": "", - "pub_key": [ - 1, - "5770B4DD55B3E08B7F5711C48B516347D8C33F47C30C226315D21AA64E0DFF2E" - ] - } - ] -} -``` - -And the `priv_validator.json`: - -``` -{ - "address": "4F4D895F882A18E1D1FC608D102601DA8D3570E5", - "last_height": 0, - "last_round": 0, - "last_signature": null, - "last_signbytes": "", - "last_step": 0, - "priv_key": [ - 1, - "F9FA3CD435BDAE54D0BCA8F1BC289D718C23D855C6DB21E8543F5E4F457E62805770B4DD55B3E08B7F5711C48B516347D8C33F47C30C226315D21AA64E0DFF2E" - ], - "pub_key": [ - 1, - "5770B4DD55B3E08B7F5711C48B516347D8C33F47C30C226315D21AA64E0DFF2E" - ] -} -``` - -The `priv_validator.json` actually contains a private key, and should thus be kept absolutely secret; -for now we work with the plain text. -Note the `last_` fields, which are used to prevent us from signing conflicting messages. - -Note also that the `pub_key` (the public key) in the `priv_validator.json` is also present in the `genesis.json`. - -The genesis file contains the list of public keys which may participate in the consensus, -and their corresponding voting power. -Greater than 2/3 of the voting power must be active (ie. the corresponding private keys must be producing signatures) -for the consensus to make progress. -In our case, the genesis file contains the public key of our `priv_validator.json`, -so a tendermint node started with the default root directory will be able to make new blocks, -as we've already seen. - -If we want to add more nodes to the network, we have two choices: -we can add a new validator node, who will also participate in the consensus -by proposing blocks and voting on them, -or we can add a new non-validator node, who will not participate directly, -but will verify and keep up with the consensus protocol. - -### Peers - -To connect to peers on start-up, specify them in the `config.toml` or on the command line. - -For instance, - -``` -tendermint node --p2p.seeds "1.2.3.4:46656,5.6.7.8:46656" -``` - -Alternatively, you can use the `/dial_seeds` endpoint of the RPC to specify peers for a running node to connect to: - -``` -curl --data-urlencode "seeds=[\"1.2.3.4:46656\",\"5.6.7.8:46656\"]" localhost:46657/dial_seeds -``` - -Additionally, the peer-exchange protocol can be enabled using the `--pex` flag, -though this feature is [still under development](https://github.com/tendermint/tendermint/issues/598) -If `--pex` is enabled, peers will gossip about known peers and form a more resilient network. - -### Adding a Non-Validator - -Adding a non-validator is simple. Just copy the original `genesis.json` to `~/.tendermint` on the new machine -and start the node, specifying seeds as necessary. -If no seeds are specified, the node won't make any blocks, because it's not a validator, -and it won't hear about any blocks, because it's not connected to the other peer. - -### Adding a Validator - -The easiest way to add new validators is to do it in the `genesis.json`, before starting the network. -For instance, we could make a new `priv_validator.json`, and copy it's `pub_key` into the above genesis. - -We can generate a new `priv_validator.json` with the command: - -``` -tendermint gen_validator -``` - -Now we can update our genesis file. For instance, if the new `priv_validator.json` looks like: - -``` -{ - "address": "AC379688105901436A34A65F185C115B8BB277A1", - "last_height": 0, - "last_round": 0, - "last_signature": null, - "last_signbytes": "", - "last_step": 0, - "priv_key": [ - 1, - "0D2ED337D748ADF79BE28559B9E59EBE1ABBA0BAFE6D65FCB9797985329B950C8F2B5AACAACC9FCE41881349743B0CFDE190DF0177744568D4E82A18F0B7DF94" - ], - "pub_key": [ - 1, - "8F2B5AACAACC9FCE41881349743B0CFDE190DF0177744568D4E82A18F0B7DF94" - ] -} -``` - -then the new `genesis.json` will be: - -``` -{ - "app_hash": "", - "chain_id": "test-chain-HZw6TB", - "genesis_time": "0001-01-01T00:00:00.000Z", - "validators": [ - { - "amount": 10, - "name": "", - "pub_key": [ - 1, - "5770B4DD55B3E08B7F5711C48B516347D8C33F47C30C226315D21AA64E0DFF2E" - ] - }, - { - "amount": 10, - "name": "", - "pub_key": [ - 1, - "8F2B5AACAACC9FCE41881349743B0CFDE190DF0177744568D4E82A18F0B7DF94" - ] - } - ] -} -``` - -Update the `genesis.json` in `~/.tendermint`. Copy the genesis file and the new `priv_validator.json` -to the `~/.tendermint` on a new machine. - -Now run `tendermint node` on both machines, and use either `--p2p.seeds` or the `/dial_seeds` to get them to peer up. -They should start making blocks, and will only continue to do so as long as both of them are online. - -To make a Tendermint network that can tolerate one of the validators failing, you need at least four validator nodes (> 2/3). - -Updating validators in a live network is supported but must be explicitly programmed by the application developer. -See the [application developers guide](/docs/guides/app-development#Handshake) for more details. - -### Local Network - -To run a network locally, say on a single machine, you must change the `_laddr` fields in the `config.toml` (or using the flags) -so that the listening addresses of the various sockets don't conflict. -Additionally, you must set `addrbook_strict=false` in the `config.toml`, -otherwise Tendermint's p2p library will deny making connections to peers with the same IP address. - -## More - -Got a couple nodes talking to each other using the dummy app? -Try a more sophisticated app like [Ethermint](https://github.com/tendermint/ethermint), -or learn more about building your own in the [Application Developer's Guide](/docs/guides/app-development). diff --git a/docs/index.rst b/docs/index.rst index dcceab227..7d46b1d64 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -27,3 +27,8 @@ See our [Community](/community) page for more ways to collaborate. You can also [get in touch with the team](/contact). Most importantly, enjoy! + +Next Steps +---------- + + diff --git a/docs/install-from-source.rst b/docs/install-from-source.rst new file mode 100644 index 000000000..a132bf68b --- /dev/null +++ b/docs/install-from-source.rst @@ -0,0 +1,113 @@ +Install from Source +=================== + +This page provides instructions on installing Tendermint from source. To +download pre-built binaries, see the `Download page `__. + +Install Go +---------- + +Make sure you have `installed Go `__ and +set the ``GOPATH``. + +Install Tendermint +------------------ + +You should be able to install the latest with a simple + +:: + + go get github.com/tendermint/tendermint/cmd/tendermint + +Run ``tendermint --help`` for more. + +If the installation failed, a dependency may been updated and become +incompatible with the latest Tendermint master branch. We solve this +using the ``glide`` tool for dependency management. + +Fist, install ``glide``: + +:: + + go get github.com/Masterminds/glide + +Now we can fetch the correct versions of each dependency by running: + +:: + + cd $GOPATH/src/github.com/tendermint/tendermint + glide install + go install ./cmd/tendermint + +Note that even though ``go get`` originally failed, the repository was +still cloned to the correct location in the ``$GOPATH``. + +The latest Tendermint Core version is now installed. + +Reinstall +~~~~~~~~~ + +If you already have Tendermint installed, and you make updates, simply + +:: + + cd $GOPATH/src/github.com/tendermint/tendermint + go install ./cmd/tendermint + +To upgrade, there are a few options: + +- set a new ``$GOPATH`` and run + ``go get github.com/tendermint/tendermint/cmd/tendermint``. This + makes a fresh copy of everything for the new version. +- run ``go get -u github.com/tendermint/tendermint/cmd/tendermint``, + where the ``-u`` fetches the latest updates for the repository and + its dependencies +- fetch and checkout the latest master branch in + ``$GOPATH/src/github.com/tendermint/tendermint``, and then run + ``glide install && go install ./cmd/tendermint`` as above. + +Note the first two options should usually work, but may fail. If they +do, use ``glide``, as above: + +:: + + cd $GOPATH/src/github.com/tendermint/tendermint + glide install + go install ./cmd/tendermint + +Since the third option just uses ``glide`` right away, it should always +work. + +Troubleshooting +~~~~~~~~~~~~~~~ + +If ``go get`` failing bothers you, fetch the code using ``git``: + +:: + + mkdir -p $GOPATH/src/github.com/tendermint + git clone https://github.com/tendermint/tendermint $GOPATH/src/github.com/tendermint/tendermint + cd $GOPATH/src/github.com/tendermint/tendermint + glide install + go install ./cmd/tendermint + +Run +~~~ + +To start a one-node blockchain with a simple in-process application: + +:: + + tendermint init + tendermint node --proxy_app=dummy + +See the `App Development `__ guide for +more details on building applications, and the `Using +Tendermint `__ guide for more details +about using the ``tendermint`` program. + +Next Step +--------- + +Learn how to `create your first ABCI +app `__. diff --git a/docs/using-tendermint.rst b/docs/using-tendermint.rst new file mode 100644 index 000000000..5d7f95c90 --- /dev/null +++ b/docs/using-tendermint.rst @@ -0,0 +1,365 @@ +Using Tendermint +================ + +This is a guide to using the ``tendermint`` program from the command +line. It assumes only that you `have tendermint installed `__ +and have some rudimentary idea of what Tendermint and ABCI are. + +You can see the help menu with ``tendermint --help``, and the version +number with ``tendermint version``. + +Directory Root +-------------- + +The default directory for blockchain data is ``~/.tendermint``. Override +this by setting the ``TMROOT`` environment variable. + +Initialize +---------- + +Initialize the root directory by running: + +:: + + tendermint init + +This will create a new private key (``priv_validator.json``), and a +genesis file (``genesis.json``) containing the associated public key. +This is all that's necessary to run a local testnet with one validator. + +For more elaborate initialization, see our `testnet deployment +tool `__. + +Run +--- + +To run a tendermint node, use + +:: + + tendermint node + +By default, Tendermint will try to connect to a abci appliction on +`127.0.0.1:46658 <127.0.0.1:46658>`__. If you have the ``dummy`` ABCI +app installed, run it in another window. If you don't, kill tendermint +and run an in-process version with + +:: + + tendermint node --proxy_app=dummy + +After a few seconds you should see blocks start streaming in. Note that +blocks are produced regularly, even if there are no transactions. This +changes `with this pull +request `__. + +Tendermint supports in-process versions of the dummy, counter, and nil +apps that ship as examples in the `ABCI +repository `__. It's easy to compile +your own app in-process with tendermint if it's written in Go. If your +app is not written in Go, simply run it in another process, and use the +``--proxy_app`` flag to specify the address of the socket it is +listening on, for instance + +:: + + tendermint node --proxy_app=/var/run/abci.sock + +Transactions +------------ + +To send a transaction, use ``curl`` to make requests to the Tendermint +RPC server: + +:: + + curl http://localhost:46657/broadcast_tx_commit?tx=\"abcd\" + +For handling responses, we recommend you `install the jsonpp +tool `__ to pretty print the JSON. + +We can see the chain's status at the ``/status`` end-point: + +:: + + curl http://localhost:46657/status | jsonpp + +and the ``latest_app_hash`` in particular: + +:: + + curl http://localhost:46657/status | jsonpp | grep app_hash + +Visit http://localhost:46657 in your browser to see the list of other +endpoints. Some take no arguments (like ``/status``), while others +specify the argument name and use ``_`` as a placeholder. + +Reset +----- + +**WARNING: UNSAFE** Only do this in development and only if you can +afford to lose all blockchain data! + +To reset a blockchain, stop the node, remove the ``~/.tendermint/data`` +directory and run + +:: + + tendermint unsafe_reset_priv_validator + +This final step is necessary to reset the ``priv_validator.json``, which +otherwise prevents you from making conflicting votes in the consensus +(something that could get you in trouble if you do it on a real +blockchain). If you don't reset the ``priv_validator.json``, your fresh +new blockchain will not make any blocks. + +Configuration +------------- + +Tendermint uses a ``config.toml`` for configutation. For details, see +`the documentation `__. + +Notable options include the socket address of the application +(``proxy_app``), the listenting address of the tendermint peer +(``p2p.laddr``), and the listening address of the rpc server +(``rpc.laddr``). + +Some fields from the config file can be overwritten with flags. + +Broadcast API +------------- + +Earlier, we used the ``broadcast_tx_commit`` endpoint to send a +transaction. When a transaction is sent to a tendermint node, it will +run via ``CheckTx`` against the application. If it passes ``CheckTx``, +it will be included in the mempool, broadcast to other peers, and +eventually included in a block. + +Since there are multiple phases to processing a transaction, we offer +multiple endpoints to broadcast a transaction: + +:: + + /broadcast_tx_async + /broadcast_tx_sync + /broadcast_tx_commit + +These correspond to no-processing, processing through the mempool, and +processing through a block, respectively. That is, +``broadcast_tx_async``, will return right away without waiting to hear +if the transaction is even valid, while ``broadcast_tx_sync`` will +return with the result of running the transaction through ``CheckTx``. +Using ``broadcast_tx_commit`` will wait until the transaction is +committed in a block or until some timeout is reached, but will return +right away if the transaction does not pass ``CheckTx``. The return +value for ``broadcast_tx_commit`` includes two fields, ``check_tx`` and +``deliver_tx``, pertaining to the result of running the transaction +through those ABCI messages. + +The benefit of using ``broadcast_tx_commit`` is that the request returns +after the transaction is committed (ie. included in a block), but that +can take on the order of a second. For a quick result, use +``broadcast_tx_sync``, but the transaction will not be committed until +later, and by that point its effect on the state may change. + +Tendermint Networks +------------------- + +When ``tendermint init`` is run, both a ``genesis.json`` and +``priv_validator.json`` are created in ``~/.tendermint``. The +``genesis.json`` might look like: + +:: + + { + "app_hash": "", + "chain_id": "test-chain-HZw6TB", + "genesis_time": "0001-01-01T00:00:00.000Z", + "validators": [ + { + "amount": 10, + "name": "", + "pub_key": [ + 1, + "5770B4DD55B3E08B7F5711C48B516347D8C33F47C30C226315D21AA64E0DFF2E" + ] + } + ] + } + +And the ``priv_validator.json``: + +:: + + { + "address": "4F4D895F882A18E1D1FC608D102601DA8D3570E5", + "last_height": 0, + "last_round": 0, + "last_signature": null, + "last_signbytes": "", + "last_step": 0, + "priv_key": [ + 1, + "F9FA3CD435BDAE54D0BCA8F1BC289D718C23D855C6DB21E8543F5E4F457E62805770B4DD55B3E08B7F5711C48B516347D8C33F47C30C226315D21AA64E0DFF2E" + ], + "pub_key": [ + 1, + "5770B4DD55B3E08B7F5711C48B516347D8C33F47C30C226315D21AA64E0DFF2E" + ] + } + +The ``priv_validator.json`` actually contains a private key, and should +thus be kept absolutely secret; for now we work with the plain text. +Note the ``last_`` fields, which are used to prevent us from signing +conflicting messages. + +Note also that the ``pub_key`` (the public key) in the +``priv_validator.json`` is also present in the ``genesis.json``. + +The genesis file contains the list of public keys which may participate +in the consensus, and their corresponding voting power. Greater than 2/3 +of the voting power must be active (ie. the corresponding private keys +must be producing signatures) for the consensus to make progress. In our +case, the genesis file contains the public key of our +``priv_validator.json``, so a tendermint node started with the default +root directory will be able to make new blocks, as we've already seen. + +If we want to add more nodes to the network, we have two choices: we can +add a new validator node, who will also participate in the consensus by +proposing blocks and voting on them, or we can add a new non-validator +node, who will not participate directly, but will verify and keep up +with the consensus protocol. + +Peers +~~~~~ + +To connect to peers on start-up, specify them in the ``config.toml`` or +on the command line. + +For instance, + +:: + + tendermint node --p2p.seeds "1.2.3.4:46656,5.6.7.8:46656" + +Alternatively, you can use the ``/dial_seeds`` endpoint of the RPC to +specify peers for a running node to connect to: + +:: + + curl --data-urlencode "seeds=[\"1.2.3.4:46656\",\"5.6.7.8:46656\"]" localhost:46657/dial_seeds + +Additionally, the peer-exchange protocol can be enabled using the +``--pex`` flag, though this feature is `still under +development `__ If +``--pex`` is enabled, peers will gossip about known peers and form a +more resilient network. + +Adding a Non-Validator +~~~~~~~~~~~~~~~~~~~~~~ + +Adding a non-validator is simple. Just copy the original +``genesis.json`` to ``~/.tendermint`` on the new machine and start the +node, specifying seeds as necessary. If no seeds are specified, the node +won't make any blocks, because it's not a validator, and it won't hear +about any blocks, because it's not connected to the other peer. + +Adding a Validator +~~~~~~~~~~~~~~~~~~ + +The easiest way to add new validators is to do it in the +``genesis.json``, before starting the network. For instance, we could +make a new ``priv_validator.json``, and copy it's ``pub_key`` into the +above genesis. + +We can generate a new ``priv_validator.json`` with the command: + +:: + + tendermint gen_validator + +Now we can update our genesis file. For instance, if the new +``priv_validator.json`` looks like: + +:: + + { + "address": "AC379688105901436A34A65F185C115B8BB277A1", + "last_height": 0, + "last_round": 0, + "last_signature": null, + "last_signbytes": "", + "last_step": 0, + "priv_key": [ + 1, + "0D2ED337D748ADF79BE28559B9E59EBE1ABBA0BAFE6D65FCB9797985329B950C8F2B5AACAACC9FCE41881349743B0CFDE190DF0177744568D4E82A18F0B7DF94" + ], + "pub_key": [ + 1, + "8F2B5AACAACC9FCE41881349743B0CFDE190DF0177744568D4E82A18F0B7DF94" + ] + } + +then the new ``genesis.json`` will be: + +:: + + { + "app_hash": "", + "chain_id": "test-chain-HZw6TB", + "genesis_time": "0001-01-01T00:00:00.000Z", + "validators": [ + { + "amount": 10, + "name": "", + "pub_key": [ + 1, + "5770B4DD55B3E08B7F5711C48B516347D8C33F47C30C226315D21AA64E0DFF2E" + ] + }, + { + "amount": 10, + "name": "", + "pub_key": [ + 1, + "8F2B5AACAACC9FCE41881349743B0CFDE190DF0177744568D4E82A18F0B7DF94" + ] + } + ] + } + +Update the ``genesis.json`` in ``~/.tendermint``. Copy the genesis file +and the new ``priv_validator.json`` to the ``~/.tendermint`` on a new +machine. + +Now run ``tendermint node`` on both machines, and use either +``--p2p.seeds`` or the ``/dial_seeds`` to get them to peer up. They +should start making blocks, and will only continue to do so as long as +both of them are online. + +To make a Tendermint network that can tolerate one of the validators +failing, you need at least four validator nodes (> 2/3). + +Updating validators in a live network is supported but must be +explicitly programmed by the application developer. See the `application +developers guide `__ for more +details. + +Local Network +~~~~~~~~~~~~~ + +To run a network locally, say on a single machine, you must change the +``_laddr`` fields in the ``config.toml`` (or using the flags) so that +the listening addresses of the various sockets don't conflict. +Additionally, you must set ``addrbook_strict=false`` in the +``config.toml``, otherwise Tendermint's p2p library will deny making +connections to peers with the same IP address. + +More +---- + +Got a couple nodes talking to each other using the dummy app? Try a more +sophisticated app like +`Ethermint `__, or learn more +about building your own in the `Application Developer's +Guide `__.