You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

630 lines
20 KiB

  1. <!---
  2. order: 3
  3. --->
  4. # Creating an application in Java
  5. ## Guide Assumptions
  6. This guide is designed for beginners who want to get started with a Tendermint
  7. Core application from scratch. It does not assume that you have any prior
  8. experience with Tendermint Core.
  9. Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state
  10. transition machine (your application) - written in any programming language - and securely
  11. replicates it on many machines.
  12. By following along with this guide, you'll create a Tendermint Core project
  13. called kvstore, a (very) simple distributed BFT key-value store. The application (which should
  14. implementing the blockchain interface (ABCI)) will be written in Java.
  15. This guide assumes that you are not new to JVM world. If you are new please see [JVM Minimal Survival Guide](https://hadihariri.com/2013/12/29/jvm-minimal-survival-guide-for-the-dotnet-developer/#java-the-language-java-the-ecosystem-java-the-jvm) and [Gradle Docs](https://docs.gradle.org/current/userguide/userguide.html).
  16. ## Built-in app vs external app
  17. If you use Golang, you can run your app and Tendermint Core in the same process to get maximum performance.
  18. [Cosmos SDK](https://github.com/cosmos/cosmos-sdk) is written this way.
  19. Please refer to [Writing a built-in Tendermint Core application in Go](./go-built-in.md) guide for details.
  20. If you choose another language, like we did in this guide, you have to write a separate app,
  21. which will communicate with Tendermint Core via a socket (UNIX or TCP) or gRPC.
  22. This guide will show you how to build external application using RPC server.
  23. Having a separate application might give you better security guarantees as two
  24. processes would be communicating via established binary protocol. Tendermint
  25. Core will not have access to application's state.
  26. ## 1.1 Installing Java and Gradle
  27. Please refer to [the Oracle's guide for installing JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html).
  28. Verify that you have installed Java successfully:
  29. ```bash
  30. $ java -version
  31. java version "12.0.2" 2019-07-16
  32. Java(TM) SE Runtime Environment (build 12.0.2+10)
  33. Java HotSpot(TM) 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)
  34. ```
  35. You can choose any version of Java higher or equal to 8.
  36. This guide is written using Java SE Development Kit 12.
  37. Make sure you have `$JAVA_HOME` environment variable set:
  38. ```bash
  39. $ echo $JAVA_HOME
  40. /Library/Java/JavaVirtualMachines/jdk-12.0.2.jdk/Contents/Home
  41. ```
  42. For Gradle installation, please refer to [their official guide](https://gradle.org/install/).
  43. ## 1.2 Creating a new Java project
  44. We'll start by creating a new Gradle project.
  45. ```bash
  46. $ export KVSTORE_HOME=~/kvstore
  47. $ mkdir $KVSTORE_HOME
  48. $ cd $KVSTORE_HOME
  49. ```
  50. Inside the example directory run:
  51. ```bash
  52. gradle init --dsl groovy --package io.example --project-name example --type java-application --test-framework junit
  53. ```
  54. This will create a new project for you. The tree of files should look like:
  55. ```bash
  56. $ tree
  57. .
  58. |-- build.gradle
  59. |-- gradle
  60. | `-- wrapper
  61. | |-- gradle-wrapper.jar
  62. | `-- gradle-wrapper.properties
  63. |-- gradlew
  64. |-- gradlew.bat
  65. |-- settings.gradle
  66. `-- src
  67. |-- main
  68. | |-- java
  69. | | `-- io
  70. | | `-- example
  71. | | `-- App.java
  72. | `-- resources
  73. `-- test
  74. |-- java
  75. | `-- io
  76. | `-- example
  77. | `-- AppTest.java
  78. `-- resources
  79. ```
  80. When run, this should print "Hello world." to the standard output.
  81. ```bash
  82. $ ./gradlew run
  83. > Task :run
  84. Hello world.
  85. ```
  86. ## 1.3 Writing a Tendermint Core application
  87. Tendermint Core communicates with the application through the Application
  88. BlockChain Interface (ABCI). All message types are defined in the [protobuf
  89. file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto).
  90. This allows Tendermint Core to run applications written in any programming
  91. language.
  92. ### 1.3.1 Compile .proto files
  93. Add the following piece to the top of the `build.gradle`:
  94. ```groovy
  95. buildscript {
  96. repositories {
  97. mavenCentral()
  98. }
  99. dependencies {
  100. classpath 'com.google.protobuf:protobuf-gradle-plugin:0.8.8'
  101. }
  102. }
  103. ```
  104. Enable the protobuf plugin in the `plugins` section of the `build.gradle`:
  105. ```groovy
  106. plugins {
  107. id 'com.google.protobuf' version '0.8.8'
  108. }
  109. ```
  110. Add the following code to `build.gradle`:
  111. ```groovy
  112. protobuf {
  113. protoc {
  114. artifact = "com.google.protobuf:protoc:3.7.1"
  115. }
  116. plugins {
  117. grpc {
  118. artifact = 'io.grpc:protoc-gen-grpc-java:1.22.1'
  119. }
  120. }
  121. generateProtoTasks {
  122. all()*.plugins {
  123. grpc {}
  124. }
  125. }
  126. }
  127. ```
  128. Now we should be ready to compile the `*.proto` files.
  129. Copy the necessary `.proto` files to your project:
  130. ```bash
  131. mkdir -p \
  132. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/abci \
  133. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/version \
  134. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/types \
  135. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/crypto \
  136. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/libs \
  137. $KVSTORE_HOME/src/main/proto/github.com/gogo/protobuf/gogoproto
  138. cp $GOPATH/src/github.com/tendermint/tendermint/proto/tendermint/abci/types.proto \
  139. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/abci/types.proto
  140. cp $GOPATH/src/github.com/tendermint/tendermint/proto/tendermint/version/version.proto \
  141. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/version/version.proto
  142. cp $GOPATH/src/github.com/tendermint/tendermint/proto/tendermint/types/types.proto \
  143. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/types/types.proto
  144. cp $GOPATH/src/github.com/tendermint/tendermint/proto/tendermint/types/evidence.proto \
  145. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/types/evidence.proto
  146. cp $GOPATH/src/github.com/tendermint/tendermint/proto/tendermint/types/params.proto \
  147. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/types/params.proto
  148. cp $GOPATH/src/github.com/tendermint/tendermint/proto/tendermint/crypto/merkle.proto \
  149. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/crypto/merkle.proto
  150. cp $GOPATH/src/github.com/tendermint/tendermint/proto/tendermint/crypto/keys.proto \
  151. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/crypto/keys.proto
  152. cp $GOPATH/src/github.com/tendermint/tendermint/proto/tendermint/libs/types.proto \
  153. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/proto/tendermint/libs/types.proto
  154. cp $GOPATH/src/github.com/gogo/protobuf/gogoproto/gogo.proto \
  155. $KVSTORE_HOME/src/main/proto/github.com/gogo/protobuf/gogoproto/gogo.proto
  156. ```
  157. Add these dependencies to `build.gradle`:
  158. ```groovy
  159. dependencies {
  160. implementation 'io.grpc:grpc-protobuf:1.22.1'
  161. implementation 'io.grpc:grpc-netty-shaded:1.22.1'
  162. implementation 'io.grpc:grpc-stub:1.22.1'
  163. }
  164. ```
  165. To generate all protobuf-type classes run:
  166. ```bash
  167. ./gradlew generateProto
  168. ```
  169. To verify that everything went smoothly, you can inspect the `build/generated/` directory:
  170. ```bash
  171. $ tree build/generated/
  172. build/generated/
  173. |-- source
  174. | `-- proto
  175. | `-- main
  176. | |-- grpc
  177. | | `-- types
  178. | | `-- ABCIApplicationGrpc.java
  179. | `-- java
  180. | |-- com
  181. | | `-- google
  182. | | `-- protobuf
  183. | | `-- GoGoProtos.java
  184. | |-- common
  185. | | `-- Types.java
  186. | |-- merkle
  187. | | `-- Merkle.java
  188. | `-- types
  189. | `-- Types.java
  190. ```
  191. ### 1.3.2 Implementing ABCI
  192. The resulting `$KVSTORE_HOME/build/generated/source/proto/main/grpc/types/ABCIApplicationGrpc.java` file
  193. contains the abstract class `ABCIApplicationImplBase`, which is an interface we'll need to implement.
  194. Create `$KVSTORE_HOME/src/main/java/io/example/KVStoreApp.java` file with the following content:
  195. ```java
  196. package io.example;
  197. import io.grpc.stub.StreamObserver;
  198. import types.ABCIApplicationGrpc;
  199. import types.Types.*;
  200. class KVStoreApp extends ABCIApplicationGrpc.ABCIApplicationImplBase {
  201. // methods implementation
  202. }
  203. ```
  204. Now I will go through each method of `ABCIApplicationImplBase` explaining when it's called and adding
  205. required business logic.
  206. ### 1.3.3 CheckTx
  207. When a new transaction is added to the Tendermint Core, it will ask the
  208. application to check it (validate the format, signatures, etc.).
  209. ```java
  210. @Override
  211. public void checkTx(RequestCheckTx req, StreamObserver<ResponseCheckTx> responseObserver) {
  212. var tx = req.getTx();
  213. int code = validate(tx);
  214. var resp = ResponseCheckTx.newBuilder()
  215. .setCode(code)
  216. .setGasWanted(1)
  217. .build();
  218. responseObserver.onNext(resp);
  219. responseObserver.onCompleted();
  220. }
  221. private int validate(ByteString tx) {
  222. List<byte[]> parts = split(tx, '=');
  223. if (parts.size() != 2) {
  224. return 1;
  225. }
  226. byte[] key = parts.get(0);
  227. byte[] value = parts.get(1);
  228. // check if the same key=value already exists
  229. var stored = getPersistedValue(key);
  230. if (stored != null && Arrays.equals(stored, value)) {
  231. return 2;
  232. }
  233. return 0;
  234. }
  235. private List<byte[]> split(ByteString tx, char separator) {
  236. var arr = tx.toByteArray();
  237. int i;
  238. for (i = 0; i < tx.size(); i++) {
  239. if (arr[i] == (byte)separator) {
  240. break;
  241. }
  242. }
  243. if (i == tx.size()) {
  244. return Collections.emptyList();
  245. }
  246. return List.of(
  247. tx.substring(0, i).toByteArray(),
  248. tx.substring(i + 1).toByteArray()
  249. );
  250. }
  251. ```
  252. Don't worry if this does not compile yet.
  253. If the transaction does not have a form of `{bytes}={bytes}`, we return `1`
  254. code. When the same key=value already exist (same key and value), we return `2`
  255. code. For others, we return a zero code indicating that they are valid.
  256. Note that anything with non-zero code will be considered invalid (`-1`, `100`,
  257. etc.) by Tendermint Core.
  258. Valid transactions will eventually be committed given they are not too big and
  259. have enough gas. To learn more about gas, check out ["the
  260. specification"](https://docs.tendermint.com/master/spec/abci/apps.html#gas).
  261. For the underlying key-value store we'll use
  262. [JetBrains Xodus](https://github.com/JetBrains/xodus), which is a transactional schema-less embedded high-performance database written in Java.
  263. `build.gradle`:
  264. ```groovy
  265. dependencies {
  266. implementation 'org.jetbrains.xodus:xodus-environment:1.3.91'
  267. }
  268. ```
  269. ```java
  270. ...
  271. import jetbrains.exodus.ArrayByteIterable;
  272. import jetbrains.exodus.ByteIterable;
  273. import jetbrains.exodus.env.Environment;
  274. import jetbrains.exodus.env.Store;
  275. import jetbrains.exodus.env.StoreConfig;
  276. import jetbrains.exodus.env.Transaction;
  277. class KVStoreApp extends ABCIApplicationGrpc.ABCIApplicationImplBase {
  278. private Environment env;
  279. private Transaction txn = null;
  280. private Store store = null;
  281. KVStoreApp(Environment env) {
  282. this.env = env;
  283. }
  284. ...
  285. private byte[] getPersistedValue(byte[] k) {
  286. return env.computeInReadonlyTransaction(txn -> {
  287. var store = env.openStore("store", StoreConfig.WITHOUT_DUPLICATES, txn);
  288. ByteIterable byteIterable = store.get(txn, new ArrayByteIterable(k));
  289. if (byteIterable == null) {
  290. return null;
  291. }
  292. return byteIterable.getBytesUnsafe();
  293. });
  294. }
  295. }
  296. ```
  297. ### 1.3.4 BeginBlock -> DeliverTx -> EndBlock -> Commit
  298. When Tendermint Core has decided on the block, it's transferred to the
  299. application in 3 parts: `BeginBlock`, one `DeliverTx` per transaction and
  300. `EndBlock` in the end. `DeliverTx` are being transferred asynchronously, but the
  301. responses are expected to come in order.
  302. ```java
  303. @Override
  304. public void beginBlock(RequestBeginBlock req, StreamObserver<ResponseBeginBlock> responseObserver) {
  305. txn = env.beginTransaction();
  306. store = env.openStore("store", StoreConfig.WITHOUT_DUPLICATES, txn);
  307. var resp = ResponseBeginBlock.newBuilder().build();
  308. responseObserver.onNext(resp);
  309. responseObserver.onCompleted();
  310. }
  311. ```
  312. Here we begin a new transaction, which will accumulate the block's transactions and open the corresponding store.
  313. ```java
  314. @Override
  315. public void deliverTx(RequestDeliverTx req, StreamObserver<ResponseDeliverTx> responseObserver) {
  316. var tx = req.getTx();
  317. int code = validate(tx);
  318. if (code == 0) {
  319. List<byte[]> parts = split(tx, '=');
  320. var key = new ArrayByteIterable(parts.get(0));
  321. var value = new ArrayByteIterable(parts.get(1));
  322. store.put(txn, key, value);
  323. }
  324. var resp = ResponseDeliverTx.newBuilder()
  325. .setCode(code)
  326. .build();
  327. responseObserver.onNext(resp);
  328. responseObserver.onCompleted();
  329. }
  330. ```
  331. If the transaction is badly formatted or the same key=value already exist, we
  332. again return the non-zero code. Otherwise, we add it to the store.
  333. In the current design, a block can include incorrect transactions (those who
  334. passed `CheckTx`, but failed `DeliverTx` or transactions included by the proposer
  335. directly). This is done for performance reasons.
  336. Note we can't commit transactions inside the `DeliverTx` because in such case
  337. `Query`, which may be called in parallel, will return inconsistent data (i.e.
  338. it will report that some value already exist even when the actual block was not
  339. yet committed).
  340. `Commit` instructs the application to persist the new state.
  341. ```java
  342. @Override
  343. public void commit(RequestCommit req, StreamObserver<ResponseCommit> responseObserver) {
  344. txn.commit();
  345. var resp = ResponseCommit.newBuilder()
  346. .setData(ByteString.copyFrom(new byte[8]))
  347. .build();
  348. responseObserver.onNext(resp);
  349. responseObserver.onCompleted();
  350. }
  351. ```
  352. ### 1.3.5 Query
  353. Now, when the client wants to know whenever a particular key/value exist, it
  354. will call Tendermint Core RPC `/abci_query` endpoint, which in turn will call
  355. the application's `Query` method.
  356. Applications are free to provide their own APIs. But by using Tendermint Core
  357. as a proxy, clients (including [light client
  358. package](https://godoc.org/github.com/tendermint/tendermint/light)) can leverage
  359. the unified API across different applications. Plus they won't have to call the
  360. otherwise separate Tendermint Core API for additional proofs.
  361. Note we don't include a proof here.
  362. ```java
  363. @Override
  364. public void query(RequestQuery req, StreamObserver<ResponseQuery> responseObserver) {
  365. var k = req.getData().toByteArray();
  366. var v = getPersistedValue(k);
  367. var builder = ResponseQuery.newBuilder();
  368. if (v == null) {
  369. builder.setLog("does not exist");
  370. } else {
  371. builder.setLog("exists");
  372. builder.setKey(ByteString.copyFrom(k));
  373. builder.setValue(ByteString.copyFrom(v));
  374. }
  375. responseObserver.onNext(builder.build());
  376. responseObserver.onCompleted();
  377. }
  378. ```
  379. The complete specification can be found
  380. [here](https://docs.tendermint.com/master/spec/abci/).
  381. ## 1.4 Starting an application and a Tendermint Core instances
  382. Put the following code into the `$KVSTORE_HOME/src/main/java/io/example/App.java` file:
  383. ```java
  384. package io.example;
  385. import jetbrains.exodus.env.Environment;
  386. import jetbrains.exodus.env.Environments;
  387. import java.io.IOException;
  388. public class App {
  389. public static void main(String[] args) throws IOException, InterruptedException {
  390. try (Environment env = Environments.newInstance("tmp/storage")) {
  391. var app = new KVStoreApp(env);
  392. var server = new GrpcServer(app, 26658);
  393. server.start();
  394. server.blockUntilShutdown();
  395. }
  396. }
  397. }
  398. ```
  399. It is the entry point of the application.
  400. Here we create a special object `Environment`, which knows where to store the application state.
  401. Then we create and start the gRPC server to handle Tendermint Core requests.
  402. Create the `$KVSTORE_HOME/src/main/java/io/example/GrpcServer.java` file with the following content:
  403. ```java
  404. package io.example;
  405. import io.grpc.BindableService;
  406. import io.grpc.Server;
  407. import io.grpc.ServerBuilder;
  408. import java.io.IOException;
  409. class GrpcServer {
  410. private Server server;
  411. GrpcServer(BindableService service, int port) {
  412. this.server = ServerBuilder.forPort(port)
  413. .addService(service)
  414. .build();
  415. }
  416. void start() throws IOException {
  417. server.start();
  418. System.out.println("gRPC server started, listening on $port");
  419. Runtime.getRuntime().addShutdownHook(new Thread(() -> {
  420. System.out.println("shutting down gRPC server since JVM is shutting down");
  421. GrpcServer.this.stop();
  422. System.out.println("server shut down");
  423. }));
  424. }
  425. private void stop() {
  426. server.shutdown();
  427. }
  428. /**
  429. * Await termination on the main thread since the grpc library uses daemon threads.
  430. */
  431. void blockUntilShutdown() throws InterruptedException {
  432. server.awaitTermination();
  433. }
  434. }
  435. ```
  436. ## 1.5 Getting Up and Running
  437. To create a default configuration, nodeKey and private validator files, let's
  438. execute `tendermint init`. But before we do that, we will need to install
  439. Tendermint Core.
  440. ```bash
  441. $ rm -rf /tmp/example
  442. $ cd $GOPATH/src/github.com/tendermint/tendermint
  443. $ make install
  444. $ TMHOME="/tmp/example" tendermint init
  445. I[2019-07-16|18:20:36.480] Generated private validator module=main keyFile=/tmp/example/config/priv_validator_key.json stateFile=/tmp/example2/data/priv_validator_state.json
  446. I[2019-07-16|18:20:36.481] Generated node key module=main path=/tmp/example/config/node_key.json
  447. I[2019-07-16|18:20:36.482] Generated genesis file module=main path=/tmp/example/config/genesis.json
  448. ```
  449. Feel free to explore the generated files, which can be found at
  450. `/tmp/example/config` directory. Documentation on the config can be found
  451. [here](https://docs.tendermint.com/master/tendermint-core/configuration.html).
  452. We are ready to start our application:
  453. ```bash
  454. ./gradlew run
  455. gRPC server started, listening on 26658
  456. ```
  457. Then we need to start Tendermint Core and point it to our application. Staying
  458. within the application directory execute:
  459. ```bash
  460. $ TMHOME="/tmp/example" tendermint node --abci grpc --proxy_app tcp://127.0.0.1:26658
  461. I[2019-07-28|15:44:53.632] Version info module=main software=0.32.1 block=10 p2p=7
  462. I[2019-07-28|15:44:53.677] Starting Node module=main impl=Node
  463. I[2019-07-28|15:44:53.681] Started node module=main nodeInfo="{ProtocolVersion:{P2P:7 Block:10 App:0} ID_:7639e2841ccd47d5ae0f5aad3011b14049d3f452 ListenAddr:tcp://0.0.0.0:26656 Network:test-chain-Nhl3zk Version:0.32.1 Channels:4020212223303800 Moniker:Ivans-MacBook-Pro.local Other:{TxIndex:on RPCAddress:tcp://127.0.0.1:26657}}"
  464. I[2019-07-28|15:44:54.801] Executed block module=state height=8 validTxs=0 invalidTxs=0
  465. I[2019-07-28|15:44:54.814] Committed state module=state height=8 txs=0 appHash=0000000000000000
  466. ```
  467. Now open another tab in your terminal and try sending a transaction:
  468. ```bash
  469. $ curl -s 'localhost:26657/broadcast_tx_commit?tx="tendermint=rocks"'
  470. {
  471. "jsonrpc": "2.0",
  472. "id": "",
  473. "result": {
  474. "check_tx": {
  475. "gasWanted": "1"
  476. },
  477. "deliver_tx": {},
  478. "hash": "CDD3C6DFA0A08CAEDF546F9938A2EEC232209C24AA0E4201194E0AFB78A2C2BB",
  479. "height": "33"
  480. }
  481. ```
  482. Response should contain the height where this transaction was committed.
  483. Now let's check if the given key now exists and its value:
  484. ```bash
  485. $ curl -s 'localhost:26657/abci_query?data="tendermint"'
  486. {
  487. "jsonrpc": "2.0",
  488. "id": "",
  489. "result": {
  490. "response": {
  491. "log": "exists",
  492. "key": "dGVuZGVybWludA==",
  493. "value": "cm9ja3My"
  494. }
  495. }
  496. }
  497. ```
  498. `dGVuZGVybWludA==` and `cm9ja3M=` are the base64-encoding of the ASCII of `tendermint` and `rocks` accordingly.
  499. ## Outro
  500. I hope everything went smoothly and your first, but hopefully not the last,
  501. Tendermint Core application is up and running. If not, please [open an issue on
  502. Github](https://github.com/tendermint/tendermint/issues/new/choose). To dig
  503. deeper, read [the docs](https://docs.tendermint.com/master/).
  504. The full source code of this example project can be found [here](https://github.com/climber73/tendermint-abci-grpc-java).