You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

614 lines
19 KiB

  1. # Creating an application in Java
  2. ## Guide Assumptions
  3. This guide is designed for beginners who want to get started with a Tendermint
  4. Core application from scratch. It does not assume that you have any prior
  5. experience with Tendermint Core.
  6. Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state
  7. transition machine (your application) - written in any programming language - and securely
  8. replicates it on many machines.
  9. By following along with this guide, you'll create a Tendermint Core project
  10. called kvstore, a (very) simple distributed BFT key-value store. The application (which should
  11. implementing the blockchain interface (ABCI)) will be written in Java.
  12. This guide assumes that you are not new to JVM world. If you are new please see [JVM Minimal Survival Guide](https://hadihariri.com/2013/12/29/jvm-minimal-survival-guide-for-the-dotnet-developer/#java-the-language-java-the-ecosystem-java-the-jvm) and [Gradle Docs](https://docs.gradle.org/current/userguide/userguide.html).
  13. ## Built-in app vs external app
  14. If you use Golang, you can run your app and Tendermint Core in the same process to get maximum performance.
  15. [Cosmos SDK](https://github.com/cosmos/cosmos-sdk) is written this way.
  16. Please refer to [Writing a built-in Tendermint Core application in Go](./go-built-in.md) guide for details.
  17. If you choose another language, like we did in this guide, you have to write a separate app,
  18. which will communicate with Tendermint Core via a socket (UNIX or TCP) or gRPC.
  19. This guide will show you how to build external application using RPC server.
  20. Having a separate application might give you better security guarantees as two
  21. processes would be communicating via established binary protocol. Tendermint
  22. Core will not have access to application's state.
  23. ## 1.1 Installing Java and Gradle
  24. Please refer to [the Oracle's guide for installing JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html).
  25. Verify that you have installed Java successfully:
  26. ```sh
  27. $ java -version
  28. java version "12.0.2" 2019-07-16
  29. Java(TM) SE Runtime Environment (build 12.0.2+10)
  30. Java HotSpot(TM) 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)
  31. ```
  32. You can choose any version of Java higher or equal to 8.
  33. This guide is written using Java SE Development Kit 12.
  34. Make sure you have `$JAVA_HOME` environment variable set:
  35. ```sh
  36. $ echo $JAVA_HOME
  37. /Library/Java/JavaVirtualMachines/jdk-12.0.2.jdk/Contents/Home
  38. ```
  39. For Gradle installation, please refer to [their official guide](https://gradle.org/install/).
  40. ## 1.2 Creating a new Java project
  41. We'll start by creating a new Gradle project.
  42. ```sh
  43. $ export KVSTORE_HOME=~/kvstore
  44. $ mkdir $KVSTORE_HOME
  45. $ cd $KVSTORE_HOME
  46. ```
  47. Inside the example directory run:
  48. ```sh
  49. gradle init --dsl groovy --package io.example --project-name example --type java-application --test-framework junit
  50. ```
  51. This will create a new project for you. The tree of files should look like:
  52. ```sh
  53. $ tree
  54. .
  55. |-- build.gradle
  56. |-- gradle
  57. | `-- wrapper
  58. | |-- gradle-wrapper.jar
  59. | `-- gradle-wrapper.properties
  60. |-- gradlew
  61. |-- gradlew.bat
  62. |-- settings.gradle
  63. `-- src
  64. |-- main
  65. | |-- java
  66. | | `-- io
  67. | | `-- example
  68. | | `-- App.java
  69. | `-- resources
  70. `-- test
  71. |-- java
  72. | `-- io
  73. | `-- example
  74. | `-- AppTest.java
  75. `-- resources
  76. ```
  77. When run, this should print "Hello world." to the standard output.
  78. ```sh
  79. $ ./gradlew run
  80. > Task :run
  81. Hello world.
  82. ```
  83. ## 1.3 Writing a Tendermint Core application
  84. Tendermint Core communicates with the application through the Application
  85. BlockChain Interface (ABCI). All message types are defined in the [protobuf
  86. file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto).
  87. This allows Tendermint Core to run applications written in any programming
  88. language.
  89. ### 1.3.1 Compile .proto files
  90. Add the following piece to the top of the `build.gradle`:
  91. ```groovy
  92. buildscript {
  93. repositories {
  94. mavenCentral()
  95. }
  96. dependencies {
  97. classpath 'com.google.protobuf:protobuf-gradle-plugin:0.8.8'
  98. }
  99. }
  100. ```
  101. Enable the protobuf plugin in the `plugins` section of the `build.gradle`:
  102. ```groovy
  103. plugins {
  104. id 'com.google.protobuf' version '0.8.8'
  105. }
  106. ```
  107. Add the following code to `build.gradle`:
  108. ```groovy
  109. protobuf {
  110. protoc {
  111. artifact = "com.google.protobuf:protoc:3.7.1"
  112. }
  113. plugins {
  114. grpc {
  115. artifact = 'io.grpc:protoc-gen-grpc-java:1.22.1'
  116. }
  117. }
  118. generateProtoTasks {
  119. all()*.plugins {
  120. grpc {}
  121. }
  122. }
  123. }
  124. ```
  125. Now we should be ready to compile the `*.proto` files.
  126. Copy the necessary `.proto` files to your project:
  127. ```sh
  128. mkdir -p \
  129. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/abci/types \
  130. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/crypto/merkle \
  131. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/libs/common \
  132. $KVSTORE_HOME/src/main/proto/github.com/gogo/protobuf/gogoproto
  133. cp $GOPATH/src/github.com/tendermint/tendermint/abci/types/types.proto \
  134. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/abci/types/types.proto
  135. cp $GOPATH/src/github.com/tendermint/tendermint/crypto/merkle/merkle.proto \
  136. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/crypto/merkle/merkle.proto
  137. cp $GOPATH/src/github.com/tendermint/tendermint/libs/common/types.proto \
  138. $KVSTORE_HOME/src/main/proto/github.com/tendermint/tendermint/libs/common/types.proto
  139. cp $GOPATH/src/github.com/gogo/protobuf/gogoproto/gogo.proto \
  140. $KVSTORE_HOME/src/main/proto/github.com/gogo/protobuf/gogoproto/gogo.proto
  141. ```
  142. Add these dependencies to `build.gradle`:
  143. ```groovy
  144. dependencies {
  145. implementation 'io.grpc:grpc-protobuf:1.22.1'
  146. implementation 'io.grpc:grpc-netty-shaded:1.22.1'
  147. implementation 'io.grpc:grpc-stub:1.22.1'
  148. }
  149. ```
  150. To generate all protobuf-type classes run:
  151. ```sh
  152. ./gradlew generateProto
  153. ```
  154. To verify that everything went smoothly, you can inspect the `build/generated/` directory:
  155. ```sh
  156. $ tree build/generated/
  157. build/generated/
  158. |-- source
  159. | `-- proto
  160. | `-- main
  161. | |-- grpc
  162. | | `-- types
  163. | | `-- ABCIApplicationGrpc.java
  164. | `-- java
  165. | |-- com
  166. | | `-- google
  167. | | `-- protobuf
  168. | | `-- GoGoProtos.java
  169. | |-- common
  170. | | `-- Types.java
  171. | |-- merkle
  172. | | `-- Merkle.java
  173. | `-- types
  174. | `-- Types.java
  175. ```
  176. ### 1.3.2 Implementing ABCI
  177. The resulting `$KVSTORE_HOME/build/generated/source/proto/main/grpc/types/ABCIApplicationGrpc.java` file
  178. contains the abstract class `ABCIApplicationImplBase`, which is an interface we'll need to implement.
  179. Create `$KVSTORE_HOME/src/main/java/io/example/KVStoreApp.java` file with the following content:
  180. ```java
  181. package io.example;
  182. import io.grpc.stub.StreamObserver;
  183. import types.ABCIApplicationGrpc;
  184. import types.Types.*;
  185. class KVStoreApp extends ABCIApplicationGrpc.ABCIApplicationImplBase {
  186. // methods implementation
  187. }
  188. ```
  189. Now I will go through each method of `ABCIApplicationImplBase` explaining when it's called and adding
  190. required business logic.
  191. ### 1.3.3 CheckTx
  192. When a new transaction is added to the Tendermint Core, it will ask the
  193. application to check it (validate the format, signatures, etc.).
  194. ```java
  195. @Override
  196. public void checkTx(RequestCheckTx req, StreamObserver<ResponseCheckTx> responseObserver) {
  197. var tx = req.getTx();
  198. int code = validate(tx);
  199. var resp = ResponseCheckTx.newBuilder()
  200. .setCode(code)
  201. .setGasWanted(1)
  202. .build();
  203. responseObserver.onNext(resp);
  204. responseObserver.onCompleted();
  205. }
  206. private int validate(ByteString tx) {
  207. List<byte[]> parts = split(tx, '=');
  208. if (parts.size() != 2) {
  209. return 1;
  210. }
  211. byte[] key = parts.get(0);
  212. byte[] value = parts.get(1);
  213. // check if the same key=value already exists
  214. var stored = getPersistedValue(key);
  215. if (stored != null && Arrays.equals(stored, value)) {
  216. return 2;
  217. }
  218. return 0;
  219. }
  220. private List<byte[]> split(ByteString tx, char separator) {
  221. var arr = tx.toByteArray();
  222. int i;
  223. for (i = 0; i < tx.size(); i++) {
  224. if (arr[i] == (byte)separator) {
  225. break;
  226. }
  227. }
  228. if (i == tx.size()) {
  229. return Collections.emptyList();
  230. }
  231. return List.of(
  232. tx.substring(0, i).toByteArray(),
  233. tx.substring(i + 1).toByteArray()
  234. );
  235. }
  236. ```
  237. Don't worry if this does not compile yet.
  238. If the transaction does not have a form of `{bytes}={bytes}`, we return `1`
  239. code. When the same key=value already exist (same key and value), we return `2`
  240. code. For others, we return a zero code indicating that they are valid.
  241. Note that anything with non-zero code will be considered invalid (`-1`, `100`,
  242. etc.) by Tendermint Core.
  243. Valid transactions will eventually be committed given they are not too big and
  244. have enough gas. To learn more about gas, check out ["the
  245. specification"](https://tendermint.com/docs/spec/abci/apps.html#gas).
  246. For the underlying key-value store we'll use
  247. [JetBrains Xodus](https://github.com/JetBrains/xodus), which is a transactional schema-less embedded high-performance database written in Java.
  248. `build.gradle`:
  249. ```groovy
  250. dependencies {
  251. implementation 'org.jetbrains.xodus:xodus-environment:1.3.91'
  252. }
  253. ```
  254. ```java
  255. ...
  256. import jetbrains.exodus.ArrayByteIterable;
  257. import jetbrains.exodus.ByteIterable;
  258. import jetbrains.exodus.env.Environment;
  259. import jetbrains.exodus.env.Store;
  260. import jetbrains.exodus.env.StoreConfig;
  261. import jetbrains.exodus.env.Transaction;
  262. class KVStoreApp extends ABCIApplicationGrpc.ABCIApplicationImplBase {
  263. private Environment env;
  264. private Transaction txn = null;
  265. private Store store = null;
  266. KVStoreApp(Environment env) {
  267. this.env = env;
  268. }
  269. ...
  270. private byte[] getPersistedValue(byte[] k) {
  271. return env.computeInReadonlyTransaction(txn -> {
  272. var store = env.openStore("store", StoreConfig.WITHOUT_DUPLICATES, txn);
  273. ByteIterable byteIterable = store.get(txn, new ArrayByteIterable(k));
  274. if (byteIterable == null) {
  275. return null;
  276. }
  277. return byteIterable.getBytesUnsafe();
  278. });
  279. }
  280. }
  281. ```
  282. ### 1.3.4 BeginBlock -> DeliverTx -> EndBlock -> Commit
  283. When Tendermint Core has decided on the block, it's transferred to the
  284. application in 3 parts: `BeginBlock`, one `DeliverTx` per transaction and
  285. `EndBlock` in the end. `DeliverTx` are being transferred asynchronously, but the
  286. responses are expected to come in order.
  287. ```java
  288. @Override
  289. public void beginBlock(RequestBeginBlock req, StreamObserver<ResponseBeginBlock> responseObserver) {
  290. txn = env.beginTransaction();
  291. store = env.openStore("store", StoreConfig.WITHOUT_DUPLICATES, txn);
  292. var resp = ResponseBeginBlock.newBuilder().build();
  293. responseObserver.onNext(resp);
  294. responseObserver.onCompleted();
  295. }
  296. ```
  297. Here we begin a new transaction, which will accumulate the block's transactions and open the corresponding store.
  298. ```java
  299. @Override
  300. public void deliverTx(RequestDeliverTx req, StreamObserver<ResponseDeliverTx> responseObserver) {
  301. var tx = req.getTx();
  302. int code = validate(tx);
  303. if (code == 0) {
  304. List<byte[]> parts = split(tx, '=');
  305. var key = new ArrayByteIterable(parts.get(0));
  306. var value = new ArrayByteIterable(parts.get(1));
  307. store.put(txn, key, value);
  308. }
  309. var resp = ResponseDeliverTx.newBuilder()
  310. .setCode(code)
  311. .build();
  312. responseObserver.onNext(resp);
  313. responseObserver.onCompleted();
  314. }
  315. ```
  316. If the transaction is badly formatted or the same key=value already exist, we
  317. again return the non-zero code. Otherwise, we add it to the store.
  318. In the current design, a block can include incorrect transactions (those who
  319. passed `CheckTx`, but failed `DeliverTx` or transactions included by the proposer
  320. directly). This is done for performance reasons.
  321. Note we can't commit transactions inside the `DeliverTx` because in such case
  322. `Query`, which may be called in parallel, will return inconsistent data (i.e.
  323. it will report that some value already exist even when the actual block was not
  324. yet committed).
  325. `Commit` instructs the application to persist the new state.
  326. ```java
  327. @Override
  328. public void commit(RequestCommit req, StreamObserver<ResponseCommit> responseObserver) {
  329. txn.commit();
  330. var resp = ResponseCommit.newBuilder()
  331. .setData(ByteString.copyFrom(new byte[8]))
  332. .build();
  333. responseObserver.onNext(resp);
  334. responseObserver.onCompleted();
  335. }
  336. ```
  337. ### 1.3.5 Query
  338. Now, when the client wants to know whenever a particular key/value exist, it
  339. will call Tendermint Core RPC `/abci_query` endpoint, which in turn will call
  340. the application's `Query` method.
  341. Applications are free to provide their own APIs. But by using Tendermint Core
  342. as a proxy, clients (including [light client
  343. package](https://godoc.org/github.com/tendermint/tendermint/lite)) can leverage
  344. the unified API across different applications. Plus they won't have to call the
  345. otherwise separate Tendermint Core API for additional proofs.
  346. Note we don't include a proof here.
  347. ```java
  348. @Override
  349. public void query(RequestQuery req, StreamObserver<ResponseQuery> responseObserver) {
  350. var k = req.getData().toByteArray();
  351. var v = getPersistedValue(k);
  352. var builder = ResponseQuery.newBuilder();
  353. if (v == null) {
  354. builder.setLog("does not exist");
  355. } else {
  356. builder.setLog("exists");
  357. builder.setKey(ByteString.copyFrom(k));
  358. builder.setValue(ByteString.copyFrom(v));
  359. }
  360. responseObserver.onNext(builder.build());
  361. responseObserver.onCompleted();
  362. }
  363. ```
  364. The complete specification can be found
  365. [here](https://tendermint.com/docs/spec/abci/).
  366. ## 1.4 Starting an application and a Tendermint Core instances
  367. Put the following code into the `$KVSTORE_HOME/src/main/java/io/example/App.java` file:
  368. ```java
  369. package io.example;
  370. import jetbrains.exodus.env.Environment;
  371. import jetbrains.exodus.env.Environments;
  372. import java.io.IOException;
  373. public class App {
  374. public static void main(String[] args) throws IOException, InterruptedException {
  375. try (Environment env = Environments.newInstance("tmp/storage")) {
  376. var app = new KVStoreApp(env);
  377. var server = new GrpcServer(app, 26658);
  378. server.start();
  379. server.blockUntilShutdown();
  380. }
  381. }
  382. }
  383. ```
  384. It is the entry point of the application.
  385. Here we create a special object `Environment`, which knows where to store the application state.
  386. Then we create and start the gRPC server to handle Tendermint Core requests.
  387. Create the `$KVSTORE_HOME/src/main/java/io/example/GrpcServer.java` file with the following content:
  388. ```java
  389. package io.example;
  390. import io.grpc.BindableService;
  391. import io.grpc.Server;
  392. import io.grpc.ServerBuilder;
  393. import java.io.IOException;
  394. class GrpcServer {
  395. private Server server;
  396. GrpcServer(BindableService service, int port) {
  397. this.server = ServerBuilder.forPort(port)
  398. .addService(service)
  399. .build();
  400. }
  401. void start() throws IOException {
  402. server.start();
  403. System.out.println("gRPC server started, listening on $port");
  404. Runtime.getRuntime().addShutdownHook(new Thread(() -> {
  405. System.out.println("shutting down gRPC server since JVM is shutting down");
  406. GrpcServer.this.stop();
  407. System.out.println("server shut down");
  408. }));
  409. }
  410. private void stop() {
  411. server.shutdown();
  412. }
  413. /**
  414. * Await termination on the main thread since the grpc library uses daemon threads.
  415. */
  416. void blockUntilShutdown() throws InterruptedException {
  417. server.awaitTermination();
  418. }
  419. }
  420. ```
  421. ## 1.5 Getting Up and Running
  422. To create a default configuration, nodeKey and private validator files, let's
  423. execute `tendermint init`. But before we do that, we will need to install
  424. Tendermint Core.
  425. ```sh
  426. $ rm -rf /tmp/example
  427. $ cd $GOPATH/src/github.com/tendermint/tendermint
  428. $ make install
  429. $ TMHOME="/tmp/example" tendermint init
  430. I[2019-07-16|18:20:36.480] Generated private validator module=main keyFile=/tmp/example/config/priv_validator_key.json stateFile=/tmp/example2/data/priv_validator_state.json
  431. I[2019-07-16|18:20:36.481] Generated node key module=main path=/tmp/example/config/node_key.json
  432. I[2019-07-16|18:20:36.482] Generated genesis file module=main path=/tmp/example/config/genesis.json
  433. ```
  434. Feel free to explore the generated files, which can be found at
  435. `/tmp/example/config` directory. Documentation on the config can be found
  436. [here](https://tendermint.com/docs/tendermint-core/configuration.html).
  437. We are ready to start our application:
  438. ```sh
  439. ./gradlew run
  440. gRPC server started, listening on 26658
  441. ```
  442. Then we need to start Tendermint Core and point it to our application. Staying
  443. within the application directory execute:
  444. ```sh
  445. $ TMHOME="/tmp/example" tendermint node --abci grpc --proxy_app tcp://127.0.0.1:26658
  446. I[2019-07-28|15:44:53.632] Version info module=main software=0.32.1 block=10 p2p=7
  447. I[2019-07-28|15:44:53.677] Starting Node module=main impl=Node
  448. I[2019-07-28|15:44:53.681] Started node module=main nodeInfo="{ProtocolVersion:{P2P:7 Block:10 App:0} ID_:7639e2841ccd47d5ae0f5aad3011b14049d3f452 ListenAddr:tcp://0.0.0.0:26656 Network:test-chain-Nhl3zk Version:0.32.1 Channels:4020212223303800 Moniker:Ivans-MacBook-Pro.local Other:{TxIndex:on RPCAddress:tcp://127.0.0.1:26657}}"
  449. I[2019-07-28|15:44:54.801] Executed block module=state height=8 validTxs=0 invalidTxs=0
  450. I[2019-07-28|15:44:54.814] Committed state module=state height=8 txs=0 appHash=0000000000000000
  451. ```
  452. Now open another tab in your terminal and try sending a transaction:
  453. ```sh
  454. $ curl -s 'localhost:26657/broadcast_tx_commit?tx="tendermint=rocks"'
  455. {
  456. "jsonrpc": "2.0",
  457. "id": "",
  458. "result": {
  459. "check_tx": {
  460. "gasWanted": "1"
  461. },
  462. "deliver_tx": {},
  463. "hash": "CDD3C6DFA0A08CAEDF546F9938A2EEC232209C24AA0E4201194E0AFB78A2C2BB",
  464. "height": "33"
  465. }
  466. ```
  467. Response should contain the height where this transaction was committed.
  468. Now let's check if the given key now exists and its value:
  469. ```sh
  470. $ curl -s 'localhost:26657/abci_query?data="tendermint"'
  471. {
  472. "jsonrpc": "2.0",
  473. "id": "",
  474. "result": {
  475. "response": {
  476. "log": "exists",
  477. "key": "dGVuZGVybWludA==",
  478. "value": "cm9ja3My"
  479. }
  480. }
  481. }
  482. ```
  483. `dGVuZGVybWludA==` and `cm9ja3M=` are the base64-encoding of the ASCII of `tendermint` and `rocks` accordingly.
  484. ## Outro
  485. I hope everything went smoothly and your first, but hopefully not the last,
  486. Tendermint Core application is up and running. If not, please [open an issue on
  487. Github](https://github.com/tendermint/tendermint/issues/new/choose). To dig
  488. deeper, read [the docs](https://tendermint.com/docs/).
  489. The full source code of this example project can be found [here](https://github.com/climber73/tendermint-abci-grpc-java).