zolfa
/
tendermint
1
0
Fork 0
Code Issues 0 Pull Requests 0 Projects 0 Releases 221 Wiki Activity
Browse Source

ADR: Protocol Buffers Management (#8029) 

* First draft of protobuf management ADR

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Pre-emptively add ADR to "Accepted" section in README

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Add missing prototool link

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Elaborate on positive consequences of decision

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Add clang-format GA to detailed design

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Fix broken link

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Add notes on automated docs generation

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Rewording and restructuring for clarity

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Grammatical fixes and elaborations

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Revise wording for clarity

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Address comments

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Update ADR to reflect current consensus on Buf

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Minor grammar fix

Signed-off-by: Thane Thomson <connect@thanethomson.com>

Co-authored-by: M. J. Fromberger <fromberger@interchain.io>
pull/8128/head
Thane Thomson 3 years ago
committed by GitHub
parent
faf123bda2
commit
4dce885994
No known key found for this signature in database GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 202 additions and 0 deletions
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. +1
    -0
      docs/architecture/README.md
  2. +201
    -0
      docs/architecture/adr-081-protobuf-mgmt.md

+ 1
- 0
docs/architecture/README.md View File

@ -85,6 +85,7 @@ Note the context/background should be written in the present tense.
- [ADR-067: Mempool Refactor](./adr-067-mempool-refactor.md) - [ADR-067: Mempool Refactor](./adr-067-mempool-refactor.md)
- [ADR-075: RPC Event Subscription Interface](./adr-075-rpc-subscription.md) - [ADR-075: RPC Event Subscription Interface](./adr-075-rpc-subscription.md)
- [ADR-076: Combine Spec and Tendermint Repositories](./adr-076-combine-spec-repo.md) - [ADR-076: Combine Spec and Tendermint Repositories](./adr-076-combine-spec-repo.md)
- [ADR-081: Protocol Buffers Management](./adr-081-protobuf-mgmt.md)
### Deprecated ### Deprecated


+ 201
- 0
docs/architecture/adr-081-protobuf-mgmt.md View File

@ -0,0 +1,201 @@
# ADR 081: Protocol Buffers Management
## Changelog
- 2022-02-28: First draft
## Status
Accepted
[Tracking issue](https://github.com/tendermint/tendermint/issues/8121)
## Context
At present, we manage the [Protocol Buffers] schema files ("protos") that define
our wire-level data formats within the Tendermint repository itself (see the
[`proto`](../../proto/) directory). Recently, we have been making use of [Buf],
both locally and in CI, in order to generate Go stubs, and lint and check
`.proto` files for breaking changes.
The version of Buf used at the time of this decision was `v1beta1`, and it was
discussed in [\#7975] and in weekly calls as to whether we should upgrade to
`v1` and harmonize our approach with that used by the Cosmos SDK. The team
managing the Cosmos SDK was primarily interested in having our protos versioned
and easily accessible from the [Buf] registry.
The three main sets of stakeholders for the `.proto` files and their needs, as
currently understood, are as follows.
1. Tendermint needs Go code generated from `.proto` files.
2. Consumers of Tendermint's `.proto` files, specifically projects that want to
interoperate with Tendermint and need to generate code for their own
programming language, want to be able to access these files in a reliable and
efficient way.
3. The Tendermint Core team wants to provide stable interfaces that are as easy
as possible to maintain, on which consumers can depend, and to be able to
notify those consumers promptly when those interfaces change. To this end, we
want to:
1. Prevent any breaking changes from being introduced in minor/patch releases
of Tendermint. Only major version updates should be able to contain
breaking interface changes.
2. Prevent generated code from diverging from the Protobuf schema files.
There was also discussion surrounding the notion of automated documentation
generation and hosting, but it is not clear at this time whether this would be
that valuable to any of our stakeholders. What will, of course, be valuable at
minimum would be better documentation (in comments) of the `.proto` files
themselves.
## Alternative Approaches
### Meeting stakeholders' needs
1. Go stub generation from protos. We could use:
1. [Buf]. This approach has been rather cumbersome up to this point, and it
is not clear what Buf really provides beyond that which `protoc` provides
to justify the additional complexity in configuring Buf for stub
generation.
2. [protoc] - the Protocol Buffers compiler.
2. Notification of breaking changes:
1. Buf in CI for all pull requests to *release* branches only (and not on
`master`).
2. Buf in CI on every pull request to every branch (this was the case at the
time of this decision, and the team decided that the signal-to-noise ratio
for this approach was too low to be of value).
3. `.proto` linting:
1. Buf in CI on every pull request
4. `.proto` formatting:
1. [clang-format] locally and a [clang-format GitHub Action] in CI to check
that files are formatted properly on every pull request.
5. Sharing of `.proto` files in a versioned, reliable manner:
1. Consumers could simply clone the Tendermint repository, check out a
specific commit, tag or branch and manually copy out all of the `.proto`
files they need. This requires no effort from the Tendermint Core team and
will continue to be an option for consumers. The drawback of this approach
is that it requires manual coding/scripting to implement and is brittle in
the face of bigger changes.
2. Uploading our `.proto` files to Buf's registry on every release. This is
by far the most seamless for consumers of our `.proto` files, but requires
the dependency on Buf. This has the additional benefit that the Buf
registry will automatically [generate and host
documentation][buf-docs-gen] for these protos.
3. We could create a process that, upon release, creates a `.zip` file
containing our `.proto` files.
### Popular alternatives to Buf
[Prototool] was not considered as it appears deprecated, and the ecosystem seems
to be converging on Buf at this time.
### Tooling complexity
The more tools we have in our build/CI processes, the more complex and fragile
repository/CI management becomes, and the longer it takes to onboard new team
members. Maintainability is a core concern here.
### Buf sustainability and costs
One of the primary considerations regarding the usage of Buf is whether, for
example, access to its registry will eventually become a
paid-for/subscription-based service and whether this is valuable enough for us
and the ecosystem to pay for such a service. At this time, it appears as though
Buf will never charge for hosting open source projects' protos.
Another consideration was Buf's sustainability as a project - what happens when
their resources run out? Will there be a strong and broad enough open source
community to continue maintaining it?
### Local Buf usage options
Local usage of Buf (i.e. not in CI) can be accomplished in two ways:
1. Installing the relevant tools individually.
2. By way of its [Docker image][buf-docker].
Local installation of Buf requires developers to manually keep their toolchains
up-to-date. The Docker option comes with a number of complexities, including
how the file system permissions of code generated by a Docker container differ
between platforms (e.g. on Linux, Buf-generated code ends up being owned by
`root`).
The trouble with the Docker-based approach is that we make use of the
[gogoprotobuf] plugin for `protoc`. Continuing to use the Docker-based approach
to using Buf will mean that we will have to continue building our own custom
Docker image with embedded gogoprotobuf.
Along these lines, we could eventually consider coming up with a [Nix]- or
[redo]-based approach to developer tooling to ensure tooling consistency across
the team and for anyone who wants to be able to contribute to Tendermint.
## Decision
1. We will adopt Buf for now for proto generation, linting, breakage checking
and its registry (mainly in CI, with optional usage locally).
2. Failing CI when checking for breaking changes in `.proto` files will only
happen when performing minor/patch releases.
3. Local tooling will be favored over Docker-based tooling.
## Detailed Design
We currently aim to:
1. Update to Buf `v1` to facilitate linting, breakage checking and uploading to
the Buf registry.
2. Configure CI appropriately for proto management:
1. Uploading protos to the Buf registry on every release (e.g. the
[approach][cosmos-sdk-buf-registry-ci] used by the Cosmos SDK).
2. Linting on every pull request (e.g. the
[approach][cosmos-sdk-buf-linting-ci] used by the Cosmos SDK). The linter
passing should be considered a requirement for accepting PRs.
3. Checking for breaking changes in minor/patch version releases and failing
CI accordingly - see [\#8003].
4. Add [clang-format GitHub Action] to check `.proto` file formatting. Format
checking should be considered a requirement for accepting PRs.
3. Update the Tendermint [`Makefile`](../../Makefile) to primarily facilitate
local Protobuf stub generation, linting, formatting and breaking change
checking. More specifically:
1. This includes removing the dependency on Docker and introducing the
dependency on local toolchain installation. CI-based equivalents, where
relevant, will rely on specific GitHub Actions instead of the Makefile.
2. Go code generation will rely on `protoc` directly.
## Consequences
### Positive
- We will still offer Go stub generation, proto linting and breakage checking.
- Breakage checking will only happen on minor/patch releases to increase the
signal-to-noise ratio in CI.
- Versioned protos will be made available via Buf's registry upon every release.
### Negative
- Developers/contributors will need to install the relevant Protocol
Buffers-related tooling (Buf, gogoprotobuf, clang-format) locally in order to
build, lint, format and check `.proto` files for breaking changes.
### Neutral
## References
- [Protocol Buffers]
- [Buf]
- [\#7975]
- [protoc] - The Protocol Buffers compiler
[Protocol Buffers]: https://developers.google.com/protocol-buffers
[Buf]: https://buf.build/
[\#7975]: https://github.com/tendermint/tendermint/pull/7975
[protoc]: https://github.com/protocolbuffers/protobuf
[clang-format]: https://clang.llvm.org/docs/ClangFormat.html
[clang-format GitHub Action]: https://github.com/marketplace/actions/clang-format-github-action
[buf-docker]: https://hub.docker.com/r/bufbuild/buf
[cosmos-sdk-buf-registry-ci]: https://github.com/cosmos/cosmos-sdk/blob/e6571906043b6751951a42b6546431b1c38b05bd/.github/workflows/proto-registry.yml
[cosmos-sdk-buf-linting-ci]: https://github.com/cosmos/cosmos-sdk/blob/e6571906043b6751951a42b6546431b1c38b05bd/.github/workflows/proto.yml#L15
[\#8003]: https://github.com/tendermint/tendermint/issues/8003
[Nix]: https://nixos.org/
[gogoprotobuf]: https://github.com/gogo/protobuf
[Prototool]: https://github.com/uber/prototool
[buf-docs-gen]: https://docs.buf.build/bsr/documentation
[redo]: https://redo.readthedocs.io/en/latest/

Write Preview
Loading…
Cancel
Save