tendermint

3995 Commits

183 Branches

291 MiB

Tree: 1e3951c61c

Commit Graph

Author	SHA1	Message	Date
Anton Kaliaev	4fc06e9d2a	[libs/pubsub] fix memory leak Refs #1755 I started with writing a test for wsConnection (WebsocketManager) where I: - create a WS connection - do a simple echo call - close it No leaking goroutines, nor any leaking memory were detected. For useful shortcuts see my blog post https://blog.cosmos.network/debugging-the-memory-leak-in-tendermint-210186711420 Then I went to the rpc tests to see if calling Subscribe results in memory growth. It did. I used a slightly modified version of TestHeaderEvents function: ``` func TestHeaderEvents(t testing.T) { // memory heap before f, err := os.Create("/tmp/mem1.mprof") if err != nil { t.Fatal(err) } pprof.WriteHeapProfile(f) f.Close() for i := 0; i < 100; i++ { c := getHTTPClient() err = c.Start() require.Nil(t, err) evtTyp := types.EventNewBlockHeader evt, err := client.WaitForOneEvent(c, evtTyp, waitForEventTimeout) require.Nil(t, err) _, ok := evt.(types.EventDataNewBlockHeader) require.True(t, ok) c.Stop() c = nil } runtime.GC() // memory heap before f, err = os.Create("/tmp/mem2.mprof") if err != nil { t.Fatal(err) } pprof.WriteHeapProfile(f) f.Close() // dump all running goroutines time.Sleep(10 time.Second) pprof.Lookup("goroutine").WriteTo(os.Stdout, 1) } ``` ``` Showing nodes accounting for 35159.16kB, 100% of 35159.16kB total Showing top 10 nodes out of 48 flat flat% sum% cum cum% 32022.23kB 91.08% 91.08% 32022.23kB 91.08% github.com/tendermint/tendermint/libs/pubsub/query.(QueryParser).Init 1056.33kB 3.00% 94.08% 1056.33kB 3.00% bufio.NewReaderSize 528.17kB 1.50% 95.58% 528.17kB 1.50% bufio.NewWriterSize 528.17kB 1.50% 97.09% 528.17kB 1.50% github.com/tendermint/tendermint/consensus.NewConsensusState 512.19kB 1.46% 98.54% 512.19kB 1.46% runtime.malg 512.08kB 1.46% 100% 512.08kB 1.46% syscall.ByteSliceFromString 0 0% 100% 512.08kB 1.46% github.com/tendermint/tendermint/consensus.(ConsensusState).(github.com/tendermint/tendermint/consensus.defaultDecideProposal)-fm 0 0% 100% 512.08kB 1.46% github.com/tendermint/tendermint/consensus.(ConsensusState).addVote 0 0% 100% 512.08kB 1.46% github.com/tendermint/tendermint/consensus.(ConsensusState).defaultDecideProposal 0 0% 100% 512.08kB 1.46% github.com/tendermint/tendermint/consensus.(ConsensusState).enterNewRound ``` 100 subscriptions produce 32MB. Again, no additional goroutines are running after the end of the test (wsConnection readRoutine and writeRoutine both finishes). It means that some exiting goroutine or object is holding a reference to the Query objects, which are leaking.** One of them is pubsub#loop. It's using state.queries to map queries to clients and state.clients to map clients to queries. Before this commit, we're not thoroughly cleaning state.queries, which was the reason for memory leakage.	7 years ago
Anton Kaliaev	6004587347	expect all tags to be strings (#1498 ) * expect all tags to be strings Refs #1369 * port changes from https://github.com/tendermint/tmlibs/pull/204 Refs #1369	7 years ago
Anton Kaliaev	bb9aa85d22	copy events and pubsub packages from tmlibs Refs #847	7 years ago

Author

SHA1

Message

Date

Anton Kaliaev

4fc06e9d2a

[libs/pubsub] fix memory leak

Refs #1755

I started with writing a test for wsConnection (WebsocketManager) where
I:

- create a WS connection
- do a simple echo call
- close it

No leaking goroutines, nor any leaking memory were detected.

For useful shortcuts see my blog post
https://blog.cosmos.network/debugging-the-memory-leak-in-tendermint-210186711420

Then I went to the rpc tests to see if calling Subscribe results in
memory growth. It did.

I used a slightly modified version of TestHeaderEvents function:

```
func TestHeaderEvents(t *testing.T) {
	// memory heap before
	f, err := os.Create("/tmp/mem1.mprof")
	if err != nil {
		t.Fatal(err)
	}
	pprof.WriteHeapProfile(f)
	f.Close()

	for i := 0; i < 100; i++ {
		c := getHTTPClient()
		err = c.Start()
		require.Nil(t, err)

		evtTyp := types.EventNewBlockHeader
		evt, err := client.WaitForOneEvent(c, evtTyp, waitForEventTimeout)
		require.Nil(t, err)
		_, ok := evt.(types.EventDataNewBlockHeader)
		require.True(t, ok)

		c.Stop()
		c = nil
	}

	runtime.GC()

	// memory heap before
	f, err = os.Create("/tmp/mem2.mprof")
	if err != nil {
		t.Fatal(err)
	}
	pprof.WriteHeapProfile(f)
	f.Close()

	// dump all running goroutines
	time.Sleep(10 * time.Second)
	pprof.Lookup("goroutine").WriteTo(os.Stdout, 1)
}
```

```
Showing nodes accounting for 35159.16kB, 100% of 35159.16kB total
Showing top 10 nodes out of 48
      flat  flat%   sum%        cum   cum%
32022.23kB 91.08% 91.08% 32022.23kB 91.08%  github.com/tendermint/tendermint/libs/pubsub/query.(*QueryParser).Init
 1056.33kB  3.00% 94.08%  1056.33kB  3.00%  bufio.NewReaderSize
  528.17kB  1.50% 95.58%   528.17kB  1.50%  bufio.NewWriterSize
  528.17kB  1.50% 97.09%   528.17kB  1.50%  github.com/tendermint/tendermint/consensus.NewConsensusState
  512.19kB  1.46% 98.54%   512.19kB  1.46%  runtime.malg
  512.08kB  1.46%   100%   512.08kB  1.46%  syscall.ByteSliceFromString
         0     0%   100%   512.08kB  1.46%  github.com/tendermint/tendermint/consensus.(*ConsensusState).(github.com/tendermint/tendermint/consensus.defaultDecideProposal)-fm
         0     0%   100%   512.08kB  1.46%  github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote
         0     0%   100%   512.08kB  1.46%  github.com/tendermint/tendermint/consensus.(*ConsensusState).defaultDecideProposal
         0     0%   100%   512.08kB  1.46%  github.com/tendermint/tendermint/consensus.(*ConsensusState).enterNewRound
```

100 subscriptions produce 32MB.

Again, no additional goroutines are running after the end of the test
(wsConnection readRoutine and writeRoutine both finishes). **It means
that some exiting goroutine or object is holding a reference to the
*Query objects, which are leaking.**

One of them is pubsub#loop. It's using state.queries to map queries to
clients and state.clients to map clients to queries.

Before this commit, we're not thoroughly cleaning state.queries, which
was the reason for memory leakage.

7 years ago

Anton Kaliaev

6004587347

expect all tags to be strings (#1498 )

* expect all tags to be strings

Refs #1369

* port changes from https://github.com/tendermint/tmlibs/pull/204

Refs #1369

7 years ago

Anton Kaliaev

bb9aa85d22

copy events and pubsub packages from tmlibs

Refs #847

7 years ago

3 Commits (1e3951c61c1799d1c8a195c8fcf748fe451d6ddf)