Fix evm indexer sync logic

[jeongseup]

PR(Pull Request) Overview

I'm Jeongseup from Cosmostation. And I found a bug to sync blocks at evm indexer service side in the core.

In core at EVM indexer service side, there is breaking point not sync new blocks. that occur a new node cannot sync and index new evm txs.

As you know, the evm indexer service is subscribing to sync a new block.

the full logic is located at (https://github.com/InjectiveFoundation/injective-core/blob/v1.16.0-beta.4/injective-chain/server/indexer_service.go#L48)

But, in this issue, I just took snippet for understanding easily. Source codes at bottom we can see the bug points I already point out with this mark ⚠️. By that points make our node's evm indexer service stuck at that height.

The thing is when the node used external public snapshot, and the snapshot got nil block or block_result at start sync height, eventually evm indexer service in the node cannot sync new blocks

And then, I just fixed with continue. However, this progress is not correct way absolutly. Some of node operators like us or other service provider, they really want to index all blocks without expected case. For that, we need to fix this more.

for {
		if latestBlock <= lastBlock {
			// nothing to index. wait for signal of new block

			select {
			case <-newBlockSignal:
			case <-time.After(NewBlockWaitTimeout):
			}
			continue
		}
		var (
			err         error
			block       *ctypes.ResultBlock
			blockResult *ctypes.ResultBlockResults
		)
		for i := lastBlock + 1; i <= latestBlock; i++ {
			block, err = eis.client.Block(ctx, &i)
			if err != nil {
				if eis.allowGap && strings.Contains(err.Error(), NotFoundErr) {
					continue
				}
				eis.Logger.Error("failed to fetch block", "height", i, "err", err)
				break
			}
                        // ⚠️ by using snapshot, this will be nil or not found -> bug point
			blockResult, err = eis.client.BlockResults(ctx, &i) 
			if err != nil {
   			       // ⚠️ we expected this point can handle above situatiton but it didn't. -> bug point
				if eis.allowGap && strings.Contains(err.Error(), NotFoundErr) { 
					continue
				}
				eis.Logger.Error("failed to fetch block result", "height", i, "err", err)
				break
			}
			if err := eis.txIdxr.IndexBlock(block.Block, blockResult.TxResults); err != nil {
				eis.Logger.Error("failed to index block", "height", i, "err", err)
			}
			lastBlock = blockResult.Height
		}
		if err != nil {
			time.Sleep(ErrorBackoffDuration)
		}
	}

Changes

• Bug fix

Related Issue

Please reference the Issue number this PR addresses: #11

[jeongseup]

Could you plz check this PR for your node? @achilleas-kal @jennamp

[kakysha]

Hey @jeongseup, in the loop of the code snippet you mentioning there is already a continue statement which is triggered when --json-rpc.allow-indexer-gap flag is true. So just set that flag to true via config or CLI flag when you start the node.

[jeongseup]

@kakysha Thank you for your feedback. But in our app.toml, the key was already true because the default value is true

# AllowIndexerGap allow block gap for the custom transaction indexer for the EVM (ethereum transactions).
allow-indexer-gap = true

But, at that time, EVM indexer service in our node was not working. Could you please test about that with https://polkachu.com/testnets/injective/snapshots this snapshot?

[jeongseup]

If it's really working well, this PR isn't needed to fix it. Sorry to bother your team.

[kakysha]

I tried restoring from the Polkachu testnet snapshot and I don't see any errors in the log, neither my blockResults on that lines that you changed are never nil @jeongseup.

Need more info to reproduce the problem.

[maxim-inj]

@kakysha this is a good suggestion, because

if eis.allowGap && strings.Contains(err.Error(), NotFoundErr) {
     continue
}

It only continues if error explicitly contains a pattern (+ that pattern is from cometbft 0.37.4). The error might be different (yet ephemeral), that will result in break unfortunately. It's not about allowGap only.

My idea of a fix is to treat every error as ephemeral, have single retry, finally always continue.

[maxim-inj]

++ oke you're right, it breaks only the internal loop

[jeongseup]

Umm.. but anyway working well with the allowGap flag(--json-rpc.allow-indexer-gap). @kakysha

I didn't know what was the exact problem.

However, our backend team have a problem to query about eth_getTransactionReceipt method at evm-indexer module. and then I found a something stuck moment.

curl --location 'https://testnet.sentry.chain.json-rpc.injective.network/' \
--header 'Content-Type: application/json' \
--header 'Cookie: lb=a5e16ea18caf85bb44b0f90de9924f7d86e8f3145eeaa6ed0115e2893f69c8ff' \
--data '
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "eth_getTransactionReceipt",
    "params": ["0xfcbb2be314ebce4d703fb70d676859f021829914498a6694406059968dd38a31"]
}
'

=> Response
{"jsonrpc":"2.0","id":1,"result":null}

As you already said and pointed out, optimistically it should be skipped(by continue in the code). But our side might be not.

I thought the point was the evm indexing stuck was caused by failed to fetch block result... logs

That's why I suggested this PR.

Anyway, now the node is up and running and then eth_getTransactionReceipt method is also working well Really. I'm not sure why, the log is still being after running a injective fullnode. try like that injectived start --json-rpc.allow-indexer-gap true --log-level error with new snapshot data.

Thank you for reviewing this PR. If it was something like bother, I'm sorry to you guys. @kakysha @maxim-inj