Optimize record file query count in web3

[bilyana-gospodinova]

Problem

As a result of the performed testing in #12164 the following observations were made:

1. Among all record files the latest one is read the most frequently. It is always the same for all requests in the 2-second window that is valid. Currently it is cached for 500ms and this can be increased up to 2s or at least 1s as we don't need it refreshed that frequently.
2. There are way too many DB requests that are made concurrently to read the latest record file before it gets cached. In numbers: with only 20% of the real world traffic over a 20-second period there are over 850 actual DB queries (not repository calls) for the latest record file. Even in 3-4 ms period about 20 actual DB calls are made (the average time for the repository call completion is about 80ms). This takes 20 DB connections from the pool that can be used for other queries that are really needed.

Solution

Find a way to optimize the record file DB queries. Ideally, only 1 thread should make the call for the latest record file and all others should just wait for it and read it. Then after 2 seconds when the file expires, it needs to be read again.
Another suggestion is to make the repository call lazily and skip the record file reading wherever possible.

Alternatives

No response

[sdimitrov9]

Logs on the db container were activated to confirm that there is unexpected number of get latest record file queries executed against the database. Hundreds of api requests were executed within 20 seconds, which resulted in ~200
"select * from record_file order by consensus_end desc limit 1" logs in the db container which proves there is a cache stampede indeed(otherwise the queries should have around 40, since the cache expires on 500ms). Using scheduler to preload the latest record file and synchronizing the caffeine cache (sync = true) resolved the problem with the excessive number of queries, but unfortunately the performance tests didn't show noticeable improvement(if any) in execution time.
The next step is to look if we can entirely cut calls for the latest records files in certain cases.