Add prepared statements performance optimisation to increase observation performance x2 #534
Description
I've faced an issue with tarantool/metrics poor performance in a following scenario:
local function observability_middleware(next)
local called_func = _G.onedb_state.initing_func
local called_app = _G.onedb_state.loading_app
return function (...)
local timer = es_timer:new()
timer:start()
local result = { pcall(next, ...) }
local ok = result[1]
timer:stop()
onedb_stat.api_requests_latency:observe(timer:latency(), { method = called_func, app = called_app })
if not ok then
onedb_stat.api_requests_counter:inc(1, { method = called_func, app = called_app, state = 'fail' })
error(result[2])
end
onedb_stat.api_requests_counter:inc(1, { method = called_func, app = called_app, state = 'success' })
return unpack(result, 2, table.maxn(result))
end
end^ The idea is to wrap each applications' API function with a given middleware to track latency/throughput/errors on a per-function basis
I've performed a synthetic performance test, on my setup I've had the following results:
- tarantool with no middleware got 40k RPS,
; - tarantool with middleware got 15k RPS,
.
In the second falmegraph I see an issue with slow histogram observations. Three main reasons for that are:
- intensive
label_pairscopying to fill in thelelabel; - intensive work with strings in
make_keyfor each observation; - GC pressure caused by temporary strings/tables deallocation.
My idea is to introduce "prepared statements". My labels have low cardinality, which means I could make an object that have:
- a collector reference;
- immutable
label_parisset; - cached
key, that is computed once and then could be used for infinite number of new observations.
Usage example:
local Histogram = require('metrics.collectors.histogram')
local h = Histogram:new('hist', 'some histogram', {2, 4})
local prepared= h:prepare({label = 1})
prepared:observe(3)
prepared:observe(5)
prepared:observe(7)Metadata
Metadata
Assignees
Labels
No labels