Handle multiple metrics per benchmark

[wjones127]

In lance-format/lance#5461, I plan on adding a bunch of new benchmark metrics:

1. Peak memory use
2. Total number of allocations
3. Read IOPs
4. Write IOPs
5. Total bytes read
6. Total bytes written

I'd like to publish these here. Ideally these would all get nice units in the graph.

I'd also like this to not pollute the benchmark UI too much. It would be nice to be able to view one benchmark and have a dropdown to see different metrics or something like that.

[westonpace]

I'd also like this to not pollute the benchmark UI too much.

What's your workflow / use case? The primary workflow I have in mind is detecting regressions. In that case why would you want more than one metric per chart? I feel like that would introduce more clicking versus just scrolling from "most likely has regression to leak likely has regression"

[westonpace]

As for

I plan on adding a bunch of new benchmark metrics:

You can just publish multiple results. Perhaps you can add the result to the benchmark_name? For example foo_benchmark/read_iops and foo_benchmark/write_iops?

Alternatively, we can make results with the same benchmark_name but different values in the units field (read_iops, write_iops)?

Or we can introduce a new column for "metric name" or something like that.

[wjones127]

What's your workflow / use case?

My main ask is this: If I'm looking at a chart for one metric on a benchmark (say "total bytes read") I'd like it to be easy to navigate to another metric for the same benchmark (say "total bytes written").

The primary workflow I have in mind is detecting regressions.

I think the other workflow I have in mind is: I have a benchmark I know about, I want to see what it's results are, if it's changed and if the results seem sensible to me. So basically, exploring results rather than just looking for the biggest regressions.

In that case why would you want more than one metric per chart? I feel like that would introduce more clicking versus just scrolling from "most likely has regression to leak likely has regression"

To be clear, I don't want multiple metrics put into a single chart, as the scales won't match. Just want to feel like I can go from one metric to another from the same benchmark fast.

I think what I'm imaginging is a separate view. So on the page you might have the existing view as a "Top regressions page", and then there would be another tab that is "Browse benchmarks". And from there we could click through a hierachy of:

1. Repo (Such as lance, lancedb, etc.)
2. Benchmark file (Such as test_random_access.py)
3. Benchmark (Such as test_simple_random_access)
4. Benchmark labels (Such as {"dataset": "tpch", "rows_per_take": 100})
5. Metric (Such as runtime, peak_memory, etc.)

Or something like that.

[westonpace]

Ok. I think what is probably missing today in your hierarchy is 2 (benchmark file) and 4 (benchmark labels). I was thinking of recording 2 as a "benchmark group" such as lance-io or lance-file or something like that.

For number 4 I agree that would be useful. I was thinking another chart that might be nice for a sort of performance guide is the ability to look at performance vs. parameter (e.g. performance vs nprobes).

The view your describing makes sense. I think I'll prioritize reporting 2 & 4 and save the work on the view for once we have a pretty good story in place for regression.