From c291ee666934e3eb678ed32bc1ec00801510d38b Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 29 Jan 2026 14:46:58 +0800 Subject: [PATCH 01/28] Proposal for label/values discovery API Signed-off-by: Andrew Hall --- proposals/0000-new-labels-values-api.md | 485 ++++++++++++++++++++++++ 1 file changed, 485 insertions(+) create mode 100644 proposals/0000-new-labels-values-api.md diff --git a/proposals/0000-new-labels-values-api.md b/proposals/0000-new-labels-values-api.md new file mode 100644 index 0000000..04c8c97 --- /dev/null +++ b/proposals/0000-new-labels-values-api.md @@ -0,0 +1,485 @@ +## V2 API for labels and values discovery + +* **Owners:** + * Ismail Simsek [@itsmylife](https://github.com/itsmylife) [ismail.simsek@grafana.com](mailto:ismail.simsek@grafana.com) + * Andrew Hall [@tcp13equals2](https://github.com/tcp13equals2) [andrew.hall@grafana.com](mailto:andrew.hall@grafana.com) + +* **Implementation Status:** Not implemented + +* **Related Issues and PRs:** + +* **Other docs or links:** +* [Grafana - Performance improvements for high cardinality metrics #100376](https://github.com/grafana/grafana/issues/100376) +* [Grafana - Autocomplete desired API flow](https://mermaid.live/edit#pako:eNrFVNtu00AQ_ZXVPlQgXN9ix4lFgrgJ-lBU1PICQdbGO0lW2LtmL2ncKP_OrlMLo5aiSpXwg6Udj8-ZOWd29rgUFHCOFfw0wEt4x8haknrBkX0aIjUrWUO4Rl8UyLvR10aLUtRNBRru-XpxhohCnw3IFr0h5Q_g1AWPmce3wz2dz4dAObpqG1BogcvGLPAxb5jg8i_OcvTh_RUKSMOCbRxUZAmVCihTpdiCfPVyKYN5I2HFdjMLc1KxmulZHJ4oIXWxbGcreWy5PWG8rAyFQgtNqpmW5rYVy3F6p7T9llQGVI6-deUVRpE1LLCHbo-aVeyGaCa4C_q-_93rSunAcxSloYc2RBW1kBbNkR2GcvzRpiV38uTociOukRYNikMkQZlKK_TCMlo01AF7SLkUB9oLdq-wH5l2unZ4510yYhzpDSAyyENUioaKa_5fpH8aUrFaKehY_07_CJsbkCVw_dvo0qqnHmnxA-Zy2Omhu8-SMHDuVoJQoM-HI_JJWIdcz53FXqdMr9q_blPxVPepMA_qij28lozi3HXv4RpkTdwR7x37AtuJq-3w2cIxhRWxLbu6DvY3uzi-ClH3f0ph1pv-YBpKdL-i-qAb3MuWl_3ZAuB8j3c4j7Kpn4zjcBJFozjM4lHm4RbncRj56SSaJOk4TUfhOMkOHr7pOEM_m0zjcJSMJ3EWjqKph4EyLeT5cUt2y9IWZfcYyLfCcI3zNE0OvwCMfMHW) +* [Grafana - Metrics Browser/Explore desired API flow](https://mermaid.live/edit#pako:eNqllm1v2zYQx7_KgS-KBlVs-UGILMwZ1u4BxZatW7Y3mweDlk4WUUlUSSqJYvi790hKgfNQN23zwokux__d_e581I6lMkOWMI0fWqxT_FHwreLVqgb6abgyIhUNrw38o1E9tr5uRZmhAq7hAo0SqR5Mj31_ePfW-v3ZourgNU_fY51Zo_f0nzbK6fl5r5HAHw3WGqpeeqPk9V0avQ95k0YCv_z0N4x5I8ZX03HvP86ETuUVqu-_26jxeSkqYZZR-ELUadlmuCY3nnHDl0a16EV_lwZBiW1hQObghP9C06qDJK6FKcB0DerAyba1MDoATsVsuBYpiDqXXo3Onx5Ws-s1EvhvV_MKE1ixwphmrSx9bfTaSMPLFfPKNoh1SWVbGyqbBWCDWdNwwNpGo9Heff7fH7MaCUymYRhAwfW6kop0bJH7--goNws8gctCXkMulDYQhXeVvrQS4OROjjXpEktMzUDoUzV9tmsl32D5sGlec_mE4ouhn59g7eUI9YqRSCEzy2rFtOGm1Ws79t5AU9hIUZse5X2I8SHBnJf6OMIY-BUXFLlE8PGfw815PkztebjGB2fGV7wkPt_Kzas4btR-z2gezv0fkbc8Hrboy0BNor5qH-2QkvsK2gFwvAJX93GKXKUFarB9rlYMXubt7W0H2tlPvnJZOI0lCT7A9cSC-NmFq7hxWdARqbrA_l63mm_Rc0plbbioUa0HBzTp6LNrwtVE7l7K98Aq343ZsCvslrgfYb3pDOonuzX_wm7NI3A8Tn2RGSjUbWmODvebUqTvbVN-kzyDC4pErZE1bUdhBC-HJfOV7ZF5rtEt8-ds9SOAiY3deJTety_QGm8e7M8wHLsdWhIDzE6eHvNezE_6r9gl8LZnZI_5-ybDnBNxcNXCKzgccXjlsm74VtTcCGK8pGmsaV7cHrIsiFV_ceUl3ghn7nl2LGBbJTKW2PoCmi5VcfvIdjbRFTMFVtQ7e-v0WdjltKdjdKv_K2U1nFSy3RbDQ9tQ3OFtYjDy1sjLrk6HZxJgyY7dsGQymY2m03C6mJydTeNodhYHrGMJPY_O4sUink_mYRjF0T5gty7khNo2DSezeE7_mi3ixTxgmAkj1YV_oXHvNZQUrXdUb-wNypJZvP8IvLvw7A) +* [Grafana - Metrics Drilldown desired API flow](https://mermaid.live/edit#pako:eNq1Vl2T2jYU_St39LCTTFgwLF7AU7aTdttMp036se1LQ4YRlsCaGsuRZIjL8N9zJdngEJNl28k-7GBb9-uce67ujsSScRIRzd8XPIv5vaArRdezDPAvp8qIWOQ0M_CX5urzt_dKpCmT2wyohtfcKBHr48vPz7_87Sd78veCqxK-o_E_PGP2pT_p_9tI13d3By8R_JrzTAM7xGJUJwtJFfMGb6ThoMQqMSCX0LD7RVIGRuZAYyM2HNY-QW91OIexMIMIXv3wJ_RoLnqbQa862WNCx3LD1bffLFTvLhVrYab9ILjSUpn5opw6x8KUVyKL04LxORpSRg2dGlVwHwidX39az65yH8HbGUmMyefKwq-NnhtpaDojHRdvRtZ8LVU5LzRdcYxnuMZv0O123_kT7ngE_UEQdCCheo7HeQQ2-P60TMzBQhvBQyK3gFXAWmrTis05Ih54ymOjq8NwJvkL8E3pgqen8Hqv0xafVx758Ah8jOyLjKaIfVsTuIB_cFMobBwfDKwpZ7AoXbCGA3jmgNjQFEPCUihtnp-lzjtzzGHv5lJkxlIyI9pQU-i51VOTP5NI1srZuEnYkqb6McbGdSFLqbwYRLaCo84eI81ZnyZ6KVe9hlXPI3UhaU25LJUfM-VZeL1rBy_2tEd2GAz9j7B-EwY37ToInwpqPwRfGdjKqh5o4un6yvaoQ7bj0Pky3i_zPBWukVKDZg3gplVRj4wt6xK2OC41Di_PtKMZloUyCVeu4q0wCfAPQhvbBVUs6kKzryBA_52aOHn7ru3ArqXM_VVjaobB0yV1VM-MiAwj4AXVyvvoHO2PTwW6oQJ_ptzKyvlE9C19rliLLUJe4fvlVhoBZUwYIXGq1N6Rv4qaRXmZSk99QBOUi9Vam1wq1f9N739Wucu7QDi059k950qyAtFoJXv4ZJEPQ6gB0fDME-JugjZxPm_ylErcHn7EectpnPjp4D-c8tCo7hXPuKLYd7jxIH92WreE6dQZTT_F4Gu5P0LqI-D3ZqEt0N3XWxa4WePC-cvHrlQHQNuH5cGfn5g_8xK5wP6wgkLc672pd-gVeFEp5ZpuqeJV89fzyd_ZLzCJFV7aVh8wxVRiBI0d10HtM3W3uUI9bezKWS02pENWSjAS2d2og6NFral9JDub-oygytc4XHC4E8aXtEid3PZohoX_LeW6tlSyWCX1Q5HjslcvzfVLWhj5UGZx_YwOSLQjH0h0G3aHwWQcjkfBzW3_ZjDukJJE16Og3x0Owttw0h9NbsejcN8h_7qQ_e7NKBj1g-Fk0L9B0-GgQzgOCKle-73dre-YFNLB1feyyAyJJvuPHWAAkw) +* [Grafana - Possible streaming API flow](https://mermaid.live/edit#pako:eNqNVU1v2zgQ_SsDXisnlpTEqQ4BEqsttkDT7qq5LHyZSGObXYlUScpJGuS_71CU6tj5qg-mSLwZvnnzwXtR6opEJiz97EiVlEtcGWwWCvjXonGylC0qB_NaEi9o4ZPBJSocTp4i88KjvhndkFtTZyFHh4XuTElPwR62B7_A8j9S1VPs9yK_8Nh-LZw2uGKPAXepHYHekBloRR6UebettlhDChP4oKqJ0xNe2NoQNlKt4Ea6NVzmn4uvl6Ov4GFydpYXGXz68B0OsZWHm-Swxmuq7WElbemvCmgsndwg354X4SD85wU78FFl8FGbGzQV2N-XGq-2dXsOPPqxC79nJyGSvzsydxAYwEYiSEcGWYQ9Jx79high2kEDrwuWa6ilIpAWpKqoZfl9soMoW0LefPI7rAt0bBc_4jpIFkhm8ENfR-zPOuTCikBhQ7bFsQpYIMYHYn9q0wdk5GrtQC9htP2GxlIfQMTKMnUDsmmokqxHffdWAMnLAbS6iqDUyiH7NsyGe-U18i_jnyd-3nqlwWmmbbva2Qi6tvJZvPrrLdrpy7QtmY306rHzVkvOO_je0tVr3F8xGiqsdnDeOV3qpq2JOU5g7pNUAy65FMF2yyV3qq-bIZpgt9NR461X1ltQTaWjCjCweIK_9VGdX2vj5iyq0XVN5gD9fgvlaG6DLvNacxmw_Iq9Sq22mF6k21D8PDfaoXd2MNteCbh_yPYjy8LSEFU-mXCD1tP1CnRuzz58sZQEX8gZWfIg62Tti3EythonelBvx3Q3vR-l4nl17ZO8F8Femg2t2E0EvzR37juo_LKscbWjzTOZftvu-WodghjzH3FF1BDH40wyVJLc0FBk_QAPnxXtDae9w-3Ye3SYFyISKyMrkTnTUSQaMg36rbj34IXgx6Khhcj4s6IlcsEtxEI9sBk_Fv9q3YyWRner9bgJ_TW8cuMhclkXd6oc9-xAZPfiVmSTeDo9SNLZ6TSezpJZcnRyHIk7Pk_ShM-PT2azeJam8TQ-eojEr_7W5GB6Eqez4_fp-9P49Oj0eBYJHkY8qr-Et7Z_cplXP6rmulOOjZL04X-xgmzc) + +> TL;DR: This design doc proposes a richer API for searching and discovering metric names, labels, and values to power web applications. + +## Why + +The existing Prometheus [metadata API endpoints](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-metadata) are used by web applications (including Grafana) to query series metadata, such as label names and label values. + +These endpoints are heavily relied upon by the Grafana web application for core discovery workflows, including: + +* __Autocomplete__ — triggered as users type queries +* __Metrics Explorer / Builder__ — a click-based interface for browsing metrics and label dimensions +* __Metrics Drilldown__ — dynamically generating metric panels by iterating over label values + +As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, these existing endpoints have increasingly become a limiting factor. + +In practice, this manifests as performance bottlenecks in the Grafana UI and constrains the implementation of user-requested discovery features, particularly those that require iterative or interactive exploration of large metadata sets. + +Grafana has attempted to mitigate these limitations through a range of client-side optimizations and workarounds (see [#100376](https://github.com/grafana/grafana/issues/100376) +). However, these approaches add complexity and cannot fully address the underlying scalability and efficiency constraints of the current metadata APIs. + +### Pitfalls of the current solution + +Pitfalls of the current metadata endpoints; + +* __Limited client-side filtering__: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. +* __Limited matching capabilities__: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). +* __Unordered `limit` results__: `limit` returns an arbitrary subset of values rather than returning the most relevant or most frequent values, which causes drilldown UIs to display incomplete or unrepresentative results. +* __No metadata about further results__: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). +* __No pagination/cursors__: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. +* __Fragmented calls for full context__: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. +* __No streaming / incremental delivery__: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and perceived latency. + +## Goals + +Effective data queries cannot be written if what needs to be queried cannot be identified first. Improving metric name, label name and label value discovery reduces Mean Time to Execute Query (MTEQ). This can be rephrased as “users spend less time hunting for the right metrics and more time analyzing data.” + +* __Faster metric name and label identification__ - users spend less time hunting for the right metrics and can execute queries sooner, reducing the time from "I need to monitor X" to "I'm looking at data for X" +* __Better autocomplete performance__ - latency-sensitive autocomplete becomes usable at scale by fetching only relevant results instead of downloading large (43MB+) datasets just to filter client-side +* __Scalable high-cardinality handling__ -server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion +* __Reduced bandwidth/resource consumption__ - server-side filtering, streaming and/or pagination eliminate wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user +* __Improved AI/ML agent efficiency__ - metadata-rich APIs and streaming capabilities enable AI agents (like Grafana Assistant) to make informed decisions, process data progressively, and stop early when confident, reducing redundant API calls and improving response times + +### Audience + +* Operators seeking to discover metrics, labels and values + +## Non-Goals + +* Changing/enhancing/deprecating any existing Prometheus API endpoints related to labels, values and metadata +* Addressing OpenTelemetry hidden label limitations. Resource attributes stored in `target_info` rather than on every series and therefor not surfaced in the existing labels' endpoint. + +## How + +The following API endpoints are proposed. Each endpoint should return data in a NDJSON / chunked encoding streaming format. + +### `GET /api/v1/search/metrics` + +An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric. + +#### Request + +**Method:** `GET` `POST` +**Path:** `/api/v1/search/metrics` + +**Query parameters:** + +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------------|----------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `search` | string | No | N/A | The search string to be used for matching against metric names.

Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold.

The matching score of a fuzzy match must be >= this value.

A value of 100 disables any fuzz matching. | +| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | +| `sort_by` | alpha
cardinality
frequency
score | No | N/A | A flag to define how matching metrics should be sorted in the response.

* __alpha__ - metrics are sorted by alphabetical order
* __cardinality__ - metrics are sorted by their cardinality
* __frequency__ - metrics are sorted by their frequency of use
* __score__ - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. +| `sort_dir` | asc
dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. +| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. +| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. +| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. +| `start` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `end` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | +| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk.
A value of 0 indicates that the server can determine the batch size, which may be variable. | +| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | + | + +**Notes:** +- If `search` is omitted or empty, all metric names are matched. +- The fuzzy search could be a Jaro-Winkler or Levenshtein match. A Jaro-Winkler returns a score between 0..1 which can be easily scaled to a 0..100 threshold. A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100` +- Support for `cursor` based pagination is desirable but not essential. +- `start` and `end` could have default values set which align to a reasonable look-back period. ie last 24 hours +- `include_*` should use `json:"omitempty"` to not serialize the enriched attributes if they have not been requested / initialised + +Example fuzzy matching; + +* Jaro-Winkler, mimir vs mimer = 0.953 +* Levenshtein, mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8 + +#### Response + +**Status codes:** + +Use existing Prometheus API status codes. + +**Body (JSON):** + +* Content-Type: application/x-ndjson; charset=utf-8 + +##### Example of NDJSON batched result set - no include_* flags set + +```ndjson +{ + "results": [ + { "name": "go_gc_duration_seconds" }, + { "name": "go_gc_duration_seconds_count" }, + { "name": "go_gc_duration_seconds_sum" } + ] +} + +{ + "results": [ + { "name": "go_gc_gogc_percent" }, + { "name": "go_gc_gomemlimit_bytes" } + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +##### Example of NDJSON batched result set - with include_* flags set + +```ndjson +{ + "results": [ + { + "name": "activity_tracker_failed_total", + "frequency": 1003, + "cardinality": 10, + "type": "counter", + "help": "How many times has activity tracker failed to insert new activity", + "unit": "" + }, + { + "name": "activity_tracker_free_slots", + "frequency": 4, + "cardinality": 50, + "type": "gauge", + "help": "Number of free slots in activity file.", + "unit": "" + } + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +##### Example of pagination token (if supported) + +If cursor based pagination is supported, the `has_more` attribute is not required. The presence of the `cursor` attribute in the response indicates that there is more results available. + +```ndjson +{ + "results": [ + ... + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "cursor": "" +} +``` + +##### Example no results matching + +```ndjson +{ + "results": [ ], +} + +{ + "status": "success", + "has_more": false +} +``` + +##### Example API error + +As per existing endpoints. + +```ndjson +{ + "status": "error", + "errorType": "bad_data", + "error": "1:4: parse error: unexpected \"(\"", +} +``` + +### `GET /api/v1/search/labels` + +An endpoint specific to searching for label names. + +#### Request + +**Method:** `GET` `POST` +**Path:** `/api/v1/search/labels` + +**Query parameters:** + +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------------|----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | +| `search` | string | No | N/A | The search string to be used for matching against label names.

Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha
cardinality
frequency
score | No | N/A | A flag to define how matching labels should be sorted in the response.

* __alpha__ - labels are sorted by alphabetical order
* __cardinality__ - labels are sorted by their value cardinality
* __frequency__ - labels are sorted by their frequency of use
* __score__ - labels are sorted by a matching score.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. +| `sort_dir` | asc
dsc | No | asc | As per above endpoint +| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. +| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. +| `start` | rfc3339
unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339
unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | +| + +**Notes:** +- If `search` is omitted or empty, all label names are matched. + +#### Response + +**Status codes:** + +Use existing Prometheus API status codes. + +**Body (JSON):** + +* Content-Type: application/x-ndjson; charset=utf-8 + +##### Example of NDJSON batched result set - no include_* flags set + +```ndjson +{ + "results": [ + { "name": "cluster" }, + { "name": "container" }, + { "name": "instance" } + ] +} + +{ + "results": [ + { "name": "job" }, + { "name": "namespace" } + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +##### Example of NDJSON batched result set - with include_* flags set + +```ndjson +{ + "results": [ + { + "name": "cluster", + "frequency": 1003, + "cardinality": 10 + }, + { + "name": "container", + "frequency": 4, + "cardinality": 50 + } + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +### `GET /api/v1/search/values` + +An endpoint specific to searching for label values. + +#### Request + +**Method:** `GET` `POST` +**Path:** `/api/v1/search/values` + +**Query parameters:** + +| Name | Type | Required | Default | Description | +|-----------------------|------------------------------------------|----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | +| `label` | string | Yes | N/A | The label the user is requesting values for. | +| `search` | string | No | N/A | The search string to be used for matching against label values.

Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha
frequency
score | No | N/A | A flag to define how matching values should be sorted in the response.

* __alpha__ - values are sorted by alphabetical order
* __frequency__ - values are sorted by their frequency of use
* __score__ - values are sorted by a matching score.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. +| `sort_dir` | asc
dsc | No | asc | As per above endpoint +| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. +| `start` | rfc3339
unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339
unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | +| + +**Notes:** +- If `search` is omitted or empty, all label values are matched. +- The `label` parameter has been deliberately added as a required query parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. + +#### Response + +**Status codes:** + +Use existing Prometheus API status codes. + +**Body (JSON):** + +* Content-Type: application/x-ndjson; charset=utf-8 + +### Example of NDJSON batched result set - no include_* flags set + + +```ndjson +{ + "results": [ + { "name": "cluster1" }, + { "name": "cluster2" }, + { "name": "cluster3" } + ] +} + +{ + "results": [ + { "name": "cluster4" }, + { "name": "cluster5" } + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +### Example of NDJSON batched result set - with include_* flags set + +```ndjson +{ + "results": [ + { + "name": "cluster1", + "frequency": 1003 + }, + { + "name": "cluster1", + "frequency": 4 + } + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": false +} +``` + +### Additional notes + +For the `frequency` and `cardinality` enrichments, an idea is to support a "maximum reported threshold". + +For instance, the API request could include a `max-reported-frequency=100`. As the server side indexes the frequency of use it can stop once it reaches a tally of 100, allowing it to return earlier. + +From an operator perspective, they may not care for the exact value just that it's a value which is exceeding a given threshold. + +On a similar thought process, a "minimum frequency" could be supported. For instance, the API request could set a `min-frequency=100`. + +The server would only return metric/label/value matches if their associated frequency was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. + +With the combination of this min value and the `limit` it would allow the server to return more quickly. + +For instance, this could be used for searches such as "find me 10 metrics whose name contains `cpu` with a cardinality > 100". + +### Testing and verification + +It should be possible to validate these new endpoints via comparing to the existing endpoints. + +Queries to existing endpoints can be constructed to extract a full set of labels and values and this data can be client side processed and compared to the equivalent new endpoint responses. + +### Migration + +These are new endpoints and do not change or alter any existing functionality. No migration is required. + +### Known unknowns + +* confirm feasibility of including frequency and cardinality in these API responses +* confirm feasibility of supporting cursor based pagination for these new endpoints +* confirm any performance / response time constraints for these new endpoints +* specific choice of fuzzy search algorithm + +## Alternatives + +### 1. Can we not just use the existing Prometheus endpoints? + +For large scale environments with client-side optimizations; + +* High-cardinality scenarios remain challenging (43MB responses, timeouts) +* OpenTelemetry adoption increases frequency of these scenarios +* Frontend complexity grows (caching, throttling, workarounds) +* User experience compromises persist (limited search, arbitrary limits) + +### 2. Can we extend/enhance the existing Prometheus endpoints? + +The existing API design and response format is constrained. + +The existing response format returns collections of strings which does not support additional record enrichment. + +Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes. + +### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \__name__? + +Separating this endpoint allows for the additional enrichment options to be included in both the parameters and response. + +Although these could be applied to the values endpoint, it would result in the values endpoint have different behaviours (input parameters and response format) depending on the label name. + +A dedicated endpoint also allows for future enrichments to be added which are only relevant to metric names. + +## Action Plan + +The tasks to do in order to migrate to the new idea. + +* [ ] Task one `` +* [ ] Task two `` ... From 447b619e9c852b26f3f29646c36d28a169342f7e Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 29 Jan 2026 14:52:57 +0800 Subject: [PATCH 02/28] Lint Signed-off-by: Andrew Hall --- ...s-api.md => 0074-new-labels-values-api.md} | 256 ++++++++++++------ 1 file changed, 177 insertions(+), 79 deletions(-) rename proposals/{0000-new-labels-values-api.md => 0074-new-labels-values-api.md} (69%) diff --git a/proposals/0000-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md similarity index 69% rename from proposals/0000-new-labels-values-api.md rename to proposals/0074-new-labels-values-api.md index 04c8c97..8b8bf91 100644 --- a/proposals/0000-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -3,7 +3,7 @@ * **Owners:** * Ismail Simsek [@itsmylife](https://github.com/itsmylife) [ismail.simsek@grafana.com](mailto:ismail.simsek@grafana.com) * Andrew Hall [@tcp13equals2](https://github.com/tcp13equals2) [andrew.hall@grafana.com](mailto:andrew.hall@grafana.com) - + * **Implementation Status:** Not implemented * **Related Issues and PRs:** @@ -23,9 +23,9 @@ The existing Prometheus [metadata API endpoints](https://github.com/prometheus/p These endpoints are heavily relied upon by the Grafana web application for core discovery workflows, including: -* __Autocomplete__ — triggered as users type queries -* __Metrics Explorer / Builder__ — a click-based interface for browsing metrics and label dimensions -* __Metrics Drilldown__ — dynamically generating metric panels by iterating over label values +* **Autocomplete** — triggered as users type queries +* **Metrics Explorer / Builder** — a click-based interface for browsing metrics and label dimensions +* **Metrics Drilldown** — dynamically generating metric panels by iterating over label values As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, these existing endpoints have increasingly become a limiting factor. @@ -38,23 +38,23 @@ Grafana has attempted to mitigate these limitations through a range of client-si Pitfalls of the current metadata endpoints; -* __Limited client-side filtering__: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. -* __Limited matching capabilities__: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). -* __Unordered `limit` results__: `limit` returns an arbitrary subset of values rather than returning the most relevant or most frequent values, which causes drilldown UIs to display incomplete or unrepresentative results. -* __No metadata about further results__: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). -* __No pagination/cursors__: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. -* __Fragmented calls for full context__: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. -* __No streaming / incremental delivery__: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and perceived latency. +* **Limited client-side filtering**: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. +* **Limited matching capabilities**: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). +* **Unordered `limit` results**: `limit` returns an arbitrary subset of values rather than returning the most relevant or most frequent values, which causes drilldown UIs to display incomplete or unrepresentative results. +* **No metadata about further results**: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). +* **No pagination/cursors**: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. +* **Fragmented calls for full context**: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. +* **No streaming / incremental delivery**: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and perceived latency. ## Goals Effective data queries cannot be written if what needs to be queried cannot be identified first. Improving metric name, label name and label value discovery reduces Mean Time to Execute Query (MTEQ). This can be rephrased as “users spend less time hunting for the right metrics and more time analyzing data.” -* __Faster metric name and label identification__ - users spend less time hunting for the right metrics and can execute queries sooner, reducing the time from "I need to monitor X" to "I'm looking at data for X" -* __Better autocomplete performance__ - latency-sensitive autocomplete becomes usable at scale by fetching only relevant results instead of downloading large (43MB+) datasets just to filter client-side -* __Scalable high-cardinality handling__ -server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion -* __Reduced bandwidth/resource consumption__ - server-side filtering, streaming and/or pagination eliminate wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user -* __Improved AI/ML agent efficiency__ - metadata-rich APIs and streaming capabilities enable AI agents (like Grafana Assistant) to make informed decisions, process data progressively, and stop early when confident, reducing redundant API calls and improving response times +* **Faster metric name and label identification** - users spend less time hunting for the right metrics and can execute queries sooner, reducing the time from "I need to monitor X" to "I'm looking at data for X" +* **Better autocomplete performance** - latency-sensitive autocomplete becomes usable at scale by fetching only relevant results instead of downloading large (43MB+) datasets just to filter client-side +* **Scalable high-cardinality handling** -server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion +* **Reduced bandwidth/resource consumption** - server-side filtering, streaming and/or pagination eliminate wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user +* **Improved AI/ML agent efficiency** - metadata-rich APIs and streaming capabilities enable AI agents (like Grafana Assistant) to make informed decisions, process data progressively, and stop early when confident, reducing redundant API calls and improving response times ### Audience @@ -67,7 +67,7 @@ Effective data queries cannot be written if what needs to be queried cannot be i ## How -The following API endpoints are proposed. Each endpoint should return data in a NDJSON / chunked encoding streaming format. +The following API endpoints are proposed. Each endpoint should return data in a NDJSON / chunked encoding streaming format. ### `GET /api/v1/search/metrics` @@ -75,38 +75,79 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai #### Request -**Method:** `GET` `POST` +**Method:** `GET` `POST` + **Path:** `/api/v1/search/metrics` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------------|----------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `search` | string | No | N/A | The search string to be used for matching against metric names.

Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold.

The matching score of a fuzzy match must be >= this value.

A value of 100 disables any fuzz matching. | -| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | -| `sort_by` | alpha
cardinality
frequency
score | No | N/A | A flag to define how matching metrics should be sorted in the response.

* __alpha__ - metrics are sorted by alphabetical order
* __cardinality__ - metrics are sorted by their cardinality
* __frequency__ - metrics are sorted by their frequency of use
* __score__ - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. -| `sort_dir` | asc
dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. -| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. -| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. -| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. -| `start` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `end` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | -| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk.
A value of 0 indicates that the server can determine the batch size, which may be variable. | -| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | - | +| Name | Type | Required | Default | Description | +|----------|--------|----------|---------|-----------------------------------------------------------------| +| `search` | string | No | N/A | The search string to be used for matching against metric names. | +|
| | | | | + +
+Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold. +
+ +
+The matching score of a fuzzy match must be >= this value. +
+ +
+A value of 100 disables any fuzz matching. | +| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | +| `sort_by` | alpha +
+cardinality +
+frequency +
+score | No | N/A | A flag to define how matching metrics should be sorted in the response. +
+ +
+* __alpha__ - metrics are sorted by alphabetical order +
+* __cardinality__ - metrics are sorted by their cardinality +
+* __frequency__ - metrics are sorted by their frequency of use +
+* __score__ - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. +
+ +
+Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc +
+dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. | +| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. | +| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. | +| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. | +| `start` | rfc3339 +
+unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `end` | rfc3339 +
+unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | +| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. +
+A value of 0 indicates that the server can determine the batch size, which may be variable. | +| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | +| | | | | | **Notes:** - If `search` is omitted or empty, all metric names are matched. -- The fuzzy search could be a Jaro-Winkler or Levenshtein match. A Jaro-Winkler returns a score between 0..1 which can be easily scaled to a 0..100 threshold. A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100` +- The fuzzy search could be a Jaro-Winkler or Levenshtein match. A Jaro-Winkler returns a score between 0..1 which can be easily scaled to a 0..100 threshold. A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100` - Support for `cursor` based pagination is desirable but not essential. - `start` and `end` could have default values set which align to a reasonable look-back period. ie last 24 hours - `include_*` should use `json:"omitempty"` to not serialize the enriched attributes if they have not been requested / initialised Example fuzzy matching; -* Jaro-Winkler, mimir vs mimer = 0.953 +* Jaro-Winkler, mimir vs mimer = 0.953 * Levenshtein, mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8 #### Response @@ -229,31 +270,62 @@ As per existing endpoints. ### `GET /api/v1/search/labels` -An endpoint specific to searching for label names. +An endpoint specific to searching for label names. #### Request -**Method:** `GET` `POST` +**Method:** `GET` `POST` + **Path:** `/api/v1/search/labels` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------------|----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | -| `search` | string | No | N/A | The search string to be used for matching against label names.

Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha
cardinality
frequency
score | No | N/A | A flag to define how matching labels should be sorted in the response.

* __alpha__ - labels are sorted by alphabetical order
* __cardinality__ - labels are sorted by their value cardinality
* __frequency__ - labels are sorted by their frequency of use
* __score__ - labels are sorted by a matching score.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. -| `sort_dir` | asc
dsc | No | asc | As per above endpoint -| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. -| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. -| `start` | rfc3339
unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339
unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | -| +| Name | Type | Required | Default | Description | +|-----------|--------------------|----------|---------|-------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | +| `search` | string | No | N/A | The search string to be used for matching against label names. | +|
| | | | | + +
+Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha +
+cardinality +
+frequency +
+score | No | N/A | A flag to define how matching labels should be sorted in the response. +
+ +
+* __alpha__ - labels are sorted by alphabetical order +
+* __cardinality__ - labels are sorted by their value cardinality +
+* __frequency__ - labels are sorted by their frequency of use +
+* __score__ - labels are sorted by a matching score. +
+ +
+Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc +
+dsc | No | asc | As per above endpoint | +| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. | +| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. | +| `start` | rfc3339 +
+unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339 +
+unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | +| | | | | | **Notes:** - If `search` is omitted or empty, all label names are matched. @@ -328,27 +400,54 @@ An endpoint specific to searching for label values. #### Request -**Method:** `GET` `POST` +**Method:** `GET` `POST` + **Path:** `/api/v1/search/values` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|------------------------------------------|----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | -| `label` | string | Yes | N/A | The label the user is requesting values for. | -| `search` | string | No | N/A | The search string to be used for matching against label values.

Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha
frequency
score | No | N/A | A flag to define how matching values should be sorted in the response.

* __alpha__ - values are sorted by alphabetical order
* __frequency__ - values are sorted by their frequency of use
* __score__ - values are sorted by a matching score.

Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. -| `sort_dir` | asc
dsc | No | asc | As per above endpoint -| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. -| `start` | rfc3339
unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339
unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | -| +| Name | Type | Required | Default | Description | +|-----------|--------------------|----------|---------|------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | +| `label` | string | Yes | N/A | The label the user is requesting values for. | +| `search` | string | No | N/A | The search string to be used for matching against label values. | +|
| | | | | + +
+Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha +
+frequency +
+score | No | N/A | A flag to define how matching values should be sorted in the response. +
+ +
+* __alpha__ - values are sorted by alphabetical order +
+* __frequency__ - values are sorted by their frequency of use +
+* __score__ - values are sorted by a matching score. +
+ +
+Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc +
+dsc | No | asc | As per above endpoint | +| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. | +| `start` | rfc3339 +
+unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339 +
+unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | +| | | | | | **Notes:** - If `search` is omitted or empty, all label values are matched. @@ -366,7 +465,6 @@ Use existing Prometheus API status codes. ### Example of NDJSON batched result set - no include_* flags set - ```ndjson { "results": [ @@ -419,15 +517,15 @@ Use existing Prometheus API status codes. ### Additional notes -For the `frequency` and `cardinality` enrichments, an idea is to support a "maximum reported threshold". +For the `frequency` and `cardinality` enrichments, an idea is to support a "maximum reported threshold". -For instance, the API request could include a `max-reported-frequency=100`. As the server side indexes the frequency of use it can stop once it reaches a tally of 100, allowing it to return earlier. +For instance, the API request could include a `max-reported-frequency=100`. As the server side indexes the frequency of use it can stop once it reaches a tally of 100, allowing it to return earlier. From an operator perspective, they may not care for the exact value just that it's a value which is exceeding a given threshold. -On a similar thought process, a "minimum frequency" could be supported. For instance, the API request could set a `min-frequency=100`. +On a similar thought process, a "minimum frequency" could be supported. For instance, the API request could set a `min-frequency=100`. -The server would only return metric/label/value matches if their associated frequency was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. +The server would only return metric/label/value matches if their associated frequency was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. With the combination of this min value and the `limit` it would allow the server to return more quickly. @@ -435,7 +533,7 @@ For instance, this could be used for searches such as "find me 10 metrics whose ### Testing and verification -It should be possible to validate these new endpoints via comparing to the existing endpoints. +It should be possible to validate these new endpoints via comparing to the existing endpoints. Queries to existing endpoints can be constructed to extract a full set of labels and values and this data can be client side processed and compared to the equivalent new endpoint responses. @@ -465,11 +563,11 @@ For large scale environments with client-side optimizations; The existing API design and response format is constrained. -The existing response format returns collections of strings which does not support additional record enrichment. +The existing response format returns collections of strings which does not support additional record enrichment. Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes. -### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \__name__? +### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \_*name*_? Separating this endpoint allows for the additional enrichment options to be included in both the parameters and response. From ca400791b5246be6d66ecbdb27c28a9e0f33aa4a Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 30 Jan 2026 09:31:12 +0800 Subject: [PATCH 03/28] Lint Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 8b8bf91..f114e64 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -84,7 +84,9 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai | Name | Type | Required | Default | Description | |----------|--------|----------|---------|-----------------------------------------------------------------| | `search` | string | No | N/A | The search string to be used for matching against metric names. | +| | | | | | |
| | | | | +| | | | | |
Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | @@ -92,7 +94,7 @@ Metric names are matched if they contain this search string or a fuzzy match mee

-The matching score of a fuzzy match must be >= this value. +The matching score of a fuzzy match must be >= this value.

@@ -131,8 +133,8 @@ unix_timestamp | No | | As per the current labe | `end` | rfc3339
unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | -| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. +| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | +| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk.
A value of 0 indicates that the server can determine the batch size, which may be variable. | | `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | @@ -284,7 +286,9 @@ An endpoint specific to searching for label names. |-----------|--------------------|----------|---------|-------------------------------------------------------------------------------------------------------------| | `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | | `search` | string | No | N/A | The search string to be used for matching against label names. | +| | | | | | |
| | | | | +| | | | | |
Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | @@ -411,7 +415,9 @@ An endpoint specific to searching for label values. | `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | | `label` | string | Yes | N/A | The label the user is requesting values for. | | `search` | string | No | N/A | The search string to be used for matching against label values. | +| | | | | | |
| | | | | +| | | | | |
Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | From d9d5928edfc434c745808c5e3ddea4c018cab438 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 30 Jan 2026 09:45:23 +0800 Subject: [PATCH 04/28] Lint Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 195 ++++++------------------ 1 file changed, 45 insertions(+), 150 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index f114e64..2522c4e 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -81,64 +81,21 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai **Query parameters:** -| Name | Type | Required | Default | Description | -|----------|--------|----------|---------|-----------------------------------------------------------------| -| `search` | string | No | N/A | The search string to be used for matching against metric names. | -| | | | | | -|
| | | | | -| | | | | | - -
-Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold. -
- -
-The matching score of a fuzzy match must be >= this value. -
- -
-A value of 100 disables any fuzz matching. | -| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | -| `sort_by` | alpha -
-cardinality -
-frequency -
-score | No | N/A | A flag to define how matching metrics should be sorted in the response. -
- -
-* __alpha__ - metrics are sorted by alphabetical order -
-* __cardinality__ - metrics are sorted by their cardinality -
-* __frequency__ - metrics are sorted by their frequency of use -
-* __score__ - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. -
- -
-Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc -
-dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. | -| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. | -| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. | -| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. | -| `start` | rfc3339 -
-unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `end` | rfc3339 -
-unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | -| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. -
-A value of 0 indicates that the server can determine the batch size, which may be variable. | -| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | -| | | | | | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `search` | string | No | N/A | The search string to be used for matching against metric names. Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold. The matching score of a fuzzy match must be >= this value. A value of 100 disables any fuzz matching. | +| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | +| `sort_by` | alpha / cardinality / frequency / score | No | N/A | A flag to define how matching metrics should be sorted in the response. **alpha** - metrics are sorted by alphabetical order. **cardinality** - metrics are sorted by their cardinality. **frequency** - metrics are sorted by their frequency of use. **score** - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc / dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. | +| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. | +| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. | +| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. | +| `start` | rfc3339 / unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `end` | rfc3339 / unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | +| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | +| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. A value of 0 indicates that the server can determine the batch size, which may be variable. | +| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | **Notes:** - If `search` is omitted or empty, all metric names are matched. @@ -282,54 +239,21 @@ An endpoint specific to searching for label names. **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------|--------------------|----------|---------|-------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | -| `search` | string | No | N/A | The search string to be used for matching against label names. | -| | | | | | -|
| | | | | -| | | | | | - -
-Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha -
-cardinality -
-frequency -
-score | No | N/A | A flag to define how matching labels should be sorted in the response. -
- -
-* __alpha__ - labels are sorted by alphabetical order -
-* __cardinality__ - labels are sorted by their value cardinality -
-* __frequency__ - labels are sorted by their frequency of use -
-* __score__ - labels are sorted by a matching score. -
- -
-Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc -
-dsc | No | asc | As per above endpoint | -| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. | -| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. | -| `start` | rfc3339 -
-unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339 -
-unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | -| | | | | | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | +| `search` | string | No | N/A | The search string to be used for matching against label names. Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha / cardinality / frequency / score | No | N/A | A flag to define how matching labels should be sorted in the response. **alpha** - labels are sorted by alphabetical order. **cardinality** - labels are sorted by their value cardinality. **frequency** - labels are sorted by their frequency of use. **score** - labels are sorted by a matching score. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint | +| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. | +| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | **Notes:** - If `search` is omitted or empty, all label names are matched. @@ -410,50 +334,21 @@ An endpoint specific to searching for label values. **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------|--------------------|----------|---------|------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | -| `label` | string | Yes | N/A | The label the user is requesting values for. | -| `search` | string | No | N/A | The search string to be used for matching against label values. | -| | | | | | -|
| | | | | -| | | | | | - -
-Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha -
-frequency -
-score | No | N/A | A flag to define how matching values should be sorted in the response. -
- -
-* __alpha__ - values are sorted by alphabetical order -
-* __frequency__ - values are sorted by their frequency of use -
-* __score__ - values are sorted by a matching score. -
- -
-Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc -
-dsc | No | asc | As per above endpoint | -| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. | -| `start` | rfc3339 -
-unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339 -
-unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | -| | | | | | +| Name | Type | Required | Default | Description | +|---------------------|---------------------------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | +| `label` | string | Yes | N/A | The label the user is requesting values for. | +| `search` | string | No | N/A | The search string to be used for matching against label values. Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | +| `fuzz_threshold` | int | No | 100 | As per above endpoint | +| `case_sensitive` | bool | No | true | As per above endpoint | +| `sort_by` | alpha / frequency / score | No | N/A | A flag to define how matching values should be sorted in the response. **alpha** - values are sorted by alphabetical order. **frequency** - values are sorted by their frequency of use. **score** - values are sorted by a matching score. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint | +| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint | +| `limit` | int | No | 100 | As per above endpoint | +| `batch_size` | int | No | 100 | As per above endpoint | +| `cursor` | string | No | N/A | As per above endpoint | **Notes:** - If `search` is omitted or empty, all label values are matched. From c71bd78da1e4f6562a5aa55d60ac0850b583d8e4 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 30 Jan 2026 12:13:34 +0800 Subject: [PATCH 05/28] Lint Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 215 +++++++++++++++--------- 1 file changed, 136 insertions(+), 79 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 2522c4e..2f1fb23 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -1,8 +1,8 @@ ## V2 API for labels and values discovery * **Owners:** - * Ismail Simsek [@itsmylife](https://github.com/itsmylife) [ismail.simsek@grafana.com](mailto:ismail.simsek@grafana.com) - * Andrew Hall [@tcp13equals2](https://github.com/tcp13equals2) [andrew.hall@grafana.com](mailto:andrew.hall@grafana.com) + * Ismail Simsek [@itsmylife](https://github.com/itsmylife) ismail.simsek@grafana.com + * Andrew Hall [@tcp13equals2](https://github.com/tcp13equals2) andrew.hall@grafana.com * **Implementation Status:** Not implemented @@ -27,16 +27,16 @@ These endpoints are heavily relied upon by the Grafana web application for core * **Metrics Explorer / Builder** — a click-based interface for browsing metrics and label dimensions * **Metrics Drilldown** — dynamically generating metric panels by iterating over label values -As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, these existing endpoints have increasingly become a limiting factor. +As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, the existing Prometheus endpoints have increasingly become a limiting factor. -In practice, this manifests as performance bottlenecks in the Grafana UI and constrains the implementation of user-requested discovery features, particularly those that require iterative or interactive exploration of large metadata sets. +This manifests as performance bottlenecks in the Grafana UI and constrains the implementation of user-requested discovery features, particularly those that require iterative or interactive exploration of large metadata sets. Grafana has attempted to mitigate these limitations through a range of client-side optimizations and workarounds (see [#100376](https://github.com/grafana/grafana/issues/100376) ). However, these approaches add complexity and cannot fully address the underlying scalability and efficiency constraints of the current metadata APIs. ### Pitfalls of the current solution -Pitfalls of the current metadata endpoints; +Pitfalls of the current metadata endpoints: * **Limited client-side filtering**: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. * **Limited matching capabilities**: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). @@ -44,7 +44,7 @@ Pitfalls of the current metadata endpoints; * **No metadata about further results**: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). * **No pagination/cursors**: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. * **Fragmented calls for full context**: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. -* **No streaming / incremental delivery**: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and perceived latency. +* **No streaming / incremental delivery**: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and low latency interactions. ## Goals @@ -52,62 +52,113 @@ Effective data queries cannot be written if what needs to be queried cannot be i * **Faster metric name and label identification** - users spend less time hunting for the right metrics and can execute queries sooner, reducing the time from "I need to monitor X" to "I'm looking at data for X" * **Better autocomplete performance** - latency-sensitive autocomplete becomes usable at scale by fetching only relevant results instead of downloading large (43MB+) datasets just to filter client-side -* **Scalable high-cardinality handling** -server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion +* **Scalable high-cardinality handling** - server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion * **Reduced bandwidth/resource consumption** - server-side filtering, streaming and/or pagination eliminate wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user * **Improved AI/ML agent efficiency** - metadata-rich APIs and streaming capabilities enable AI agents (like Grafana Assistant) to make informed decisions, process data progressively, and stop early when confident, reducing redundant API calls and improving response times ### Audience -* Operators seeking to discover metrics, labels and values +* Operators seeking to discover metrics, labels and values - operators seeking to create new queries, perform adhoc investigations, create new rule/alerts, cardinality exploration etc ## Non-Goals -* Changing/enhancing/deprecating any existing Prometheus API endpoints related to labels, values and metadata -* Addressing OpenTelemetry hidden label limitations. Resource attributes stored in `target_info` rather than on every series and therefor not surfaced in the existing labels' endpoint. +* Deprecating / replacing any existing Prometheus API endpoints +* Addressing OpenTelemetry hidden label limitations. Resource attributes stored in `target_info` rather than on every series and therefore not surfaced in the existing labels' endpoint. ## How -The following API endpoints are proposed. Each endpoint should return data in a NDJSON / chunked encoding streaming format. +A new set of `search` endpoints are proposed. -### `GET /api/v1/search/metrics` +* /api/v1/search/metric_names +* /api/v1/search/label_names +* /api/v1/search/label_values -An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric. +Each endpoint allows for the specific searching, filtering, sorting of metric names, label names and label values respectively. + +Note that; + +* the existing Prometheus API labels/values parameter set is re-used in each endpoint, with additional parameters introduced to deliver the desired functionality +* the response format allows for enriched data to be added to each metric name, label name or label value record - rather than just a collection of strings +* the response is NDJSON (`application/x-ndjson`), allowing for a streamed chunked encoding response + +### `GET /api/v1/search/metric_names` + +An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric name. #### Request **Method:** `GET` `POST` -**Path:** `/api/v1/search/metrics` +**Path:** `/api/v1/search/metric_names` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `search` | string | No | N/A | The search string to be used for matching against metric names. Metric names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | An integer value between 0 - 100 which is used to provide a fuzzy match threshold. The matching score of a fuzzy match must be >= this value. A value of 100 disables any fuzz matching. | -| `case_sensitive` | bool | No | true | A flag to toggle whether the metric name search is case sensitive or not. | -| `sort_by` | alpha / cardinality / frequency / score | No | N/A | A flag to define how matching metrics should be sorted in the response. **alpha** - metrics are sorted by alphabetical order. **cardinality** - metrics are sorted by their cardinality. **frequency** - metrics are sorted by their frequency of use. **score** - metrics are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc / dsc | No | asc | A flag to define the ordering of the sort. Only valid when `sort_by` is set. | -| `include_frequency` | bool | No | false | A flag to request the metric frequency be included in the response. | -| `include_cardinality` | bool | No | false | A flag to request the metric cardinality be included in the response. | -| `include_metadata` | bool | No | false | A flag to request the metric metadata (units, type, description) be included in the response. | -| `start` | rfc3339 / unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `end` | rfc3339 / unix_timestamp | No | | As per the current labels/values endpoint. The time range to be considered for series inspection. | -| `limit` | int | No | 100 | An integer value >= 0. The maximum number of results to return. The overall result set will be limited to this number after any ordering has been applied. | -| `batch_size` | int | No | 100 | An integer value >= 0. The desired size of each batch of results sent in each response chunk. A value of 0 indicates that the server can determine the batch size, which may be variable. | -| `cursor` | string | No | N/A | Request the next page of results. Note that this parameter can only be used with the `limit` parameter. It is not valid with any other parameter. | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|---------|------------------------------------------------------------------------------| +| `search` | string | No | | The search string to be used for matching metric names. | +| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | +| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | +| `sort_by` | alpha / cardinality / frequency / score | No | | Request how matching metrics should be sorted in the response. | +| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | +| `include_frequency` | bool | No | false | Request metric frequency. | +| `include_cardinality` | bool | No | false | Request metric cardinality. | +| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | +| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | +| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | +| `cursor` | string | No | | Request the next page of results. | **Notes:** -- If `search` is omitted or empty, all metric names are matched. -- The fuzzy search could be a Jaro-Winkler or Levenshtein match. A Jaro-Winkler returns a score between 0..1 which can be easily scaled to a 0..100 threshold. A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100` -- Support for `cursor` based pagination is desirable but not essential. -- `start` and `end` could have default values set which align to a reasonable look-back period. ie last 24 hours -- `include_*` should use `json:"omitempty"` to not serialize the enriched attributes if they have not been requested / initialised -Example fuzzy matching; +***search*** + +The given `search` value will be used to match metric names. + +A match is found if the metric name contains this search value or if a fuzzy match between the metric name and this search meets the given `fuzz_threshold`. + +If `search` is omitted or empty, all metric names are matched. + +***fuzz_threshold*** + +The fuzz matching score must meet or exceed this threshold to be considered a match. + +A value of 100 disables any fuzz matching. + +The fuzzy search could be a Jaro-Winkler or Levenshtein match. + +A Jaro-Winkler returns a score between 0..1 which can be easily scaled to 0..100. mimir vs mimer = 0.953. A `fuzz_threshold` of 95 or below would allow this match. + +A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100`. mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8. A `fuzz_threshold` of 80 or below would allow this match. + +***sort_by*** + +* **alpha** - metric names are sorted by alphabetical order. +* **cardinality** - metric names are sorted by their cardinality. +* **frequency** - metric names are sorted by their frequency of use. +* **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. -* Jaro-Winkler, mimir vs mimer = 0.953 -* Levenshtein, mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8 +Note that `sort_by` is optional, and it is valid for there to be no sorting requested. This allows the server to return / stream results back immediately without the need for any server-side buffering. + +***batch_size*** + +The desired size of each batch of results sent in each response chunk. + +A value of 0 indicates that the server can determine the batch size, which may be variable. + +***cursor*** + +Note that cursor based pagination is desirable but not essential. + +Note that this parameter can only be used with the `limit` parameter. The request should fail if the `cursor` parameter is set with any other parameter. + +***start/end*** + +It is proposed that these could have default values which align to a reasonable look-back period. ie last 24 hours + +***include_*** + +The implementation of `include_*` should use `json:"omitempty"` to not serialize the enriched attributes if they have not been requested / initialised #### Response @@ -227,7 +278,7 @@ As per existing endpoints. } ``` -### `GET /api/v1/search/labels` +### `GET /api/v1/search/label_names` An endpoint specific to searching for label names. @@ -235,28 +286,34 @@ An endpoint specific to searching for label names. **Method:** `GET` `POST` -**Path:** `/api/v1/search/labels` +**Path:** `/api/v1/search/label_names` **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the labels search to these metrics/series. | -| `search` | string | No | N/A | The search string to be used for matching against label names. Label names are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha / cardinality / frequency / score | No | N/A | A flag to define how matching labels should be sorted in the response. **alpha** - labels are sorted by alphabetical order. **cardinality** - labels are sorted by their value cardinality. **frequency** - labels are sorted by their frequency of use. **score** - labels are sorted by a matching score. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint | -| `include_frequency` | bool | No | false | A flag to request the label frequency be included in the response. | -| `include_cardinality` | bool | No | false | A flag to request the label cardinality be included in the response. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|---------|----------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | string | No | | The search string to be used for matching against label names. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / cardinality / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request label frequency. | +| `include_cardinality` | bool | No | false | Request label cardinality. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | +| `cursor` | string | No | | As per above endpoint. | **Notes:** -- If `search` is omitted or empty, all label names are matched. + +***sort_by*** + +* **alpha** - label names are sorted by alphabetical order. +* **cardinality** - label names are sorted by their cardinality. +* **frequency** - label names are sorted by their frequency of use. +* **score** - label names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. #### Response @@ -322,7 +379,7 @@ Use existing Prometheus API status codes. } ``` -### `GET /api/v1/search/values` +### `GET /api/v1/search/label_values` An endpoint specific to searching for label values. @@ -330,29 +387,28 @@ An endpoint specific to searching for label values. **Method:** `GET` `POST` -**Path:** `/api/v1/search/values` +**Path:** `/api/v1/search/label_values` **Query parameters:** -| Name | Type | Required | Default | Description | -|---------------------|---------------------------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | N/A | Series selector - as per existing labels/values endpoints. Limit the search to these metrics/series. | -| `label` | string | Yes | N/A | The label the user is requesting values for. | -| `search` | string | No | N/A | The search string to be used for matching against label values. Label values are matched if they contain this search string or a fuzzy match meets the required threshold. | -| `fuzz_threshold` | int | No | 100 | As per above endpoint | -| `case_sensitive` | bool | No | true | As per above endpoint | -| `sort_by` | alpha / frequency / score | No | N/A | A flag to define how matching values should be sorted in the response. **alpha** - values are sorted by alphabetical order. **frequency** - values are sorted by their frequency of use. **score** - values are sorted by a matching score. Note that no sort_by is allowed and this allows the server to return / stream results back immediately without the need for any server-side buffering. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint | -| `include_frequency` | bool | No | false | A flag to request the value frequency be included in the response. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint | -| `limit` | int | No | 100 | As per above endpoint | -| `batch_size` | int | No | 100 | As per above endpoint | -| `cursor` | string | No | N/A | As per above endpoint | +| Name | Type | Required | Default | Description | +|---------------------|---------------------------|----------|---------|-----------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `label` | string | Yes | | The label the user is requesting values for. | +| `search` | string | No | | The search string to be used for matching against label values. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request value frequency. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | +| `cursor` | string | No | | As per above endpoint. | **Notes:** -- If `search` is omitted or empty, all label values are matched. -- The `label` parameter has been deliberately added as a required query parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. +- The `label` parameter has been deliberately added as a required `query` parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. #### Response @@ -364,7 +420,7 @@ Use existing Prometheus API status codes. * Content-Type: application/x-ndjson; charset=utf-8 -### Example of NDJSON batched result set - no include_* flags set +##### Example of NDJSON batched result set - no include_* flags set ```ndjson { @@ -388,7 +444,7 @@ Use existing Prometheus API status codes. } ``` -### Example of NDJSON batched result set - with include_* flags set +##### Example of NDJSON batched result set - with include_* flags set ```ndjson { @@ -398,7 +454,7 @@ Use existing Prometheus API status codes. "frequency": 1003 }, { - "name": "cluster1", + "name": "cluster2", "frequency": 4 } ] @@ -448,12 +504,13 @@ These are new endpoints and do not change or alter any existing functionality. N * confirm feasibility of supporting cursor based pagination for these new endpoints * confirm any performance / response time constraints for these new endpoints * specific choice of fuzzy search algorithm +* specific implementation of the search result ordering for auto-complete scenarios ## Alternatives ### 1. Can we not just use the existing Prometheus endpoints? -For large scale environments with client-side optimizations; +For large scale environments even with client-side optimizations; * High-cardinality scenarios remain challenging (43MB responses, timeouts) * OpenTelemetry adoption increases frequency of these scenarios @@ -466,7 +523,7 @@ The existing API design and response format is constrained. The existing response format returns collections of strings which does not support additional record enrichment. -Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes. +Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes to these endpoints. ### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \_*name*_? From 5b45b0e98e07100c448c6d27ea9e9953ed5e044c Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:36:41 +0800 Subject: [PATCH 06/28] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 2f1fb23..0dd2e71 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -87,7 +87,9 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai #### Request -**Method:** `GET` `POST` +**Method:** `GET` or `POST` + +`POST` is recommended when query parameters may exceed URL length limits. **Path:** `/api/v1/search/metric_names` From 7c3ab0b3aef89b71d32eb237f8e9de4a3f18f126 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:37:05 +0800 Subject: [PATCH 07/28] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 0dd2e71..fd696f1 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -137,7 +137,8 @@ A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance * **alpha** - metric names are sorted by alphabetical order. * **cardinality** - metric names are sorted by their cardinality. -* **frequency** - metric names are sorted by their frequency of use. +* **frequency** - metric names are sorted by their frequency of use (i.e., the number of active series containing this metric name within the queried time range). +* ``` * **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that `sort_by` is optional, and it is valid for there to be no sorting requested. This allows the server to return / stream results back immediately without the need for any server-side buffering. From 38d4a6e8eb0f070bd1710841965cdd0dd570ebda Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:37:25 +0800 Subject: [PATCH 08/28] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index fd696f1..94d81cf 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -287,7 +287,9 @@ An endpoint specific to searching for label names. #### Request -**Method:** `GET` `POST` +**Method:** `GET` or `POST` + +`POST` is recommended when `match[]` selectors may exceed URL length limits. **Path:** `/api/v1/search/label_names` From 8d3f2e77263aadcbe1470ef371b04951b18b138f Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:37:37 +0800 Subject: [PATCH 09/28] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 94d81cf..ba9504b 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -317,7 +317,7 @@ An endpoint specific to searching for label names. * **alpha** - label names are sorted by alphabetical order. * **cardinality** - label names are sorted by their cardinality. -* **frequency** - label names are sorted by their frequency of use. +* **frequency** - label names are sorted by their frequency of use (i.e., the number of active series containing this label within the queried time range). * **score** - label names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. #### Response From 2009a7b8583b58a56220b6ba70c323a51a9c7439 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:37:48 +0800 Subject: [PATCH 10/28] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index ba9504b..207b645 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -390,7 +390,9 @@ An endpoint specific to searching for label values. #### Request -**Method:** `GET` `POST` +**Method:** `GET` or `POST` + +`POST` is recommended when `match[]` selectors may exceed URL length limits. **Path:** `/api/v1/search/label_values` From 8aaf30c92ddafb2a93341ff4087e30f3b8bb7a72 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:38:12 +0800 Subject: [PATCH 11/28] Update proposals/0074-new-labels-values-api.md Co-authored-by: Arve Knudsen Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 207b645..d412670 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -544,5 +544,10 @@ A dedicated endpoint also allows for future enrichments to be added which are on The tasks to do in order to migrate to the new idea. -* [ ] Task one `` -* [ ] Task two `` ... +* [ ] Finalize proposal based on community feedback +* [ ] Create Prometheus implementation issue for `/api/v1/search/metric_names` +* [ ] Create Prometheus implementation issue for `/api/v1/search/label_names` +* [ ] Create Prometheus implementation issue for `/api/v1/search/label_values` +* [ ] Implement endpoints in Prometheus +* [ ] Integrate new endpoints into Prometheus UI +* [ ] Update Prometheus documentation From 43905db1b26fa7b08e0cf003eb9769dbe9b2dedd Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 11:53:49 +0800 Subject: [PATCH 12/28] Update 0074-new-labels-values-api.md Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index d412670..1b4e15b 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -97,6 +97,7 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai | Name | Type | Required | Default | Description | |-----------------------|-----------------------------------------|----------|---------|------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | | `search` | string | No | | The search string to be used for matching metric names. | | `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | | `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | @@ -121,6 +122,12 @@ A match is found if the metric name contains this search value or if a fuzzy mat If `search` is omitted or empty, all metric names are matched. +The `match[]` param can be used as an alternative to using the `search` or it can be used in conjunction with the `search`. + +For example; +* `match[]={cluster=prod}&search=` - find me all metric names which have a `cluster=prod` label +* `match[]={cluster=prod}&search=cpu` - find me all metric names which contain `cpu` and which have a `cluster=prod` label + ***fuzz_threshold*** The fuzz matching score must meet or exceed this threshold to be considered a match. From 89d8e07d2549066e4e3cd263403185e1bf95f837 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 12:07:31 +0800 Subject: [PATCH 13/28] PR feedback Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 1b4e15b..a7b2870 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -36,7 +36,18 @@ Grafana has attempted to mitigate these limitations through a range of client-si ### Pitfalls of the current solution -Pitfalls of the current metadata endpoints: +The current metadata APIs provide limited server-side filtering, ordering, and aggregation capabilities. As a result, client applications are often required to retrieve large, unfiltered datasets and then perform filtering, sorting, and relevance ranking locally. + +In addition, clients must frequently combine data from multiple API endpoints to assemble the information needed for a single user interaction or view. + +In practice, this leads to: +* unnecessarily **large response payloads** +* **increased backend load** +* a degraded user experience, as **clients must wait** for full responses before rendering meaningful results + +For interactive discovery workflows, this significantly increases the time-to-first-result presented to the user and constrains the ability to build responsive, incremental interfaces. + +Some specific examples highlighted by Grafana web application developers include; * **Limited client-side filtering**: `match[]` only filters series on the server. For label-name autocomplete clients must download all label names and then filter locally, which is slow at scale. * **Limited matching capabilities**: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). From 9feba6b783f5ccedb1b2184cb044be2d92430016 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 4 Feb 2026 12:44:06 +0800 Subject: [PATCH 14/28] PR feedback Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index a7b2870..6b3259f 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -157,6 +157,7 @@ A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance * **cardinality** - metric names are sorted by their cardinality. * **frequency** - metric names are sorted by their frequency of use (i.e., the number of active series containing this metric name within the queried time range). * ``` + ``` * **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that `sort_by` is optional, and it is valid for there to be no sorting requested. This allows the server to return / stream results back immediately without the need for any server-side buffering. From 5b020b6c5c877eafc6294e133ffe0d0fbfae0875 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 5 Feb 2026 08:17:33 +0800 Subject: [PATCH 15/28] Rename cursor param to next_token Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 6b3259f..64214ea 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -121,7 +121,7 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai | `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | | `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | | `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | -| `cursor` | string | No | | Request the next page of results. | +| `next_token` | string | No | | Request the next page of results. | **Notes:** @@ -168,11 +168,11 @@ The desired size of each batch of results sent in each response chunk. A value of 0 indicates that the server can determine the batch size, which may be variable. -***cursor*** +***next_token*** -Note that cursor based pagination is desirable but not essential. +Note that pagination is desirable but not essential to this proposal. -Note that this parameter can only be used with the `limit` parameter. The request should fail if the `cursor` parameter is set with any other parameter. +Note that this parameter can only be used with the `limit` parameter. The request should fail if the `next_token` parameter is set with any other parameter. ***start/end*** @@ -254,7 +254,7 @@ Use existing Prometheus API status codes. ##### Example of pagination token (if supported) -If cursor based pagination is supported, the `has_more` attribute is not required. The presence of the `cursor` attribute in the response indicates that there is more results available. +If pagination is supported, the `has_more` attribute is not required. The presence of the `next_token` attribute in the response indicates that there is more results available. ```ndjson { @@ -271,7 +271,7 @@ If cursor based pagination is supported, the `has_more` attribute is not require { "status": "success", - "cursor": "" + "next_token": "" } ``` @@ -328,7 +328,7 @@ An endpoint specific to searching for label names. | `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | | `limit` | int >= 0 | No | 100 | As per above endpoint. | | `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `cursor` | string | No | | As per above endpoint. | +| `next_token` | string | No | | As per above endpoint. | **Notes:** @@ -431,7 +431,7 @@ An endpoint specific to searching for label values. | `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | | `limit` | int >= 0 | No | 100 | As per above endpoint. | | `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `cursor` | string | No | | As per above endpoint. | +| `next_token` | string | No | | As per above endpoint. | **Notes:** - The `label` parameter has been deliberately added as a required `query` parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. From bb56ea4c92c90b87bc2fd2ca7bf6ea15bb9247ce Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 5 Feb 2026 09:04:07 +0800 Subject: [PATCH 16/28] Add extra section about frequency and cardinality Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 36 +++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 64214ea..b883bd0 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -27,7 +27,7 @@ These endpoints are heavily relied upon by the Grafana web application for core * **Metrics Explorer / Builder** — a click-based interface for browsing metrics and label dimensions * **Metrics Drilldown** — dynamically generating metric panels by iterating over label values -As environments have grown in active series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, the existing Prometheus endpoints have increasingly become a limiting factor. +As environments have grown in series count, label counts, cardinality, and exploratory user activity, and as the Grafana web application itself has grown in complexity, the existing Prometheus endpoints have increasingly become a limiting factor. This manifests as performance bottlenecks in the Grafana UI and constrains the implementation of user-requested discovery features, particularly those that require iterative or interactive exploration of large metadata sets. @@ -125,6 +125,38 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai **Notes:** +***frequency & cardinality*** + +For the purposes of this document, the following definitions apply; + +* **frequency** — number of occurrences across a dataset within the queried time range. +* **cardinality** — the number of distinct time series within the queried time range; equivalently, the number of unique label-value combinations for a given metric or label. + +For example consider these samples; + +```text +http_requests_total{method="GET", status="200", instance="a"} +http_requests_total{method="GET", status="200", instance="b"} +http_requests_total{method="POST", status="200", instance="a"} +http_requests_total{method="POST", status="500", instance="a"} +http_requests_total{method="POST", status="500", instance="b"} + +cpu_usage_seconds_total{core="0", instance="a"} +cpu_usage_seconds_total{core="1", instance="a"} +cpu_usage_seconds_total{core="0", instance="b"} +cpu_usage_seconds_total{core="0", instance="b"} +``` + +| Name | Type | Frequency | Cardinality | +|-----------------------------------|-------------|-----------|-------------| +| http_requests_total | metric_name | 5 | 5 | +| http_requests_total{method="GET"} | metric_name | 2 | 2 | +| cpu_usage_seconds_total | metric_name | 4 | 3 | +| method | label_name | 5 | 2 | +| core | label_name | 4 | 2 | +| method="GET" | label_value | 2 | N/A | +| GET | label_value | 2 | N/A | + ***search*** The given `search` value will be used to match metric names. @@ -155,7 +187,7 @@ A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance * **alpha** - metric names are sorted by alphabetical order. * **cardinality** - metric names are sorted by their cardinality. -* **frequency** - metric names are sorted by their frequency of use (i.e., the number of active series containing this metric name within the queried time range). +* **frequency** - metric names are sorted by their frequency. * ``` ``` * **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. From c801cdc83e3a348da8a622b8bbc0788ddbe15593 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 5 Feb 2026 09:38:50 +0800 Subject: [PATCH 17/28] Add section for API extensibility Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 58 ++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index b883bd0..e60efd4 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -546,6 +546,62 @@ With the combination of this min value and the `limit` it would allow the server For instance, this could be used for searches such as "find me 10 metrics whose name contains `cpu` with a cardinality > 100". +### Extensibility for Mimir, Thanos, Cortex + +The introduction of returning collections of records rather than collections of strings allows for different implementations to provide additional record decorations. + +This would allow each implementation to provide a custom set of additional request parameters which results in additional data in the response. + +To avoid conflicts, the response record format could be extended to include an `extensions` map which returns the custom fields. The `extensions` would be omitted completely in the pure Prometheus implementation. + +```json +{ + "results": [ + { + "name": "cluster1", + "frequency": 1003, + "extensions": { + "mimir": { + "active_series": 10, + "owned_series" : 5, + ... + } + } + }, + { + "name": "cluster2", + "frequency": 4, + "extensions": { + ... + } + } + ] +} +``` + +The request to enable the extension could be implicit in the requested params. For instance the params could be `mimir.include_active_series=true`. + +To be more standards compliant, the extension could be requested in via the `Accept` header. ie `Accept: application/x-ndjson; charset=utf-8; extensions=mimir`. This would also be reflected in the returned `Content-Type`. + +If required for client side applications, a simple endpoint which returns the supported extensions and their data model could be exposed. + +*GET /api/v1/search/extensions* + +```json +{ + "mimir": { + "active_series": { + "type": "int", + "help": "some string explaining the field" + }, + ... + } +} + +``` + +If required, these extensions could also be versioned. + ### Testing and verification It should be possible to validate these new endpoints via comparing to the existing endpoints. @@ -583,7 +639,7 @@ The existing response format returns collections of strings which does not suppo Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes to these endpoints. -### 3 Why do we need a 3rd endpoint specific to metric names - it's just a special case of \_*name*_? +### 3. Why do we need a 3rd endpoint specific to metric names - it's just a special case of \_*name*_? Separating this endpoint allows for the additional enrichment options to be included in both the parameters and response. From 195a96d617ede81718d7b77d89e83ee902627b65 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 08:48:49 +0800 Subject: [PATCH 18/28] Move pagination into feature request rather then limitation Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index e60efd4..8cf68c1 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -53,10 +53,11 @@ Some specific examples highlighted by Grafana web application developers include * **Limited matching capabilities**: `match[]` supports regex matching of label values, but does not support fuzzy or typo-tolerant matching that discovery UIs require (e.g., user types container.memory and expects container_memory). * **Unordered `limit` results**: `limit` returns an arbitrary subset of values rather than returning the most relevant or most frequent values, which causes drilldown UIs to display incomplete or unrepresentative results. * **No metadata about further results**: responses do not indicate whether `limit` truncated the result set or how many total values exist, preventing the UI from making informed choices (show “more” affordance, refine query, etc.). -* **No pagination/cursors**: there is no offset/cursor mechanism; UIs cannot page through thousands of label values without fetching very large responses that may time out or overload the server. * **Fragmented calls for full context**: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. * **No streaming / incremental delivery**: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and low latency interactions. +It is also noted that as part of this proposal there is also a **feature** request from the web application developers to support **pagination** (cursor/offset). *A request has been mde to the web application developers to confirm if this is needed assuming batched streaming of responses are supported.* + ## Goals Effective data queries cannot be written if what needs to be queried cannot be identified first. Improving metric name, label name and label value discovery reduces Mean Time to Execute Query (MTEQ). This can be rephrased as “users spend less time hunting for the right metrics and more time analyzing data.” From daa1e946c9b0f3faa0e663e505f83618319a7532 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 09:15:20 +0800 Subject: [PATCH 19/28] Add warning example Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 73 ++++++++++++++++++++++++- 1 file changed, 70 insertions(+), 3 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 8cf68c1..bcab7e0 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -89,9 +89,10 @@ Each endpoint allows for the specific searching, filtering, sorting of metric na Note that; -* the existing Prometheus API labels/values parameter set is re-used in each endpoint, with additional parameters introduced to deliver the desired functionality -* the response format allows for enriched data to be added to each metric name, label name or label value record - rather than just a collection of strings -* the response is NDJSON (`application/x-ndjson`), allowing for a streamed chunked encoding response +* the existing Prometheus API labels/values parameter set is re-used in each endpoint, with additional parameters introduced to deliver the desired functionality. +* the response format allows for enriched data to be added to each metric name, label name or label value record - rather than just a collection of strings. +* the response is NDJSON (`application/x-ndjson`), allowing for a streamed chunked encoding response. +* the existing Prometheus API response status, errors, warnings messages should be maintained. Note their placement in the response has been adapted for the streaming response format. ### `GET /api/v1/search/metric_names` @@ -333,6 +334,72 @@ As per existing endpoints. } ``` +##### Example of has_more + +Note - The `has_more` flag could replace the need for the existing Prometheus `results truncated due to limit` warning; + +```ndjson +{ + "results": [ + { + "name": "activity_tracker_failed_total", + }, + { + "name": "activity_tracker_free_slots", + } + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": true +} +``` + +##### Example of warnings + +Warnings can be added to either to an individual batch or to the overall request. + +As per the existing labels/values endpoints the `warnings` attribute is omitted in the response if there are none. + +If informational messages are also required they could be added in a similar way to a batch or the end message. + +```ndjson +{ + "results": [ + { + "name": "activity_tracker_failed_total", + }, + { + "name": "activity_tracker_free_slots", + } + ], + "warnings": [ + "some_warning_relevant_to_this_batch" + ] +} + +{ + "results": [ + ... + ] +} + +{ + "status": "success", + "has_more": true, + "warnings": [ + "some_other_warning_relevant_to_the_overall_request" + ] +} +``` + ### `GET /api/v1/search/label_names` An endpoint specific to searching for label names. From f4a7917544f172b92b309d5d2ba5270205aa04c4 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 09:45:07 +0800 Subject: [PATCH 20/28] Fix use of frequency Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 101 ++++++++++++------------ 1 file changed, 52 insertions(+), 49 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index bcab7e0..dfbb5c8 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -87,52 +87,29 @@ A new set of `search` endpoints are proposed. Each endpoint allows for the specific searching, filtering, sorting of metric names, label names and label values respectively. -Note that; +### Implementation notes * the existing Prometheus API labels/values parameter set is re-used in each endpoint, with additional parameters introduced to deliver the desired functionality. * the response format allows for enriched data to be added to each metric name, label name or label value record - rather than just a collection of strings. * the response is NDJSON (`application/x-ndjson`), allowing for a streamed chunked encoding response. * the existing Prometheus API response status, errors, warnings messages should be maintained. Note their placement in the response has been adapted for the streaming response format. -### `GET /api/v1/search/metric_names` - -An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric name. - -#### Request - -**Method:** `GET` or `POST` - -`POST` is recommended when query parameters may exceed URL length limits. +#### frequency & cardinality -**Path:** `/api/v1/search/metric_names` +For the purposes of this document, the following definitions apply; -**Query parameters:** +* **frequency** — the number of occurrences of a label, or label value across time series within the queried time range. A single time series contributes at most one occurrence. +* **cardinality** — the number of distinct time series within the queried time range; equivalently, the number of unique label-value combinations for a given metric or label. -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------|----------|---------|------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `search` | string | No | | The search string to be used for matching metric names. | -| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | -| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | -| `sort_by` | alpha / cardinality / frequency / score | No | | Request how matching metrics should be sorted in the response. | -| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | -| `include_frequency` | bool | No | false | Request metric frequency. | -| `include_cardinality` | bool | No | false | Request metric cardinality. | -| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | -| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | -| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | -| `next_token` | string | No | | Request the next page of results. | +Cardinality answers "How many unique series exist?". -**Notes:** +Frequency answers "How commonly does this label / value appear across series?" -***frequency & cardinality*** +For a metric name, cardinality and frequency are equivalent so only cardinality is considered. -For the purposes of this document, the following definitions apply; +For a label name, cardinality and frequency are independent. -* **frequency** — number of occurrences across a dataset within the queried time range. -* **cardinality** — the number of distinct time series within the queried time range; equivalently, the number of unique label-value combinations for a given metric or label. +For a label value, only frequency is relevant. For example consider these samples; @@ -146,19 +123,49 @@ http_requests_total{method="POST", status="500", instance="b"} cpu_usage_seconds_total{core="0", instance="a"} cpu_usage_seconds_total{core="1", instance="a"} cpu_usage_seconds_total{core="0", instance="b"} -cpu_usage_seconds_total{core="0", instance="b"} ``` | Name | Type | Frequency | Cardinality | |-----------------------------------|-------------|-----------|-------------| -| http_requests_total | metric_name | 5 | 5 | -| http_requests_total{method="GET"} | metric_name | 2 | 2 | -| cpu_usage_seconds_total | metric_name | 4 | 3 | +| http_requests_total | metric_name | N/A | 5 | +| http_requests_total{method="GET"} | metric_name | N/A | 2 | | method | label_name | 5 | 2 | | core | label_name | 4 | 2 | | method="GET" | label_value | 2 | N/A | | GET | label_value | 2 | N/A | +### `GET /api/v1/search/metric_names` + +An endpoint specific to searching for metric names (\_\_name__ values) and obtaining an enriched record for each metric name. + +#### Request + +**Method:** `GET` or `POST` + +`POST` is recommended when query parameters may exceed URL length limits. + +**Path:** `/api/v1/search/metric_names` + +**Query parameters:** + +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------|----------|---------|------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | string | No | | The search string to be used for matching metric names. | +| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | +| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | +| `sort_by` | alpha / cardinality / score | No | | Request how matching metrics should be sorted in the response. | +| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | +| `include_cardinality` | bool | No | false | Request metric cardinality. | +| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | +| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | +| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | +| `next_token` | string | No | | Request the next page of results. | + +**Notes:** + ***search*** The given `search` value will be used to match metric names. @@ -189,9 +196,6 @@ A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance * **alpha** - metric names are sorted by alphabetical order. * **cardinality** - metric names are sorted by their cardinality. -* **frequency** - metric names are sorted by their frequency. -* ``` - ``` * **score** - metric names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. Note that `sort_by` is optional, and it is valid for there to be no sorting requested. This allows the server to return / stream results back immediately without the need for any server-side buffering. @@ -257,7 +261,6 @@ Use existing Prometheus API status codes. "results": [ { "name": "activity_tracker_failed_total", - "frequency": 1003, "cardinality": 10, "type": "counter", "help": "How many times has activity tracker failed to insert new activity", @@ -265,7 +268,6 @@ Use existing Prometheus API status codes. }, { "name": "activity_tracker_free_slots", - "frequency": 4, "cardinality": 50, "type": "gauge", "help": "Number of free slots in activity file.", @@ -436,7 +438,7 @@ An endpoint specific to searching for label names. * **alpha** - label names are sorted by alphabetical order. * **cardinality** - label names are sorted by their cardinality. -* **frequency** - label names are sorted by their frequency of use (i.e., the number of active series containing this label within the queried time range). +* **frequency** - label names are sorted by their frequency. * **score** - label names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. #### Response @@ -485,7 +487,7 @@ Use existing Prometheus API status codes. }, { "name": "container", - "frequency": 4, + "frequency": 60, "cardinality": 50 } ] @@ -608,7 +610,7 @@ From an operator perspective, they may not care for the exact value just that it On a similar thought process, a "minimum frequency" could be supported. For instance, the API request could set a `min-frequency=100`. -The server would only return metric/label/value matches if their associated frequency was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. +The server would only return metric/label/value matches if their associated frequency/cardinality was >= 100. This allows for the discovery of high frequency or high cardinality metrics without needing to do a full search and order by. With the combination of this min value and the `limit` it would allow the server to return more quickly. @@ -630,8 +632,8 @@ To avoid conflicts, the response record format could be extended to include an ` "frequency": 1003, "extensions": { "mimir": { - "active_series": 10, - "owned_series" : 5, + "some_mimir_specific_attribute": 10, + "another_mimir_specific_attribute" : 5, ... } } @@ -658,7 +660,7 @@ If required for client side applications, a simple endpoint which returns the su ```json { "mimir": { - "active_series": { + "some_mimir_specific_attribute": { "type": "int", "help": "some string explaining the field" }, @@ -683,7 +685,7 @@ These are new endpoints and do not change or alter any existing functionality. N ### Known unknowns * confirm feasibility of including frequency and cardinality in these API responses -* confirm feasibility of supporting cursor based pagination for these new endpoints +* confirm requirement of supporting cursor based pagination for these new endpoints * confirm any performance / response time constraints for these new endpoints * specific choice of fuzzy search algorithm * specific implementation of the search result ordering for auto-complete scenarios @@ -719,6 +721,7 @@ A dedicated endpoint also allows for future enrichments to be added which are on The tasks to do in order to migrate to the new idea. +* [ ] Confirm requirement for supporting pagination or not * [ ] Finalize proposal based on community feedback * [ ] Create Prometheus implementation issue for `/api/v1/search/metric_names` * [ ] Create Prometheus implementation issue for `/api/v1/search/label_names` From b20bc305d68fcc11f62cd2c7614545cfe5ecee2c Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 09:48:29 +0800 Subject: [PATCH 21/28] Update default lookback period Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index dfbb5c8..7dd8aad 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -214,7 +214,9 @@ Note that this parameter can only be used with the `limit` parameter. The reques ***start/end*** -It is proposed that these could have default values which align to a reasonable look-back period. ie last 24 hours +It is proposed that these could have default values which align to a reasonable look-back period. + +Ideally this would fall within the WAL. A default look-back of 1 hour is proposed. ***include_*** From 9a626f93ecd3054a9fa12e3540f40623d2530a15 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 6 Feb 2026 09:57:58 +0800 Subject: [PATCH 22/28] Add fuzz_alg param Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 108 +++++++++++++----------- 1 file changed, 60 insertions(+), 48 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 7dd8aad..7d19cce 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -148,21 +148,22 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------|----------|---------|------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `search` | string | No | | The search string to be used for matching metric names. | -| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | -| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | -| `sort_by` | alpha / cardinality / score | No | | Request how matching metrics should be sorted in the response. | -| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | -| `include_cardinality` | bool | No | false | Request metric cardinality. | -| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | -| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | -| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | -| `next_token` | string | No | | Request the next page of results. | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------|----------|--------------|------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | string | No | | The search string to be used for matching metric names. | +| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | +| `fuzz_alg` | string | No | jarowrinkler | Select the fuzzy match algorithm. | +| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | +| `sort_by` | alpha / cardinality / score | No | | Request how matching metrics should be sorted in the response. | +| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | +| `include_cardinality` | bool | No | false | Request metric cardinality. | +| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | +| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | +| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | +| `next_token` | string | No | | Request the next page of results. | **Notes:** @@ -192,6 +193,14 @@ A Jaro-Winkler returns a score between 0..1 which can be easily scaled to 0..100 A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance / max(len(s), len(t)))) * 100`. mimir vs mimer = 1. Applying `similarity = 1 − (distance / max(len(s), len(t)))` = similarity = 1 − 1/5 = 0.8. A `fuzz_threshold` of 80 or below would allow this match. +***fuzz_alg*** + +Allow the client to select which fuzzy algorithm is used. Noting that the selection must be supported by the server. + +It is proposed that the default algorithm be Jaro-Winkler, and that all implementations should support this. + +It is proposed that the available fuzzy algorithms be exposed via the `/features` endpoint. + ***sort_by*** * **alpha** - metric names are sorted by alphabetical order. @@ -214,9 +223,9 @@ Note that this parameter can only be used with the `limit` parameter. The reques ***start/end*** -It is proposed that these could have default values which align to a reasonable look-back period. +It is proposed that these could have default values which align to a reasonable look-back period. -Ideally this would fall within the WAL. A default look-back of 1 hour is proposed. +Ideally this would fall within the WAL. A default look-back of 1 hour is proposed. ***include_*** @@ -418,21 +427,22 @@ An endpoint specific to searching for label names. **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------|----------|---------|----------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `search` | string | No | | The search string to be used for matching against label names. | -| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | -| `case_sensitive` | bool | No | true | As per above endpoint. | -| `sort_by` | alpha / cardinality / frequency / score | No | | As per above endpoint. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | -| `include_frequency` | bool | No | false | Request label frequency. | -| `include_cardinality` | bool | No | false | Request label cardinality. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `limit` | int >= 0 | No | 100 | As per above endpoint. | -| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `next_token` | string | No | | As per above endpoint. | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|--------------|----------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | string | No | | The search string to be used for matching against label names. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `fuzz_alg` | string | No | jarowrinkler | Select the fuzzy match algorithm. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / cardinality / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request label frequency. | +| `include_cardinality` | bool | No | false | Request label cardinality. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | +| `next_token` | string | No | | As per above endpoint. | **Notes:** @@ -521,21 +531,22 @@ An endpoint specific to searching for label values. **Query parameters:** -| Name | Type | Required | Default | Description | -|---------------------|---------------------------|----------|---------|-----------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `label` | string | Yes | | The label the user is requesting values for. | -| `search` | string | No | | The search string to be used for matching against label values. | -| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | -| `case_sensitive` | bool | No | true | As per above endpoint. | -| `sort_by` | alpha / frequency / score | No | | As per above endpoint. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | -| `include_frequency` | bool | No | false | Request value frequency. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `limit` | int >= 0 | No | 100 | As per above endpoint. | -| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `next_token` | string | No | | As per above endpoint. | +| Name | Type | Required | Default | Description | +|---------------------|---------------------------|----------|--------------|-----------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `label` | string | Yes | | The label the user is requesting values for. | +| `search` | string | No | | The search string to be used for matching against label values. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `fuzz_alg` | string | No | jarowrinkler | Select the fuzzy match algorithm. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request value frequency. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | +| `next_token` | string | No | | As per above endpoint. | **Notes:** - The `label` parameter has been deliberately added as a required `query` parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. @@ -655,7 +666,7 @@ The request to enable the extension could be implicit in the requested params. F To be more standards compliant, the extension could be requested in via the `Accept` header. ie `Accept: application/x-ndjson; charset=utf-8; extensions=mimir`. This would also be reflected in the returned `Content-Type`. -If required for client side applications, a simple endpoint which returns the supported extensions and their data model could be exposed. +If required for client side applications, a simple endpoint which returns the supported extensions and their data model could be exposed. Noting that the existing Prometheus `/api/v1/features` could also be used. *GET /api/v1/search/extensions* @@ -728,6 +739,7 @@ The tasks to do in order to migrate to the new idea. * [ ] Create Prometheus implementation issue for `/api/v1/search/metric_names` * [ ] Create Prometheus implementation issue for `/api/v1/search/label_names` * [ ] Create Prometheus implementation issue for `/api/v1/search/label_values` +* [ ] Modify existing Prometheus `/features` endpoint to expose supported fuzzy algorithm choices * [ ] Implement endpoints in Prometheus * [ ] Integrate new endpoints into Prometheus UI * [ ] Update Prometheus documentation From a55442d04d0ec0636936fa3955c074767743029d Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 11 Feb 2026 08:59:47 +0800 Subject: [PATCH 23/28] Add note about metadata limit Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 7d19cce..5916b9c 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -267,6 +267,10 @@ Use existing Prometheus API status codes. ##### Example of NDJSON batched result set - with include_* flags set +Note on the inclusion of metadata. Although the existing `/api/v1/metadata` endpoint accepts a `limit_per_metric` flag, the proposal here is to return a single metadata record. + +The record included in this response would be the same as the first record returned in the current `/api/v1/metadata` endpoint (for a given metric). + ```ndjson { "results": [ From d4bc5961aafaca72587437d58c03519e350d1f47 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 25 Feb 2026 15:49:04 +0800 Subject: [PATCH 24/28] Remove pagination references, make search multiple, add notes around fuzzy search/ordering, add proposed interfaces Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 280 ++++++++++++++++-------- 1 file changed, 189 insertions(+), 91 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 5916b9c..753bc83 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -56,16 +56,14 @@ Some specific examples highlighted by Grafana web application developers include * **Fragmented calls for full context**: clients must call multiple endpoints (labels, metric metadata, series queries) to assemble a page of enriched metric information, increasing latency and backend load. * **No streaming / incremental delivery**: the API returns the full result set only after query completion; clients cannot render partial results quickly for improved interactivity and low latency interactions. -It is also noted that as part of this proposal there is also a **feature** request from the web application developers to support **pagination** (cursor/offset). *A request has been mde to the web application developers to confirm if this is needed assuming batched streaming of responses are supported.* - ## Goals Effective data queries cannot be written if what needs to be queried cannot be identified first. Improving metric name, label name and label value discovery reduces Mean Time to Execute Query (MTEQ). This can be rephrased as “users spend less time hunting for the right metrics and more time analyzing data.” * **Faster metric name and label identification** - users spend less time hunting for the right metrics and can execute queries sooner, reducing the time from "I need to monitor X" to "I'm looking at data for X" * **Better autocomplete performance** - latency-sensitive autocomplete becomes usable at scale by fetching only relevant results instead of downloading large (43MB+) datasets just to filter client-side -* **Scalable high-cardinality handling** - server-side filtering, streaming and/or pagination enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion -* **Reduced bandwidth/resource consumption** - server-side filtering, streaming and/or pagination eliminate wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user +* **Scalable high-cardinality handling** - server-side filtering, streaming enables users to browse thousands of labels/values without timeouts, incomplete data, or memory exhaustion +* **Reduced bandwidth/resource consumption** - server-side filtering, streaming eliminates wasteful large responses, and enriched responses eliminate additional API calls reducing overall latency for the user * **Improved AI/ML agent efficiency** - metadata-rich APIs and streaming capabilities enable AI agents (like Grafana Assistant) to make informed decisions, process data progressively, and stop early when confident, reducing redundant API calls and improving response times ### Audience @@ -76,6 +74,7 @@ Effective data queries cannot be written if what needs to be queried cannot be i * Deprecating / replacing any existing Prometheus API endpoints * Addressing OpenTelemetry hidden label limitations. Resource attributes stored in `target_info` rather than on every series and therefore not surfaced in the existing labels' endpoint. +* Supporting pagination of API responses. ## How @@ -148,30 +147,31 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------|----------|--------------|------------------------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `search` | string | No | | The search string to be used for matching metric names. | -| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | -| `fuzz_alg` | string | No | jarowrinkler | Select the fuzzy match algorithm. | -| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | -| `sort_by` | alpha / cardinality / score | No | | Request how matching metrics should be sorted in the response. | -| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | -| `include_cardinality` | bool | No | false | Request metric cardinality. | -| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | -| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | -| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | -| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | -| `next_token` | string | No | | Request the next page of results. | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------|----------|---------|------------------------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | []string | No | | The search strings to be used for matching metric names. | +| `fuzz_threshold` | int [0..100] | No | 100 | Set the fuzzy match threshold. | +| `fuzz_alg` | string | No | jaro | Select the fuzzy match algorithm. | +| `case_sensitive` | bool | No | true | Toggle case sensitivity in string matching. | +| `sort_by` | alpha / cardinality / score | No | | Request how matching metrics should be sorted in the response. | +| `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | +| `include_cardinality` | bool | No | false | Request metric cardinality. | +| `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | +| `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | +| `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | +| `batch_size` | int >= 0 | No | 100 | The desired number of records per batch. | **Notes:** ***search*** -The given `search` value will be used to match metric names. +The given `search` values to be used to match metric names. + +Multiple search values can be specified. -A match is found if the metric name contains this search value or if a fuzzy match between the metric name and this search meets the given `fuzz_threshold`. +A match is found if the metric name contains one of these search values or if a fuzzy match between the metric name and one of these search terms meets the given `fuzz_threshold`. If `search` is omitted or empty, all metric names are matched. @@ -180,14 +180,15 @@ The `match[]` param can be used as an alternative to using the `search` or it ca For example; * `match[]={cluster=prod}&search=` - find me all metric names which have a `cluster=prod` label * `match[]={cluster=prod}&search=cpu` - find me all metric names which contain `cpu` and which have a `cluster=prod` label +* `match[]={cluster=prod}&search=cpu&search=mem` - find me all metric names which contain `cpu` or `mem` and which have a `cluster=prod` label ***fuzz_threshold*** The fuzz matching score must meet or exceed this threshold to be considered a match. -A value of 100 disables any fuzz matching. +A value of 0 disables any fuzz matching. -The fuzzy search could be a Jaro-Winkler or Levenshtein match. +The fuzzy search could be a Jaro, Jaro-Winkler or Levenshtein match. A Jaro-Winkler returns a score between 0..1 which can be easily scaled to 0..100. mimir vs mimer = 0.953. A `fuzz_threshold` of 95 or below would allow this match. @@ -197,7 +198,7 @@ A Levenshtein match can be scaled to 0..100 with `similarity = (1 − (distance Allow the client to select which fuzzy algorithm is used. Noting that the selection must be supported by the server. -It is proposed that the default algorithm be Jaro-Winkler, and that all implementations should support this. +It is proposed that the default search algorithm be Jaro, and that all implementations should support this. It is proposed that the available fuzzy algorithms be exposed via the `/features` endpoint. @@ -215,17 +216,15 @@ The desired size of each batch of results sent in each response chunk. A value of 0 indicates that the server can determine the batch size, which may be variable. -***next_token*** - -Note that pagination is desirable but not essential to this proposal. - -Note that this parameter can only be used with the `limit` parameter. The request should fail if the `next_token` parameter is set with any other parameter. - ***start/end*** It is proposed that these could have default values which align to a reasonable look-back period. -Ideally this would fall within the WAL. A default look-back of 1 hour is proposed. +Ideally this would fall within the WAL. A default look-back of 1 hour is proposed if no start/end is specified. + +In the scenario of start being set but no end, end should default to now (plus a small sane addition to compensate for clock drift and execution delay). + +In the scenario of end being set but no start, the start should default to the 1 hour range (as above). ***include_*** @@ -303,29 +302,6 @@ The record included in this response would be the same as the first record retur } ``` -##### Example of pagination token (if supported) - -If pagination is supported, the `has_more` attribute is not required. The presence of the `next_token` attribute in the response indicates that there is more results available. - -```ndjson -{ - "results": [ - ... - ] -} - -{ - "results": [ - ... - ] -} - -{ - "status": "success", - "next_token": "" -} -``` - ##### Example no results matching ```ndjson @@ -431,22 +407,21 @@ An endpoint specific to searching for label names. **Query parameters:** -| Name | Type | Required | Default | Description | -|-----------------------|-----------------------------------------|----------|--------------|----------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `search` | string | No | | The search string to be used for matching against label names. | -| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | -| `fuzz_alg` | string | No | jarowrinkler | Select the fuzzy match algorithm. | -| `case_sensitive` | bool | No | true | As per above endpoint. | -| `sort_by` | alpha / cardinality / frequency / score | No | | As per above endpoint. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | -| `include_frequency` | bool | No | false | Request label frequency. | -| `include_cardinality` | bool | No | false | Request label cardinality. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `limit` | int >= 0 | No | 100 | As per above endpoint. | -| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `next_token` | string | No | | As per above endpoint. | +| Name | Type | Required | Default | Description | +|-----------------------|-----------------------------------------|----------|---------|-----------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `search` | []string | No | | The search strings to be used for matching against label names. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `fuzz_alg` | string | No | jaro | Select the fuzzy match algorithm. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / cardinality / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request label frequency. | +| `include_cardinality` | bool | No | false | Request label cardinality. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | **Notes:** @@ -455,7 +430,7 @@ An endpoint specific to searching for label names. * **alpha** - label names are sorted by alphabetical order. * **cardinality** - label names are sorted by their cardinality. * **frequency** - label names are sorted by their frequency. -* **score** - label names are sorted by a matching score. Weighting is given to matches that start with the given search string. The ordering should be optimised for auto-complete use cases. +* **score** - label names are sorted by a matching score. Weighting is given to matches that start with the given search strings. The ordering should be optimised for auto-complete use cases. #### Response @@ -535,22 +510,21 @@ An endpoint specific to searching for label values. **Query parameters:** -| Name | Type | Required | Default | Description | -|---------------------|---------------------------|----------|--------------|-----------------------------------------------------------------| -| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | -| `label` | string | Yes | | The label the user is requesting values for. | -| `search` | string | No | | The search string to be used for matching against label values. | -| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | -| `fuzz_alg` | string | No | jarowrinkler | Select the fuzzy match algorithm. | -| `case_sensitive` | bool | No | true | As per above endpoint. | -| `sort_by` | alpha / frequency / score | No | | As per above endpoint. | -| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | -| `include_frequency` | bool | No | false | Request value frequency. | -| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | -| `limit` | int >= 0 | No | 100 | As per above endpoint. | -| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | -| `next_token` | string | No | | As per above endpoint. | +| Name | Type | Required | Default | Description | +|---------------------|---------------------------|----------|---------|------------------------------------------------------------------| +| `match[]` | string / selectors | No | | Series selector - as per existing labels/values endpoints. | +| `label` | string | Yes | | The label the user is requesting values for. | +| `search` | []string | No | | The search strings to be used for matching against label values. | +| `fuzz_threshold` | int [0..100] | No | 100 | As per above endpoint. | +| `fuzz_alg` | string | No | jaro | Select the fuzzy match algorithm. | +| `case_sensitive` | bool | No | true | As per above endpoint. | +| `sort_by` | alpha / frequency / score | No | | As per above endpoint. | +| `sort_dir` | asc / dsc | No | asc | As per above endpoint. | +| `include_frequency` | bool | No | false | Request value frequency. | +| `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | +| `limit` | int >= 0 | No | 100 | As per above endpoint. | +| `batch_size` | int >= 0 | No | 100 | As per above endpoint. | **Notes:** - The `label` parameter has been deliberately added as a required `query` parameter to avoid issues with the existing [values](https://github.com/prometheus/prometheus/blob/main/docs/querying/api.md#querying-label-values) endpoint which requires the label to be included as a path parameter. @@ -633,6 +607,130 @@ With the combination of this min value and the `limit` it would allow the server For instance, this could be used for searches such as "find me 10 metrics whose name contains `cpu` with a cardinality > 100". +### Fuzz searching and order by score + +There are a range of use cases which should be explored around the use of the search, fuzz_threshold and sort by `score`. + +#### Simple filter (no fuzz) + +This scenario requires the search strings to be set, but the `fuzz_threshold` to be disabled. + +The user is requesting to match on any records which ‘contains’ the given search strings. + +Noting that the search filter is applied after Matchers have been applied. + +It is proposed that the string matching is simply `contains` and does not extend to regular expression matching. + +#### Simple filter (with fuzz) + +This scenario requires the search strings to be set and `fuzz_threshold` and fuzz algorithm selected. + +The user is requesting to match on any records which when applied to the fuzzy algorithm return a score greater than the given threshold. + +It is proposed that this search should also inherently also apply the simple search filter of ‘contains’. The search returns records which contain the given search string `or` meet the fuzz threshold score. + +The reason for this is that in a fuzzy implementation such as Jaro and Jaro-Winkler, results which contain the given search string are easily excluded depending on their position in the record. + +See the table below with example scores for searching for `cpu`. + +__It is proposed for this search, the fuzz algorithm should be Jaro only. No prefix boost. The prefix boosting is more relevant to the use of ordering.__ + +| Metric name | Jaro (no prefix boost) | Jaro-Winkler (p=0.1, max prefix=4) | +|----------------------------------------------------------------------|------------------------|------------------------------------| +| admission_admitted_elastic_cpu | 0.0 | 0.0 | +| admission_admitted_elastic_cpu_bulk_normal_pri | 0.0 | 0.0 | +| admission_elastic_cpu_acquired_nanos | 0.45 | 0.45 | +| asserts:instance:node_cpus:count | 0.45 | 0.45 | +| asserts:namespace:kube_pod_container_resource_requests_cpu_cores:sum | 0.57 | 0.57 | +| admission_elastic_cpu_max_available_nanos | 0.69 | 0.69 | +| admission_granter_cpu_load_short_period_duration_kv | 0.69 | 0.69 | +| admission_cpu_acquired_nanos | 0.70 | 0.70 | +| cpu_admission_admitted_elastic | 0.70 | 0.79 | + +### Order by score + +This scenario requires the search strings to be set and the fuzz search is either enabled or disabled. + +In either case, we have a list of records which need to be ordered in a manner which is logical for auto-completion. + +__It is proposed to use Jaro-Winkler scores to provide the sort order.__ This will inherently give a higher score to those matches with a prefix match on the given search string. Note - experimentation may be needed on the Jaro-Winkler max-prefix input option. + +Ideally, if the search filtering has already calculated the Jaro score for a record this can be re-used and only the prefix-boost needs to be calculated and added to the score. + +For searches which did not use any fuzzy filtering, the records would be passed through a fresh Jaro-Winkler calculation. + +### storage.LabelQuerier interface changes + +The following discusses a proposal for storage querier interface changes to support this API implementation. + +The existing Labels/Values API leverages the [storage.LabelQuerier](https://github.com/prometheus/prometheus/blob/main/storage/interface.go#L175-L189) interface. + +Additional parameters are passed to the search functions using the [storage.LabelHints](https://github.com/prometheus/prometheus/blob/main/storage/interface.go#L254-L257) structure. + +#### New interfaces + +```go +// FilteredResult is an intermediate result which is created when applying a Filter when searching for label/values. +// A fuzzy match score (if any) is returned so that it can be used by any subsequent comparators. +type FilteredResult struct { + Value string // The metric name, label name or label value. + Score float64 // Relevance score in [0.0, 1.0]. 1.0 is returned if no fuzzy filter was applied in the filtering. +} + +// Filter is used to reduce the set of labels/values. +type Filter interface { + // Accept returns whether the value should be included (accepted). + // Score is the input from the user indicating the fuzzy score threshold required for a match. 0 requests no fuzzy matching, 100 requests an exact match. + Accept(value string) (accepted bool, score float64) +} + +// Comparator is used for ordering the returned SearchResults. +// The implementation may use the SearchResult.Score as is, or may use this as an input to its own algorithm for sorting. +// For instance, a Filter may have applied a Jaro fuzzy filter and this is returned in the SearchResult.Score. The Comparator implementation may add a prefix boost score +// to the Jaro score, there by applying a Jaro-Winkler sort ordering. +type Comparator interface { + Compare(a, b FilteredResult) int +} + +// SearchHints is the input to the labels/values Searcher. It allows for a filter and comparator function to be passed into the Searcher. +// Note that both the filter and comparator are optional. +type SearchHints struct { + // Filter determines which values to include and their relevance scores. + // A nil Filter accepts all values and will return a SearchResult.Score of 1.0 + Filter Filter + + // Limit is the maximum number of results to return. + // Use 0 to disable limiting. + Limit int + + // Compare is used for ordering results. + // A nil value means NO sort ordering is applied. + Compare Comparator +} + +// SearcherValueSet is an iterator returned from the Searcher label/value search functions. +type SearcherValueSet interface { + Next() bool + At() string + Warnings() annotations.Annotations + Err() error + // close this iterator and releases its resources. This does not close the Searcher. The iterator should be closed before the Searcher is closed. + Close() +} + +// Searcher allows for the searching, filtering and ordering of label names and values. The result set is accessible via an SearcherValueSet iterator. +type Searcher interface { + // SearchLabelNames returns label names matching the search criteria. + // The SearcherValueSet iterator is ordered by any given Comparator. + SearchLabelNames(ctx context.Context, hints *SearchHints, matchers ...*labels.Matcher) (SearcherValueSet, error) + + // SearchLabelValues returns label values for the given label name. + // The SearcherValueSet iterator is ordered by any given Comparator. + SearchLabelValues(ctx context.Context, name string, hints *SearchHints, matchers ...*labels.Matcher) (SearcherValueSet, error) +} + +``` + ### Extensibility for Mimir, Thanos, Cortex The introduction of returning collections of records rather than collections of strings allows for different implementations to provide additional record decorations. @@ -702,7 +800,7 @@ These are new endpoints and do not change or alter any existing functionality. N ### Known unknowns * confirm feasibility of including frequency and cardinality in these API responses -* confirm requirement of supporting cursor based pagination for these new endpoints +* confirm requirement of supporting cursor based pagination for these new endpoints - confirmed as not required * confirm any performance / response time constraints for these new endpoints * specific choice of fuzzy search algorithm * specific implementation of the search result ordering for auto-complete scenarios @@ -724,7 +822,7 @@ The existing API design and response format is constrained. The existing response format returns collections of strings which does not support additional record enrichment. -Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting, support pagination etc - these would all be significant internal functional changes to these endpoints. +Although the existing endpoints could be adapted to (optionally) return streaming/batched results, apply filtering and sorting etc - these would all be significant internal functional changes to these endpoints. ### 3. Why do we need a 3rd endpoint specific to metric names - it's just a special case of \_*name*_? @@ -738,7 +836,7 @@ A dedicated endpoint also allows for future enrichments to be added which are on The tasks to do in order to migrate to the new idea. -* [ ] Confirm requirement for supporting pagination or not +* [x] Confirm requirement for supporting pagination or not - not required * [ ] Finalize proposal based on community feedback * [ ] Create Prometheus implementation issue for `/api/v1/search/metric_names` * [ ] Create Prometheus implementation issue for `/api/v1/search/label_names` From 5430015a333ecce7c8beb53efedbae5ac2fc6b89 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Wed, 25 Feb 2026 15:50:35 +0800 Subject: [PATCH 25/28] Lint Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 38 ++++++++++++------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 753bc83..f1ab205 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -629,13 +629,13 @@ The user is requesting to match on any records which when applied to the fuzzy a It is proposed that this search should also inherently also apply the simple search filter of ‘contains’. The search returns records which contain the given search string `or` meet the fuzz threshold score. -The reason for this is that in a fuzzy implementation such as Jaro and Jaro-Winkler, results which contain the given search string are easily excluded depending on their position in the record. +The reason for this is that in a fuzzy implementation such as Jaro and Jaro-Winkler, results which contain the given search string are easily excluded depending on their position in the record. See the table below with example scores for searching for `cpu`. -__It is proposed for this search, the fuzz algorithm should be Jaro only. No prefix boost. The prefix boosting is more relevant to the use of ordering.__ +**It is proposed for this search, the fuzz algorithm should be Jaro only. No prefix boost. The prefix boosting is more relevant to the use of ordering.** -| Metric name | Jaro (no prefix boost) | Jaro-Winkler (p=0.1, max prefix=4) | +| Metric name | Jaro (no prefix boost) | Jaro-Winkler (p=0.1, max prefix=4) | |----------------------------------------------------------------------|------------------------|------------------------------------| | admission_admitted_elastic_cpu | 0.0 | 0.0 | | admission_admitted_elastic_cpu_bulk_normal_pri | 0.0 | 0.0 | @@ -653,7 +653,7 @@ This scenario requires the search strings to be set and the fuzz search is eithe In either case, we have a list of records which need to be ordered in a manner which is logical for auto-completion. -__It is proposed to use Jaro-Winkler scores to provide the sort order.__ This will inherently give a higher score to those matches with a prefix match on the given search string. Note - experimentation may be needed on the Jaro-Winkler max-prefix input option. +**It is proposed to use Jaro-Winkler scores to provide the sort order.** This will inherently give a higher score to those matches with a prefix match on the given search string. Note - experimentation may be needed on the Jaro-Winkler max-prefix input option. Ideally, if the search filtering has already calculated the Jaro score for a record this can be re-used and only the prefix-boost needs to be calculated and added to the score. @@ -663,7 +663,7 @@ For searches which did not use any fuzzy filtering, the records would be passed The following discusses a proposal for storage querier interface changes to support this API implementation. -The existing Labels/Values API leverages the [storage.LabelQuerier](https://github.com/prometheus/prometheus/blob/main/storage/interface.go#L175-L189) interface. +The existing Labels/Values API leverages the [storage.LabelQuerier](https://github.com/prometheus/prometheus/blob/main/storage/interface.go#L175-L189) interface. Additional parameters are passed to the search functions using the [storage.LabelHints](https://github.com/prometheus/prometheus/blob/main/storage/interface.go#L254-L257) structure. @@ -673,8 +673,8 @@ Additional parameters are passed to the search functions using the [storage.Labe // FilteredResult is an intermediate result which is created when applying a Filter when searching for label/values. // A fuzzy match score (if any) is returned so that it can be used by any subsequent comparators. type FilteredResult struct { - Value string // The metric name, label name or label value. - Score float64 // Relevance score in [0.0, 1.0]. 1.0 is returned if no fuzzy filter was applied in the filtering. + Value string // The metric name, label name or label value. + Score float64 // Relevance score in [0.0, 1.0]. 1.0 is returned if no fuzzy filter was applied in the filtering. } // Filter is used to reduce the set of labels/values. @@ -710,23 +710,23 @@ type SearchHints struct { // SearcherValueSet is an iterator returned from the Searcher label/value search functions. type SearcherValueSet interface { - Next() bool - At() string - Warnings() annotations.Annotations - Err() error - // close this iterator and releases its resources. This does not close the Searcher. The iterator should be closed before the Searcher is closed. - Close() + Next() bool + At() string + Warnings() annotations.Annotations + Err() error + // close this iterator and releases its resources. This does not close the Searcher. The iterator should be closed before the Searcher is closed. + Close() } // Searcher allows for the searching, filtering and ordering of label names and values. The result set is accessible via an SearcherValueSet iterator. type Searcher interface { - // SearchLabelNames returns label names matching the search criteria. + // SearchLabelNames returns label names matching the search criteria. // The SearcherValueSet iterator is ordered by any given Comparator. SearchLabelNames(ctx context.Context, hints *SearchHints, matchers ...*labels.Matcher) (SearcherValueSet, error) - - // SearchLabelValues returns label values for the given label name. - // The SearcherValueSet iterator is ordered by any given Comparator. - SearchLabelValues(ctx context.Context, name string, hints *SearchHints, matchers ...*labels.Matcher) (SearcherValueSet, error) + + // SearchLabelValues returns label values for the given label name. + // The SearcherValueSet iterator is ordered by any given Comparator. + SearchLabelValues(ctx context.Context, name string, hints *SearchHints, matchers ...*labels.Matcher) (SearcherValueSet, error) } ``` @@ -836,7 +836,7 @@ A dedicated endpoint also allows for future enrichments to be added which are on The tasks to do in order to migrate to the new idea. -* [x] Confirm requirement for supporting pagination or not - not required +* [X] Confirm requirement for supporting pagination or not - not required * [ ] Finalize proposal based on community feedback * [ ] Create Prometheus implementation issue for `/api/v1/search/metric_names` * [ ] Create Prometheus implementation issue for `/api/v1/search/label_names` From 6c4e2b1dc7e89cc5f6b7ab3b32a06682b5248105 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Fri, 27 Feb 2026 13:51:33 +0800 Subject: [PATCH 26/28] Update the iterator to return the string + score not just the string Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index f1ab205..9281406 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -711,7 +711,7 @@ type SearchHints struct { // SearcherValueSet is an iterator returned from the Searcher label/value search functions. type SearcherValueSet interface { Next() bool - At() string + At() FilteredResult Warnings() annotations.Annotations Err() error // close this iterator and releases its resources. This does not close the Searcher. The iterator should be closed before the Searcher is closed. From e41ec76b7375cafc826347c43af47b6fd8bf1d42 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Tue, 3 Mar 2026 10:58:34 +0800 Subject: [PATCH 27/28] Added include_score option Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index 9281406..fadae7f 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -158,6 +158,7 @@ An endpoint specific to searching for metric names (\_\_name__ values) and obtai | `sort_dir` | asc / dsc | No | asc | Request the ordering of the sort. Only valid when `sort_by` is set. | | `include_cardinality` | bool | No | false | Request metric cardinality. | | `include_metadata` | bool | No | false | Request metric metadata (units, type, description). | +| `include_score` | bool | No | false | Request the fuzz search score to be returned for reach result. | | `start` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | | `end` | rfc3339 / unix_timestamp | No | | As per the existing labels/values endpoint. | | `limit` | int >= 0 | No | 100 | The maximum number of results to return after any ordering has been applied. | @@ -275,6 +276,7 @@ The record included in this response would be the same as the first record retur "results": [ { "name": "activity_tracker_failed_total", + "score": 0.76, "cardinality": 10, "type": "counter", "help": "How many times has activity tracker failed to insert new activity", @@ -282,6 +284,7 @@ The record included in this response would be the same as the first record retur }, { "name": "activity_tracker_free_slots", + "score": 0.50, "cardinality": 50, "type": "gauge", "help": "Number of free slots in activity file.", @@ -418,6 +421,7 @@ An endpoint specific to searching for label names. | `sort_dir` | asc / dsc | No | asc | As per above endpoint. | | `include_frequency` | bool | No | false | Request label frequency. | | `include_cardinality` | bool | No | false | Request label cardinality. | +| `include_score` | bool | No | false | Request the fuzz search score to be returned for reach result. | | `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | | `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | | `limit` | int >= 0 | No | 100 | As per above endpoint. | @@ -521,6 +525,7 @@ An endpoint specific to searching for label values. | `sort_by` | alpha / frequency / score | No | | As per above endpoint. | | `sort_dir` | asc / dsc | No | asc | As per above endpoint. | | `include_frequency` | bool | No | false | Request value frequency. | +| `include_score` | bool | No | false | Request the fuzz search score to be returned for reach result. | | `start` | rfc3339 / unix_timestamp | No | | As per above endpoint. | | `end` | rfc3339 / unix_timestamp | No | | As per above endpoint. | | `limit` | int >= 0 | No | 100 | As per above endpoint. | From 5d75519113ed10f6d444c822bb195ea76eb9f746 Mon Sep 17 00:00:00 2001 From: Andrew Hall Date: Thu, 5 Mar 2026 14:00:35 +0800 Subject: [PATCH 28/28] Fix incorrect frequency example Signed-off-by: Andrew Hall --- proposals/0074-new-labels-values-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0074-new-labels-values-api.md b/proposals/0074-new-labels-values-api.md index fadae7f..e70685e 100644 --- a/proposals/0074-new-labels-values-api.md +++ b/proposals/0074-new-labels-values-api.md @@ -129,7 +129,7 @@ cpu_usage_seconds_total{core="0", instance="b"} | http_requests_total | metric_name | N/A | 5 | | http_requests_total{method="GET"} | metric_name | N/A | 2 | | method | label_name | 5 | 2 | -| core | label_name | 4 | 2 | +| core | label_name | 3 | 2 | | method="GET" | label_value | 2 | N/A | | GET | label_value | 2 | N/A |