Bulk API failed with HttpHostConnectException: Unable to connect to the data source, response 422

[chandrams]

Describe the bug
Specifying a time range that is older than 15 days in the bulk configuration, fails with
HttpHostConnectException: Unable to connect to the data source, unexpected response 422

For example, for the time range

  "time_range": {
                "start": "2024-12-17T08:00:00.000Z",
                "end": "2024-12-17T14:00:00.000Z"
    }

How to reproduce it

• Setup thanos with tsdb blocks for a time range beyond 15 days
• Deploy kruize with thanos as datasource
• Invoke bulk API with time_range of the tsdb blocks, above exception is seen

Expected behavior
No exception should be thrown and recommendations should be generated

Relevant logs

{
    "summary": {
        "status": "FAILED",
        "total_experiments": 0,
        "processed_experiments": 0,
        "notifications": {
            "503": {
                "type": "ERROR",
                "message": "HttpHostConnectException: Unable to connect to the data source. Please try again later. (Unexpected response status: 422)",
                "code": 503
            }
        },
        "input": {
            "filter": {
                "exclude": {
                    "namespace": [],
                    "workload": [],
                    "containers": [],
                    "labels": {}
                },
                "include": {
                    "namespace": [],
                    "workload": [],
                    "containers": [],
                    "labels": {
                        "org_id": "org-1",
                        "cluster_id": "eu-1-1"
                    }
                }
            },
            "time_range": {
                "start": "2024-12-17T08:00:00.000Z",
                "end": "2024-12-17T14:00:00.000Z"
            },
            "datasource": "thanos",
            "webhook": null
        },

Environment:

• Kubernetes Cluster openshift