Unable to update datasource

[navpreet-securitas]

Describe the bug
Operator fails to update existing datasource. tempo data source was created via GrafanaDataSource CRD. When now i try to update url in datasource , it gives error :

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  creationTimestamp: "2025-10-29T10:46:05Z"
  finalizers:
  - operator.grafana.com/finalizer
  generation: 2
  labels:
    kustomize.toolkit.fluxcd.io/name: grafana
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: tempo
  namespace: monitoring
  resourceVersion: "52104481"
  uid: 3aa9f134-c7ef-453c-b281-1a011e46e62f
spec:
  allowCrossNamespaceImport: false
  datasource:
    access: proxy
    editable: true
    jsonData:
      httpMethod: GET
      lokiSearch:
        datasourceUid: Loki
      nodeGraph:
        enabled: true
      search:
        hide: false
      serviceMap:
        datasourceUid: Prometheus-Thanos
      tracesToLogs:
        datasourceUid: Loki
        filterBySpanID: true
        filterByTraceID: true
        spanEndTimeShift: 1h
        spanStartTimeShift: -5m
        tags:
        - job
        - instance
        - pod
        - namespace
    name: Tempo
    type: tempo
    url: http://grafana-tempo-query-frontend.monitoring.svc.cluster.local:3200
  instanceSelector:
    matchLabels:
      dashboards: grafana
  resyncPeriod: 10m0s
status:
  conditions:
  - lastTransitionTime: "2025-10-29T10:59:25Z"
    message: |-
      Datasource failed to be applied for 1 out of 1 instances. Errors:
      - monitoring/grafana: [PUT /datasources/uid/{uid}][403] updateDataSourceByUidForbidden {"message":"Cannot update read-only data source"}
    observedGeneration: 2
    reason: ApplyFailed
    status: "False"
    type: DatasourceSynchronized

Version
v5.20.0

To Reproduce
Steps to reproduce the behavior:

1. create a datasource via CRD. it gets cerated and available in grafana deployment
2. then update a field in same datasource defination and apply file.(in our case done by fluxcd)
3. checking status of datasource : it gives

message: |-
     Datasource failed to be applied for 1 out of 1 instances. Errors:
     - monitoring/grafana: [PUT /datasources/uid/{uid}][403] updateDataSourceByUidForbidden {"message":"Cannot update read-only data source"}
   observedGeneration: 2
   reason: ApplyFailed
   status: "False"
   type: DatasourceSynchronized

4. logs in grafana-operator shows:

2025-10-29T11:04:43Z    ERROR   Reconciler error        {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"azure-monitor","namespace":"monitoring"}, "namespace": "monitoring", "name": "azure-monitor", "reconcileID": "cd819416-0711-4a1f-91c1-bf50c688cd68", "error": "failed to apply to all instances: map[monitoring/grafana:[PUT /datasources/uid/{uid}][403] updateDataSourceByUidForbidden {\"message\":\"Cannot update read-only data source\"}]"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:474
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:421
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:296

Expected behavior
datasource to be updated automatically.

Suspect component/Location where the bug might be occurring
Please provide this if you know where this bug might occur otherwise leave as unknown

Screenshots
If applicable, add screenshots to help explain your problem.

Runtime (please complete the following information):

• OS: Linux
• Grafana Operator Version: v5.0.0
• Environment: Kubernetes
• Deployment type: Helm/kustomize

Additional context
Add any other context about the problem here.

[Baarsgaard]

Are you by any chance in the middle of migrating away from a file provisioned datasource?
Specifically located in provisioning/datasources/ similar to: https://github.com/grafana/grafana/blob/main/conf/provisioning/datasources/sample.yaml

Datasources provisioned through files cannot be edited via the UI or the API unless they are provisioned with editable: true
In older versions of Grafana, there was the additional field readOnly: false which sometimes have to be set to enable editing via the API.

Alternatively, provisioning a deletion config for the datasource will allow the operator to create the datasource through the API from scratch.
https://github.com/grafana/grafana/blob/6093afddd4991a390030000f8c43eaf222849cf4/conf/provisioning/datasources/sample.yaml#L4-L7
Be vary of cleaning up the deletion config again as it will cause Grafana to delete the config on every restart otherwise.

[navpreet-securitas]

yea , now i get to the root of this.
previously we were deploying grafana directly (without operator) and then in that HR we had datasources defined as:

datasources: 
       datasources.yaml:
         apiVersion: 1
         datasources:
             - name: Tempo
                type: tempo
                access: proxy
                url: http://grafana-tempo-query-frontend.monitoring.svc.cluster.local:3100

and then when we moved to grafana operator , we use same database. Probably with grafana deployment using same old database( which have datasources provisioned) is considering these datasources as provisioned ones. even though with operator we provided datasources as crds.

@Baarsgaard does this makes sense ?
and since now we are using crd / operator , how can we remove old datasources and recreate them via crd ?

[Baarsgaard]

You have three ways of solving it.

• Provision a deletion config as shown here: https://github.com/grafana/grafana/blob/6093afddd4991a390030000f8c43eaf222849cf4/conf/provisioning/datasources/sample.yaml#L4-L7
  And restarting Grafana

• Updating the datasource through file provisioning by setting editable: true
  Restart Grafana
  Then delete the provisioning config afterwards so the datasource does not reset when Grafana restarts.

• Connect to the database and update the datasource row directly to enable editing.