Pipe: Implemented tree auth check for source + write-back-sink #16531

Caideyipi · 2025-09-30T06:45:39Z

Description

As the title said.

This PR has:

Key changed/added classes (or packages if there are too many classes) in this PR

* fix * fix * fix * fix * fix * fix * fix * fix * refactor * fix * fix * partial * grass * Update PipeConfigTreePrivilegeParseVisitor.java * Feature/client hide password (#16468) * session hide password * fix start-cli.sh * fix start-cli.bat * client hide password * fix client warning * echo cmd line * fix * Update PipeConfigTreePrivilegeParseVisitor.java * partial * partial * fix * fix * partial --------- Co-authored-by: Hongzhi Gao <[email protected]>

…to fix-audit-logger

* fix * fix * fix * fix * fix * fix * fix * fix * refactor * fix * fix * partial * grass * Update PipeConfigTreePrivilegeParseVisitor.java * Feature/client hide password (#16468) * session hide password * fix start-cli.sh * fix start-cli.bat * client hide password * fix client warning * echo cmd line * fix * Update PipeConfigTreePrivilegeParseVisitor.java * partial * partial * fix * fix * partial --------- Co-authored-by: Hongzhi Gao <[email protected]>

…to fix-audit-logger

…it-logger

# Conflicts: # integration-test/src/test/java/org/apache/iotdb/pipe/it/dual/treemodel/auto/basic/IoTDBTreePatternFormatIT.java # integration-test/src/test/java/org/apache/iotdb/pipe/it/dual/treemodel/manual/IoTDBPipeInclusionIT.java # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/source/PipeConfigTreePatternParseVisitor.java # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/auth/AuthorInfo.java # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/schema/CNPhysicalPlanGenerator.java # iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/manager/pipe/source/PipeConfigTreePatternParseVisitorTest.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/connection/PipeEventCollector.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/parser/TsFileInsertionEventParser.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/parser/TsFileInsertionEventParserProvider.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/parser/query/TsFileInsertionEventQueryParser.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/parser/scan/TsFileInsertionEventScanParser.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/parser/table/TsFileInsertionEventTableParser.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/receiver/visitor/PipeTableStatementDataTypeConvertExecutionVisitor.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/config/executor/ClusterConfigTaskExecutor.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/converter/LoadTableStatementDataTypeConvertExecutionVisitor.java # iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/datastructure/pattern/IoTDBTreePattern.java

# Conflicts: # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/source/PipeConfigTreePatternParseVisitor.java # iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/connection/PipeEventCollector.java # iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/datastructure/pattern/IoTDBTreePattern.java

…it-logger

...test/src/test/java/org/apache/iotdb/pipe/it/dual/treemodel/manual/IoTDBPipePermissionIT.java

jt2594838 · 2025-11-19T09:47:27Z

...test/src/test/java/org/apache/iotdb/pipe/it/dual/treemodel/manual/IoTDBPipePermissionIT.java

+    // A user shall only see its own pipe
+    try (final SyncConfigNodeIServiceClient client =
+        (SyncConfigNodeIServiceClient) senderEnv.getLeaderConfigNodeConnection()) {
+      Assert.assertEquals(
+          1, client.showPipe(new TShowPipeReq().setUserName("thulab")).pipeInfoList.size());
+    } catch (Exception e) {
+      fail(e.getMessage());
+    }


You can show another user's pipe simply by setting the request's username?

Yes, but if the config client can be initialized outside of IoTDB, a user can do everything without permission, because the permissions are checked on DataNode. So users shall not have the chance to directly "set" the request's username

jt2594838 · 2025-11-19T09:57:04Z

...e/src/main/java/org/apache/iotdb/confignode/manager/pipe/source/IoTDBConfigRegionSource.java

+    if (!Boolean.TRUE.equals(isTableDatabasePlan)) {
+      final Optional<ConfigPhysicalPlan> result = treePrivilegeParseVisitor.process(plan, userName);
+      if (result.isPresent()) {
+        return Optional.of(
+            new PipeConfigRegionWritePlanEvent(result.get(), event.isGeneratedByPipe()));
+      }
    }
-
-    final Optional<ConfigPhysicalPlan> result =
-        TABLE_PRIVILEGE_PARSE_VISITOR.process(plan, userName);
-    if (result.isPresent()) {
-      return Optional.of(
-          new PipeConfigRegionWritePlanEvent(result.get(), event.isGeneratedByPipe()));
+    if (!Boolean.FALSE.equals(isTableDatabasePlan)) {


Reverse the conditions to remove unnecessary !.

This cannot be done because other than table database, the planNodes are processed by both of the visitor but will only be filtered by one and pass directly through another. The ! is to tell 'null + FALSE' and 'TRUE + NULL'

jt2594838 · 2025-11-19T10:03:11Z

...ava/org/apache/iotdb/confignode/manager/pipe/source/PipeConfigTreePrivilegeParseVisitor.java

+  public boolean canShowSchemaTemplate(final String templateName, final String userName) {
+    try {
+      return ConfigNode.getInstance()
+                  .getConfigManager()
+                  .getPermissionManager()
+                  .checkUserPrivileges(userName, new PrivilegeUnion(PrivilegeType.SYSTEM))
+                  .getStatus()
+                  .getCode()
+              == TSStatusCode.SUCCESS_STATUS.getStatusCode()
+          || ConfigNode.getInstance()
+              .getConfigManager()
+              .getClusterSchemaManager()
+              .getPathsSetTemplate(templateName, ALL_MATCH_SCOPE)
+              .getPathList()
+              .stream()
+              .anyMatch(
+                  path -> {
+                    try {
+                      return ConfigNode.getInstance()
+                              .getConfigManager()
+                              .getPermissionManager()
+                              .checkUserPrivileges(
+                                  userName,
+                                  new PrivilegeUnion(
+                                      Collections.singletonList(
+                                          new PartialPath(path)
+                                              .concatNode(MULTI_LEVEL_PATH_WILDCARD)),
+                                      PrivilegeType.READ_SCHEMA))
+                              .getStatus()
+                              .getCode()
+                          == TSStatusCode.SUCCESS_STATUS.getStatusCode();
+                    } catch (final IllegalPathException e) {
+                      throw new RuntimeException(e);
+                    }
+                  });
+    } catch (final Exception e) {
+      LOGGER.warn(
+          "Un-parse-able path name encountered during template privilege trimming, please check",
+          e);
+      return false;
+    }
+  }


Add a test to cover this？

...ava/org/apache/iotdb/confignode/manager/pipe/source/PipeConfigTreePrivilegeParseVisitor.java

jt2594838 · 2025-11-19T10:45:52Z

...ode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/PipeTsFileInsertionEvent.java

+        final Set<IDeviceID> devices = getDeviceSet();
+        if (Objects.nonNull(devices)) {
+          final List<MeasurementPath> measurementList = new ArrayList<>();
+          for (final IDeviceID device : devices) {
+            measurementList.add(new MeasurementPath(device, IoTDBConstant.ONE_LEVEL_PATH_WILDCARD));
+          }
+          final TSStatus status =
+              AuthorityChecker.getAccessControl()
+                  .checkSeriesPrivilege4Pipe(
+                      new UserEntity(Long.parseLong(userId), userName, cliHostname),
+                      measurementList,
+                      PrivilegeType.READ_DATA);
+          if (TSStatusCode.SUCCESS_STATUS.getStatusCode() != status.getCode()) {
+            if (skipIfNoPrivileges) {
+              shouldParse4Privilege = true;
+            } else {
+              throw new AccessDeniedException(status.getMessage());
+            }
+          }
+        } else {
+          shouldParse4Privilege = true;
+        }
+      }


If the file only contains root.db1.d1.s1 (on which the user has privileges), but the user does not have privileges on root.db1.s2, and skipIfNoPrivileges = false, then the file will not be transferred?

Of course, "skipIf == false" means the pipe will stop for any data/schema without privilege.

...e/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/writeback/WriteBackSink.java

jt2594838 · 2025-11-19T11:02:15Z

...node/src/main/java/org/apache/iotdb/db/pipe/source/schemaregion/IoTDBSchemaRegionSource.java

-        TABLE_PRIVILEGE_PARSE_VISITOR.process(
-            ((PipeSchemaRegionWritePlanEvent) event).getPlanNode(), userEntity);
+        treePrivilegeParseVisitor
+            .process(((PipeSchemaRegionWritePlanEvent) event).getPlanNode(), userEntity)
+            .flatMap(planNode -> TABLE_PRIVILEGE_PARSE_VISITOR.process(planNode, userEntity));


The same node is processed by both treeVisitor and tableVisitor?

This is OK because one kind of node will only be parsed by one visitor. In the other one it just "pass by"

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/parser/ASTVisitor.java

...mons/src/main/java/org/apache/iotdb/commons/pipe/datastructure/pattern/IoTDBTreePattern.java

…it-logger

sonarqubecloud · 2025-12-04T02:04:17Z

Quality Gate passed

Issues
81 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.0% Duplication on New Code

See analysis details on SonarQube Cloud

CRZbulabula and others added 30 commits September 24, 2025 09:38

remove internal auditor

88178c4

stash 4 collaborate

ca82a9a

Update PermissionManager.java

2373adb

entity

3c5d69d

FIX COMPILE BUGS

cffbe6d

Merge branch 'fix-audit-logger' of https://github.com/apache/iotdb in…

6efdb68

…to fix-audit-logger

fix

8f23025

fix

342b253

fix

5af7057

fix

cd6ad89

fix

11c78af

little

df187a1

fix

13e4fdd

remove internal auditor

2c6ead7

stash 4 collaborate

8858086

FIX COMPILE BUGS

8e96f8d

Update PermissionManager.java

b42cd44

entity

de63e47

fix

f953015

fix

8b13d1d

fix

f03f520

fix

4629947

fix

3202503

little

ede612e

fix

d38d829

Merge branch 'fix-audit-logger' of https://github.com/apache/iotdb in…

27530de

…to fix-audit-logger

partial

ec8fa89

append log 4 tree

d8e534a

Caideyipi added 18 commits October 10, 2025 10:18

fix

4337d4a

fix

8f64acd

unwebbed-fish

b0d8269

Merge branch 'master' of https://github.com/apache/iotdb into fix-aud…

327973e

…it-logger

refactor

f4344ec

fix

3996d4e

fix

e5dd2aa

add-IT

7ea8163

fix

c45305c

fix

d22189f

fix

7fb2c09

user-null-fix

9bb3dcd

fix-some

c86cf29

user

0df40e3

Merge branch 'master' of https://github.com/apache/iotdb into fix-aud…

90dcbd3

…it-logger

Merge branch 'master' of https://github.com/apache/iotdb into fix-aud…

9a4b680

…it-logger

jt2594838 reviewed Nov 20, 2025

View reviewed changes

Caideyipi added 10 commits December 3, 2025 16:29

Merge branch 'master' of https://github.com/apache/iotdb into fix-aud…

088bd1c

…it-logger

refactor

86c9cf9

ap

0353c6b

fix

4ebc890

fix

ab881e1

fix

09c0b70

apply

45d3200

codecov

c08b962

fix

3d6ff40

f

705f73d

Pipe: Implemented tree auth check for source + write-back-sink #16531

Are you sure you want to change the base?

Pipe: Implemented tree auth check for source + write-back-sink #16531

Conversation

Caideyipi commented Sep 30, 2025

Description

Key changed/added classes (or packages if there are too many classes) in this PR

Uh oh!

Uh oh!

jt2594838 Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Caideyipi Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jt2594838 Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Caideyipi Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

jt2594838 Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jt2594838 Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Caideyipi Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jt2594838 Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Caideyipi Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Dec 4, 2025

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Caideyipi Dec 3, 2025 •

edited

Loading

Caideyipi Dec 3, 2025 •

edited

Loading