Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -55,4 +55,7 @@ dolphinscheduler-master/logs
dolphinscheduler-api/logs
__pycache__
ds_schema_check_test

# Agent
docs/superpowers/
.claude/worktrees
46 changes: 44 additions & 2 deletions dolphinscheduler-api/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,50 @@ REST API server. Entry point for the UI and external clients (curl, Python SDK).

## Tests

- Unit tests in `src/test/java`.
- **Integration tests live in `dolphinscheduler-api-test`** (separate module, Docker Compose + Testcontainers).
Unit tests live in `src/test/java`. **Integration tests live in `dolphinscheduler-api-test`** (separate module, Docker Compose + Testcontainers).

### Running unit tests from the command line

The CI flow (`.github/workflows/unit-test.yml`) runs in two phases. Mirror it locally:

**One-time setup** — install BOM + upstream deps to local m2:

```bash
export MAVEN_OPTS="-Xmx4g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m"
./mvnw install -B \
-pl "dolphinscheduler-bom,dolphinscheduler-api" \
-am -DskipTests=true -Dspotless.skip=true -DskipUT=true \
-Djacoco.skip=true -Danalyze.skip=true
```

**Run tests** — use `verify` (NOT `test`), no `-am`, no `clean`:

```bash
./mvnw verify -B -pl "dolphinscheduler-api" \
-Dmaven.test.skip=false -Dspotless.skip=true -DskipUT=false -Danalyze.skip=true \
-Dsurefire.printSummary=true -Dsurefire.useFile=false \
-Dsurefire.reportFormat=plain -Dsurefire.redirectTestOutputToFile=false \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dtest='EnvironmentServiceTest#testVerifyEnvironment'
```

Drop the `-Dtest=...` filter to run the whole module.

### Why the unusual flags

- **`verify` not `test`** — root pom uses `jacoco-maven-plugin` in offline-instrument + restore-instrumented-classes mode. Only the `verify` lifecycle runs the `restore` goal, so `test` alone leaves `target/classes` instrumented and the next build fails with "Cannot process instrumented class".
- **`dolphinscheduler-bom` must be in the install `-pl`** — without it, the installed pom lacks effective dependencyManagement and downstream compile/test classpaths drop transitive deps like `commons-collections4` and `oshi-core`.
- **`-DskipTests=true` (not `-Dmaven.test.skip=true`) during install** — `maven.test.skip` skips test-source attach which breaks downstream modules that depend on the test-jar.
- **No `-am` on the test run** — `-am` re-instruments upstream modules with jacoco and triggers the same "Cannot process instrumented class" failure.
- **No `clean` between `install` and `verify`** — clean wipes `target/classes` and the next compile re-resolves transitive deps from the (still installed) pom; some resolutions fail with `[WARNING] The POM for ... is invalid` and the build can't see `commons-collections4` etc.

### If things go sideways

- `Cannot process instrumented class` — re-run with `-Djacoco.skip=true`, or wipe `target/` for the affected module and re-run.
- `NoClassDefFoundError: oshi/SystemInfo` or `commons-collections4/CollectionUtils` at test runtime — the bom didn't get installed; redo the setup step including `dolphinscheduler-bom` in `-pl`.
- `class file contains wrong class` — local `~/.m2/.../dolphinscheduler-*-dev-SNAPSHOT.jar` is stale/corrupt. Delete the offending `dev-SNAPSHOT` directory under `~/.m2/repository/org/apache/dolphinscheduler/` and re-run install.

Surefire XML reports land at `target/surefire-reports/TEST-<class>.xml` for the per-test counts.

## Related modules

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@

import org.apache.dolphinscheduler.api.audit.OperatorLog;
import org.apache.dolphinscheduler.api.audit.enums.AuditType;
import org.apache.dolphinscheduler.api.dto.EnvironmentDto;
import org.apache.dolphinscheduler.api.exceptions.ApiException;
import org.apache.dolphinscheduler.api.service.EnvironmentService;
import org.apache.dolphinscheduler.api.utils.Result;
Expand All @@ -34,7 +35,7 @@
import org.apache.dolphinscheduler.dao.entity.User;
import org.apache.dolphinscheduler.plugin.task.api.utils.ParameterUtils;

import java.util.Map;
import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
Expand Down Expand Up @@ -127,7 +128,6 @@
* query environment details by code
*
* @param environmentCode environment code
* @return environment detail information
*/
@Operation(summary = "queryEnvironmentByCode", description = "QUERY_ENVIRONMENT_BY_CODE_NOTES")
@Parameters({
Expand All @@ -136,11 +136,10 @@
@GetMapping(value = "/query-by-code")
@ResponseStatus(HttpStatus.OK)
@ApiException(QUERY_ENVIRONMENT_BY_CODE_ERROR)
public Result queryEnvironmentByCode(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
@RequestParam("environmentCode") Long environmentCode) {

Map<String, Object> result = environmentService.queryEnvironmentByCode(environmentCode);
return returnDataList(result);
public Result<EnvironmentDto> queryEnvironmentByCode(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,

Check notice

Code scanning / CodeQL

Useless parameter Note

The parameter 'loginUser' is never used.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ignore, will remove the loginUser at another PR, we should use ThreadLocal to hold this.

@RequestParam("environmentCode") Long environmentCode) {
EnvironmentDto dto = environmentService.queryEnvironmentByCode(environmentCode);
return Result.success(dto);
}

/**
Expand Down Expand Up @@ -175,7 +174,6 @@
*
* @param loginUser login user
* @param environmentCode environment code
* @return delete result code
*/
@Operation(summary = "deleteEnvironmentByCode", description = "DELETE_ENVIRONMENT_BY_CODE_NOTES")
@Parameters({
Expand All @@ -185,34 +183,31 @@
@ResponseStatus(HttpStatus.OK)
@ApiException(DELETE_ENVIRONMENT_ERROR)
@OperatorLog(auditType = AuditType.ENVIRONMENT_DELETE)
public Result deleteEnvironment(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
@RequestParam("environmentCode") Long environmentCode) {

Map<String, Object> result = environmentService.deleteEnvironmentByCode(loginUser, environmentCode);
return returnDataList(result);
public Result<Void> deleteEnvironment(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
@RequestParam("environmentCode") Long environmentCode) {
environmentService.deleteEnvironmentByCode(loginUser, environmentCode);
return Result.success();
}

/**
* query all environment list
*
* @param loginUser login user
* @return all environment list
*/
@Operation(summary = "queryAllEnvironmentList", description = "QUERY_ALL_ENVIRONMENT_LIST_NOTES")
@GetMapping(value = "/query-environment-list")
@ResponseStatus(HttpStatus.OK)
@ApiException(QUERY_ENVIRONMENT_ERROR)
public Result queryAllEnvironmentList(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) {
Map<String, Object> result = environmentService.queryAllEnvironmentList(loginUser);
return returnDataList(result);
public Result<List<EnvironmentDto>> queryAllEnvironmentList(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) {
List<EnvironmentDto> dtos = environmentService.queryAllEnvironmentList(loginUser);
return Result.success(dtos);
}

/**
* verify environment and environment name
*
* @param loginUser login user
* @param environmentName environment name
* @return true if the environment name not exists, otherwise return false
*/
@Operation(summary = "verifyEnvironment", description = "VERIFY_ENVIRONMENT_NOTES")
@Parameters({
Expand All @@ -221,9 +216,9 @@
@PostMapping(value = "/verify-environment")
@ResponseStatus(HttpStatus.OK)
@ApiException(VERIFY_ENVIRONMENT_ERROR)
public Result verifyEnvironment(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
@RequestParam(value = "environmentName") String environmentName) {
Map<String, Object> result = environmentService.verifyEnvironment(environmentName);
return returnDataList(result);
public Result<Void> verifyEnvironment(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,

Check notice

Code scanning / CodeQL

Useless parameter Note

The parameter 'loginUser' is never used.
Comment thread
ruanwenjun marked this conversation as resolved.
Dismissed
@RequestParam(value = "environmentName") String environmentName) {
environmentService.verifyEnvironment(environmentName);
return Result.success();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
package org.apache.dolphinscheduler.api.python;

import org.apache.dolphinscheduler.api.configuration.ApiConfig;
import org.apache.dolphinscheduler.api.dto.EnvironmentDto;
import org.apache.dolphinscheduler.api.dto.resources.ResourceComponent;
import org.apache.dolphinscheduler.api.dto.workflow.WorkflowTriggerRequest;
import org.apache.dolphinscheduler.api.enums.Status;
Expand Down Expand Up @@ -610,15 +609,13 @@ public Map<String, Object> getResourcesFileInfo(String fullName) {
* @param environmentName name of the environment
*/
public Long getEnvironmentInfo(String environmentName) {
Map<String, Object> result = environmentService.queryEnvironmentByName(environmentName);

if (result.get("data") == null) {
try {
return environmentService.queryEnvironmentByName(environmentName).getCode();
} catch (ServiceException e) {
String msg = String.format("Can not find valid environment by name %s", environmentName);
log.error(msg);
throw new IllegalArgumentException(msg);
}
EnvironmentDto environmentDto = EnvironmentDto.class.cast(result.get("data"));
return environmentDto.getCode();
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,12 @@

package org.apache.dolphinscheduler.api.service;

import org.apache.dolphinscheduler.api.dto.EnvironmentDto;
import org.apache.dolphinscheduler.api.utils.Result;
import org.apache.dolphinscheduler.dao.entity.Environment;
import org.apache.dolphinscheduler.dao.entity.User;

import java.util.Map;
import java.util.List;

public interface EnvironmentService {

Expand All @@ -41,22 +42,22 @@ public interface EnvironmentService {
*
* @param name environment name
*/
Map<String, Object> queryEnvironmentByName(String name);
EnvironmentDto queryEnvironmentByName(String name);

/**
* query environment
*
* @param code environment code
*/
Map<String, Object> queryEnvironmentByCode(Long code);
EnvironmentDto queryEnvironmentByCode(Long code);

/**
* delete environment
*
* @param loginUser login user
* @param code environment code
*/
Map<String, Object> deleteEnvironmentByCode(User loginUser, Long code);
void deleteEnvironmentByCode(User loginUser, Long code);

/**
* update environment
Expand Down Expand Up @@ -85,16 +86,14 @@ Environment updateEnvironmentByCode(User loginUser, Long code, String name, Stri
* query all environment
*
* @param loginUser
* @return all environment list
*/
Map<String, Object> queryAllEnvironmentList(User loginUser);
List<EnvironmentDto> queryAllEnvironmentList(User loginUser);

/**
* verify environment name
*
* @param environmentName environment name
* @return true if the environment name not exists, otherwise return false
*/
Map<String, Object> verifyEnvironment(String environmentName);
void verifyEnvironment(String environmentName);

}
Loading
Loading