feat: configuration of project sources by victor-linroth-sensmetry · Pull Request #130 · sensmetry/sysand

victor-linroth-sensmetry · 2025-11-28T11:08:34Z

Main new features added by this PR:

Config options for project sources in sysand.toml (or given config). Allows for portable specification of local and remote sources.
Command line options --editable, --path, --url-src and --url-kpar for sysand add that automatically updates config file. These also get removed with sysand remove.
Documentation explaining the above.

In order to simplify implementing the above the core library was build out with the following changes:

New macro crate with ProjectRead and ProjectMut derive macros for easy enum composition of project structs.
New CachedProject<Local, Remote> struct for when the Local project is easier to access (e.g. on disk or in memory) vs Remote, but you still want .sources() to give the remote sources.
Implementation of CombinedResolver simplified using derive(ProjectRead) and CachedProject.
New ProjectReference<Project> struct. When working with ReadEnvironment the projects need to be clonable, but this might not always be a good idea, e.g. some project hold ownership of temp dirs. ProjectReference is basically and Arc wrapper for a project to be used as a compatibility layer.
New AnyProject struct combining the different projects supported for source configuration. Should be extended to include Git projects in future.
LocalSrcProject and LocalKparProject now has optional nominal paths in addition to the regular path. This comes from the choice to have configured source always be relative to the project root but this not always being available everywhere. (Could perhaps be simplified if we added a Global Context).

In addition there where some small changes that came about when experimenting with different solutions:

MemoryEnv can now be used with a generic project.
MemoryResolver can be conveniently constructed from iterator/array/Vec.

These ended up not being used here, but they could be useful at some later point so where left in.

Lastly, the verbose/quiet options were removed from the config. They were mostly placeholders to begin with and now that there are more real options present in the config they don't serve much of a purpose. Removing them also means that the logger can be initialized almost immediately on start, opening up the potential for logging more things, like e.g. the loading of the config files.

tilowiklundSensmetry · 2025-11-28T11:34:16Z

Is there a particular reason for expanding CombinedResolver with an Override case, as opposed to having a generic OverrideResolver<T> and use StandardResolver = OverrideResolver<CombinedResolver>?

victor-linroth-sensmetry · 2025-11-28T11:38:56Z

Is there a particular reason for expanding CombinedResolver with an Override case, as opposed to having a generic OverrideResolver<T> and use StandardResolver = OverrideResolver<CombinedResolver>?

No particular reason, just the first thing that came to mind. Do you think that would be a better solution?

tilowiklundSensmetry · 2025-11-28T12:03:55Z

Is there a particular reason for expanding CombinedResolver with an Override case, as opposed to having a generic OverrideResolver<T> and use StandardResolver = OverrideResolver<CombinedResolver>?

No particular reason, just the first thing that came to mind. Do you think that would be a better solution?

It might be easier to test/mock, and be a bit easier to reuse elsewhere. You can simply have OverrideResolver<Memory/Null> instead of having to create a CombinedResolver<Memory/Null, Memory/Null, Memory/Null, Override, Memory/Null>.

No very strong preference though, so don't bother if it's a lot of work to refactor.

tilowiklundSensmetry · 2025-11-28T12:05:29Z

Everything looks good to me in principle, would be a good to have a proper code review from @andrius-puksta-sensmetry before merging though.

victor-linroth-sensmetry · 2025-11-28T12:08:16Z

No very strong preference though, so don't bother if it's a lot of work to refactor.

Well, this was precisely the type of feedback I was looking for, so I'll definitely consider it.

Everything looks good to me in principle, would be a good to have a proper code review from @andrius-puksta-sensmetry before merging though.

It's still a draft so far. I want to add some quality of life command line stuff also.

victor-linroth-sensmetry · 2026-01-20T12:21:07Z

Currently sysand_core::lock::Source and typed_path::Utf8UnixPath have to be in scope when using the ProjectRead macro. This is certainly less than ideal but it's not obvious to me what the best solution would be.

The problem stem from the fact that reference of types in macros should be absolute but the absolute paths vary depending on if you are inside or outside the crate. E.g. it's ::sysand_core::lock::Source if outside and ::crate::lock::Source if inside. If typed_path::Utf8UnixPath is part of the public API it should probably be reexported by sysand_core and then it's the same problem there.

One could try to add them as associated types to ProjectRead but then you introduce the additional complexity that it can vary with trait implementations. I'm unsure if thats desirable or necessary.

Another solution would be to refactor the crates so that they are always used outside of the crate where they are exported. Would essentially mean that all the project stuff goes into it's own crate (except for AnyProject which is derived using the macro).

Edit: I think the appropriate course of action is to factor out the projects into their own crate, but this would be a pretty wide reaching change so best to do as a separate PR after this is merged.

victor-linroth-sensmetry · 2026-01-20T12:22:04Z

Unsure if core/tests/project_no_derive.rs has any value left now that I have it working.

victor-linroth-sensmetry · 2026-01-20T20:58:14Z

Still left to make sure Windows paths are handled properly (don't think they are at the moment).

Edit: Since the sources only have paths that are Utf8UnixPaths, I think this should be covered.

core/src/env/memory.rs

sysand/src/cli.rs

Signed-off-by: victor.linroth.sensmetry <victor.linroth@sensmetry.com>

andrius-puksta-sensmetry · 2026-02-10T16:15:06Z

core/src/project/local_src.rs

 /// re-interpreted as filesystem-native paths relative to `project_path`.
 #[derive(Clone, Debug)]
 pub struct LocalSrcProject {
+    pub nominal_path: Option<Utf8PathBuf>,


Why is this an Option?

Only needed if .sources() is called, otherwise unnecessary.

andrius-puksta-sensmetry · 2026-02-10T16:31:46Z

docs/src/commands/add.md

+- `--editable <EDITABLE>`: Path to local editable interchange project
+- `--path <PATH>`: Path to local interchange project
+- `--url-src <URL_SRC>`: URL to remote interchange project
+- `--url-kpar <URL_KPAR>`: URL to remote interchange project archive (KPAR)


Update flag names

andrius-puksta-sensmetry · 2026-02-10T16:42:21Z

sysand/src/commands/clone.rs


-    let mut local_project = LocalSrcProject { project_path };
+    let mut local_project = LocalSrcProject {
+        nominal_path: Some(project_path.clone()),


This is incorrect. project_path is absolute (line 77), and nominal_path must be relative to workspace root.

Unless nominal_path here means something different.

It's basically whatever path you want to be returned when calling .sources() on the project, so not necessarily relative to workspace root (although that might be the only usage at the moment).

I think this would be better solved if we had some form of Global Context object that could be used when calling .sources() so the caller gets some control over what type of path they want project represented by. Would make try_from_source simpler too. Such an object would require quite the bit of refactoring though so better done as a separate PR.

Also, this particular seems to be unnecessary and can just be None.

andrius-puksta-sensmetry · 2026-02-10T16:45:28Z

sysand/src/cli.rs

+    /// Path to local editable interchange project
+    #[arg(long, group = "source")]
+    pub as_editable: Option<String>,
+    /// Path to local interchange project
+    #[arg(long, group = "source")]
+    pub as_path: Option<String>,
+    /// URL to remote interchange project
+    #[arg(long, group = "source")]
+    pub as_url_src: Option<String>,
+    /// URL to remote interchange project archive (KPAR)
+    #[arg(long, group = "source")]
+    pub as_url_kpar: Option<String>,


Should mention here that these add a custom source to config file.

andrius-puksta-sensmetry · 2026-02-11T07:49:13Z

sysand/src/commands/info.rs

+                MemoryResolver::from(overrides),
+                standard_resolver(
+                    cwd,
+                    if local_env_path.is_dir() {


This is eating errors. Maybe we should add is_file() and is_dir() to wrapfs, but make them e.g. fn is_file() -> Result<bool, SomeError>? This is mostly a nice-to-have, so can be done later.

andrius-puksta-sensmetry · 2026-02-11T08:18:13Z

sysand/src/commands/env.rs

+    let mut overrides = Vec::new();
+    for config_project in &config.projects {
+        for identifier in &config_project.identifiers {
+            let mut projects = Vec::new();
+            for source in &config_project.sources {
+                projects.push(ProjectReference::new(AnyProject::try_from_source(
+                    source.clone(),
+                    project_root.clone(),
+                    auth_policy.clone(),
+                    client.clone(),
+                    runtime.clone(),
+                )?));
+            }
+            overrides.push((Iri::parse(identifier.as_str())?.into(), projects));
+        }
+    }


Move this out into a function, it's done almost identically in 4 places

andrius-puksta-sensmetry · 2026-02-11T09:14:57Z

sysand/src/commands/env.rs

+            nominal_path: Some(path.as_str().into()),
            project_path: path.as_str().into(),


nominal_path should be made relative to sysand_env parent. Since it's not currently done in most places, a comment would be fine for now.

andrius-puksta-sensmetry · 2026-02-11T09:16:33Z

sysand/src/commands/env.rs

+        let override_resolver = PriorityResolver::new(
+            MemoryResolver::from(overrides),
            MemoryResolver {
                iri_predicate: AcceptAll {},
                projects: memory_projects,
            },
+        );
+        // TODO: Move out the runtime
+        let resolver = PriorityResolver::new(
+            override_resolver,
            standard_resolver(


Ok for now, but if we add one more layer of PriorityResolver, something will have to be done.

andrius-puksta-sensmetry · 2026-02-11T09:23:15Z

sysand/src/commands/add.rs

+    Ok(src_path)
+}
+
+fn relativize(path: &Utf8Path, root: &Utf8Path) -> Utf8PathBuf {


Suggested change

fn relativize(path: &Utf8Path, root: &Utf8Path) -> Utf8PathBuf {

/// Create a relative path from `root` to `path`. Both of them must either

/// be absolute or otherwise start with the same prefix.

fn relativize(path: &Utf8Path, root: &Utf8Path) -> Utf8PathBuf {

Also maybe return Option<Utf8PathBuf> so that caller can decide what to do in case relative path cannot be constructed.

This function is useful (i.e. should be used, but currently isn't) in lots of places, so it should be moved to some utils file.

andrius-puksta-sensmetry · 2026-02-11T09:23:41Z

sysand/src/commands/add.rs

+    // If prefixes (e.g. C: vs D: on Windows) differ, no relative path is possible.
+    if path.components().next() != root.components().next() {
+        return path.to_path_buf();
+    }


Not sure what to do about Windows UNC vs non-UNC paths, as they always have different first components. Relevant here is that std's canonicalize returns UNC paths on Windows, but calling that on both inputs here would be kind of wasteful.

Windows root-relative paths are also not handled correctly here.

It may be a good idea after all to canonicalize both paths (or ensuring only canonical paths are given as args here) to avoid subtle bugs due to symlinks and Windows weirndess.

andrius-puksta-sensmetry · 2026-02-11T12:41:49Z

core/src/project/any.rs

+            Source::Editable { editable } => {
+                let nominal_path = editable.to_path_buf();
+                let project = LocalSrcProject {
+                    nominal_path: Some(nominal_path.to_string().into()),
+                    project_path: project_root.as_ref().join(nominal_path.as_str()),
+                };
+                Ok(AnyProject::Editable(
+                    EditableProject::<LocalSrcProject>::new(nominal_path.as_str().into(), project),
+                ))
+            }


Suggested change

Source::Editable { editable } => {

let nominal_path = editable.to_path_buf();

let project = LocalSrcProject {

nominal_path: Some(nominal_path.to_string().into()),

project_path: project_root.as_ref().join(nominal_path.as_str()),

};

Ok(AnyProject::Editable(

EditableProject::<LocalSrcProject>::new(nominal_path.as_str().into(), project),

))

}

Source::Editable {

editable: nominal_path,

} => {

let project = LocalSrcProject {

nominal_path: Some(nominal_path.to_string().into()),

project_path: project_root.as_ref().join(nominal_path.as_str()),

};

Ok(AnyProject::Editable(

EditableProject::<LocalSrcProject>::new(

nominal_path.into_string().into(),

project,

),

))

}

andrius-puksta-sensmetry · 2026-02-11T12:43:59Z

core/src/project/any.rs

+                .map_err(TryFromSourceError::LocalKpar)?,
+            )),
+            Source::LocalSrc { src_path } => {
+                let nominal_path = src_path.as_str().into();


Suggested change

let nominal_path = src_path.as_str().into();

let nominal_path = src_path.into_string().into();

andrius-puksta-sensmetry · 2026-02-11T12:45:29Z

core/src/project/any.rs

+            _ => Err(TryFromSourceError::UnsupportedSource(format!(
+                "{:?}",
+                source
+            ))),


Suggested change

_ => Err(TryFromSourceError::UnsupportedSource(format!(

"{:?}",

source

))),

_ => Err(TryFromSourceError::UnsupportedSource(format!("{source:?}"))),

andrius-puksta-sensmetry · 2026-02-11T13:08:06Z

core/src/resolve/combined.rs

+                    return Ok(ResolutionOutcome::Unresolvable(format!(
+                        "failed to resolve as file: {:?}",
+                        msg,
+                    )));


Suggested change

return Ok(ResolutionOutcome::Unresolvable(format!(

"failed to resolve as file: {:?}",

msg,

)));

return Ok(ResolutionOutcome::Unresolvable(format!(

"failed to resolve as file: {msg}",

)));

andrius-puksta-sensmetry · 2026-02-11T13:20:37Z