Unable to replicate keyless `cosign sign ...` using basic Env

[tarilabs]

Description

I'm trying to do the python analogous of:

        echo $TUF_URL
        echo $FULCIO_URL
        cosign initialize --mirror=$TUF_URL --root=$TUF_URL/root.json
        cosign sign -y --fulcio-url=$FULCIO_URL --identity-token=$TOKEN <SOMETHING TO SIGN>

programmatically using sigstore (python) but I'm not able to understand how to Initalize/TUF and how to use the Token I have "on hand" which I typically pass as $TOKEN above.

I've followed the documentation, and I'm currently stuck at trying to find the equivalent of initializing.

If I try:

from pathlib import Path
from sigstore.models import ClientTrustConfig
from sigstore.oidc import Issuer
from sigstore.sign import SigningContext

artifact_path = Path("README.txt")

trust_config = ClientTrustConfig.from_tuf(os.environ["TUF_URL"])

I get:

FileNotFoundError: [Errno 2] No such file or directory: '/opt/app-root/src/.local/share/sigstore-python/tuf/<TUF_URL VALUE>/root.json'

The above exception was the direct cause of the following exception:

TUFError                                  Traceback (most recent call last)
Cell In[12], line 8
      4 from sigstore.sign import SigningContext
      6 artifact_path = Path("README.txt")
----> 8 trust_config = ClientTrustConfig.from_tuf("asd")

File [/opt/app-root/lib64/python3.12/site-packages/sigstore/models.py:963](https://....openshiftapps.com/opt/app-root/lib64/python3.12/site-packages/sigstore/models.py#line=962), in ClientTrustConfig.from_tuf(cls, url, offline, bootstrap_root)
    951 @classmethod
    952 def from_tuf(
    953     cls,
   (...)    956     bootstrap_root: Path | None = None,
    957 ) -> ClientTrustConfig:
    958     """Create a new trust config from a TUF repository.
    959 
    960     If `offline`, will use data in local TUF cache. Otherwise will
    961     update the trust config from remote TUF repository.
    962     """
--> 963     updater = TrustUpdater(url, offline, bootstrap_root)
    965     tr_path = updater.get_trusted_root_path()
    966     inner_tr = trustroot_v1.TrustedRoot.from_json(Path(tr_path).read_bytes())

File [/opt/app-root/lib64/python3.12/site-packages/sigstore/_internal/tuf.py:129](https://....openshiftapps.com/opt/app-root/lib64/python3.12/site-packages/sigstore/_internal/tuf.py#line=128), in TrustUpdater.__init__(self, url, offline, bootstrap_root)
    127     self._updater.refresh()
    128 except Exception as e:
--> 129     raise TUFError("Failed to refresh TUF metadata") from e

TUFError: Failed to refresh TUF metadata

So, I assume I need to download from the TUF server the root.json and place it in that location?
I do so with:

import os, urllib.parse, requests, pathlib

tuf_url = os.environ["TUF_URL"]
encoded = urllib.parse.quote(tuf_url, safe='')
dest = pathlib.Path(f"/opt/app-root/src/.local/share/sigstore-python/tuf/{encoded}")
dest.mkdir(parents=True, exist_ok=True)

r = requests.get(f"{tuf_url}/root.json")
r.raise_for_status()

(dest / "root.json").write_bytes(r.content)
print(f"Saved to {dest / 'root.json'}")

Then, I try again the above

from pathlib import Path
from sigstore.models import ClientTrustConfig
from sigstore.oidc import Issuer
from sigstore.sign import SigningContext

artifact_path = Path("README.txt")

trust_config = ClientTrustConfig.from_tuf(os.environ["TUF_URL"])

But this time I get:

ValidationError                           Traceback (most recent call last)
Cell In[11], line 8
      4 from sigstore.sign import SigningContext
      6 artifact_path = Path("README.txt")
----> 8 trust_config = ClientTrustConfig.from_tuf(os.environ["TUF_URL"])

File [/opt/app-root/lib64/python3.12/site-packages/sigstore/models.py:966](https://....openshiftapps.com/opt/app-root/lib64/python3.12/site-packages/sigstore/models.py#line=965), in ClientTrustConfig.from_tuf(cls, url, offline, bootstrap_root)
    963 updater = TrustUpdater(url, offline, bootstrap_root)
    965 tr_path = updater.get_trusted_root_path()
--> 966 inner_tr = trustroot_v1.TrustedRoot.from_json(Path(tr_path).read_bytes())
    968 try:
    969     sc_path = updater.get_signing_config_path()

File [/opt/app-root/lib64/python3.12/site-packages/sigstore_models/_core.py:22](https://....openshiftapps.com/opt/app-root/lib64/python3.12/site-packages/sigstore_models/_core.py#line=21), in Base.from_json(cls, json)
     20 @classmethod
     21 def from_json(cls, json: str | bytes) -> te.Self:
---> 22     return cls.model_validate_json(json)

File [/opt/app-root/lib64/python3.12/site-packages/pydantic/main.py:766](https://....openshiftapps.com/opt/app-root/lib64/python3.12/site-packages/pydantic/main.py#line=765), in BaseModel.model_validate_json(cls, json_data, strict, extra, context, by_alias, by_name)
    760 if by_alias is False and by_name is not True:
    761     raise PydanticUserError(
    762         'At least one of `by_alias` or `by_name` must be set to True.',
    763         code='validate-by-alias-and-name-false',
    764     )
--> 766 return cls.__pydantic_validator__.validate_json(
    767     json_data, strict=strict, extra=extra, context=context, by_alias=by_alias, by_name=by_name
    768 )

ValidationError: 2 validation errors for TrustedRoot
tlogs.0.logId
  Field required [type=missing, input_value={'baseUrl': 'https://reko...jdepp+ybrrskst3yax5c=/'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
ctlogs.0.logId
  Field required [type=missing, input_value={'baseUrl': 'https://ctlo...oof955zem9hfxi8ejbz4=/'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing

How do I setup the root of trust?
How do I pass the Fulcio URL and my Token when signing?

When I'm trying this overall keyless sign flow inside a K8s Pod with cosign (binary) it works with the top above snippet without any issue, keyless signing using the SA Token. My Fulcio instance seems to connects to the setup OIDC instance and the artifact is signed with the expected identity.

Thanks in advance!

Version

Collecting sigstore
  Downloading sigstore-4.1.0-py3-none-any.whl.metadata (16 kB)

while using tutorial for cosign

cosign: 2.5.3

[woodruffw]

Hi @tarilabs, thanks for the report.

TUFError: Failed to refresh TUF metadata

This indicates that the TUF repository URL you've passed in isn't valid -- can you confirm that that URL actually points to a TUF repository?

For context, sigstore-python wraps tuf.ngclient.Updater for TUF metadata fetches, and the error you're seeing happens here:

sigstore-python/sigstore/_internal/tuf.py

Lines 127 to 129 in 55f1d18

	self._updater.refresh()
	except Exception as e:
	raise TUFError("Failed to refresh TUF metadata") from e

which corresponds to this part of python-tuf:

https://github.com/theupdateframework/python-tuf/blob/f54248c61a92cd84627facf1b530f3b12fefbd38/tuf/ngclient/updater.py#L145-L172

We could probably improve the quality of the error messages there, but the error itself strongly suggests that the URL you gave doesn't have one or more necessary TUF metadata files (e.g. root.json).

[woodruffw]

(I see you have root.json based on your manual example, so maybe it's another one of the files. The validation errors looks suspicious, but those might be secondary -- we only validate the actual TUF responses later on, so the update failure suggests a more basic incompatibility between your TUF repo and what sigstore-python expects.)

[tarilabs]

can you confirm that that URL actually points to a TUF repository?

yes,

TUF_URL=...
curl -s $TUF_URL/root.json | jq .

I get:

{
  "signed": {
    "_type": "root",
    "spec_version": "1.0.0",
    "consistent_snapshot": true,
    "version": 1,
    "expires": "2026-10-21T09:43:58Z",
    "keys": ...,
    "roles": ...
  },
  "signatures": [
...
  ]
}

But I'm a bit confused now

the error itself strongly suggests that the URL you gave doesn't have one or more necessary TUF metadata files (e.g. root.json)

since

trust_config = ClientTrustConfig.from_tuf(os.environ["TUF_URL"])

complaints about a file missing /opt/app-root/src/.local/share/sigstore-python/tuf/<TUF_URL VALUE>/root.json, not about unable to reach the TUF repository? 🤔

[woodruffw]

complaints about a file missing /opt/app-root/src/.local/share/sigstore-python/tuf/<TUF_URL VALUE>/root.json, not about unable to reach the TUF repository? 🤔

Yeah, I think the error stack is unfortunately very confusing here: what it's saying (I think) is that it failed to refresh the metadata remotely, so we tried to read <TUF_URL_VALUE>/root.json locally as a fallback, which then failed (because we have no cached version).

[tarilabs]

thanks again for the feedback so far, here's the complete stack of the error:

from pathlib import Path
from sigstore.models import ClientTrustConfig
from sigstore.oidc import Issuer
from sigstore.sign import SigningContext

artifact_path = Path("README.txt")

trust_config = ClientTrustConfig.from_tuf(os.environ["TUF_URL"])

results in:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
File /opt/app-root/lib64/python3.12/site-packages/sigstore/_internal/tuf.py:117, in TrustUpdater.__init__(self, url, offline, bootstrap_root)
    116 try:
--> 117     self._updater = Updater(
    118         metadata_dir=str(self._metadata_dir),
    119         metadata_base_url=url,
    120         target_base_url=parse.urljoin(f"{url}/", "targets/"),
    121         target_dir=str(self._targets_dir),
    122         config=UpdaterConfig(
    123             app_user_agent=f"sigstore-python/{__version__}"
    124         ),
    125         bootstrap=root_json,
    126     )
    127     self._updater.refresh()

File /opt/app-root/lib64/python3.12/site-packages/tuf/ngclient/updater.py:136, in Updater.__init__(self, metadata_dir, metadata_base_url, target_dir, target_base_url, fetcher, config, bootstrap)
    134 if not bootstrap:
    135     # if no root was provided, use the cached non-versioned root.json
--> 136     bootstrap = self._load_local_metadata(Root.type)
    138 # Load the initial root, make sure it's cached

File /opt/app-root/lib64/python3.12/site-packages/tuf/ngclient/updater.py:317, in Updater._load_local_metadata(self, rolename)
    316 encoded_name = parse.quote(rolename, "")
--> 317 with open(os.path.join(self._dir, f"{encoded_name}.json"), "rb") as f:
    318     return f.read()

FileNotFoundError: [Errno 2] No such file or directory: '/opt/app-root/src/.local/share/sigstore-python/tuf/<TUF_URL value encoded here>/root.json'

The above exception was the direct cause of the following exception:

TUFError                                  Traceback (most recent call last)
Cell In[16], line 8
      4 from sigstore.sign import SigningContext
      6 artifact_path = Path("README.txt")
----> 8 trust_config = ClientTrustConfig.from_tuf(os.environ["TUF_URL"])

File /opt/app-root/lib64/python3.12/site-packages/sigstore/models.py:963, in ClientTrustConfig.from_tuf(cls, url, offline, bootstrap_root)
    951 @classmethod
    952 def from_tuf(
    953     cls,
   (...)    956     bootstrap_root: Path | None = None,
    957 ) -> ClientTrustConfig:
    958     """Create a new trust config from a TUF repository.
    959 
    960     If `offline`, will use data in local TUF cache. Otherwise will
    961     update the trust config from remote TUF repository.
    962     """
--> 963     updater = TrustUpdater(url, offline, bootstrap_root)
    965     tr_path = updater.get_trusted_root_path()
    966     inner_tr = trustroot_v1.TrustedRoot.from_json(Path(tr_path).read_bytes())

File /opt/app-root/lib64/python3.12/site-packages/sigstore/_internal/tuf.py:129, in TrustUpdater.__init__(self, url, offline, bootstrap_root)
    127     self._updater.refresh()
    128 except Exception as e:
--> 129     raise TUFError("Failed to refresh TUF metadata") from e

TUFError: Failed to refresh TUF metadata

[jku]

trust_config = ClientTrustConfig.from_tuf(os.environ["TUF_URL"])

this will only work for tuf instances where the root.json is embedded in the sigstore-python sources (meaning public good sigstore.dev), or the instance has been initialized already. You'll need this to use your own TUF repo (note that this only works with the newest version but you seem to be on it already):

ClientTrustConfig.from_tuf(url, path_to_initial_root)`

you only need to provide the initial root once as it is cached in the local TUF cache, but it is not a problem to always provide it (in fact it is more secure to do so in some situations).

[jku]

Your workaround should have worked though I think... so it does imply that sigstore-python is not happy with the contents of the TUF repository (trusted_root.json):

ValidationError: 2 validation errors for TrustedRoot
tlogs.0.logId
  Field required [type=missing, input_value={'baseUrl': 'https://reko...jdepp+ybrrskst3yax5c=/'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
ctlogs.0.logId
  Field required [type=missing, input_value={'baseUrl': 'https://ctlo...oof955zem9hfxi8ejbz4=/'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing

[tarilabs]

it does imply that sigstore-python is not happy with the contents of the TUF repository (trusted_root.json):

the $TUF_URL/root.json contains the json per this previous #1609 (comment)

how can I debu*g this? 🤔 😕

[jku]

In your workaround example (where you copied the TUF root.json in place) it looks like the TUF update has completed successfully so the TUF root.json is likely fine.

But TUF is really just the delivery mechanism for sigstores configuration files trusted_root.json and signing_config.v0.2.json and it seems the first of those is not what sigstore-python expects.

[jku]

Also, maybe you are aware but I'll mention that you don't need to code to replicate the cosign example:

cosign:

cosign initialize --mirror=$TUF_URL --root=$TUF_URL/root.json
cosign sign -y --fulcio-url=$FULCIO_URL --identity-token=$TOKEN <SOMETHING TO SIGN>

sigstore-python 4.1 should work with something like this:

sigstore --instance $TUF_URL trust-instance $LOCAL_PATH_TO_ROOT_JSON
sigstore --instance $TUF_URL sign --identity-token=$TOKEN <SOMETHING TO SIGN>

(but the trusted_root.json error is likely still there)

I've filed #1610 to document how to use the CLI with custom instances

[tarilabs]

maybe you are aware but I'll mention that you don't need to code to replicate the cosign example

thanks @jku I'm very new, so kindly let me ask a basic question. I run this python code in an "ephemeral" pod in k8s, so I need to "init" to my understanding before I can proceed to sign.

using cosign, I do:

cosign initialize --mirror=$TUF_URL --root=$TUF_URL/root.json

with

ClientTrustConfig.from_tuf(url, path_to_initial_root)

I get I should pass the value of $TUF_URL into the url parameter, but how do I create the path_to_initial_root, please? Any documentation you could point me to, please?

[jku]

using cosign, I do:

cosign initialize --mirror=$TUF_URL --root=$TUF_URL/root.json

with

ClientTrustConfig.from_tuf(url, path_to_initial_root)
I get I should pass the value of $TUF_URL into the url parameter, but how do I create the path_to_initial_root, please? Any documentation you could point me to, please?

I think I overlooked that the variable should be named (bootstrap_root=path_to_initial_root) so maybe that's what did not work...

Anyway, sigstore-python expects you to have the initial root as a local file, examples of doing this:

command line example

TUF_URL=https://sigstore.github.io/root-signing-staging

# trust the instance with trust-on-first-use 
curl -o root.json $TUF_URL/12.root.json
sigstore --instance $TUF_URL trust-instance root.json

# sign using the instance
sigstore --instance $TUF_URL sign README.md

code example (just the clienttrustconfig part)

If you want to do the same in code you'd need to do the curl equivalent with e.g. urllib.request:

import urllib.request
from pathlib import Path
from sigstore.models import ClientTrustConfig

url = "https://sigstore.github.io/root-signing-staging"
urllib.request.urlretrieve(f"{url}/12.root.json", "./root.json")
config = ClientTrustConfig.from_tuf(url, bootstrap_root=Path("./root.json"))

Hope that helps

[tarilabs]

I am using:

seems to me downloading exactly the same per previous #1609 (comment)

resulting in:

ps, not sure I got why 12. in the 12.root.json 🤔

[jku]

Yes, as I mentioned it looks like the TUF part works fine but the payload (sigstore configuration file trusted_root.json) seems to be something sigstore-python does not understand.

A valid trusted_root.json looks like https://github.com/sigstore/root-signing-staging/blob/main/targets/trusted_root.json Would you like to share the trusted_root.json you are using?

not sure I got why 12. in the 12.root.json

any version of root.json should work: some TUF repositories provide a non-standard "unversioned" URL that points to the last root.json version but the repository I used in the example does not do that so I chose the last version.