Reconciling GitHub artifact_source_id vs artifact_id

[ccerv1]

In March 2025, we made an important change to standardize how we arrive at an artifact_id across different sources. Previously, in the case of GitHub data -- and only in the case of GitHub data -- we included the artifact_source_id in the hashing function. This id mapped to the node_id field in gharchive data.

Back then, the primary implication is that you would NOT be able to derive the OSO artifact_id from a GitHub URL. You would need to ping the GitHub API or do some historic lookup to get the node_id, and then derive your OSO artifact_id. As result, it was difficult to join different registries of GitHub URLs (eg, OSSD and OP Atlas or Crypto Ecosystems).

@ravenac95 asked about this again in October 2025, and here was my response in Discord:

• We decided to use our registries as our source of truth for artifact identification.
• Separately, we have models for artifact "discovery". For instance, we "discover" all artifacts that are downstream from a deployer and all packages that are linked to a repo.
• We should probably do the same thing for GitHub URLs, eg, if you query artifacts_by_project_v1 for https://github.com/opensource-observer/oso, you should also get https://github.com/hypercerts-org/oso as an artifact that we own.
• If we did this, then I would also advocate for making regular updates to OSS Directory that monitor for renaming events ...
• The business logic could be that we assume all historic aliases of a repo belong to the current repo owner unless they are specifically enumerated in other project file. For example, https://github.com/hypercerts-org/oso would belong to opensource-observer (the project) unless there was some other project that included https://github.com/hypercerts-org/oso (not just https://github.com/hypercerts-org) in its list of github urls

Yesterday, I started doing something along these lines to track repo lineage / renaming events and noticed something even stranger (see notebook here if you want to play around).

Here's a good example:

SELECT 
    time,
    event_source_id,
    to_artifact_namespace,
    to_artifact_name,
    to_artifact_source_id
FROM int_events__github
WHERE
    to_artifact_id = 'EXACMsonfj+4vw1KkFmZ8ZWWgmYRFuVN/kzClwF8hTI='
	AND event_type = 'COMMIT_CODE'
	AND time BETWEEN DATE('2022-08-01') AND DATE('2022-09-30')
ORDER BY 1

There are 10 distinct artifact_source_id for the same repo over this two month period.

If we take each artifact_source_id and its corresponding minimum and maximum timestamps, we can see that all intervals appear sequential and non-overlapping (each min_time begins after the previous max_time ends).

GitHub does NOT have different repo ids for branches of the same repo.

However, I did find a discussion about GitHub at one point (around this time) about this very issue:

it appears that there are three ID's, the Rest API ID, the GraphQL API ID and there is a new Next ID for GraphQL. This all makes perfect sense 👺 .

I'd have to do more digging to see how pervasive this issue is. In the meantime, it seems like moving to the URL-based method of deriving an OSO artifact_id was definitely the right call!

[ccerv1]

FWIW, here are GPT's thoughts on all this:

Understanding Sequential Non-Overlapping GitHub Artifact IDs

This pattern makes sense given how GitHub IDs behaved circa 2022–2023.

What’s Likely Happening

• Multiple node_id generations surfaced in the wild.
  Around that period, GitHub introduced “Next IDs” for GraphQL while REST kept its own numeric repo.id.
  GHArchive appears to have recorded events populated by different producers (REST vs GraphQL old vs GraphQL-next).
  The same repo can show up with different node_id values in contiguous blocks of time — explaining your sequential, non-overlapping intervals for one URL.

• Renames/transfers don’t explain the pattern.
  A repo keeps working across renames and org transfers; GraphQL global IDs should be stable, but in practice you’re seeing migration artifacts.
  These clean temporal boundaries align with backend rollouts rather than repo lifecycle events.

• Because OSO’s old hash included artifact_source_id (GitHub node_id), those swaps created multiple OSO artifact_ids for one URL until your March 2025 change.

---

Quick Validations to Run

-- 1) Identify ID “families” by prefix/length
SELECT
  to_artifact_source_id,
  LENGTH(to_artifact_source_id) AS len,
  SUBSTR(to_artifact_source_id,1,8) AS prefix,
  MIN(time) AS first_seen,
  MAX(time) AS last_seen,
  COUNT(*) AS n_events
FROM int_events__github
WHERE to_artifact_id = '<artifact_id>'
GROUP BY 1,2,3
ORDER BY first_seen

-- 2) Check correlation with event producers
SELECT
  DATE_TRUNC('day',time) AS d,
  ANY_VALUE(event_source_id) IGNORE NULLS,
  COUNT(*) AS n,
  APPROX_DISTINCT(to_artifact_source_id) AS n_ids
FROM int_events__github
WHERE to_artifact_namespace='<owner>' AND to_artifact_name='<repo>'
GROUP BY 1
ORDER BY 1

You’ll typically see two or more ID families (e.g., old base64 vs new R_kg...) occupying distinct time windows.

---

Recommended Remediation

1. Keep URL-based artifact_id as canonical.
   It’s human-auditable and immune to node_id churn.

2. Add an alias table with validity ranges.
   Example schema:

   repo_id_aliases(
     repo_url,
     node_id,
     valid_from,
     valid_to,
     discovered_source
   )
   

   Populate via your min/max scan so historic joins on node_id map back to canonical URLs.

3. Prefer REST numeric repo.id when available.
   Historically more stable than node_id.

4. Discovery rules for lineage.

   • Treat owner/repo case-insensitively.
   • Follow GitHub redirects.
   • Union historical aliases (e.g., hypercerts-org/oso → opensource-observer/oso) unless another project explicitly claims it.

5. Tests and monitoring.

   • Assert 1 canonical artifact per URL per day in artifacts_by_project_v1.
   • Alert if a URL shows >1 distinct node_id in a 7-day window.
   • Snapshot alias deltas to track new ID families.

---

Helper Function for Alias Ranges

def build_node_id_alias_ranges(df):
    """
    df: columns ['repo_url','to_artifact_source_id','time']
    returns: alias ranges per node_id
    """
    d = df.copy()
    d['time'] = pd.to_datetime(d['time'])
    g = (
        d.sort_values('time')
         .groupby(['repo_url','to_artifact_source_id'])
         .agg(valid_from=('time','min'),
              valid_to=('time','max'),
              n=('time','size'))
         .reset_index()
         .sort_values(['repo_url','valid_from'])
    )
    return g

---

Bottom Line

Your observation is consistent with GitHub ID migration artifacts leaking into event streams.
Switching OSO’s canonical artifact_id to a URL-derived hash was the right call.
Add a lightweight alias table (with time bounds) and prefer numeric repo.id for stable, historical reconciliation.

[ravenac95]

Discussion from 2025-10-28:

So one part of this problem is that we our ossd.repositories assets doesn't reference previously retrieved node_id values. This is part of the problem with using ossd as a source of truth. Naturally, however, ossd is temporarily true and most closely true to the time of the commit. So we should, as soon as possible essentially translate ossd references into node_id values. This would then allow for us to properly track history and hopefully then properly rely on artifact_source_id instead of something looser.