Unable to override hexpm repo_url with local/private Hex repo (Artifactory); api_url honored but repo_url stays https://repo.hex.pm

[kolomnik]

Hello team,

Context

• We must forbid direct fetches from hex.pm and use an internal JFrog Artifactory Hex repository instead.
• Following https://rebar3.org/docs/configuration/configuration/#hex-repos-and-indexes, we configured a local repo in ~/.config/rebar3/rebar.config and rely on the rebar3_hex plugin.

Configuration

~/.config/rebar3/rebar.config (sanitized):

{plugins, [
    %% pulled from an internal mirror
    {rebar3_hex, {git, "<internal_mirror>", "v7.0.11"}}
]}.

{hex, [
   {repos, [
        #{name => <<"hex-local">>,
          api_url =>  <<"https://somehost/artifactory/api/hex/hex-local">>,
          repo_url => <<"https://somehost/artifactory/hex-local">>,
          repo_public_key =>
            <<"-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----\n">>,
          api_key => <<"Bearer cmVm...">>,
          repo_key => <<"Bearer cmVm....">>
        }
    ]},
   {repos, replace, [
      #{name => <<"hexpm">>,
        api_url =>  <<"https://somehost/artifactory/api/hex/hex-local">>,
        repo_url => <<"https://somehost/artifactory/hex-local">>,
        repo_public_key =>
          <<"-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----\n">>,
        api_key => <<"Bearer cmVm...">>,
        repo_key => <<"Bearer cmVm....">>
       }
   ]}
]}.

Reproduction steps

1. Minimal project with a public dependency:

%% rebar.config
{deps, [verl]}.

2. Run with diagnostics:

DIAGNOSTIC=1 rebar3 deps

Observed behavior

In the diagnostic output, api_url is correctly replaced with the Artifactory endpoint, but repo_url remains set to https://repo.hex.pm:

===> Making request to get package verl from repo hexpm (#{name => <<"hexpm">>,
         api_url => <<"https://somehost/artifactory/api/hex/hex-local">>,
         repo_url => <<"https://repo.hex.pm">>, repo_verify_origin => true,
         repo_name => <<"hexpm">>, repo_organization => undefined,
         repo_verify => true})

Expected behavior

• Both api_url and repo_url for the hexpm repository should be overridden so that index queries and tarball downloads come from Artifactory.
• The chosen repository should be reflected in rebar.lock so the entire team resolves consistently.

Questions

1. Is repo_url override supported via {hex, [{repos, replace, [...]}]}? If so, is there a different key that controls the tarball download host?
2. Is there precedence or caching that could keep repo_url at its default (for example rebar.lock, ~/.cache/rebar3/hex, or registry metadata)? What is the proper way to clear or refresh it?
3. How should ~/.hex/hex.config auth_key map into rebar3 configuration? Are api_key and repo_key the correct fields for Artifactory, and should the values be in Bearer <token> form?
4. Are there canonical examples of configuring rebar3 with a self-hosted Hex repository (Artifactory or mini_repo) that fully override hexpm?

Environment

• rebar3: 3.24.0
• rebar3_hex: v7.0.11 (from internal mirror)
• hex_core: 0.10.3
• Erlang/OTP: 27.3.3
• OS: Ubuntu

Additional notes and attempts

• Removing hexpm from the repo list and leaving only the local repo still showed repo_url => https://repo.hex.pm in diagnostics when the repo name is hexpm.
• Clearing ~/.cache/rebar3 and removing rebar.lock did not change the behavior.
• Early attempts may have mixed endpoints; the example above consistently points to the same Artifactory repo for both api_url and repo_url.

Thanks for your help. If this belongs under rebar3_hex or hex_core instead of rebar3, please let me know and I can refile.

[ferd]

the {repos, replace, List} key should be the only one you specify, which will 'replace' the default config with the list you provide. If you submit {repos, List}, these get prepended to the default value.

The documentation is probably unclear but the two options are mutually exclusive, and 'replace' needs to be first:

rebar3/apps/rebar/src/rebar_hex_repos.erl

Lines 82 to 96 in b62659b

	%% A user's list of repos are merged by name while keeping the order
	%% intact. The order is based on the first use of a repo by name in the
	%% list. The default repo is appended to the user's list.
	repos(HexConfig) ->
	HexDefaultConfig = default_repo(),
	case [R || R <- HexConfig, element(1, R) =:= repos] of
	[] ->
	[HexDefaultConfig];
	%% we only care if the first element is a replace entry
	[{repos, replace, Repos} | _]->
	merge_repos(Repos);
	Repos ->
	RepoList = repo_list(Repos),
	merge_repos(RepoList ++ [HexDefaultConfig])
	end.

[kolomnik]

Thanks for your reply. I have removed {repos, List}, but the issue with repo_url remains:

$ DIAGNOSTIC=1 rebar3 deps
===> Load global config file /home/user/.config/rebar3/rebar.config
===> Expanded command sequence to be run: [app_discovery,install_deps,deps]
===> Running provider: app_discovery
===> Found top-level apps: [test_deps]
        using config: [{src_dirs,["src"]},{lib_dirs,["apps/*","lib/*","."]}]
===> Running provider: install_deps
===> Verifying dependencies...
===> Fetching verl v1.1.1
===> Making request to get package verl from repo hexpm (#{name => <<"hexpm">>,
         api_url =>
             <<"https://somehost/artifactory/api/hex/ccs-hex-local">>,
         repo_url => <<"https://repo.hex.pm">>,repo_name => <<"hexpm">>})
^C
$
$ cat ~/.config/rebar3/rebar.config
{hex, [
   {repos, replace, [
      #{name => <<"hexpm">>,
        api_url =>  <<"https://somehost/artifactory/api/hex/hex-local">>,
        repo_url =>  <<"https://somehost/artifactory/hex-local">>,
        repo_public_key =>
          <<"-----BEGIN PUBLIC KEY-----
...-----END PUBLIC KEY-----\n">>,
        api_key => <<"Bearer cmVmd...">>,
        repo_key => <<"Bearer cmVmd...">>
       }
   ]}
]}.

Would be also great to get some information about all hex repo map fields and what they are for. Especially how are they related to ~/.hex/hex.config:

{'$repos',#{<<"hex-local">> =>
                #{auth_key =>
                      <<"Bearer cmVmd...">>,
                  public_key =>
                      <<"-----BEGIN PUBLIC KEY-----...-----END PUBLIC KEY-----\n">>,
                  url =>
                      <<"https://somehost/artifactory/api/hex/hex-local">>,
                  fetch_public_key => nil},
            <<"hexpm">> => #{url => <<"https://repo.hex.pm">>}}}.

[kolomnik]

Can it be that:

rebar3/apps/rebar/src/rebar_hex_repos.erl

Lines 142 to 143 in b62659b

	update_repo_list(R=#{name := N}, [H=#{name := HN} | Rest]) when N =:= HN ->
	[maps:merge(R, H) | Rest];

Maps merge parameters order is incorrect; it appears as if default repo overwrite replacement.

[kolomnik]

Hello @ferd,
I believe I've identified the actual cause, which differs from my initial assumption. The issue lies in a function that iterates over all defined repos:

rebar3/apps/rebar/src/rebar_hex_repos.erl

Lines 133 to 140 in b62659b

	maybe_override_default_repo_url(Repos, State) ->
	lists:map(fun(Repo=#{repo_name := ?PUBLIC_HEX_REPO}) ->
	Repo#{repo_url => rebar_state:default_hex_repo_url_override(State)};
	(Repo=#{name := ?PUBLIC_HEX_REPO}) ->
	Repo#{repo_url => rebar_state:default_hex_repo_url_override(State)};
	(Repo) ->
	Repo
	end, Repos).

This function always overrides repo_url — regardless of whether it is already defined or what value it has — with either the default Hex.pm DEFAULT_CDN or the value of the HEX_CDN environment variable. This is problematic when overriding the default Hex.pm repo, since it forces me to also set repo_public_key, api_url, api_key, and repo_key in the Hex repo configuration, and also define the HEX_CDN environment variable externally.

Wouldn't it be more appropriate to only perform this override if HEX_CDN is defined and different from the value in the Hex.pm repo? I suspect there may be a deliberate reasoning behind the current behavior, but it’s unclear to me at the moment.

Thanks !

[ferd]

That's a good candidate yeah. Frankly I'm not quite sure what the rationale was anymore for this one. Maybe @tsloughter or @starbelly would know, but absent any other insight it might be worth assuming this always was a bug (or a deprecated behavior when hex changed some configs in the past) and just making it work for now.

[tsloughter]

The idea may have been that you always must use the official hex repo for any hex repo named ?PUBLIC_HEX_REPO. maybe that is a wrong assumption, I don't recall the actual rational.