Poudriere in Highly Secure Environment with Proxy Access

[vermaden]

Hi,

I need to use Poudriere in a highly secured environment without direct Internet connection and without DNS that reaches out to the outside world.

To setup the Poudriere Jails the env(1) settings work.

# \
  env HTTP_PROXY="http://proxy.freebsd.xyz:3128/" \
      HTTPS_PROXY="https://proxy.freebsd.xyz:3128/" \
      FTP_PROXY="http://proxy.freebsd.xyz:3128/" \
      poudriere jail -c -j 14-3-R-amd64 -v 14.3-RELEASE

To fetch Ports tree git(1) option does the job - while env(1) method does not work.

# git config --system http.proxy http://proxy.freebsd.xyz:3128/

# poudriere ports -c -p default
[00:00:00] Creating default fs at /var/local/poudriere/ports/default... done
[00:00:00] Cloning the ports tree...

Now - while the above steps work - I am not able to make poudriere bulk to work.

None of the methods above work.

Bare metal FreeBSD host in the same environment needs these settings to make pkg(8) do bootstrap and work.

This PKG_ENV at the end of /usr/local/etc/pkg.conf file:

# tail -6 /usr/local/etc/pkg.conf

PKG_ENV {
        HTTP_PROXY: "http://proxy.freebsd.xyz:3128"
        HTTPS_PROXY: "https://proxy.freebsd.xyz:3128"
        FTP_PROXY: "http://proxy.freebsd.xyz:3128"
}

But that is not all.

The pkg+ prefix needs to be removed from url: and mirror_type: needs to be changed from srv to none.

After these changes pkg(8) works.

- url: "pkg+https://pkg.FreeBSD.org/${ABI}/latest",
+ url: "https://pkg.FreeBSD.org/${ABI}/latest",
- mirror_type: "srv",
+ mirror_type: "none",

Now - my questions:

Which part of Poudriere I need to modify that:

• I will have this custom pkg.conf inside FreeBSD Jail before the build bulk process.
• I will have git(1) option http.proxy defined as http://proxy.freebsd.xyz:3128 value.

I assume it should be put somewhere just after Jail start and before bulk process of building packages.

This is how it fails 'live':

Thanks for help.

Regards,
vermaden

[michael-o]

This always worked for me:

$ cat /usr/local/etc/poudriere.conf | grep export
export HTTP_PROXY=http://de.coia.siemens.net:9400
export HTTPS_PROXY=http://de.coia.siemens.net:9400
export FTP_PROXY=http://de.coia.siemens.net:9400
export NO_PROXY="localhost,.siemens.net,.siemens.com,.siemens.de,.siemens.cloud,.siemens.io,.innomotics.net,.innomotics.com"

[vermaden]

Thank You - that solves the PROXY part (http://proxy.freebsd.xyz:3128) but the remaining pkg(8) config remains open - and it fails the same way at fetch phase.

So I still need to put custom pkg(8) config into the Jail.

[michael-o]

Interesting. Can you access the environment of pkg? I am curious where it comes from...

[vermaden]

Can You ask the question in different words? What do You mean by accessing the pkg(8) environment?

[michael-o]

Can You ask the question in different words? What do You mean by accessing the pkg(8) environment?

Yes, sure. When pkg(8) is run, is it on the jailhost or the jail? What is the env? Maybe patch poudriere and run env before pkg(8)? This helped me to understand hooks better.

[vermaden]

Its for pkg(8) in a Poudriere Jail.

The pkg(8) on a host already has working pkg(8) config with proxy as in the 1st post here.

[michael-o]

Its for pkg(8) in a Poudriere Jail.

Can you check the env there?

The pkg(8) on a host already has working pkg(8) config with proxy as in the 1st post here.

The env from poudriere.conf isn't passed along? It is tedious to provide this over and over again.

What you should do is what I do with login.conf:

# cat ./base/files/patch-etc_login.conf
--- /etc/login.conf     2024-07-02 20:06:10.102418000 +0200
+++ /etc/login.conf     2024-07-02 20:08:04.774989000 +0200
@@ -25,9 +25,13 @@
        :passwd_format=sha512:\
        :copyright=/etc/COPYRIGHT:\
        :welcome=/var/run/motd:\
-       :setenv=BLOCKSIZE=K:\
+       :setenv=BLOCKSIZE=K,LSCOLORS=ExGxFxdxCxDxDxhbadExEx,CLICOLOR=YES,LESS=-x4 -RFK,\
+                       HTTP_PROXY=http\c//de.coia.siemens.net\c9400,\
+                       HTTPS_PROXY=http\c//de.coia.siemens.net\c9400,\
+                       FTP_PROXY=http\c//de.coia.siemens.net\c9400,\
+                       NO_PROXY="localhost,.siemens.net,.siemens.com,.siemens.de,.siemens.cloud,.siemens.io,.innomotics.net,.innomotics.com":\
        :mail=/var/mail/$:\
-       :path=/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin ~/bin:\
+       :path=/sbin /bin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin ~/bin:\
        :nologin=/var/run/nologin:\
        :cputime=unlimited:\
        :datasize=unlimited:\

Since pkg uses curl, I expect it to work which in fact did for the past 15+ years.

[vermaden]

Can you check the env there?

How?

What you should do is what I do with login.conf:

I added my proxy servers to the setenv variable at /etc/login.conf file on the host but nothing changed - same error.

[michael-o]

Can you check the env there?

How?

What you should do is what I do with login.conf:

I added my proxy servers to the setenv variable at /etc/login.conf file on the host but nothing changed - same error.

Let me double check and get back to you!

[michael-o]

Something isn't right with pkg's debug levels, so I had to apply a local patch to show it to you:

# git diff
diff --git a/libpkg/fetch_libcurl.c b/libpkg/fetch_libcurl.c
index 690e344e2..84d9fac3e 100644
--- a/libpkg/fetch_libcurl.c
+++ b/libpkg/fetch_libcurl.c
@@ -153,9 +153,7 @@ curl_do_fetch(struct curl_userdata *data, CURL *cl, struct curl_repodata *cr)
    curl_easy_setopt(cl, CURLOPT_PRIVATE, &data);
    if (data->offset > 0)
        curl_easy_setopt(cl, CURLOPT_RESUME_FROM_LARGE, data->offset);
-   if (ctx.debug_flags & PKG_DBG_FETCH && ctx.debug_level >= 1)
        curl_easy_setopt(cl, CURLOPT_VERBOSE, 1L);
-   if (ctx.debug_flags & PKG_DBG_FETCH && ctx.debug_level >= 1)
        curl_easy_setopt(cl, CURLOPT_DEBUGFUNCTION, my_trace);

    /* compat with libfetch */

See:

root@deblndw011x:/tmp/pkg (main *=)
# env | grep PROXY
NO_PROXY=localhost,.siemens.net,.siemens.com,.siemens.de,.siemens.cloud,.siemens.io,.innomotics.net,.innomotics.com
FTP_PROXY=http://de.coia.siemens.net:9400
HTTPS_PROXY=http://de.coia.siemens.net:9400
HTTP_PROXY=http://de.coia.siemens.net:9400
root@deblndw011x:/tmp/pkg (main *=)
#   ./src/pkg update | less
== Info: Uses proxy env variable NO_PROXY == 'localhost,.siemens.net,.siemens.com,.siemens.de,.siemens.cloud,.siemens.io,.innomotics.net,.innomotics.com'
== Info: Couldn't find host dw-eng-rsc.innomotics.net in the .netrc file; using defaults
== Info: Host dw-eng-rsc.innomotics.net:443 was resolved.
== Info: IPv6: (none)
== Info: IPv4: 10.64.105.147
== Info:   Trying 10.64.105.147:443...
== Info: ALPN: curl offers http/1.1
=> Send SSL data, 0000000005 bytes (0x00000005)
0000: .....
== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
=> Send SSL data, 0000000512 bytes (0x00000200)

Now let's remove NO_PROXY and it will use the proxy, the communication is expected to fail:

root@deblndw011x:/tmp/pkg (main *=)
# NO_PROXY=""  ./src/pkg update
Updating common repository catalogue...
== Info: Couldn't find host dw-eng-rsc.innomotics.net in the .netrc file; using defaults
== Info: Host de.coia.siemens.net:9400 was resolved.
== Info: IPv6: (none)
== Info: IPv4: 185.46.212.91, 185.46.212.98
== Info:   Trying 185.46.212.91:9400...
== Info: CONNECT: no ALPN negotiated
== Info: allocate connect buffer
== Info: Establish HTTP proxy tunnel to dw-eng-rsc.innomotics.net:443
=> Send header, 0000000141 bytes (0x0000008d)
0000: CONNECT dw-eng-rsc.innomotics.net:443 HTTP/1.1
0030: Host: dw-eng-rsc.innomotics.net:443
0055: User-Agent: pkg/2.4.99
006d: Proxy-Connection: Keep-Alive
008b:
<= Recv header, 0000000026 bytes (0x0000001a)
0000: HTTP/1.0 502 Bad Gateway
<= Recv header, 0000000021 bytes (0x00000015)
0000: Server: Zscaler/6.2
<= Recv header, 0000000025 bytes (0x00000019)
0000: Content-Type: text/html
<= Recv header, 0000000019 bytes (0x00000013)
0000: Connection: close
<= Recv header, 0000000002 bytes (0x00000002)
0000:
== Info: CONNECT tunnel failed, response 502
== Info: closing connection #0
== Info: Couldn't find host dw-eng-rsc.innomotics.net in the .netrc file; using defaults
== Info: Hostname de.coia.siemens.net was found in DNS cache
== Info:   Trying 185.46.212.91:9400...
== Info: CONNECT: no ALPN negotiated
== Info: allocate connect buffer
== Info: Establish HTTP proxy tunnel to dw-eng-rsc.innomotics.net:443
=> Send header, 0000000141 bytes (0x0000008d)
0000: CONNECT dw-eng-rsc.innomotics.net:443 HTTP/1.1

So it does work.

[michael-o]

@vermaden are you able to replicate?

[vermaden]

Its not that easy.

The environment I try to do these things are really strict.

For example its not possible to pkg bootstrap online - it needs to be done in offline mode.

root@15base:~ # pkg update
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64/latest, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:15:amd64/latest/Latest/pkg.pkg: Transient resolver failure
A pre-built version of pkg could not be found for your system.
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64/kmods_latest, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:15:amd64/kmods_latest/Latest/pkg.pkg: Transient resolver failure
A pre-built version of pkg could not be found for your system.
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64/base_latest, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:15:amd64/base_latest/Latest/pkg.pkg: Transient resolver failure
A pre-built version of pkg could not be found for your system.

... and proxy ENV variables are invisible for pkg(8) command.

root@15base:~ # export HTTP_PROXY="http://proxy.freebsd.xyz:3128"
root@15base:~ # export HTTPS_PROXY="https://proxy.freebsd.xyz:3128"
root@15base:~ # export FTP_PROXY="http://proxy.freebsd.xyz:3128"
root@15base:~ # pkg update
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64/latest, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:15:amd64/latest/Latest/pkg.pkg: Transient resolver failure
A pre-built version of pkg could not be found for your system.
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64/kmods_latest, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:15:amd64/kmods_latest/Latest/pkg.pkg: Transient resolver failure
A pre-built version of pkg could not be found for your system.
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64/base_latest, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:15:amd64/base_latest/Latest/pkg.pkg: Transient resolver failure
A pre-built version of pkg could not be found for your system.

I will get back with more details after I upload new 15.0-BETA4 image there.

[michael-o]

All of this does not look right. Only the proxy should resolve the target host. I recommend to fire up truss and so whether it talks to the proxy. I am afraid that it does not.