Make sshd logs available

[kailash-everlaw]

Image I'm using: v0.12.3

Issue or Feature Request:

Right now, I don’t think logs from the admin container's sshd are available anywhere. This means that if someone SSHs to a Bottlerocket node through the admin container there is no audit trail. It would be nice to have the logs available somewhere so that we have a record when someone SSHs to a Bottlerocket node. Ideally, the logs would be collectible using a logging agent too so they can be stored in central logging system.

I found bottlerocket-os/bottlerocket#1583 which is similar but I think a little different, since it seems more related to debugging Bottlerocket issues without using the admin container, while my use case is more focused on viewing logs from services running inside the admin container.

Workaround
One hacky workaround for the current behavior that I have tried out is updating the admin container start script to run the following commands before launched systemd:

/usr/lib/systemd/systemd-journald &
while [ ! -d "/run/log/journal" ];
do
    sleep 1
done
journalctl -f &

Launching systemd-journald makes the logs viewable using journalctl inside the admin container:

[root@admin]# journalctl -u sshd                                                                                                                                                      
-- Logs begin at Wed 2025-11-05 05:16:10 UTC, end at Wed 2025-11-05 05:17:59 UTC. --
Nov 05 05:16:11 ip-10-104-73-161.ec2.internal sshd[3244]: Server listening on 0.0.0.0 port 22.
Nov 05 05:16:11 ip-10-104-73-161.ec2.internal sshd[3244]: Server listening on :: port 22.
Nov 05 05:17:45 ip-10-104-73-161.ec2.internal sshd[3244]: Accepted publickey for ec2-user from 10.104.2.109 port 36012 ssh2: RSA SHA256:9gwqMOcR6cYW7wHKde1F8nSjQyWo5xnLEXGIoa/A/ug
Nov 05 05:17:45 ip-10-104-73-161.ec2.internal sshd[7137]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)

And running journalctl -f causes the contents of the journal to be streamed to stdout and get picked up by the host's journal:

[root@admin]# sudo sheltie                                                                                                                                                             
bash-5.1# journalctl -u host-containers@admin
...
Nov 05 05:17:45 ip-10-104-73-161.ec2.internal host-containers@admin[2276]: Nov 05 05:17:45 ip-10-104-73-161.ec2.internal sshd[3244]: Accepted publickey for ec2-user from 10.104.2.109 port 36012 ssh2: RSA SHA256:9gwqMOcR6cYW7wHKde1F8nSjQyWo5xnLEXGIoa/A/ug
Nov 05 05:17:45 ip-10-104-73-161.ec2.internal host-containers@admin[2276]: Nov 05 05:17:45 ip-10-104-73-161.ec2.internal sshd[7137]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)

I haven't looked too deeply into the implications of this, but there are at least a couple potential downsides:

• The journal for the admin container uses volatile storage, which limits the potential size
• The logs in the host journal have systemd-related metadata (e.g. timestamps and hostnames) twice

It also causes logs from other services in the admin container to be stored in the journal (e.g. running sudo sheltie gets logged). This may actually be beneficial but I'm not very familiar with everything that could be running in an admin container.

[maherthomsi]

Hey @kailash-everlaw,

Sorry for the delayed response, I wanted to update you that we are discussing this feature as an option and will keep this isuse open. Any future updates will be posted here.

Thank you again for reaching out!