gluon-core: entropy daemon not available on all platforms

[blocktrron]

Bug report

What is the problem?

urngd is not compatible with every platform. The timer is to coarse to generate jitter entropy from it on some platforms.

The kernel ringbuffer will in this case read

[   10.250566] urngd: jent-rng init failed, err: 2

This problem exists at least on ramips-mt7620 and bcm-2708.

What is the expected behaviour?

Have an entropy daemon for generating random ssh host keys as well as fastd keys.

Gluon Version:

Gluon 2020.2

Site Configuration:

Not site-dependent if haveged is not included in the site.

Custom patches:

[neocturne]

Any ideas how to solve this? Does haveged work on these platforms?

[blocktrron]

Yes, haveged works. AFAIR was haveged only a recommendation in the docs (cit needed), mentioning this in the docs might be fine.

Otherwise, adding a dependency for haveged and removing urngd from the package set is also a viable solution.

[neocturne]

It would be interesting to know if haveged actually produces good entropy on these platforms, and if it does, if urngd can be made to do the same. I can have a look at the code this week, but it's unlikely that I'll find out anything without the hardware in question...

[neocturne]

Upstream jitterentropy-rngd used by urng recently gained a fallback mode for systems without high-resolution timers. We should evaluate if urng can be updated to use the new version without increasing its size too much.

[neocturne]

Well... the fallback mode "works", but urngd now consumes 100% CPU for several minutes on boot (tested on brcm2708).

On brcm2708, urngd is not included by default anymore on OpenWrt master, instead /dev/random is sourced by the hwrng (this seems to be handled fully in kernel now?).

One remaining question would be how to deal with ramips-mt7620 (and other targets with this issue that we don't know of)

[neocturne]

urng has been updated to use libjitterentropy-rngd with fallback support some time ago, so the original issue should be fixed. Now the question is how urngd behaves on mt7620...

[neocturne]

Hmm, I don't think I own any mt7620 devices. It would be great if someone could install a current Gluon (2023.2.x should be sufficient) and check if urngd uses a lot of CPU after boot.

[neocturne]

Found an Archer C2 in my stash; as expected, urngd runs now, but takes a long time to generate enough entropy on boot (2-3 minutes for dropbear to generate its RSA keys, 3-4 minutes for urngd to finish and stop hogging the CPU). Not optimal, but overall acceptable behavior, as the system is working fine once it has booted, and the generated entropy should be good enough.

We may consider disabling RSA host keys for dropbear, as all SSH clients should support Ed25519 these days (support was added to OpenSSH in version 6.5 from 2014).

[Djfe]

A fix was just pushed upstream. You might want to compare performance in openwrt main with the fix
openwrt/openwrt@eec11fb

I quite like the idea of dropping RSA host keys

[Djfe]

this following text was due to a great misunderstandment (host key vs client key), but I'll leave it here in a separate comment but mark it so it stays hidden

but fear the implications of locking people out of their devices. Especially devices like the NWA55AXE without a reset button. (yes, you can still fix this serially, but that not so nice with a device attached to some mast outdoors)

brainstorming workarounds (probably way to specific):
We could add a script that increases the major version by one in-case no RSA key is installed on the device to allow updating it to a new major version release without RSA.
Alternatively the script could increase major version if atleast one ED25519 key is present.
Alternatively this could only be done/enforced for devices without a reset key. But I dislike that approach

Note: The new sysupgrade doesn't have to contain an image that identifies as a new major release. it's enough if the metadata of the sysupgrade says so. (avoids having to keep this change around forever but would require an intermediate release)

Overall new communities should be ED25519 only. We could add specific settings to site.conf or specific package names to image-customization.lua to allow for communities themselves to switch to this mode while we figure out how to do a transition.

Aachen likely has tons of devices out there with just rsa keys (even if most of them are by Freifunk members we have no way of reaching everyone affected)

I just had an idea for an actual alternative to the above (increase of the major version):
Add entries to site.json for alternative update paths for devices with no keys or just ED25519-keys.
Those can be used by communities to transition devices into a new release without RSA.
Devices won't accept these no-rsa images over the regular old update path. either due to new metadata or a specific file-name for no-rsa images.

[rotanid]

We may consider disabling RSA host keys for dropbear, as all SSH clients should support Ed25519 these days (support was added to OpenSSH in version 6.5 from 2014).

if that is only for the router's host keys and one can still use RSA as ssh client key?
and would this change generate new keys on all nodes or leave old rsa host keys?

[neocturne]

@rotanid Yes, host key and client key types are independent.

Currently all nodes have both RSA and Ed25519 keys; I don't see a reason to delete existing RSA keys, so only new/firstbooted nodes would become Ed25519 only. There is no configuration option specifically for key generation at the moment however, so this would require a change in OpenWrt.

[neocturne]

Some numbers from my Archer C2:

• From the start of boot, Dropbear takes ~140s (measured by /proc/uptime) to generate its keys before an SSH connection is possible
• urngd takes ~240s before it settles

openwrt/openwrt@eec11fb does not have any effect on this (just because the kernel module doesn't fail to load anymore, the device doesn't suddenly have a good entropy source)

Disabling RSA key generation, dropbear starts almost immediately (even before netifd finishes setting up br-setup). I'm not sure if it also has an effect on urngd (by consuming less entropy that has to be generated) - it only took 220s to settle in my second try, but that might also have been... random.