Revpi Core S + Bookworm leads to high sysload

[TriNiklas]

We investigated a problem with Revpi Core S modules and the new bookworm release.
On a fresh installation we see sysloads around 90%. This happend on five different Revpi Core S modules.
We tested the lite and default image with the same results. A downgrade to bullseye solved it for us.

Our Setup is as follow:
- Revpi Core S
- Revpi AIO
- Revpi DIO

We catched following difference between Bookworm and Bullseye via command

Code: Select all

ps -eo state,pid,cmd | grep '^D'

Bookworm:

bookworm output

bookworm.png (5.9 KiB) Viewed 1449 times

Bullseye:

bullseye output

Bullseye.png (2.66 KiB) Viewed 1449 times

We see that in Bookworm more processes are uninterruptle sleeping. That could lead to the high sysload.
We have the problem only on Revpi Core S not on Revpi Core SE.

We have sos reports for bookworm-lite, bookworm-default and bullseye-lite. If needed i could send them to a support mail.

From our side this is a critical bug. We have alot of Revpi Core S still in use and we currently can't switch to bookworm.
Without the possibility to update os releases remotely, this means alot of work for us soon, if this is not addressed.

[nicolaiB]

Hi,

which version of Core S is this? 1.1 or 1.0?

Nicolai

[TriNiklas]

Here are the firmware versions of the module. Ignore that the AIO and DIO are not configured. We configuried them during the tests and made sure they are correctly configured.

What we catched is that Core SE and Core S share the same type of 95. Is this intended?

[nicolaiB]

Please try if the following change brings back the load to previous numbers:

1. Open /lib/udev/rules.d/50-revpi.rules
2. Change

Code: Select all

ACTION=="add", SUBSYSTEM=="net", DEVPATH=="*/spi0.0/net/*eth*", NAME="pileft", TAG+="systemd", ENV{SYSTEMD_WANTS}="pileft-quirks.service"
ACTION=="add", SUBSYSTEM=="net", DEVPATH=="*/spi0.1/net/*eth*", NAME="piright", TAG+="systemd", ENV{SYSTEMD_WANTS}="piright-quirks.service"

to (remove `SUBSYSTEM=="net", `)

Code: Select all

ACTION=="add", DEVPATH=="*/spi0.0/net/*eth*", NAME="pileft", TAG+="systemd", ENV{SYSTEMD_WANTS}="pileft-quirks.service"
ACTION=="add", DEVPATH=="*/spi0.1/net/*eth*", NAME="piright", TAG+="systemd", ENV{SYSTEMD_WANTS}="piright-quirks.service"

3. Save the file and reboot.

[TriNiklas]

I changed the two lines and it seems it fixed it. The sysload is now around 25%-30% on the lite image.
The irqpoll/pileft and irqpoll/piright are not anymore in a blocking state.

Thanks for the fast help/troubleshoot.

Further questions i have:

- Is this a problem in general or did we somehow triggered it by setting up the revpi incorrectly?
- If it is a general problem, will this be fixed via a new os image or can this also be fixed with apt later?

[nicolaiB]

Thanks for testing it!

No, this isn't anything related to a bad update. We will patch the file and make a new release of the package revpi-base-files (which contains the udev rules). After that the rules can be fixed by the usual apt upgrade. It will be also part of the next image release in December.

[TriNiklas]

Ok thank you. This helps us alot.

[nicolaiB]

We have just published revpi-base-files (1.2.1-1+deb12+1), which fixes the reported issue. You can update the affected systems with the usual apt update / apt upgrade.

[TriNiklas]

Perfect. We tested it and it seems to work.

Thanks for the fast fix.