Hi all,
I have a Connect 5 with 2 AIO modules connected to it. Each AIO is using the two RTD readouts with 3-wire measurements.
For completion, these are ingested in a Node-Red flow via the revpi-getpin node in a 1Hz frequency, and put in an influxDB database.
It works well most of the time but sometimes, seemingly at random, the RTD values of the AIO2 (and jsut form AIO2) get stuck at a constant value, independent of the actual temperature/resistance of the probe. With pyTest in the terminal the values are the same as read through the Node-RED node. Rebooting the RevPi brings it back to correct working state.
Any idea what might be happening? Or how to debug?
Thanks!
Ricardo
RTD from AIO module stuck
Re: RTD from AIO module stuck
Which image (/etc/revpi/image-release) and kernel (uname -a) are you using?
Are there any error messages in the kernel log (/var/log/kern.log)?
BR
Nicolai
Are there any error messages in the kernel log (/var/log/kern.log)?
BR
Nicolai
Nicolai
Re: RTD from AIO module stuck
Hi Nicolai,
image: 250528-revpi-bookworm-arm64-default
kernel: Linux RevPi151017 6.12.87-revpi0-rpi-v8 #1+deb12+1 SMP PREEMPT_RT Fri May 8 11:04:59 UTC 2026 aarch64 GNU/Linux
There are logs indeed! Recent logs are on the latest reboot but around the time the RTDs got stuck there are full sections with:
These go on and on until reboot:
Best,
Ricardo
image: 250528-revpi-bookworm-arm64-default
kernel: Linux RevPi151017 6.12.87-revpi0-rpi-v8 #1+deb12+1 SMP PREEMPT_RT Fri May 8 11:04:59 UTC 2026 aarch64 GNU/Linux
There are logs indeed! Recent logs are on the latest reboot but around the time the RTDs got stuck there are full sections with:
Code: Select all
2026-06-09T03:41:38.317636+01:00 RevPi151017 kernel: [60172.416335] piControl: too many communication errors -> set inputs to default 0 0 25 0 0 0 0 0
2026-06-09T03:41:38.332132+01:00 RevPi151017 kernel: [60172.430844] piControl: too many communication errors -> set inputs to default 0 0 26 0 0 0 0 0
2026-06-09T03:41:38.346632+01:00 RevPi151017 kernel: [60172.445527] piControl: too many communication errors -> set inputs to default 0 0 27 0 0 0 0 0
2026-06-09T03:41:38.361137+01:00 RevPi151017 kernel: [60172.459838] piControl: too many communication errors -> set inputs to default 0 0 28 0 0 0 0 0
2026-06-09T03:41:38.375634+01:00 RevPi151017 kernel: [60172.474374] piControl: too many communication errors -> set inputs to default 0 0 29 0 0 0 0 0
2026-06-09T03:41:38.390133+01:00 RevPi151017 kernel: [60172.488832] piControl: too many communication errors -> set inputs to default 0 0 30 0 0 0 0 0
2026-06-09T03:41:38.404636+01:00 RevPi151017 kernel: [60172.503377] piControl: too many communication errors -> set inputs to default 0 0 31 0 0 0 0 0
2026-06-09T03:41:38.419136+01:00 RevPi151017 kernel: [60172.517992] piControl: too many communication errors -> set inputs to default 0 0 32 0 0 0 0 0
2026-06-09T03:41:38.433631+01:00 RevPi151017 kernel: [60172.532337] piControl: too many communication errors -> set inputs to default 0 0 33 0 0 0 0 0
2026-06-09T03:41:38.448132+01:00 RevPi151017 kernel: [60172.546837] piControl: too many communication errors -> set inputs to default 0 0 34 0 0 0 0 0
2026-06-09T03:41:38.462639+01:00 RevPi151017 kernel: [60172.561455] piControl: too many communication errors -> set inputs to default 0 0 35 0 0 0 0 0
2026-06-09T03:41:38.477134+01:00 RevPi151017 kernel: [60172.576071] piControl: too many communication errors -> set inputs to default 0 0 36 0 0 0 0 0
2026-06-09T03:41:38.491637+01:00 RevPi151017 kernel: [60172.590334] piControl: too many communication errors -> set inputs to default 0 0 37 0 0 0 0 0
2026-06-09T03:41:38.506135+01:00 RevPi151017 kernel: [60172.604905] piControl: Error during piBridge communicationCode: Select all
2026-06-09T14:48:05.504711+01:00 RevPi151017 kernel: [100159.970794] piControl: Error during piBridge communication
2026-06-09T14:48:05.504726+01:00 RevPi151017 kernel: [100159.970799] piControl: too many communication errors -> set inputs to default 0 0 255 0 0 0 0 0
2026-06-09T14:48:05.520170+01:00 RevPi151017 kernel: [100159.985164] piControl: too many communication errors -> set inputs to default 0 0 255 0 0 0 0 0
2026-06-09T14:48:05.536922+01:00 RevPi151017 kernel: [100159.999739] piControl: too many communication errors -> set inputs to default 0 0 255 0 0 0 0 0
2026-06-09T14:48:05.549174+01:00 RevPi151017 kernel: [100160.014353] piControl: too many communication errors -> set inputs to default 0 0 255 0 0 0 0 0
2026-06-09T14:48:05.563758+01:00 RevPi151017 kernel: [100160.028690] piControl: too many communication errors -> set inputs to default 0 0 255 0 0 0 0 0
2026-06-09T14:48:05.578240+01:00 RevPi151017 kernel: [100160.043195] piControl: too many communication errors -> set inputs to default 0 0 255 0 0 0 0 0
2026-06-09T14:48:05.593132+01:00 RevPi151017 kernel: [100160.058520] piControl: too many communication errors -> set inputs to default 0 0 255 0 0 0 0 0
2026-06-09T14:48:24.627908+01:00 RevPi151017 kernel: [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x414fd0b1]Ricardo
Re: RTD from AIO module stuck
Hi Ricardo,
could you please share the output of
ps -eLo cls,rtprio,psr,comm
?
Thanks
could you please share the output of
ps -eLo cls,rtprio,psr,comm
?
Thanks
Nicolai
Re: RTD from AIO module stuck
Hi Nicolai,
That would be this:
At the time of the errors I was sampling the RTDs at 1Hz. I've now change to a period of 5 s and it did not get stuck again yet (2 days and counting).
Thanks for the help!
Best,
Ricardo
That would be this:
Code: Select all
ps -eLo cls,rtprio,psr,comm
CLS RTPRIO PSR COMMAND
TS - 0 systemd
TS - 3 kthreadd
TS - 0 pool_workqueue_release
TS - 0 kworker/R-kvfree_rcu_reclaim
TS - 0 kworker/R-rcu_gp
TS - 0 kworker/R-sync_wq
TS - 0 kworker/R-slub_flushwq
TS - 0 kworker/R-netns
TS - 1 kworker/u16:0-ipv6_addrconf
TS - 0 kworker/R-mm_percpu_wq
TS - 0 rcu_tasks_kthread
TS - 0 rcu_tasks_rude_kthread
TS - 0 rcu_tasks_trace_kthread
TS - 0 ksoftirqd/0
TS - 1 pr/legacy
FF 1 3 rcu_preempt
FF 1 0 rcub/0
TS - 0 rcu_exp_par_gp_kthread_worker/0
FF 1 0 rcuc/0
TS - 1 rcu_exp_gp_kthread_worker
FF 99 0 migration/0
FF 1 0 irq_work/0
TS - 0 cpuhp/0
TS - 1 cpuhp/1
FF 1 1 irq_work/1
FF 99 1 migration/1
FF 1 1 rcuc/1
TS - 1 ksoftirqd/1
TS - 1 kworker/1:0H-events_highpri
TS - 2 cpuhp/2
FF 1 2 irq_work/2
FF 99 2 migration/2
FF 1 2 rcuc/2
TS - 2 ksoftirqd/2
TS - 3 cpuhp/3
FF 1 3 irq_work/3
FF 99 3 migration/3
FF 1 3 rcuc/3
TS - 3 ksoftirqd/3
TS - 3 kworker/3:0H-events_highpri
TS - 0 kdevtmpfs
TS - 3 kworker/R-inet_frag_wq
TS - 1 kauditd
FF 50 0 irq/15-107c013880.mailbox
TS - 3 khungtaskd
TS - 3 oom_reaper
TS - 3 kworker/R-writeback
TS - 3 kcompactd0
TS - 3 kcompactd1
TS - 0 kcompactd2
TS - 3 kcompactd3
TS - 0 kcompactd4
TS - 3 kcompactd5
TS - 1 kcompactd6
TS - 1 kcompactd7
TS - 3 kworker/R-kintegrityd
TS - 3 kworker/R-kblockd
TS - 3 kworker/R-blkcg_punt_bio
TS - 0 backlog_napi/0
TS - 1 backlog_napi/1
TS - 2 backlog_napi/2
TS - 3 backlog_napi/3
FF 50 3 watchdogd
TS - 3 kworker/R-quota_events_unbound
TS - 3 kworker/3:1H-kblockd
TS - 0 kworker/R-rpciod
TS - 0 kworker/R-xprtiod
TS - 0 kswapd0
TS - 0 kswapd1
TS - 0 kswapd2
TS - 0 kswapd3
TS - 0 kswapd4
TS - 0 kswapd5
TS - 0 kswapd6
TS - 2 kswapd7
TS - 0 kworker/R-nfsiod
TS - 0 kworker/R-kthrotld
TS - 1 hwrng
TS - 0 kworker/R-iscsi_conn_cleanup
TS - 0 kworker/R-nvme-wq
TS - 0 kworker/R-nvme-reset-wq
TS - 0 kworker/R-nvme-delete-wq
FF 50 0 irq/34-1000480000.usb
FF 50 0 irq/34-dwc2_hsotg:usb1
TS - 0 kworker/R-uas
FF 50 0 irq/35-107d508200.i2c
TS - 1 kworker/1:1H-kblockd
TS - 2 kworker/u21:0-hci0
TS - 0 kworker/u22:0
TS - 0 kworker/u23:0
TS - 0 kworker/u24:0
TS - 0 kworker/u25:0
FF 50 0 irq/41-PCIe PME
FF 50 0 irq/41-aerdrv
FF 50 0 irq/41-s-aerdrv
FF 50 0 irq/45-PCIe PME
FF 50 0 irq/45-aerdrv
FF 50 0 irq/45-s-aerdrv
TS - 0 kworker/R-sdhci
FF 50 2 irq/168-mmc0
TS - 0 kworker/R-sdhci
FF 50 2 irq/168-s-mmc0
FF 50 2 irq/169-mmc1
FF 50 2 irq/169-s-mmc1
FF 50 1 irq/113-eth%d
TS - 2 ptp0
FF 50 0 irq/147-dw_axi_dmac_platform
FF 50 0 irq/138-xhci-hcd:usb2
FF 50 1 irq/143-xhci-hcd:usb4
TS - 0 kworker/0:1H-kblockd
TS - 2 kworker/R-mmc_complete
TS - 2 kworker/2:1H-kblockd
FF 50 1 irq/115-1f00074000.i2c
FF 50 3 irq/127-1f00054000.spi
TS - 1 spi1
TS - 0 kworker/0:2-mm_percpu_wq
FF 50 2 irq/171-1-0022
FF 50 2 irq/172-1-0021
FF 50 1 irq/173-pwr_button
FF 50 1 irq/175-107c580000.hvs
FF 50 1 irq/176-107c580000.hvs
FF 50 2 irq/177-107c580000.hvs
FF 50 0 irq/178-vc4 hdmi hpd connected
FF 50 1 irq/179-vc4 hdmi hpd disconnected
TS - 1 cec-vc4-hdmi-0
FF 50 0 irq/22-DMA IRQ
FF 50 1 irq/181-vc4 hdmi cec rx
FF 50 1 irq/181-s-vc4 hdmi cec rx
FF 50 1 irq/182-vc4 hdmi cec tx
FF 50 1 irq/182-s-vc4 hdmi cec tx
FF 50 0 irq/23-DMA IRQ
FF 50 0 irq/24-DMA IRQ
FF 50 0 irq/180-107d004000.spi
TS - 0 spi10
TS - 3 kworker/R-v3d_bin
TS - 3 kworker/R-v3d_render
TS - 1 kworker/R-v3d_tfu
TS - 0 kworker/R-v3d_csd
TS - 0 kworker/R-v3d_cache_clean
TS - 0 kworker/R-v3d_cpu
FF 50 0 irq/183-v3d_core0
FF 50 0 irq/184-v3d_hub
FF 50 0 irq/185-107c500000.mop
FF 50 0 irq/186-107c501000.moplet
FF 50 0 irq/187-vc4 crtc
FF 50 0 card1-crtc0
FF 50 0 card1-crtc1
FF 50 0 card1-crtc2
TS - 2 kworker/R-md
TS - 1 kworker/R-md_bitmap
TS - 2 kworker/R-raid5wq
TS - 3 jbd2/mmcblk0p2-8
TS - 0 kworker/R-ext4-rsv-conversion
TS - 1 kworker/R-mld
TS - 2 kworker/R-ipv6_addrconf
TS - 2 kworker/u16:1-ipv6_addrconf
TS - 2 systemd-journal
FF 50 3 irq/150-uart-pl011
TS - 0 systemd-udevd
FF 54 0 piControl I/O
FF 50 1 revpi_gate_rcv
FF 50 1 irq/165-1f00008000.mailbox
FF 50 0 irq/190-pispbe
FF 50 0 irq/191-1000800000.codec
FF 50 0 irq/191-s-1000800000.codec
TS - 2 kworker/R-tpm_dev_wq
TS - 2 kworker/R-cfg80211
FF 50 2 irq/192-tpm0
TS - 3 kworker/R-brcmf_wq/mmc1:0001:1
TS - 2 brcmf_wdog/mmc1:0001:1
FF 50 0 irq/33-ttyS0
TS - 0 kworker/u21:2-hci0
TS - 0 avahi-daemon
TS - 1 bluetoothd
TS - 0 dbus-daemon
TS - 0 node-red
TS - 3 node
TS - 3 node
TS - 3 node
TS - 3 node
TS - 3 node
TS - 2 node
TS - 0 node
TS - 3 node
TS - 0 node
TS - 3 node
TS - 2 polkitd
TS - 1 gmain
TS - 2 gdbus
TS - 3 rsyslogd
TS - 2 in:imuxsock
TS - 1 in:imklog
TS - 3 rs:main Q:Reg
TS - 2 systemd-logind
TS - 3 thd
TS - 2 avahi-daemon
FF 50 0 irq/193-lan743x
FF 50 0 irq/194-lan743x
FF 50 0 irq/195-lan743x
FF 50 0 irq/196-lan743x
FF 50 0 irq/197-lan743x
FF 50 2 irq/198-lan743x
TS - 0 ptp1
TS - 2 NetworkManager
TS - 3 gmain
TS - 1 gdbus
TS - 0 wpa_supplicant
FF 50 0 irq/199-lan743x
FF 50 0 irq/200-lan743x
FF 50 2 irq/201-lan743x
FF 50 2 irq/202-lan743x
FF 50 2 irq/203-lan743x
FF 50 1 irq/204-lan743x
TS - 1 ptp2
TS - 0 dhclient
TS - 2 ModemManager
TS - 2 gmain
TS - 1 gdbus
FF 50 0 irq/170-1f00100000.ethernet-ffffffff:00
TS - 0 systemd-timesyn
TS - 2 sd-resolve
FF 50 2 irq/205-lan743x
FF 50 0 irq/206-lan743x
FF 50 0 irq/207-lan743x
FF 50 2 irq/208-lan743x
FF 50 2 irq/209-lan743x
FF 50 2 irq/210-lan743x
TS - 2 ptp4
TS - 0 noderedrevpinod
TS - 0 noderedrevpinod
TS - 2 noderedrevpinod
TS - 0 noderedrevpinod
TS - 0 noderedrevpinod
TS - 3 piModbusMaster
RR 20 2 piModbusMaster
TS - 1 influxd
TS - 3 influxd
TS - 2 influxd
TS - 1 influxd
TS - 0 influxd
TS - 1 influxd
TS - 1 influxd
TS - 0 influxd
TS - 2 influxd
TS - 3 influxd
TS - 1 influxd
TS - 3 influxd
TS - 1 influxd
TS - 0 influxd
TS - 1 influxd
TS - 2 influxd
TS - 2 influxd
TS - 1 sshd
TS - 2 agetty
TS - 0 dhcpd
TS - 2 exim4
TS - 1 revpipyloadd
TS - 2 revpipyloadd
FF 50 1 irq/152-uart-pl011
TS - 3 apache2
TS - 0 grafana
TS - 3 grafana
TS - 1 grafana
TS - 0 grafana
TS - 2 grafana
TS - 0 grafana
TS - 1 grafana
TS - 1 grafana
TS - 2 grafana
TS - 0 grafana
TS - 3 grafana
TS - 0 grafana
TS - 1 grafana
TS - 1 grafana
TS - 2 grafana
TS - 1 grafana
TS - 3 grafana
TS - 1 grafana
TS - 2 grafana
TS - 2 grafana
TS - 2 grafana
TS - 3 grafana
TS - 2 grafana
TS - 1 grafana
TS - 0 gpx_grafana_ela
TS - 0 gpx_grafana_ela
TS - 0 gpx_grafana_ela
TS - 0 gpx_grafana_ela
TS - 0 gpx_grafana_ela
TS - 0 gpx_grafana_ela
TS - 3 gpx_grafana_ela
TS - 2 gpx_grafana_ela
TS - 0 gpx_grafana_ela
TS - 0 gpx_grafana_ela
TS - 0 gpx_grafana_ela
TS - 1 rtkit-daemon
TS - 0 rtkit-daemon
RR 99 1 rtkit-daemon
TS - 2 krfcommd
TS - 0 udisksd
TS - 0 gmain
TS - 0 gdbus
TS - 2 probing-thread
TS - 0 cleanup
TS - 1 rpcbind
TS - 1 kworker/1:0-mm_percpu_wq
TS - 2 kworker/u19:0-writeback
TS - 2 apache2
TS - 2 apache2
TS - 0 apache2
TS - 3 apache2
TS - 0 apache2
TS - 0 cupsd
TS - 1 cups-browsed
TS - 3 gmain
TS - 3 gdbus
TS - 2 kworker/u19:3-flush-179:0
TS - 0 kworker/u17:0-flush-179:0
TS - 1 kworker/u18:1-kvfree_rcu_reclaim
TS - 2 kworker/2:2-mm_percpu_wq
TS - 0 kworker/u17:1-kvfree_rcu_reclaim
TS - 2 kworker/2:2H
TS - 3 kworker/u20:1-events_unbound
TS - 0 kworker/0:2H
TS - 2 kworker/u19:1-events_unbound
TS - 1 kworker/u18:0-events_unbound
TS - 0 apache2
TS - 0 apache2
TS - 1 apache2
TS - 0 apache2
TS - 1 apache2
TS - 0 kworker/0:0-cgroup_release
TS - 3 kworker/u20:0-events_unbound
TS - 2 kworker/2:0-cgroup_free
TS - 3 kworker/u20:2-kvfree_rcu_reclaim
TS - 1 kworker/u18:2-kvfree_rcu_reclaim
TS - 1 kworker/1:2-cgroup_release
TS - 3 kworker/3:1-mm_percpu_wq
TS - 0 kworker/u17:3-events_unbound
TS - 3 kworker/3:0-mm_percpu_wq
TS - 3 kworker/3:2-events_freezable
TS - 2 sshd
TS - 1 systemd
TS - 3 (sd-pam)
TS - 1 kworker/1:1-cgroup_free
TS - 1 pipewire
TS - 1 module-rt
TS - 1 data-loop.0
TS - 1 pipewire
TS - 0 module-rt
TS - 0 wireplumber
TS - 0 module-rt
TS - 3 gmain
TS - 3 pool-wireplumbe
TS - 1 gdbus
TS - 1 data-loop.0
TS - 1 pipewire-pulse
TS - 3 module-rt
TS - 2 data-loop.0
TS - 1 dbus-daemon
TS - 1 sshd
TS - 1 bash
TS - 2 psThanks for the help!
Best,
Ricardo
Re: RTD from AIO module stuck
A quick update that it happened again, now on the RTDs of the AIO1 module.
Similar error log. Back to expected behaviour after reboot.
Best,
Ricardo
Similar error log. Back to expected behaviour after reboot.
Best,
Ricardo
Re: RTD from AIO module stuck
Can you please create a SOS report and send it to support@kunbus.com? Thanks
Nicolai