Is Your Linux Version Hiding Interrupt CPU Usage From You? | Tanel Poder Consulting

[utterances-bot]

Is Your Linux Version Hiding Interrupt CPU Usage From You? | Tanel Poder Consulting

TL;DR: Some Linux distros and even different kernel versions within a distro have disabled IRQ time accounting. In such case, your monitoring tools will report zero time spent in hardware interrupt handlers (shown as %hi, %irq, hiq, etc). It’s easy to check how your kernel is behaving by looking at /proc/stat and you can still measure interrupt CPU usage using perf and a little creativity.
Index Index Problem Explanation Kernel configuration options Measuring IRQ time with perf Different distros and kernel versions Summary Here’s a reminder that your Linux kernel may be not breaking down hardware interrupt CPU usage separately in its /proc/stat CPU time reporting, depending on the kernel’s compile-time set

https://tanelpoder.com/posts/linux-hiding-interrupt-cpu-usage/

[nerkn]

For full stack dev perspective you see something is consuming cpu resources but you can't locate anything, like your server under light load slows. Thanks for this reaserch, and i can understand ubuntu devs choose to become op sys for dumps.

[jkstill]

This bit from the VMware paper caught my eye: "There is also a timer that can generate periodic interrupts at any power-of- two rate from 2Hz to 8,192Hz"

Once upon a time (Oracle 8 I think) I found that Oracle was measuring time in binary (1024) milliseconds.

Now I thin I know why.

[tanelpoder]

Ha indeed. I think there was also the issue of some operating systems doing quick math by shifting bits instead of proper multiplication/division (so x1000 became x1024 thanks to the bit shift). Not everywhere, of course, but in some sleep/wait duration related places...

[sukwoo1414]

Hello, I have some questions about time accounting of linux. My 2 core system with a main process which has about 30 threads, is making weird symptom. When I see top or htop, cpu utilization % is very low, but my main process(which has 30 threads)'s cpu usage is very high.
For example,
core 0 utilization: 13%
core 1 utilization: 5%
My main process' cpu usage: 58% (and a single thread in this process has cpu usage of 25%, which is pinned in core 1)

It doesn't have any IOwait time. How can this possible?? I checked /proc/stat and /proc//stat, and I saw weird thing.
In /proc/stat, user and sys time is growing very slowly (about 5~10 in 1 sec), and idle time is growing far more faster(about 90 in 1 sec).
However in /proc//stat of my main process, utime and stime is increasing 2~30 per seconds.
Why does usr and sys time not increasing in /proc/stat, but utime and stime is growing so faster in /proc//stat??? This makes strange core utilization and process's cpu usage.
Does values in /proc/stat and /proc//stat reliable?
Below is my kernel config.
CONFIG_IRQ_TIME_ACCOUNTING is not set
CONFIG_NO_HZ_FULL=y
COFNIG_VIRT_CPU_ACCOUNTING_GEN=y

[tanelpoder]

What distro & kernel version are you using?

Perhaps there's some accounting/summarization error somewhere in /proc/PID/stat and you need to look into the thread level detail instead /proc/PID/task/TID/stat... use top -H (or just press "H" in top) to see individual thread's CPU breakdown.

[sukwoo1414]

Thank's for reply!! It's Xilinx's Petalinux(which is Yocto based), v5.4 and PREEMPT-RT.
I also looked at /proc/PID/task/TID/stat and top -H or threaded view of htop.
A single thread pinned in core 1 has cpu usage of 25% and core 1's cpu utilization is just 5%. Also utime and stime value in /proc/PID/task/TID/stat (which has 25% cpu usage) is growing far faster than that of /proc/stat. So I concluded htop or top has no problem. But how can I say Linux kernel's CPU accounting is wrong??? I thought that CPU time accounting of Linux is mature enough...

[liupzmin]

Thanks for your research，your article has settled my doubts，I‘m deeply grateful.