LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 10-22-2015, 07:49 AM   #1
Kippers
LQ Newbie
 
Registered: Sep 2015
Posts: 17

Rep: Reputation: Disabled
Kernel interrupts impacts on CPU-load


Hey there..

I have implementet Derek Molloys Loadable Kernel Module (see listing 4). It uses a Kernel Module to register a kernel interrupt on a GPIO rising edge. So every time there is a rising edge on a certain GPIO-pin, an interrupt (ISR) runs. The only thing happening in the interrupt, is counting up an integer. I'm running debian on the beaglebone (Linux beaglebone 3.8.13-bone47).

I put a square wave signal onto the GPIO, causing the interrupts to trigger with a certain frequency. If I turn the frequency up to somewhere above 10kHz, the processor freezes. I don't expect the the processor to be able to follow up to this pace, but I expect the load to be visible by the "top" command. Here is what I see:
http://i.imgur.com/Ar7nwu5.png

This measurement is taken with 10 kHz kernel interrupts running, but i still only get:

%Cpu(s): 0.0 hi

"hi" is defined as: "time spent servicing hardware interrupts" in man top
How can that be? How can i measure the impact the kernel interrupt has on the CPU's idletime?
 
Old 10-22-2015, 08:55 AM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 4,140

Rep: Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263
My guess is that something else is happening. 10 kHz is not fast enough to put any load on the CPU. Most likely you are freezing the first time that your interrupt overlaps with a previous one. Check your code.
 
1 members found this post helpful.
Old 10-29-2015, 10:52 AM   #3
Kippers
LQ Newbie
 
Registered: Sep 2015
Posts: 17

Original Poster
Rep: Reputation: Disabled
What should I look for in my code? I'm just attaching an interrupt to a GPIO-event from gpio.h, and then having a really simple ISR counting a variable up.
So, say I solve the problem in the code, do you then know how to measure the modules impact on the CPU-load?
The module isn't running in a thread, so I guess I shouldn't be visible on top?

Last edited by Kippers; 10-29-2015 at 10:53 AM. Reason: typo and formatting
 
Old 10-29-2015, 05:09 PM   #4
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,659
Blog Entries: 4

Rep: Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941
The load will not be visible on "top." Since a kernel module is "part of the kernel," all interrupt-processing is part of kernel overhead.

Everything that you describe, though, "is trivial." Therefore, undoubtedly you are mis-reading something here ... interpreting the symptoms that you see "wrongly."

"10,000 times a second" is no big deal for a CPU that executes hundreds of millions of ops per second, therefore there is a very strong probability that you are doing something wrong in your code or, much(!) more likely, in whatever userland program is interacting with your module's interface.

When you say, "the processor freezes," what exactly do you mean and exactly how do you know? There's an excellent chance that it is the (graphic?) user interface that is freezing. You might well find that Ctrl+Alt+2 or somesuch brings you to a terminal screen, even when you think the machine is "frozen." (Another Ctrl+Alt+# sequence should return you to the GUI.) If this is the case, then the problem is not in the kernel at all, but in userland.

Last edited by sundialsvcs; 10-29-2015 at 05:11 PM.
 
Old 11-05-2015, 07:48 AM   #5
Kippers
LQ Newbie
 
Registered: Sep 2015
Posts: 17

Original Poster
Rep: Reputation: Disabled
I have stripped the IRQ function (ISR) down even further. Now it is like this:

Code:
static irq_handler_t ebbgpio_irq_handler(unsigned int irq, void *dev_id, struct pt_regs *regs) {
    return (irq_handler_t) IRQ_HANDLED;
}
Nothing happens when it is handling the interrupt.. So the freeze can't occur due to multiple simultaneous interrupts trying to access a shared variable.

I'm not running any GUI, I'm running it directly through the commandline. I have the beaglebone connected to my pc, via a USB-UART cable, so PuTTY is the most graphic user interface I get. The same happens when I connect via ethernet. I'm not running any other userland processes, as far as I know. When I run top, top it selves is the most demanding process.
By 'freeze' I mean: I have the kernel module inserted, when I turn the interrupt frequency up to somewhere around 10kHz, I get no response in PuTTY when I press the return key, then when I stop the interrupts from occuring, I get the responses delayed.

After this happens, I get the following messages:

Quote:
[12981.802097] omap_i2c 44e0b000.i2c: controller timed out
[12982.811840] omap_i2c 44e0b000.i2c: controller timed out
[12982.819654] dummy 0-0034: Error -110 reading from cec:0xfe
[12983.825497] omap_i2c 44e0b000.i2c: controller timed out
[12983.833325] tps65217 0-0024: Read from reg 0xf failed
[12981.802097] omap_i2c 44e0b000.i2c: controller timed out

I'm running the LKM with a kboject, in order to access the device from userland, maybe this is part of the reason?

Now that I know that I can't use top to meassure the impact the kernel interrupts has on the cpu load, do you then know any other method to measuring this? I wanted to use mpstat, but it seems that it doesn't come with "Linux beaglebone 3.8.13-bone47".

Sorry, if my langauge isn't always "correct", I'm new to this environment..
 
Old 11-06-2015, 08:42 AM   #6
Kippers
LQ Newbie
 
Registered: Sep 2015
Posts: 17

Original Poster
Rep: Reputation: Disabled
So I figured out that mpstat is part of the sysstat package. So when I use mpstat and iostat (also part of sysstat) to measure the impact of the kernel interrupt on the CPU-load, I still don't see any numbers changing before processor freezes.

I'm I going about this the wrong way? Is it even possible to measure kernel interrupts impact on CPU-load?
 
Old 11-06-2015, 07:38 PM   #7
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,659
Blog Entries: 4

Rep: Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941
"The Processor" does not "freeze."

Something is causing your user interface to freeze, or something is causing the system-at-large to be unable to service processes and threads.

There is an enormous difference between the two. Please describe the user-environment that you are using when you conclude that "the computer 'freezes.'" (Exactly(!) how do you know?)

Of one thing I am quite sure: "10 kiloHertz" is not(!) "a load" for a CPU that executes billions of operations per second. (In fact, this is a frequency that you could easily hear.)

I suspect that you are "barking up the wrong tree" here because you have a fundamental mis-conception as to the fundamental nature of the actual problem. ("Show of hands, please, of how many other people this same thing has happened to?" And we all said, "d'oh!", did we not??)

You're dealing with a bug that is somehow standing in the way of your ability to identify the true nature of what the bug is . . .

Try booting your Linux in single-user mode and trying again. (Use su username to pass from "all-powerful root mode to that of "an ordinary user," then cd to that user's home directory ...) (Now, you are "your same old non-privileged self," but, "there is nothing else here.")

Last edited by sundialsvcs; 11-06-2015 at 07:42 PM.
 
Old 11-10-2015, 09:59 AM   #8
Kippers
LQ Newbie
 
Registered: Sep 2015
Posts: 17

Original Poster
Rep: Reputation: Disabled
I'm not running on a traditional PC! It's running on a Beaglebone Black, an embedded processor, running "Linux beaglebone 3.8.13-bone47". I don't have a graphical user interface. I'm communicating with the Beaglebone from my Windows PC, thourh a USB cable using PuTTY. I get the same behavior whether I'm running as a root or as my "non-privileged"-used.

I guess knowing the interrupt frequency itselves isn't enough to determine the CPU-load. One interrupt may cause more than one execution(very likely).
I ran the tests again, this time logged in as root. I tried different interrupt frequencies, ranging from 1Hz to 20kHz. When I hit 20kHz, PuTTY became unresponsive, nothing happened when I pressed any keys. I got the following error meassage in the terminal:
Quote:
[ 1564.039226] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:1:15]
[ 1564.064735] BUG: scheduling while atomic: kworker/0:1/15/0x40010100
[ 1585.219861] musb_g_ep0_irq 710: SetupEnd came in a wrong ep0stage wait
Even the small heartbeat LED connected to the processor, stopped beating. Is this evidence of a processor freeze?

During the tests, I looked at top, cpufreq-ino, mpstat and iostat at every choosen frequency, and I didnt see any significant change in any of the values between the different tests.

How can I investigate what causes the processor to freeze?
How can I measure the impact the kernel interrupts has on the CPU-load?
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Configure cpu interrupts manually honeybadger Programming 11 12-20-2010 12:25 PM
Interrupts being processed by only 1 cpu duryodhan Slackware 35 09-10-2007 07:48 PM
FAQ of Kernel changes and the impacts? brundles Linux - General 1 05-11-2005 09:59 AM
kernel 2.4.20-28.7 & cpu load Rodion Red Hat 1 02-18-2005 07:31 PM
hardware interrupts using too much cpu Rocker Linux - General 4 10-12-2004 05:55 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 11:09 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration