Runaway CPU usage for 64bit BitCoin (Linux Client)

[knightmb]

I did a forum search, didn't see this mentioned by anyone yet.

I have 5 Bit Coin clients, some running Windows Bit Coin, others running Linux Bit Coin.

I only have one 64 bit Linux system, but when I run Bit Coin, after about 10 minutes, the CPU usage runs away and brings the system to a stall unless I kill the Bit Coin application.

Now I have 2 other computers running a 32 bit Linux Bit Coin client and they work just fine, no runaway CPU.

I've tried running the 32 bit Bit Coin on the 64 bit Linux system, but the same thing still happens. My 64 bit Linux system has a dual-core processor, and even when I choose the "use only 1 processor" option, after a while Bit Coin ends up using both CPU to the 100% max that it can.

Is this a bug that only affects 64 bit Linux users at the moment?

Using the latest version 0.3.0 from the front page of the website.

System OS: Linux Mandriva 2010.0 (64 bit)
RAM 16GB
Processor - Intel Dual-Core 3.06 CPU

Feedback or advice welcome.  


_{Edit by Xunie: I merged this thread with the other "Bitcoin slowing down your Linux system?" thread, so things my be confusing for a second, heh.}

[Xunie]

There has been a problem in settings the "nice" level on Linux.
A programmer who wasn't familiar with Unix/Linux used the highest priority (-20) instead of the lowest priority (20)!

So, we have a bug.

[lachesis]

I never noticed responsiveness issues. Perhaps I should nice Bitcoin until the fix.

[lachesis]

I wondered about that, but I've never noticed any issues on a dual-core 64bit system.

[knightmb]

There has been a problem in settings the "nice" level on Linux.
A programmer who wasn't familiar with Unix/Linux used the highest priority (-20) instead of the lowest priority (20)!

So, we have a bug.

Yeah, that always gets me too. You would think a negative number would mean less and positive number means more, but it turns out "Nice" works the opposite of that, you want it to be more "nice" so you want a higher number, LOL.

I checked the 32bit Linux clients, they run a Nice level of 0, so I guess it's running equal time with everything else.

On the Windows clients, Bit Coin runs at "normal" priority as well.

It's easy to start the process in low priority via command line for both of them, so for now I can do that to manually control their priority 

[Xunie]

It automatically raises it, I don't think nice 20 ./bitcoin will help to be honest.
I think some people have set higher priorities for themselves in /etc/security/limits.conf, and when -20 isn't available as a nice level, a lower priority is tried.
So, I think the dirty hack of editing /etc/security/limits.conf might fix it.

I wondered about that, but I've never noticed any issues on a dual-core 64bit system.

I don't have any problems on my quad core system here either!

[knightmb]

It automatically raises it, I don't think nice 20 ./bitcoin will help to be honest.
I think some people have set higher priorities for themselves in /etc/security/limits.conf, and when -20 isn't available as a nice level, a lower priority is tried.
So, I think the dirty hack of editing /etc/security/limits.conf might fix it.

I wondered about that, but I've never noticed any issues on a dual-core 64bit system.

I don't have any problems on my quad core system here either!

That may be the issue because you can't "Nice 20" anything, 19 is the max 

So far, I've "Nice 19" 3 clients (one 64 bit and two 32 bit) for the last hour and they haven't changed the Nice level on me yet.

I figured since the process is started at that level, it would remain for all child processes spawned afterwards. I could be wrong though, I'll check again in another few hours to see.

[knightmb]

On a side note, I've tracked down the other GUI issue.


The "minimize to tray instead of taskbar" is what was eating up all the CPU on my system. After I turned this off, the issue was resolved with Runaway CPU.

This only seems to affect the 64 bit Client, as the 32 bit Clients I have don't seem to be affected by this.

I did notice on the 64 bit Client, what happens is, it spawns multiple "tray" icons until X server finally kills over, so I guess I should submit that as a bug to somewhere? 

[asdfman]

knightmb - just curious, if you check not just the main process, but all the thread processes (ps -eflL | grep bitcoin - need the L, or else it will just show the main process), do you see the other thread processes change the priority after a little while ?  When I change the priority or renice, it will just stick for the main process, but some of threads will change priority everytime it goes to the generate coin function..

there may be some confusion in thinking that nice is working in that if people dont use -L to list all threads, they just see the PID of the main process, and think that the nice/renice has worked, when in fact the 2 subthreads that are maxing the CPU are the ones that are changing the priority back..

I myself just ripped out the whole priority changing function, so that whatever priority I set sticks permanently..

my post about ripping out priority change function is in : http://bitcointalk.org/index.php?topic=72.0
also shows the whole process tree.. in that code pasting, the main process would stay to whatever you nice/renice to, but 2 of the threads would change back, until i ripped out the function

also http://bitcointalk.org/index.php?topic=285.0 describes the bug with bitcoin changing the threads to use the highest priority instead of lowest

[knightmb]

knightmb - just curious, if you check not just the main process, but all the thread processes (ps -eflL | grep bitcoin - need the L, or else it will just show the main process), do you see the other thread processes change the priority after a little while ?  When I change the priority or renice, it will just stick for the main process, but some of threads will change priority everytime it goes to the generate coin function..

Feedback from the ps -elfL | grep bitcoin command

Code:

[knightmb@KnightMB ~]$ ps -eflL | grep bitcoin
0 S knightmb 11200  6781 11200  0   10  99  19 - 114169 poll_s 17:17 ?       00:00:02 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200  6781 11202  0   10  99  19 - 114169 hrtime 17:17 ?       00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200  6781 11203  0   10  99  19 - 114169 hrtime 17:17 ?       00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200  6781 11204  0   10  99  19 - 114169 hrtime 17:17 ?       00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200  6781 11212  0   10  99  19 - 114169 sk_wai 17:17 ?       00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 R knightmb 11200  6781 11213  0   10  99  19 - 114169 -     17:17 ?        00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200  6781 11214  0   10  99  19 - 114169 hrtime 17:17 ?       00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200  6781 11215  0   10  82   2 - 114169 hrtime 17:17 ?       00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 R knightmb 11200  6781 11216 86   10  99  19 - 114169 -     17:17 ?        00:40:37 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 R knightmb 11200  6781 11219 85   10  99  19 - 114169 -     17:17 ?        00:40:11 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
0 R knightmb 13291 12282 13291  0    1  80   0 -  1842 -      18:04 pts/1    00:00:00 grep --color bitcoin

Anything in that look out of place? I did notice one that was nice to "2" instead of "19"


[edit] I see now from the other topic, I'm running the client, the other was the server daemon, might be a bug with just the daemon, not sure though.

[asdfman]

yep, that 2 is exactly what im talking about..
wait a little longer and another thread might reset too.. after my change all my threads all stay 19 forever....

[knightmb]

yep, that 2 is exactly what im talking about..
wait a little longer and another thread might reset too.. after my change all my threads all stay 19 forever....

Ah I see now. Since you've already dove into the code and disabled the function, is there any reason why the function in question wouldn't just set everything to lowest priority by default? Why would it need to change it around? I'm afraid to dive in myself, otherwise I'll spend days fixing it, hehe.

[asdfman]

from what I saw, there was an error in util.h basically the THREAD_PRIORITY_LOWEST was defined as PRIO_MIN.. the intention was to switch to the "lowest" priority, but in the linux kernel source in resourch.h PRIO_MIN is actually the lowest NUMBER (-20), meaning it actually changed to the HIGHEST priority.. and then after that function is done, it will change priority again (to either 0 or 2, not as bad as -20) but either way it is changing it, so either way no matter what you nice/renice the threads to, it will switch to diff priority levels automatically, with those particular threads that switch priorities never being the most unobtrusive at 19..

Its been stated in the latest source code, this min/max is fixed, so if you dl the latest source tree and compile the min/max reversal should be fixed... personally I didnt care of having any of the threads change priority on its own at all, so I just removed the function alltogether..

[knightmb]

from what I saw, there was an error in util.h basically the THREAD_PRIORITY_LOWEST was defined as PRIO_MIN.. the intention was to switch to the "lowest" priority, but in the linux kernel source in resourch.h PRIO_MIN is actually the lowest NUMBER (-20), meaning it actually changed to the HIGHEST priority.. and then after that function is done, it will change priority again (to either 0 or 2, not as bad as -20) but either way it is changing it, so either way no matter what you nice/renice the threads to, it will switch to diff priority levels automatically, with those particular threads that switch priorities never being the most unobtrusive at 19..

Yeah, I finally found it as well (I knew I shouldn't have peaked in the code, LOL). I'm going to try manually setting it to hard code 19 and see if that makes a difference after a compile. If so, it might be a more elegant solution that removing the function all together since it appears it's trying to balance out when to use more CPU depending on the system load.

[asdfman]

hardcoding to 19 should work, also you could just change PRIO_MIN to PRIO_MAX which would basically do the same thing, (technically PRIO_MAX is defined as 20 in the linux kernel source)

I agree that there authors saw some practicality in changing the priority depending on what the function is doing, but personally Id rather just have it not touch the priority at all, as the program will work just fine sitting at lowest priority all day.. I dont want it ever stealing any priority from any process regardless of what the thread is doing.. (maybe it would be a good suggestion to make this a command line parameter to disable/enable automatic priority adjustment - Im sure theres many who would prefer to keep the automatic priority scheduling in place, but also plenty that don't want any part of it)

[knightmb]

hardcoding to 19 should work, also you could just change PRIO_MIN to PRIO_MAX which would basically do the same thing, (technically PRIO_MAX is defined as 20 in the linux kernel source)

I agree that there authors saw some practicality in changing the priority depending on what the function is doing, but personally Id rather just have it not touch the priority at all, as the program will work just fine sitting at lowest priority all day.. I dont want it ever stealing any priority from any process regardless of what the thread is doing.. (maybe it would be a good suggestion to make this a command line parameter to disable/enable automatic priority adjustment - Im sure theres many who would prefer to keep the automatic priority scheduling in place, but also plenty that don't want any part of it)

Good point. I just wanted to test if it made a difference (I suspect it will). But after reading the other topics, seems the fix was already checked into the SVN, just not in time for the release so there is no sense in me trying a fix that which is already fixed code wise, hehe.

I agree though, it just needs to run at the lowest prioirty at *all* times, I mean it's suppose to run off of idle CPU and a nice level of 2 isn't far from *shared* CPU cycles than *idle* CPU cycles in my opinion.

[aceat64]

One fairly simple workaround is to run bitcoind as a user other than root. Since bitcoin does not require root permissions it will run just fine and it won't be able to set a negative niceness (only root can do that). That will ensure that the highest priority bitcoind can use is 0 (normal).

[satoshi]

After it initially tries incorrectly to set itself to the lowest priority, the generate thread only changes its priority again temporarily when it finds a block.  When you've found a block, you should want it to hurry up and broadcast it as soon a possible before someone else finds one and makes yours invalid.  The generate thread only changes to higher priority for less than a second every few days.

There should be a 0.3.1 release for this soon.  There are a few other issues we need to look at fixing in 0.3.1 before making a release.

On a side note, I've tracked down the other GUI issue.

The "minimize to tray instead of taskbar" is what was eating up all the CPU on my system. After I turned this off, the issue was resolved with Runaway CPU.

This only seems to affect the 64 bit Client, as the 32 bit Clients I have don't seem to be affected by this.

I did notice on the 64 bit Client, what happens is, it spawns multiple "tray" icons until X server finally kills over, so I guess I should submit that as a bug to somewhere? 

That's interesting.  I know the minimize to tray on Ubuntu is very clunky, but I didn't know it had a CPU peg problem too.  Anyone else able to reproduce this problem?  We had this feature disabled on Linux before, but then it seemed better to have the imperfect UI than to lose the feature entirely.  I'm thinking we should disable it again on Linux.

[knightmb]

After it initially tries incorrectly to set itself to the lowest priority, the generate thread only changes its priority again temporarily when it finds a block.  When you've found a block, you should want it to hurry up and broadcast it as soon a possible before someone else finds one and makes yours invalid.  The generate thread only changes to higher priority for less than a second every few days.

There should be a 0.3.1 release for this soon.  There are a few other issues we need to look at fixing in 0.3.1 before making a release.

On a side note, I've tracked down the other GUI issue.

The "minimize to tray instead of taskbar" is what was eating up all the CPU on my system. After I turned this off, the issue was resolved with Runaway CPU.

This only seems to affect the 64 bit Client, as the 32 bit Clients I have don't seem to be affected by this.

I did notice on the 64 bit Client, what happens is, it spawns multiple "tray" icons until X server finally kills over, so I guess I should submit that as a bug to somewhere? 

That's interesting.  I know the minimize to tray on Ubuntu is very clunky, but I didn't know it had a CPU peg problem too.  Anyone else able to reproduce this problem?  We had this feature disabled on Linux before, but then it seemed better to have the imperfect UI than to lose the feature entirely.  I'm thinking we should disable it again on Linux.

It only seems to happen if
A) you minimize to tray, later bring it back up, minimize again, bring it back up. It starts stacking tray icons out (but they are blank, but still labeled as BitCoin)
B) sometimes the program does it randomly

I'll turn it back on with mine to make a screen shot, doesn't take long for it to happen, hehe.

[knightmb]

Ok, easy to replicate (well for me anyway)

64bit Client Linux

Having either the "minimize to tray instead of taskbar" or "minimize on close" options checked will cause the bug.

First thing you notice (as per the first screen-shot), the tasktray is being filled up with invisible icons "spaces". If you hover the mouse over them, they all say "Bitcoin (not connected" but the one that does have an Bit Coin icon, just says "Bitcoin" and that's what you can bring the window back up with.

If you just have "minimize on close" checked, it still sends it to task tray causing the bugs.

After a while, all your CPU will be consumed by the X server, even after you close Bitcoin, I think some process stay stuck in memory, not sure.

The 32bit Client for Linux doesn't have this exact issue. I will still make at least one invisible Bitcoin icon, but doesn't tear up the CPU later. So the 32 bit client sort-a has the bug, but not in a way to bring the system down after a short while.