|
Title: Bitcoin Node CPU Spikes Post by: udevNull on March 01, 2017, 05:10:11 AM Hey guys.
Quick question regarding running a node, not sure if this place to ask or if https://bitcointalk.org/index.php?board=6.0 would be better. Please let me know. I'm monitoring CPU usage of the node using grafana which gets it's data, ultimately, from collectd. I'm running 0.13.2. My question is regarding random CPU spikes that occur. Any ideas what could be causing that? They can last for up to 10 minutes (often only spike for a minute or two) and reach around 70% - 80% CPU usage. I've been seeing this for a while now (the last 3/4 months of active monitoring) on different versions of bitcoin core and on different machines (different hardware, same OS, Ubuntu 16.04 LTS). Is this some kind of a clean up process that runs or is it something else? Thanks guys. Title: Re: Bitcoin Node CPU Spikes Post by: achow101 on March 01, 2017, 05:13:22 AM It could be when there are spikes in transactions that need verification. Could also be new blocks coming in that need to be checked.
Title: Re: Bitcoin Node CPU Spikes Post by: udevNull on March 01, 2017, 05:31:12 AM Yeah, I checked that first as well but they don't correspond with new blocks. i.e. last night I had a spike between 23:16 - 23:25 (@72% CPU Usage). The next spike occured at 00:13 (@63% CPU Usage). The next one at 01:20 (@81% CPU Usage).
Using collectd, I've been able to narrow it down specifically to the bitcoind process. If it were related to blocks, it would occur far more frequently. The server is also a C4 aws instance. So it's not like it's lacking in terms of CPU power. Title: Re: Bitcoin Node CPU Spikes Post by: -ck on March 01, 2017, 05:38:15 AM You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing.
Title: Re: Bitcoin Node CPU Spikes Post by: udevNull on March 01, 2017, 05:41:06 AM You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing. Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes? Title: Re: Bitcoin Node CPU Spikes Post by: shorena on March 01, 2017, 07:57:06 AM You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing. Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes? Yeah, not that long though. Turned out my node was connected to a service that was pushing several thousand transactions to it every X minutes. If you have a network spike (from a few kbit/s to several mbit/s) at the same times that might be it. Title: Re: Bitcoin Node CPU Spikes Post by: udevNull on March 01, 2017, 08:49:53 AM You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing. Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes? Yeah, not that long though. Turned out my node was connected to a service that was pushing several thousand transactions to it every X minutes. If you have a network spike (from a few kbit/s to several mbit/s) at the same times that might be it. That's quite interesting, how long did your spikes last and how did you pick that up? Title: Re: Bitcoin Node CPU Spikes Post by: HI-TEC99 on March 01, 2017, 10:28:07 AM You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing. Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes? Yeah, not that long though. Turned out my node was connected to a service that was pushing several thousand transactions to it every X minutes. If you have a network spike (from a few kbit/s to several mbit/s) at the same times that might be it. A redditer was complaining about high node CPU usage a year ago. He identified a high number of connections correlating with high CPU usage, and speculated it might be due to suboptimally configured nodes pushing dust and spam transactions. His error log was full of nonstandard transaction/dust errors. UdevNull's node's error log might contain the same errors if he's experiencing the same problem. https://www.reddit.com/r/Bitcoin/comments/3xp22r/fully_synced_bitcoin_core_is_using_50_cpu_is/cy6l0oi/ Quote I'm running bitcoind on several different servers, some of them are reachable via a public IP, other aren't. With public IP: version v0.11.0, minrelaytxfee=0.00004 CPU fluctuates between 20% and 50% 113 connections 37135 transactions in mempool, 463 MB total I see a lot of ERROR messages in debug.log . 2015-12-21 13:51:16 ERROR: AcceptToMemoryPool: free transaction rejected by rate limiter 2015-12-21 13:51:16 ERROR: AcceptToMemoryPool: nonstandard transaction: dust Without public IP: version v0.11.1, no config tweaks CPU usage often stays in 0-2% range, 9 connections 35766 transactions in mempool, 428 MB very few ERROR messages in debug.log So it looks like CPU usage correlates with the number of connections. I'm not sure if there is any specific cause (possible causes: old/misconfigured nodes pushing dust & spam transactions; thin clients catching up with the blockchain using bloom queries), or just the sheer number of connections. You can restrict the number of connection with maxconnections config parameter. Title: Re: Bitcoin Node CPU Spikes Post by: shorena on March 06, 2017, 10:35:42 AM You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing. Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes? Yeah, not that long though. Turned out my node was connected to a service that was pushing several thousand transactions to it every X minutes. If you have a network spike (from a few kbit/s to several mbit/s) at the same times that might be it. That's quite interesting, how long did your spikes last and how did you pick that up? Just a few seconds. I found the IP in question using tcptrack. The node was just regular XT node, nothing special about it. With nmap I found they had ports 80 and 443 (among others) open so I just entered the IP in a browser and it redirected me to their main HP. If you want to check the debug.log for a possible origin node as HI-TEC99 suggested make sure you have logips=1 set |