Bitcoin Forum
November 19, 2017, 09:14:36 AM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: Bitcoin Node CPU Spikes  (Read 648 times)
udevNull
Newbie
*
Offline Offline

Activity: 10


View Profile
March 01, 2017, 05:10:11 AM
 #1

Hey guys.

Quick question regarding running a node, not sure if this place to ask or if https://bitcointalk.org/index.php?board=6.0 would be better. Please let me know.

I'm monitoring CPU usage of the node using grafana which gets it's data, ultimately, from collectd. I'm running 0.13.2.
My question is regarding random CPU spikes that occur. Any ideas what could be causing that?
They can last for up to 10 minutes (often only spike for a minute or two) and reach around 70% - 80% CPU usage. I've been seeing this for a while now (the last 3/4 months of active monitoring) on different versions of bitcoin core and on different machines (different hardware, same OS, Ubuntu 16.04 LTS). Is this some kind of a clean up process that runs or is it something else?

Thanks guys.
1511082876
Hero Member
*
Offline Offline

Posts: 1511082876

View Profile Personal Message (Offline)

Ignore
1511082876
Reply with quote  #2

1511082876
Report to moderator
1511082876
Hero Member
*
Offline Offline

Posts: 1511082876

View Profile Personal Message (Offline)

Ignore
1511082876
Reply with quote  #2

1511082876
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1511082876
Hero Member
*
Offline Offline

Posts: 1511082876

View Profile Personal Message (Offline)

Ignore
1511082876
Reply with quote  #2

1511082876
Report to moderator
1511082876
Hero Member
*
Offline Offline

Posts: 1511082876

View Profile Personal Message (Offline)

Ignore
1511082876
Reply with quote  #2

1511082876
Report to moderator
achow101
Moderator
Legendary
*
Offline Offline

Activity: 1218


17kKQppUsngUiByDsce4JXoZEjjpvX9bpR


View Profile WWW
March 01, 2017, 05:13:22 AM
 #2

It could be when there are spikes in transactions that need verification. Could also be new blocks coming in that need to be checked.

          ▄█████▄
        ▄█████████▄
      ▄████▀   ▀████▄
    ▄████▀   ▄ ▄█▀████▄
  ▄████▀   ▄███▀   ▀████▄
▄████▀   ▄███▀   ▄   ▀████▄
█████   ███▀   ▄███   █████
▀████▄   ▀██▄▄███▀   ▄████▀
  ▀████▄   ▀███▀   ▄████▀
    ▀████▄       ▄████▀
      ▀████▄   ▄████▀
        ▀███  ████▀
          ▀█▄███▀
.
|
.
|
          ▄█████▄
        ▄█████████▄
      ▄████▀   ▀████▄
    ▄████▀   ▄ ▄█▀████▄
  ▄████▀   ▄███▀   ▀████▄
▄████▀   ▄███▀   ▄   ▀████▄
█████   ███▀   ▄███   █████
▀████▄   ▀██▄▄███▀   ▄████▀
  ▀████▄   ▀███▀   ▄████▀
    ▀████▄       ▄████▀
      ▀████▄   ▄████▀
        ▀███  ████▀
          ▀█▄███▀
unthy
udevNull
Newbie
*
Offline Offline

Activity: 10


View Profile
March 01, 2017, 05:31:12 AM
 #3

Yeah, I checked that first as well but they don't correspond with new blocks. i.e. last night I had a spike between 23:16 - 23:25 (@72% CPU Usage). The next spike occured at 00:13 (@63% CPU Usage). The next one at 01:20 (@81% CPU Usage).

Using collectd, I've been able to narrow it down specifically to the bitcoind process. If it were related to blocks, it would occur far more frequently. The server is also a C4 aws instance. So it's not like it's lacking in terms of CPU power.
-ck
Staff
Legendary
*
Offline Offline

Activity: 2338


Ruu \o/


View Profile WWW
March 01, 2017, 05:38:15 AM
 #4

You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing.

Primary developer/maintainer for cgminer and ckpool/ckproxy.
ZERO FEE Pooled mining at ckpool.org 1% Fee Solo mining at solo.ckpool.org
-ck
udevNull
Newbie
*
Offline Offline

Activity: 10


View Profile
March 01, 2017, 05:41:06 AM
 #5

You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing.

Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes?
shorena
Legendary
*
Offline Offline

Activity: 1400


ALL escrow is signed! https://keybase.io/verify


View Profile WWW
March 01, 2017, 07:57:06 AM
 #6

You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing.

Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes?

Yeah, not that long though. Turned out my node was connected to a service that was pushing several thousand transactions to it every X minutes. If you have a network spike (from a few kbit/s to several mbit/s) at the same times that might be it.

udevNull
Newbie
*
Offline Offline

Activity: 10


View Profile
March 01, 2017, 08:49:53 AM
 #7

You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing.

Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes?

Yeah, not that long though. Turned out my node was connected to a service that was pushing several thousand transactions to it every X minutes. If you have a network spike (from a few kbit/s to several mbit/s) at the same times that might be it.

That's quite interesting, how long did your spikes last and how did you pick that up?
HI-TEC99
Legendary
*
Offline Offline

Activity: 1050



View Profile
March 01, 2017, 10:28:07 AM
 #8

You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing.

Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes?

Yeah, not that long though. Turned out my node was connected to a service that was pushing several thousand transactions to it every X minutes. If you have a network spike (from a few kbit/s to several mbit/s) at the same times that might be it.


A redditer was complaining about high node CPU usage a year ago. He identified a high number of connections correlating with high CPU usage, and speculated it might be due to suboptimally configured nodes pushing dust and spam transactions. His error log was full of nonstandard transaction/dust errors.

UdevNull's node's error log might contain the same errors if he's experiencing the same problem.

https://www.reddit.com/r/Bitcoin/comments/3xp22r/fully_synced_bitcoin_core_is_using_50_cpu_is/cy6l0oi/

Quote

I'm running bitcoind on several different servers, some of them are reachable via a public IP, other aren't.

With public IP:
version v0.11.0, minrelaytxfee=0.00004
CPU fluctuates between 20% and 50%
113 connections
37135 transactions in mempool, 463 MB total

I see a lot of ERROR messages in debug.log
.
2015-12-21 13:51:16 ERROR: AcceptToMemoryPool: free transaction rejected by rate limiter
2015-12-21 13:51:16 ERROR: AcceptToMemoryPool: nonstandard transaction: dust

Without public IP:
version v0.11.1, no config tweaks
CPU usage often stays in 0-2% range,
9 connections
35766 transactions in mempool, 428 MB

very few ERROR messages in debug.log

So it looks like CPU usage correlates with the number of connections. I'm not sure if there is any specific cause (possible causes: old/misconfigured nodes pushing dust & spam transactions; thin clients catching up with the blockchain using bloom queries), or just the sheer number of connections.

You can restrict the number of connection with maxconnections config parameter.
shorena
Legendary
*
Offline Offline

Activity: 1400


ALL escrow is signed! https://keybase.io/verify


View Profile WWW
March 06, 2017, 10:35:42 AM
 #9

You're on linux so bring up 'top' while it's happening, press H to enable thread mode, and see what thread(s) is/are consuming CPU at the time. The bitcoin daemon has lots of threads that are named so you may be able to track which particular thread is consuming CPU and hence figure out what it's doing.

Right, okay. I can give that a shot. So nobody else has experienced similar CPU spikes?

Yeah, not that long though. Turned out my node was connected to a service that was pushing several thousand transactions to it every X minutes. If you have a network spike (from a few kbit/s to several mbit/s) at the same times that might be it.

That's quite interesting, how long did your spikes last and how did you pick that up?

Just a few seconds. I found the IP in question using tcptrack. The node was just regular XT node, nothing special about it. With nmap I found they had ports 80 and 443 (among others) open so I just entered the IP in a browser and it redirected me to their main HP.



If you want to check the debug.log for a possible origin node as HI-TEC99 suggested make sure you have logips=1 set

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!