bitcoin core 0.16.3: high resource usage

[ajeeshb]

One week before I was running 0.12.0 version is running and now it is upgraded to 0.16.3 and found to be high CPU and disk IO consuming for real bitcoin network. But for the Testnet it is running normal resource usage.

The real bitcoin network is running with the following options
  exec start-stop-daemon \
    --start \
    --pidfile "$BITCOIND_PIDFILE" \
    --chuid $BITCOIND_USER:$BITCOIND_GROUP \
    --exec "$BITCOIND_BIN" \
    -- \
    -pid="$BITCOIND_PIDFILE" \
    -conf="$BITCOIND_CONFIGFILE" \
    -datadir="$BITCOIND_DATADIR" \
    -daemon \
    -debug=prune

And Testnet is with the following option

-daemon -testnet -reindex

[ajeeshb]

After adding the flag -reindex  now the memory hike is reduced to normal usage, but cpu is still showing high.

[ajeeshb]

debug.log

2018-11-13 11:49:25 Reindexing block file blk00695.dat...
2018-11-13 11:49:31 Loaded 148 blocks from external file in 5161ms

System spec
8 GB RAM
cpu 2 core [Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz]

Reindexing is still running and it already takes 1.30 hours now, How much time normally it will take to complete the reindexing?

[bob123]

High ressource usage while reindexing is completely normal.
 
Core is verifying each block currently. This results in high CPU/HDD usage.

If you are running the GUI, you should be able to see how long it takes at the bottom bar.

According to your log, you are currently at blk00695.dat.
Without the GUI, you can check how much blk**.dat files there are to estimate the time it still needs to index.

[ajeeshb]

But after some time I am getting the following error message.

2018-11-13 12:11:21 ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness nonce size (code 16)

[ajeeshb]

After some time I got the below message

2018-11-13 12:43:38 ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness nonce size (code 16)
2018-11-13 12:43:38 ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness nonce size (code 16)
2018-11-13 12:43:38 ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness nonce size (code 16)
2018-11-13 12:43:38 Reindexing finished
2018-11-13 12:43:38 Pre-allocating up to position 0x100000 in rev00000.dat


2018-11-13 13:34:09 UpdateTip: new best=00xxxx0 height=304948 version=0xxx2 log2_work=79.070092 tx=40xx99 date='2014-06-09 13:40:16' progress=0.105260 cache=437.5MiB(3xx3txo)

Is it normal behavior, can you please put some light on this?

[jackg]

After some time I got the below message

2018-11-13 12:43:38 ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness nonce size (code 16)
2018-11-13 12:43:38 ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness nonce size (code 16)
2018-11-13 12:43:38 ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness nonce size (code 16)
2018-11-13 12:43:38 Reindexing finished
2018-11-13 12:43:38 Pre-allocating up to position 0x100000 in rev00000.dat


2018-11-13 13:34:09 UpdateTip: new best=00xxxx0 height=304948 version=0xxx2 log2_work=79.070092 tx=40xx99 date='2014-06-09 13:40:16' progress=0.105260 cache=437.5MiB(3xx3txo)

Is it normal behavior, can you please put some light on this?

I think the error is the nodes broadcasting to yours and not your nose itself. The issue resolved as it gets an updatetip so it accepts the new block. Unless this has caused some sort of an infinite loop with the same update statement being printed to the log (with the same height)...

[ajeeshb]

I still see the UpdateTip logs for the last 3 hours and it is still running.

UpdateTip: new best=0xx height=416621 version=0x20000001 log2_work=84.xxx tx=13xx date='2016-06-16 22:42:30' progress=0.355548 cache=396.4MiB(xtxo)

What this really doing?

During this update, my CPU load should be high?

[jackg]

I still see the UpdateTip logs for the last 3 hours and it is still running.

UpdateTip: new best=0xx height=416621 version=0x20000001 log2_work=84.xxx tx=13xx date='2016-06-16 22:42:30' progress=0.355548 cache=396.4MiB(xtxo)

What this really doing?

During this update, my CPU load should be high?

Yes. From as far as i know. The update tip means you’re validating and downloading a block.
The height of that block is 416621, meaning you have some 100000-200000 to go so you’re.a substantial way through the blockchain but not all the waay there
The processes get more intense as you go through and try to download an even fuller block (you have that to look forward to ).

When you download a block, you have to check the hash of the block matches the nonce and also that it matches the data that is given by the miner.

[ajeeshb]

After validating and downloading the block now my server is stable as before when i run 0.12.0 version. But now i can see the following message in my debug.log

2018-11-14 11:43:26 Potential stale tip detected, will try using extra outbound peer (last tip update: 2679 seconds ago)
2018-11-14 11:43:26 New outbound peer connected: version: 70015, blocks=550051, peer=170
2018-11-14 11:51:05 connect() to 92.117.120.207:8333 failed after select(): No route to host (113)

I can see lots of :8333 failed after select(): with Network is unreachable,Connection refused,No route to host in the log.

[jackg]

Can you try updating the software, maybe you’re blocked due to the recent vulnerability by some nodes.

[ajeeshb]

But i am already synced then how it is blocking?

[jackg]

But i am already synced then how it is blocking?

The debug log claims you aren’t synced.

When you open core how does the screen look, can you take a screenshot an upload it (using a sit such as imgur).

[Lauda]

But i am already synced then how it is blocking?

This is what you're looking for:


A -reindex will not just cause "high resource usage" it will try to use ALL available resources (maxing out CPU and/or I/O depending on which one is the constraint, and using as much memory as you've allocated for it). I once tried it on a VPS out of curiosity, the complete thing was practically inaccessible until a (pruned) sync was complete.