bitcoin 0.62, 100% CPU load and stops working, possible DOS attack.

[sergio]

Hi I am having an strange problem since about 3AM today.

I have been running the bitcoind 0.62 just fine for several weeks, and before that 0.53.

However recently the bitcoind started having a 100%  load and it stops working, if I kill  the daemon, the stop command will not work when at 100% load, and restart the daemon a few minutes later after killing it. it will work fine for a very short time, and then the load will be 100% again.

Since it was working fine for a long time, and it works fine when I run it at home, I am thinking of maybe a possible DOS attack.

Any recommendations on what the problem could be, or what to do.

The problem also happens with version 0.53 which used to work fine.

[Stephen Gornick]

Operating system?

Debug.log?

[Diapolo]

Can you please try 0.6.3.RC1 and report back.

Dia

[sergio]

OS Centos 5.5 but compiled statically on Debian 6.0 (will not compile on Centos 5.5).

I am starting clean, backed up everything, and starting downloading a fresh block chain.

I am will give bitcoind 0.63RC1 a try and report back.

The debug.log before starting clean was like this, this is the end of the log, the log file is huge over 82000 lines long:
Not sure to what version this log belongs to since, I tested 3 version of bitcoind 0.53, 0,62, and 0.63RC1, after I started having this problem.

06/23/12 02:26:37 received: tx (257 bytes)
ERROR: ConnectInputs() : c33c410c97 mapTransactions prev not found 857aaeb44f
ERROR: AcceptToMemoryPool() : ConnectInputs failed c33c410c97
06/23/12 02:26:38 sending: addr (31 bytes)
ResendWalletTransactions()
06/23/12 02:26:39 received: tx (260 bytes)
ERROR: ConnectInputs() : cba242ade5 mapTransactions prev not found 3806d647e5
ERROR: AcceptToMemoryPool() : ConnectInputs failed cba242ade5
06/23/12 02:26:39 received: tx (258 bytes)
ERROR: ConnectInputs() : 1d6d14002c mapTransactions prev not found 69901b2d0d
ERROR: AcceptToMemoryPool() : ConnectInputs failed 1d6d14002c
06/23/12 02:26:39 received: tx (258 bytes)
ERROR: ConnectInputs() : 191c9787c0 mapTransactions prev not found a68b23daec
ERROR: AcceptToMemoryPool() : ConnectInputs failed 191c9787c0
06/23/12 02:26:39 received: tx (438 bytes)
ERROR: ConnectInputs() : 45cd10b8de mapTransactions prev not found e6af7b55bb
ERROR: AcceptToMemoryPool() : ConnectInputs failed 45cd10b8de
06/23/12 02:26:39 received: tx (258 bytes)
Rate limit dFreeCount: 5758.21 => 6016.21

It looks like like a network problem, but the network is fine.

bitcoind uses a huge CPU load, and becomes unresponsive.

The very strange thing , I tested version 0.53 and same problem was happening.
a while back I was using 0.53, and then 0.62 without problems, and then about a day ago suddenly I start having problems.

In a few hours I will report back, after a fresh  block chain has been downloaded, and I will test 0.62. and 0.63RC1.

[sergio]

update.
After a fresh block chain and fresh new wallter bitcoind 0.63RC1 seems to run fine, however as soon as I load my old wallet I get a db error:

# bitcoind -server -daemon
bitcoin server starting
[root@vpn3000 .bitcoin]# terminate called after throwing an instance of 'DbRunRecoveryException'
  what():  DbEnv::txn_checkpoint: DB_RUNRECOVERY: Fatal error, run database recovery

and on the debug log:
Bitcoin version v0.6.3.0-g6e0c5e3-beta ()
Default data directory /root/.bitcoin
Loading addresses...
dbenv.open LogDir=/root/.bitcoin/database ErrorFile=/root/.bitcoin/db.log
Loaded 10520 addresses
 addresses               101ms
Loading block index...
LoadBlockIndex(): hashBestChain=0000000000000106b7c2  height=185975
Verifying last 2500 blocks at level 1
 block index           13996ms
Loading wallet...

The error recommends to " run database recovery", how does one run database recovery on bitcoind.
The block chain is fresh, recently downloaded, it seems that is the wallet causing the problem.
If I use the new empty wallet everything works fine.

Note: Before I downloaded the fresh block chain, I was not getting an error, I  was getting a 100% load on bitcoind
and bitcoind was unresponsive, so downloading a fresh blockchain helped somehow.

How can the wallet get corrupted?

After testing with bitcoind 0.62 I have the same error.


here is the debug.log with 0.62
Bitcoin version vCLIENT_VERSION_MAJOR.CLIENT_VERSION_MINOR.CLIENT_VERSION_REVISION.CLIENT_VERSI ON_BUILD-g8ff1873-beta ()
Default data directory /root/.bitcoin
Loading addresses...
dbenv.open LogDir=/root/.bitcoin/database ErrorFile=/root/.bitcoin/db.log
Loaded 11592 addresses
 addresses               111ms
Loading block index...
LoadBlockIndex(): hashBestChain=0000000000000932bdb3  height=185978
Verifying last 2500 blocks at level 1
 block index           14024ms
Loading wallet...

and here is the exact error with 0.62:
# bitcoind -server -daemon
bitcoin server starting
[root@vpn3000 sbin]# terminate called after throwing an instance of 'DbRunRecoveryException'
  what():  DbEnv::txn_checkpoint: DB_RUNRECOVERY: Fatal error, run database recovery


Any recommendations on how to repair the wallet?

[Stephen Gornick]

Any recommendations on how to repair the wallet?

There's a report as a response to this:
 - http://bitcoin.stackexchange.com/a/3538/153

which said to remove everything except wallet.dat  and to restart the client.

(of course, make sure you have backups of your wallet.dat first)

You might try this with the blockchain (blk*.dat) there two, ... so it starts with just those three files.

[sergio]

Thanks for the link it helped solve the problem, one of the comments was to use an older bitcoin client which it worked, the other comment to use a clean block chain which works only with the older bitcoin client which the newer bitcoin client there is the error about the db.

Bitcoind 0.53 reads the wallet fine and it is now working fine.

It must have been a change in the way 0.62 reads the wallet most likely a bug.
0.63RC1 has the same problem.

At this point I have 2 options, transfer the bitcoins to a new wallet and then use 0.63RC1 or 0.62 with a new wallet or continue to use 0.53 which seems to work fine with the existing wallet.

Thanks, this problem has been solved.

[Diapolo]

Thanks for the link it helped solve the problem, one of the comments was to use an older bitcoin client which it worked, the other comment to use a clean block chain which works only with the older bitcoin client which the newer bitcoin client there is the error about the db.

Bitcoind 0.53 reads the wallet fine and it is now working fine.

It must have been a change in the way 0.62 reads the wallet most likely a bug.
0.63RC1 has the same problem.

At this point I have 2 options, transfer the bitcoins to a new wallet and then use 0.63RC1 or 0.62 with a new wallet or continue to use 0.53 which seems to work fine with the existing wallet.

Thanks, this problem has been solved.

Can you try to start the client with -upgradewallet as parameter, does this help?

Dia

[Pieter Wuille]

Can you try to start the client with -upgradewallet as parameter, does this help?

Dia

That won't help.

My guess is that his wallet is actually corrupted, but old clients fail to notice it. 0.6.0 introduced more strict consistency checks for wallets at startup.

[sergio]

The command -upgradewallet fixed the problem it now works fine with 0.63RC1.

What is strange is I thought the wallet update automatically as this used to be the case for older bitcoin clients.

When I used the command -upgradewallet, I am no longer getting the db error.

What is very strange is that before things broke it was running fine on 0.62, then when I got a clean block chain is when I started to get the db error on new bitcoin versions,
but now that I ran -upgradewallet it fixed the db problem with the new bitcoin versions and everything appears to be running fine.

Thanks for the help, both the link provided, and the recommendation to run  -upgradewallet helped.