wallet migration problem

[nklak]

Hi, I have strange issue so I need help to solve situation.

I use ubuntu 18.04., 8 cores 24GB RAM.

I had old node (v0.16.3) with wallet version 159900, and about 100 accounts/addresses in wallet.dat file and I have daily backups of wallet.dat.  I also deployed new node (v17.1) and fully synced, then stopped old node, transferred wallet.dat to new node and started new node. It started fine, everything is working as it should because I did this many times before with both btc and other coins.
But everything was not fine. First symptom is that bitcoind with copied wallet.dat is causing high CPU usage (100%) , bitcoin-cli commands are very slow to return balance or list of accounts, and after a while (5-15 minutes) it will crash. Then I reverted to old same version (v0.16.3) and recopied original wallet.dat. Same issue again. With newly created wallet (empty) it is working fine, so there is definitevely some issue with old wallet file itself. I also tried various options as full resync and salvagewallet, but issue persisted. I also opted for another approach, I dumped private keys from original wallet, and imported into fresh new wallet and did resync. Now, I have full balance, ntc is working normaly (low cpu usage and no crashes), but now all balance is on default account ("") and not in separate accounts / addresses as it was originaly. I have option to manualy transfer proper values to each account (with move) but this is last resort option. I would like to get opinion or possible solution for this issue. Db.log do not show any errors or warnings when starting or while working, but clearly bitcoind has issue with my original wallet.dat file. As I mentioned I was using this process many time before (copying to other node or restoring from backup) with zero issues but this time it is hard to catch the cause of problem and to find solution.

Thanks in advance.

[achow101]

When it crashes, does it crash with an error? If so, what is that error?

Please post the contents of the debug.log file.

[BitMaxz]

Have you tried to restart the PC/Pi after you installed the v17.1?

If not yet try it first. Then run the node again with -rescan. Let see if it won't crash again.

[nklak]

It is getting more interesting. Issue is not with wallet but with network connection (rpc). For node control I am tunneling rpc calls via ssh. Now, when node is disconnected from tunnel it is started and working fine. As soon as I establish ssh tunnel bitcoind is shooting to 100% cpu usage and it crashes soon. It goes like this

when conenction is established this is first error
socket recv error Connection reset by peer (104)
node still works (including remote RPC via tunnel) but it is getting really slow in responding

after a while (about 10 minutes) it is unreachable via RPC or CLI  with followng error (multiple entries)

WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting

when closing bitcoind there is an error at shutdown

~HTTPRequest: Unhandled request

So At least I found cause (but still do not know what can be a problem). I have many nodes tunneling on this server, but issues is only with this. Currently I am setting up new node on another separate server (and network) for test.

[Deathwing]

It is getting more interesting. Issue is not with wallet but with network connection (rpc). For node control I am tunneling rpc calls via ssh. Now, when node is disconnected from tunnel it is started and working fine. As soon as I establish ssh tunnel bitcoind is shooting to 100% cpu usage and it crashes soon. It goes like this

when conenction is established this is first error
socket recv error Connection reset by peer (104)
node still works (including remote RPC via tunnel) but it is getting really slow in responding

after a while (about 10 minutes) it is unreachable via RPC or CLI  with followng error (multiple entries)

WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting

when closing bitcoind there is an error at shutdown

~HTTPRequest: Unhandled request

So At least I found cause (but still do not know what can be a problem). I have many nodes tunneling on this server, but issues is only with this. Currently I am setting up new node on another separate server (and network) for test.

Considering that the CPU usage is hitting 100% when you are using SSH Tunnel. It might be due to the tunnel itself. When you count in Encryption and such with SSH Tunneling, it might be putting way too much load on the CPU, therefore, starting to push out errors. While this is just a hypothesis, connecting to several nodes and tunnelling them through SSH is probably what is causing this issue. Due to network congestion/CPU overload it just fails.

[nc50lc]

WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting

You can also consider the log's suggestion.

Add rpcworkqueue=64 to your data directory's bitcoin.conf file, create if your don't have one.
Or add it as a start parameter.

[nklak]

Already did that but it just make it last longer before crash.

Anyway, I was up all night and made different approach. I deployed new node to other VM on separate network and have it synced in about 14h. Old wallet works perfectly, and everything is as it should be.
I would like to thank to the community for giving constructive suggestions. I think of my self as expert but there is always some new problem to tackle with.

Thank guys.

best regards