Node stuck on some commands...

[Jodmapht]

Lately I have been noticing that my node gets stuck for a very long time with some commands, e.g.:
bitcoin-cli getrpcinfo
{
  "active_commands": [
    {
      "method": "getmempoolentry",
      "duration": 6030622998
    }
]}

I also saw with with getrawtransaction. How can I diagnose what is going on exactly?
Thanks!

[nc50lc]

It only happens to me when my node is catching-up to the tip of the blockchain.
e.g.: when I just opened Bitcoin Core after a day/few hours of downtime.

If it's not the case, you can enable advance debugging with "rpc" flag, reproduce the issue and then consult your debug.log file for related entries.
e.g., start bitcoind with: bitcoind -debug=rpc

[Jodmapht]

It only happens to me when my node is catching-up to the tip of the blockchain.
e.g.: when I just opened Bitcoin Core after a day/few hours of downtime.

If it's not the case, you can enable advance debugging with "rpc" flag, reproduce the issue and then consult your debug.log file for related entries.
e.g., start bitcoind with: bitcoind -debug=rpc

Ok no it is not the case. Also I noticed that I cannot shutdown the process gracefully, it hangs forever in b-shutoff, I have to hard kill it, even if I have dbcache set to only 300. I will try rpc debugging.

[nc50lc]

-snip-
Also I noticed that I cannot shutdown the process gracefully, it hangs forever in b-shutoff, I have to hard kill it, even if I have dbcache set to only 300. I will try rpc debugging.

Frequently doing that "hard kill" might corrupt your blockchain.
If you've finished "Initial Block Download" (IBD) or already at the tip, then it may be a hardware-related issue, you can also check your debug.log for other errors.

[alecfisker]

post your config file
copy data folder to another node on same network, make sure memory drive space enough
try prune mode usually helps

[Jodmapht]

-snip-
Also I noticed that I cannot shutdown the process gracefully, it hangs forever in b-shutoff, I have to hard kill it, even if I have dbcache set to only 300. I will try rpc debugging.

Frequently doing that "hard kill" might corrupt your blockchain.
If you've finished "Initial Block Download" (IBD) or already at the tip, then it may be a hardware-related issue, you can also check your debug.log for other errors.

I had this node running for about 1.5 year. I don't see any error about my hardware in dmesg. I don't see any error in debug.log. Here is my config:

bitcoin.conf:

Code:

server=1
prune=0
disablewallet=0
txindex=1
blockfilterindex=1
proxy=127.0.0.1:9050
onion=127.0.0.1:9050
proxyrandomize=1
listen=1
listenonion=1

rpcallowip=127.0.0.1

shrinkdebugfile=1

# is required by Fail2Ban described below
logips=1

# magic RBP optimisations
maxconnections=32
whitelist=127.0.0.1
maxuploadtarget=5000

dbcache=300
maxorphantx=10
maxmempool=50

#For btcrpcexplorer
rpcworkqueue=100

#Disable DNS to prevent DNS deanonymization
dnsseed=0
dns=0

#Add seed nodes
seednode=wxvp2d4rspn7tqyu.onion
seednode=bk5ejfe56xakvtkk.onion
seednode=bpdlwholl7rnkrkw.onion
seednode=hhiv5pnxenvbf4am.onion
seednode=4iuf2zac6aq3ndrb.onion
seednode=nkf5e6b7pl4jfd4a.onion
seednode=xqzfakpeuvrobvpj.onion
seednode=tsyvzsqwa2kkf6b2.onion

includeconf=../etc/bitcoin_main.conf

includeconf=../etc/bitcoin_testnet.conf

bitcoin_main.conf:

Code:

[main]
onlynet=onion

bind=127.0.0.1:8333
rpcbind=127.0.0.1
rpccookiefile=/opt/local/bitcoind/var/cookie

zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333

whitelist = bloomfilter@127.0.0.1

#And/or add some nodes
addnode=gyn2vguc35viks2b.onion
addnode=kvd44sw7skb5folw.onion
addnode=nkf5e6b7pl4jfd4a.onion
addnode=yu7sezmixhmyljn4.onion
addnode=3ffk7iumtx3cegbi.onion
addnode=3nmbbakinewlgdln.onion
addnode=4j77gihpokxu2kj4.onion
addnode=546esc6botbjfbxb.onion
addnode=5at7sq5nm76xijkd.onion
addnode=77mx2jsxaoyesz2p.onion
addnode=7g7j54btiaxhtsiy.onion
addnode=a6obdgzn67l7exu3.onion
addnode=ab64h7olpl7qpxci.onion
addnode=am2a4rahltfuxz6l.onion
addnode=azuxls4ihrr2mep7.onion
addnode=bitcoin7bi4op7wb.onion
addnode=bitcoinostk4e4re.onion
addnode=bk7yp6epnmcllq72.onion
addnode=bmutjfrj5btseddb.onion
addnode=ceeji4qpfs3ms3zc.onion
addnode=clexmzqio7yhdao4.onion
addnode=gb5ypqt63du3wfhn.onion
addnode=h2vlpudzphzqxutd.onion
addnode=n42h7r6oumcfsbrs.onion:4176
addnode=ncwk3lutemffcpc4.onion
addnode=okdzjarwekbshnof.onion
addnode=pjghcivzkoersesd.onion
addnode=rw7ocjltix26mefn.onion
addnode=uws7itep7o3yinxo.onion
addnode=vk3qjdehyy4dwcxw.onion
addnode=vqpye2k5rcqvj5mq.onion
addnode=wpi7rpvhnndl52ee.onion

[Jodmapht]

I left my node running overnight without bisq and I did not experience an issue with CPU. Could the issue be related to the usage of bloom filters by bisq? I upgraded Bitcoin core, started using CLN and Bisq at around the same time and this is when the issues I mentioned started. I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!

[nc50lc]

I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!

Possible, since RPC works although slow, the issue may be your hardware can't handle all those processes at the same time.
(based from the dbcache, RPi?)

I'm not using Bisq so I can't help any further but I've read that it has high CPU usage during the sync process.

[Jodmapht]

I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!

Possible, since RPC works although slow, the issue may be your hardware can't handle all those processes at the same time.
(based from the dbcache, RPi?)

I'm not using Bisq so I can't help any further but I've read that it has high CPU usage during the sync process.

Yes during Bisq's sync process, there is heavy load on my node, but then it goes away once sync is complete after a minute or so. The stuck transaction occurs typically hours after Bisq's launch. Yesterday it occurred about 12h after launch.
Yes the hardware is a RPI4 with 8 GB of RAM.

bitcoin_main.conf:

Code:

[main]
onlynet=onion

bind=127.0.0.1:8333
rpcbind=127.0.0.1
rpccookiefile=/opt/local/bitcoind/var/cookie

zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333

whitelist = bloomfilter@127.0.0.1

#And/or add some nodes
addnode=gyn2vguc35viks2b.onion
addnode=kvd44sw7skb5folw.onion
addnode=nkf5e6b7pl4jfd4a.onion
addnode=yu7sezmixhmyljn4.onion
addnode=3ffk7iumtx3cegbi.onion
addnode=3nmbbakinewlgdln.onion
addnode=4j77gihpokxu2kj4.onion
addnode=546esc6botbjfbxb.onion
addnode=5at7sq5nm76xijkd.onion
addnode=77mx2jsxaoyesz2p.onion
addnode=7g7j54btiaxhtsiy.onion
addnode=a6obdgzn67l7exu3.onion
addnode=ab64h7olpl7qpxci.onion
addnode=am2a4rahltfuxz6l.onion
addnode=azuxls4ihrr2mep7.onion
addnode=bitcoin7bi4op7wb.onion
addnode=bitcoinostk4e4re.onion
addnode=bk7yp6epnmcllq72.onion
addnode=bmutjfrj5btseddb.onion
addnode=ceeji4qpfs3ms3zc.onion
addnode=clexmzqio7yhdao4.onion
addnode=gb5ypqt63du3wfhn.onion
addnode=h2vlpudzphzqxutd.onion
addnode=n42h7r6oumcfsbrs.onion:4176
addnode=ncwk3lutemffcpc4.onion
addnode=okdzjarwekbshnof.onion
addnode=pjghcivzkoersesd.onion
addnode=rw7ocjltix26mefn.onion
addnode=uws7itep7o3yinxo.onion
addnode=vk3qjdehyy4dwcxw.onion
addnode=vqpye2k5rcqvj5mq.onion
addnode=wpi7rpvhnndl52ee.onion

Slightly off-topic, but you need to know .onion link V2 is dead[1] where Tor[1] and Bitcoin Core[2] already remove the support about a year ago. I would recommend you to update your configuration file with Bitcoin node which use .onion link V3 instead. If you don't know which node you should add, you can check list of node[3] on Bitcoin Core repository as starter.

[1] https://support.torproject.org/onionservices/v2-deprecation/
[2] https://github.com/bitcoin/bitcoin/pull/22050
[3] https://github.com/bitcoin/bitcoin/blob/master/contrib/seeds/nodes_main.txt#L781-L792

Thanks for that, I will fix it!

I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!

Possible, since RPC works although slow, the issue may be your hardware can't handle all those processes at the same time.
(based from the dbcache, RPi?)

I'm not using Bisq so I can't help any further but I've read that it has high CPU usage during the sync process.

Yes during Bisq's sync process, there is heavy load on my node, but then it goes away once sync is complete after a minute or so. The stuck transaction occurs typically hours after Bisq's launch. Yesterday it occurred about 12h after launch.
Yes the hardware is a RPI4 with 8 GB of RAM.

Is there a way to cancel a stuck RPC command without killing the the whole process?

[moderator's note: consecutive posts merged]

[NotATether]

Is there a way to cancel a stuck RPC command without killing the the whole process?

You have to sever the network connection between the RPC client and the daemon. Briefly toggling on/off Airplane mode or just physically unplugging and plugging back your Ethernet port should work.

Note - don't do this if the node and client are on the same machine, or you will cripple the node as well.

PS. txindex creates a lot of extra data to parse, which makes some RPC calls that deal with transactions slower.

[Jodmapht]

Is there a way to cancel a stuck RPC command without killing the the whole process?

You have to sever the network connection between the RPC client and the daemon. Briefly toggling on/off Airplane mode or just physically unplugging and plugging back your Ethernet port should work.

Note - don't do this if the node and client are on the same machine, or you will cripple the node as well.

PS. txindex creates a lot of extra data to parse, which makes some RPC calls that deal with transactions slower.

Ok I can try that then. I use a SSH tunnel between the two, so I guess I can exit from SSH and see if it does the trick.

Hmm, I think my understanding of txindex is not good. Isn't it required to search for transaction by txid, unless the transaction is part of a Bitcoin Core wallet? I don't think Bisq is using a Bitcoin Core wallet. Thanks!

[seoincorporation]

I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!

Possible, since RPC works although slow, the issue may be your hardware can't handle all those processes at the same time.
(based from the dbcache, RPi?)

I'm not using Bisq so I can't help any further but I've read that it has high CPU usage during the sync process.

Yes during Bisq's sync process, there is heavy load on my node, but then it goes away once sync is complete after a minute or so. The stuck transaction occurs typically hours after Bisq's launch. Yesterday it occurred about 12h after launch.
Yes the hardware is a RPI4 with 8 GB of RAM.

Is there a way to cancel a stuck RPC command without killing the the whole process?

First, you should take a look at your server resources, for that I recommend the 'top' command, there will you see how many RAM are you using. And if you want to free some space in the ram you could use the next command:

Code:

# sync; echo 3 > /proc/sys/vm/drop_caches 

The function of that command is to Clear pagecache, dentries, and inodes. For me, it helps me when my RAM is almost full.

[NotATether]

Hmm, I think my understanding of txindex is not good. Isn't it required to search for transaction by txid, unless the transaction is part of a Bitcoin Core wallet? I don't think Bisq is using a Bitcoin Core wallet. Thanks!

Bisq requires txindex, yes. It uses the RPC functionality of Bitcoin Core, but not the wallets - Bisq has its own wallet subsystem. But your Bitcoin Core wallet itself does not and will work perfectly fine without it.

[Jodmapht]

I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!

Possible, since RPC works although slow, the issue may be your hardware can't handle all those processes at the same time.
(based from the dbcache, RPi?)

I'm not using Bisq so I can't help any further but I've read that it has high CPU usage during the sync process.

Yes during Bisq's sync process, there is heavy load on my node, but then it goes away once sync is complete after a minute or so. The stuck transaction occurs typically hours after Bisq's launch. Yesterday it occurred about 12h after launch.
Yes the hardware is a RPI4 with 8 GB of RAM.

Is there a way to cancel a stuck RPC command without killing the the whole process?

First, you should take a look at your server resources, for that I recommend the 'top' command, there will you see how many RAM are you using. And if you want to free some space in the ram you could use the next command:

Code:

# sync; echo 3 > /proc/sys/vm/drop_caches 

The function of that command is to Clear pagecache, dentries, and inodes. For me, it helps me when my RAM is almost full.

Thanks. I have plenty of free RAM available on my node, there is 8 GB on RAM on my RPI. I monitor RAM and I/O regularly on it.

Hmm, I think my understanding of txindex is not good. Isn't it required to search for transaction by txid, unless the transaction is part of a Bitcoin Core wallet? I don't think Bisq is using a Bitcoin Core wallet. Thanks!

Bisq requires txindex, yes. It uses the RPC functionality of Bitcoin Core, but not the wallets - Bisq has its own wallet subsystem. But your Bitcoin Core wallet itself does not and will work perfectly fine without it.

Ok thanks, this is consistent with the understanding I had

Is there a way to cancel a stuck RPC command without killing the the whole process?

You have to sever the network connection between the RPC client and the daemon. Briefly toggling on/off Airplane mode or just physically unplugging and plugging back your Ethernet port should work.

Note - don't do this if the node and client are on the same machine, or you will cripple the node as well.

PS. txindex creates a lot of extra data to parse, which makes some RPC calls that deal with transactions slower.

Ok I can try that then. I use a SSH tunnel between the two, so I guess I can exit from SSH and see if it does the trick.

Hmm, I think my understanding of txindex is not good. Isn't it required to search for transaction by txid, unless the transaction is part of a Bitcoin Core wallet? I don't think Bisq is using a Bitcoin Core wallet. Thanks!

Ok, so closing the SSH tunnel to the node does not kill the active command that gets stuck. Is killing the connection between the RPC client and Core what matters, or the connection between Core and the bitcoin network?

I had to do a hard kill of the process again and it corrupted the coinstatsindex I had just included, which I understand is known to be a bit flimsy compared to txindex.

[moderator's note: consecutive posts merged]

[lionheart78]

I had this node running for about 1.5 year. I don't see any error about my hardware in dmesg. I don't see any error in debug.log. Here is my config:

1.5 years seems a long time for me, if your node is running for a long time it is possible that the mempool has grown in size and the reason why the node is taking a long time to process the getmempoolentry command. In this case, you can try to clear the mempool by using the command bitcoin-cli clearmempool.

You can also try this:

To diagnose why your node is getting stuck with certain commands, you can start by checking the resource usage of your node. You can use the command top or htop on Linux or MacOS, or the Task Manager on Windows to check the CPU, memory, and disk usage of your node. If you see that the CPU or memory usage is high, it could indicate that your node is under heavy load and unable to process the commands in a timely manner.

You can also check the disk usage by using the command df -h on Linux or MacOS, or the Task Manager on Windows. This will show you the amount of free space on your hard drive and could indicate if there's any issue with disk I/O.

Another thing you can check is the network traffic of your node. You can use the command netstat -an on Linux or MacOS, or the Resource Monitor on Windows to check the number of incoming and outgoing connections. If you see a high number of connections, it could indicate that your node is experiencing high network traffic and is unable to process the commands quickly.

You can also check the debug.log file in the bitcoin data directory, which may provide more detailed information about what is happening in the node.

Additionally, you can try increasing the amount of memory and CPU resources available to the node, if possible.

If the issue persists, you can try to increase the -rpc* settings in your bitcoin.conf or use a different JSON-RPC library.

Also, the getrawtransaction command can be slow if the transaction is not in the node's local copy of the blockchain and it needs to request it from other peers.

[Jodmapht]

I had this node running for about 1.5 year. I don't see any error about my hardware in dmesg. I don't see any error in debug.log. Here is my config:

1.5 years seems a long time for me, if your node is running for a long time it is possible that the mempool has grown in size and the reason why the node is taking a long time to process the getmempoolentry command. In this case, you can try to clear the mempool by using the command bitcoin-cli clearmempool.

You can also try this:

To diagnose why your node is getting stuck with certain commands, you can start by checking the resource usage of your node. You can use the command top or htop on Linux or MacOS, or the Task Manager on Windows to check the CPU, memory, and disk usage of your node. If you see that the CPU or memory usage is high, it could indicate that your node is under heavy load and unable to process the commands in a timely manner.

You can also check the disk usage by using the command df -h on Linux or MacOS, or the Task Manager on Windows. This will show you the amount of free space on your hard drive and could indicate if there's any issue with disk I/O.

Another thing you can check is the network traffic of your node. You can use the command netstat -an on Linux or MacOS, or the Resource Monitor on Windows to check the number of incoming and outgoing connections. If you see a high number of connections, it could indicate that your node is experiencing high network traffic and is unable to process the commands quickly.

You can also check the debug.log file in the bitcoin data directory, which may provide more detailed information about what is happening in the node.

Additionally, you can try increasing the amount of memory and CPU resources available to the node, if possible.

If the issue persists, you can try to increase the -rpc* settings in your bitcoin.conf or use a different JSON-RPC library.

Also, the getrawtransaction command can be slow if the transaction is not in the node's local copy of the blockchain and it needs to request it from other peers.

Sorry I did not mean that it has been up for 1.5 years, I meant setted up and synchronized 1.5 year ago (I upgraded Core along the way).

If the node needs to request the transaction from peers it could take long, but it does not explain the CPU usage?

[nc50lc]

1.5 years seems a long time for me, if your node is running for a long time it is possible that the mempool has grown in size and the reason why the node is taking a long time to process the getmempoolentry command. In this case, you can try to clear the mempool by using the command bitcoin-cli clearmempool.

His mempool will periodically drop transactions once those are included in a block.
Even if he run the node non-stop in 1.5 years, there shouldn't be any issue with his mempool.

Also, the getrawtransaction command can be slow if the transaction is not in the node's local copy of the blockchain and it needs to request it from other peers.

getrawtransaction wont request from other peers, it'll either get from his mempool or his blockchain since he has txidex enabled.

[Jodmapht]

I had this node running for about 1.5 year. I don't see any error about my hardware in dmesg. I don't see any error in debug.log. Here is my config:

1.5 years seems a long time for me, if your node is running for a long time it is possible that the mempool has grown in size and the reason why the node is taking a long time to process the getmempoolentry command. In this case, you can try to clear the mempool by using the command bitcoin-cli clearmempool.

This is wrong. Transaction on mempool is removed either after included in a block or remain unconfirmed after 2 weeks.

You can also try this:

You can also check the disk usage by using the command df -h on Linux or MacOS, or the Task Manager on Windows. This will show you the amount of free space on your hard drive and could indicate if there's any issue with disk I/O.

This part is inaccurate. df -h only show 6 column (Filesystem, Size, Used, Avail, Use%, Mounted on) which can't be used to check disk I/O usage.

Also, the getrawtransaction command can be slow if the transaction is not in the node's local copy of the blockchain and it needs to request it from other peers.

Also wrong. If you try to execute getrawtransaction with TXID which isn't exist on mempool/blockhain you'll get this message (for reference, i use Bitcoin Core 23.0).

Code:

No such mempool or blockchain transaction. Use gettransaction for wallet transactions. (code -5)

And unless this command uses a spin loop it would not explain de 100% CPU usage for 1+ day?

[alecfisker]

shrinkdebugfile=1
listenonion=1
txindex=1
blockfilterindex=1
hardware is a RPI4 with 8 GB of RAM.
all this adds up

try improve IO some how, use faster SSD, more pcie lines
create two logical drives one for OS other for coin node also helps

[Jodmapht]

I don't think IO is the bottleneck here, my SSD is optimised as much as possible and iowait seems negligible when I have issues. It rather seems that bitcoind enters into some kind of spin loop when executing getmempoolinfo or getrawtransaction and a connectivity issue with the RPC client occurs?

Edit: So I attached gdb to the process while it was stuck on getrawtransaction. Here is the info about the thread that is stuck and using 100% CPU (this is version v23.0):

Code:

Thread 7 (Thread 0x7fb37fdf40 (LWP 850614) "b-httpworker.0"):
#0  0x0000007fbda6a92c in mcount_internal (selfpc=8709628, frompc=1142896) at ../gmon/mcount.c:153
#1  __mcount (frompc=0x53c880 <RPCResult::RPCResult(RPCResult::Type, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<RPCResult, std::allocator<RPCResult> >)+240>) at ../gmon/mcount.c:180
#2  0x000000000084e5fc in RPCResult::CheckInnerDoc (this=this@entry=0x7fb37fbe70) at rpc/util.cpp:903
#3  0x000000000053c880 in RPCResult::RPCResult (inner=..., description=..., optional=false, m_key_name=..., type=<optimized out>, this=0x7fb37fbe70) at ./rpc/util.h:301
#4  RPCResult::RPCResult (this=0x7fb37fbe70, type=<optimized out>, m_key_name=..., description=..., inner=...) at ./rpc/util.h:309
#5  0x0000000000589d4c in getrawtransaction () at rpc/rawtransaction.cpp:194
#6  0x000000000053b1b0 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (request=..., result=..., __closure=<optimized out>, __closure=<optimized out>) at ./rpc/server.h:109
#7  0x00000000005b71cc in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (__args#2=<optimized out>, __args#1=..., __args#0=..., this=0xd1c650 <RegisterRawTransactionRPCCommands(CRPCTable&)::commands+64>) at /usr/include/c++/10/bits/std_function.h:622
#8  ExecuteCommand (command=..., request=..., result=..., last_handler=true) at rpc/server.cpp:480
#9  0x00000000005b840c in ExecuteCommands (result=..., request=..., commands=...) at rpc/server.cpp:444
#10 CRPCTable::execute (this=<optimized out>, request=...) at rpc/server.cpp:464
#11 0x000000000066e218 in HTTPReq_JSONRPC (context=..., req=0x7f9c0035a0) at httprpc.cpp:202
#12 0x000000000067aa84 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1=..., __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/10/bits/std_function.h:622
#13 HTTPWorkItem::operator() (this=<optimized out>) at httpserver.cpp:54
#14 WorkQueue<HTTPClosure>::Run (this=this@entry=0x16e99150) at httpserver.cpp:112
#15 0x0000000000676480 in HTTPWorkQueueRun (queue=0x16e99150, worker_num=<optimized out>) at httpserver.cpp:342
#16 0x0000007fbdca4cac in ?? () from target:/lib/aarch64-linux-gnu/libstdc++.so.6
#17 0x0000007fbe048648 in start_thread (arg=0x7fb37fd840) at pthread_create.c:477
#18 0x0000007fbda67c1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Here is a second snapshot

Code:

Thread 7 (Thread 0x7fb37fdf40 (LWP 850614) "b-httpworker.0"):
#0  0x0000007fbda6a91c in mcount_internal (selfpc=8709628, frompc=1142896) at ../gmon/mcount.c:152
#1  __mcount (frompc=0x53c880 <RPCResult::RPCResult(RPCResult::Type, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<RPCResult, std::allocator<RPCResult> >)+240>) at ../gmon/mcount.c:180
#2  0x000000000084e5fc in RPCResult::CheckInnerDoc (this=this@entry=0x7fb37fbe70) at rpc/util.cpp:903
#3  0x000000000053c880 in RPCResult::RPCResult (inner=..., description=..., optional=false, m_key_name=..., type=<optimized out>, this=0x7fb37fbe70) at ./rpc/util.h:301
#4  RPCResult::RPCResult (this=0x7fb37fbe70, type=<optimized out>, m_key_name=..., description=..., inner=...) at ./rpc/util.h:309
#5  0x0000000000589d4c in getrawtransaction () at rpc/rawtransaction.cpp:194
#6  0x000000000053b1b0 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (request=..., result=..., __closure=<optimized out>, __closure=<optimized out>) at ./rpc/server.h:109
#7  0x00000000005b71cc in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (__args#2=<optimized out>, __args#1=..., __args#0=..., this=0xd1c650 <RegisterRawTransactionRPCCommands(CRPCTable&)::commands+64>) at /usr/include/c++/10/bits/std_function.h:622
#8  ExecuteCommand (command=..., request=..., result=..., last_handler=true) at rpc/server.cpp:480
#9  0x00000000005b840c in ExecuteCommands (result=..., request=..., commands=...) at rpc/server.cpp:444
#10 CRPCTable::execute (this=<optimized out>, request=...) at rpc/server.cpp:464
#11 0x000000000066e218 in HTTPReq_JSONRPC (context=..., req=0x7f9c0035a0) at httprpc.cpp:202
#12 0x000000000067aa84 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1=..., __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/10/bits/std_function.h:622
#13 HTTPWorkItem::operator() (this=<optimized out>) at httpserver.cpp:54
#14 WorkQueue<HTTPClosure>::Run (this=this@entry=0x16e99150) at httpserver.cpp:112
#15 0x0000000000676480 in HTTPWorkQueueRun (queue=0x16e99150, worker_num=<optimized out>) at httpserver.cpp:342
#16 0x0000007fbdca4cac in ?? () from target:/lib/aarch64-linux-gnu/libstdc++.so.6
#17 0x0000007fbe048648 in start_thread (arg=0x7fb37fd840) at pthread_create.c:477
#18 0x0000007fbda67c1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

and a third one:

Code:

Thread 7 (Thread 0x7fb37fdf40 (LWP 850614) "b-httpworker.0"):
#0  0x0000007fbda6a92c in mcount_internal (selfpc=8709628, frompc=1142896) at ../gmon/mcount.c:153
#1  __mcount (frompc=0x53c880 <RPCResult::RPCResult(RPCResult::Type, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<RPCResult, std::allocator<RPCResult> >)+240>) at ../gmon/mcount.c:180
#2  0x000000000084e5fc in RPCResult::CheckInnerDoc (this=this@entry=0x7fb37fbe70) at rpc/util.cpp:903
#3  0x000000000053c880 in RPCResult::RPCResult (inner=..., description=..., optional=false, m_key_name=..., type=<optimized out>, this=0x7fb37fbe70) at ./rpc/util.h:301
#4  RPCResult::RPCResult (this=0x7fb37fbe70, type=<optimized out>, m_key_name=..., description=..., inner=...) at ./rpc/util.h:309
#5  0x0000000000589d4c in getrawtransaction () at rpc/rawtransaction.cpp:194
#6  0x000000000053b1b0 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (request=..., result=..., __closure=<optimized out>, __closure=<optimized out>) at ./rpc/server.h:109
#7  0x00000000005b71cc in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (__args#2=<optimized out>, __args#1=..., __args#0=..., this=0xd1c650 <RegisterRawTransactionRPCCommands(CRPCTable&)::commands+64>) at /usr/include/c++/10/bits/std_function.h:622
#8  ExecuteCommand (command=..., request=..., result=..., last_handler=true) at rpc/server.cpp:480
#9  0x00000000005b840c in ExecuteCommands (result=..., request=..., commands=...) at rpc/server.cpp:444
#10 CRPCTable::execute (this=<optimized out>, request=...) at rpc/server.cpp:464
#11 0x000000000066e218 in HTTPReq_JSONRPC (context=..., req=0x7f9c0035a0) at httprpc.cpp:202
#12 0x000000000067aa84 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1=..., __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/10/bits/std_function.h:622
#13 HTTPWorkItem::operator() (this=<optimized out>) at httpserver.cpp:54
#14 WorkQueue<HTTPClosure>::Run (this=this@entry=0x16e99150) at httpserver.cpp:112
#15 0x0000000000676480 in HTTPWorkQueueRun (queue=0x16e99150, worker_num=<optimized out>) at httpserver.cpp:342
#16 0x0000007fbdca4cac in ?? () from target:/lib/aarch64-linux-gnu/libstdc++.so.6
#17 0x0000007fbe048648 in start_thread (arg=0x7fb37fd840) at pthread_create.c:477
#18 0x0000007fbda67c1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

What is going on with this?
Thanks!

Edit: Node is stuck again this morning, this time with getmempoolentry. Here is the stuck thread:

Code:

Thread 7 (Thread 0x7f8ca26f40 (LWP 1037331) "b-httpworker.0"):
#0  mcount_internal (selfpc=5489172, frompc=1141804) at ../gmon/mcount.c:153
#1  __mcount (frompc=0x53c43c <RPCResult::RPCResult(RPCResult::Type, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<RPCResult, std::allocator<RPCResult> >)+108>) at ../gmon/mcount.c:180
#2  0x000000000053c214 in std::vector<RPCResult, std::allocator<RPCResult> >::vector (this=0x7f8ca247e0, __x=...) at /usr/include/c++/10/bits/stl_vector.h:553
#3  0x000000000053c43c in RPCResult::RPCResult (this=0x7f8ca247b8, type=RPCResult::Type::STR_AMOUNT, m_key_name=..., optional=true, description=..., inner=...) at /usr/include/c++/10/bits/move.h:101
#4  0x0000000000512434 in MempoolEntryDescription () at rpc/blockchain.cpp:463
#5  0x000000000052c90c in getmempoolentry () at rpc/blockchain.cpp:764
#6  0x000000000053b1b0 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (request=..., result=..., __closure=<optimized out>, __closure=<optimized out>) at ./rpc/server.h:109
#7  0x00000000005b71cc in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (__args#2=<optimized out>, __args#1=..., __args#0=..., this=0xd1a738 <RegisterBlockchainRPCCommands(CRPCTable&)::commands+1856>) at /usr/include/c++/10/bits/std_function.h:622
#8  ExecuteCommand (command=..., request=..., result=..., last_handler=true) at rpc/server.cpp:480
#9  0x00000000005b840c in ExecuteCommands (result=..., request=..., commands=...) at rpc/server.cpp:444
#10 CRPCTable::execute (this=<optimized out>, request=...) at rpc/server.cpp:464
#11 0x000000000066e218 in HTTPReq_JSONRPC (context=..., req=0x7f7c00f0d0) at httprpc.cpp:202
#12 0x000000000067aa84 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1=..., __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/10/bits/std_function.h:622
#13 HTTPWorkItem::operator() (this=<optimized out>) at httpserver.cpp:54
#14 WorkQueue<HTTPClosure>::Run (this=this@entry=0x20a6e150) at httpserver.cpp:112
#15 0x0000000000676480 in HTTPWorkQueueRun (queue=0x20a6e150, worker_num=<optimized out>) at httpserver.cpp:342
#16 0x0000007f926fdcac in ?? () from target:/lib/aarch64-linux-gnu/libstdc++.so.6
#17 0x0000007f92aa1648 in start_thread (arg=0x7f8ca26840) at pthread_create.c:477
#18 0x0000007f924c0c1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Edit: In the release notes for v23.1, there is something about a fix for a race condition. As I think it might be the issue I am facing, I tried upgrading to v24.0.1. I will see it the issue is gone...

Edit: So far it looks like the problem is not occurring with v24.0.1, so it appears I was experiencing the race condition from v23.0 ...