 Show Posts
|
|
Pages: « 1 [2] 3 4 »
|
Which lightning client/wallet are you using?
It could mean that you or users are sending requests in quick succession but it could be anything else depending on the client.
I am using my one bitcoin/CLN node. I am able to open channels with other Lightning nodes, but not this one (ACINQ). |
|
|
|
|
Hi,
I have been trying to open a Lightning channel with a major node for about a day, but every time I get a "Rate Limit Reached" error. What does it mean exactly?
Thank you!
|
|
|
|
|
Hi,
I tried closing a channel using the fee-range feature to set a fee range that makes sense, but the fee range argument is not well documented and I must have passed it using wrong units, resulting in a stuck transaction. I need to close another channel so I need to figure this out. Could someone help me?
Here is what I used. I thought I was specifying a fee rate argument in sat/vByte for the closing tx:
lightning-cli close (channel_ID) 345600 (output_address) null null null '[45,50]'
Here is the output from lightning-cli:
# Sending closing fee offer 171sat, with range 171sat-171sat
Here is what I saw in the logs:
State changed from CHANNELD_NORMAL to CHANNELD_SHUTTING_DOWN
State changed from CHANNELD_SHUTTING_DOWN to CLOSINGD_SIGEXCHANGE
Our ideal fee is 3395sat (5023 sats/perkw), but our maximum is 171sat: using that
Peer transient failure in CLOSINGD_SIGEXCHANGE: closingd WARNING: warning channel (channel hex id): closing fee range must not be below 2683 sat
It seemed stuck there, then I restarted lightningd and I saw this:
performing quickclose in range 1697sat-7564sat
State changed from CLOSINGD_SIGEXCHANGE to CLOSINGD_COMPLETE
And the tx was submitted to the mempool with an insufficient fee rate of 20.98 sat/vByte and a total fee of 0.00003544BTC.
So what happened exactly and how should I specify the fee rate in the future to avoid this?
Thanks!
|
|
|
|
|
Hi,
After upgrading CLN to the latest tag, I noticed that my channel info was wrong when using c-lightning-REST, either when using it either through Zeus or RTL. I tried upgrading c-lightning-REST to the latest tag as well (v0.10.2), but it did not change the result. It shows my channel and it says it is open, but it shows balances of 0 for the channel. When I use lightning-cli, the balances are as expected. I do not see any error or warning in CLN's logs. Has anyone experienced this kind of problem before? Could I be missing something or is it likely to be a bug? The latest tag for the plugin is quite a bit older than for CLN,
Thanks!
|
|
|
|
it is a possibility that it was just a glitch with the explorer.
Probably an error from you if not the explorer. I mean maybe you did not check it accurately. So it is not possible that it is related to a change with respect to the longest chain?
Not related as it has 5 confirmations at the fifth block mined which is correct. I did reload the page multiple times and I also searched for the transaction a second time. I have not tried clearing the cache and the cookies through. I don't recall experiencing this issue in the past. |
|
|
|
After you transaction has 1 confirmation. As next block is mined, your transaction has 2 confirmations. As the another block is mined, it will have 3 confirmations, and so on. Maybe the blockchain explorer did not update it properly.
it is a possibility that it was just a glitch with the explorer. So it is not possible that it is related to a change with respect to the longest chain? Thanks! |
|
|
|
|
Hi,
There is this situation that happened to me where a transaction gets included in a block, then there are four additional blocks that get added while the number of confirmations for my transaction remains unchanged (as shown by btcrpcexplorer), until a sixth block gets added, which caused the number of confirmations for my transaction to jump to 5. Would this occur if there are parallel chains that develop? If it is the case, how do the transactions that were included in a shorter chain get introduced back in the main chain? Do they just get back in the mempool with the same priority as any other mempool trnasaction? Does it mean my transaction could have been easily overridden while there was only a single confirmation by spending any input utxo using a higher fee?
Thanks!
|
|
|
|
I don't think IO is the bottleneck here, my SSD is optimised as much as possible and iowait seems negligible when I have issues. It rather seems that bitcoind enters into some kind of spin loop when executing getmempoolinfo or getrawtransaction and a connectivity issue with the RPC client occurs? Edit: So I attached gdb to the process while it was stuck on getrawtransaction. Here is the info about the thread that is stuck and using 100% CPU (this is version v23.0): Thread 7 (Thread 0x7fb37fdf40 (LWP 850614) "b-httpworker.0"):
#0 0x0000007fbda6a92c in mcount_internal (selfpc=8709628, frompc=1142896) at ../gmon/mcount.c:153
#1 __mcount (frompc=0x53c880 <RPCResult::RPCResult(RPCResult::Type, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<RPCResult, std::allocator<RPCResult> >)+240>) at ../gmon/mcount.c:180
#2 0x000000000084e5fc in RPCResult::CheckInnerDoc (this=this@entry=0x7fb37fbe70) at rpc/util.cpp:903
#3 0x000000000053c880 in RPCResult::RPCResult (inner=..., description=..., optional=false, m_key_name=..., type=<optimized out>, this=0x7fb37fbe70) at ./rpc/util.h:301
#4 RPCResult::RPCResult (this=0x7fb37fbe70, type=<optimized out>, m_key_name=..., description=..., inner=...) at ./rpc/util.h:309
#5 0x0000000000589d4c in getrawtransaction () at rpc/rawtransaction.cpp:194
#6 0x000000000053b1b0 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (request=..., result=..., __closure=<optimized out>, __closure=<optimized out>) at ./rpc/server.h:109
#7 0x00000000005b71cc in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (__args#2=<optimized out>, __args#1=..., __args#0=..., this=0xd1c650 <RegisterRawTransactionRPCCommands(CRPCTable&)::commands+64>) at /usr/include/c++/10/bits/std_function.h:622
#8 ExecuteCommand (command=..., request=..., result=..., last_handler=true) at rpc/server.cpp:480
#9 0x00000000005b840c in ExecuteCommands (result=..., request=..., commands=...) at rpc/server.cpp:444
#10 CRPCTable::execute (this=<optimized out>, request=...) at rpc/server.cpp:464
#11 0x000000000066e218 in HTTPReq_JSONRPC (context=..., req=0x7f9c0035a0) at httprpc.cpp:202
#12 0x000000000067aa84 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1=..., __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/10/bits/std_function.h:622
#13 HTTPWorkItem::operator() (this=<optimized out>) at httpserver.cpp:54
#14 WorkQueue<HTTPClosure>::Run (this=this@entry=0x16e99150) at httpserver.cpp:112
#15 0x0000000000676480 in HTTPWorkQueueRun (queue=0x16e99150, worker_num=<optimized out>) at httpserver.cpp:342
#16 0x0000007fbdca4cac in ?? () from target:/lib/aarch64-linux-gnu/libstdc++.so.6
#17 0x0000007fbe048648 in start_thread (arg=0x7fb37fd840) at pthread_create.c:477
#18 0x0000007fbda67c1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Here is a second snapshot Thread 7 (Thread 0x7fb37fdf40 (LWP 850614) "b-httpworker.0"):
#0 0x0000007fbda6a91c in mcount_internal (selfpc=8709628, frompc=1142896) at ../gmon/mcount.c:152
#1 __mcount (frompc=0x53c880 <RPCResult::RPCResult(RPCResult::Type, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<RPCResult, std::allocator<RPCResult> >)+240>) at ../gmon/mcount.c:180
#2 0x000000000084e5fc in RPCResult::CheckInnerDoc (this=this@entry=0x7fb37fbe70) at rpc/util.cpp:903
#3 0x000000000053c880 in RPCResult::RPCResult (inner=..., description=..., optional=false, m_key_name=..., type=<optimized out>, this=0x7fb37fbe70) at ./rpc/util.h:301
#4 RPCResult::RPCResult (this=0x7fb37fbe70, type=<optimized out>, m_key_name=..., description=..., inner=...) at ./rpc/util.h:309
#5 0x0000000000589d4c in getrawtransaction () at rpc/rawtransaction.cpp:194
#6 0x000000000053b1b0 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (request=..., result=..., __closure=<optimized out>, __closure=<optimized out>) at ./rpc/server.h:109
#7 0x00000000005b71cc in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (__args#2=<optimized out>, __args#1=..., __args#0=..., this=0xd1c650 <RegisterRawTransactionRPCCommands(CRPCTable&)::commands+64>) at /usr/include/c++/10/bits/std_function.h:622
#8 ExecuteCommand (command=..., request=..., result=..., last_handler=true) at rpc/server.cpp:480
#9 0x00000000005b840c in ExecuteCommands (result=..., request=..., commands=...) at rpc/server.cpp:444
#10 CRPCTable::execute (this=<optimized out>, request=...) at rpc/server.cpp:464
#11 0x000000000066e218 in HTTPReq_JSONRPC (context=..., req=0x7f9c0035a0) at httprpc.cpp:202
#12 0x000000000067aa84 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1=..., __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/10/bits/std_function.h:622
#13 HTTPWorkItem::operator() (this=<optimized out>) at httpserver.cpp:54
#14 WorkQueue<HTTPClosure>::Run (this=this@entry=0x16e99150) at httpserver.cpp:112
#15 0x0000000000676480 in HTTPWorkQueueRun (queue=0x16e99150, worker_num=<optimized out>) at httpserver.cpp:342
#16 0x0000007fbdca4cac in ?? () from target:/lib/aarch64-linux-gnu/libstdc++.so.6
#17 0x0000007fbe048648 in start_thread (arg=0x7fb37fd840) at pthread_create.c:477
#18 0x0000007fbda67c1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
and a third one: Thread 7 (Thread 0x7fb37fdf40 (LWP 850614) "b-httpworker.0"):
#0 0x0000007fbda6a92c in mcount_internal (selfpc=8709628, frompc=1142896) at ../gmon/mcount.c:153
#1 __mcount (frompc=0x53c880 <RPCResult::RPCResult(RPCResult::Type, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<RPCResult, std::allocator<RPCResult> >)+240>) at ../gmon/mcount.c:180
#2 0x000000000084e5fc in RPCResult::CheckInnerDoc (this=this@entry=0x7fb37fbe70) at rpc/util.cpp:903
#3 0x000000000053c880 in RPCResult::RPCResult (inner=..., description=..., optional=false, m_key_name=..., type=<optimized out>, this=0x7fb37fbe70) at ./rpc/util.h:301
#4 RPCResult::RPCResult (this=0x7fb37fbe70, type=<optimized out>, m_key_name=..., description=..., inner=...) at ./rpc/util.h:309
#5 0x0000000000589d4c in getrawtransaction () at rpc/rawtransaction.cpp:194
#6 0x000000000053b1b0 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (request=..., result=..., __closure=<optimized out>, __closure=<optimized out>) at ./rpc/server.h:109
#7 0x00000000005b71cc in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (__args#2=<optimized out>, __args#1=..., __args#0=..., this=0xd1c650 <RegisterRawTransactionRPCCommands(CRPCTable&)::commands+64>) at /usr/include/c++/10/bits/std_function.h:622
#8 ExecuteCommand (command=..., request=..., result=..., last_handler=true) at rpc/server.cpp:480
#9 0x00000000005b840c in ExecuteCommands (result=..., request=..., commands=...) at rpc/server.cpp:444
#10 CRPCTable::execute (this=<optimized out>, request=...) at rpc/server.cpp:464
#11 0x000000000066e218 in HTTPReq_JSONRPC (context=..., req=0x7f9c0035a0) at httprpc.cpp:202
#12 0x000000000067aa84 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1=..., __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/10/bits/std_function.h:622
#13 HTTPWorkItem::operator() (this=<optimized out>) at httpserver.cpp:54
#14 WorkQueue<HTTPClosure>::Run (this=this@entry=0x16e99150) at httpserver.cpp:112
#15 0x0000000000676480 in HTTPWorkQueueRun (queue=0x16e99150, worker_num=<optimized out>) at httpserver.cpp:342
#16 0x0000007fbdca4cac in ?? () from target:/lib/aarch64-linux-gnu/libstdc++.so.6
#17 0x0000007fbe048648 in start_thread (arg=0x7fb37fd840) at pthread_create.c:477
#18 0x0000007fbda67c1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
What is going on with this? Thanks! Edit: Node is stuck again this morning, this time with getmempoolentry. Here is the stuck thread: Thread 7 (Thread 0x7f8ca26f40 (LWP 1037331) "b-httpworker.0"):
#0 mcount_internal (selfpc=5489172, frompc=1141804) at ../gmon/mcount.c:153
#1 __mcount (frompc=0x53c43c <RPCResult::RPCResult(RPCResult::Type, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<RPCResult, std::allocator<RPCResult> >)+108>) at ../gmon/mcount.c:180
#2 0x000000000053c214 in std::vector<RPCResult, std::allocator<RPCResult> >::vector (this=0x7f8ca247e0, __x=...) at /usr/include/c++/10/bits/stl_vector.h:553
#3 0x000000000053c43c in RPCResult::RPCResult (this=0x7f8ca247b8, type=RPCResult::Type::STR_AMOUNT, m_key_name=..., optional=true, description=..., inner=...) at /usr/include/c++/10/bits/move.h:101
#4 0x0000000000512434 in MempoolEntryDescription () at rpc/blockchain.cpp:463
#5 0x000000000052c90c in getmempoolentry () at rpc/blockchain.cpp:764
#6 0x000000000053b1b0 in CRPCCommand::CRPCCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, RPCHelpMan (*)())::{lambda(JSONRPCRequest const&, UniValue&, bool)#1}::operator()(JSONRPCRequest const&, UniValue&, bool) const (request=..., result=..., __closure=<optimized out>, __closure=<optimized out>) at ./rpc/server.h:109
#7 0x00000000005b71cc in std::function<bool (JSONRPCRequest const&, UniValue&, bool)>::operator()(JSONRPCRequest const&, UniValue&, bool) const (__args#2=<optimized out>, __args#1=..., __args#0=..., this=0xd1a738 <RegisterBlockchainRPCCommands(CRPCTable&)::commands+1856>) at /usr/include/c++/10/bits/std_function.h:622
#8 ExecuteCommand (command=..., request=..., result=..., last_handler=true) at rpc/server.cpp:480
#9 0x00000000005b840c in ExecuteCommands (result=..., request=..., commands=...) at rpc/server.cpp:444
#10 CRPCTable::execute (this=<optimized out>, request=...) at rpc/server.cpp:464
#11 0x000000000066e218 in HTTPReq_JSONRPC (context=..., req=0x7f7c00f0d0) at httprpc.cpp:202
#12 0x000000000067aa84 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1=..., __args#0=<optimized out>, this=<optimized out>) at /usr/include/c++/10/bits/std_function.h:622
#13 HTTPWorkItem::operator() (this=<optimized out>) at httpserver.cpp:54
#14 WorkQueue<HTTPClosure>::Run (this=this@entry=0x20a6e150) at httpserver.cpp:112
#15 0x0000000000676480 in HTTPWorkQueueRun (queue=0x20a6e150, worker_num=<optimized out>) at httpserver.cpp:342
#16 0x0000007f926fdcac in ?? () from target:/lib/aarch64-linux-gnu/libstdc++.so.6
#17 0x0000007f92aa1648 in start_thread (arg=0x7f8ca26840) at pthread_create.c:477
#18 0x0000007f924c0c1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Edit: In the release notes for v23.1, there is something about a fix for a race condition. As I think it might be the issue I am facing, I tried upgrading to v24.0.1. I will see it the issue is gone... Edit: So far it looks like the problem is not occurring with v24.0.1, so it appears I was experiencing the race condition from v23.0 ... |
|
|
|
I had this node running for about 1.5 year. I don't see any error about my hardware in dmesg. I don't see any error in debug.log. Here is my config:
1.5 years seems a long time for me, if your node is running for a long time it is possible that the mempool has grown in size and the reason why the node is taking a long time to process the getmempoolentry command. In this case, you can try to clear the mempool by using the command bitcoin-cli clearmempool. This is wrong. Transaction on mempool is removed either after included in a block or remain unconfirmed after 2 weeks. You can also try this: You can also check the disk usage by using the command df -h on Linux or MacOS, or the Task Manager on Windows. This will show you the amount of free space on your hard drive and could indicate if there's any issue with disk I/O.
This part is inaccurate. df -h only show 6 column (Filesystem, Size, Used, Avail, Use%, Mounted on) which can't be used to check disk I/O usage. Also, the getrawtransaction command can be slow if the transaction is not in the node's local copy of the blockchain and it needs to request it from other peers.
Also wrong. If you try to execute getrawtransaction with TXID which isn't exist on mempool/blockhain you'll get this message (for reference, i use Bitcoin Core 23.0). No such mempool or blockchain transaction. Use gettransaction for wallet transactions. (code -5)
And unless this command uses a spin loop it would not explain de 100% CPU usage for 1+ day? |
|
|
|
I had this node running for about 1.5 year. I don't see any error about my hardware in dmesg. I don't see any error in debug.log. Here is my config:
1.5 years seems a long time for me, if your node is running for a long time it is possible that the mempool has grown in size and the reason why the node is taking a long time to process the getmempoolentry command. In this case, you can try to clear the mempool by using the command bitcoin-cli clearmempool. You can also try this: To diagnose why your node is getting stuck with certain commands, you can start by checking the resource usage of your node. You can use the command top or htop on Linux or MacOS, or the Task Manager on Windows to check the CPU, memory, and disk usage of your node. If you see that the CPU or memory usage is high, it could indicate that your node is under heavy load and unable to process the commands in a timely manner.
You can also check the disk usage by using the command df -h on Linux or MacOS, or the Task Manager on Windows. This will show you the amount of free space on your hard drive and could indicate if there's any issue with disk I/O.
Another thing you can check is the network traffic of your node. You can use the command netstat -an on Linux or MacOS, or the Resource Monitor on Windows to check the number of incoming and outgoing connections. If you see a high number of connections, it could indicate that your node is experiencing high network traffic and is unable to process the commands quickly.
You can also check the debug.log file in the bitcoin data directory, which may provide more detailed information about what is happening in the node.
Additionally, you can try increasing the amount of memory and CPU resources available to the node, if possible.
If the issue persists, you can try to increase the -rpc* settings in your bitcoin.conf or use a different JSON-RPC library.
Also, the getrawtransaction command can be slow if the transaction is not in the node's local copy of the blockchain and it needs to request it from other peers. Sorry I did not mean that it has been up for 1.5 years, I meant setted up and synchronized 1.5 year ago (I upgraded Core along the way). If the node needs to request the transaction from peers it could take long, but it does not explain the CPU usage? |
|
|
|
I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!
Possible, since RPC works although slow, the issue may be your hardware can't handle all those processes at the same time. ( based from the dbcache, RPi?) I'm not using Bisq so I can't help any further but I've read that it has high CPU usage during the sync process. Yes during Bisq's sync process, there is heavy load on my node, but then it goes away once sync is complete after a minute or so. The stuck transaction occurs typically hours after Bisq's launch. Yesterday it occurred about 12h after launch. Yes the hardware is a RPI4 with 8 GB of RAM. Is there a way to cancel a stuck RPC command without killing the the whole process? First, you should take a look at your server resources, for that I recommend the 'top' command, there will you see how many RAM are you using. And if you want to free some space in the ram you could use the next command: # sync; echo 3 > /proc/sys/vm/drop_caches The function of that command is to Clear pagecache, dentries, and inodes. For me, it helps me when my RAM is almost full. Thanks. I have plenty of free RAM available on my node, there is 8 GB on RAM on my RPI. I monitor RAM and I/O regularly on it.
Hmm, I think my understanding of txindex is not good. Isn't it required to search for transaction by txid, unless the transaction is part of a Bitcoin Core wallet? I don't think Bisq is using a Bitcoin Core wallet. Thanks!
Bisq requires txindex, yes. It uses the RPC functionality of Bitcoin Core, but not the wallets - Bisq has its own wallet subsystem. But your Bitcoin Core wallet itself does not and will work perfectly fine without it. Ok thanks, this is consistent with the understanding I had
Is there a way to cancel a stuck RPC command without killing the the whole process?
You have to sever the network connection between the RPC client and the daemon. Briefly toggling on/off Airplane mode or just physically unplugging and plugging back your Ethernet port should work. Note - don't do this if the node and client are on the same machine, or you will cripple the node as well. PS. txindex creates a lot of extra data to parse, which makes some RPC calls that deal with transactions slower. Ok I can try that then. I use a SSH tunnel between the two, so I guess I can exit from SSH and see if it does the trick. Hmm, I think my understanding of txindex is not good. Isn't it required to search for transaction by txid, unless the transaction is part of a Bitcoin Core wallet? I don't think Bisq is using a Bitcoin Core wallet. Thanks! Ok, so closing the SSH tunnel to the node does not kill the active command that gets stuck. Is killing the connection between the RPC client and Core what matters, or the connection between Core and the bitcoin network? I had to do a hard kill of the process again and it corrupted the coinstatsindex I had just included, which I understand is known to be a bit flimsy compared to txindex. [moderator's note: consecutive posts merged] |
|
|
|
Is there a way to cancel a stuck RPC command without killing the the whole process?
You have to sever the network connection between the RPC client and the daemon. Briefly toggling on/off Airplane mode or just physically unplugging and plugging back your Ethernet port should work. Note - don't do this if the node and client are on the same machine, or you will cripple the node as well. PS. txindex creates a lot of extra data to parse, which makes some RPC calls that deal with transactions slower. Ok I can try that then. I use a SSH tunnel between the two, so I guess I can exit from SSH and see if it does the trick. Hmm, I think my understanding of txindex is not good. Isn't it required to search for transaction by txid, unless the transaction is part of a Bitcoin Core wallet? I don't think Bisq is using a Bitcoin Core wallet. Thanks! |
|
|
|
I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!
Possible, since RPC works although slow, the issue may be your hardware can't handle all those processes at the same time. ( based from the dbcache, RPi?) I'm not using Bisq so I can't help any further but I've read that it has high CPU usage during the sync process. Yes during Bisq's sync process, there is heavy load on my node, but then it goes away once sync is complete after a minute or so. The stuck transaction occurs typically hours after Bisq's launch. Yesterday it occurred about 12h after launch. Yes the hardware is a RPI4 with 8 GB of RAM.
bitcoin_main.conf: [main]
onlynet=onion
bind=127.0.0.1:8333
rpcbind=127.0.0.1
rpccookiefile=/opt/local/bitcoind/var/cookie
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333
whitelist = bloomfilter@127.0.0.1
#And/or add some nodes
addnode=gyn2vguc35viks2b.onion
addnode=kvd44sw7skb5folw.onion
addnode=nkf5e6b7pl4jfd4a.onion
addnode=yu7sezmixhmyljn4.onion
addnode=3ffk7iumtx3cegbi.onion
addnode=3nmbbakinewlgdln.onion
addnode=4j77gihpokxu2kj4.onion
addnode=546esc6botbjfbxb.onion
addnode=5at7sq5nm76xijkd.onion
addnode=77mx2jsxaoyesz2p.onion
addnode=7g7j54btiaxhtsiy.onion
addnode=a6obdgzn67l7exu3.onion
addnode=ab64h7olpl7qpxci.onion
addnode=am2a4rahltfuxz6l.onion
addnode=azuxls4ihrr2mep7.onion
addnode=bitcoin7bi4op7wb.onion
addnode=bitcoinostk4e4re.onion
addnode=bk7yp6epnmcllq72.onion
addnode=bmutjfrj5btseddb.onion
addnode=ceeji4qpfs3ms3zc.onion
addnode=clexmzqio7yhdao4.onion
addnode=gb5ypqt63du3wfhn.onion
addnode=h2vlpudzphzqxutd.onion
addnode=n42h7r6oumcfsbrs.onion:4176
addnode=ncwk3lutemffcpc4.onion
addnode=okdzjarwekbshnof.onion
addnode=pjghcivzkoersesd.onion
addnode=rw7ocjltix26mefn.onion
addnode=uws7itep7o3yinxo.onion
addnode=vk3qjdehyy4dwcxw.onion
addnode=vqpye2k5rcqvj5mq.onion
addnode=wpi7rpvhnndl52ee.onion
Slightly off-topic, but you need to know .onion link V2 is dead[1] where Tor[1] and Bitcoin Core[2] already remove the support about a year ago. I would recommend you to update your configuration file with Bitcoin node which use .onion link V3 instead. If you don't know which node you should add, you can check list of node[3] on Bitcoin Core repository as starter. [1] https://support.torproject.org/onionservices/v2-deprecation/[2] https://github.com/bitcoin/bitcoin/pull/22050[3] https://github.com/bitcoin/bitcoin/blob/master/contrib/seeds/nodes_main.txt#L781-L792Thanks for that, I will fix it! 
I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!
Possible, since RPC works although slow, the issue may be your hardware can't handle all those processes at the same time. ( based from the dbcache, RPi?) I'm not using Bisq so I can't help any further but I've read that it has high CPU usage during the sync process. Yes during Bisq's sync process, there is heavy load on my node, but then it goes away once sync is complete after a minute or so. The stuck transaction occurs typically hours after Bisq's launch. Yesterday it occurred about 12h after launch. Yes the hardware is a RPI4 with 8 GB of RAM. Is there a way to cancel a stuck RPC command without killing the the whole process? [moderator's note: consecutive posts merged] |
|
|
|
|
I left my node running overnight without bisq and I did not experience an issue with CPU. Could the issue be related to the usage of bloom filters by bisq? I upgraded Bitcoin core, started using CLN and Bisq at around the same time and this is when the issues I mentioned started. I think Bisq might be what triggers these issues, but I am not 100% certain. Thanks!
|
|
|
|
-snip-
Also I noticed that I cannot shutdown the process gracefully, it hangs forever in b-shutoff, I have to hard kill it, even if I have dbcache set to only 300. I will try rpc debugging.
Frequently doing that " hard kill" might corrupt your blockchain. If you've finished " Initial Block Download" ( IBD) or already at the tip, then it may be a hardware-related issue, you can also check your debug.log for other errors. I had this node running for about 1.5 year. I don't see any error about my hardware in dmesg. I don't see any error in debug.log. Here is my config: bitcoin.conf: server=1
prune=0
disablewallet=0
txindex=1
blockfilterindex=1
proxy=127.0.0.1:9050
onion=127.0.0.1:9050
proxyrandomize=1
listen=1
listenonion=1
rpcallowip=127.0.0.1
shrinkdebugfile=1
# is required by Fail2Ban described below
logips=1
# magic RBP optimisations
maxconnections=32
whitelist=127.0.0.1
maxuploadtarget=5000
dbcache=300
maxorphantx=10
maxmempool=50
#For btcrpcexplorer
rpcworkqueue=100
#Disable DNS to prevent DNS deanonymization
dnsseed=0
dns=0
#Add seed nodes
seednode=wxvp2d4rspn7tqyu.onion
seednode=bk5ejfe56xakvtkk.onion
seednode=bpdlwholl7rnkrkw.onion
seednode=hhiv5pnxenvbf4am.onion
seednode=4iuf2zac6aq3ndrb.onion
seednode=nkf5e6b7pl4jfd4a.onion
seednode=xqzfakpeuvrobvpj.onion
seednode=tsyvzsqwa2kkf6b2.onion
includeconf=../etc/bitcoin_main.conf
includeconf=../etc/bitcoin_testnet.conf
bitcoin_main.conf: [main]
onlynet=onion
bind=127.0.0.1:8333
rpcbind=127.0.0.1
rpccookiefile=/opt/local/bitcoind/var/cookie
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333
whitelist = bloomfilter@127.0.0.1
#And/or add some nodes
addnode=gyn2vguc35viks2b.onion
addnode=kvd44sw7skb5folw.onion
addnode=nkf5e6b7pl4jfd4a.onion
addnode=yu7sezmixhmyljn4.onion
addnode=3ffk7iumtx3cegbi.onion
addnode=3nmbbakinewlgdln.onion
addnode=4j77gihpokxu2kj4.onion
addnode=546esc6botbjfbxb.onion
addnode=5at7sq5nm76xijkd.onion
addnode=77mx2jsxaoyesz2p.onion
addnode=7g7j54btiaxhtsiy.onion
addnode=a6obdgzn67l7exu3.onion
addnode=ab64h7olpl7qpxci.onion
addnode=am2a4rahltfuxz6l.onion
addnode=azuxls4ihrr2mep7.onion
addnode=bitcoin7bi4op7wb.onion
addnode=bitcoinostk4e4re.onion
addnode=bk7yp6epnmcllq72.onion
addnode=bmutjfrj5btseddb.onion
addnode=ceeji4qpfs3ms3zc.onion
addnode=clexmzqio7yhdao4.onion
addnode=gb5ypqt63du3wfhn.onion
addnode=h2vlpudzphzqxutd.onion
addnode=n42h7r6oumcfsbrs.onion:4176
addnode=ncwk3lutemffcpc4.onion
addnode=okdzjarwekbshnof.onion
addnode=pjghcivzkoersesd.onion
addnode=rw7ocjltix26mefn.onion
addnode=uws7itep7o3yinxo.onion
addnode=vk3qjdehyy4dwcxw.onion
addnode=vqpye2k5rcqvj5mq.onion
addnode=wpi7rpvhnndl52ee.onion
|
|
|
|
It only happens to me when my node is catching-up to the tip of the blockchain.
e.g.: when I just opened Bitcoin Core after a day/few hours of downtime.
If it's not the case, you can enable advance debugging with "rpc" flag, reproduce the issue and then consult your debug.log file for related entries.
e.g., start bitcoind with: bitcoind -debug=rpc
Ok no it is not the case. Also I noticed that I cannot shutdown the process gracefully, it hangs forever in b-shutoff, I have to hard kill it, even if I have dbcache set to only 300. I will try rpc debugging. |
|
|
|
|
Lately I have been noticing that my node gets stuck for a very long time with some commands, e.g.:
bitcoin-cli getrpcinfo
{
"active_commands": [
{
"method": "getmempoolentry",
"duration": 6030622998
}
]}
I also saw with with getrawtransaction. How can I diagnose what is going on exactly?
Thanks!
|
|
|
|
Ok I looked at the details using htop, and what I see is that cln's bcli is writing to the disk at 10 MB/s constantly...
To me that looks like it's still trying to do something with the initial sync, but it really should not be taking this kind of time. If it's still doing it now 15+ hours from your last post and a long time after your initial seeing of the issue. I would open an issue on github. It is obviously working since you can see the channel you created in 1ml so something else is going on. Sorry I could not help more, but this looks more like it's not a configuration issue but something withing CLN / your setup. -Dave Thanks. It ended up settling down after another day. I wish I understood more what it was doing and why it was not performing these operations in memory... Either some setting has to be tweaked to get it to use ram instead of the drive OR it was just doing some table manipulation in the database and its just reading & writing until it's done. Glad it's working. Make sure to do regular backups if you have any funds in there, RPis are good and all, but you still do not have RAID in case of a disk issue. Been there, done that. -Dave Thanks, Do you think using PostgreSQL instead of SQLite could have prevented this? Yes I got some Lexar USB sticks with a metal enclosure I will use for backup... Thanks! |
|
|
|
Ok I looked at the details using htop, and what I see is that cln's bcli is writing to the disk at 10 MB/s constantly...
To me that looks like it's still trying to do something with the initial sync, but it really should not be taking this kind of time. If it's still doing it now 15+ hours from your last post and a long time after your initial seeing of the issue. I would open an issue on github. It is obviously working since you can see the channel you created in 1ml so something else is going on. Sorry I could not help more, but this looks more like it's not a configuration issue but something withing CLN / your setup. -Dave Thanks. It ended up settling down after another day. I wish I understood more what it was doing and why it was not performing these operations in memory... |
|
|
|
These seem to all apply to LND and not CLN though, correct? I can open an issue on the CLN's repository.
Sorry yeah, I am remote into my desktop that is logged into the forum and the last part "There might be something similar in CLightning" did not make it. Ok, so it looks like this could be all normal and due to the fact that my LN node is new. It is supposed to calm down...
At least for the CLN part, I'm not sure why core is using anything. Mine only has activity when blocks are found and even then not much. How are you powering the RPi? I have seen some 5V adapters cause issues. This is one of my favorites.... 5% over voltage with no load, add a load and I can get it to 5.5V / 10% over voltage, then bad things tm happen.  -Dave Do you recall having your service using a significant amount of CPU and IO for a few days initially? When comparing to downloading and verifying the blockchain, CLN does not seem to be downloading much data or using much disk space, but it saturates the SSD IO and puts significant load on the CPU. My whole bitcoin directory for CLN currently uses only 82M of disk space. I see about 20 KB/s or less of up or down traffic for the node. lightningd at 36% and bitcoind at 110% CPU core usage. I have been using the official 15W power supply. I have not had any obvious issue with it. What do you have exactly? I have not done any overclocking for my node, but as I mentioned got UASP and trim working for the SSD, and I use it for the OS as well, I do not use any SD card, I activated noatime. I would expect IO performance to be quite optimized for my RPI.
Maybe I should bump up bitcoind's dbcache temporarily to help CLN syncing faster with less IO operations? I had dbcache configured at 7368 MB when I downloaded and verified the blockchain, but since I brought it down to 300 MB... Maybe it is bitcoind that is currently creating the IO strain and not lightningd...
I will try increasing dbcache to 5000
Ok I looked at the details using htop, and what I see is that cln's bcli is writing to the disk at 10 MB/s constantly... [moderator's note: consecutive posts merged] |
|
|
|
|
