 Show Posts
|
|
Pages: [1] 2 »
|
I have two questions regarding ZMQ raw block notifications: 1 - The default high watermark for bitcoin core is 1000. If I understand correctly this means that up to 1000 raw blocks can be stored in memory (send buffer), which could translate to approximately 2GB, if there is a subscriber who forgets to read the notifications. This may lead to a crash due to insufficient memory in many machines, so it would be safer to set it to 100 and that would still allow the notification subscriber to be ~14h late in reading them, correct? 2 - It seems like raw block notifications are not published twice for the same height. An issue was just published on the bitcoin repository, but there are no comments yet: https://github.com/bitcoin/bitcoin/issues/20376 . Is this expected behavior or a bug? |
|
|
|
Yes, using newer versions of Electrumx is out of the question and I am looking for an alternative. Currently, electrs seems like a good candidate but I need stuff issues like this https://github.com/romanz/electrs/issues/79 solved! I need to query a lot of scriptPubKeys are not necessarily mine (like a block explorer). Any suggestions are greatly appreciated! Maybe I will try to get in touch with some block explorers and know owners of electrum servers and ask them what they are using (and if they are using electrumx, what they will switch to). |
|
|
|
EPS just keeps track of a set of scriptPubKeys associated with the user's seed, correct? I am not necessarily looking for something lightweight, I need something that can answer a lot of queries about any scriptPubKey fast. Also, what exactly do you mean by "very bad at handling mempool transactions"?  I mean stuff like this https://github.com/romanz/electrs/issues/79 |
|
|
|
Based on average read speed, do you use USB 2 (whether the port is USB 2 or the external HDD uses USB 2) ?
I think it's USB 3 (the external HDD is the Seagate device - bus 004) >lsusb
Bus 002 Device 002: ID 8087:8000 Intel Corp.
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 007: ID 0bc2:231a Seagate RSS LLC
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID 138a:0017 Validity Sensors, Inc. Fingerprint Reader
Bus 003 Device 010: ID 0458:0186 KYE Systems Corp. (Mouse Systems)
Bus 003 Device 002: ID 1058:10b8 Western Digital Technologies, Inc. Elements Portable (WDBU6Y, WDBUZG)
Bus 003 Device 016: ID 1b3f:2008 Generalplus Technology Inc.
Bus 003 Device 005: ID 04ca:7035 Lite-On Technology Corp.
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub May not get any answers due to the ElectrumX dev committing to BSV, but have you tried logging an issue on the ElectrumX GitHub? (or maybe the fork of ElectrumX that is still supporting BTC) I know... I guess electrs will take the lead But last time I checked electrs was very bad at handling mempool transactions... any other electrum server implementation besides electrumX or electrs? |
|
|
|
I changed from a 5 year-old 2TB external HDD to a recent 4TB external HDD and I think the problem is still there. I haven't experimented much today... I have already spent ~ 3 mBTC in the past two days in bitcoin fees  I ran "iotop -a" and waited for the numbers to stabilize. The result is: https://i.imgur.com/vJu9ZTP.png I realised that "iostat" gives the average I/O stats and not the instantaneous ones. I moved a large file from the external HDD into my laptop's SSD and saw that it averaged around 30 MB/s avg-cpu: %user %nice %system %iowait %steal %idle
4.3% 0.0% 3.2% 24.5% 0.0% 68.1%
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sde
293.00 36.0M 0.0k 36.0M 0.0k If the problem is I/O speed from the drive, I am not worried because I will launch my app on AWS with an SSD... I am more concerned about excluding other causes. |
|
|
|
This issue is occuring locally on my laptop (Lenovo Thinkpad W541) which is connected to an external HDD with 2T that stores the bitcoin blockchain and eletrumx data. My laptop specs: Ubuntu 18.04.4 16 GB RAM/ Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz/ ~100 GB partition SSD (using a HDD with 2T to store the blockchain/electrumx data) I have some open apps running in the background like chrome, telegram, keybase, pycharm, IntelliJ ... I don't see any specific app consuming a lot of resources. Here is the output of iostat when the problem occurred: avg-cpu: %user %nice %system %iowait %steal %idle
32.1% 0.0% 9.0% 7.2% 0.0% 51.7%
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
loop0
0.00 0.0k 0.0k 6.3M 0.0k
loop1
0.00 0.0k 0.0k 3.7M 0.0k
loop2
0.00 0.0k 0.0k 111.0k 0.0k
loop3
0.01 0.0k 0.0k 10.8M 0.0k
loop4
0.00 0.0k 0.0k 118.0k 0.0k
loop5
0.00 0.0k 0.0k 5.6M 0.0k
loop6
0.00 0.0k 0.0k 813.0k 0.0k
loop7
0.00 0.0k 0.0k 44.0k 0.0k
sda
17.17 151.9k 162.7k 163.9G 175.6G
loop8
0.00 0.0k 0.0k 5.6M 0.0k
loop9
0.01 0.0k 0.0k 6.5M 0.0k
loop10
0.53 0.5k 0.0k 588.2M 0.0k
loop11
0.00 0.0k 0.0k 6.4M 0.0k
loop12
0.00 0.0k 0.0k 78.0k 0.0k
loop13
0.01 0.0k 0.0k 6.6M 0.0k
loop14
0.00 0.0k 0.0k 3.3M 0.0k
loop15
0.00 0.0k 0.0k 192.0k 0.0k
loop16
0.00 0.0k 0.0k 758.0k 0.0k
loop17
0.00 0.0k 0.0k 2.5M 0.0k
loop18
0.00 0.0k 0.0k 3.5M 0.0k
loop19
0.01 0.0k 0.0k 8.0M 0.0k
loop20
0.17 0.2k 0.0k 183.5M 0.0k
loop21
0.15 0.2k 0.0k 169.5M 0.0k
loop22
0.21 0.2k 0.0k 231.2M 0.0k
loop23
0.07 0.1k 0.0k 80.3M 0.0k
loop24
0.00 0.0k 0.0k 6.0M 0.0k
loop25
0.00 0.0k 0.0k 5.7M 0.0k
loop26
0.00 0.0k 0.0k 5.6M 0.0k
loop27
0.01 0.0k 0.0k 6.7M 0.0k
loop28
0.00 0.0k 0.0k 493.0k 0.0k
loop29
0.00 0.0k 0.0k 46.0k 0.0k
sdb
34.03 1.3M 504.2k 1.5T 544.1G
loop30
0.05 0.1k 0.0k 61.0M 0.0k
loop31
0.00 0.0k 0.0k 8.0k 0.0k
|
|
|
|
I'm running a bitcoin full node (v0.20.0) with an electrumx server (version 1.13.0) on top and I noticed that electrumx sometimes is veryyy slow to update itself with new transactions. What I am doing is: create a new signed transaction, broadcast it, make a CPFP transaction and broadcast it too. Then, I try to see if electrumx can detect the child transaction by querying electrumx for the scriptPubKeys of the parent transaction and see if anything is spending it. Around 2/3 of the times, electrumx updates quickly and can detect the child transaction but around 1/3 of the times it takes many minutes, like 15 minutes for it to detect and answer this child transaction. I have done this experiment like 9 times. I don't see any issues with my bitcoin node, its mempool contains the child transaction and when I run bitcoin-cli getrawmempool true | jq '."parent_txid"' I see the child txid in the "spentby" field. Here is my electrumx.conf configuration: DB_DIRECTORY = /media/pedro/Portable_2TB/Bitcoin/electrumx/data
DAEMON_URL = http://user:***@127.0.0.1
COIN = "BitcoinSegwit"
NET = "mainnet"
ALLOW_ROOT=1
PEER_DISCOVERY= self
SERVICES = rpc://localhost:8000,tcp://localhost:50001
INITIAL_CONCURRENT=500
COST_SOFT_LIMIT=10000000
COST_HARD_LIMIT=10000000
BANDWIDTH_UNIT_COST=1 NOTE: my bitcoin node gives a weird answer to `bitcoin-cli getchaintips` but looks good otherwise. Here is the answer: >bitcoin-cli getchaintips
[
{
"height": 644185,
"hash": "00000000000000000002fc463486fa70cc80cecdb0ebdfd8f7e2f056cbeb929c",
"branchlen": 0,
"status": "active"
},
{
"height": 638519,
"hash": "00000000000000000011398b899fb881c61bd6e8670270fbc8e225c11fd95d26",
"branchlen": 1,
"status": "valid-fork"
}
] I tried restarting the node and running 'bitcoin-cli reconsiderblock 00000000000000000011398b899fb881c61bd6e8670270fbc8e225c11fd95d26' but it didn't solve, could this be related? It's working fine otherwise it seems. |
|
|
|
Yes it would. maxuploadtarget only restricts fetching historical blocks, it won't restrict anything about you sending transactions. That's great, I think I will use it then. Yes, enable pruning. You can set the pruning limit so high that nothing actually gets pruned, but you'll still signal yourself as pruned to the network. An option should probably get added to do this more directly, you should open a feature request for a "-nodelimited=1" option. I am using txindex and according to the command line help, it's not compatible with prunning. What you signal is mostly moot however, if you're not even listening for connections from outside. Wouldn't my outbout 8 peers receive whatever I broadcast with sendrawtransaction? I will set maxconnections to 20 then, just to be safe. Edit: Newbie question, how to add code in the wiki's markdown? Is this what you mean?  |
|
|
|
Thank you for all the suggestions. I think I pulled together those that work best for. Here is how I am setting up my bitcoin.conf: maxconnections=8 // no more than the 8 outbound connections my node will attempt
addnode=123.123.123.123 // my external node that I am sure it will always be up
addnode-122.122.122.122 // some node geographically close to me with an up-to-date block height (maybe repeat this step)
peerbloomfilters=0 // disable SPV clients from doing block filtering
blockfilterindex=0 // I think this is unnecessary because it's the default, but just making sure
Setting a low with -maxuploadtarget won't work for me because my application will broadcast many transactions (possibly new to the network), so it's very important that these broadcasts are done properly. I also don't want to prune because I need the whole tx history. @JuleAdka suggestion seems interesting. I took a look at BIP 159 ( https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki) which introduced NODE_NETWORK_LIMITED. Disabling NODE_NETWORK might be a good way to make sure nobody tries to download historical blocks from my node. Is there a way to disable this service flag? Searched through the options and didn't find a way to do it (bitcoind --help | grep "service"). |
|
|
|
I am planning about launching a bandwidth-intensive service. My application will be running on a server with a full node, and I was wondering if I could disable peers from downloading historical blocks from my node to optimize the bandwidth for my application. Actually, stopping them from downloading any block from my node would also work I know many of you will not like this idea because it's selfish, but I already run a second, sometimes third, node. I am aware of the "blocksonly" mode, but it's not an option as I need a mempool. |
|
|
|
Except "batch on replacement" is ludicrously hard to do.
That's why I say they should batch in the first place.  I actually think it's the hardest (pure) programming problem I've worked on, and don't feel like I'd even be able to do it given another 6 months. I've never done "logic programming" but I almost feel like something like that would be essential, where you sort of logically describe all the high level concepts and ask it to solve what you should do. As just trying to handle all the cases imperatively just seems impossible to not end up in an exploded spaghetti nightmare of a gazillion states.
[Bit off topic, just ranting here incase you have any insights]
That was all I had to offer. I think the right way to solve that isn't to write code for it, it's to write (or steal) a logic-relation engine. In particular handling all the cases were an earlier partial payment confirms, and then making sure that your follwup payment conflicts with the earlier complete payments (either by being a child of the partial or more directly) ... just a gnarly mess. For most people the best advise right now is batch in the first place. @RHavar may I see what you have tried? I'm interested in this subject. My name for it was "dynamic batching". |
|
|
|
|
But BIP 174 was merged like two years. It's about PSBT. There are a bunch of RPC commands to make PSBTs and all. For a reader interested in informing himself on what BIPs made it to the bitcoin core code, he would just ignore BIP 174. Even if there are some details that are left to be implemented, there should be a status like "Partially merged" or some indication that the BIP has made it into the codebase.
|
|
|
|
There is BIP 199 and then the next BIP number is 300. why? These BIP numbers seem so random. Also, the status of the BIPs could seem a bit outdated, no? For example, BIP 174 says "proposed" but it should be "final", no? It has been merged about two years ago... https://github.com/bitcoin/bips |
|
|
|
|
What alternative do you propose? Leave the private key encrypted with the heir PGP private key somewhere in the house, to be found when you die?
|
|
|
|
|
true, but if you are making a 'will' and have the coins in cold storage it's highly likely that you won't spend them.
|
|
|
|
I have two questions regarding the generalization of message signing/verification that has until recently only been done with legacy addresses through the RPC commands signmessage/verifymessage. 1 - Why was it chosen to prepend the scriptPubKey (when not P2PKH) to the preimage? To me, this adds unnecessary complexity which will slow down the adoption of BIP 322 by bitcoin libraries. 2 - In https://bitcoinops.org/en/topics/generic-signmessage , it can be read "This means a signed message can be produced for any script or address that a wallet would be able to spend. Additionally, two or more wallets can cooperate to create a BIP322 signed message for multisig scripts.". Additionally, BIP322 states that in order to sign: ( https://github.com/bitcoin/bips/blob/master/bip-0322.mediawiki#signing) "Derive the private key privkey for the scriptPubKey". I just don't know how this is possible, it should be impossible to derive a private key from a scriptPubKey of a multisig for example. For a multisig there are a set of private and public keys of the owners, can there be a single private key of the whole multisig that can sign anything without the other's approval? And even if yes, how can the public key of this derived master multisig private key match the public key hash which is a hash of the locking script? |
|
|
|
Recently I learned about bitcoin-cli reconsiderblock <block_hash> , which saved me from having to start the synchronization from scratch (was getting constantly ERROR: AcceptBlockHeader: block 00.. is marked invalid and the blockheader had a invalid difficulty). Does anybody know about other hidden RPC commands that are not listed under ? |
|
|
|
|
