Bitcoin Forum
July 23, 2024, 09:30:27 AM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
  Home Help Search Login Register More  
  Show Posts
Pages: [1] 2 »
1  Bitcoin / Development & Technical Discussion / ZMQ raw block questions on: November 12, 2020, 10:27:27 PM
I have two questions regarding ZMQ raw block notifications:

1 - The default high watermark for bitcoin core is 1000. If I understand correctly this means that up to 1000 raw blocks can be stored in memory (send buffer), which could translate to approximately 2GB, if there is a subscriber who forgets to read the notifications. This may lead to a crash due to insufficient memory in many machines, so it would be safer to set it to 100 and that would still allow the notification subscriber to be ~14h late in reading them, correct?

2 - It seems like raw block notifications are not published twice for the same height. An issue was just published on the bitcoin repository, but there are no comments yet: https://github.com/bitcoin/bitcoin/issues/20376 . Is this expected behavior or a bug?
2  Bitcoin / Development & Technical Discussion / Re: Electrumx not updating with mempool transactions on: August 25, 2020, 12:31:38 AM
Yes, using newer versions of Electrumx is out of the question and I am looking for an alternative. Currently, electrs seems like a good candidate but I need stuff issues like this https://github.com/romanz/electrs/issues/79 solved!

I need to query a lot of scriptPubKeys are not necessarily mine (like a block explorer). Any suggestions are greatly appreciated! Maybe I will try to get in touch with some block explorers and know owners of electrum servers and ask them what they are using (and if they are using electrumx, what they will switch to).
3  Bitcoin / Development & Technical Discussion / Re: Electrumx not updating with mempool transactions on: August 22, 2020, 05:36:36 PM
EPS just keeps track of a set of scriptPubKeys associated with the user's seed, correct? I am not necessarily looking for something lightweight, I need something that can answer a lot of queries about any scriptPubKey fast.

Also, what exactly do you mean by "very bad at handling mempool transactions"? Huh

I mean stuff like this https://github.com/romanz/electrs/issues/79
4  Bitcoin / Development & Technical Discussion / Re: Electrumx not updating with mempool transactions on: August 20, 2020, 11:26:47 PM
Based on average read speed, do you use USB 2 (whether the port is USB 2 or the external HDD uses USB 2) ?

I think it's USB 3 (the external HDD is the Seagate device - bus 004)

Code:
>lsusb
Bus 002 Device 002: ID 8087:8000 Intel Corp.
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 007: ID 0bc2:231a Seagate RSS LLC
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID 138a:0017 Validity Sensors, Inc. Fingerprint Reader
Bus 003 Device 010: ID 0458:0186 KYE Systems Corp. (Mouse Systems)
Bus 003 Device 002: ID 1058:10b8 Western Digital Technologies, Inc. Elements Portable (WDBU6Y, WDBUZG)
Bus 003 Device 016: ID 1b3f:2008 Generalplus Technology Inc.
Bus 003 Device 005: ID 04ca:7035 Lite-On Technology Corp.
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

May not get any answers due to the ElectrumX dev committing to BSV, but have you tried logging an issue on the ElectrumX GitHub? Huh (or maybe the fork of ElectrumX that is still supporting BTC)

I know... I guess electrs will take the lead But last time I checked electrs was very bad at handling mempool transactions... any other electrum server implementation besides electrumX or electrs?
5  Bitcoin / Development & Technical Discussion / Re: Electrumx not updating with mempool transactions on: August 20, 2020, 02:32:11 AM
I changed from a 5 year-old 2TB external HDD to a recent 4TB external HDD and I think the problem is still there. I haven't experimented much today... I have already spent ~ 3 mBTC in the past two days in bitcoin fees Tongue

I ran "iotop -a" and waited for the numbers to stabilize. The result is:
https://i.imgur.com/vJu9ZTP.png 

I realised that "iostat" gives the average I/O stats and not the instantaneous ones. I moved a large file from the external HDD into my laptop's SSD and saw that it averaged around 30 MB/s

Code:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.3%    0.0%    3.2%   24.5%    0.0%   68.1%

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sde
                293.00        36.0M         0.0k      36.0M       0.0k

If the problem is I/O speed from the drive, I am not worried because I will launch my app on AWS with an SSD... I am more concerned about excluding other causes.

6  Bitcoin / Development & Technical Discussion / Re: Electrumx not updating with mempool transactions on: August 18, 2020, 02:35:46 PM
This issue is occuring locally on my laptop (Lenovo Thinkpad W541)  which is connected to an external HDD with 2T that stores the bitcoin blockchain and eletrumx data.

My laptop specs:
Ubuntu 18.04.4
16 GB RAM/ Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz/ ~100 GB partition SSD (using a HDD with 2T to store the blockchain/electrumx data)

I have some open apps running in the background like chrome, telegram, keybase, pycharm, IntelliJ ... I don't see any specific app consuming a lot of resources.

Here is the output of iostat when the problem occurred:

Code:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          32.1%    0.0%    9.0%    7.2%    0.0%   51.7%

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
loop0
                  0.00         0.0k         0.0k       6.3M       0.0k
loop1
                  0.00         0.0k         0.0k       3.7M       0.0k
loop2
                  0.00         0.0k         0.0k     111.0k       0.0k
loop3
                  0.01         0.0k         0.0k      10.8M       0.0k
loop4
                  0.00         0.0k         0.0k     118.0k       0.0k
loop5
                  0.00         0.0k         0.0k       5.6M       0.0k
loop6
                  0.00         0.0k         0.0k     813.0k       0.0k
loop7
                  0.00         0.0k         0.0k      44.0k       0.0k
sda
                 17.17       151.9k       162.7k     163.9G     175.6G
loop8
                  0.00         0.0k         0.0k       5.6M       0.0k
loop9
                  0.01         0.0k         0.0k       6.5M       0.0k
loop10
                  0.53         0.5k         0.0k     588.2M       0.0k
loop11
                  0.00         0.0k         0.0k       6.4M       0.0k
loop12
                  0.00         0.0k         0.0k      78.0k       0.0k
loop13
                  0.01         0.0k         0.0k       6.6M       0.0k
loop14
                  0.00         0.0k         0.0k       3.3M       0.0k
loop15
                  0.00         0.0k         0.0k     192.0k       0.0k
loop16
                  0.00         0.0k         0.0k     758.0k       0.0k
loop17
                  0.00         0.0k         0.0k       2.5M       0.0k
loop18
                  0.00         0.0k         0.0k       3.5M       0.0k
loop19
                  0.01         0.0k         0.0k       8.0M       0.0k
loop20
                  0.17         0.2k         0.0k     183.5M       0.0k
loop21
                  0.15         0.2k         0.0k     169.5M       0.0k
loop22
                  0.21         0.2k         0.0k     231.2M       0.0k
loop23
                  0.07         0.1k         0.0k      80.3M       0.0k
loop24
                  0.00         0.0k         0.0k       6.0M       0.0k
loop25
                  0.00         0.0k         0.0k       5.7M       0.0k
loop26
                  0.00         0.0k         0.0k       5.6M       0.0k
loop27
                  0.01         0.0k         0.0k       6.7M       0.0k
loop28
                  0.00         0.0k         0.0k     493.0k       0.0k
loop29
                  0.00         0.0k         0.0k      46.0k       0.0k
sdb
                 34.03         1.3M       504.2k       1.5T     544.1G
loop30
                  0.05         0.1k         0.0k      61.0M       0.0k
loop31
                  0.00         0.0k         0.0k       8.0k       0.0k
7  Bitcoin / Development & Technical Discussion / Electrumx not updating with mempool transactions on: August 17, 2020, 09:24:59 PM
I'm running a bitcoin full node (v0.20.0) with an electrumx server (version 1.13.0) on top and I noticed that electrumx sometimes is veryyy slow to update itself with new transactions. What I am doing is: create a new signed transaction, broadcast it, make a CPFP transaction and broadcast it too. Then, I try to see if electrumx can detect the child transaction by querying electrumx for the scriptPubKeys of the parent transaction and see if anything is spending it. Around 2/3 of the times, electrumx updates quickly and can detect the child transaction but around 1/3 of the times it takes many minutes, like 15 minutes for it to detect and answer this child transaction. I have done this experiment like 9 times. I don't see any issues with my bitcoin node, its mempool contains the child transaction and when I run

Code:
bitcoin-cli getrawmempool true | jq '."parent_txid"'

I see the child txid in the "spentby" field.


Here is my electrumx.conf configuration:

Code:
DB_DIRECTORY = /media/pedro/Portable_2TB/Bitcoin/electrumx/data
DAEMON_URL = http://user:***@127.0.0.1
COIN = "BitcoinSegwit"
NET = "mainnet"
ALLOW_ROOT=1

PEER_DISCOVERY= self
SERVICES = rpc://localhost:8000,tcp://localhost:50001

INITIAL_CONCURRENT=500
COST_SOFT_LIMIT=10000000
COST_HARD_LIMIT=10000000
BANDWIDTH_UNIT_COST=1


NOTE: my bitcoin node gives a weird answer to `bitcoin-cli getchaintips` but looks good otherwise. Here is the answer:

Code:
>bitcoin-cli getchaintips
[
  {
    "height": 644185,
    "hash": "00000000000000000002fc463486fa70cc80cecdb0ebdfd8f7e2f056cbeb929c",
    "branchlen": 0,
    "status": "active"
  },
  {
    "height": 638519,
    "hash": "00000000000000000011398b899fb881c61bd6e8670270fbc8e225c11fd95d26",
    "branchlen": 1,
    "status": "valid-fork"
  }
]

I tried restarting the node and running 'bitcoin-cli reconsiderblock 00000000000000000011398b899fb881c61bd6e8670270fbc8e225c11fd95d26' but it didn't solve, could this be related? It's working fine otherwise it seems.
8  Bitcoin / Development & Technical Discussion / Re: Selfish full node for production? on: August 12, 2020, 09:32:39 PM
Yes it would. maxuploadtarget only restricts fetching historical blocks, it won't restrict anything about you sending transactions.
That's great, I think I will use it then.

Yes, enable pruning. You can set the pruning limit so high that nothing actually gets pruned, but you'll still signal yourself as pruned to the network. An option should probably get added to do this more directly, you should open a feature request for a "-nodelimited=1" option.
I am using txindex and according to the command line help, it's not compatible with prunning.

What you signal is mostly moot however, if you're not even listening for connections from outside.
Wouldn't my outbout 8 peers receive whatever I broadcast with sendrawtransaction? I will set maxconnections to 20 then, just to be safe.

Edit: Newbie question, how to add code in the wiki's markdown?
Is this what you mean?
9  Bitcoin / Development & Technical Discussion / Re: Selfish full node for production? on: August 12, 2020, 02:55:39 PM
Thank you for all the suggestions. I think I pulled together those that work best for. Here is how I am setting up my bitcoin.conf:

Code:
maxconnections=8 // no more than the 8 outbound connections my node will attempt
addnode=123.123.123.123 // my external node that I am sure it will always be up
addnode-122.122.122.122 // some node geographically close to me with an up-to-date block height (maybe repeat this step)

peerbloomfilters=0 // disable SPV clients from doing block filtering
blockfilterindex=0 // I think this is unnecessary because it's the default, but just making sure


Setting a low with -maxuploadtarget won't work for me because my application will broadcast many transactions (possibly new to the network), so it's very important that these broadcasts are done properly.
I also don't want to prune because I need the whole tx history.

@JuleAdka suggestion seems interesting. I took a look at BIP 159 (https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki) which introduced NODE_NETWORK_LIMITED. Disabling NODE_NETWORK might be a good way to make sure nobody tries to download historical blocks from my node. Is there a way to disable this service flag? Searched through the options and didn't find a way to do it (bitcoind --help | grep "service").
10  Bitcoin / Development & Technical Discussion / Selfish full node for production? on: August 11, 2020, 08:21:09 PM
I am planning about launching a bandwidth-intensive service. My application will be running on a server with a full node, and I was wondering if I could disable peers from downloading historical blocks from my node to optimize the bandwidth for my application. Actually, stopping them from downloading any block from my node would also work Tongue

I know many of you will not like this idea because it's selfish, but I already run a second, sometimes third, node. I am aware of the "blocksonly" mode, but it's not an option as I need a mempool.
11  Bitcoin / Development & Technical Discussion / Re: Joining mempool RBF transactions on: August 11, 2020, 07:21:18 PM
Except "batch on replacement" is ludicrously hard to do.
That's why I say they should batch in the first place. Smiley

Quote
I actually think it's the hardest (pure) programming problem I've worked on, and don't feel like I'd even be able to do it given another 6 months. I've never done "logic programming" but I almost feel like something like that would be essential, where you sort of logically describe all the high level concepts and ask it to solve what you should do. As just trying to handle all the cases imperatively just seems impossible to not end up in an exploded spaghetti nightmare of a gazillion states.
[Bit off topic, just ranting here incase you have any insights]
That was all I had to offer. I think the right way to solve that isn't to write code for it, it's to write (or steal) a logic-relation engine.

In particular handling all the cases were an earlier partial payment confirms, and then making sure that your follwup payment conflicts with the earlier complete payments (either by being a child of the partial or more directly) ... just a gnarly mess.

For most people the best advise right now is batch in the first place.

@RHavar may I see what you have tried? I'm interested in this subject. My name for it was "dynamic batching".
12  Bitcoin / Development & Technical Discussion / Re: BIP numbering and status mess on: July 13, 2020, 01:47:34 PM
But BIP 174 was merged like two years. It's about PSBT. There are a bunch of RPC commands to make PSBTs and all. For a reader interested in informing himself on what BIPs made it to the bitcoin core code, he would just ignore BIP 174. Even if there are some details that are left to be implemented, there should be a status like "Partially merged" or some indication that the BIP has made it into the codebase.
13  Bitcoin / Development & Technical Discussion / BIP numbering and status mess on: July 13, 2020, 01:13:48 PM
There is BIP 199 and then the next BIP number is 300. why? These BIP numbers seem so random.

Also, the status of the BIPs could seem a bit outdated, no? For example, BIP 174 says "proposed" but it should be "final", no? It has been merged about two years ago...

https://github.com/bitcoin/bips
14  Bitcoin / Development & Technical Discussion / Re: Insure GUI - create a locked sweeping transaction for life insurance on: July 12, 2020, 11:10:24 PM
What alternative do you propose? Leave the private key encrypted with the heir PGP private key somewhere in the house, to be found when you die?
15  Bitcoin / Development & Technical Discussion / Re: Insure GUI - create a locked sweeping transaction for life insurance on: July 12, 2020, 10:16:02 PM
true, but if you are making a 'will' and have the coins in cold storage it's highly likely that you won't spend them.
16  Bitcoin / Development & Technical Discussion / Insure GUI - create a locked sweeping transaction for life insurance on: July 12, 2020, 09:12:36 PM
A cool project was posted in /r/bitcoin today: https://github.com/lacksfish/insure-gui (original post: https://www.reddit.com/r/Bitcoin/comments/hpyfjt/ive_built_a_tool_making_it_easy_to_inherit/)

It's an easy way to create a sweeping transaction from a Ledger device that is only valid no sooner than one year from the moment of creation (user can set this parameter). The application then creates a pdf which you can email to your heir.

The main disadvantage is that you have to move the funds to another address of yours to invalidate that sweeping transaction. Any similar projects like this existed before? Thoughts?
17  Bitcoin / Development & Technical Discussion / Questions about generic signmessage (BIP322) on: July 11, 2020, 07:45:44 PM
I have two questions regarding the generalization of message signing/verification that has until recently only been done with legacy addresses through the RPC commands signmessage/verifymessage.

1 - Why was it chosen to prepend the scriptPubKey (when not P2PKH) to the preimage? To me, this adds unnecessary complexity which will slow down the adoption of BIP 322 by bitcoin libraries.

2 - In https://bitcoinops.org/en/topics/generic-signmessage , it can be read "This means a signed message can be produced for any script or address that a wallet would be able to spend. Additionally, two or more wallets can cooperate to create a BIP322 signed message for multisig scripts.". Additionally, BIP322 states that in order to sign: (https://github.com/bitcoin/bips/blob/master/bip-0322.mediawiki#signing) "Derive the private key privkey for the scriptPubKey". I just don't know how this is possible, it should be impossible to derive a private key from a scriptPubKey of a multisig for example. For a multisig there are a set of private and public keys of the owners, can there be a single private key of the whole multisig that can sign anything without the other's approval? And even if yes, how can the public key of this derived master multisig private key match the public key hash which is a hash of the locking script?
18  Bitcoin / Development & Technical Discussion / Re: Hidden bitcoin core RPC commands on: July 01, 2020, 07:38:51 PM
Thank you both!
19  Bitcoin / Development & Technical Discussion / Hidden bitcoin core RPC commands on: July 01, 2020, 05:31:49 PM
Recently I learned about
Code:
bitcoin-cli reconsiderblock <block_hash>
, which saved me from having to start the synchronization from scratch (was getting constantly ERROR: AcceptBlockHeader: block 00.. is marked invalid and the blockheader had a invalid difficulty).

Does anybody know about other hidden RPC commands that are not listed under
Code:
bitcoin-cli help
?
20  Bitcoin / Press / Re: [2018-07-01] Bacloud accepts Bitcoin payments through the Lightning Network on: July 02, 2018, 09:14:34 PM
Added to http://lightningnetworkstores.com
Pages: [1] 2 »
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!