No, it only copies what it's inside a mempool into the other.

It's doesn't transfer them, but if both nodes are already connected to the bitcoin network, both will receive the new transactions. This program is intended to get old transactions to a node from another since they are not retransmitted normally by the bitcoin network. Anyway in a future, a ZMQ channel can be added to transfer them also. But be aware that it's difficult to get both mempools to be the same if there are conflicting txs with children. etc... among them (which is the normal thing by the way)
The program retransmit a snapshot of a mempool. If a block arrives while running, the evicted txs wont be retransmitted or accepted by the nodes.
Yes, there is a summary at the beginning and at the end of the execution. But again, be aware it's almost impossible to get both mempools the same.
It helps in the sense that you can shutdown one node, increase it's limits, reboot it again, and make a transfer of txs from another node to the rebooted one. The majority of the txs will be reloaded through mempool.dat file, but the program can transfer the txs that were lost while configuring the node. If any of those tx conflicts with mempool they will be rejected by the node.

I've coded mempoolcp

https://github.com/dev7ba/mempoolcp

What is mempoolcp?

Mempoolpc is a command-line program to copy the mempool from one bitcoin node to another.

How does it works?

Through bitcoin nodes rpc interface, this program uses getrawmempool(verbose), getmempoolentry, getrawtransaction and sendrawtransaction rpc calls to copy the mempool between nodes.

Mempoolpc takes into account the dependencies between transactions and the fact that you can't send a child tx before a parent tx, or a parent tx before a grandparent tx... because otherwise, the sent transactions could be denied by the receiving node.

Mempoolcp is fast, as fast as rust serde is. Also, mempoolcp use multithreading when possible.

It has two modes of operation: a faster one using more memory and a normal one using less. The faster uses getrawmempool_verbose (a heavy call that uses a lot of memory if there are many txs). and then getrawtransaction + sendrawTransaction for each transaction. The normal mode uses getrawmempool (without verbose), then getmempoolentry + getrawtransaction + sendrawTransaction for each transaction.

Configuration is done via the command line or via mempoolcp.conf in a file (to avoid using passwords in the shell). It can actively ask for the user and password if needed.

It has an option to choose network (ports): mainnet, testnet, regtest...

It is compatible with any limitancestorcount value in bitcoin.conf

Currently only support user/password authorization.

Which problems solves?

At least 3 questions in bitcoin stack exchange:

https://bitcoin.stackexchange.com/questions/63675/synchronize-mempool-between-3-nodes
https://bitcoin.stackexchange.com/questions/53638/are-there-any-ways-to-sync-mempool-from-another-nodes-faster
https://bitcoin.stackexchange.com/questions/93231/how-to-achieve-fast-mempool-synchronization-at-startup

You have complete self hosting instructions in https://github.com/mempoolexplorer/mempoolexplorer There's a section about bitcoin.conf

Let me know if you had any problems following the instructions to change them accordingly. Thanks.


Yep!, a point to be aware of.

Thanks DaveF. Yesterday I was fighting with the impostor syndrome.

Hi there.

I have calculated the average profit because of txs fees (no subsidy) per block , for each miner, in the last 4 months.

You can find it here:

https://mempoolexplorer.com/miner

You can order ascending or descending by column, and change values to USD,BTC or Sats.

I was expecting for all pools having a decent % mining share (Binance, AntPool, ViaBTC, F2Pool, Slush, Foundry USA and Poolin) to have this value more or less the same. I was expecting this because mining activity is independent of mempool size, and all miners on average will have the same luck obtaining tx fees during a long period of time.

But I found to have almost a 15% variance between mining pools. (rejecting small pools as btc.com, ultimus, 1Thash... because not having enough data to be representative).

The causes of this can be:

- Optimizations in txs selection algorithm by pool operatos-> See https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md (less than 0,2% improvement).

- Better Network connectivity among miners -> See https://www.dsn.kastel.kit.edu/bitcoin/ (Txs and Block propagation time are really small to have any impact)

- Txs included by miners themselves with no broadcast-> I do not take into account this txs by comparing against my mempool when block arrives.

- Empty blocks-> I do not take into consideration for the average the empty blocks, and anyways they are only a few.

- Transaction accelerators-> This is my main point.

Could be that txs accelerators are the main factor explaining those differences? If that is the case, Poolin.com is the most used txs accelerator since they choose much worse sat/vByte ratio txs in average than the other miners (See Average lost reward column). And they are effectively way down the ranking of on-chain profit.

Have you ever seen this statistic anywhere else?

Well, maybe it's not laziness, One must have priorities, I've posted this just in case somebody find it useful/interesting.


I think something like this is the most plausible answer, nevertheless I felt the urge to share it with the community.


Yes, I made a special effort in having a mining queue which place CPFP Transactions in it's correct place, if that weren't the case, the mining queue wouldn't be accurate. Glad to see somebody has noticed it.
I'm proud of the hierarchical visual representation in https://mempoolexplorer.com/mempool and in the graphical representation of dependencies between transactions, but the algorithm I've used it's the same as in bitcoind getBlockTemplate but generalized for the entire mempool, not only the first block. You can check it at https://github.com/mempoolexplorer/txMemPool/blob/master/src/main/java/com/mempoolexplorer/txmempool/entites/miningqueue/MiningQueue.java

As I've stated in https://mempoolexplorer.com/faq#aims this begun as a side project to learn microservices in java using spring, be aware of the amount of services needed to make it work (i.e. administration server, configuration server, gateway service... etc). I think you can ignore all that paraphernalia and focus on the code I've pointed you to. I'm planning to remove the microservices architecture sooner than later anyway...

If you are a corporation/individual you should run your own bitcoin node as they do not require trust.

But if you want to play with bitcoin rpc without installing a node you can use this page

https://chainquery.com/bitcoin-cli#

Yes, I know, The distinction on the webpage is because "linear" chains are pretty common, at first I thought they were CPFP to the extreme, so I colored the transactions from red to green to distinguish this case. But I found a lot of chains of this kind with transactions having two outputs: one being a OP_RETURN and the other the utxo dependency of another transaction. Maybe you can get some of them just now, they are pretty common.


Yes, something more to the todo list. But not sure if worthy. Maybe just for fun.

Indeed, Txs with red color has the most sat/Vbyte and the green the less.

Thanks a lot for your guesswork. Not really sure for what could this be useful anyway.

Two months ago, while developing my webpage about mempool statistics, I found that there was a huge dependency graph having more than 700 transactions in (my) mempool.
I could capture some minor examples having around 470 transactions (see below)

Of course no mempool rule is broken, i.e. max mempool ancestors for a transaction, but I'm wondering about what can be doing this kind of dependency graphs.

I posted a question about it in bitcoin.stackexchange without luck:
https://bitcoin.stackexchange.com/questions/110723/huge-dependency-graphs-for-transactions-in-mempool

The best answer given to me by https://b10c.me/projects/ was that maybe there was an agent running out of utxos in its wallet, and then started using its non-confirmed utxos in the mempool, forming the dependency graphs.
If that is the case, the graphs seemed not having a structure to minimize its depth for not exceeding the max transactions ancestors rule in the mempool (i.e. a tree), and seemed pretty unstructured?. (but investigating about the best strategy in this case and recognize its visual shape is something I haven't done. Take that with a grain of salt)
So although the former answer is the most plausible, I feel it is worthy to share this with you.

What can be creating these dependency graphs?
Are they worthy to investigate or has been investigated before?

Captured transaction graphs:
https://gist.github.com/dev7ba/c144c68127b97082652bc860cc95edf6
https://gist.github.com/dev7ba/50167d6e336100698ab2599123444efc
These .json are offered by the backend of my web page you can call it here: https://mempoolexplorer.com/txmempool/miningQueueAPI/tx/{txIdHere} to obtain a transaction statistics currently in mempool.
See txDependenciesInfo.nodes and txDependenciesInfo.edges data to obtain the dependency info (if any) for queried transaction.

You can visually see the transactions graphs currently in (my) mempool here: https://mempoolexplorer.com/txsGraphs.
These graphs can be linear: forming a long chain of dependencies, or non linear: forming a Direct Acyclic Graph.
When clicked on the number of transactions on a graph, a list of the transactions ids contained in that graph is shown. You can go to the graph by clicking any of these transactions id. The result is shown in the main page along with its position in the mining queue (from my node's perspective) and other useful data.
To avoid the annoying constant refreshing please check "lock mempool" in the upper right corner.

Be aware that I currently don't have any historic section of dependencies graphs or something like that in the webpage. The results show are in real time, so if mempool is almost empty maybe no graphs or very small ones are shown.

Note that the following graphs are minor examples compared to the huge ones I'm talking about. (sorry, I didn't take a screenshot for them, only the .json returned by my backend)



Bonus Points: This webpage has other sections discussed in https://mempoolexplorer.com/faq and also here https://bitcointalk.org/index.php?topic=5383112.0 that may be of interest to you. I'd be glad receiving feedback.

Thanks Danny, I'll take this into account: https://www.viabtc.com/tools/txaccelerator/

So having into account both PrimeNumber7 and DannyHammilton answers we have these issues:

1- A pool will be paid to include a transaction with a small fee (or not fee at all)
2- A pool reserves a small percentage for low fee transactions.
3- A person mess around with a tx and do not put the charge back output into the tx. So the fees goes moon.

I'm going to deal with them:

1- That's something I've already see and asked about it: https://bitcoin.stackexchange.com/questions/93471/ive-found-two-mined-txs-with-no-fee.
But no problem: As you can see in this section https://mempoolexplorer.com/block/last/BITCOIND I can easily see the txs which has not been relayed to my node when the block arrived.
I've already discounted them in the column "Lost reward excl. not in our mempool txs" from here: https://mempoolexplorer.com/block/BITCOIND but not here yet: https://mempoolexplorer.com/miner. I'll do it.
2- If a pool reserves a small percentage for low fees tx there's nothing can be done. But I think this must be taken into the "final result" to see how much money they are spending onto this.
3- Easy: Check if a transaction deviates a lot from block average. The threshold will be high since I've seen in a regular basis tx's paying a lot more than it's needed.

I have a lot to do. since this needs more graphs and a lot of other things I need to solve first.

But I'd like somebody answer the question if running several instances in different geographical places and compare the results would be a good way to start and make the problem of block propagation time less concerning.

Thanks in advance.

Yes, before running the instance I let my bitcoin node run for some time until there are some blocks not completely full and I also check that my mempool have more or less the same number of transactions than others like this: https://mempool.observer/

This problem has caused me some headaches as you can see here: https://bitcoin.stackexchange.com/questions/99717/transaction-being-ignored-by-miners-i-mean-ignored-not-not-mined-because-low/99822?noredirect=1#comment113126_99822


As far as I know, Mining with other bitcoin node than bitcoincore is reckless since the consensus code is nothing to mess with. So I'd like to focus in pools not in implentations. Anyway, note that here: https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md you have a possible improvement over the regular greedy algorithm.

If a mempool is implementing or not a better algorithm is not worthy to know now. But maybe in the future when the fees gets higher.

Well, I started replicating the getBlockTemplate RCP from bitcoind into my code, but for the whole mempool, just to show the hierarchical representation in the first section.

Then I realized that I can compare the incoming block with my template and compare the results and search for ignored transactions in case of miners behaving wrongly. But this webpage https://miningpool.observer/ did it also, so now I can also compare it's results with mine's.

Then I realized that I can draw the transaction dependency graph and I asked this other question regarding big in-mempool transaction dependecy graphs https://bitcoin.stackexchange.com/questions/110723/huge-dependency-graphs-for-transactions-in-mempool but nobody answered.

And then I made the section we are discussing with, yes, having in mind which mempool is selecting/receiving the best transactions.

Maybe I should try to run several instances with different node geographical position to see how the miners statistics behave. What do you think?

I need to clean up the code since currently uses to much memory and needs a bitcoin node with fully downloaded blockchain to operate with, I don't want to spend too much money in the process.

This project was to learn microservices in java, and I got into the rabbit hole... so imagine...

Hello there!

I'm building a webpage: https://mempoolexplorer.com with bitcoin mempool statistics and I have some sections there that I don't really know if they are worthy so I'm asking here for help, and also, for a constructive criticism of the webpage since I'm dealing with what to do next.

I've posted this same question in bitcoin.stackexchange without luck, And I've realized that maybe is a too general question to be asked there so I'll try here:

Is there a way to know how good a miner is choosing its transanctions to be mined from a fee maximization point of view?

Well, this is the way I've tried:

When a mined block arrives to my node, I compare it against a transaction selection algorithm using my mempool. (I know my mempool it's different than the miner's one, and that's the point so keep reading ;-) )

The transaction selection algorithm sorts transactions in the usual greedy way: using the regular Ancestor Set Based (ASB) algorithm defined [here][1], and considering transaction dependencies and [CPFP][2], so basically is the same as invoking getblockTemplate with the state of my mempool just before the block arrives. (I also compare the incoming block by using the last getBlockTemplate result of bitcoind but as it's only calculated each 30s the results are not accurate).

An example of a comparison is shown [here][3], (my webpage, in development). I divide the transactions in sets whether are ignored by the miner (from my node point of view), are in common with me, Ignored by me, Relayed to me, or not relayed to me.

And, of course, I also measure the fees I'd collected against the miner's block (having into account the incoming block coin base transaction size for better accuracy). You can check the results of the last blocks [here][4], and an aggregate for the miners [here][5].

This could give me an idea about how good a miner is selecting its transactions and/or has good network connectivity (in average, and against my mempool), but as I have no way of knowing the state of the miners mempool before they mine a block (not when I receive it), this effort seems inaccurate.

I make the assumption that a received block and my block template can be compared. But as the block propagation time is not, by any means, negligible, the comparison is skewed giving the idea that my algorithm/network situation is better. The reality is that I am receiving transactions while the block is being propagated to me, and thus, I have normally a bigger mempool than the miner when the block was mined. A bigger mempool gives my algorithm the opportunity to search for better transactions which had no opportunity to be mined by miners.

So, Is the accuracy of this methodology worthy? I've assumed its not. But as block propagation time is getting smaller (see [this][6]) And the results for some blocks/miners seems really different (see [this][7] block for example) I have my doubts.

NOTE: Be aware that I've started the webpage only 2-3 hours ago so only last blocks will be shown.

Same question in stack exchange:
https://bitcoin.stackexchange.com/questions/111940/is-there-a-way-to-know-how-good-a-miner-is-choosing-its-transanctions-to-be-mine

  [1]: https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md
  [2]: https://bitcoinops.org/en/topics/cpfp/
  [3]: https://mempoolexplorer.com/block/last/OURS
  [4]: https://mempoolexplorer.com/block/OURS
  [5]: https://mempoolexplorer.com/miner
  [6]: https://www.dsn.kastel.kit.edu/bitcoin/?ref=hackernoon.com#latency
  [7]: https://mempoolexplorer.com/block/719478/BITCOIND

Yes I haven't explained myself correctly.

Thanks for the answer.

I can't figure out why it's an horrible idea, I mean, old clients could still validate transactions in 1mb block. Only newer clients will be aware of transactions in >1mb new blocks. Also, you can make addresses between the two blocks not compatible: if only bitcoins can be transferred from (current) 1mb block chain to >1mb new blockchain (and not in reverse), the amount of bitcoins in 1mb blockchain decreases as it increases in >1mb blockchain. You transfer wealth from one blockchain to the other gracefully.

When all bitcoins are transfered to the >1mb new block set, current (old) blockchain is only in charge of verifiying the proof of work and collect block rewards which can be transfered ¿immediately? to the new >1mb blockchain. To validate >1mb blocks, its merkle root hash can be included as an obligatory (via soft- fork) op_return transaction with zero fees in current blockchain. Fees for transactions in >1mb blockchain could be sent to the same coinbase address than 1mb block. It's like a one-way peg sidechain that uses the current mainchain proof of work.

Also note that bitcoin users that do not want to upgrade, still can use the 1mb blockchain if they want. They only see that an increasing amount of bitcoins are moving to (new blockchain's) addresses that can't be sent back to current blockchain because miners agreed to forbid it (via softfork)

I hope I'm explaining myself correctly. I've been thinking all day about it and I think that having old clients unaware of the other blockchain transactions it's a fair (¿and inevitable?) price to pay for scaling via soft-fork.

Do I have missed something?

Also I apologize for my English grammar. Thanks in advance for the answers.

Hi there, I hope not to ask a stupid question but...

Is there any way to arbitrary increase the block size via softfork? I mean, apart from Segregated Witness, Schnorr signatures or MAST which optimizes transaction size, and regardless of storage/bandwidth problems like the Great Firewall of China. Has anybody discovered/studied how to do that via softfork?

Yo ya conozco a Luiscar. Y quiero conocer a mas gente. ¡Me apunto!

Despues del DDOS a mtGox, bitstamp... etc las casas de cambio tienen precios muy diferentes, ¡incluso de 5 dólares!

¿Alguien tiene idea de porqué es esto?

Muchas gracias