A way to know how good a miner is choosing its transanctions

[JRamos]

Hello there!

I'm building a webpage: https://mempoolexplorer.com with bitcoin mempool statistics and I have some sections there that I don't really know if they are worthy so I'm asking here for help, and also, for a constructive criticism of the webpage since I'm dealing with what to do next.

I've posted this same question in bitcoin.stackexchange without luck, And I've realized that maybe is a too general question to be asked there so I'll try here:

Is there a way to know how good a miner is choosing its transanctions to be mined from a fee maximization point of view?

Well, this is the way I've tried:

When a mined block arrives to my node, I compare it against a transaction selection algorithm using my mempool. (I know my mempool it's different than the miner's one, and that's the point so keep reading ;-) )

The transaction selection algorithm sorts transactions in the usual greedy way: using the regular Ancestor Set Based (ASB) algorithm defined [here][1], and considering transaction dependencies and [CPFP][2], so basically is the same as invoking getblockTemplate with the state of my mempool just before the block arrives. (I also compare the incoming block by using the last getBlockTemplate result of bitcoind but as it's only calculated each 30s the results are not accurate).

An example of a comparison is shown [here][3], (my webpage, in development). I divide the transactions in sets whether are ignored by the miner (from my node point of view), are in common with me, Ignored by me, Relayed to me, or not relayed to me.

And, of course, I also measure the fees I'd collected against the miner's block (having into account the incoming block coin base transaction size for better accuracy). You can check the results of the last blocks [here][4], and an aggregate for the miners [here][5].

This could give me an idea about how good a miner is selecting its transactions and/or has good network connectivity (in average, and against my mempool), but as I have no way of knowing the state of the miners mempool before they mine a block (not when I receive it), this effort seems inaccurate.

I make the assumption that a received block and my block template can be compared. But as the block propagation time is not, by any means, negligible, the comparison is skewed giving the idea that my algorithm/network situation is better. The reality is that I am receiving transactions while the block is being propagated to me, and thus, I have normally a bigger mempool than the miner when the block was mined. A bigger mempool gives my algorithm the opportunity to search for better transactions which had no opportunity to be mined by miners.

So, Is the accuracy of this methodology worthy? I've assumed its not. But as block propagation time is getting smaller (see [this][6]) And the results for some blocks/miners seems really different (see [this][7] block for example) I have my doubts.

NOTE: Be aware that I've started the webpage only 2-3 hours ago so only last blocks will be shown.

Same question in stack exchange:
https://bitcoin.stackexchange.com/questions/111940/is-there-a-way-to-know-how-good-a-miner-is-choosing-its-transanctions-to-be-mine

  [1]: https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md
  [2]: https://bitcoinops.org/en/topics/cpfp/
  [3]: https://mempoolexplorer.com/block/last/OURS
  [4]: https://mempoolexplorer.com/block/OURS
  [5]: https://mempoolexplorer.com/miner
  [6]: https://www.dsn.kastel.kit.edu/bitcoin/?ref=hackernoon.com#latency
  [7]: https://mempoolexplorer.com/block/719478/BITCOIND

[1714694010]

1714694010

[1714694010]

1714694010

[1714694010]

1714694010

[1714694010]

1714694010

[1714694010]

1714694010

[1714694010]

1714694010

[mocacinno]

What you're doing is interesting... I'm not sure how much real-world use this information will have, other than convincing people to switch pools?

I guess there are multiple reasons why a miner/pool would have a different selection of transactions:

• Because they have a different mempool (like you said, you receive new transactions while their block was already solved, it just didn't reach you yet), but also because they could have different settings at to which transactions to reject, different settings for pruning, different settings for mempool size,... or they could have been running far longer than you have and have historic transactions in their mempool... Maybe they even have "private" transactions in their mempool (it happens that they receive a non-standard tx from somebody directly). On the other side, nobody is stopping them from removing transactions from their mempool, or rejecting them (eventough i don't know a real world usecase where this would be usefull)
• Because they change tx priority... Because somebody asked them nicely, payed them, or because it's their own transaction... Maybe even just for fun
• Because they mined an empty block, they just weren't ready with building the transaction selection, building the merkle tree and building the block header... So they started out with an empty block while building the "real" one, but solved the empty one instead... Or maybe it was a protest, or they weren't interested in the block rewards
• Because they modified the algorithm of tx selection...

Now, in your case... If you systematically find that you could have built a block with a better coinbase reward because a certain pool systematically builds blocks with non-ideal transactions, you might make a valid point and have interesting information on your hands, especially if you're talking about a pool that shares the fees with their miners...
This being said, they might have been payed to include transactions, raised the priority of their own transactions or maybe they had to make the choice between broadcasting an empty block or losing the block reward altogether, so i don't know if you can draw conclusions without knowing why they did what they did....

One other interesting thing you might want to consider: not everybody is running bitcoind (the reference client), and not everybody is running the same version... For example, i have ran btcd for a while (but it was not performant enough), and i've also run a heavily patched reference client that was built to run an explorer on top of it... Maybe those different full node implementations behave differently when it comes to rejecting transactions, maintaining mempools and selecting transactions? I personally have no idear if this is true (i never checked), but since you seem to be "digging" into this subject, it might be something to investigate So, the question would not only be: which pool makes the most optimal blocks, but also: which full node implementation and which settings makes the most optimal blocks

[JRamos]

I'm not sure how much real-world use this information will have, other than convincing people to switch pools?

Well, I started replicating the getBlockTemplate RCP from bitcoind into my code, but for the whole mempool, just to show the hierarchical representation in the first section.

Then I realized that I can compare the incoming block with my template and compare the results and search for ignored transactions in case of miners behaving wrongly. But this webpage https://miningpool.observer/ did it also, so now I can also compare it's results with mine's.

Then I realized that I can draw the transaction dependency graph and I asked this other question regarding big in-mempool transaction dependecy graphs https://bitcoin.stackexchange.com/questions/110723/huge-dependency-graphs-for-transactions-in-mempool but nobody answered.

And then I made the section we are discussing with, yes, having in mind which mempool is selecting/receiving the best transactions.

Maybe I should try to run several instances with different node geographical position to see how the miners statistics behave. What do you think?

I need to clean up the code since currently uses to much memory and needs a bitcoin node with fully downloaded blockchain to operate with, I don't want to spend too much money in the process.

This project was to learn microservices in java, and I got into the rabbit hole... so imagine...

[JRamos]

or they could have been running far longer than you have and have historic transactions in their mempool.

Yes, before running the instance I let my bitcoin node run for some time until there are some blocks not completely full and I also check that my mempool have more or less the same number of transactions than others like this: https://mempool.observer/

This problem has caused me some headaches as you can see here: https://bitcoin.stackexchange.com/questions/99717/transaction-being-ignored-by-miners-i-mean-ignored-not-not-mined-because-low/99822?noredirect=1#comment113126_99822

So, the question would not only be: which pool makes the most optimal blocks, but also: which full node implementation and which settings makes the most optimal blocks

As far as I know, Mining with other bitcoin node than bitcoincore is reckless since the consensus code is nothing to mess with. So I'd like to focus in pools not in implentations. Anyway, note that here: https://gist.github.com/Xekyo/5cb413fe9f26dbce57abfd344ebbfaf2#file-candidate-set-based-block-building-md you have a possible improvement over the regular greedy algorithm.

If a mempool is implementing or not a better algorithm is not worthy to know now. But maybe in the future when the fees gets higher.

[DannyHamilton]

Something to keep in mind is that sometimes a pool will be paid directly to include a transaction with a small fee.  So, while it may appear that it would have been beneficial to include some other transaction, that's only because you are unaware of that separate payment that was made directly to the pool.

Additionally, some pools may choose to reserve a small percentage of the available block space for extremely small fee transactions, as a charitable service to the community.  I'm pretty sure that viaBTC provides a limited number of requests per hour for very small fee transactions to be included in their next block. Services like this may result in smaller rewards for any single block but may help encourage and incentivize the use of Bitcoin overall which can result in higher fees and overall higher revenue in the long term.

[PrimeNumber7]

This being said, they might have been payed to include transactions, raised the priority of their own transactions or maybe they had to make the choice between broadcasting an empty block or losing the block reward altogether, so i don't know if you can draw conclusions without knowing why they did what they did....

Something to keep in mind is that sometimes a pool will be paid directly to include a transaction with a small fee.  So, while it may appear that it would have been beneficial to include some other transaction, that's only because you are unaware of that separate payment that was made directly to the pool.

This is really the key to the OP's problem/question.

The "blockchain" fee is neither an upper bound nor a lower bound as to the actual fee that a user will pay. There have been instances in which, due to human error, a person has included a fee that is orders of magnitude higher than the going rate for a similar transaction, and pool that mined the transaction has refunded the tx fee back to the user. Similarly, as is noted above, a pool may receive an off-chain payment for confirming a transaction that is in addition to the "blockchain" fee paid. These off-chain payments may be a subscription-like service, or it could be an ad-hoc like service.

[LoyceV]

Something to keep in mind is that sometimes a pool will be paid directly to include a transaction with a small fee.

From what I've seen, that fee is not "small":

Correct... I just logged in and it is currently quoting the following amounts to accelerate 644417ca91a9c16e1cb11ec0ed5a0928e203119d4f1bc00a93eb7da3ece1feb3:

0.0045472 BTC ≈ 228.90

I'm not sure if they share the revenue from those "sales" with the mining pool users though.

English is hard.

I misread a word: "with a small fee" is obviously not the same as "for a small fee".

I'm pretty sure that viaBTC provides a limited number of requests per hour for very small fee transactions to be included in their next block. Services like this may result in smaller rewards for any single block but may help encourage and incentivize the use of Bitcoin overall which can result in higher fees and overall higher revenue in the long term.

My assumption is they use the free transaction acceleration to "advertise" their paid service.

[JRamos]

So having into account both PrimeNumber7 and DannyHammilton answers we have these issues:

1- A pool will be paid to include a transaction with a small fee (or not fee at all)
2- A pool reserves a small percentage for low fee transactions.
3- A person mess around with a tx and do not put the charge back output into the tx. So the fees goes moon.

I'm going to deal with them:

1- That's something I've already see and asked about it: https://bitcoin.stackexchange.com/questions/93471/ive-found-two-mined-txs-with-no-fee.
But no problem: As you can see in this section https://mempoolexplorer.com/block/last/BITCOIND I can easily see the txs which has not been relayed to my node when the block arrived.
I've already discounted them in the column "Lost reward excl. not in our mempool txs" from here: https://mempoolexplorer.com/block/BITCOIND but not here yet: https://mempoolexplorer.com/miner. I'll do it.
2- If a pool reserves a small percentage for low fees tx there's nothing can be done. But I think this must be taken into the "final result" to see how much money they are spending onto this.
3- Easy: Check if a transaction deviates a lot from block average. The threshold will be high since I've seen in a regular basis tx's paying a lot more than it's needed.

I have a lot to do. since this needs more graphs and a lot of other things I need to solve first.

But I'd like somebody answer the question if running several instances in different geographical places and compare the results would be a good way to start and make the problem of block propagation time less concerning.

Thanks in advance.

[mocacinno]

--snip--
As far as I know, Mining with other bitcoin node than bitcoincore is reckless since the consensus code is nothing to mess with.
--snip--

btcd has been around since 2013. I stopped using it because the performance wasn't all that good (i had btcd and bitcoind both running on different machines, i switched to one node running bitcoind afterwards)... Bad performance was fine in 2014/2015 (which was around the time i used it iirc), but around ~2017(ish) the amount of broadcasted transactions rose untill the point most of the blocks were filled to the brim. The sheer size of the blockchain at that point in time, combined with full blocks made it obvious to me that i had to move away.

The reason for my move from btcd+bitcoind (two different nodes) to 1 node running bitcoind was 100% performance related, and had nothing to do with the consesus algo. AFAIK, btcd is updated frequently and is a perfectly valid (albeit, less performant) full node that can be used as a mining node. I'm not sure if anybody is doing this tough, and my experience with the project stopped many years ago, so i cannot comment on recent topics.

[DannyHamilton]

Something to keep in mind is that sometimes a pool will be paid directly to include a transaction with a small fee.

From what I've seen, that fee is not "small":

English is hard.

Let's try that again, but I'll make an effort to be a bit more clear in what I'm saying...

Something to keep in mind is that sometimes when someone sends a transaction with a small transaction fee, they may later choose to make a completely separate payment to a pool to incentivize that pool to include that low-fee transaction in one of their blocks. That separate payment might be made with a completely separate bitcoin transaction, in which case it may show up in the same block as the low-fee transaction or it may show up in an earlier (or later) block. That separate payment might also be paid via a completely different method, such as Lightning Network, an altcoin, physical cash in local currency, or local currency electronic payment (ACH, SWIFT, Wire, Venmo, etc).

[JRamos]

Thanks Danny, I'll take this into account: https://www.viabtc.com/tools/txaccelerator/

[PrimeNumber7]

So having into account both PrimeNumber7 and DannyHammilton answers we have these issues:

1- A pool will be paid to include a transaction with a small fee (or not fee at all)
2- A pool reserves a small percentage for low fee transactions.
3- A person mess around with a tx and do not put the charge back output into the tx. So the fees goes moon.

I'm going to deal with them:

1- That's something I've already see and asked about it: https://bitcoin.stackexchange.com/questions/93471/ive-found-two-mined-txs-with-no-fee.
But no problem: As you can see in this section https://mempoolexplorer.com/block/last/BITCOIND I can easily see the txs which has not been relayed to my node when the block arrived.
I've already discounted them in the column "Lost reward excl. not in our mempool txs" from here: https://mempoolexplorer.com/block/BITCOIND but not here yet: https://mempoolexplorer.com/miner. I'll do it.
2- If a pool reserves a small percentage for low fees tx there's nothing can be done. But I think this must be taken into the "final result" to see how much money they are spending onto this.
3- Easy: Check if a transaction deviates a lot from block average. The threshold will be high since I've seen in a regular basis tx's paying a lot more than it's needed.

I have a lot to do. since this needs more graphs and a lot of other things I need to solve first.

But I'd like somebody answer the question if running several instances in different geographical places and compare the results would be a good way to start and make the problem of block propagation time less concerning.

Thanks in advance.

1 and 2 are likely the same issue. Pools that accept low fee transactions for "free" may be using their "free" service as a means to advertise their paid service to include a low fee tx in their block for a fee. Some of the time, these low fee transactions may not qualify to get broadcast based on most nodes' relay rules, however, I think most of the time, these will be transactions with RBF disabled (not opted-into) that are in the mempool of most nodes, making a double-spend difficult.

3 - A pool that is solely taking the on-chain tx fee into consideration for which transactions to include in their blocks are solving something similar to a knapsack problem. CPFP transactions make this a little more tricky, especially if there is a chain of transactions, some of which may cause the transaction date to get confirmed (for example if the tx fee values were: Low, Low, Low, High, Low, Low, High, High, Low, Low, Very High). In general, pools will choose which transactions to include in a greedy manner, by choosing the highest tx fee rate transactions first, until there is very little block space left, at which point they may bypass a higher tx fee rate transaction in favor of a slightly lower tx fee rate transaction if it means the total tx fee increases compared to including the higher tx fee rate transaction (for example, a pool may choose to not include a transaction that would result in one unit of fees in favor of including two transactions that result in two units of fees being paid to the pool -- this may be possible if the former resulted in some block space being unused and the later resulted in lesser block space being unused).

If you do some research on the knapsack problem, you will discover that the average tx fee is not going to give you much information on if a transaction has an unusually high tx fee. There may be other measure that might hint at if a transaction has an unusually high tx fee.