What happens to "unselected" transactions?

[PCW]

One thing has been puzzling me.

In reading several topics in this board, I understand that solo miners (if they still exist) and pools have the option to select the transactions that will be included in their blocks.

So if a pool in creating a new block selects e.g., 6 transactions #1, #2, #4, #5, #6 and #7, deciding to leave out #3 which e.g., had a lower fee than other transactions.

They calculate the merkle root(s), create the block header(s), etc. and pool members start hashing their respective headers.

Someone in the pool uses a nonce that causes the hash to be lower than the target, and that block is added to the blockchain.

My question simply is, how does transaction #3 ever get to the blockchain?

[1715385433]

1715385433

[1715385433]

1715385433

[jackg]

There's a new block every 10 minutes. If the network is saturated with high fee transactions, the transaction will take a space where #3 is viable to add then it will be added, after about 2 weeks transactions can be dropped from the network although may or may not be depending on the node.

If a low fee transaction is sent, you can cpfp with it if you received it (so you spend the funds immediately, even to the same address but with a higher fee to compensate) or you can rbf if your wallet is compatible with it where you can double spend your transaction with a higher fee (miners favour higher fee txs so they will drop the low fee one).

[gmaxwell]

It gets picked up in some subsequent block or it doesn't.  Unless/until one of its inputs is spend some other way the transaction remains valid and can be included ones it makes sense to include it.


I'm not sure how approximate you intended your example to be. The actual behaviour is that miners collect up all translations they receive (above some very low threshold minimum fee to prevent DOS attacks) and queue them. Then they try to construct blocks that make the most fee they can, leaving out whatever doesn't fit-- which get included later when there are fewer higher paying things available.

E.g. This graph from a couple years ago shows the fees available for a block immediately before a new block was found (red), and the fees available in a block created instantly after a block was found (green).   So all those fees in green were already available but wouldn't fit.



As the subsidy goes away having a backlog is important to keeping hashrate up ... otherwise miners would have to turn off and stop mining as soon as a block is found and wait for there to be enough transactions to keep mining.

[PCW]

Thanks for the explanation thus far.

Maybe if I understood the mechanism that allows transactions to be available for selection, this would make more sense.

The idea that transaction #3 (or whatever) under certain circumstances could take hours to get to the blockchain (or never get there if hanging out to dry for 2 weeks) seems "inefficient".

Could someone explain where these transactions are before they get into a block, and how a pool gains access to them to select (or not).

Thanks.

[alani123]

Users with a full node can broadcast the transaction and the owner can actually choose to re-broadcast it with higher fees. Although that requires some more technical steps.

However, the simplest solution is to just wait as a miner would eventually pick it up form the mempool. Although with some delay probably.

[nc50lc]

Could someone explain where these transactions are before they get into a block, and how a pool gains access to them to select (or not).

The node where it was broadcast will save it in its mempool then relay that particular TX to its peers and those peers will also save and relay it as long as it complies with their relay/default settings;
after a short while, it will eventually reach a miner/pool's node.

For the meantime if it didn't get mined, it'll stay in the node's "mempool" as long as it wasn't "mined",
eg. saved in the data directory of their bitcoin core as mempool.dat.

[tranthidung]

Users with a full node can broadcast the transaction and the owner can actually choose to re-broadcast it with higher fees. Although that requires some more technical steps.

However, the simplest solution is to just wait as a miner would eventually pick it up form the mempool. Although with some delay probably.

If one uses the Electrum wallet, there are three options to do:

• Waiting patiently over long period to see that transaction confirmed or cancelled (I don't think people are patiently enough to wait for cancel of their transactions. In constrast if people are patiently enough, after hours or days their transactions will be confirmed).
• Use the RBF (Replace-by-Fee) option
• Use the Child-pays-for-parent transactions

Please see details:

What is Replace-by-fee?

To solve this problem, the concept of replace-by-fee was developed: by requiring replacements to pay for not only its own cost, but also the fee of the transactions being replaced, the DoS risk was strictly less than the risk of flooding with separate transactions.

If you have made a transaction that is unconfirmed, you can:

• Wait for a long time. Eventually, your transaction will either be confirmed or cancelled. This might take several days.
• Increase the transaction fee. This is only possible for “replaceable” transactions. To create this type of transaction, you must have checked “Replaceable” on the send tab before sending the transaction. If you’re not seeing the “Replaceable” option on the send tab go to Tools menu > Preferences > Fees tab and set “Propose Replace-By-Fee” to “Always”. Transactions that are replaceable have the word “Replaceable” in the date column on the history tab. To increase the fee of a replaceable transaction right click on its entry on the history tab and choose “Increase Fee”. Set an appropriate fee and click on “OK”. A window will popup with the unsigned transaction. Click on “Sign” and then “Broadcast”.

There is another option for you: A child pays for parent transactions

• Create a “Child Pays for Parent” transaction. A CPFP is a new transaction that pays a high fee in order to compensate for the small fee of its parent transaction. It can be done by the recipient of the funds, or by the sender, if the transaction has a change output. To create a CPFP transaction right click on the unconfirmed transaction on the history tab and choose “Child pays for parent”. Set an appropriate fee and click on “OK”. A window will popup with the unsigned transaction. Click on “Sign” and then “Broadcast”.

It is better for senders to check suggested good transaction fees before set-up fees for their transactions to avoid unexpected stuck transactions as well as some technical steps to boost transactions with higher fees, and to save more time (less waiting time).

• https://whatthefee.io/ ^[1]
• https://coinb.in/#fees ^[2]

According to ^[1], the suggested fees at the moment is 1 satoshi/byte.
According to ^[2], one can move their transactions with 4 satoshis/byte of fees and has 95% of probability to get confirmations after 2 to 3 hours.

Remember to check your wallet version is the newest one. If not, download the newest version from official site and verify it before installing and making transactions.
Don't trust, verify!

If you do it the first time (with RBF or CPFP), let try it with your very small fund (for initial transactions and for transaction accelerator steps).

Lastly, you can find services if you still don't know how to do it yourself by searching with keyword "Bitcointalk.org + bitcoin accelerator".
In my opinion, I don't see reason to lose my satoshis with too high fees or with bitcoin accelerator services.

I know a free service: https://bitaccelerate.com/. If you are interested (OP), you can give it a try. As I suggested previously, when you try to do something for the first time, try it with very small fund.

References:

• Electrum and RFB (replace by fee)
• {NOV 2019} Fees are low, use this opportunity to Consolidate your small inputs
• Make sure to avoid wasting BTC for too high fees – step by step guide (Electrum)
• Bitcoin Transaction Fees - Everything in one
• https://en.bitcoin.it/wiki/Techniques_to_reduce_transaction_fees
• https://freedomnode.com/blog/75/how-to-fix-slow-bitcoin-transactions-with-replace-by-fee
• https://hackernoon.com/holy-cow-i-sent-a-bitcoin-transaction-with-too-low-fees-are-my-coins-lost-forever-7a865e2e45ba

[/list]

[PCW]

Everything posted has been interesting; indeed learning is generally interesting, but enough philosophy...

Both alani123 and nc50lc mentioned the mempool - a clue.

At the advise of some random post found by searching mempool.dat, I went to my full node wallet (which continues to just eat up disk space but it's what I started with so...) and typed:

Code:

bitcoin-cli getrawmempool true

and before me appeared data, which led to a better understanding and more questions:

The file was 9MB in size and dated 2 days ago so I wasn't sure if that meant that I didn't have the last 2 days of transactions or, as I've seen before, the OS doesn't change the date even through some file has been updated

9MB of transactions that weren't picked up?  Seems like a lot. (in JSON format, each transaction took 767 bytes so = approx 11,700 transactions)

I then noted 2 fields in particular:

Code:

 "time": 1573801222,
 "height": 603870,

If this is a transaction that hasn't seen a block yet, why does it have a height.

I didn't know how to translate the date field - help appreciated.

So the questions are:

Does the mempool only contain "unselected" transactions?

If so, why does it have a block height?

What does this date field represent?  I'm sure there's an easy way to translate it e.g., some online tool - if anyone knows of one, please let me know.

If this does contain transactions already in the blockchain, which field indicates that it was never selected (or never made its way to the blockchain)?

[PCW]

I know a free service: https://bitaccelerate.com/. If you are interested (OP), you can give it a try. As I suggested previously, when you try to do something for the first time, try it with very small fund.

This is a fascinating aspect of BTC that I had no idea existed.  Thanks.

[nc50lc]

So the questions are:

Does the mempool only contain "unselected" transactions?

If so, why does it have a block height?

What does this date field represent?  I'm sure there's an easy way to translate it e.g., some online tool - if anyone knows of one, please let me know.

If this does contain transactions already in the blockchain, which field indicates that it was never selected (or never made its way to the blockchain)?

Those are all of the "unconfirmed transactions" that your node had accepted, both newly broadcast and not yet mined.
For the time, it's Epoch timestamp and you can use this online tool (as you requested): https://www.epochconverter.com/

You can find the answers in this bitcoin core documentation, including why it has a block height: https://bitcoincore.org/en/doc/0.16.0/rpc/blockchain/getrawmempool/
And everything RPC command-related info by using the menu on the right side.

[ranochigo]

I then noted 2 fields in particular:

Code:

 "time": 1573801222,
 "height": 603870,

If this is a transaction that hasn't seen a block yet, why does it have a height.

The block height field takes reference from the time that your node sees the unconfirmed transaction. It does not mean that the transaction is included in the block but it simply means that your node saw it when the latest block is X. It is of course quite inaccurate to be used to gauge the time where the transaction was sent.

Does the mempool only contain "unselected" transactions?

It contains transactions that are in not yet included in a block. When it sees a transaction, the node automatically purges the transactions that are in that block from the mempool.

If this does contain transactions already in the blockchain, which field indicates that it was never selected (or never made its way to the blockchain)?

It doesn't. There is no use for the client to keep multiple copies of the same transaction, especially when Bitcoin Core is resource intensive enough.

[alani123]

Users with a full node can broadcast the transaction and the owner can actually choose to re-broadcast it with higher fees. Although that requires some more technical steps.

However, the simplest solution is to just wait as a miner would eventually pick it up form the mempool. Although with some delay probably.

If one uses the Electrum wallet, there are three options to do:
[snip]

Very detailed answer and thanks for putting out up-to date step-by-step instructions. Maybe you should make a separate OP with a tutorial on the matter.

Also I've gotta say that getting meritted by Greg Maxwell is in my book one of my top 'Hi mum' kind of moments moments for my time in this forum. I'll cherish it.

[RHavar]

As the subsidy goes away having a backlog is important to keeping hashrate up ... otherwise miners would have to turn off and stop mining as soon as a block is found and wait for there to be enough transactions to keep mining.

Hmm. I'm probably missing something obvious, but I don't see why that is a problem? I can even see some upsides if that happens (reduced block variance, higher block density during peak times...)



The case I'd be worried about is if it became more profitable to re-mine the best-block to get those txfees, than try move the chain forward.

[gmaxwell]

As the subsidy goes away having a backlog is important to keeping hashrate up ... otherwise miners would have to turn off and stop mining as soon as a block is found and wait for there to be enough transactions to keep mining.

Hmm. I'm probably missing something obvious, but I don't see why that is a problem? I can even see some upsides if that happens (reduced block variance, higher block density during peak times...)

The case I'd be worried about is if it became more profitable to re-mine the best-block to get those txfees, than try move the chain forward.

That is that case, honest miners would turn off ... which also means that it would be more profitable to attack.

(Though just turning off alone is absolutely a problem too-- because it means that the chain temporarily stops gaining security, and it's advantage vs a lower hashrate attacker which simply doesn't stop is reduced).

Reduced block variance is bad too: the convergence of the bitcoin consensus algorithm comes purely from variance, the fact that when there is a chain split even if the hashrates are equal on each side eventually one side will get lucky relative to the other and get decisively ahead from the perspective of every node even given propagation delays.  Lower variance, higher reorg length. If lower variance were acceptable due to fast propagation and/or tolerance of reorgs, then it would be much better to simply reduce the interblock interval than have mining randomly come in synchronized bursts as soon as enough fees show up.

Higher block density I agree could be useful, depending on if the system is more limited by peak traffic compared to overall volume. Both sorts of limits exist in the system, e.g. strength against selfish-mining like attacks care about maximum block propagation delays in the face of adversarially constructed blocks ... while catchup/IBD costs care about overall volume.  To the extent that the latter is limiting it would be good to have a daily swing in capacity.