My concern was that perhaps by the time i complete a for loop on lets say 30 of the transactions in the mempool and get each corresponding values, i may have missed new entries in the mempool which would also have to be looped through...
Absolutely. You can only capture a moment-in-time view of your mempool. Then at another time you can capture another moment-in-time view. This would be true even if the getrawmempool command included values. You'd still only be able to capture the values of the transactions that happened to be in your mempool at the exact moment that you ran the command.
btw just out of curiosity would you know how services like blochain.info do it?
I'm not sure that they've ever publicly shared their internal processes. If it were me, I might just pull down the source for Bitcoin Core and then modify it to connect directly to my database and store all the information that I wanted to capture as it received transactions and blocks from peers. For the purposes of scalability I might actually have it write that information to some sort of in-memory queue. That way I could have multiple custom nodes all writing to the queue simultaneously and then have a de-duplication process that pulls from the queue and updates the database. This would protect me from some of the performance issues that might arise from establishing DB connections and write-locks.
also, on a related note, is it possible for my node to miss any unconfirmed bitcoin transactions if, for some reason, they don't ever show up in my mempool. Or is there a low chance of this happening? thanks
It's possible that your node receives transactions in a different order with different timestamps than other nodes.
It's possible that your node receives transactions that some other nodes ONLY receive in a block (never entering their mempool).
It's possible that other nodes receive transactions that your node ONLY receives in a block (never entering your mempool).
It's possible that your node will receive transactions that some other nodes never receive AT ALL.
It's possible that other nodes receive transactions that your node never receives AT ALL.
It is possible that your node:
- Receives some transactions into your mempool
- Receives a block (Block_A1) containing those transactions and therefore removes them from your mempool
- Receives a replacement block (Block_A2) that doesn't include those transactions
- Receives a third block (Block_B2) that builds on top of Block_A2 (forcing a re-org for you)
- Moves some of the transactions from Block_A1 back into the mempool
- Receives another block (Block_C2) that invalidates some of those transactions that moved back to your mempool, causing them to be abandoned, permanently. (note that this process could technically repeat multiple times for the same transaction moving from mempool to a block, back to mempool due to a re-org, back to a block, back to mempool due to another re-org, back to a block, and so on until eventually either there is no re-org and remains in a block or it is permanently abandoned.)
It is also possible that your node receives a transaction, it remains in the mempool for many days, it is eventually purged from the mempool without being confirmed in order to make room for more transactions, and then it is received again (a second time, or third, or fourth, etc).
It is also possible that your node receives a transaction, it eventually expires out of your mempool unconfirmed, and a replacement transaction is received that spends ALL of the exact same inputs and sends value to all the exact same Txout-scripts/scriptPubKeys, BUT is a completely different transaction ID (due perhaps to slightly different values assigned to those outputs?)
It is also possible that the arrival time of a transaction in your mempool (timestamp) could be LATER than the timestamp of the block that it eventually is included in.
These are the scenarios that come immediately to mind. I suspect there are others that I could come up with if I thought about it a bit longer.
Such is the nature of a decentralized system. You can't count on the fact that you'll receive EVERYTHING that every other node receives, or that any of them will receive EVERYTHING that you receive, or that those things which you and some other node do both receive will arrive in the same order for both of you.Some things to ask yourself about your code:
- If a transaction that I never saw in my mempool shows up in a block, do I want to add it to the list of unconfirmed transactions that I had generated prior to the arrival of the block?
- If a confirmed transaction once again becomes unconfirmed (due to a re-org), do I want to keep track of that fact?
- If an unconfirmed transaction in my mempool becomes invalidated by some other transaction that arrives in a block, do I still include the (now invalidated) transaction in my list of unconfirmed transactions?
- If the timestamp of when I receive a transaction is LATER than the timestamp of the block that it is eventually included in, do I want to modify the timestamp of the transaction to be equal to the block timestamp?
