Obtaining all transactions since a given txid

[jon_smark]

Hi guys,

I would very much like to support Bitcoin in a web site I'm developing.  I've been playing around with the JSON API, and for the most part it fulfills my needs.  There's one glaring omission, however: with the present API I can see no way of obtaining all transactions (either associated with an account or all of them) that have happened since a fixed point in the past (since a given transaction ID, for example).

I realise that 'listtransactions' will give me a list of the last N transactions, but there's no way to know beforehand what is the value N of transactions that have occurred since I last polled the Bitcoin server.  Also, 'listtransactions' requires a mandatory account name; it would be useful if an hypothetical 'listtransactionssince' command could also return all transactions, regardless of the account.

Your thoughts?

Thank you!
Jon

[1713563794]

1713563794

[1713563794]

1713563794

[Gavin Andresen]

0.3.20 will support listtransactions '*'  for all transactions.

Transactions since <txid> ....  sounds dangerous.  Are you trying to mirror the transaction database that bitcoin keeps?  If you are, are you sure you'll deal properly with (rare) block chain re-organizations where transactions that you thought had one confirmation turn out to be invalid in a new fork of the chain?

I can imagine subtle bugs-- you ask for transactions since <txid> but that txid turns out to be an orphan transaction (double-spend on the losing side of a block chain re-org) so bitcoin returns... what?

All non-orphan transactions with transaction time equal to or greater than a given timestamp might work-- but what is your use case?

[jon_smark]

Hi,

Thanks for the prompt reply!  Obviously I have no intention of replicating all the chain sanitising that the bitcoin daemon performs.  My idea was to get all the confirmed/valid/kosher transactions since a fixed point in the past.  There's more than one way to specify what this fixed point should be, but assuming the txid is unique, it would be a good candidate.  Timestamps could of course also be used, as long as the corner cases involving multiple transactions with the same timestamp are cared for.

Here's my use case: suppose I have a potentially large number of users making transfers directed towards me (each using a separate account).  My application will need to periodically poll the bitcoin daemon to find out what new transfers have been confirmed, so it can update the user's account/assets on the application side.

I realise that a PUSH mechanism could also be used, ie, the bitcoin daemon could be given a callback to invoke every time a transaction is confirmed.  However, there are a few things that can go wrong in that scenario over which I have little control (would for instance the bitcoin daemon be smart enough to issue a retry if the other end did not reply?).  Hence, I prefer the periodic polling mechanism, since I'm in control of the error situations.

Cheers,
Jon

[Gavin Andresen]

Here's my use case: suppose I have a potentially large number of users making transfers directed towards me (each using a separate account).  My application will need to periodically poll the bitcoin daemon to find out what new transfers have been confirmed, so it can update the user's account/assets on the application side.

Two thoughts:

listaccounts [minconfirmations]  (also new in 0.3.20, if I'm remembering correctly) will give you all accounts and their balances.  If an account's balance doesn't change, you know it hasn't received coins.

And why do you need to periodically update the user's account/assets?  Could you just query bitcoind for their current balance when they fetch their 'account status' page (or wherever you show the user how many bitcoins they have)?  Are you doing something with the received bitcoins without an explicit action on the part of the user, and does that something care about the total balance or individual transactions?

[jon_smark]

Hi,

listaccounts [minconfirmations]  (also new in 0.3.20, if I'm remembering correctly) will give you all accounts and their balances.  If an account's balance doesn't change, you know it hasn't received coins.

If no other solution is available, that could work.  However, for the sake of transparency I would very much like to present the user with a full list of confirmed transactions associated with their account.  Also, I'm a bit concerned about the overhead of 'listaccounts'.  Suppose there are thousands of accounts, but only a handful have changed their balance since my last query.  It seems a bit silly having to go through all of them computing the delta in the account balance.

And why do you need to periodically update the user's account/assets?  Could you just query bitcoind for their current balance when they fetch their 'account status' page (or wherever you show the user how many bitcoins they have)?  Are you doing something with the received bitcoins without an explicit action on the part of the user, and does that something care about the total balance or individual transactions?

Yes, there are "behind the scenes" actions that must take place regardless of user intervention.  Suppose the user has bought something; I want to be able to ship the item as soon as the bitcoin transaction is confirmed.  Sure, I could have a list of outstanding orders and do a periodic polling checking all associated accounts for a changed balance, but it seems far more efficient if I could ask the bitcoin daemon "look, the last transaction I got has txid of XXYYZZ.  Please give me a list of all confirmed transactions which have occurred since then".

Thanks again for your attention, it is much appreciated!
Jon

[bitcoinex]

It is important not to miss a transaction, becouse listtransactions do not report number of transactions - it is gives only list latest from them.

[jon_smark]

It is important not to miss a transaction, becouse listtransactions do not report number of transactions - it is gives only list latest from them.

Yes, that is why I think 'listtransactions' has very little use for a reliable ecommerce system, and would very much prefer an API method that would return all confirmed transactions since a fixed txid in the past...

[davout]

What language/framework are you going to develop your app with ?

Also, be aware that accounts tracking is unreliable as of now since it will silently discard subcent amounts, that happened on bitcoin central, people had a reported balance that was different when asking to the DB and to the client.

The bitcoin central code is open sourced, you can study it to see how it's managed.

[Gavin Andresen]

How do you think the code should deal with the issue of "a fixed txid might no longer be valid" ?  Or transactions that you THOUGHT had N confirmations and were in a block with timestamp T are re-organized to have M confirmations and are in a block with timestamp T2 ?

tcatm and I chatted in IRC about this problem-- he has the problem of how to keep the Javascript GUI (which shows you ever transaction) up-to-date.

Seems like you really want an efficient way of asking "Do I know about exactly the same set of transactions as bitcoind" (either for the entire wallet or for an account).  That's assuming transaction data can't change (somebody double-check that assumption for me-- if the transaction data changes, then the txid changes... I think).

That led to some brainstorming of computing a hash of all transaction ids (maybe just XOR them all together).  Or maybe something more complicated like a merkle tree...   but that seems like overkill.

Is there a good, simple solution to synchronizing "I have this set of stuff, give me everything I don't have" ?

[davout]

Is there a good, simple solution to synchronizing "I have this set of stuff, give me everything I don't have" ?

Not that I know of.
Answer I found is unelegant polling.

Also iirc there was a consensus on the fact there had to be a point at which you decide that the transaction won't be reversed and consider the payment as valid.

Something I should implement in BC is check that confirmations keep going up for every transaction. If the number of confirmations for a given tx stops going up even though the block count increased means that the transaction got invalidated and should be taken care of specifically.

* davout opens a git issue

[Gavin Andresen]

Yes, there are "behind the scenes" actions that must take place regardless of user intervention.  Suppose the user has bought something; I want to be able to ship the item as soon as the bitcoin transaction is confirmed.

Transactions are confirmed when new blocks are accepted; that's the idea behind my 95%-finished 'monitorreceived' code.  It will POST to a URL when new wallet transactions and blocks are accepted.  The block information includes all transaction ids (and a link to the previous block).

As you note, you still have the issue of what to do if your code misses a POST.  That shouldn't be a big issue, though, because you'll resynchronize the next time a block is found.  Worst case, you'll ship to the customer 1 block (10 minutes) later than is optimal if your connection to bitcoind is lost.

[jon_smark]

How do you think the code should deal with the issue of "a fixed txid might no longer be valid" ?  Or transactions that you THOUGHT had N confirmations and were in a block with timestamp T are re-organized to have M confirmations and are in a block with timestamp T2 ?

Here's how I think that problem should be solved:

• Upon an initial invocation, the client requests a list of all transactions with a minimal number N of confirmations.
• The Bitcoind daemon dutifully obliges.
• The client stores the txid of the last transaction on top of a stack.
• The client now requests a list of all transactions with at least N confirmations which have occurred since the txid on top of the stack.
• The Bitcoind daemon dutifully obliges.
• The client again stores the txid of the last transaction on top of a stack.  The stack has now two elements.
• Some time passes, and the client requests once more all transactions which have occurred since the txid that is now on top of the stack.
• There's a problem however, since the requested txid has been invalidated.  The bitcoind daemon returns an error reporting this situation.
• The client now knows that all transactions which occurred between the two top elements on the stack must be invalidated. It removes the top element from the stack and tries a new request from the "last known good point".
• The Bitcoind daemon either obliges or returns an error (forcing the client to pop yet another element from the stack and invalidate a bunch of transactions).

Note that my proposal puts the burden of dealing with those hairy circumstances on the ecommerce client, not on the Bitcoind daemon.  While I realise that some lazy programmers might now bother to implement this algorithm in their client application, even they can avoid the chances of catastrophe simply by setting the number N of minimum confirmations to be rather high.  It is my understanding that the chances of a chain being invalidated diminish exponentially with N.  Therefore, if you can't be bothered to deal with hairy situations you could just set N=100 and take comfort in the fact that the universe will suffer from heat death before there's a chance you'll ever run into problems...

What do you think?

Best regards,
Jon

[Gavin Andresen]

Nice, I like your proposal.

Do you know C++?

[ribuck]

ote that my proposal puts the burden of dealing with those hairy circumstances on the ecommerce client, not on the Bitcoind daemon.

That's the only way it will work in the real world. It's a business decision when to regard a transaction as complete, and how (or whether) to handle invalidated transactions. Different businesses will have different policies and procedures.

[jon_smark]

Nice, I like your proposal.

Do you know C++?

Actually, I was hoping that by putting on the client the onus of dealing with the hairy stuff (and by client I mean whichever application is sending JSON-RPC requests), my proposal would make this API method relatively straightforward to implement on the Bitcoin daemon.  It doesn't sound too different from the already existing 'listtransactions' method, or is it?

I have used C++ in a previous life, but haven't done anything serious with it in almost 10 years. Almost all my programming these days is done in Ocaml.  In any case, I don't know enough about the inner workings of Bitcoin to even contemplate implementing this myself with any degree of confidence.

Cheers,
Jon

[jon_smark]

Hallo again,

Okay, I have taken a brief look at the code in rpc.cpp (from https://github.com/bitcoin/bitcoin/blob/master/rpc.cpp), and it seems the listtransactionssince feature I requested is not that different from the code that is already in place.

In function listtransactions, there is already a loop reversing backwards from the last transaction until a given count of transactions is reached.  In a hypothetical listtransactionssince function, the termination condition for this loop should be instead the discovery of a transaction whose txid matches the given parameter.

As to the question of how should we deal with the situation where the past transaction has been invalidated, I assume that that the given txid will never be found, and the function will find it itself looping all the way to the end.  So, one straightforward implementation is to simply return an error if we reach the end without ever finding the given txid.  However, for performance reasons, it makes sense to break out of the loop and return an error much sooner than that.  So, why not provide the function with an optional extra parameter (a timestamp) which indicates the last time in the past until which we should still be looking for txid.  If the loop finds itself testing a transaction whose timestamp is older than this cutoff, than we can break out of the loop earlier and report the error condition.

Again, note that this puts on the client the burden of providing a timestamp which make sense.

Your thoughts?

Best regards,
Jon

[Gavin Andresen]

If an optional time param is needed, maybe get rid of the "since <txid>" notion and just make it "give me all receive transactions for [account] with [minconfirmations] that have a time greater than [time]".

[jon_smark]

If an optional time param is needed, maybe get rid of the "since <txid>" notion and just make it "give me all receive transactions for [account] with [minconfirmations] that have a time greater than [time]".

I thought of that as well, but how would the client know if a previously obtained list of transactions had been invalidated?  Moreover, the txid can be assumed to be unique, even if we consider the set of all transactions including those in invalidated branches.  Does this property also hold for transaction timestamps in Bitcoin?  My guess is not, which makes relying solely on timestamps a bad idea.

[Gavin Andresen]

I thought of that as well, but how would the client know if a previously obtained list of transactions had been invalidated?

Huh?  I don't see how passing <txid> makes that problem any better.

Let's say you've got transactions A B C D (in that order).  With times T1, T1, T2 and T2 (A&B happened 2 blocks back, you got C&D in the last block).

Now imagine a block-chain re-org, so B and C are now invalid.

Ask for all transactions after D and you get nothing-- same as if you asked for all transactions with time greater than T2.

[jon_smark]

Let's say you've got transactions A B C D (in that order).  With times T1, T1, T2 and T2 (A&B happened 2 blocks back, you got C&D in the last block).
Now imagine a block-chain re-org, so B and C are now invalid.

Okay, let us consider that scenario.  Step by step, just to make sure we are on the same page:

• The Bitcoind daemon knows of transactions [A, B, C, D] with timestamps [T1, T1, T2, T2].
• Upon first invocation, the client asks for a list of all transactions.  It receives [A, B, C, D] (and timestamps) as response.
• There's a block chain re-org invalidating [C, D].
• Four new transactions [E, F, G, H] with timestamps [T2, T2, T3, T3] become known to the daemon. According to the daemon, the current chain is [A, B, E, F, G, H].
• If the client now asks for all transactions since T2, it will receive only [G, H] and no error condition, becoming therefore oblivious to the invalidation of [C, D] and to the existence of [E, F].  If, on the other hand, the client were to ask for all transactions since D, the daemon would have returned (correctly) an error.

Am I missing something?

Cheers, (and thanks for your attention!)
Jon