Bitcoin Forum

Hi guys,

I would very much like to support Bitcoin in a web site I'm developing.  I've been playing around with the JSON API, and for the most part it fulfills my needs.  There's one glaring omission, however: with the present API I can see no way of obtaining all transactions (either associated with an account or all of them) that have happened since a fixed point in the past (since a given transaction ID, for example).

I realise that 'listtransactions' will give me a list of the last N transactions, but there's no way to know beforehand what is the value N of transactions that have occurred since I last polled the Bitcoin server.  Also, 'listtransactions' requires a mandatory account name; it would be useful if an hypothetical 'listtransactionssince' command could also return all transactions, regardless of the account.

Your thoughts?

Thank you!
Jon

0.3.20 will support listtransactions '*'  for all transactions.

Transactions since <txid> ....  sounds dangerous.  Are you trying to mirror the transaction database that bitcoin keeps?  If you are, are you sure you'll deal properly with (rare) block chain re-organizations where transactions that you thought had one confirmation turn out to be invalid in a new fork of the chain?

I can imagine subtle bugs-- you ask for transactions since <txid> but that txid turns out to be an orphan transaction (double-spend on the losing side of a block chain re-org) so bitcoin returns... what?

All non-orphan transactions with transaction time equal to or greater than a given timestamp might work-- but what is your use case?

Hi,

Thanks for the prompt reply!  Obviously I have no intention of replicating all the chain sanitising that the bitcoin daemon performs.  My idea was to get all the confirmed/valid/kosher transactions since a fixed point in the past.  There's more than one way to specify what this fixed point should be, but assuming the txid is unique, it would be a good candidate.  Timestamps could of course also be used, as long as the corner cases involving multiple transactions with the same timestamp are cared for.

Here's my use case: suppose I have a potentially large number of users making transfers directed towards me (each using a separate account).  My application will need to periodically poll the bitcoin daemon to find out what new transfers have been confirmed, so it can update the user's account/assets on the application side.

I realise that a PUSH mechanism could also be used, ie, the bitcoin daemon could be given a callback to invoke every time a transaction is confirmed.  However, there are a few things that can go wrong in that scenario over which I have little control (would for instance the bitcoin daemon be smart enough to issue a retry if the other end did not reply?).  Hence, I prefer the periodic polling mechanism, since I'm in control of the error situations.

Cheers,
Jon

Two thoughts:

listaccounts [minconfirmations]  (also new in 0.3.20, if I'm remembering correctly) will give you all accounts and their balances.  If an account's balance doesn't change, you know it hasn't received coins.

And why do you need to periodically update the user's account/assets?  Could you just query bitcoind for their current balance when they fetch their 'account status' page (or wherever you show the user how many bitcoins they have)?  Are you doing something with the received bitcoins without an explicit action on the part of the user, and does that something care about the total balance or individual transactions?

Hi,


If no other solution is available, that could work.  However, for the sake of transparency I would very much like to present the user with a full list of confirmed transactions associated with their account.  Also, I'm a bit concerned about the overhead of 'listaccounts'.  Suppose there are thousands of accounts, but only a handful have changed their balance since my last query.  It seems a bit silly having to go through all of them computing the delta in the account balance.


Yes, there are "behind the scenes" actions that must take place regardless of user intervention.  Suppose the user has bought something; I want to be able to ship the item as soon as the bitcoin transaction is confirmed.  Sure, I could have a list of outstanding orders and do a periodic polling checking all associated accounts for a changed balance, but it seems far more efficient if I could ask the bitcoin daemon "look, the last transaction I got has txid of XXYYZZ.  Please give me a list of all confirmed transactions which have occurred since then".

Thanks again for your attention, it is much appreciated!
Jon

It is important not to miss a transaction, becouse listtransactions do not report number of transactions - it is gives only list latest from them.

Yes, that is why I think 'listtransactions' has very little use for a reliable ecommerce system, and would very much prefer an API method that would return all confirmed transactions since a fixed txid in the past...

What language/framework are you going to develop your app with ?

Also, be aware that accounts tracking is unreliable as of now since it will silently discard subcent amounts, that happened on bitcoin central, people had a reported balance that was different when asking to the DB and to the client.

The bitcoin central code is open sourced, you can study it to see how it's managed.

How do you think the code should deal with the issue of "a fixed txid might no longer be valid" ?  Or transactions that you THOUGHT had N confirmations and were in a block with timestamp T are re-organized to have M confirmations and are in a block with timestamp T2 ?

tcatm and I chatted in IRC about this problem-- he has the problem of how to keep the Javascript GUI (which shows you ever transaction) up-to-date.

Seems like you really want an efficient way of asking "Do I know about exactly the same set of transactions as bitcoind" (either for the entire wallet or for an account).  That's assuming transaction data can't change (somebody double-check that assumption for me-- if the transaction data changes, then the txid changes... I think).

That led to some brainstorming of computing a hash of all transaction ids (maybe just XOR them all together).  Or maybe something more complicated like a merkle tree...   but that seems like overkill.

Is there a good, simple solution to synchronizing "I have this set of stuff, give me everything I don't have" ?

Not that I know of.
Answer I found is unelegant polling.

Also iirc there was a consensus on the fact there had to be a point at which you decide that the transaction won't be reversed and consider the payment as valid.

Something I should implement in BC is check that confirmations keep going up for every transaction. If the number of confirmations for a given tx stops going up even though the block count increased means that the transaction got invalidated and should be taken care of specifically.
/me opens a git issue :)

Transactions are confirmed when new blocks are accepted; that's the idea behind my 95%-finished 'monitorreceived' code.  It will POST to a URL when new wallet transactions and blocks are accepted.  The block information includes all transaction ids (and a link to the previous block).

As you note, you still have the issue of what to do if your code misses a POST.  That shouldn't be a big issue, though, because you'll resynchronize the next time a block is found.  Worst case, you'll ship to the customer 1 block (10 minutes) later than is optimal if your connection to bitcoind is lost.

Here's how I think that problem should be solved:

• Upon an initial invocation, the client requests a list of all transactions with a minimal number N of confirmations.
• The Bitcoind daemon dutifully obliges.
• The client stores the txid of the last transaction on top of a stack.
• The client now requests a list of all transactions with at least N confirmations which have occurred since the txid on top of the stack.
• The Bitcoind daemon dutifully obliges.
• The client again stores the txid of the last transaction on top of a stack.  The stack has now two elements.
• Some time passes, and the client requests once more all transactions which have occurred since the txid that is now on top of the stack.
• There's a problem however, since the requested txid has been invalidated.  The bitcoind daemon returns an error reporting this situation.
• The client now knows that all transactions which occurred between the two top elements on the stack must be invalidated. It removes the top element from the stack and tries a new request from the "last known good point".
• The Bitcoind daemon either obliges or returns an error (forcing the client to pop yet another element from the stack and invalidate a bunch of transactions).

Note that my proposal puts the burden of dealing with those hairy circumstances on the ecommerce client, not on the Bitcoind daemon.  While I realise that some lazy programmers might now bother to implement this algorithm in their client application, even they can avoid the chances of catastrophe simply by setting the number N of minimum confirmations to be rather high.  It is my understanding that the chances of a chain being invalidated diminish exponentially with N.  Therefore, if you can't be bothered to deal with hairy situations you could just set N=100 and take comfort in the fact that the universe will suffer from heat death before there's a chance you'll ever run into problems...

What do you think?

Best regards,
Jon

Nice, I like your proposal.

Do you know C++?

That's the only way it will work in the real world. It's a business decision when to regard a transaction as complete, and how (or whether) to handle invalidated transactions. Different businesses will have different policies and procedures.

Actually, I was hoping that by putting on the client the onus of dealing with the hairy stuff (and by client I mean whichever application is sending JSON-RPC requests), my proposal would make this API method relatively straightforward to implement on the Bitcoin daemon.  It doesn't sound too different from the already existing 'listtransactions' method, or is it?

I have used C++ in a previous life, but haven't done anything serious with it in almost 10 years. Almost all my programming these days is done in Ocaml.  In any case, I don't know enough about the inner workings of Bitcoin to even contemplate implementing this myself with any degree of confidence.

Cheers,
Jon

Hallo again,

Okay, I have taken a brief look at the code in rpc.cpp (from https://github.com/bitcoin/bitcoin/blob/master/rpc.cpp (https://github.com/bitcoin/bitcoin/blob/master/rpc.cpp)), and it seems the listtransactionssince feature I requested is not that different from the code that is already in place.

In function listtransactions, there is already a loop reversing backwards from the last transaction until a given count of transactions is reached.  In a hypothetical listtransactionssince function, the termination condition for this loop should be instead the discovery of a transaction whose txid matches the given parameter.

As to the question of how should we deal with the situation where the past transaction has been invalidated, I assume that that the given txid will never be found, and the function will find it itself looping all the way to the end.  So, one straightforward implementation is to simply return an error if we reach the end without ever finding the given txid.  However, for performance reasons, it makes sense to break out of the loop and return an error much sooner than that.  So, why not provide the function with an optional extra parameter (a timestamp) which indicates the last time in the past until which we should still be looking for txid.  If the loop finds itself testing a transaction whose timestamp is older than this cutoff, than we can break out of the loop earlier and report the error condition.

Again, note that this puts on the client the burden of providing a timestamp which make sense.

Your thoughts?

Best regards,
Jon

If an optional time param is needed, maybe get rid of the "since <txid>" notion and just make it "give me all receive transactions for [account] with [minconfirmations] that have a time greater than [time]".

I thought of that as well, but how would the client know if a previously obtained list of transactions had been invalidated?  Moreover, the txid can be assumed to be unique, even if we consider the set of all transactions including those in invalidated branches.  Does this property also hold for transaction timestamps in Bitcoin?  My guess is not, which makes relying solely on timestamps a bad idea.

Huh?  I don't see how passing <txid> makes that problem any better.

Let's say you've got transactions A B C D (in that order).  With times T1, T1, T2 and T2 (A&B happened 2 blocks back, you got C&D in the last block).

Now imagine a block-chain re-org, so B and C are now invalid.

Ask for all transactions after D and you get nothing-- same as if you asked for all transactions with time greater than T2.

Okay, let us consider that scenario.  Step by step, just to make sure we are on the same page:

• The Bitcoind daemon knows of transactions [A, B, C, D] with timestamps [T1, T1, T2, T2].
• Upon first invocation, the client asks for a list of all transactions.  It receives [A, B, C, D] (and timestamps) as response.
• There's a block chain re-org invalidating [C, D].
• Four new transactions [E, F, G, H] with timestamps [T2, T2, T3, T3] become known to the daemon. According to the daemon, the current chain is [A, B, E, F, G, H].
• If the client now asks for all transactions since T2, it will receive only [G, H] and no error condition, becoming therefore oblivious to the invalidation of [C, D] and to the existence of [E, F].  If, on the other hand, the client were to ask for all transactions since D, the daemon would have returned (correctly) an error.

Am I missing something?

Cheers, (and thanks for your attention!)
Jon

No, there is a block chain re-org invalidating B and C.  D is still valid in my scenario.

So the client remains oblivious to B and C being invalid, whether it asks for transactions older than T2 or transactions after D.

Okay, I was under the impression that if B and C were invalidated, then D would be invalidated too.  (As I noted a few posts ago, one of my assumptions was that the txid was unique across the entire tree).

So, we're back to square one: we need to find some ID which is unique across the entire tree.  Is there such a thing as a block hash?  I assume that when there is a fork in the block chain, both children of block N will number themselves N+1.  Though one of them will be invalidated eventually, it's not possible to use the block number as a unique ID.  Hence why some sort of hash which uniquely identifies a block would be useful.

In terms of the API, the client would invoke listtransactionssince with the hash of the last known block.  If found, the daemon would return the hash of the last confirmed block in the chain, together with all transactions which have occurred between the two blocks.  And instead of a timestamp, the optional parameter indicating early breakout criteria would be the block number.

(Alternatively, there could be two separate API calls: listblockssince and getblocktransactions).

As far as the client goes, the algorithm to ensure a correct transaction history -- even in the context of possible invalidations -- would be similar to the one I've already described.  Just s/txid/blockhash/.

Anyway, as I mentioned I'm not that familiar with the inner workings of the Bitcoin protocol, so I'm making lots of assumptions.  I hope you can fill the blanks, because without an API that allows clients to have a fool-proof way of ensuring transaction consistency it will be very hard for Bitcoin to be taken seriously for e-commerce applications.

Cheers,
Jon

The fool-proof way of dealing with your use case (customer orders something, you want to ship after you're sure payment has cleared):

+ Give each customer an account.  When they order, use getaccountaddress to get a bitcoin address to which they can send payment.

+ Every N minutes ask bitcoin either the balance (with minimum 6 confirmations) for accounts with pending orders or all accounts

+ If the account balance is enough to pay for the order, ship it and move the coins from the customer's account to a PAID account.
   If not... either wait or tell the customer they paid the wrong amount or maybe refund any extra they sent (you'll have to ask them for a refund address).

The inelegant polling will eventually be fixed by bitcoin POSTing when new blocks or transactions arrive, but I think you'll still need to ask bitcoin what the account's current balance is-- trust me, you really don't want to recreate all the bitcoin logic dealing with double-spent transactions or block chain reorganizations.

If you grow to handling thousands of orders per day (which would be a very good problem to have) you'll want to buy or build a version of bitcoin optimized for high-volume transaction websites.  Or maybe you'll run 20 bitcoinds, each handling 1/20'th of the customers -- I dunno, I don't spend a lot of time worrying about problems I'll have when my project is outrageously successful.

Well, that's the straightforward procedure which the Bitcoin daemon already caters for, and which a lazy programmer might implement in their client.  It's awfully inefficient though, and it won't scale at all.  I guess that's my fundamental problem with it.  But it's not the only one, though:

Suppose there are multiple items associated with a customer (for instance, adverts whose impression count may be asynchronously "recharged" at any moment).  With the simple algorithm above, one must have a separate account for each asset.  Since each customer can have multiple assets, things can quickly grow out of control.  Alternatively, one could group all assets from one user into one account and distinguish between assets via their address.  However, the "simple" algorithm is suddenly much more complicated: after an initial polling of all accounts, one would have to loop through all accounts with a changed balance, doing an additional API invocation to get all address balances associated with that account, and comparing each one to the previously stored balance. Just think of all the extra pressure this would put on the database backend that stores the account balances!

Anyway, my point is that the simplicity of the algorithm you describe is only valid for a narrow range of scenarios.  There are situations (such as mine) where it would actually make the client logic more complex and unable to scale beyond triviality.

Note that I'm not advocating that clients should be forced to deal with block chain re-organisations.  On the contrary, I think the client of the API should have the freedom to implement whatever scheme they desire.  I will not argue that there are plenty of people for whom the simple algorithm will suffice.  There are however also people who would like a more efficient and scalable way of dealing with the Bitcoin daemon...

But going back to my original question: is there a unique way to identify a block even in the face of reorgs?

Sure, every block has a unique hash.

But I don't think that helps at all; you might see that transactions A B C D are in block #100,000 with hash H1, but after a block chain re-org block #100,000 might contain transactions A D (with block hash H2).

The probability of that happening rapidly approaches zero as the block gets confirmed; after 6 confirmations you can safely assume it just won't happen.

Good idea. This will work. The client can also keep a stack of block hashes if desired to reduce work in case of a reorg (which happens more often than you might think).

It won't work for 0-confirmation transactions, of course.

Bitcoin Block Explorer does something similar: it uses the current block number for ETag/If-None-Match caching on certain pages.

Hallo again,

I have a new proposal.  This one is IMHO simpler and more robust.  Moreover, it caters for two kinds of clients of the API: 1) those who don't want to bother with block chain reorgs and therefore can just play it safe and request only transactions which have a high number of confirmations (and therefore have an infinitesimal chance of belonging to a block which might be discarded down the road), and 2) those who want absolute control, 100% robustness, and are willing to deal with possible block chain reorgs.

My suggestion is to base the API call not on transactions but on blocks.  After all, transactions are grouped into blocks, and saying that a given transaction has N confirmations means that all transactions in that same block also have N confirmations.  It therefore makes more sense for the client application to receive transactions in block-based batches.

So, I propose a gettransactionssince method with the following parameters:


If given, the optional parameter account restricts the result set to transactions pertaining to that account.  The also optional parameter blkhash indicates the block hash of the last block the client knows about.  When this parameter is given, the Bitcoin daemon is supposed to return all transactions which have occurred since this block (but not including the transactions in the block itself!).  If not given, then all transactions since the beginning should be returned.

Very important: together with a set of transactions, the method gettransactionssince should also return the block hash and the block number of the last block in the chain with minconf confirmations (and whose transactions are possibly contained in the result set).  This way, the client can in subsequent invocations tell the Bitcoin daemon what it already knows about.

Note the minconf parameter.  I suggest setting its default value to a number so high that the probability of a reorg affecting the returned blocks is practically zero.  This way, "dumb" clients can just call gettransactionssince using a different blkhash each time, and not worry about any other book-keeping.

As for the blknum parameter, it can be used for an early breakout of the cycle in the event that there's been a reorg and therefore blkhash will not be found.  If it's not given, then the daemon will have to loop all the way to the beginning before it can detect and report an error situation.

Smarter clients can use blknum together with a lower minconf value to ensure a lower latency for confirmations; they should however take care of some extra book-keeping in case there's been in fact a reorg.  The algorithm for this is not that complicated, but it is in any case the sole responsibility of the client to implement it correctly; the Bitcoin daemon should not have to care.

Anyway, what do you think?

Best regards,
Jon

It doesn't 'feel right' to me.

Seems like the ideal API would be:

"Hey bitcoin, I want to keep track of all transactions for account FOO (or all accounts) that have [minconf] confirmations.  Please POST them to [url]."

  or

"Hey bitcoin, I want to keep track of all transactions for account FOO (or all accounts) that have [minconf] confirmations.  I'll be polling you to see if there are any new ones every once in a while, I'll pass you [unique_token] so you know it is me."

... at least for the simple case.  You'd get back two lists of transactions:  new transactions with [minconf] that you haven't been told about before (maybe empty in the polling case), and a list of transactions you were told about before that now have less than [minconf] confirmations because of a block chain re-org (always empty if [minconf] is big enough).



For the "I really want to shoot myself in the foot and deal with block-chain reorgs myself" you can call getblock and/or monitorblock to get all the gory details about which transactions are in which blocks.

As I explain below, whether it feels right or not is mostly a function of the role one envisions for the Bitcoin daemon.


I agree that would also be useful.  But note that it requires some extra book-keeping on the Bitcoin daemon, in order to keep track of possible lost messages.


That's essentially what I want, with the difference that my proposal does not require the Bitcoin daemon to be stateful in respect to this method implementation; it is instead the client's responsibility to keep track of the state.

Choosing one approach or another is a matter of deciding what role should the bitcoind daemon play.  Should it be a relatively dumb gateway to the Bitcoin protocol? (in which case my approach makes more sense) Or should it make life as easy as possible for clients, even at the expense of being a more complex daemon? (in which case your solution is appropriate)  If this fundamental decision has already been made, then the choice of approach should be straightforward.


I'm running version 0.3.19 and I cannot find neither getblock nor monitorblock.  Are these new methods coming for 0.3.20?  Well, if getblock does what I think it does (ie, return all information available about a given block number, including block hash and transactions) then it would of course be an alternative way to implement the "shoot-myself-in-the-foot" client.  Everybody can be happy, yay!

A very large site would probably want use getblock, anyway, since it allows you to talk to Bitcoin only once per block instead of querying it every time you want to do anything.

They're not in 0.3.20; maybe 0.3.21.
  https://github.com/gavinandresen/bitcoin-git/tree/monitorreceived
... is the not-yet-ready-for-prime-time branch they're on.