Bitcoin Forum
December 04, 2016, 04:40:36 PM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1]
  Print  
Author Topic: JSON-RPC method idea: list transactions newer than a given txid  (Read 5418 times)
davux
Sr. Member
****
Offline Offline

Activity: 289


Firstbits.com/1davux


View Profile WWW
December 08, 2010, 05:07:21 PM
 #1

It would be great to have a JSON-RPC method for listing new transactions that are newer than a particular transaction id. This would enable developpers to watch new transactions easily, by just keeping track of the latest known txid and polling that method at the rate of their choice.

A possible way to do it would be to extend listttransactions so that it accepts an optional txid argument:
Code:
listtransactions <account> [count=10] [txid]

If txid is given, don't show it, nor any transaction older than it.

(As a sidenote, maybe listtransactions, with or without txid, should accept filtering by transaction category: generate, send, receive, move.)

I'm sorry I can't propose any patch here, as I can't do any decent C++.

1DavuxH9tLqU4c7zvG387aTG4mA7BcRpp2
México (Oaxaca) – France - Leeds
1480869636
Hero Member
*
Offline Offline

Posts: 1480869636

View Profile Personal Message (Offline)

Ignore
1480869636
Reply with quote  #2

1480869636
Report to moderator
1480869636
Hero Member
*
Offline Offline

Posts: 1480869636

View Profile Personal Message (Offline)

Ignore
1480869636
Reply with quote  #2

1480869636
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480869636
Hero Member
*
Offline Offline

Posts: 1480869636

View Profile Personal Message (Offline)

Ignore
1480869636
Reply with quote  #2

1480869636
Report to moderator
1480869636
Hero Member
*
Offline Offline

Posts: 1480869636

View Profile Personal Message (Offline)

Ignore
1480869636
Reply with quote  #2

1480869636
Report to moderator
nelisky
Legendary
*
Offline Offline

Activity: 1554


View Profile
December 08, 2010, 05:44:01 PM
 #2

+1

I haven't got my hands dirty there yet, and would rather spend what time I find to do import/export of keys, but that is something that would help some of my scripts a lot.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
December 08, 2010, 07:01:02 PM
 #3

It would be great to have a JSON-RPC method for listing new transactions that are newer than a particular transaction id. This would enable developpers to watch new transactions easily, by just keeping track of the latest known txid and polling that method at the rate of their choice.

If we're adding code to watch for new transactions, then we should be including gavin's monitor patches:
https://github.com/gavinandresen/bitcoin-git/tree/monitorreceived

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
genjix
Legendary
*
expert
Offline Offline

Activity: 1232


View Profile
December 08, 2010, 08:20:57 PM
 #4

Surely since the WalletTx are added sequentially to the wallet, we can assume that older txid's appear earlier in the wallet?

Then it's simply trivial to do a linear lookup and then continue on spitting out the txid's after that one.

Can anyone confirm this?
satoshi
Founder
Sr. Member
*
qt
Offline Offline

Activity: 364


View Profile
December 08, 2010, 08:21:49 PM
 #5

It's not safe to use listtransactions this way.

I know I've been criticized for being reluctant about listtransactions.  Let me explain my reluctance.

Transactions are dynamic.  Past transactions can become unconfirmed, go away and come back, become invalid and disappear, or be replaced by a different double-spend.  Their date can change, their order can change.

Programmers are naturally inclined to want to use listtransactions like this: feed me the new transactions since I last asked, and I'll keep my own tally or static record of them.  This will seem to work in all regular use, but if you use the amounts for anything, it is highly exploitable:
1) How do you know if a past transaction becomes invalid and disappears?
2) When there's a block-chain reorg, it would be easy to double-count transactions when they get confirmed again.
3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.

The model where you assume you only need to see new transactions because you've already seen previous transactions is not true.  Old transactions can change at any time.

Any time you take an action based on payment amounts received, you always need to go back to bitcoin and ask for a current balance total (or use move or sendfrom), and be ready for the possibility that it can go down.

Now that we have the Accounts feature making it easier to do it the right way, we're better prepared to have listtransactions.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
December 08, 2010, 08:22:42 PM
 #6

Surely since the WalletTx are added sequentially to the wallet, we can assume that older txid's appear earlier in the wallet?

Then it's simply trivial to do a linear lookup and then continue on spitting out the txid's after that one.

The wallet is a key/value db4 database (and a key/value map in RAM).

Neither data structure is ordered by time.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
mikegogulski
Sr. Member
****
Offline Offline

Activity: 360



View Profile WWW
December 08, 2010, 08:41:41 PM
 #7

I can see the motivation for davux's request. In developing the e-commerce plugins I've released, I've wondered about the viability long term of using a "dump" method against the wallet, since over time the data returned is just going to grow and grow.

What I'd prefer is being able to attach a JSON-RPC callback URL to an account or an address, maybe to be invoked each time a confirmation or any other status change to that account or address comes in, until the callback is cleared. That would eliminate the need for polling the wallet entirely.

I threw a "feature request" post in about that someplace.

FREE ROSS ULBRICHT, allegedly one of the Dread Pirates Roberts of the Silk Road
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
December 08, 2010, 08:46:03 PM
 #8

I know I've been criticized for being reluctant about listtransactions.  Let me explain my reluctance.

Transactions are dynamic.  Past transactions can become unconfirmed, go away and come back, become invalid and disappear, or be replaced by a different double-spend.  Their date can change, their order can change.

Programmers are naturally inclined to want to use listtransactions like this: feed me the new transactions since I last asked, and I'll keep my own tally or static record of them.  This will seem to work in all regular use, but if you use the amounts for anything, it is highly exploitable:
1) How do you know if a past transaction becomes invalid and disappears?
2) When there's a block-chain reorg, it would be easy to double-count transactions when they get confirmed again.
3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.

At some point, a website or person accepting a transaction must take this risk.  It is unavoidable, whether you use listreceivedbyaddress or listtransactions.  This is why listtransactions reluctance seems so unusual.

Almost every exchange or website accepting bitcoins achieves a binary decisionpoint, where the transaction is accepted, and goods are shipped or money is exchanged.  After that binary decisionpoint, even if the block chain is reorg'd or transactions disappear, there is nothing the website can do but take a loss (or pursue a refund outside of bitcoin).

From the website's point of view, there is zero effective difference between 'listtransactions 6' and 'listreceivedbyaddress 6', because the end result to the website operator is the same:  the goods have been shipped / order accepted / money exchanged.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
satoshi
Founder
Sr. Member
*
qt
Offline Offline

Activity: 364


View Profile
December 08, 2010, 10:36:45 PM
 #9

Then how do you cope with the issues I listed in the message you quoted?
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
December 08, 2010, 11:07:22 PM
 #10

I think the important point is that these things are unavoidable unsafe after a certain point, even for listreceivedbyaddress.  So I think it's helpful to compare current website behavior under mainline bitcoin, with listtransactions.   Let us call this the confirmation point.  Addressing your queries...

1) How do you know if a past transaction becomes invalid and disappears?

bitcoinmarket and mtgox and other sites seem to consider 6 confirmations their "confirmation point", the moment at which a transaction may be considered "safe." If a past transaction becomes invalid and disappears, the website cannot avoid potential loss, because the user has already received their PayPal-USD or LR-USD or Pecunix GAU and disappeared.

Same for a web store or brick-and-mortar store.  There is a confirmation point at which the customer receives goods.  If a TX becomes invalid after that, the store takes an unavoidable loss, because the customer is already gone with the purchased goods.

2) When there's a block-chain reorg, it would be easy to double-count transactions when they get confirmed again.

Let us assume that confirmation count for txid 0x1234 is bouncing wildly between 0 to 10 to 0 and back and forth, due to a large number of block reorgs.

Whether it is listreceivedbyaddress or listtransactions, you still have a binary confirmation point, a moment in time, at which the transaction crosses the "approved by store" level of confidence.  At that confirmation point, the customer leaves with purchased goods, and store takes a loss regardless of further block chain or TX behavior.

I do agree a programmer may make a mistake, and assume that number of confirmations will always increase.  But that human mistake, too, will cause danger when used with listreceivedbyaddress.

3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.

On this point, I agree that listtransactions presents some additional danger here, due to changing txid, if and only if the new double-spend matches destination bitcoin address being sought by the JSON-RPC user.

Nevertheless, in this case too, the user's security rests entirely on their level of confidence:  if the TX is replaced before 6 confirmations, software will likely not notice anything.  If the TX is replaced after 6 confirmations, the customer has already left with purchased goods, and the bitcoin user takes a loss.


This is simply inherent to bitcoin itself.  A block chain reorg might happen after 50 blocks.  But websites do not want to make their users wait 50 blocks before receiving goods.  It is the well-known snack machine problem.  listtransactions does not add anything to this problem, beyond that which is already vulnerable through listreceivedbyaddress.

Transactions can and will be replaced after the binary "confirmation point."  All users of bitcoin must figure this into their business plans, just like they account for credit card chargeback risk or shoplifting risk.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
davux
Sr. Member
****
Offline Offline

Activity: 289


Firstbits.com/1davux


View Profile WWW
December 08, 2010, 11:40:59 PM
 #11

Any time you take an action based on payment amounts received, you always need to go back to bitcoin and ask for a current balance total (or use move or sendfrom), and be ready for the possibility that it can go down.

So it would be a design problem in the developper's app, not a problem inherent to listtransactions. What you're saying is: if listtransactions is used without care, it can be dangerous. OK, but many programmatic tools have traps too. For example, programming with threads is full of design traps. That doesn't mean that that tool shouldn't be made available to developpers, but that the documentation must warn strongly about those traps.

My point: let's make listtransactions available, but document it precisely, like any API, and explain the design mistakes that shouldn't be made when using it.

1DavuxH9tLqU4c7zvG387aTG4mA7BcRpp2
México (Oaxaca) – France - Leeds
satoshi
Founder
Sr. Member
*
qt
Offline Offline

Activity: 364


View Profile
December 09, 2010, 12:12:17 AM
 #12

I'm not talking about the normal risk for a given minconf level, I'm talking about additional pitfalls from listtransactions when used this way.

2) When there's a block-chain reorg, it would be easy to double-count transactions when they get confirmed again.
The OP's example of listtransactions <account> [count=10] [txid] seems to imply and it would be very easy for programmers to assume that if they pass in the last txid of the previous call to listtransactions, they will never see the same transaction more than once, which is not the case.  It would be very easy to double-count payments if you don't maintain your own persistent map or dictionary to track which txid's you've already accepted.

It doesn't seem right to have a function that seems tailor made to be used a certain obvious way, and that way is a non-obvious trap.

3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.
listtransactions does not add anything to this problem, beyond that which is already vulnerable through listreceivedbyaddress.
Suppose both spends are to the same address.  getreceivedbyaddress would always count only one or the other spend at any given time, never both.

Using listtransactions, it would be very easy to count both.  You see the first spend, you count it.  You see the second spend, you count it.  Total is double counted.
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
December 09, 2010, 12:41:44 AM
 #13

I'll and add another reason not to have a "list transactions that happened after <txid>" :

move "transactions" don't have a transaction id, but they do affect account balances (and are listed in listtransactions).

Your code is going to get really messy if you expect to call listtransactions and then squirrel away the txid of the last item returned.  If it was "category":"move",  there WILL be no txid...

RE: eliminating polling:  at some point fairly soon, I plan on cleaning up my "monitorreceived" patch, to POST to a URL when transactions come in or blocks are accepted... but I need to do some Deep Thinking to redesign based on lessons learned from 'accounts'.  It might turn into a very minimal API, where the notification is "Hey, txid <123ae4221...> just got to N confirmations, you might want to call gettransaction and getbalance to get up-to-date."

How often do you get the chance to work on a potentially world-changing project?
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
December 09, 2010, 12:58:05 AM
 #14

I'll and add another reason not to have a "list transactions that happened after <txid>" :

I agree with you and satoshi about "txs after <txid>".  My listtransactions (now xlisttransactions) patch pointedly does not have that feature, and never has.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
ribuck
Donator
Legendary
*
Offline Offline

Activity: 826


View Profile
December 09, 2010, 11:48:56 AM
 #15

If a past transaction becomes invalid and disappears, the website cannot avoid potential loss, because the user has already received their PayPal-USD or LR-USD or Pecunix GAU and disappeared.

That's not always the case. Sometimes the customer will have a balance at the website. The operator of the website will certainly want to know if a past transactions becomes invalid and disappears.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1470


View Profile
December 09, 2010, 04:13:50 PM
 #16

If a past transaction becomes invalid and disappears, the website cannot avoid potential loss, because the user has already received their PayPal-USD or LR-USD or Pecunix GAU and disappeared.

That's not always the case. Sometimes the customer will have a balance at the website. The operator of the website will certainly want to know if a past transactions becomes invalid and disappears.

Sure, and that's easy enough to track with transactions.

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
satoshi
Founder
Sr. Member
*
qt
Offline Offline

Activity: 364


View Profile
December 09, 2010, 06:08:08 PM
 #17

I agree with you and satoshi about "txs after <txid>".  My listtransactions (now xlisttransactions) patch pointedly does not have that feature, and never has.
As long as the interface is designed for things like showing the user the last N transactions history, it's fine, now that we have the Accounts feature making it easier to do payment detection the right way.

Gavin, could listtransactions have an option to list transactions for all accounts?

I'm not sure what the interface could be, maybe:
listtransactions <JSON null type> [count]

It would be hard to do that from the command line though.

I can't think of a good solution for the interface, that's the problem.  Maybe "*" special case like "" is.  Everyone would have to make sure no user can create account name "*".

Sure, and that's easy enough to track with transactions.
I don't get how that's "easy" to track with transactions.
davux
Sr. Member
****
Offline Offline

Activity: 289


Firstbits.com/1davux


View Profile WWW
December 09, 2010, 07:23:52 PM
 #18

Gavin, could listtransactions have an option to list transactions for all accounts?

That would be great!

I'm not sure what the interface could be, maybe:
listtransactions <JSON null type> [count]

It would be hard to do that from the command line though.

What about the empty string? It's easy to use in a programmatic context (e.g. in Python), and also in a shell context:
Code:
bitcoind listtransactions ''

Moreover it would be consistent with the current API, since the empty string already means "any account", for example when used with "sendfrom" (and probably in other situations). Let's not use several special cases that are special in subtle, different ways. The simpler, the better.

1DavuxH9tLqU4c7zvG387aTG4mA7BcRpp2
México (Oaxaca) – France - Leeds
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
December 09, 2010, 07:41:33 PM
 #19

Gavin, could listtransactions have an option to list transactions for all accounts?

I'm not sure what the interface could be, maybe:
listtransactions <JSON null type> [count]

It would be hard to do that from the command line though.

I can't think of a good solution for the interface, that's the problem.  Maybe "*" special case like "" is.  Everyone would have to make sure no user can create account name "*".


Yes, listtransactions "*" <count> is possible.  The other account routines could return a new "invalid account name" error if given "*".

I've got two issues with it, though:

1.  listtransactions "*" will have to iterate over every transaction in the wallet (transactions are not indexed by time), which will be slow for large wallets and will get slower over time.  And indexing transactions just so that listtransactions * is fast violates the "an optional feature shouldn't cost anything if it is not used" principle.

2. What is the use case for "list the last N transactions across all accounts" ?   The only one I can come up with is developing an alternate GUI that communicates with bitcoind via the JSON-RPC, but to support that at least a couple of other features would have to be added at the same time (e.g. listtransactions would needs to add account and bitcoin address information to the objects it returns....)

How often do you get the chance to work on a potentially world-changing project?
abourget
Newbie
*
Offline Offline

Activity: 8


View Profile
July 31, 2013, 08:18:06 PM
 #20

What is the new state of affairs regarding this issue, after 3 years ?

Is there a way to poll or get notified when things happen on the network ?

thanks Smiley

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!