Rescan on btcd and bitcoind gives a different set of unspents

[arnuschky]

Currently testing Bitcoin Core Daemon version v0.10.0.0-g047a898 and btcwallet version 0.5.1-alpha.
Wallets have the same 14 addresses, freshly rescanned, no activity since.

Calls:

Code:

$ bitcoin-cli listunspent|grep txid|wc -l
169
$ btcctl listunspent|grep txid|wc -l
173

So bitcoin core is missing 4 transactions. These are known to http://tbtc.blockr.io/ and have been confirmed since a long time (different blocks). Does this mean that bitcoin core has a bug here? Or am I missing something?

[1714893341]

1714893341

[1714893341]

1714893341

[gmaxwell]

listunspent has nothing to do with the utxo set in Bitcoin core, its a wallet function.

You haven't provided basically anything to go on here:   No addresses where someone could attempt a reproduction, no list of missing transactions, no information about how you created this state (e.g. which side did you import keys into and how), no information about how the misisng scriptpubkeys are shaped.  I'm not trying to be snarky but http://www.catb.org/esr/faqs/smart-questions.html  covers some important life skills which too few people are taught and which you might benefit from!   There are basically only two possible answers to your question:  "I have no idea and you've not given enough information to figure anything out"  or "This sounds like an already known bug, but you haven't given enough information to be completely sure".  I'm not aware of any known bug that sounds like this and likely would be aware of it if there were a known one on the Bitcoin Core side.

Why are you assuming this is a Bitcoin Core bug?-- could it be that btcd is displaying txouts that have no business being in your wallet (e.g. weird scriptpubkeys or already spent txouts)?  (I'm not arguing that it isn't, or that you don't have a reason to-- you've just not specified that reason whatever it is, and to investigate I must know!). It would be somewhat surprising if there were some systemic reason for entries to now show up in listunspent and no one were reporting it, but there is always a first one.

Do you have a reason to believe this just isn't missing change addresses and transactions involving them?  Try locating a missing entry and run validateaddress  with the address the missing entry has on the wallet that is missing it.

If you send me the _whole_ listunspent output from each, and a precise description of how you created the wallets, I'll look into it-- with that I should be able to reproduce the behavior.  Feel free to use PGP to preserve your privacy (and that of those you've transacted with).   If you're unable to share this information with me, I'll still be glad to help, but it will be much harder.

[arnuschky]

Yes, I realized later that my question was kind of incomplete/posed badly - apologies for not taking the time to properly phrase it.
Not problem with getting a snarky reply, the question deserved it.

What I meant to ask was whether there is a known bug or whether I was overlooking something obvious before I debug it more thoroughly.

I tracked it down in the meantime. It turns out that btcd includes unspents with zero amount while bitcoin core does not.
Including a zero-amount output might be more complete (it is an unspent output, after all) but it's rather useless as the output can't be spent.
No idea how I missed it yesterday, I must have been tired...

I opened a bug with the `btcd` folks, let's see what they say. Sorry for the noise, people. 

[gmaxwell]

Zero amount outputs can be spent; a bitcoin transaction need not involve any bitcoins-- this is part of the premise by which some of these embedded consensus altcoins hope to replace the Bitcoin currency.. They're non-standard to create, so they need miner cooperation.

Now I'm extra interested, since that sounds like a bug unless they're actually unspendable for some other reason.

[arnuschky]

Ah, I thought that they are not spendable at all, but yes, I now remember reading that Eligus includes some 'non-standard' transactions, right?

Here is an example transaction on testnet:
1d6e3301bf2ccb4abd61109e9bfd8935061b8641ee8ae3f1058d03b71fbba7b9 (vout 0 is the unspent missing from my bitcoind)

I can dig up more data tomorrow, have to go now.

[arnuschky]

I deleted the wallet and imported only a single private key into it. After scanning finished, I ended up with the following:

Code:

$ bitcoin-cli listunspent
[
    {
        "txid" : "7e7723e9bde41663e9581e9440aebd522d64b665966f92a6f04a1ea6a512ecab",
        "vout" : 1,
        "address" : "mgQnKMysJokhb5x7u3ASjTaxsurqTH4obp",
        "account" : "",
        "scriptPubKey" : "76a91409cd4d1fc8fcb00f976538bc56c8d616796810ff88ac",
        "amount" : 0.00010000,
        "confirmations" : 142125,
        "spendable" : true
    }
]

btcd/btcwallet gives me the following:

Code:

[

	{
		"txid": "7e7723e9bde41663e9581e9440aebd522d64b665966f92a6f04a1ea6a512ecab",
		"vout": 1,
		"address": "mgQnKMysJokhb5x7u3ASjTaxsurqTH4obp",
		"account": "imported",
		"scriptPubKey": "76a91409cd4d1fc8fcb00f976538bc56c8d616796810ff88ac",
		"amount": 0.0001,
		"confirmations": 142127
	},
	{
		"txid": "1d6e3301bf2ccb4abd61109e9bfd8935061b8641ee8ae3f1058d03b71fbba7b9",
		"vout": 0,
		"address": "mgQnKMysJokhb5x7u3ASjTaxsurqTH4obp",
		"account": "imported",
		"scriptPubKey": "76a91409cd4d1fc8fcb00f976538bc56c8d616796810ff88ac",
		"amount": 0,
		"confirmations": 254254
	},
	{
		"txid": "eb8289d13f0dc240be6a154506fbe94ea73d3dfe910de704298d5d92aa2dfcbb",
		"vout": 0,
		"address": "mgQnKMysJokhb5x7u3ASjTaxsurqTH4obp",
		"account": "imported",
		"scriptPubKey": "76a91409cd4d1fc8fcb00f976538bc56c8d616796810ff88ac",
		"amount": 0,
		"confirmations": 255628
	},
	{
		"txid": "26c210136af16e163af51eea47e5b59965ffe83e216f3e074af31739b493ee1f",
		"vout": 0,
		"address": "mgQnKMysJokhb5x7u3ASjTaxsurqTH4obp",
		"account": "imported",
		"scriptPubKey": "76a91409cd4d1fc8fcb00f976538bc56c8d616796810ff88ac",
		"amount": 0,
		"confirmations": 255630
	}
]

This is confirmed by other parties, ie https://api.chain.com/v2/testnet3/addresses/mgQnKMysJokhb5x7u3ASjTaxsurqTH4obp/unspents?api-key-id=DEMO-4a5e1e4 or https://www.blocktrail.com/tBTC/tx/1d6e3301bf2ccb4abd61109e9bfd8935061b8641ee8ae3f1058d03b71fbba7b9

Nothing weird in the output scripts either.

I'll wipe the chain and let it resync, it's an old test machine and I am not 100% that everything is in order there.

[arnuschky]

Redownloaded the blockchain, and rescanned the wallet. Same result.

The fact that all tested 3rd party services indicates that the issue is with my system rather than with bitcoin core itself, but I don't know what else could cause this issue.

Currently using latest bitcoind on Ubuntu 64bit 12.04.
I can spin up a fresh instance and test there, if necessary.

Any tips on how to proceed further?

[arnuschky]

Josh Rickmar over at btcd found the reason. Bitcoin core's wallet explicitly filters zero-amount outputs after all:

From CWallet::AvailableCoins in src/wallet/wallet.cpp:

Code:

            for (unsigned int i = 0; i < pcoin->vout.size(); i++) {
                isminetype mine = IsMine(pcoin->vout[i]);
                if (!(IsSpent(wtxid, i)) && mine != ISMINE_NO &&
                    !IsLockedCoin((*it).first, i) && pcoin->vout[i].nValue > 0 &&
                    (!coinControl || !coinControl->HasSelected() || coinControl->IsSelected((*it).first, i)))
                        vCoins.push_back(COutput(pcoin, i, nDepth, (mine & ISMINE_SPENDABLE) != ISMINE_NO));
            }

This behavior is different from all the other APIs that I've tested. If no objections, gmaxwell, I'd file a bug.

[gmaxwell]

Feel like testing a patch?

Also PRed at: https://github.com/bitcoin/bitcoin/pull/6036


diff --git a/src/wallet/rpcwallet.cpp b/src/wallet/rpcwallet.cpp
index e03cd5b..c31c09d 100644
--- a/src/wallet/rpcwallet.cpp
+++ b/src/wallet/rpcwallet.cpp
@@ -2301,7 +2301,7 @@ Value listunspent(const Array& params, bool fHelp)
     vector<COutput> vecOutputs;
     assert(pwalletMain != NULL);
     LOCK2(cs_main, pwalletMain->cs_wallet);
-    pwalletMain->AvailableCoins(vecOutputs, false);
+    pwalletMain->AvailableCoins(vecOutputs, false, NULL, true);
     BOOST_FOREACH(const COutput& out, vecOutputs) {
         if (out.nDepth < nMinDepth || out.nDepth > nMaxDepth)
             continue;
diff --git a/src/wallet/wallet.cpp b/src/wallet/wallet.cpp
index 2566b27..1ed0c03 100644
--- a/src/wallet/wallet.cpp
+++ b/src/wallet/wallet.cpp
@@ -1481,7 +1481,7 @@ CAmount CWallet::GetImmatureWatchOnlyBalance() const
 /**
  * populate vCoins with vector of available COutputs.
  */
-void CWallet::AvailableCoins(vector<COutput>& vCoins, bool fOnlyConfirmed, const CCoinControl *coinControl) const
+void CWallet::AvailableCoins(vector<COutput>& vCoins, bool fOnlyConfirmed, const CCoinControl *coinControl, bool fIncludeZeroValue) const
 {
     vCoins.clear();
 
@@ -1508,7 +1508,7 @@ void CWallet::AvailableCoins(vector<COutput>& vCoins, bool fOnlyConfirmed, const
             for (unsigned int i = 0; i < pcoin->vout.size(); i++) {
                 isminetype mine = IsMine(pcoin->vout);
                 if (!(IsSpent(wtxid, i)) && mine != ISMINE_NO &&
-                    !IsLockedCoin((*it).first, i) && pcoin->vout.nValue > 0 &&
+                    !IsLockedCoin((*it).first, i) && (pcoin->vout.nValue > 0 || fIncludeZeroValue) &&
                     (!coinControl || !coinControl->HasSelected() || coinControl->IsSelected((*it).first, i)))
                         vCoins.push_back(COutput(pcoin, i, nDepth, (mine & ISMINE_SPENDABLE) != ISMINE_NO));
             }
diff --git a/src/wallet/wallet.h b/src/wallet/wallet.h
index 4dbb0e2..c5f876f 100644
--- a/src/wallet/wallet.h
+++ b/src/wallet/wallet.h
@@ -540,7 +540,7 @@ public:
     //! check whether we are allowed to upgrade (or already support) to the named feature
     bool CanSupportFeature(enum WalletFeature wf) { AssertLockHeld(cs_wallet); return nWalletMaxVersion >= wf; }
 
-    void AvailableCoins(std::vector<COutput>& vCoins, bool fOnlyConfirmed=true, const CCoinControl *coinControl = NULL) const;
+    void AvailableCoins(std::vector<COutput>& vCoins, bool fOnlyConfirmed=true, const CCoinControl *coinControl = NULL, bool fIncludeZeroValue=false) const;
     bool SelectCoinsMinConf(const CAmount& nTargetValue, int nConfMine, int nConfTheirs, std::vector<COutput> vCoins, std::set<std::pair<const CWalletTx*,unsigned int> >& setCoinsRet, CAmount& nValueRet) const;
 
     bool IsSpent(const uint256& hash, unsigned int n) const;

[arnuschky]

Will try tomorrow, no time today.

[arnuschky]

For the record, gmaxwell's patch works fine: https://github.com/bitcoin/bitcoin/pull/6036