By my reading of Bitcoin Core's source code, the RPC call gettxoutproof won't work if pruning is enabled and if the relevant block has been deleted from disk.

I'm interested in this for Electrum Personal Server. Electrum wallet requests a merkle proof for each transaction otherwise it will display it as "Not Verified". For a non-pruned node this can be easily found with gettxoutproof and sent to Electrum, but if the node is pruned this often won't work.

What might be the best solution for my situation? The only idea I can come up with is to write code for Bitcoin Core that stores transaction merkle proofs in wallet.dat along with the rest of the transactions. Those proofs could then be accessed via the gettransaction RPC call.

[gateway]

Electrum uses Electrum servers which can spy on or lie to wallet users. Electrum wallets may be tricked into accepting transactions of BitcoinXT or some other hostile hard fork proposal. And the servers have a list of all the user's bitcoin addresses which is harmful to privacy. Right now Electrum comes packaged with Tails OS which has the tagline "Privacy for anyone anywhere", which is simply not true when using Electrum with default servers.

Another way to synchronize a wallet would be to download full blocks and/or point the wallet to your own full node. One of Electrum's values is "instant on" so those methods will probably never be added to Electrum itself.

Probably a better way would be to create a gateway, a script that behaves like an Electrum server but which obtains bitcoin network information in another way. To use it you would just run the gateway script and point your Electrum wallet to localhost. The script would be separate from the Electrum software itself so Electrum's raison d'etre which includes "instant on" would be preserved.

This is basically a proposal for an improved lightweightness of Electrum server. The goal is to keep all of Electrum's feature-richness of hardware wallet integration, multisignature, offline signing, coin control, etc but with better privacy, full node verification and minimal resource requirement compared to a full server.

Here are some gateway script ideas:

Type 1
The gateway script adds watch-only addresses to Bitcoin Core's wallet via the RPC protocol. Updates to the wallet.dat are transmitted to Electrum via the gateway.
Features: Full verification, security and privacy from a full node. Works with pruning and blocksonly. Reasonably simple to code.
Downsides: Adding wallets with historical transaction being added will require rescanning. Users must configure the gateway script with their Electrum master public keys before using.

Type 1 would be combining Bitcoin Core with Electrum. All of Bitcoin's configuration tuning like -blocksonly and -pruning can be used and txindex can be switched off. It would provide a way to use Electrum wallet with as low resource usage as possible while still being a full node. It would also be reasonably simple to code, only requiring Electrum's protocol and Bitcoin's RPC to be implemented and have them connected together.
I've heard a story of a tourist travelling through rural South America with a laptop, every couple of days at the cheap hostels they stayed they would run Bitcoin with -blocksonly to catch up. The type 1 gateway script would allow them to use Electrum if that was their preferred wallet.

Regarding rescanning. The user should have all their Electrum wallets imported into Bitcoin Core's wallet.dat file, so then they can switch between them without rescanning. If a brand new empty wallet with no transactions is created, there is also no need to rescan. The only time a rescan is needed is if new wallets are added which have historical transactions.

Type 2
The gateway script connects to the bitcoin p2p network and downloads full blocks (possibly with committed block bloom filters).
Features: Excellent privacy, almost as good as full node. SPV security just like Electrum today. Doesn't require any setup.
Downsides: Uses (much) more initial bandwidth than an Electrum server. Slower startup. Fairly complicated to code. Requires using new technology (committed bloom filters) before it can be practical.

Type 2 would be trading off bandwidth and speed for privacy. The wallet downloads some full blocks which takes longer, but in return it gets much better privacy. This script could be added to Tails OS as a pre-installed pre-configured script. Note that scanning blocks only needs to start from the wallet creation date, so if the user creates their wallet right now then the gateway script won't need to scan any old blocks.


Thoughts?

Pooled mining in bitcoin contributes to miner centralization. P2Pool is one solution but has bad scalability; additional hashers require the coinbase transaction to be larger, bigger miners joining increase the variance of payouts for everyone else, and smaller miners must pay extra to consolidate dust payouts. In this email I propose an improved scheme using payment channels which would allow far more individual hashers to mine on p2pool and result in a much lower payout variance.


== Intro ==

P2Pool is a decentralized pool that works by creating a P2P network of hashers. These hashers work on a chain of shares similar to Bitcoin's blockchain. Each hasher works on a block that includes payouts to the previous shares' owners and the node itself. The point of pooling is to reduce the variance of payout, even though on average the reward is the same (or less with fees). The demand for insurance, and the liquid markets for options show that variance does have costs that people are willing to pay to avoid.

Here is an example of a p2pool coinbase transaction: https://blockchain.info/tx/d1a1e125ed332483b6e8e2f128581efc397582fe4c950dc48fadbc0ea4008022

It is 5803 bytes in size, which at a fee rate of 350 sat/b is worth 0.02031050 btc of block space that p2pool cannot sell to any other transaction. As bitcoin inflation goes down and miners are funded more by fees, this puts p2pool at more and more of a disadvantage compared to trusted-third-party mining pools.

As each hasher is paid to their own bitcoin address, this limits the number of hashers taking part as adding more individual people to the payout transaction increases its size. Also small payouts cost a disproportionate amount in miner fees to actually spend, which hurts small miners who are essential to a decentralized mining ecosystem.

This could maybe be solved by keeping a separate balance state for each user that is independent from the payouts, and make payouts only when that balance state exceeds some reasonable threshold. But this increases the variance which goes against the aim of pooled mining.


== Payment Channels ==

P2pool's problems described above come from the fact that it's payout uses on-chain payments. We know the blockchain doesn't scale so it's natural to look for off-chain solutions.

What's needed is a way to use off-chain payments where any number of payments can be sent to each individual hasher without using the blockchain. Then all the parameters of p2pool can be modified to drastically reduce the payout variance. The N of the pay-per-last-N-shares (PPLNS) of p2pool can be increased to something like 6-12 months of shares and so as long as a small miner can mine a share every few months they will always get a payout when p2pool finds a block. The payment channels would be in a hub-and-spokes system and would work in a similar way to coinswap, lightning network, atomic cross-chain swaps or any other contract involving hashlocks and timelocks.

There would still be a sharechain but with hashers paying the entire block reward to a hub. This hub would have a one-way payment channel open to every hasher in p2pool and there would be created a situation where if the hub gets paid then the hashers cannot fail to get paid. Because cheating is impossible, the hub and hashers will agree to just release the money to each other without resorting to the blockchain.

The coinbase address scriptPubKey where block rewards are paid to would be this:

Code:

 2of2 multisig of hub + successful hasher
       OR
  hub pubkey + H(X)

A 2of2 multisig between the hub and the "successful" hasher which found the block, although with a hashlock and timelock. H(X) is a hash value, where the preimage X is a randomly-generated nonce which is kept secret by the hub, but X must be revealed if the hub spends via that execution path.

Each payment channel between the hub and a hasher is a 2of2 multisig between them.

The hashers mine a sharechain, a solved share contains the hashers pubkey. The hub keeps up with the sharechain and announces partially-signed transactions going to each hasher. The transactions are updated states of the payment channel, they pay money to each hasher in proportion to the work that the hasher contributed to the sharechain.

This is the form of an updated state of the payment channel:

Code:

2of2 multisig UTXO ----> [total - balance btc] hub-pubkey
                   ----> [balance btc]         hasher-pubkey + H(X)

If broadcasted, this transaction would allow the hasher to take the balance of the payment channel but only if it knows the value of X.

If a hasher is successful and finds a share that is also a valid bitcoin block, they broadcast it to the network.

Now, the hub can spend the block reward money on its own but only by revealing X. Each hasher could then take that X and combine it with the updated state transaction to get their money. So if the hub gets paid then the hashers cannot fail to get paid. Since defecting is pointless, the hub creates a new partially-signed transaction simply spending the 2of2 multisig output to each hasher, which now does not have any hashlock conditions. A hasher could at any time sign with their own private key and broadcast to get the money settled on the blockchain. After that, the successful hasher signs the 2of2 multisig block reward sending the block reward money to the hub. The successful hasher gets a small bonus via an updated payment channel state for finding the block, to discourage withholding same as today's p2pool. If everything goes well then the hub never reveals X, for mining a new block it announces a new value of H(X).

These payment channels can be kept open indefinitely, as new blocks are found by p2pool the hub creates new updated states with more money going to each hasher. When the hasher wants to stop mining and get the money, they can add their own signature and broadcast the most recent state to the network.

== Hubs ==

The hub is a central point of failure. It cannot steal the money, but if it gets DDOS'd or just becomes evil then the whole thing would stop working. This problem could be mitigated by having a federated system, where there are several hubs to choose from and hashers have payment channels open with each of them. It's worth noting that if someone has a strong botnet they could probably DDOS individual p2pool hashers in the same way they DDOS hubs or even centralized mining pools.

The hub would need to own many bitcoins in order to have payment channels while waiting for blocks to be mined. Maybe 50 times the block reward which today would be about 650 bitcoins. The hub should receive a small percentage of each block reward to provide them with an incentive, we know from JoinMarket that this percentage will probably be around 0.1% or less for large amounts of bitcoin. Prospostive hub operators should write their bids on a forum somewhere and have their details added to some list on github. Hashers should have an interface for blacklisting, whitelisting, lowering and raising priority for certain hubs in case the hub operators behave badly.

As well as the smart contract, there are iterated prisoner's dilemma effects between the hub and the hashers. If the hub cooperates it can expect to make a predictable low-risk income from its held bitcoins for a long time to come, if it does something bad then the hashers can easily call off the deal. The hub operator will require a lot of profit in order to burn its reputation and future income stream, and by damaging the bitcoin ecosystem it will have indirectly damaged its own held bitcoins. A fair pricing plan will probably have the hub taking a small percent to start with and then 12 months later that percentage goes up to take into account the hub's improved reputation.


== Transaction Selection ==

All the hashers and hub need to know the exact value of the block reward in advance, this means they must know what the miner fees will be. This is probably the most serious problem with this proposal.

One possible way to solve this is to mine transactions into shares and so use the sharechain to make all the hashers and hubs come to consensus about exactly which transactions they will mine, and so exactly what the total miner fee will be. A problem here is that this consensus mechanism is slow, immediately after a bitcoin block is found all the p2pool hashers will have to wait 30-120 seconds before they know what transactions to mine, so this would make them uncompetitive as a mining operation.

Another way to deal with this is to have the hub just choose all the transactions, announcing the transactions, total miner fee and merkle root for the hashers to mine. This would work but allows the hub to control and censor bitcoin transactions, which mostly defeats the point of p2pool as an improvement to bitcoin miner centralization.

Another way is to have the hashers and hub estimate what the total miner fee value will be. The estimate could start from the median miner fee of the last few blocks, or from the next 1MB of the mempool. The hub would announce all the partially-signed transactions to every hasher, and then periodically (say every 60 seconds) announce updated versions depending on how the mempool changes. Let's analyze what happens if the estimated and actual rewards are different. If the actual block reward is lower than the estimated reward, then the hub can update the payment channel state to slightly lower values to take that into account. The hashers can't use the higher channel state without knowing X. The successful hasher will get their bonus for finding the block which should help in encouraging them to actually sign the hub's payout transaction. If the actual block reward is higher than the estimated reward, the hub would hopefully still update the hasher's payment channel states because of the interated-prisoners-dilemma effects. But if the actual reward is much higher then the hub may find it profitable to burn its reputation and take the money by revealing X, one situation where this might happen is if someone accidentally pays a very high miner fee and a hasher mines it without it being taken into account in the hub's regular payment channel state updates. Apart from that very specific situation, this scheme of estimating the total miner fee should work. Note that hashers can choose which hub to mine for from their federation of hubs, they should avoid mining transactions which make the block reward much higher than the announced payment channel state, this should avoid the problem.


== Some Notes ==

*) Block rewards are locked for 100 blocks before they can be spent, so the cooperative signatures should be exchanged after 100 blocks just in case the block gets made stale/orphaned. While the hashers are waiting for the 100 reward maturity period, they should mine with another hub as the payout.

*) Today's p2pool has a feature for donating to individual hashers, this could be replicated in the payment channel system by having the hashers sign a LN payment code using the same pubkey that appears in the sharechain. Each LN payment code is then donated to in proportion to the work in the sharechain. After p2pool matures there should be an organized social movement asking for donations to the p2pool hashers. The economic majority has an incentive to support decentralized mining, and in the past when the status quo p2pool was bigger then it did receive them, so this is a realistic ask. The UASF movement was successful so we know such social movements are possible. The donation drive should continue until all public centralized mining pools have low hash power.

*) Each hasher should probably be made to pay some bitcoins into the payment channel address too, to stop DOSers locking up all the hub's bitcoins. If the hasher doesn't find a share within some timeout then the hub should close the payment channel. Another way to rate-limit DOS is for the hub to require a solved PoW challenge before opening a channel.

*) Now that we have segwit all these payment channel schemes are much easier to code.

*) The hashers must keep their money locked up in the payment channel for months before enough collects. This could be a problem because some miners don't really want to hold bitcoin long term. I wonder if theres some way to link up these channels to LN so they can sold straight away. They could also use futures contracts to sell the coins today at a discount and actually deliver the coins later when they close the channel. Edit: according to roasbeef such a connection with LN is indeed possible, then hashers would get bitcoins in their LN wallet which would be available for spending immediately. This would work by having bidirectional payment channels between hub and hashers, and the hub having open payment channels with the rest of the network. Then hashers can send their mining income via the hub to anyone in the lightning network

*) In order to stop holdups or ransom, there is an added OP_CSV execution path in the coinbase. The coinbase output actually has this script:

Code:

 2of2 multisig of hub + successful hasher
       OR
  hub pubkey + H(X)
       OR
  successful hasher pubkey + OP_CSV 6 months

Then the hub has a time limit to spending the coinbase and revealing X, if they don't then the successful hasher can steal the entire coinbase after 6 months.


== References ==

*) https://en.bitcoin.it/wiki/P2Pool how p2pool works

*) https://bitcointalk.org/index.php?topic=18313.msg13057899#msg13057899 the scaleability problems of p2pool

*) https://bitcointalk.org/index.php?topic=18313.msg20943086#msg20943086 making the PPLNS window longer

*) book: The Evolution of Co-Operation by Robert Axelrod, for explaining iterated prisoner's dilemma effects in detail

Thanks to the p2pool developer veqtrus for reviewing this

This is also posted on the bitcoin dev mailing list: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-August/014893.html

I've been trying to spend from a 2of3 multisig address, using the same procedure as from this gist: https://gist.github.com/gavinandresen/3966071

When trying to sign I get the "error": "Operation not valid with the current stack size"

What's really strange is I did this exact procedure some time in October 2015 and it worked. I must've been using a version of core around 0.10.2 or 0.11.0. If nobody has any idea I can try downgrading to one of those versions and seeing if it works.

Here's what I did:

Generated 3 keypairs (privkey, pubkey and address)

Code:

cMpfNfuasJcrCv8RKMG35C8YtUVpaheFbNMwDuBXVvdcjaeYE7oh
024c5965fb18701569ec09e8d2979c45e3d6e0c60ddb3c9d6af0692124d3890e8a
mgMZZtLSX1cU9jhAdQWMqMyoPv4JncZqcU

cVM5Kaoim2MKzT8MxVHHAu9BixJKTJgMhqape1eHgjHGhFPkf2mn
03c9084cf82f74f764829a7d3c842462e03f4bbaf59981cb3b98cd3974438e5771
mrdkuGp2wcXhcmtN7RwwUs7CYevJpYuLnV

cU4mHYwU2X1oTWBfYBGYFN69oYdrpWnBMrWd4nCoeRPUzmRY89vM
0256d87a1997bad87ebdd409a3437ecec02885300a1bb0a1159e289ae66af78b34
mospdWZdHtym59yMwUUi8w4HcWQxd7sQsv

Create 2of3 multisig address

Code:

createmultisig 2 '["024c5965fb18701569ec09e8d2979c45e3d6e0c60ddb3c9d6af0692124d3890e8a","03c9084cf82f74f764829a7d3c842462e03f4bbaf59981cb3b98cd3974438e5771","0256d87a1997bad87ebdd409a3437ecec02885300a1bb0a1159e289ae66af78b34"]'

{
  "address": "2N9nn3aMYzTjs2ccTRAvDUmdKykL9YjM2cj",
  "redeemScript": "5221024c5965fb18701569ec09e8d2979c45e3d6e0c60ddb3c9d6af0692124d3890e8a2103c9084cf82f74f764829a7d3c842462e03f4bbaf59981cb3b98cd3974438e5771210256d87a1997bad87ebdd409a3437ecec02885300a1bb0a1159e289ae66af78b3453ae"
}

Import address and fund it

Code:

importaddress 2N9nn3aMYzTjs2ccTRAvDUmdKykL9YjM2cj "" false

//sent 0.1btc to 2N9nn3aMYzTjs2ccTRAvDUmdKykL9YjM2cj

listunspent
[
  {
    "txid": "ff65917ac6cbe36d55893d9e1ce63cad1140c2ab38e9c7d099170b16d1a55d70",
    "vout": 0,
    "address": "2N9nn3aMYzTjs2ccTRAvDUmdKykL9YjM2cj",
    "account": "",
    "scriptPubKey": "a914b579aabfd350ed75ee056ff4d6d56718c74f55df87",
    "amount": 0.10000000,
    "confirmations": 1,
    "spendable": false,
    "solvable": false
  },

Create spending tx, sending to mqZa1zbDg5RHVYbX6ea1QxPuuxAyuGVSoK

Code:

createrawtransaction '[{"txid":"ff65917ac6cbe36d55893d9e1ce63cad1140c2ab38e9c7d099170b16d1a55d70","vout":0,"scriptPubKey":"a914b579aabfd350ed75ee056ff4d6d56718c74f55df87","redeemScript":"5221024c5965fb18701569ec09e8d2979c45e3d6e0c60ddb3c9d6af0692124d3890e8a2103c9084cf82f74f764829a7d3c842462e03f4bbaf59981cb3b98cd3974438e5771210256d87a1997bad87ebdd409a3437ecec02885300a1bb0a1159e289ae66af78b3453ae"}]' '{"mqZa1zbDg5RHVYbX6ea1QxPuuxAyuGVSoK":0.0999}'

0100000001705da5d1160b1799d0c7e938abc24011ad3ce61c9e3d89556de3cbc67a9165ff0000000000ffffffff01706f9800000000001976a9146e2fe8ef5dcb05fd258e2f51d3f15ba3781cd96b88ac00000000


decoderawtransaction 0100000001705da5d1160b1799d0c7e938abc24011ad3ce61c9e3d89556de3cbc67a9165ff0000000000ffffffff01706f9800000000001976a9146e2fe8ef5dcb05fd258e2f51d3f15ba3781cd96b88ac00000000
{
  "txid": "a8cfe9dd4fb8534caf4236b8fdf87d28aee5d74154eb88dab60c4de865bebfea",
  "hash": "a8cfe9dd4fb8534caf4236b8fdf87d28aee5d74154eb88dab60c4de865bebfea",
  "size": 85,
  "vsize": 85,
  "version": 1,
  "locktime": 0,
  "vin": [
    {
      "txid": "ff65917ac6cbe36d55893d9e1ce63cad1140c2ab38e9c7d099170b16d1a55d70",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.09990000,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 6e2fe8ef5dcb05fd258e2f51d3f15ba3781cd96b OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a9146e2fe8ef5dcb05fd258e2f51d3f15ba3781cd96b88ac",
        "reqSigs": 1,
        "type": "pubkeyhash",
        "addresses": [
          "mqZa1zbDg5RHVYbX6ea1QxPuuxAyuGVSoK"
        ]
      }
    }
  ]
}

Sign with the first privkey

Code:

signrawtransaction '0100000001705da5d1160b1799d0c7e938abc24011ad3ce61c9e3d89556de3cbc67a9165ff0000000000ffffffff01706f9800000000001976a9146e2fe8ef5dcb05fd258e2f51d3f15ba3781cd96b88ac00000000' '[{"txid":"ff65917ac6cbe36d55893d9e1ce63cad1140c2ab38e9c7d099170b16d1a55d70","vout":0,"scriptPubKey":"a914b579aabfd350ed75ee056ff4d6d56718c74f55df87","redeemScript":"5221024c5965fb18701569ec09e8d2979c45e3d6e0c60ddb3c9d6af0692124d3890e8a2103c9084cf82f74f764829a7d3c842462e03f4bbaf59981cb3b98cd3974438e5771210256d87a1997bad87ebdd409a3437ecec02885300a1bb0a1159e289ae66af78b3453ae"}]' '["cMpfNfuasJcrCv8RKMG35C8YtUVpaheFbNMwDuBXVvdcjaeYE7oh"]'


{
  "hex": "0100000001705da5d1160b1799d0c7e938abc24011ad3ce61c9e3d89556de3cbc67a9165ff00000000b5004830450221009d4f252bc47a8ab7078c3f93929bd091da175ed115bbe4953a3704d1e8027e12022005ab1434847d64bc6551ce6044b970becad36fc0cf6380847c33b241a30fc085014c695221024c5965fb18701569ec09e8d2979c45e3d6e0c60ddb3c9d6af0692124d3890e8a2103c9084cf82f74f764829a7d3c842462e03f4bbaf59981cb3b98cd3974438e5771210256d87a1997bad87ebdd409a3437ecec02885300a1bb0a1159e289ae66af78b3453aeffffffff01706f9800000000001976a9146e2fe8ef5dcb05fd258e2f51d3f15ba3781cd96b88ac00000000",
  "complete": false,
  "errors": [
    {
      "txid": "ff65917ac6cbe36d55893d9e1ce63cad1140c2ab38e9c7d099170b16d1a55d70",
      "vout": 0,
      "scriptSig": "004830450221009d4f252bc47a8ab7078c3f93929bd091da175ed115bbe4953a3704d1e8027e12022005ab1434847d64bc6551ce6044b970becad36fc0cf6380847c33b241a30fc085014c695221024c5965fb18701569ec09e8d2979c45e3d6e0c60ddb3c9d6af0692124d3890e8a2103c9084cf82f74f764829a7d3c842462e03f4bbaf59981cb3b98cd3974438e5771210256d87a1997bad87ebdd409a3437ecec02885300a1bb0a1159e289ae66af78b3453ae",
      "sequence": 4294967295,
      "error": "Operation not valid with the current stack size"
    }
  ]
}

As far as I can tell, everything is done in exactly the same was as in Gavin's gist from 2012, and I did this procedure in October 2015 and it worked.

Vidim sad u hrvatskoj da nema nikakve ponude za trgovanje bitcoin za gotovinu putem pošte.

Ja mislim da je to jako dobar način jer moze bit jako anoniman način i događa se totalno van bankovnog sustava. U suprotnost od susretanja ljudi u živo i trgovanje gotovinom, sa gotovinom kroz poštu može se trgovat bili gdje u državi ili gdje god se prihvačaju kune. I nemoraš se srest u isto vrime.

Ja sam više puti koristio ovaj način kupovanja i prodavanja bitcoina jer u državi da živim banke jako nevole bitcoin i zatvorili bi ti račun odma ako primjetu da trgujuš sa bitcoinom.

Staviš papirnate novce (nikako kovacine) u kuvertu sa malo presavijenog papira, da se nemože vidit novce ako kuvertu držiš nad svitlom. Neki ljudi pošalju sa tracked mail ali ja nikad nisam i nikad nije bilo problema. Ali ako razmisliš koliko pisma i kuverta putuju svaki dan onda nema razloga da neko pogleda baš na tvoju kuvertu.

Onaj tko prima gotovinu može snimat sebe na kameru dok otvara kuvertu, i naravno provjeri da su novce pravi.

Sve je to malo više rizično nego nesto poput bank transfer (i ako neki bank transfer se može preokrenut a gotovina nikad nemože). To plus činjenica da pošta treba malo više vrimena onda se može očekivat da gotovinu kroz poštu ima veče označavanje.

It could be interesting to have a list of Bitcoin days of observance. A bit like those "On This Day" segments where people reflect on history for 5 minutes.

Here's all the ones I could think of that could work.


3rd January 2009 - Genesis Block Day

11th January 2009 - Hal Finney downloads and installs bitcoin https://twitter.com/halfin/status/1110302988

12th January 2009 - First bitcoin transaction (to Hal Finney)

27th January 2011 - Silk road opens. Ross Ulbricht's first ever shill thread

25th Feburary 2018 - Lightning Pizza Day

1st April 2012 - p2sh soft fork activates

5th April 1975 - Satoshi's Birthday

7th May 2015 - First JoinMarket coinjoin

22nd May 2010 - Pizza Day

?? 2010 - First GPU mining

?? - First p2pool block

7th June 2012 - localbitcoins opens

4th July 2015 - BIP66 soft fork activates, results in accidental chain split.

9th July 2016 - Second bitcoin halvening

28th July 2015 - Catastrophic near miss revealed. Disclosure of consensus bug that was fixed by BIP66 soft fork.

1st August 2017 - "UASF day" BIP148 trigger date for activating UASF for segwit.

?? 2012 - Bitcoin appears on a TV show for the first time

31st October 2008 - Bitcoin whitepaper published

12th November 2017 - First bitcoin block with miner fee income surpassing inflation subsidy. See also.

28th November 2012 - First bitcoin halvening

12 December 2010 - Satoshi's last public contact

[Create Escrow Account]

Quite frankly consumer protection in bitcoin is shot. Enthusiasts actually go around talking up the "no chargebacks" property as though it's a good thing. Great for the merchant if they actually get any bitcoin sales, but customers will choose something else when they can.

Lack of consumer protection further hurts merchants; in two ways: One, it lowers their margins because of lemon-market effects; honest consumers will pay less because they're not sure if they're going to get ripped off; Two, it amplifies the power of reputational ransom.. since people squaking loudly is the only real sign of fraud you'll get, you have to pay attention.. but anyone can squak loudly, and demand money (or free products) from you to shut up.

Right now if a deal goes bad, consumers have to resort to public shaming on internet forums, or legal processes. Many in the bitcoin community are libertarians who want to stop relying on the police and courts. Moving away from using the legal infrastructure of the state doesn't have to be for political reasons, it's also much cheaper to avoid lawyers and legal processes.

Bitcoin does have a way to implement better consumer protection. It's called multisig. The idea being that a special kind of bitcoin address is created from three keys. If any two of the private key-holder, the money can be moved. So to create a robust consumer protection model you give the buyer, seller and an arbitrator each one key. The buyer pays into the multisig address, if the buyer receives the service and is happy, her and the seller can both use their private keys to send the money to the seller. If they can't agree then the arbitrator can use her private key to tie-break and distribute the money as they see fit. Also the arbitrator can't steal the money on his own as they only have one key, so they don't need to spend time and money on security.

Multisig is used today but only for security. People keep one set of keys in a backup, the other two keys are held on a hot wallet and third-party web wallet or security service. Malware and the web wallet cant steal coins from the hot wallet, and if the web wallet disappears the person can just open up their backup to move the money. Plenty of wallets can do this including Electrum 2.0. But this model has nothing to do with consumer protection.

There have been some attempts at multisig consumer protection but from what I can see most did not get adopted. Often they rely on users manipulating raw ECDSA keys. The GUIs that do this should have big buttons with names normal people understand, like Create Escrow Account, Fund Escrow, Receive Escrow Money, Sign Off Payment, Get A Refund. The accounts feature in Electrum is perfect for this, when people sign up to their new payment processor or marketplace they get a xpub BIP32 key, the merchant also gives them a xpub key. Clicking the Create Escrow Account button simply uses these to obtain public keys and the redeem script.

I'm nowhere near the first person to come up with this. Here is Mike Hearn talking about the concept in 2012(!) https://www.youtube.com/watch?v=mD4L7xDNCmA

Theres a whole website based on the idea, https://www.bitrated.com/ which is actually really good IMO. I think a reason it isn't seeing more adoption is that it doesn't slot into existing infrastructure. I think it only can work for OTC trades between people, so two people decide on a trade and agree to both sign up to bitrated.com, it has no scope with a payment process like Bitpay or an existing marketplace like https://cryptothrift.com/. Plus because it runs in a browser, bitrated might not be agreeable for people who prefer to keep their private keys on their hard disk. (Although bitrated can be used entirely with local private keys, it even gives you the relevant commands on the site, but I dunno if really any of its users bother)

Any consumer protection has to be part of bitcoin wallets. That software already handles our regular private keys, its only a small step to also handling multisig keys. Hopefully it would be compatible with bitrated.com (which already has an API ready to use). Another benefit of this approach is it can give people a *choice* of mutually agreeable arbritartors, not just a monopoly tied to the credit card or payment processor.

Now there also exists the BIP70 payment protocol. I don't see it having any ability to use multisig consumer protection but I'm sure thats just an oversight.

Bitcoin CAN have good consumer protection. Instead of talking up bitcoin's "no chargebacks" we should be talking about how bitcoin has much better security than credit cards so that the only chargebacks that happen will be because of merchant fraud, not because some hacker in vietnam stole the customer's credit card.

Quite frankly the credit card security system is awful, bitcoin doesn't have to work hard to beat it. https://np.reddit.com/r/personalfinance/comments/3lf9hr/bank_is_refusing_to_refund_fraudulent_visa/cv5umvx

I don't think writing an Electrum plugin that does this is too hard. Sadly it's been more than two years since p2sh multisig was released before the beginnings of adoption were seen.

I'm working on a bitcoin application which talks to Bitcoin Core. It uses RPC commands to request information about which bitcoin addresses from a HD seed have UTXOs are on them. I'm thinking about making this wallet synchronization work with a pruned node.

Given a HD seed, one way to synchronize the wallet would be to import a lot of watch-only addresses into Bitcoin Core and then restart it with -rescan. This would require the pruned node to redownload the entire blockchain again [possibly multiple times(!) if I didn't import enough watch-only addresses] and then use listunspent RPC.

So I'm thinking to query the UTXO set. I wouldn't get transaction history information but that's okay for these uses. I'd need a way to go from address to UXTOs.
1) One way would be to simply use the RPC calls getbestblockhash, getblock and gettxout to step through all outputs in the entire blockchain, checking if our addresses match any of them.
2) A second way would be to write some new RPC calls that allow dumping of the UXTO set database with pagination in the manner of listtransaction, then check if our addresses match.
3) A third way would be to open the UTXO BerkeleyDB database file (can this be done while Bitcoin Core is running?) and then check if our addresses match.

Another issue is that currently pruning disables the Bitcoin Core wallet so -walletnotify probably wont work. This doesn't have to be a problem as I could just poll getrawmempool and use -blocknotify instead.

Thoughts? What's the best way to proceed?

I couldn't find a way to query what the electrum server thinks about the blockchain through the GUI.

During the chain fork after bip66, this information was important. I was able to hack up this plugin.

Code:

'''
GET BLOCK HEADER

An electrum plugin for querying the electrum server about it's block headers

HOW TO
1. place plugin in electrum/plugins
2. start electrum on the command line
3. in the gui go Tools -> Plugins
4. enable the GetServerBlockHeaderData and press button
5. input a block height and watch the shell terminal (not electrum's console tab) for the result

example output
[{u'nonce': 697162691, u'prev_block_hash': u'00000000000000000e20bcf213a0bbd6be88d5fede6b060c737f7f8b7f1df504', u'timestamp': 1436136522, u'merkle_root': u'0fa543c7aa9f2b3b6d9755024e586c443cc1cf1cb513f9177ada9c718a1be3e6', u'block_height': 364001, u'version': 3, u'bits': 404111758}]

'''

from PyQt4 import QtCore
from PyQt4.QtGui import *

from electrum.plugins import BasePlugin, hook
from electrum.i18n import _
from electrum.wallet import Abstract_Wallet

from electrum_gui.qt.util import EnterButton

import time, traceback, threading, socket, json

class Plugin(BasePlugin):

    gui = None
    daemon_comm = None

    def fullname(self):
        return 'GetServerBlockHeaderData'

    def description(self):
        return _("Get block header data from the server, useful for obtaining data when the chain forks")

    def requires_settings(self):
        return True

    def settings_widget(self, window):
        return EnterButton(_('GetBlockHeader'), self.settings_dialog)

    def settings_dialog(self):
        print 'pressed settings'
        #self.daemon_comm.send_json({'command': 'echo', 'stuff': 'here'})
        d = QDialog()
        d.setWindowTitle("Settings")

        blockheight = QInputDialog.getText(None, 'Block Height', 'Input block height you wish to get the header of')
        if not blockheight[1]:
            return
        blockheight = int(blockheight[0])
        raw = self.gui.network.synchronous_get([ ('blockchain.block.get_header', [blockheight]) ])
        print raw
        
        #blockhash = 10
        #raw = self.gui.network.synchronous_get([ ('blockchain.block.get_chunk', [blockhash]) ])
        #jsonraw = str(json.loads(raw))
        #print jsonraw.keys()

        #if d.exec_():
        #    return True
        #else:
        #    return False

    @hook
    def set_enabled(self, enabled):
        BasePlugin.set_enabled(self, enabled)
        print 'set enabled = ' + str(enabled)
        '''
        if enabled:
            self.start_daemon_connection()
        else:
            self.stop_daemon_connection()
        if not enabled and isinstance(self.gui.main_window.wallet, JoinMarketWallet):
            print 'returning wallet to normal'
            self.gui.main_window.wallet = self.gui.main_window.wallet.underlying_wallet
         '''

    @hook
    def load_wallet(self, wallet):
        print 'load wallet'
        #if self.gui and self.gui.main_window.wallet and not isinstance(self.gui.main_window.wallet, JoinMarketWallet):
        #    print 'creating JM wallet'
        #    self.gui.main_window.wallet = JoinMarketWallet(self.gui.main_window.wallet, self.gui.main_window.wallet)

    #in ./gui/qt/main_window.py there is broadcast_transaction()

    @hook
    def init_qt(self, gui):
        print 'init gui'
        self.gui = gui
        #if self.gui.main_window.wallet and not isinstance(self.gui.main_window.wallet, JoinMarketWallet):
        #    print 'creating JM wallet'
        #    self.gui.main_window.wallet = JoinMarketWallet(self.gui.main_window.wallet, self.gui.main_window.wallet)

    @hook
    def make_unsigned_transaction(self, tx):
        '''called when the user presses send or edits the send dialog'''
        print 'make unsigned tx'
        #print str(tx)

        #see https://electrum.orain.org/wiki/Stratum_protocol_specification

        '''
        #get the address and value like this
        tx_hash = '6719c7e225972bb2f935d0f923748a69868fab094fc302bbd64b7d10d89a16b0'
        print 'getting txhash = ' + tx_hash
        raw = self.gui.network.synchronous_get([ ('blockchain.transaction.get',[tx_hash]) ])
        print str(raw)

        #check its unspent and confirmed
        #addr = '1JfbZRwdDHKZmuiZgYArJZhcuuzuw2HuMu'
        addr = '1EtnpHTBthhXbjLn2TRfJMXu982fjPKcwM'
        print 'getting addr listunspent = ' + addr
        data = self.gui.network.synchronous_get([ ('blockchain.address.listunspent', [addr]) ])
        print str(data)
        #taker needs the following information
        # info on tx being spent, unconfirmed or genuinly just in the utxo set
        #  the value and scriptpubkey (/ address) of such a utxo
        # pushtx, should be easy
        '''
        if len(tx.outputs) > 2:
            self.print_error('cant make coinjoins with more than one output address yet')
            return

        cj_addr = None
        change_addr = None
        cj_amount = 0
        for otype, addr, value in tx.outputs:
            if otype != 'address':
                self.print_error('whoops, cant send to places other than addresses')
                return
            if self.wallet.is_change(addr):
                change_addr = addr
            else:
                cj_addr = addr
                cj_amount = value
        print 'cj=' + cj_addr + ' change=' + change_addr

    @hook
    def sign_transaction(self, tx, password):
        '''called when the user types in password after send is clicked'''
        print 'sign tx ' + str(type(tx))
        print str(tx)
        #tx is of instance Transaction
        # need to somehow find out which is the change address
        address = ''
        if len(address) > 0:
            priv = self.wallet.get_private_key(address, password)
            print 'priv = ' + str(priv)
        #the tx needs enough inputs to pay the coinjoin fee, check it has enough
        # if not, repeat the process in make_unsigned_transaction() to get more
        # trustedcoin.py adds a fee to a tx, look how they do it
        #  they override Wallet class and override get_tx_fee()
        # for us it might be worth overriding the function pointer
        #if/when the user clicks boardcast, then send it to electrum or somehow halt the broadcast
        # and send it to a maker
        # need to add a hook to electrum that has the ability to halt a broadcast
        # ThomasV says he would accept such a hook
        # probably best is that it is able to raise an exception

    @hook
    def transaction_dialog(self, d):
        '''called when the transaction is displayed to the user right before broadcast'''
        print 'transaction dialog ' + str(type(d))

    @hook
    def create_send_tab(self, grid):
        print 'create send tab, put coinjoin fee amount here that updates as the user types in amounts'
        #self.start_daemon_connection()
        #maybe get access to the address/amount field, to be able to tell between output and change
        #better way would be to hook mktx() so you can see outputs

Luckily this page helped https://electrum.orain.org/wiki/Stratum_protocol_specification

blockchain.block.get_chunk looked like a useful method too but I couldn't figure it out straight away

[JoinMarket]

JoinMarket is built for those who:
1. Want to use coinjoin to improve the privacy of their bitcoin transactions or reduce the miner fees they pay.
2. Want to earn an income from their investment bitcoins

Coinjoin is a kind of bitcoin transaction which combines multiple people's transactions together in an effort to improve privacy, especially as a way to regain privacy after it's been degraded by bad practices. It can also reduce the size of transactions and so require less miner fees. The concept has enormous potential, but it has not seen widespread usage so far despite the multiple projects that implement it. I believe this is because the incentive structure was not right.

A coinjoin transaction requires other people to take part. The right resources (coins) have to be in the right place, at the right time, in the right quantity. This isn't a software or tech problem, its an economic problem. I propose a new kind of market is created that would allocate these resources in the best way.

In practice this would work by allowing coinjoin transactions to be paid-for. On one side there will be time-rich coinjoiners who will wait around and be willing to coinjoin at any time. On the other side will be time-stressed coinjoiners who can coinjoin instantly for a price.

This will naturally attract investors, holders of bitcoin who don't want to transact but just want to earn the coinjoin fee. They would be taking on only a very small amount of risk; Their private keys would never leave their computer, the software would only sign transactions that are valid and pay the correct amount of coins. So it's safe to say that investors would pour in, resulting in the coinjoin fees being very low.

Elevator Pitch for Coinjoin Users / People who desire privacy.

With Joinmarket nobody can steal your coins, you would mix them on your own computer. The software will never sign a transaction unless your coins are going to exactly the right address in the right amount.

Other mixing services mix your coins with other users, resulting in you getting back coins possibly related to drugs, stolen money or other illegal or immoral practices. This is highly undesirable if you just don't want your employer knowing which non-profits you support from your salary. Because of the incentives of JoinMarket, you have access to a huge amount of clean, untainted bitcoins to mix with at a very low price. Many of the bitcoins you're mixing with will be bought from regulated exchanges, owned by legitimate holders of bitcoin.

Not to mention you'd likely be paying lower fees for mixing than other services, because of the competition between willing-to-wait coinjoiners. Indeed if you don't mind waiting, you can wait around with a low offer fee and maybe another impatient coinjoiner will join with you.

Elevator Pitch for Investors

Firstly I'd like to clarify what I mean by investing. I don't want you to give your bitcoins to me. I dont want you to give your bitcoins to anybody. The private keys would be safely held on your own computer, known only by you and your wallet.

Features:
1. Earn an income from your investment bitcoins.
2. Very low risk. Your coins have to be on an online computer, but the software would only sign transactions that are valid and pay you the correct amount.
3. No commitment, withdraw your bitcoins at any time.
4. Improves the privacy of the bitcoin transactions, which makes bitcoin as a currency more useful and thus increases its value.
5. Improves the fungibility of bitcoin, since the distinction between 'clean' and 'dirty' bitcoins will be meaningless. Eliminates this particular systemic risk to bitcoin.

Downside:
1. Your return is likely to be quite low. Low risk = low reward.
2. You don't get paid unless people who desire privacy actually use this. If you're an investor you have an incentive to tell people about JoinMarket and contribute to it.

The Software

I've been writing an implementation of the idea. Right now the coinjoiner bots meet in an IRC channel. The bots announce their orders in an open-outcry trading pit style. Transaction data is sent between users as IRC private messages. Authenticated encryption is used to stop the IRC server eavesdropping. I have plans one day to move away from IRC entirely and have the users meet in some kind of peer to peer network.

How to try
Download the repository from https://github.com/JoinMarket-Org/joinmarket-clientserver and read the README file
There is an internal HD wallet. It is generated from a wallet encrypted on your hard drive.

There's also plenty of information on the github docs: https://github.com/JoinMarket-Org/joinmarket-clientserver/tree/master/docs

Code:

$python wallet-tool.py generate
   Generates a new wallet. Asks for your encryption passphrase and gives you a 12-word recover seed to write down on paper, a la Electrum or Mycelium.

$ python wallet-tool.py [wallet file name]
  This will print out addresses from the HD wallet, send some coins to the first receive address

$ python yield-generator.py [wallet file]
  Becomes an investor bot, being online indefinitely and doing coinjoin for the purpose of profit

$ python sendpayment.py -N 4 [wallet file] [amount-in-satoshi] [destination address]
  Chooses the cheapest offer to do a 5-party coinjoin to send money to a destination address

If you're a frugal user and don't feel like paying to coinjoin if you dont have to, use this command
$ python patientsendpayments.py -N 1 -w 2 [wallet file] [amount in satoshi] [destination address]
  Announces orders and waits to coinjoin for a maximum of 2 hours. Once that time it up cancels the
  orders and pays to do a 2-party coinjoin.

$ python ob-watcher.py
  Starts a local http server which you can connect to and will display the orderbook as well as some graphs

$ python wallet-tool.py recover
  Will prompt for the 12 word recover seed and create a new encrypted wallet file

$ python wallet-tool.py -p [wallet file]
  Will print out addresses along with private keys which can be used to sweep the coins away

Example coinjoins transactions made with JoinMarket:
14 parties, amount 0.01btc https://blockchain.info/tx/55eac9d4a4159d4ba355122c6c18f85293c19ae358306a3773ec3a5d053e2f1b
11 parties, amount 0.57960945btc https://blockchain.info/tx/402d3e1df685d1fdf82f36b220079c1bf44db227df2d676625ebcbee3f6cb22a
8 parties, amount 2.6btc https://blockchain.info/tx/722bb2662cb2ef9b4a2693e52ba82c44cea1042349f1aa6e71e28a3947aa4144
7 parties, amount 0.02btc https://blockchain.info/tx/e38e45d004ef913ee4e952d1f185dbea91cc8cc56e22aa7f767a3edec952a4a9
3 parties, amount 25btc https://blockchain.info/tx/da1a2259be752dd6b5162221989181b7334bd0acbbc1bca31596e5bc32375770
4 parties, amount 40btc https://blockchain.info/tx/3b97544488cac0271a80b20822597342648d19ed02ac25041bd8d35e624d8e6b
3 parties, amount 12btc https://blockchain.info/tx/d91278e125673f5b9201456b0c36efac3b2b6700fdd04fc2227352a63f941170
4 parties, amount 0.84btc (~$200 at contemporary exchange rate) https://blockchain.info/tx/7d588d52d1cece7a18d663c977d6143016b5b326404bbf286bc024d5d54fcecb
4 parties, amount 0.1btc https://blockchain.info/tx/b85a3b563474ca98ba1809460e61a50053899c21f9869afb6a3a6d4b4cb00b7c
3 parties, amount 0.19btc https://blockchain.info/tx/2e2cd9204f97a2260ba5b1fd446f394eeff79daa973e37dd29794a9ee667bf64

Further Development
Github: https://github.com/chris-belcher/joinmarket
IRC: irc.freenode.net #joinmarket
Twitter: https://www.twitter.com/joinmarket
Reddit: http://www.reddit.com/r/joinmarket

If you can program, contribute code. The project needs development works more than anything else right now. A list of suggested projects are here https://github.com/chris-belcher/joinmarket/wiki/What-can-I-do-for-JoinMarket%3F

If you require privacy, JoinMarket can be used right now on the mainnet, albeit with a command-line interface. Many people are already using it.

Bitcoin Core wallet could be integrated by using the -walletbroadcast=0. Integration on the command line using json-rpc calls is already done, use the --rpcwallet flag on sendpayment.py

I have plans to create an Electrum plugin which does this, it would be a simple checkbox and thereafter all transactions would be coinjoined.

If you run your own yield generator bot, you can earn an income from your bitcoin savings and increase the privacy and anonymous set of others.

People trust systems more if they think they know how it works. If you're good with graphics, you can help the project by creating an infographic for how it all works. For example how coinjoin transactions in the joinmarket software are communicated, created, checked and signed.

Snazzy websites are better for getting the word out than big walls of text on a forum.

Further Reading
gmaxwell's original coinjoin post. https://bitcointalk.org/index.php?topic=279249.0
my original Joinmarket suggestion / writeup. https://bitcointalk.org/index.php?topic=279249.msg9384411#msg9384411
Mike Hearn's blog post about privacy and coinjoin https://medium.com/@octskyward/merge-avoidance-7f95a386692f

I've coded this application which is essentially a GUI around raw transactions plus an ascii-armor format for transactions similar to PGP ascii armor. The result is an implementation of CoinJoin that could conceivably be used today by non-programmers, albeit still vunerable to denial-of-service and other attacks.

Here is an group of screenshots demonstrating the application being used.
https://www.dropbox.com/s/spzbfx1ij1rrc02/cj-demo.png (click Download to get the fullsize image. size: 2.3MB)

Other implementations I've seen require that peers have to want to do a CoinJoin at essentially the same time. Given that bitcoin transaction volume peaks at about 1 tx / second, it is quite unlikely that there will be someone else wanting to transact exactly the same amount as you that is needed for CoinJoin to improve privacy.

A solution might be to allow people to share their transaction parts asynchronously. This application makes no assumptions about how the CoinJoin peers communicate, only that they can send each other ascii-armored transaction parts in a private way. They could post them on Tor hidden service forums, Bitmessage chans, I2P eepsites, Freenet pages or shared some other way.

Here is the code. One file of python along with vbuterin's pybitcointools and a socks5 library to allow anonymous lookups through tor or a ssh tunnel. Execute run.bat on Windows or ./run.sh on the command line for Linux.

https://github.com/chris-belcher/coinjumble