Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: Vitalik Buterin on August 10, 2013, 05:32:44 PM



Title: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: Vitalik Buterin on August 10, 2013, 05:32:44 PM
I've been writing my own Python library for handling Bitcoin elliptic curve arithmetic for a while now, and just yesterday I made a serious push to have it include many more features. It now supports compressed keys, RFC6979 deterministic ECDSA message signatures (compatible with Electrum; see tests.py), transaction construction and signing and even Electrum-style wallets.

Here's the repo: https://github.com/vbuterin/pybitcointools

I deliberately designed the library to be simple and no-nonsense. No hundred-line monolithic functions, no classes, no wrapper objects, just hex/binary/numbers in and hex/binary/numbers out. You don't need to spend a lot of time "learning" it; no matter what format your data uses, there's some way of handling it in there somewhere. Note that this is NOT a fully functional wallet; it does not support transaction generation or verification, networking or anything to do with blocks. You still need *coind or *coin-qt along with the importprivkey command to do that. It's not a full node, but does have the basic features for constructing Bitcoin transactions and includes two functions for fetching history and pushing transactions with blockchain.info.

Here are some basic usage scenarios:

> from pybitcointools import *
> priv = sha256('some big long brainwallet password')
> priv
'57c617d9b4e1f7af6ec97ca2ff57e94a28279a7eedd4d12a99fa11170e94f5a4'
> pub = privtopub(priv)
> pub
'0420f34c2786b4bae593e22596631b025f3ff46e200fc1d4b52ef49bbdc2ed00b26c584b7e32523 fb01be2294a1f8a5eb0cf71a203cc034ced46ea92a8df16c6e9'
> addr = pubtoaddr(pub)
> addr
'1CQLd3bhw4EzaURHbKCwM5YZbUQfA4ReY6'


Transaction stuff:


> h = history(addr)
> h
[{'output': u'97f7c7d8ac85e40c255f8a763b6cd9a68f3a94d2e93e8bfa08f977b92e55465e:0', 'value': 50000, 'address': u'1CQLd3bhw4EzaURHbKCwM5YZbUQfA4ReY6'}, {'output': u'4cc806bb04f730c445c60b3e0f4f44b54769a1c196ca37d8d4002135e4abd171:1', 'value': 50000, 'address': u'1CQLd3bhw4EzaURHbKCwM5YZbUQfA4ReY6'}]
> outs = [{'value': 90000, 'address': '16iw1MQ1sy1DtRPYw3ao1bCamoyBJtRB4t'}]
> tx = mktx(h,outs)
> tx
'01000000025e46552eb977f908fa8b3ee9d2943a8fa6d96c3b768a5f250ce485acd8c7f79700000 00000ffffffff71d1abe4352100d4d837ca96c1a16947b5444f0f3e0bc645c430f704bb06c84c01 00000000ffffffff01905f0100000000001976a9143ec6c3ed8dfc3ceabcc1cbdb0c5aef4e2d028 73c88ac00000000'
> tx2 = sign(tx,0,priv)
> tx2
'01000000025e46552eb977f908fa8b3ee9d2943a8fa6d96c3b768a5f250ce485acd8c7f79700000 0008b483045022100fc9ec3f6c66f630604e76309092ae00b48d39a83f8683bbf9d6310084e70ea bd022058333d7a1d2158529ce39f9b48dea23dedefbe85028cdceab34e1ee9b1518c3201410420f 34c2786b4bae593e22596631b025f3ff46e200fc1d4b52ef49bbdc2ed00b26c584b7e32523fb01b e2294a1f8a5eb0cf71a203cc034ced46ea92a8df16c6e9ffffffff71d1abe4352100d4d837ca96c 1a16947b5444f0f3e0bc645c430f704bb06c84c0100000000ffffffff01905f0100000000001976 a9143ec6c3ed8dfc3ceabcc1cbdb0c5aef4e2d02873c88ac00000000'
> tx3 = sign(tx2,1,priv)
> tx3
'01000000025e46552eb977f908fa8b3ee9d2943a8fa6d96c3b768a5f250ce485acd8c7f79700000 0008b483045022100fc9ec3f6c66f630604e76309092ae00b48d39a83f8683bbf9d6310084e70ea bd022058333d7a1d2158529ce39f9b48dea23dedefbe85028cdceab34e1ee9b1518c3201410420f 34c2786b4bae593e22596631b025f3ff46e200fc1d4b52ef49bbdc2ed00b26c584b7e32523fb01b e2294a1f8a5eb0cf71a203cc034ced46ea92a8df16c6e9ffffffff71d1abe4352100d4d837ca96c 1a16947b5444f0f3e0bc645c430f704bb06c84c010000008c493046022100da7fa563ce34af5a4c 8167a2978cb5517ded494e52a29ea4103ff2c67bce77c102210094a18bda1109591a82d5cf5e444 6b12b3c399401c0f668755ac7f614eb3baa7701410420f34c2786b4bae593e22596631b025f3ff4 6e200fc1d4b52ef49bbdc2ed00b26c584b7e32523fb01be2294a1f8a5eb0cf71a203cc034ced46e a92a8df16c6e9ffffffff01905f0100000000001976a9143ec6c3ed8dfc3ceabcc1cbdb0c5aef4e 2d02873c88ac00000000'
> pushtx(tx3)
'Transaction Submitted'


ECDSA (Electrum-compatible):

> sig = ecdsa_sign("Hallo world",sha256("123"))
> sig
'G9vmnqdlS9hmP7qT6ReRJEqldYn7f6xc47ado3zg7g4LKpLgVmpmSttDNAAPSizww7zzKDamsyD7bhh fAqaAJ0s='
> ecdsa_verify("Hallo world",sig,privtopub(sha256("123")))
True
> ecdsa_verify("Hallo world!",sig,privtopub(sha256("123")))
False
> ecdsa_recover("Hallo world",sig)
'04be686ed7f0539affbaf634f3bcc2b235e8e220e7be57e9397ab1c14c39137eb43705125aac75a 865268ef33c53897c141bd092cf4d1a306b2a57e37e1386826d'
> privtopub(sha256("123"))
'04be686ed7f0539affbaf634f3bcc2b235e8e220e7be57e9397ab1c14c39137eb43705125aac75a 865268ef33c53897c141bd092cf4d1a306b2a57e37e1386826d'


Electrum:

> seed = sha256("cow")[:32]
> mpk = electrum_mpk(seed)
> for i in range(5): print electrum_privkey(seed,i,0), pubkey_to_address(privtopub(electrum_privkey(seed,i,0)))
daa19f768ea1a7bc8e33d83f9401ef237f5164ce39bf8000ec9630d088cec028 1GG9cGb7KW6qNBCxcU7Y3NH3mPRz9YSxgG
c45ad49da1ccb0ba30258902ed3de66af571de7421f9c73ffc0a8c2bde9ca596 19zxmnE4hmTwPFxhoDAxjtfnKQcRES4Sm2
c31100a56692314248a73f29b767ca252c3847eae15ce5d06e7e7f67a63ee5df 17mJpsyEVykxuf5H4YWsyhatXFyuJmkuoh
302362a97a2db566c5c65b875388db7cce246355faa72f15b486111c2f380bf3 16PGWQwcXM37iY7dPHqqVFit4GXsifRuAM
869424a50260cca7a7fa475c85e48ddf0e47abb0474df59ee45e72e6b95c8333 15QgAJmroGtExvVqVd8MGs1ZZvwYsk39i7
> for i in range(5): print pubkey_to_address(electrum_pubkey(mpk,i,0))
1GG9cGb7KW6qNBCxcU7Y3NH3mPRz9YSxgG
19zxmnE4hmTwPFxhoDAxjtfnKQcRES4Sm2
17mJpsyEVykxuf5H4YWsyhatXFyuJmkuoh
16PGWQwcXM37iY7dPHqqVFit4GXsifRuAM
15QgAJmroGtExvVqVd8MGs1ZZvwYsk39i7


Much faster Electrum by precomputing the secret exponent:

> seed = sha256("cow")[:32]
> secexp = electrum_stretch(seed)
> for i in range(5): print electrum_privkey(secexp,i,0), pubkey_to_address(privtopub(electrum_privkey(secexp,i,0)))
daa19f768ea1a7bc8e33d83f9401ef237f5164ce39bf8000ec9630d088cec028 1GG9cGb7KW6qNBCxcU7Y3NH3mPRz9YSxgG
c45ad49da1ccb0ba30258902ed3de66af571de7421f9c73ffc0a8c2bde9ca596 19zxmnE4hmTwPFxhoDAxjtfnKQcRES4Sm2
c31100a56692314248a73f29b767ca252c3847eae15ce5d06e7e7f67a63ee5df 17mJpsyEVykxuf5H4YWsyhatXFyuJmkuoh
302362a97a2db566c5c65b875388db7cce246355faa72f15b486111c2f380bf3 16PGWQwcXM37iY7dPHqqVFit4GXsifRuAM
869424a50260cca7a7fa475c85e48ddf0e47abb0474df59ee45e72e6b95c8333 15QgAJmroGtExvVqVd8MGs1ZZvwYsk39i7


BIP32:

> bip32_master_key('123456789')
'xprv9s21ZrQH143K2zm1WhDFQqHVdgzoyBWfghiTfgdVJDSjZtTupL9eeg7391hLNjHQNAbd1qLobce ZcaYvrKZ81E3SwtiZ2UbPsP2zm9HeDJo'
> bip32_ckd(bip32_master_key('123456789'),0)
'xprv9ujNqB68cg8G1owbyxEwLHMBYbB1RCyrcra4i8BciNhBrsWeQaEj8FBPNnRmGedK1Q9A5DD3vLe c8me5pBPSwdhckMPQPTWBMtiesu688LN'
> bip32_extract_key(bip32_ckd(bip32_privtopub(bip32_master_key('123456789')),0))
'02e7f3eb95c70480f3f850198017644a84654325b2a4b038268546ea60c1814f6f'
> bip32_extract_key(bip32_privtopub(bip32_ckd(bip32_master_key('123456789'),0)))
'02e7f3eb95c70480f3f850198017644a84654325b2a4b038268546ea60c1814f6f'
> privtopub(bip32_extract_key(bip32_ckd(bip32_master_key('123456789'),0)))
'02e7f3eb95c70480f3f850198017644a84654325b2a4b038268546ea60c1814f6f'


(Those are compressed public keys, which BIP0032 makes standard)

This is still an early work in progress; ideally, with help from others, I'd like to this extended to handle more networking options than just blockchain.info, and serve as a universal altcoin light client. That's a far-off goal though; better have the more basic features ironed out first. Feel free to provide any suggestions!


Title: Re: Python Bitcoin ECC library, supports ECDSA, Electrum, altcoins (no txes/blocks)
Post by: Remember remember the 5th of November on August 10, 2013, 05:57:01 PM
Nice, congratulations. Though I would've suggested a different language cause simply Python ain't all that fast when it comes to these expensive operations there.


Title: Re: Python Bitcoin ECC library, supports ECDSA, Electrum, altcoins (no txes/blocks)
Post by: Vitalik Buterin on August 10, 2013, 06:51:44 PM
Nice, congratulations. Though I would've suggested a different language cause simply Python ain't all that fast when it comes to these expensive operations there.

If you want fast I strongly recommend you look at the C++ -based libbitcoin and sx: http://bitcoinmagazine.com/what-libbitcoin-and-sx-are-and-why-they-matter/ sx doesn't support elliptic curve arithmetic or ECDSA message signing, but it's great for the actual Bitcoin stuff!

And I realize there are plenty of inefficiencies in my code as used above with converting between numbers and hex and back. I'm deliberately targeting developer ease of use. No dependency-chasing, no ./configure, make, sudo make install, just a Python file that you can even copy+paste single functions from.


Title: Re: Python Bitcoin library, supports signing, Electrum, transactions (no blocks)
Post by: Vitalik Buterin on October 09, 2013, 06:39:20 PM
See my latest commits:

http://github.com/vbuterin/pybitcointools

New features:

* history: returns the history of an address in the form of outputs, including whether or not each output has been spent (uses blockchain.info)
* pushtx: pushes a transaction with blockchain.info/pushtx
* mktx, sign (build transactions)
* deterministic (RFC6979) ECDSA signing, so those evil random number generators (http://bitcoinmagazine.com/6251/critical-vulnerability-found-in-android-wallets/) will never hurt you again.
* pybtctool: a command-line tool for calling any of the functions in the library
* A setup.py script to make the library installable
* An actual README.md file (more coming soon)

The basic transaction tools have been confirmed to work; feel free to start building apps on top of it!


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: yakov on October 22, 2013, 10:39:47 PM
I quite like your library.
It took me a while to realise that it actually does handle wallet import format. It works if you put WIF into privtopub() although it took me some searching and thinking to realise you can use bin_to_b58check() to make WIF from a binary private key. Maybe a priv_to_wif() function should be made?

So to generate keys for a paper wallet. (Maybe on a livecd with no browser where you cant load bitaddress.org)


>>> priv = random_key()
>>> bin_to_b58check(priv.decode("hex"), 0x80)
'5JjCeDuxt9jWdb7nw43w7zQSB53MYpVi1aXe7XoxrhebWB3ft5k'
>>> privtoaddr(priv)
'1Jx1tHXNLRHyDQUxhNAaWdMRRk5vUmSEeC'
>>>



I'm thinking of maybe implementing BIP38, mostly for making encrypted paper wallets.
What kind of license is this under? Since some pure python AES256 implementations I'm looking at are under Apache license. I never really paid much attention to licenses but I should ask how it goes here before I just copypaste parts of it into this library.
Also there seems to be no pure python impl of scrypt. I know this goes against your intent (which I agree with) of having no dependency chasing. I imagine I'll try to code BIP38 and if it goes well try to code a pure python scrypt.


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: gmaxwell on October 23, 2013, 06:26:55 AM
I'm thinking of maybe implementing BIP38, mostly for making encrypted paper wallets.
There is no BIP38.


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: waxwing on October 23, 2013, 04:18:06 PM
This seems to be very inline with what I'm trying to do, so I greatly appreciate it.
Would it be possible to get a quick walkthrough of how to implement multisig transactions, along the lines of Gavin Andreesen's walkthrough for 2of3: https://gist.github.com/gavinandresen/3966071 ?

Edit: maybe it would be better for me to ask for help with a specific problem, so here goes:

I attempted to construct a spend FROM  a multisig address:

the 2 of 3 multisig was created with:
Code:
mscript = mk_multisig_script(pubs,2,3) 
msigaddr = scriptaddr(mscript)

and pubs is a list of 3 public addresses, with corresponding list privs and list addresses.

I tried to spend thusly:

Code:
outs = [{'value': 20000, 'address': '38izn5J4EECw5NNFBE27fwLyxkaBAsavSF'},{'value':60000,'address':addresses[2]}]
tx3 = mktx(history(msigaddr),outs)
sig1 = multisign(tx3,0,mscript,privs[0])
print "Verifying sig1:",verify_tx_input(tx3,0,mscript,sig1,pubs[0])
sig2 = multisign(tx3,0,mscript,privs[2])
print "Verifying sig2:",verify_tx_input(tx3,0,mscript,sig2,pubs[2])
tx4 = apply_multisignatures(tx3,0,mscript,[sig1,sig2])
print deserialize(tx4)
pushtx(tx4)

#for ref, history(msigaddr) gives: [{'output': u'aba22dd8cab30ffcfb0c57eb20d3dc9412de4e28fda6a10375b5292a03892467:0', 'value': 100000, 'address': u'38izn5J4EECw5NNFBE27fwLyxkaBAsavSF'}]

I get an "Invalid Signature" exception. What did I do wrong?


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: waxwing on October 24, 2013, 04:55:43 PM
A further question; possibly related, possibly not.

Creating a multisig address with bitcoind:

Code:
>_ ~/.bitcoin# bitcoind createmultisig 2 '["0453170015ba9dd27388e51e824a8d5516b7627c025e3727232d9803ffd9c2b99d3ac338b28656aa6461e751758ce22399128f4a290a0ba705ae137baadd548f24","048c3acf5cad1a4c18467ea64382b1e50d2a5f4cc9060219d360c5f50a8ef7c01875295a3b0801c06240e888afce9c17d2bbd295bea9aafb0b6d7d1c69b0e154c5","04a8b14eb8ae0a6decb12e3b8b3c907e2084fd74d30bd00b93e687b0bf7a450055706aed7f2aaf5ae1bff08b49a0310af3f1cc5b949607f522810ed9a139f6a69e"]'
{
    "address" : "3Ej7teuzS7AGQGFdGPfRpyCwzRUq7fUHoQ",
    "redeemScript" : "52410453170015ba9dd27388e51e824a8d5516b7627c025e3727232d9803ffd9c2b99d3ac338b28656aa6461e751758ce22399128f4a290a0ba705ae137baadd548f2441048c3acf5cad1a4c18467ea64382b1e50d2a5f4cc9060219d360c5f50a8ef7c01875295a3b0801c06240e888afce9c17d2bbd295bea9aafb0b6d7d1c69b0e154c54104a8b14eb8ae0a6decb12e3b8b3c907e2084fd74d30bd00b93e687b0bf7a450055706aed7f2aaf5ae1bff08b49a0310af3f1cc5b949607f522810ed9a139f6a69e53ae"
}

Trying to create the same multisig address with pybitcointools:
Code:
 pubs = [same 3 pub keys as above]
mscript = mk_multisig_script(pubs,2,3)
msigaddr = scriptaddr(mscript)
print msigaddr
print mscript

gives this output:
Code:
38izn5J4EECw5NNFBE27fwLyxkaBAsavSF
 52410453170015ba9dd27388e51e824a8d5516b7627c025e3727232d9803ffd9c2b99d3ac338b28656aa6461e751758ce22399128f4a290a0ba705ae137baadd548f2441048c3acf5cad1a4c18467ea64382b1e50d2a5f4cc9060219d360c5f50a8ef7c01875295a3b0801c06240e888afce9c17d2bbd295bea9aafb0b6d7d1c69b0e154c54104a8b14eb8ae0a6decb12e3b8b3c907e2084fd74d30bd00b93e687b0bf7a450055706aed7f2aaf5ae1bff08b49a0310af3f1cc5b949607f522810ed9a139f6a69e53ae

The address generated is different, but the redeem script is the same. Something must be wrong..?


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: Undecidable on October 24, 2013, 09:51:25 PM
Your problem is that mk_multisig_script returns a hex encoded string and scriptaddr expects binary.  Fix:

Code:
msigaddr = scriptaddr(mscript.decode('hex'))


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: waxwing on October 24, 2013, 10:24:47 PM
Your problem is that mk_multisig_script returns a hex encoded string and scriptaddr expects binary.  Fix:

Code:
msigaddr = scriptaddr(mscript.decode('hex'))

Many thanks. That solved that one. Do you happen to know the answer to my previous question too? (i.e. I'm still getting "Invalid Signature" after signing with the two private keys).


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: Vitalik Buterin on October 25, 2013, 05:49:20 AM
Okay, made some fixes.

1. scriptaddr takes hex or bin input
2. mk_multisig_script no longer sorts public keys (at first I did this so that people could deterministically make a multisig out of some keys without having to remember the order; this incident now convinced me that such a decision was inappropriate for a low-level library like pybitcointools). This should fix some problems as signature validation issues can arise if multisig signatures are in the wrong order.

However, the "Invalid Signature" problems are not entirely the fault of (2); rather, it seems like blockchain.info is screwing up. I spent some time debugging this; then however I tried pushing the same transaction using `electrum sendrawtransaction [tx]` and everything worked fine. I've run this again; same result.

Also, here are the instructions for how to make a multisig with pybtctool (the included command line wrapper for pybitcointools), copied straight from the command line:

Code:
vub@vub-U200: pybtctool random_key > k1
vub@vub-U200: pybtctool random_key > k2
vub@vub-U200: pybtctool random_key > k3

vub@vub-U200: cat k1 | pybtctool -s privtopub > p1
vub@vub-U200: cat k2 | pybtctool -s privtopub > p2
vub@vub-U200: cat k3 | pybtctool -s privtopub > p3

vub@vub-U200: cat p1 p2 p3 | pybtctool -s mk_multisig_script 2 3
524104b469d4fafd936e8630a5377223c5b7acfa9d2948b96845cc332607ec8528e6018abe46af616567ca44fca2d703a400dfdae73d0e21265c4dbeaf09e6294aa41941048ac7607d833aedb50c7be390955513f40327ff83891b53db2b241af64526ac17b37e7599f5ab2d4c7152779470dbc95c91526217de7bc002197ded7410526bfe4104cad74f0cf89dca7e13f37257e81308772fe3c7280f656ac88367867c54256ed606aa4dd2dc8ce4e73abb3159974f11657d2a365cee6fe86eeec4a8ce529cecd153ae
vub@vub-U200: cat p1 p2 p3 | pybtctool -s mk_multisig_script 2 3 > script

vub@vub-U200: pybtctool multisign `cat tx` 0 `cat script` `cat k1` > s1
vub@vub-U200: pybtctool multisign `cat tx` 0 `cat script` `cat k2` > s2
vub@vub-U200: pybtctool multisign `cat tx` 0 `cat script` `cat k3` > s3

vub@vub-U200: pybtctool apply_multisignatures `cat tx` 0 `cat script` `cat s1` `cat s3` > stx
vub@vub-U200: pybtctool apply_multisignatures `cat tx` 0 `cat script` `cat s1` `cat s3`
010000000125b98a02b8eb1ef3efb471cc87d6f028b53eadf29fe1cfb73fd0151188b6fe9300000000fd5f0100493046022100bcb0e128a7f4f39575f538979ecce4d49142ef6b613b4a98609bc14a7ae5ee0c02210094aca62aeae8301711560171087d8f069b79016d037f4e806df6c2efe12354e3014830450220257a3130f1d47ddf06b6a33882a1f1295cd8506f1ba1383b8b4cb5a40813c7ff022100ccea8cf93c54540f770e805a61585fb0ae82d6b6f68929b65afb0d6a9c0aa4ab014cc9524104b469d4fafd936e8630a5377223c5b7acfa9d2948b96845cc332607ec8528e6018abe46af616567ca44fca2d703a400dfdae73d0e21265c4dbeaf09e6294aa41941048ac7607d833aedb50c7be390955513f40327ff83891b53db2b241af64526ac17b37e7599f5ab2d4c7152779470dbc95c91526217de7bc002197ded7410526bfe4104cad74f0cf89dca7e13f37257e81308772fe3c7280f656ac88367867c54256ed606aa4dd2dc8ce4e73abb3159974f11657d2a365cee6fe86eeec4a8ce529cecd153aeffffffff0110270000000000001976a91405778c85c60c69598ea5bc896f872415fa7100b288ac00000000
vub@vub-U200: pybtctool pushtx `cat stx`
Traceback (most recent call last):
  File "/usr/local/bin/pybtctool", line 14, in <module>
    o = vars()[cmd](*args)
  File "/usr/local/lib/python2.7/dist-packages/pybitcointools/bci.py", line 57, in pushtx
    return make_request('http://blockchain.info/pushtx','tx='+tx)
  File "/usr/local/lib/python2.7/dist-packages/pybitcointools/bci.py", line 13, in make_request
    raise Exception(p)
Exception: Invalid Signature
vub@vub-U200: electrum sendrawtransaction `cat stx`
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/electrum/interface.py", line 329, in init_tcp
    s.connect(( self.host.encode('ascii'), int(self.port)))
  File "/usr/lib/python2.7/ssl.py", line 331, in connect
    self._real_connect(addr, False)
  File "/usr/lib/python2.7/ssl.py", line 324, in _real_connect
    raise e
timeout: timed out
Using random server...
electrum.stepkrav.pw:50002:s
Connected to electrum.stepkrav.pw:50002
"914be4d2d48426959844520264e75f4c751b93cef8b5c13b2365bebcd9aa7181"

The instructions with plain pybitcointools are pretty much exactly the same.


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: waxwing on October 25, 2013, 08:56:14 AM
Okay, made some fixes.

1. scriptaddr takes hex or bin input
2. mk_multisig_script no longer sorts public keys (at first I did this so that people could deterministically make a multisig out of some keys without having to remember the order; this incident now convinced me that such a decision was inappropriate for a low-level library like pybitcointools). This should fix some problems as signature validation issues can arise if multisig signatures are in the wrong order.

However, the "Invalid Signature" problems are not entirely the fault of (2); rather, it seems like blockchain.info is screwing up. I spent some time debugging this; then however I tried pushing the same transaction using `electrum sendrawtransaction [tx]` and everything worked fine. I've run this again; same result.

Also, here are the instructions for how to make a multisig with pybtctool (the included command line wrapper for pybitcointools), copied straight from the command line:

Code:
vub@vub-U200: pybtctool random_key > k1
vub@vub-U200: pybtctool random_key > k2
vub@vub-U200: pybtctool random_key > k3

vub@vub-U200: cat k1 | pybtctool -s privtopub > p1
vub@vub-U200: cat k2 | pybtctool -s privtopub > p2
vub@vub-U200: cat k3 | pybtctool -s privtopub > p3

vub@vub-U200: cat p1 p2 p3 | pybtctool -s mk_multisig_script 2 3
524104b469d4fafd936e8630a5377223c5b7acfa9d2948b96845cc332607ec8528e6018abe46af616567ca44fca2d703a400dfdae73d0e21265c4dbeaf09e6294aa41941048ac7607d833aedb50c7be390955513f40327ff83891b53db2b241af64526ac17b37e7599f5ab2d4c7152779470dbc95c91526217de7bc002197ded7410526bfe4104cad74f0cf89dca7e13f37257e81308772fe3c7280f656ac88367867c54256ed606aa4dd2dc8ce4e73abb3159974f11657d2a365cee6fe86eeec4a8ce529cecd153ae
vub@vub-U200: cat p1 p2 p3 | pybtctool -s mk_multisig_script 2 3 > script

vub@vub-U200: pybtctool multisign `cat tx` 0 `cat script` `cat k1` > s1
vub@vub-U200: pybtctool multisign `cat tx` 0 `cat script` `cat k2` > s2
vub@vub-U200: pybtctool multisign `cat tx` 0 `cat script` `cat k3` > s3

vub@vub-U200: pybtctool apply_multisignatures `cat tx` 0 `cat script` `cat s1` `cat s2` `cat s3` > stx
vub@vub-U200: pybtctool apply_multisignatures `cat tx` 0 `cat script` `cat s1` `cat s2` `cat s3`
010000000125b98a02b8eb1ef3efb471cc87d6f028b53eadf29fe1cfb73fd0151188b6fe9300000000fda80100493046022100bcb0e128a7f4f39575f538979ecce4d49142ef6b613b4a98609bc14a7ae5ee0c02210094aca62aeae8301711560171087d8f069b79016d037f4e806df6c2efe12354e30148304502207aebad2cafeabe58f4313cf380f48a3bf4b773819f2e617cdc6e68eddef99b380221009062bfbc94efce86807c7fd01d94211c75041e49f9564b900f3457e2676ea5b1014830450220257a3130f1d47ddf06b6a33882a1f1295cd8506f1ba1383b8b4cb5a40813c7ff022100ccea8cf93c54540f770e805a61585fb0ae82d6b6f68929b65afb0d6a9c0aa4ab014cc9524104b469d4fafd936e8630a5377223c5b7acfa9d2948b96845cc332607ec8528e6018abe46af616567ca44fca2d703a400dfdae73d0e21265c4dbeaf09e6294aa41941048ac7607d833aedb50c7be390955513f40327ff83891b53db2b241af64526ac17b37e7599f5ab2d4c7152779470dbc95c91526217de7bc002197ded7410526bfe4104cad74f0cf89dca7e13f37257e81308772fe3c7280f656ac88367867c54256ed606aa4dd2dc8ce4e73abb3159974f11657d2a365cee6fe86eeec4a8ce529cecd153aeffffffff0110270000000000001976a91405778c85c60c69598ea5bc896f872415fa7100b288ac00000000
vub@vub-U200: pybtctool pushtx `cat stx`
Traceback (most recent call last):
  File "/usr/local/bin/pybtctool", line 14, in <module>
    o = vars()[cmd](*args)
  File "/usr/local/lib/python2.7/dist-packages/pybitcointools/bci.py", line 57, in pushtx
    return make_request('http://blockchain.info/pushtx','tx='+tx)
  File "/usr/local/lib/python2.7/dist-packages/pybitcointools/bci.py", line 13, in make_request
    raise Exception(p)
Exception: Invalid Signature
vub@vub-U200: electrum sendrawtransaction `cat stx`
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/electrum/interface.py", line 329, in init_tcp
    s.connect(( self.host.encode('ascii'), int(self.port)))
  File "/usr/lib/python2.7/ssl.py", line 331, in connect
    self._real_connect(addr, False)
  File "/usr/lib/python2.7/ssl.py", line 324, in _real_connect
    raise e
timeout: timed out
Using random server...
electrum.stepkrav.pw:50002:s
Connected to electrum.stepkrav.pw:50002
"914be4d2d48426959844520264e75f4c751b93cef8b5c13b2365bebcd9aa7181"

The instructions with plain pybitcointools are pretty much exactly the same.

Thanks muchly.
About sorting the pubkeys, I agree; I was a bit bewildered at first why you did it, then I saw the point, but finally I agree it's less confusing to just not do it.
About blockchain.info, the reason I didn't suspect it being the problem is that at least one person (and maybe others I read online) told me that they definitely do accept multisig txs. Is there any contact/support person there I could ask for confirmation, I wonder?

And one last question about this post: Why are you applying all 3 signatures? Isn't this 2 of 3?


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: Vitalik Buterin on October 25, 2013, 09:37:56 AM
Thanks muchly.
About sorting the pubkeys, I agree; I was a bit bewildered at first why you did it, then I saw the point, but finally I agree it's less confusing to just not do it.
About blockchain.info, the reason I didn't suspect it being the problem is that at least one person (and maybe others I read online) told me that they definitely do accept multisig txs. Is there any contact/support person there I could ask for confirmation, I wonder?

And one last question about this post: Why are you applying all 3 signatures? Isn't this 2 of 3?

Yep, sorry, meant to apply only 2 of 3. I made a mistake at first and copied over the wrong lines from my command line history. Retroactively fixed in my above comment.

As for bugs, I submitted it to the thread: https://bitcointalk.org/index.php?topic=40264.msg3408171#msg3408171

Also, one suggestion request: should I made deserialize call deserialize_script automatically? That way deserialization would decompose the transaction all the way, including breaking down the script. If not, then I'll probably make a separate showtx method which would do a full decomposition and also try to calculate the input addresses for standard pubkey and p2sh transactions.


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: waxwing on October 25, 2013, 10:20:35 AM
Can I ask which version of electrum you're using? I have 0.9 installed on my vps (because they're in the process of implementing multisig in the new version) but I can't get it to work right now with that, at least not yet.

Edit: ignore that; I was calling sendtx wrongly within Electrum. I finally managed to send my first multisig payment on the network :)

I find Electrum really flaky though; it took me three tries of running my script before the tx got accepted by a server. Sometimes it just hangs (messages like SSL Handshake timeout and others), and weirdly the second time I sent the transaction I got an error code -22 ( (False, u"error: {u'message': u'TX rejected', u'code': -22}")), but the third time it got sent  ((True, u'3e1dbe4c661b83b3d5ff22abf0e545924350c666c65ca875ac6aed741fb902b7')).
Any hints on how to make it robust, bearing in mind I'm trying to do everything within Python?

About your deserialize question: I'll have to take another look at the code, not sure if I can comment, but I'll try.


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: waxwing on October 25, 2013, 06:03:10 PM
Also, one suggestion request: should I made deserialize call deserialize_script automatically? That way deserialization would decompose the transaction all the way, including breaking down the script. If not, then I'll probably make a separate showtx method which would do a full decomposition and also try to calculate the input addresses for standard pubkey and p2sh transactions.

I'm not knowledgeable enough in the field to be very familiar with what people want, but having tried it out, I can definitely see a use for it (script deserialization). Whether there are use cases involving not doing it, such as to tie up the format with what bitcoind produces, I don't know, but I guess so. The obvious answer is to make it optional with a default of not doing it, i.e. deserialize(tx, deserialize_script=False)

Edit: a small thought, you might want to translate 174 to OP_CHECKMULTISIG or something like that. Being a noob in Bitcoin development I was confused by that kind of thing at first :)


Title: Re: Python Bitcoin ECC library, supports signing, Electrum, transactions (no blocks)
Post by: Vitalik Buterin on October 28, 2013, 06:07:46 AM
I quite like your library.
It took me a while to realise that it actually does handle wallet import format. It works if you put WIF into privtopub() although it took me some searching and thinking to realise you can use bin_to_b58check() to make WIF from a binary private key. Maybe a priv_to_wif() function should be made?

Done, although not quite in that form. It's:

> key = sha256('123')
> key
'a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3'
> encode_privkey(key,'wif')
'5K5ZyFA5NqiBzCbiaTh5KYJmSh7dXdk65DvGkcx9G6ydsj83TM1'


Or just
Code:
cat key | pybtctool -s encode_privkey wif

What kind of license is this under?

Public domain; if that's not accepted in your jurisdiction then MIT. Just added the license file today.

Quote
Also there seems to be no pure python impl of scrypt. I know this goes against your intent (which I agree with) of having no dependency chasing.

Relying on an external C++ library for scrypt is fine; doing it pure python forces you to set lower parameters for practicality, which weakens security. My general rule is, if it's in a default Python installation, it's probably fine.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: waxwing on October 29, 2013, 12:58:18 PM
In signing multisig transactions with pybitcointools.apply_multisignatures, I found that you have to apply the signatures in the correct order.

I can't find a clear explanation anywhere of how the signatures are applied, but here (https://en.bitcoin.it/wiki/Script) it says:
Quote
For each signature and public key pair, OP_CHECKSIG is executed. If more public keys than signatures are listed, some key/sig pairs can fail. All signatures need to match a public key. If all signatures are valid, 1 is returned, 0 otherwise
That's a bit ambiguous ("All sigs need to match a public key"), but it looks like it means that each sig-pubkey is checked as a pair. I don't get how this works.

E.g. pubkeys PA, PB,PC. We apply signatures SA SB. First sig SA matches first pubkey PA, second matches second, so it returns 1. But what if we apply SA SC? It seems from my experiments that it works, but that if we apply SC SA, it doesn't. Is that what's supposed to happen?



Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: deepceleron on October 29, 2013, 01:21:04 PM
This library seems to have no problem putting invalid private keys through math without validation:
Code:
>>> pybitcointools.privtopub('0000000000000000000000000000000000000000000000000000000000000000')
'0400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'
>>> pybitcointools.privtopub('ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffee')
'04d3c3c891a4e05e520b01928b19cf7b56b7debc725dd915a42763a6c94bcdce54e171bf8daa5328e7ef955a8d60a9e72cc6fe5ac918eac04134d692a34654c6c4'
>>> pybitcointools.get_privkey_format('ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffee')
'hex'


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: Vitalik Buterin on October 29, 2013, 01:52:59 PM
In signing multisig transactions with pybitcointools.apply_multisignatures, I found that you have to apply the signatures in the correct order.
...

E.g. pubkeys PA, PB,PC. We apply signatures SA SB. First sig SA matches first pubkey PA, second matches second, so it returns 1. But what if we apply SA SC? It seems from my experiments that it works, but that if we apply SC SA, it doesn't. Is that what's supposed to happen?

Yes, that's how it's supposed to work. It's a Bitcoin protocol thing; it's done this way for efficiency reasons as I understand, so signature verification can be linear (try the pubkeys in order) instead of quadratic (try every sig with every pubkey to see if any combination works).


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: waxwing on October 29, 2013, 02:11:41 PM
In signing multisig transactions with pybitcointools.apply_multisignatures, I found that you have to apply the signatures in the correct order.
...

E.g. pubkeys PA, PB,PC. We apply signatures SA SB. First sig SA matches first pubkey PA, second matches second, so it returns 1. But what if we apply SA SC? It seems from my experiments that it works, but that if we apply SC SA, it doesn't. Is that what's supposed to happen?

Yes, that's how it's supposed to work. It's a Bitcoin protocol thing; it's done this way for efficiency reasons as I understand, so signature verification can be linear (try the pubkeys in order) instead of quadratic (try every sig with every pubkey to see if any combination works).

Thanks. That makes sense. Only a tiny bit of extra code to write :)


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: danneu on November 09, 2013, 07:12:28 PM
Huge thanks for sharing.

This is the most intelligible/simple implementation I've found yet, and not just for ECC.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: deepceleron on November 21, 2013, 11:47:19 AM
You can grab line 573-590 out of here if you want to add a "print paper wallet" def print_wallet(priv) to your code:

http://we.lovebitco.in/dice2key.py

Or grab the whole thing to line 483 if you want a pretty version of main.py, but there's still some warnings for python 2.7:

Expected type 'Number', got 'str | unicode' instead (at line 429) (edit: this appears to be an invalid warning I can't make go away)
Unexpected argument (at line 92) (incorrect call argument) (edit: VB fixed this)
472-475 "Q and Qr" are uppercase
redundant parenthesis. 94, 98, 103, 107, 112, 162, 167, 169

along with more PEP8 if: statements on same line, spacings etc.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: phelix on November 28, 2013, 07:39:15 PM
Quote
Thanks for the commit!
My pleasure. PBT is great.


Title: Re: Python Bitcoin ECC library, supports ECDSA, Electrum, altcoins (no txes/blocks)
Post by: sdp on December 03, 2013, 01:22:50 PM
Nice, congratulations. Though I would've suggested a different language cause simply Python ain't all that fast when it comes to these expensive operations there.


This reminds me when someone mentioned to me that I shouldn't use iostreams in C++.  That it was faster to use C-style printf functions.  After some bench marking and some inspection he was right that C++ iostreams are slower.  Each << is a function call.  I think this argument is similar.  In the end it doesn't matter.  Computers are fast today.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: phelix on December 13, 2013, 10:48:39 AM
I assume mktx will put anything not spent explicitly to mining fees?


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: waxwing on December 17, 2013, 06:37:22 PM
Are there any practical limits in numbers of parties to a multisig transaction? I am thinking of a scenario like 6 of 10 of 8 of 15. Does the size of the script start to break anything at any point?

Quote
I assume mktx will put anything not spent explicitly to mining fees?
Yes, it works like that here as that's part of the bitcoin protocol.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: justusranvier on December 17, 2013, 06:38:33 PM
Are there any practical limits in numbers of parties to a multisig transaction? I am thinking of a scenario like 6 of 10 of 8 of 15. Does the size of the script start to break anything at any point?
n=20 is the maximum allowed by protocol.

n>3 is currently non-standard, so you have to make a private arrangement with a miner to get those transactions included in a block.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: waxwing on December 17, 2013, 07:03:25 PM
Are there any practical limits in numbers of parties to a multisig transaction? I am thinking of a scenario like 6 of 10 of 8 of 15. Does the size of the script start to break anything at any point?
n=20 is the maximum allowed by protocol.

n>3 is currently non-standard, so you have to make a private arrangement with a miner to get those transactions included in a block.

Ah thanks that is a very important point for me ... I did notice the darkwallet guys using a 3 of 5 address, I guess I can ask them about that. I only have one special use for large numbers, maybe I can work around it.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: waxwing on December 23, 2013, 04:49:30 PM
(On the above posts, Vitalik seems to have added an "eligius_pushtx" method to allow the more exotic multisig txs to go through (eligius processes them apparently) so that's cool :) )

Question: I want to do verifying messages with addresses from external wallets
..[snipped]

Question can be ignored, the code is already there in ecdsa_recover. Thanks :)


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: natb on December 29, 2013, 04:25:22 PM
Vitalik, this is exactly what I was looking for. I'm going to replace my HW wallet test harness with your library (the one I was using before was functional, but I'd say not the cleanest library in the world). Thanks for sharing - I'll let you know if I run across any issues or have any improvement ideas.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: phelix on January 23, 2014, 08:37:29 AM
the time part of the two random functions looses entropy:

str(int(time.time())**7)   is probably meant to be
str(int(time.time()**7))   or
str(time.time()**7)

It would be nice to be able to add some base entropy by hammering your tinfoil hat on the keyboard at start.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: waxwing on March 02, 2014, 01:18:37 PM
I opened an issue with regard to signature padding here: https://github.com/vbuterin/pybitcointools/issues/9

I think fixing it might be a bit above my paygrade (or at least would take me a while), so if anyone can help I'd appreciate it.

Edit: Vitalik pushed a fix already, thanks (although it needs testing, which might be tricky).


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: Wanesst on March 03, 2014, 07:24:04 PM
That's cool!  But a different language can be better cause this ain't all that fast


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: yakov on March 04, 2014, 01:05:06 AM
What are you coding that needs such breakneck speed?


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: hobbes on March 07, 2014, 06:12:36 PM
My application of pybitcointools: offbit - Minimalistic offline key generation, offline TX signing and Tor protected TX broadcasting.

https://bitcointalk.org/index.php?topic=488915.0


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: hamish.macewan on March 23, 2014, 06:23:18 PM
I deliberately designed the library to be simple and no-nonsense.
Thanks for the effort, and I feel bad having to ask for assistance when so much has already been given.

What is required before you reach this point
Quote
> from pybitcointools import *

The command line examples are fine, but this "> from pybitcointools import *" only caused me grief.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: instagibbs on March 23, 2014, 07:00:20 PM
I deliberately designed the library to be simple and no-nonsense.
Thanks for the effort, and I feel bad having to ask for assistance when so much has already been given.

What is required before you reach this point
Quote
> from pybitcointools import *

The command line examples are fine, but this "> from pybitcointools import *" only caused me grief.

Same here. I couldn't get it to import, even with adding the correct folders to path(even inside the script using sys.path).

I'm no linux expert though so I'm probably doing something wrong.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: waxwing on March 25, 2014, 04:07:22 PM
I deliberately designed the library to be simple and no-nonsense.
Thanks for the effort, and I feel bad having to ask for assistance when so much has already been given.

What is required before you reach this point
Quote
> from pybitcointools import *

The command line examples are fine, but this "> from pybitcointools import *" only caused me grief.

"from pybitcointools import *" is Python.

If you have Python installed, you can just type 'python' (assuming it's in your path) at the command line to start the Python interpreter, and then you will get
 
>

from there you can do the above command.

Feel free to ask further questions as necessary. People will be happy to help.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: gabridome on March 28, 2014, 02:40:23 PM
Thank you for all you are doing for us.
I have scarse experience in python and programming in general but I like a lot your tutorial on multisig (http://bitcoinmagazine.com/11113/pybitcointools-multisig-tutorial/).

I get the following error when I type the first line of code:
>>> from pybitcointools import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pybitcointools

also using the command line tool I get instead:
 k1=`pybtctool random_key`
Traceback (most recent call last):
  File "/usr/local/bin/pybtctool", line 8, in <module>
    cmd = sys.argv[2]
IndexError: list index out of range

I work on Mac OS. I have installed the tools with:

sudo pip install bitcoin

Thank you.


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: tongbao on March 30, 2014, 10:01:31 AM
nice work, put a footprint here [feasibility on multiple public keys]


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: instagibbs on April 01, 2014, 02:40:07 PM
Thank you for all you are doing for us.
I have scarse experience in python and programming in general but I like a lot your tutorial on multisig (http://bitcoinmagazine.com/11113/pybitcointools-multisig-tutorial/).

I get the following error when I type the first line of code:
>>> from pybitcointools import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pybitcointools

also using the command line tool I get instead:
 k1=`pybtctool random_key`
Traceback (most recent call last):
  File "/usr/local/bin/pybtctool", line 8, in <module>
    cmd = sys.argv[2]
IndexError: list index out of range

I work on Mac OS. I have installed the tools with:

sudo pip install bitcoin

Thank you.

Same exact issue. The command-line tool works just fine. Can someone post what is in their python path?


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: rapport on May 14, 2014, 10:14:26 AM
Try "from bitcoin import *"

I get the following error when I type the first line of code:
>>> from pybitcointools import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pybitcointools


Title: Re: Python Bitcoin ECC library, supports signing, transactions, determinstic wallets
Post by: Tafelpoot on December 23, 2014, 11:46:02 AM
I'm making a UI on top of this, similar to electrum.
I really like your clean code!