Pywallet 2.2: manage your wallet [Update required]

[BRules]

hi man, tks for the update, it will for sure speed things a little, but here's a problem:

When making a request to delete a transaction, all the parameters are being passed through GET Method, and the GET method has some limits regarding the length of the URL (http://classicasp.aspfaq.com/forms/what-is-the-limit-on-querystring/get/url-parameters.html). This length depend on the server and also the client, for example, until and including Internet Explorer 8 the maximum URL length is 2.083 characters. So, if I have about 100000 transactions to delete, and lets suppose that I will be able to delete 500 transactions per request, taking the average of 11 seconds to complete each request in my wallet of 242MB, it will take about 36 minutes to complete the deletion. This is a great advance instead of waiting 12 days to delete the transactions one by one, but as I said before, using the "all" keyword it took less than 20 seconds. I still think that reading a file with the keys to delete will still be the best way.

Another problem, When I tried to merge two wallets, in one of then the password contained the character #, and I received an error. when I checked the URL being called, I saw that it stripped all the remaining characters including the character #, so I'm assuming that the problem is that the URL isn't being encoded.


One more question:
- After deleting all the transactions from the wallet, it dropped the file size by a few KB. is it normal? why the size didn't drop more?


And once again, tks for your great tool.

[jackjack]

Yeah I forgot while doing it, I'll make this soon.

I never tried with such a passphrase. Try with replacing # with %23.
I'll look at that when I'm back home. I think it would just require to put an encodeURIComponent.

As for the small size drop, I don't know, it depends how many transactions your wallet contained. Compare the dumps before and after deletion to be sure there's no transaction left.

[BRules]

Just created the dump with my wallet with "no transactions" (actually it now has 1 transaction as I moved back my coins), and I confirmed that it has only one transaction.

Copied all the text of the JSON dump and saved it to a text file and the text file size is 32MB (noting that I have about 70k addresses generated).

And using the wallet in the bitcoin-qt it is showing only one transaction too.

before deleting the transactions it contained about 230000 transactions (didn't made the dump with the wallet file with all transactions as I think it will take too much time as it took about 11 minutes with the wallet without the transactions).

Regards.

[BRules]

by the way, I received the following error when making the dump (but it completed successfully):

2013-05-22 18:36:52-0300 [HTTPChannel,2,127.0.0.1] Wallet data not recognized: {'__type__': 'orderposnext', '__value__': '\xc1\x8b\x03\x00\x00\x00\x00\x00', '__key__': '\x0corderposnext'}

[jackjack]

I just pushed the new version: no more '#' bugs in passphrases and you can use file for deletions

Format: 1st line=type ('tx' or 'key'), then 1line=1data to delete
Code: pywallet.py --multidelete file.txt [--wallet wallet.dat] [--datadir /bitcoin/directory]

Code: (file.txt)

tx
b0b0865e723101b5a720a79d805efd544c59fa2ceebffc724a12cd5fe683057f
07941322e6dd5b15b3bde0739648d49526e19ea78b8f4085e943fca473cbfdbf
etc...

by the way, I received the following error when making the dump (but it completed successfully):

2013-05-22 18:36:52-0300 [HTTPChannel,2,127.0.0.1] Wallet data not recognized: {'__type__': 'orderposnext', '__value__': '\xc1\x8b\x03\x00\x00\x00\x00\x00', '__key__': '\x0corderposnext'}

That just means that the 'orderposnext' entry in your wallet isn't recognized. That is because I personally don't know what it is and I doubt it's useful.
Pywallet will keep it inside your wallet when you import/delete/dump/whatever though, it just doesn't understand it

[BRules]

Hi,

I downloaded your new pywallet, modified my program to generate a file with all my transactions ids and started pywallet with the multidelete option.

After 15 minutes running I decided to interrupt the execution of the pywallet and start investigating your python code to try to determine where the problem is (I never worked with python, but as a programmer I haven't problem undertanding it.) and I think I found it.

The problem is that my file has about 200k transactions and my wallet has at least 200k+70k(addresses) keys. Well, if for each transaction in the file, it will interact trough all the keys of the wallet (I didnt' see anywhere in the code where you break the iteratioon of the second "for" when finding the key), so the pywallet will need to do a total of 54 billions of iterations.

So I was thinking a bit and I have a question and a suggestion.

Question: in the line of code "db.delete(key)", if the "key" don't exist, will it raise an error or it will pass without problem?

My suggestion: when deleting transactions, instead of applying the modifications on the "key" to compare to the "keydel", why not just transforming the "keydel" to the "key" format and deleting it without iterating through all the wallet keys?

Regards.

[jackjack]

Question: in the line of code "db.delete(key)", if the "key" don't exist, will it raise an error or it will pass without problem?

It looks like that it passes without problem. Actually the delete function searches in all the DB to find all the keys to delete, so my method is pretty dumb...
I made it at the beginning of my fork, which corresponds to the beginning of my learning of Python/serious programming. The more I look at the code I made, the more I find it ugly

My suggestion: when deleting transactions, instead of applying the modifications on the "key" to compare to the "keydel", why not just transforming the "keydel" to the "key" format and deleting it without iterating through all the wallet keys?

Yeah that's better that what's done currently, but as I wrote the delete function will take care of that even better

[jackjack]

I completely forgot to work on it, I'll do this in the following hours

[jackjack]

I made a small improvement
I'm not really sure it would change a lot though. I don't think it might make it faster than now, because of the way Bitcoin stores the txs

[BRules]

hi, I looked your new code and, as you said, maybe it won't change the performance to much.

I was thinking in something like this:

at line 1779

   for i in range(len(kd)):
      keydel=kd[ i ]
      if typedel == "tx" and keydel <different> "all":
         <apply modifications to keydel to transform it to the key format>
         db.delete(key)
         deleted_items+=1
      else:
         for (key, value) in db.items():
            .
            .
            .

Do you think it is possible?

Regards

[cheesylard]

Can this work on jailbroken iOS?

[jackjack]

That's not possible, and that's what I meant by "the way Bitcoin stores txs": the key is "tx"+hash(tax). So you can apply what you proposed only if you know the hash.

Hmm, wait, I just get it, yes we have the hashes because you use the file to store them... I don't know why I thought you were using 'all'
There's some hope for speed improvement then!

Can this work on jailbroken iOS?

Yes absolutely. I made it work on iOS 5.
Look if there's a Python port for your iOS. If so that should work.

[jackjack]

It's done, please try

To make it faster I don't catch exceptions so if you provide at least one transaction that isn't in your wallet it will crash without telling you which one
Maybe you prefer catching exceptions and avoiding all the deletions to be canceled just because one failed, just tell me what is better

I added the exit code 1 in case of failure




To everybody
Do you think pywallet would be more practical as a Qt app?
As I said earlier I'm wondering about recoding everything. It's far from sure but if I decide to I'd like to know how to make it more user-friendly.

[K1773R]

It's done, please try

To make it faster I don't catch exceptions so if you provide at least one transaction that isn't in your wallet it will crash without telling you which one
Maybe you prefer catching exceptions and avoiding all the deletions to be canceled just because one failed, just tell me what is better

I added the exit code 1 in case of failure




To everybody
Do you think pywallet would be more practical as a Qt app?
As I said earlier I'm wondering about recoding everything. It's far from sure but if I decide to I'd like to know how to make it more user-friendly.

full CLI (ie the same capabilites like the WUI) and a WUI should be doing fine.

[BRules]

just tried and it tooked 5.5 minutes to delete all the 232386 transactions that was in my text file.

Great job, and thank you.

but after deleting it's showing the following:

"232386 elements deleted
Error: do not try to delete a non-existing transaction."

looks like it's raising an error on the command "exit(0)"


a GUI interface will be cool, as far I start earning some money with my project I will support this development.

regards

[jackjack]

I'm glad that worked and that is quick
I didn't even think that exit would raise an exception, it's ok now


You both confirm what I was afraid of, that that'd nice to have CLI+GUI+WUI
I think I won't use the twisted package anymore to deal with the WUI though, I'll manage sockets myself
I'm fed up with dependencies

[dserrano5]

To everybody
Do you think pywallet would be more practical as a Qt app?

Just as K1773R, I'd like to have a full working command-line interface, and am comfortable enough with a web browser for point n click usage.

[jackjack]

Yeah, the CLI is the priority anyway

[phelix]

Yeah, the CLI is the priority anyway

+1 to command line

It would be nice if the project was separated in three files to be able to look over it more easily:
- core functions
- command line interface
- gui/web interface

[jackjack]

I like putting everything into a single file because I find it easier to track and easier for newbies
I can split it in three though and then push the four files to github