Latest posts of: etotheipi

Well, now there's a use for my first post: the invalidated transaction will happen as soon as the first block is produced that has one of them included. If 90% of the world sees TxA first, and 10% see TxB first, there's probably about 90% chance that TxA will end up being the valid one. But regardless of the ratio, that will happen on the next block... so 1-20 minutes (dependent on global luck), for the network to "agree" which one is valid.

I suppose it's possible to end up with both blocks being solved at the same time, then you end up with another race to see which block is valid, in the exact same way there was a race to see which Tx was valid. But that's a boundary case, and it really only extends the validity "agreement" for another one more block.

To be more direct... once you get that first confirmation, the invalid Tx is purged from the network pretty quickly. I don't see why there's a problem getting the client to figure that out, too

So, as an update: I finally got some of my tools together and I have implemented the blockchain loading. Reading all the headers from blk0001.dat, and organzing the chain, with orphan checking, etc, takes approximately 1 second on my system. I am running a AMD Phenom II X4 840, which is a modest quad-core system, though I suspect only one core is being used since I didn't implement any threading.

So it concerns me that Mike's post says it takes approx 13s to load the headers. Is there a bug in the client code for executing this process? Is it just simply inefficient? According to Mike's post, this is the biggest bottleneck to getting the GUI open, so perhaps it's worth looking at what is going on here. Perhaps Mike has the capability to profile what part of loading the headers is slow...?

I get it. Using wallets on multiple computers are not handled well with the current client. But I don't see why this has to be a problem. The client can easily check each of its "0/Unconfirmed" transactions to see that they are permanently invalid. Why isn't it a simple patch to resolve this issue? Is there something I'm missing?

On the other hand, I have the opposite problem, now. 0/Unconfirmed transaction for months that is sitting there solely because because I didn't include a transaction fee (to see what happens, now I know). My client claims I don't have the money, but it was also never officially spent. I need a method to "pre-spend" this money by using the same TxOuts in a new transaction that will get handled. It won't fix the client (in fact, what will the client do??), but at least I'll get access to the money again. I'm very tempted to apply my python tools to find the relevant TxOuts, and manually construct a new Tx broadcast...

So, why can't the client see that the transaction is invalid and remove it from the list (or tell the user it's invalid)? After all, the fact that TxOuts have been used is easily verifiable.

Say, once a day, the client will scan the block chain for invalidating information about all "0/Unconfirmed" transactions in the wallet. If there are Tx's that use the outputs, that Tx can never be valid again and the user should be given some indication as such. Even if the Tx is technically still valid, it might be worth giving the user some feedback that it's still valid, just not handled yet.

So is there actual information in those TxOuts? Or was he just destroying coins for fun? It looks like gibberish to me...

I agree about burning bridges, and how it's probably not a good idea. However, I'll journey down this thought experiment a little more: The BTC destruction script mentioned in (1) could include a list of transaction IDs/hashes of the forkcoins that are taking their place. If forkcoin is anything like BTC, the forkcoin minting process (even if completely different than BTC) should still have uniquely identifiable coinbase/creation transactions where currency units are injected into the system. Then you can easily get a 1-1 mapping of BTC destroyed to forkcoins created.

I just realized I answered a question that you didn't ask. Whoops! Well hopefully you at least learned something from it!

I'm not sure what the official client would tell the users, but I can tell you what would happen in the network:

(1) A and B both broadcast transactions at the same time, using the same previous transaction "outputs" (you have to reference previous transactions to create new ones)
(2) Some nodes get word of TxA first, other nodes get word of TxB first. Whichever one they get first they will add to their queue and start trying to include them in a block
(3) When nodes get word of the second Tx, they will reject it because it conflicts with the first.
(4) The next block that is generated will include TxA or TxB, but not both. Let's assume it's TxA.
(5) The miners working with TxB will receive word of this next block, and switch to mining for the new one.
(6) They reconstruct the block, but realize that TxB is no longer valid because TxA (in the prev block) already used up the transaction outputs.
(6) TxB will be removed from the miner's Tx pool and never considered again.

So to answer your question more directly: one of the transactions will go through, and the other will be rejected. No one knows which one until the next block is generated. The probability that one wins over the other depends on what proportion of all miners receive TxA first vs those that receive TxB first.

Quote

New problems.

1) The exchanger needs to be able to prove that it has destroyed the bitcoins, or no one will trust it.

2) The exchanger needs to be able to prove that it has created no more newcoins than the number of bitcoins destroyed, or no one will trust it.

So, the exchange needs to be a trap door, or the whole thing falls apart. But, by being a trap door, it brings fear back in, since no one will give up their bitcoins unless they are pretty confident that the new system is not only viable, but will eventually dominate. Most people would be better off using a floating exchange, at least until the question of dominance has been settled.

Neither of these are problems:

(1) Coins can be verifiably "destroyed" by creating a transaction script that can't ever evaluate to true. This transaction enters the network and becomes public. Everyone can see it, and see that the PkScript is impossible to satisfy, ever. That's pretty good "proof". And since it's a custom script anyway, they could even include a message in the script which explains exactly the source and purpose (and signature) so that there is no confusion about whose coins were destroyed and why.

(2) If the new currency is anything like BTC, then similarly, the currency is all public information. Unless we're talking about non-decentralized currencies... all units of the currency should be traceable, both in time and quantity. If it's not, I don't see how this currency can really "compete" with BTC, since centralization and closed-ness would pretty much make the "fork" a non-fork.

Thanks. I've mentioned repeatedly in other posts before -- I can't grok the C++ code. Despite my experience with C++, it is completely incomprehendable to me, and I was hoping someone already knew the answer to this. Yesterday I did try to look at it, I saw a CDB class using a "Db*" and all it's derived classes, with a large web of inheritance. I never actually found where the data was stored, or how it's accessed, or where various methods are defined (probably all over the inheritance graph). Perhaps data is stored indexed by pair<hash, something> objects, but I can't tell for sure.

The even more frustrating part is that even if I develop a solution that's remarkably faster (I'm just about there, I'm reading an organizing headers nearly instantaneously from file), there's no way I can contribute to that code base, because there's no way I could ever get in that code and make a meaningful patch. I'm not sure what to even do with my code, besides keep it to myself and try to develop my own client...

Sorry my C++ skills aren't up to par. But that's why we have forums to discuss things. So in case all you were suggesting was to look at the code... I've done that. I've tried countless times to read the code and never understand what I find. Perhaps you can offer suggestions for how to understand it better...

-Eto

What is even in the log file? I never knew that it existed (and was a concern) until this thread...

I am much more concerned about the blockchain file, because that's a critical part of the protocol. Presumably, the log file can be trimmed... but the blk0001.dat cannot. And the more successful bitcoin becomes, the more that filesize is going to spiral out of control.

I guess there's no options for the miners, they're going to have to hold the whole file no matter what. But for the users, a reduced set will become necessary. Maybe not just yet, but eventually. Once the BTC network starts processing 1000 transactions per block, the blockchain is going to grow about 10-30 MB per day... or possibly 10 GB per year. It can still be handled by the miners, but the average user isn't going to want to hold that much data just to use the program.

Does anyone know how the headers and tx data is stored in RAM? If everything is referenced by 32-byte hashes, accessing blocks/tx's by hash could be very slow if we're not using a good data structure. A binary search tree probably isn't even good enough. A radix/patricia tree would be ideal, since you can access any object in about 40 clock cycles or less.

If someone is already doing some kind of speed profiling of the code, I'm wondering how long it takes for block/tx access based on hash? Maybe you could pick out a block hash, and put in a for loop to get the version of that block a million times and time how long it takes. I thrive on data structures, pointers, and low-level optimizations like this, I'd love to contribute.

I'm saying the latter -- I don't see why the client ever needs to distinguish between those two, it only matters whether the transaction is valid.

The only difference might be that a 0-confirmation transaction is a bit less trustworthy on a light node, because it doesn't have the ability to verify the transaction itself. It only knows for sure when it sees that transaction in a block, or can guess with high confidence that it wouldn't have received that Tx data unless it was valid, since invalid Tx's don't get very far in the network.

I'm working on some block-chain analysis tools right now, and playing/testing with the 374 MB of data up to block 136496. I'm not quite there yet, but I share your intrigue and might take a shot at that calculation in the next few days.

Quote

I was thinking about how to remove transactions when all the outputs have been used, but it seems to me that at least the transaction hash must be kept, because without it, it's impossible to tell the difference between an orphan transaction and a double-spend. Although since neither are fully valid, it might make sense to discard them both, and if the orphan later becomes a non-orphan, hope that it will be retransmitted.

I don't see why you need to keep that transaction. The entire state of the network (and everyone's balances) can be determined solely by the set of unused TxOut objects and the hash/index of their parent Tx object. And that's all the information you need to sign new transactions. All the TxIns and previous TxOuts are only necessary for blockchain verification, but that is done by the miners before they include them in a block. Your client can get away with storing just the TxOut information above, and trust that they are valid because they were part of the longest blockchain (which is difficult to fake), and would not be be there if they weren't valid.

You can store all the Tx's in a tree data structure, whose values are arrays of TxOut objects. As TxOut's are spent, you can remove them from the array, saving about 40 bytes for each of them. When the last TxOut in the array is spend, you can also remove the Tx node, which saves another 40 bytes (approx). You would only need to keep that data (and/or its hash) if you were concerned about verifying the transaction history.

When you say "forward the port 8333," do you mean port forwarding on my router? I tried that (any traffic to my DMZ IP address at port 8333 will go directly to my computer's port 8333). It doesn't make much sense to me, but maybe that's what you meant. I'll try it when I get home.

If you look back to Mike Hearn's post, he suggests that my 0.3.24 client won't be able to get the blockchain except from other 0.3.24 clients... which is how I thought it worked, too (but I haven't gotten to the networking part of the protocol yet, so what do I know?)

And after all this discussion, have we determined if there is a bottleneck that can be addressed? Assume everyone magically upgrades to 0.3.24 today, and the throttling bug disappears. Do we still expect speed issues? Should we be brainstorming better ideas for multi-threading, make GUI more accessible while your system is working in the background?

And finally, any reason you know why it takes 12 seconds to read the headers from blk0001.dat? That's how long it takes my python script to do it (with hashing and blockchain organization), so I assumed it could be done dramatically faster in C++ code...

But, given the difficulty getting people to upgrade their clients, it sounds like this is going to be a extremely persistent problem without doing something different. We should consider ways to allow clients with different versions to accommodate each other, at least for blockchain downloads. Rather than disconnecting clients with different versions immediately, maybe give the opportunity to exchange getdata() requests. And maybe there should also be some kind of announcement message as part of the client, which checks bitcoin.org for newer version and alerts the user they should upgrade.

The key problem (at least for me), is that if you are not up-to-date on the block headers, you can't do anything else. The other day I tried sending someone coins, wondering what would happen, and the client said "Payment Sent!" but stayed at "0/Unconfirmed" for 8 hours until it caught up to the top of the chain.

If you only need one good peer, why does my client take 1 minute per block to download? It behaves as if it's searching for a node, eventually finds one, gets one block, then gets disconnected and has to start over. The other day I was only 5 blocks behind, but it was taking like 3 minutes per block, so it took forever to catch up to the blockchain. Yes, this is linux, client 0.3.24.

P.S. - Perhaps I'm an outlier... does anyone else have these kinds of problems getting blocks from the network?

(1) Does it take 13s to load just the headers? Or the entire blk0001.dat? Right now, I can open and scan the 11MB of block headers in C++ in about 1s. The block data takes considerably longer since there's 400 MB.

(2) I noticed that my client in Linux takes a tremendous amount of time to download updated blocks--perhaps 1-4 per minute. Even when I close the client for only a day, it can take hours for it to get caught up. But when I boot my windows XP VM that has a different, "savings account" wallet on it, that client gets up to speed almost immediately (probably, 1 per sec as Mike suggested). I recognize that there is a peer/version discovery issue, but why would it be so dramatically different on the two systems? This makes it almost unusable on Linux, unless I open the client a full 1-8 hours before I want to send someone money. It's faster for me to boot my VM, load the BTC client there, and send directly from my savings account. 8 hours later, when my linux client is ready, I reimburse my savings account...

(3) Could the finding of node discovery/version issues warrant looking into using the bit-torrent protocol (or similar) for moving blocks around? The version should not matter for moving this data around, but the necessity of version matching to make a connection forces this to become a bottleneck. Also, I thought the getdata() request gets up to 500 blocks from the referenced hash...? Doesn't that mean, unless you are more than 500 block behind, you only need to find a good peer once, to get the updated blockchain?

For my OP_CHECKSIG code I tested using the following two transactions:

Tx2 uses an output from Tx1. I wish I could give credit to whoever gave this to me... but I can't remember....

Code:

tx1raw= '\x01\x00\x00\x00\x02\x0f{\x7f\xb8mL\xf6F\x05\x8eA\xd3\xb0\x07\x18?\xdfysn\xd1\x9b*th\xab\xc5\xbd\x04\xb1n\x91\x00\x00\x00\x00\x8cI0F\x02!\x00\xb2\xee9\xd2\xfc\xc2\xe5TJW\xc3\x0f{NI\xcf\xb8""fm\x03O\xb9\x0e"4\x8e\x17\xe2\x8e\x0f\x02!\x00\xdb\x91\xc3\x19\x9c\xc7\xb4\x1dMz\xfc\xe0\xcc\xb4\xce\xb4$\xb9GmQ\xc0aBX=\xafS\xce\n\x9bf\x01A\x04\xc3"\x15\xa9\t0\x11\xbd<A(:\xce=\x00,f`w\xb2J`[<\xfc\x8fq\x01\x9a\x0fC\xdff\xf3\x89\xf3\xd9\xa6!\x88\xa4\x94\xb8i\xdc~_\x9d\xff\xc9\x8av\xd3\x08\x8a!\xe9\xb78\xec\x9e\xba\x98\xcb\xff\xff\xff\xff\x97\x00A%R\x8f{^\xd34e\xca\xaa\xe0!\xc0\xb8\x15\xf3\xe6\xa3pvA\xd5\xa0\xbc\xa4?\xc1II\x01\x00\x00\x00\x8aG0D\x02 3\xd0,.\x89o\x1a\x12RH\x8dSL\xfb\x08\xab\xf3\xe7\xea\x90\xab\xa7\xbaoW\xab\xf1\x89\xce\xf1\xd87\x02 \x05f\x8duP\x13\xb0\xe5\x9a*\xf5\x14_\x10\xef\xe6.\xa7\x16\xd33&\x8b\x0bZ>\xfb\xd8-\x149\xbe\x01A\x04\xc3"\x15\xa9\t0\x11\xbd<A(:\xce=\x00,f`w\xb2J`[<\xfc\x8fq\x01\x9a\x0fC\xdff\xf3\x89\xf3\xd9\xa6!\x88\xa4\x94\xb8i\xdc~_\x9d\xff\xc9\x8av\xd3\x08\x8a!\xe9\xb78\xec\x9e\xba\x98\xcb\xff\xff\xff\xff\x01\x00\xc2\xeb\x0b\x00\x00\x00\x00\x19v\xa9\x14\x02\xbfK(\x89\xc6\xad\xa8\x19\x0c%.p\xbd\xe1\xa1\x90\x9f\x96\x17\x88\xac\x00\x00\x00\x00'
tx2raw= "\x01\x00\x00\x00\x030\xf3p\x1f\x9b\xc4dU/pIW\x91\x04\x08\x17\xcewz\xd5\xed\xe1nR\x9f\xcd\x0c\x0e\x94\x91V\x94\x00\x00\x00\x00\x8cI0F\x02!\x00\xf5tk\x0b%OZ7\xe7RQE\x9cz#\xb6\xdf\xcb\x86\x8a\xc7F~\xdd\x9ao\xdd\x1d\x96\x98q\xbe\x02!\x00\x88\x94\x8a\xea)\xb6\x91a\xca4\x1cI\xc0&\x86\xa8\x1d\x8c\xbbs\x94\x0f\x91\x7f\xa0\xedqThm>[\x01A\x04G\xd4\x90V\x1f9l\x8a\x9e\xfc\x14Hk\xc1\x98\x88K\xa1\x83y\xbc\xac.\x0b\xe2\xd8RQ4\xabt/0\x1a\x9a\xca6`n])\xaa#\x8a\x9e)\x93\x001PB=\xf6\x92Ecd-J\xfe\x9b\xf4\xfe(\xff\xff\xff\xffr\x14+\xf7hl\xe9,m\xe5\xb73e\xbf\xb9\xd5\x9b\xb6\x0c,\x80\x98-YX\xc1\xe6\xa3\xb0\x8e\xa6\x89\x00\x00\x00\x00JI0F\x02!\x00\xbc\xe4:\xd3\xac\xbcy\xb0$~T\xc8\xc9\x1e\xac\x1c\xf9\x03u\x05\x00\x0e\x01\xd1\xfd\x81\x18T\xd8[\xc2\x1a\x02!\x00\x99*oo/\xebob\xd3po;\x9a\xaa\xb8\x8d\x9f\x112\x95j\x1d\xff\xa9&\xcdUn\xd5S`\xdf\x01\xff\xff\xff\xff\xd2\x81(\xbb\xb6 |\x1c=\nc\x0c\xc6\x19\xdc~{\xeaV\xac\x19\xa1\xda\xb1'\xc6,x\xfa\x1bc,\x00\x00\x00\x00IH0E\x02  \x97W6\x81aSw\x08\xfd)\xd8\x9b\xb1\xe9\xd6H\x00yI\xec\xfd\xedx\x9bQ\xa9c$\xcbe\x18\x02!\x00\xcd\x0f|0!9\x16H+n\x16m\x8aO+\x98\x1fw~\xb1\x84\xcd\x8aI_\x1b=6\x90\xfb\xbf-\x01\xff\xff\xff\xff\x01\x00\xa6\xf7_\x02\x00\x00\x00\x19v\xa9\x14\x9e5\xd9<w\x92\xbd\xca\xadV\x97\xdd\xeb\xf0CS\xd9\xa5\xe1\x96\x88\xac\x00\x00\x00\x00"

Is this only for a single private key? Can I supply a file with a list of private keys instead? I started to try to figure out the mechanism by which the code tracks the keys and writes them to the wallet file, but so far I've been unsuccessful. I also didn't try very hard...

I agree this needs to be improved, and I agree there should be a splash screen or a disabled GUI while things load. It would seem that you could store temporary information from the last time the client was closed, to display at startup but gray-out the send-money button. Then the user can browse the previous transaction data, create new addresses, etc, while the backend goes through and re-verifies the blockchain. You could even put the progress bar under the "send" button to show the user why it's grayed out. This should solve a lot of the current speed problems if the multithreading can be done correctly (don't ask me to do it, I've never written MT code).

But, I've been thinking a lot about it how to optimize the file-reading and I'm working a prototype right now. Unfortunately, it is without much context about what is already there in the client code, but this is fun for me so I'm working on the ultimate-efficiency blockchain file-loading. I'd like to hear any thoughts on this process...

By my calculation, we have about 400 MB of blockchain information, which for most HDDs should take about 5 seconds to read. That means, at least while the client is designed to maintain the entire blockchain, we'll never beat 5 seconds load time for the average user. This may not be too bad right now, but it's not going to get any better. And of course, it takes a lot longer than that because the blockchain has to be processed/verified as it is read in. My thought is... what if it doesn't take any extra time?

(1) The blockchain data is stored in the file in exactly the same binary format as would be stored in RAM.
(2) In memory, allocate 400 MB of storage behind a raw uint8_t pointer
(3) Do a direct copy "blk0001file.read(ramChunkPtr, fileSize)"
(4) Use BlkHeaderPtr and TxPtr objects -- the members are pointers to locations in this chunk of memory instead of copies
(5) Write the accessor methods to simply dereference the pointers, any time

The overhead of storing these pointers should be small compared to the blockchain itself. And there's two options for assigning the pointers... either walk through the data and assign them manually (which would be slowed slightly be all the var_ints), or store the variable byte-locations in another file, (known from the last client shutdown). Hell, you could do the exact same memory chunk idea with this file, but even faster because they are all constant-width fields (and dramatically less data). Either way, this should be about the best time-efficiency possible, paid for with a slight reduction in space efficiency.

I have produced a few hour-long seminars at work, with the goal of adding extremely detailed visualizations to describe the mechanisms of cryptography. So, for me, these visualizations were quite easy as I have become extremely efficient with Inkscape. The hard part was understanding the algorithms to begin with! Once I finally got my OP_CHECKSIG code working, I knew that I could leverage my Inkscape skills to benefit others (and myself, as these images are great for my own reference).

If you feel that there are more algorithms/concepts that could benefit from such visualization, I will be happy to produce them, as long as I understand it. Perhaps others could help by injecting these images into the wikis at appropriate places. Of course, if you believe there is an error in one of them, please let me know.

On that note, I would like to make a few variants of the OP_CHECKSIG diagram, one for each hashtype. Unfortunately, I don't fully understand them myself. I need SIGHASH_NONE, SIGHASH_SINGLE, SIGHASH_ANYONECANPAY. I have read the descriptions, but still don't understand them well enough to complete the visualization. Perhaps someone that does understand can explain them in the context of the my diagram for SIGHASH_ALL.

-Eto