Thank you for the clarifications.


... and at this time my client would tell the client A to stop sending the blocks and send me bunch of others.


How about marking the stored block chain as good after the client properly exits and then at the startup look for this mark and do the check only if the mark is not there?


Good to know. But why it is then accessing the disk so much? I am using the latest client. True, the speed went waaaaaaaaaaay up from the 0.6.x I was using before but still it touches the disk quite a bit.


Well, maybe it might seem dubious in your eyes but you cannot be sure if there isn't a valid reason on my side for the request. It would be nice if I could control that stuff.


Good point. Thank you for pointing this out. One other question for me to ask would be "how about improving the leveldb so others can benefit from it as well?".


Good to know. Thank you. I did not do much statistics but


Maybe I might start liking the UTXO and stuff and disliking Windows instead (the client is running on Windows). I now remember that Windows is doing pretty crappy job at managing the disk and the files on it. Once I find the time for this, I will try to run the thing on (modern) Linux and see what happens.

Now I am starting to think that Windows is pretty crappy system for a task like this. I do know its disk cache sucks a lot. I also now remembered that recently I realized that the Windows's filesystem also sucks a lot (when compared with Linux filesystems) (have you ever tried to defragment NTFS and make sure that it really is defragmented? That is the thing I now remember doing recently and realizing it to be pretty impossible and concluding that NTFS sucks). Now I think that the leveldb guys and you, Bitcoin guys did an admirable job at forcing the stupid Windows to behave under such load (20+ GB of growing data + 0.5 GB of heavily updated data).

I think the problem here is not the size of the blockchain itself. The problem is how the blockchain is handled.

While using the client I found several problems. The first problem is that the client wastes bandwidth by downloading blocks that it already has. I suspect the P2P protocol does not have provision for a node telling "I already have block X, please send me blocks A, B and Z instead". I can see this problem in the debug.log file which is littered by "ERROR: ProcessBlock() : already have block X" where X runs from a certain number consecutively for several such messages then it jumps and again runs consecutively. I think this not only wastes the bandwidth but also the time because the other nodes spend time sending these useless blocks instead of blocks that make progress. This happens pretty frequently at my node which is connected behind a firewall.

The second problem is the "Reading the list of blocks" and "Validating blocks" actions which takes a lot of time. Well, my question is why the client needs to "read the list of blocks" and "validate the blocks" every time it starts up. Well, the "read the list of blocks" is not taking that much time but "validate the blocks" is 10 minute operation. You know, once the blocks are validated, why they need to be revalidated at every program startup ?

The third problem is that the client is "jumping over the data like goat over cemetery" while doing these two actions. This is MUCH SLOWER than reading the data in sequence. Why it needs to jump over the data so much? Maybe implement some caching?

The fourth problem is why the program splits the blockchain into 125 MB chunks? That is inefficient in Windows where opening and closing a file is pretty expensive operation. In my blockchain directory the first 10 GB are stored in 5 files (well, in fact 10 because I need to count the revXXXXX files) because they were downloaded by a 0.6.3 BETA client but the remaining 9 GB is spread over 75 files. Is there a way to reconfigure these storage parameters? And once I change them, is there a way to tell the client to repackage the blockchain so it is stored according to my wishes? I prefer "few large files" over "many small files" on Windows because "many small files" is inefficient.

A similar problem is with the "chainstate" data which is only 0.5 GB but is littered into 229 files. Well, that might not be your fault as I understand that these fileis actually belong to some sort of general purpose database which was recently replaced and actually is much faster now but I believe that this data could be handled more efficiently if it was in a single file (maybe developing a special purpose database?)

Also regarding the size of the blockchain, there are two things that should be done. The first thing is that the coinbase transaction can be as big as the miner wants (and some coinbase transactions weight few tens of KB, storing various stuff, see "Hidden Surprises in Bitcoin Blockchain" search on Google and especially this blog) so putting a limit to it for example 128 or even 64 bytes would be good (but the limit should not be too small because otherwise we could run into a bunch of blocks with no solution). And the second thing would be when storing the blockchain, extract the addresses and especially the public keys out of the block data, store them into some sort of index file and in the block data replace them with indices. That could reduce the size of the stored blockchain pretty significantly.

This rebuttal is pretty good! I like it

However, remember that you are THE BIG BAD GOVERNMENT. You can do pretty much anything.

(I don't believe a good government would wish to destroy such a wonderful thing as Bitcoin, this is why you are a BAD government)

So:

5. Seize the factory(ies) of the company(ies) making the ASICs ("We suspect you have hashish here, we are closing down this factory until further notice").
6. Keep the factory running (you don't want all the good people working in it to lose their jobs because you know there is no hashish) and deploy the resulting mining hardware as soon as possible while you are searching for the (nonexistent) hashish. Make sure you are building that mining megarig somewhere out of the way so nobody sees you (especially keep reporters away). The other 75% of miners is struggling as they are looking for another factory(ies) to support their growth while you are growing by leaps and bounds.
7. Continue doing 6 until you capture 75% of the total network hashrate or even more.
8. Returrn the seized ("borrowed") factories back to the owners along with a huge pile of Bitcoins ("we apologize for our mistake, we thought you had some hashish hidden here, took us 5+ years to search it all, please accept this pile of BTC and goodbye"). This helps to cover the tracks of your real intents with the seizure. Everyone will think this is the end of the story except you.
9. Continue growing the covertly established mining megarig as outlined in the scenario.

Additionally the seizure (pardon, "borrowing") would very likely send the cryptocurrency exchange rates into sky very quickly as somebody pointed out, making the step 8 here much easier to do. Especially if big press releases would be made at or around the event ("Government seized the factories of the largest SHA256 ASIC ")

And don't forget to also make big press releases when doing the step 8 to support more exchange rate rise 

Well, as I already said in a previous post, this is bound to fail because of the flexibility of the Bitcoin core developers so the end of the attack is going to look like this:

10. Once you got 99% of the hashrate and you put all your order so you won't get hit by the resulting chaos in Bitcoin economy.
11. Pull the plug.
12. Realize that the chaos dissipates in 2 days because the Bitcoin core developers issued a "critical update" killing your bloated hashrate.
13. Fix your megarig to accept the new rules and then push the plug back.
14. After a while try again by going to 11.
15. After several loops between 11 and 14 realize that the Bitcoin core developers will always fix that damn thing in 3 days or less so you're not getting anywhere.
16. Give up.

Or:

15. After several loops between 11 and 14 realize that the Bitcoin core developers implemented proof of stake and other nasty countermeasures in the Bitcoin protocol so you are hopelessly screwed.
16. Give up.

Ok, BIG BAD GOVERNMENT. Case closed. Maybe it is time to build a nice global Bitcoin address ownership register and require people to register their addresses there. You futile effort already produced that huge mining megarig so financing this should not be problem this time. The only problem is that you still want the Bitcoin network destruction and that is NOT going to happen so you won't get what you want ...

That is true if you want to increase the current rate by a factor of 100 in a couple of days or months. But if you look at my scenario and think about it a little, you can see that they don't need to hurry with the attack too much. They can start small and fund next stages of the expansion of their hashing capacity from the profits made by the previous stages. They could start with as little as 25% of the current hashing capacity (or, maybe even 10%). Given the current exchange rates that would give them %45000 per hour of income at the beginning which would all spend on more hashers and electricity. They could even repay the "loan" taken from the taxpayer's money well before they reach into 50% of the hashing capacity. And the more they conquer, the more income they will get from it.

You know, the point of the scenario is that nothing in the network is going to indicate that something bad is going to happen ... until they pull the plug. They could even recoup the price of the hardware (by selling the generated coins that are abundantly flowing out of their 99% of hashing power) they used to pull out the attack before they pull the plug.

I don't believe in "provable decentralized timestamps" and I did not expected to have these. I was thinking about some way how all nodes in the network could agree on "what time is it" with reasonable accuracy. The "reasonable accuracy" would be good enough to make near-real-time difficulty adjustments but it would be not sufficient for transaction ordering so you would still need mining. The near-real-time difficulty adjustment would remove the need for "emergency updates" killing the difficulty if 99% of the hash power would suddenly disappear into thin air.

Well, yes but you don't care about the expenses when you don't have to pay it out of your pocket (remember, it is the *government*. All they have to do is to yell loudly enough into the crowd that cryptocurrencies are only used by cyber criminals to evade punishment for their drug trafficking and child pornography and the crowd won't mind allowing them to use their taxes for taking it down).


Just today I was thinking about it (actually before reading your post) and I realized that it would be pretty stupid government to attempt to do it. Yes, they might kill Bitcoin as it is but the Bitcoin developers would simply sit down, figure out why it happened, designed a fix and deploy an emergency update that would change the protocol starting from block X (exactly as you say here). What an intelligent government would more likely to do would be to establish some service where people could register their Bitcoin addresses/wallets. The government does not need the private keys, it would just need to know "person with this ID owns this(ese) address(es)". Then those who are not interested in shady business practices would simply comply and continue as usual. This would allow them to focus investigation to those who don't want their ownership of an address becoming public.


This type of attack would most likely not work (modify the client to reject empty blocks when there are transactions to be commited), however a workable alternative would be spamming the network with bitdust, including this bitdust into their blocks. They could also include extraorbitant fees in those transactions to keep people from moving bitcoin with fees lower than this. However even this would most likely not work well.

The strategy I presented allows the attacker to keep himself hidden all the time until he decides to pull the plug. That would cause much more disruption of the network than acting "strange" from the very start of the attack.

They could also try to mine the coins and dump them to an unspendable address to reduce the money supply. However I believe now that it is pretty late for this kind of attack. They could only hope that the resulting reduction of the money supply would cause the money to be awkward to use (imagine how you are going to work with currency with its smallest unit valued at $1000 ... But that problem still can be fixed by starting to penalize coin hoarding by draining the coins out of addresses which show inbound but no outbound activity for too long (I saw some discussion on a more advanced version of cryptocurrency where there were proposals to reward maintenance of full nodes. The resulting cryptocurrency is more robust against malicious proofs of work. See proof of stake for more info.

So right now I can conclude that the scenario I proposed would work ... for a few hours or maybe days in the worst case. Then a patch would be issued, everything would return back to normal and the attacker would find itself wasting a lot of effort in vain. Sure, he would have pretty nice income stream now but that was not what he wanted .

Well, if that would be true then the difficulty adjustment would also not work. In order to be able to adjust the difficulty you need to know how long it took to generate the last 2016 blocks. So there *is* some knowledge of time. It is not exact, but it is there and it is sufficiently precise to make adjustments of difficulty that keep the block generation from getting runaway.

My idea is to use these ideas from that code that do the difficulty adjustments to implement the scenario described. Regarding the time source I was thinking about some sort of NTP-like time propagation protocol which the clients would use to track something called "current network time". A NTP service could be used by random nodes to obtain the current time but the real NTP queries should be very infrequent and made only by a few nodes, essentially the network should collectively keep track of time. The nodes shall poke throughout the network, asking random node about the time (and JUST only about the time) and then update their clock accordingly.

Well, this discussion was interesting but it died off about 2 years ago and nowhere I saw discussed the possibilities of Bitcoin destruction which I see for more than 2 years. I saw that almost since I first learned about Bitcoin but I didn't pay too much attention to these worries until about now. Essentially the Bitcoin protocol has a critical design flaw that can be exploited to efficiently kill it and I even suspect that it already started to happen. The design flaw is that "Bitcoin difficulty is adjusted every 2016 blocks so that a block is generated roughly every 10 minutes" (source: Description of difficulty on BitCoin wiki). To see why this is a critical vulnerability we need to look at the adversary and his goal.

The adversary is somebody who wants the Bitcoin dead and is willing to do anything to kill it. He has sufficiently deep pockets to accumulate any amount of haspower at a rate much larger than an average Joe could muster. And he is not afraid to empty his pockets just to see the Bitcoin shot down from the sky.

If you think nobody would wants that because "it makes little sense", "it would be more reasonable to just play nice and profit" etc, then I want to show you one such adversary: government.

Why the government would want to kill Bitcoin? For the same reason for which they destroyed eGold and Liberty Reserve. Now speculations exist that Bitcoin might be the next target and rumors exist that "hackers" switch from Liberty Reserve to Bitcoin. Moreover there are analyses about how "anonymous digital currency" is used for money laundering, for example from McAfee and Thomson Reuters, even National Drug Intelligence Center made an analysis on digital currency usage within organized crime circles. The government thus does not like electronic money systems providing anonymous transactions and anonymous accounts and Bitcoin can provide just that. Yes, they can ban Bitcoin, outlaw the mining hardware, prosecute those who operate exchanges, shut down "Bitcoin mixing services" (those are doing the actual money laundering by breaking the links between the source of the money and the current owner) but now with the coins seized from Silk Road 1.0 and Dread Pirate Roberts they have enough resources to destroy Bitcoin without even spending much of taxpayer's money. The attack would be just like this:

1. Buy enough ASIC minig hardware to cover about 25% of the current hash rate using tax payer's money. They have to "borrow" the money from the taxes rather than use the bitcoins directly because they don't want the rest of the world to know what they are doing.
2. Start mining. Use large count of addresses to receive the mined coins so nobody will notice. They even may choose to join one or more pools to further masquerade as "the little guy next door".
3. Use the mined profits and/or more taxpayer's money to purchase more mining hardware.
4. Repeating steps 2 and 3 until you get 99% of the hashing power under your control.
5. Convert as many of your BitCoins as possible to "legal tender".
6. Wait for the next difficulty change and mine a few blocks past it.
7. Switch all your mining equipment off.

During the steps 1-5 the adversary plays according to the BitCoin rules, even participating in the Bitcoin economy. No hacking, no searching for some obscure vulnerabilities somewhere, no overzealous exchanges/mixing services hunting, no seizure of mining equipment. Contrary to that, the adversary tries to hide as much as possible, pretending to be a sizeable bunch of normal people with sizeable bunch of mining equipment. The adversary can even confortably pull its own money off the BitCoin economy before it tanks. The blow of death is dealt at the very last step. To understand why it is the death blow, let's look what happens after the enemy executes the step 7.

So, just before the step 7 the adversary controls 99% of the hashrate and the difficulty is already adjusted to that state (because of the step 6). Now 99% of the hashrate vanishes literally overnight. But the difficulty stays the same because there next adjustment is some 2000 blocks away. Thus a block gets mined in 1000 minutes instead of 10 minutes because only 1% of the hashing power is now left. Transactions take over 16 hours to get into a block and it takes over 4 days to get the recommended 6 confirmations. This is much slower than cashing a check. The Bitcoin is essentially frozen.

The worst hit is dealt to the miners. This is because 120 confirmations are needed to be able to spend a mining reward. This translates into 80 days - almost 3 months. Many miners can't do any mining for so long without getting paid so they quit. This causes the hashrate to be further reduced, leading to even worse situation. Demand for mining hardware quickly abates as the public starts to see it increasingly profitable. People become increasingly disgusted and reluctant as the value of their coins plunders towards zero. Essentially the network will start a downwards spiral to its doom. The coins will become worthless because they cannot be moved at all and the classic fiat money suddenly becomes much more attractive.

And with 2000 blocks to go under these conditions it would take over 3 and a half of year before the network can recover. That is way too much of waiting. Before the next adjustment can happen, the once thriving Bitcoin is nothing but a frozen world at the periphery of Internet. And even if it would survive somehow, the adversary would simply again turn his multi-peta-byte-hashrate mining rig, mine another 2016 blocks and switch it off again, dumping another 3.5+ years of recovery to the poor remnants of Bitcoin economy.

The current design of the difficulty adjustment process is insecure because it trusts the mining nodes to actually want to do the mining and this security hole is exploited by the attack. The malicious "government" node described above is not interested in mining, it is exploiting a loophole in the mining protocol design to achieve its own malicious goal: destruction of the entire network.

I would propose a fix which would be something like "difficulty decay over time". The difficulty adjustment can be left just where it is, but it needs to be fixed a little bit. However the thing that would be (re)adjusted here (and recorded in the blocks) would be something called "baseline difficulty". The actual difficulty would be calculated like this:

1. Immediately after a block is found, the difficulty is "infinite" (a hash of 0 would be the only accepted).
2. In the first X minutes it would gradually decay to the baseline difficulty.
3. The next Y minutes it would stay at the baseline difficulty.
4. The next Z minutes it would gradually decay from the baseline difficulty all the way down to 1.
5. After (X+Y+Z) minutes the difficulty stays 1 until a new block is found.
6. Once a new block is found, the difficulty resets back to "infinity" and starts a new cycle from step 1.

My proposal for the numbers would be X=8, Y=4 and Z=8. So the first 8 minutes the difficulty decays from infinity to the baseline, the baseline stays the same for the next 4 minutes and then it is gradually reduced to 1 in the next 8 minutes. So if no block is found in the next 20 minutes, the difficulty would be reduced to 1.

The baseline difficulty readjustment also needs fixing. First it shall take into account much less than 2016 blocks, maybe only 120 or even 60 blocks. Secondly, it should calculate and account for the real difficulties used to find these blocks. And third, the readjustment shall happen at every block, not just every N-th one (where N is the count of blocks examined during difficulty adjustments).

This would protect the network from both, sudden spikes and sudden blackout in hashing power. A large increase of hashing power will increase the frequency of the blocks but only briefly because of first X minutes when the difficulty is very large (at the beginning it is MUCH larger than the baseline). A sudden blackout would reduce the block frequency to about half of the rate (one block every 20 minutes) due to the difficulty dropping to 1 after the time ellapses. And these transient changes would last only in few hours because on every block the baseline difficulty would get adapted to the new network conditions.

However I have no idea how to realize this fix. The problem here is that this is pretty disruptive change to the protocol and all the older clients would suddenly fail to synchronize to the chain with the new rules (the blocks generated with the lowered difficulty would be considered invalid by them). Essentially everyone would be forced to upgrade his client and "third party" clients would be rendered utterly obsolete. Maybe the best solution would be to prepare the fix in a branch marked as "use in emergency only" and pull it out once the "doomsday" scenario ensues.

I believe that this attack is well underway because of the recent exponential increase in the hashing power caused by the ASIC chips. I suspect that what is really happening here is large amounts of these ASIC chips purchased by some shady "goverment-like" adversary and used to push the difficulty to abnormal height as described in the steps 3 and 4 of the attack. I don't know how much time we have but it seems that not much.