Block chain size/storage and slow downloads for new users

[danielpbarron]

Why would you say it's of questionable quality and security? It has been verified by lots of people and the lead developer of Bitcoin. It would obviously be a good choice to use 0.9.3 as it is the most stable currently. You would just be wasting your time trying to make changes.

Uh.. so let me get this straight. 0.9.3 is the best version because its authors said so? Did you "verify" the code? I happen to know that real people have taken a look, and it's not pretty.

real people like who ?

http://therealbitcoin.org/mailman/listinfo/btc-dev


dont see any names .......

Scroll to the bottom of this document for the GPG key fingerprints. I'll go ahead and do your work for you:

• 1721 5D11 8B72 3950 7FAF ED98 B982 28A0 01AB FFC7 : asciilifeform
• 4F79 0794 2CA8 B89B 01E2 5A76 2AFA 1A9F D2D0 31DA : ben_vulpes
• 027A 8D7C 0FB8 A166 4372 0F40 7217 05A8 B71E ADAF : mod6
• 3DAE 5385 2AF4 7432 CE3C 9318 FF23 EAA3 D815 427F : PinkPosixPXE

See: signed code.

Or I suggest you to tell your REAL PEOPLE to create new Bitcoin.

Uh.. we're the ones who actually care about bitcoin; you're the ones trying to make a new one with your gavincoin hard fork nonsense. What do you think a hard fork is? It's a new chain! An altcoin! And it's already been proven that there's no such thing as cryptocurrencies.

[1715508744]

1715508744

[Muhammed Zakir]

Uh.. we're the ones who actually care about bitcoin; you're the ones trying to make a new one with your gavincoin hard fork nonsense. What do you think a hard fork is? It's a new chain! An altcoin! And it's already been proven that there's no such thing as cryptocurrencies.

I am not for 20 mb fork^[1] even theymos has told in a neutral way. But... what you told about Satoshi made me to say that but no offence.

[1] It needs more investigation and some improvements.

   -MZ

[danielpbarron]

Thank you for upgrading  

Yeah it's a lot easier to sync with the network when you don't bother getting all the data anymore. Kinda like how it's a lot easier to transport gold bars when you hollow them all out before-hand.

[Borisz]

I think I'll move my coins to blockchain.info, I cannot/ do not really want to run a core node consuming 30+ GB of HDD space.

[danielpbarron]

I think I'll move my coins to blockchain.info, I cannot/ do not really want to run a core node consuming 30+ GB of HDD space.

That is a terrible idea.

On their blog, Blockchain.info has disclosed that a routine update left them serving insecure code to customers using their wallet between 12:00 AM and 2:30 AM GMT today. All customers who used the Blockchain.info web wallet to interface to create wallets, generate addresses, or send transactions are reported to be affected. The problem given the scope appears to be that Blockchain.info was serving weak pseudo-random number generating software.

[Borisz]

I think I'll move my coins to blockchain.info, I cannot/ do not really want to run a core node consuming 30+ GB of HDD space.

That is a terrible idea.

On their blog, Blockchain.info has disclosed that a routine update left them serving insecure code to customers using their wallet between 12:00 AM and 2:30 AM GMT today. All customers who used the Blockchain.info web wallet to interface to create wallets, generate addresses, or send transactions are reported to be affected. The problem given the scope appears to be that Blockchain.info was serving weak pseudo-random number generating software.

Based on the description this doesn't affect me. I guess there are plenty of people who are happy with their service overall.

[danielpbarron]

I think I'll move my coins to blockchain.info, I cannot/ do not really want to run a core node consuming 30+ GB of HDD space.

That is a terrible idea.

On their blog, Blockchain.info has disclosed that a routine update left them serving insecure code to customers using their wallet between 12:00 AM and 2:30 AM GMT today. All customers who used the Blockchain.info web wallet to interface to create wallets, generate addresses, or send transactions are reported to be affected. The problem given the scope appears to be that Blockchain.info was serving weak pseudo-random number generating software.

Based on the description this doesn't affect me. I guess there are plenty of people who are happy with their service overall.

Of course it effects you, unless it's possible to keep your own copy of the javascript and never let them serve you another (a practice I doubt you were employing). Every time you open up that site in your browser you're hoping that someone doesn't give you malicious javascript. It doesn't have to be on their end; if someone is able to compromise your communications channel they can serve you any code they want. And by 'they' I mean USG.

[Borisz]

I think I'll move my coins to blockchain.info, I cannot/ do not really want to run a core node consuming 30+ GB of HDD space.

That is a terrible idea.

On their blog, Blockchain.info has disclosed that a routine update left them serving insecure code to customers using their wallet between 12:00 AM and 2:30 AM GMT today. All customers who used the Blockchain.info web wallet to interface to create wallets, generate addresses, or send transactions are reported to be affected. The problem given the scope appears to be that Blockchain.info was serving weak pseudo-random number generating software.

Based on the description this doesn't affect me. I guess there are plenty of people who are happy with their service overall.

Of course it effects you, unless it's possible to keep your own copy of the javascript and never let them serve you another (a practice I doubt you were employing). Every time you open up that site in your browser you're hoping that someone doesn't give you malicious javascript. It doesn't have to be on their end; if someone is able to compromise your communications channel they can serve you any code they want. And by 'they' I mean USG.

The incident linked happened once, for 2.5 hours.

Also based on this I should also hope that nobody has installed an undetectable malware on my computer that logs my password for my wallet.

If even cold wallets can be compromised where would my coins be safe anyway?

[danielpbarron]

I think I'll move my coins to blockchain.info, I cannot/ do not really want to run a core node consuming 30+ GB of HDD space.

That is a terrible idea.

On their blog, Blockchain.info has disclosed that a routine update left them serving insecure code to customers using their wallet between 12:00 AM and 2:30 AM GMT today. All customers who used the Blockchain.info web wallet to interface to create wallets, generate addresses, or send transactions are reported to be affected. The problem given the scope appears to be that Blockchain.info was serving weak pseudo-random number generating software.

Based on the description this doesn't affect me. I guess there are plenty of people who are happy with their service overall.

Of course it effects you, unless it's possible to keep your own copy of the javascript and never let them serve you another (a practice I doubt you were employing). Every time you open up that site in your browser you're hoping that someone doesn't give you malicious javascript. It doesn't have to be on their end; if someone is able to compromise your communications channel they can serve you any code they want. And by 'they' I mean USG.

The incident linked happened once, for 2.5 hours.

Also based on this I should also hope that nobody has installed an undetectable malware on my computer that logs my password for my wallet.

If even cold wallets can be compromised where would my coins be safe anyway?

And it could happen again..

Your coins would be safe offline -- as in, the private keys are never on a networked machine. I'm not sure how you skip from "javascript wallets aren't safe" to "cold wallets aren't safe." The point is that you re-download your wallet software every single time you open up blockchain.info. It's not a one-time thing.

[Borisz]

And it could happen again..

Your coins would be safe offline -- as in, the private keys are never on a networked machine.

I know that the private key is safe if it is offline, as long as given malware is not interfering. Which has probably similar chances as people trying to inject on my connection.

I'm not sure how you skip from "javascript wallets aren't safe" to "cold wallets aren't safe." The point is that you re-download your wallet software every single time you open up blockchain.info. It's not a one-time thing.

"If even cold wallets"...
I tried to imply here that just because your private key is on a computer - that was supposedly never connected to the internet- it doesn't mean that it is completely safe. Coupling directly to the previous statement.

I realize that online services are most likely not the best practice and I haven't used them myself as they were going down and popping up, but on my current system I cannot afford 30 something GB of missing space.

I might look into lightweight clients that do not download the entire chain, however I am not sure how secure those are. I will have to read up on that topic as well.

[danielpbarron]

I realize that online services are most likely not the best practice and I haven't used them myself as they were going down and popping up, but on my current system I cannot afford 30 something GB of missing space.

I might look into lightweight clients that do not download the entire chain, however I am not sure how secure those are. I will have to read up on that topic as well.

Oh, perhaps you should familiarize yourself with this. It's a $20 device that you can plug a used laptop hard-drive into, plug into your router, and forget about. These instructions will do for now, until we figure out how to get it working without the systemd nightmare. The thing isn't really meant to be your wallet, but you can probably figure out how to connect it to your thin client. You shouldn't rely on someone else's full node; you run a similar risk to relying on someone else's javascript.

[irfansial]

what is the bitcoin an which aresa is about to be handel rth orgidnistiom is post by bitcoin

[Borisz]

I realize that online services are most likely not the best practice and I haven't used them myself as they were going down and popping up, but on my current system I cannot afford 30 something GB of missing space.

I might look into lightweight clients that do not download the entire chain, however I am not sure how secure those are. I will have to read up on that topic as well.

Oh, perhaps you should familiarize yourself with this. It's a $20 device that you can plug a used laptop hard-drive into, plug into your router, and forget about. These instructions will do for now, until we figure out how to get it working without the systemd nightmare. The thing isn't really meant to be your wallet, but you can probably figure out how to connect it to your thin client. You shouldn't rely on someone else's full node; you run a similar risk to relying on someone else's javascript.

I know about the PogoPlug, wanted to use it for another project. I have a raspberry Pi and managed to get an altcoin client "partially" running (some node connections cut out, didn't have the time to properly troubleshoot) I think I could get bitcoin working with that. The question of storage still remains, 2.5" external HDD would do yes.

In this case I would be still concerned that I have to remotely connect to my wallet. Even though it is on my server. But I get it, it is a step better than someone else's service on the internet.

[Bernie87]

I realize that online services are most likely not the best practice and I haven't used them myself as they were going down and popping up, but on my current system I cannot afford 30 something GB of missing space.

I might look into lightweight clients that do not download the entire chain, however I am not sure how secure those are. I will have to read up on that topic as well.

Oh, perhaps you should familiarize yourself with this. It's a $20 device that you can plug a used laptop hard-drive into, plug into your router, and forget about. These instructions will do for now, until we figure out how to get it working without the systemd nightmare. The thing isn't really meant to be your wallet, but you can probably figure out how to connect it to your thin client. You shouldn't rely on someone else's full node; you run a similar risk to relying on someone else's javascript.

Interesting device (that actually costs $50, not $20, for the basic version), but the router still needs to download the whole blockchain and you need a spare disk, so that will not solve the download time problem. If demand is sufficient, a better solution would be flash memory/usb flash drive(s) sold with the initial blockchain already written on them, and a full node software update to allow it to directly read blocks from the flash media, in addition to the newer blocks on the hard disk. Flash media is ideal for that usage because it's cheap, has standard capacities similar to blockchain size, is efficient with random read-only access, and the blockchain copy can easily be updated from time to time with new blocks accumulated on the hard disk. That's also better for your hard disk health and availability.

[btcminer021]

Too many words... what's the TL;DR? My new install is taking DAYS to sync.

[2112]

Flash media is ideal for that usage because it's cheap, has standard capacities similar to blockchain size, is efficient with random read-only access, and the blockchain copy can easily be updated from time to time with new blocks accumulated on the hard disk. That's also better for your hard disk health and availability.

Unfortunately the very high write amplification caused by bitcoind will also kill any flash device in a few years.

There's quite bit of work that needs to be done to safely run Bitcoin on a flash device for extended time periods. I'm not aware of any open-source flash-specific database engines.

[danielpbarron]

If demand is sufficient, a better solution would be flash memory/usb flash drive(s) sold with the initial blockchain already written on them

That would defeat the purpose of bitcoin. What good is a blockchain that you didn't receive from the network through the lengthy process of consensus verification? You might as well just use a thin client.

[ranochigo]

Flash media is ideal for that usage because it's cheap, has standard capacities similar to blockchain size, is efficient with random read-only access, and the blockchain copy can easily be updated from time to time with new blocks accumulated on the hard disk. That's also better for your hard disk health and availability.

Unfortunately the very high write amplification caused by bitcoind will also kill any flash device in a few years.

There's quite bit of work that needs to be done to safely run Bitcoin on a flash device for extended time periods. I'm not aware of any open-source flash-specific database engines.

A normal thumbdrive would be able to last quite sometime even with high IO usage. A few years is already considered long and flash drives would be much cheaper then. A better idea is to use your old harddisk and hook it up to your computer. If you aren't in need of the features available only in Bitcoin Core/ other full chain client, don't use it. A normal SPV client is secure enough for daily usage.

[2112]

A normal thumbdrive would be able to last quite sometime even with high IO usage. A few years is already considered long and flash drives would be much cheaper then.

"A few years"? More like "a few months" or maybe even "a few weeks". A database with write-ahead-logging is a perfect example of pessimal application for flash storage: lots of small writes, forced buffer flushes to permanent storage and never read back (unless the database application crashed).

http://en.wikipedia.org/wiki/Write-ahead_logging

If "normal thumbdrive" means the typical cheap drive with controller and wear-leveling optimized for FAT32 file system then I wouldn't be surprised if the device died even before the full current blockchain synchronization was complete (when formatted using any modern file system).

[alishahid]

There have been quite a lot of threads lately with people complaining about the size of the block chain, specifically (1) how long it takes to download for new users and (2) the amount of disk space used, often combined with complaints that the "core dev team" isn't doing anything about it.

This is just a quick note to explain where we're up to with this.

• Starting from a few days ago, MultiBit is the default recommended desktop client on the bitcoin.org choose your wallet page. MultiBit is a what we call an "SPV wallet" so is capable of processing thousands of blocks per second, and its checkpoints are refreshed frequently enough that for brand new users, they will usually be synced with the chain in 5 seconds or less. I'll explain a bit more about how this works in a moment, as we have many newbies join us in recent months who may not be familiar with the details.
  
  
• At some point Bitcoin-Qt will change such that it's able to delete old blocks. The details are still being worked out, but most likely you'll be able to say "Use up to 10 GB of disk space" and it will never use more than that. Nodes will broadcast how much of the chain they have and are able to serve. New nodes that are starting from scratch will have to search out other nodes that still have the full chain and sync from them, but any node that just wasn't online for a while and needs to grab the latest parts of the chain will be able to use most of the others. By controlling disk space usage, you can also indirectly control bandwidth usage (you can't upload data you don't have).

The latter piece of work isn't done yet, basically because Pieter has been busy lately with other things (like: real life). He did the bulk of the work already last year, but some parts still need to be designed and written. Remember that nearly everyone taking part is still a volunteer except for Gavin.

Intro to SPV mode

OK, now that we're recommending MultiBit as the default wallet app for new users, what does this do? MultiBit is like the Android "Bitcoin Wallet" app by Andreas Schildbach. They're both based on the bitcoinj project that I run. Essentially, these clients download sub-parts of the block chain and then do a bunch of maths to verify that it all hangs together. Because it doesn't download the whole chain, an SPV wallet is light and fast. But because it does download and verify parts, an SPV wallet can talk to the regular P2P network because it doesn't really have to trust the remote server. This makes it more decentralised than something like Electrum or blockchain.info which relies on special servers.

How does this work? It's described in Satoshi's original white paper in the "simplified payment verification" section (hence, SPV). But here's a brief description to save you opening up your copy of his paper. Each block in the Bitcoin protocol has two parts, the header and a list of transactions. The header contains data linking the block to a place in the chain (like the hash of the previous block). A full Bitcoin node (Bitcoin-Qt/bitcoind) examines the block headers to figure out which chain of blocks has the most mining work done on it, and then verifies all the transactions in order in those blocks. The best chain determines the order the transactions are applied to the database and thus which transaction loses if there's a double spend. But (and this is crucial), the ordering is the only thing determined by miners. All the transactions still have to make sense. Miners don't have arbitrary power, if they mined a block that just magicked money out of nowhere or included bogus transactions, full nodes would all reject it.

In SPV mode things work differently. Because they don't download the full chain, they can't verify each transaction individually or build a copy of the database. Instead they verify the headers to find the best chain, and then assume the contents of the best chain must be correct. This is usually a valid assumption, because the majority of mining power is honest. However if there was to be a 51% attack then SPV wallets might display arbitrary nonsense for as long as the attack lasts. They would get back to reality once the good chain became longer (harder) than the bogus chain again.

This leads to the question of how SPV wallets find transactions that send them money, if they don't download the whole chain. The answer is they upload to the remote Bitcoin nodes a "filter", which that node applies to each transaction in the block. If the filter matches, the transaction is sent to the SPV wallet along with a mathematical proof that it was really in the chain (we call this proof a Merkle branch, after Ralph Merkle who invented them). The wallet verifies this proof and thus knows the transaction really was accepted by the majority of miners, without having to trust the server.

Because we're talking to random computers on the internet and not a trusted third party, the filter is designed to let you control your privacy. It is not a list of your addresses, as it is with the blockchain.info wallet. It's actually what we call a Bloom filter (named after Burton Howard Bloom who invented them in 1970). You can't directly get the users addresses back out of a Bloom filter, instead you have to test each one you find in the chain against it to see if it matches. Also, the filter can be made "noisy", which means it randomly matches some other addresses as well. When the Bitcoin P2P node you're downloading from finds a match, it doesn't know if it really found one of your transactions or if it was a false positive. And because there are so many P2P nodes, it's possible to split up your list of addresses and send a subset to lots of different peers, so none get an accurate idea of what's in your wallet (bitcoinj doesn't do that today though). By adjusting your false positive rate, you can decide how much bandwidth you want to spend on garbling the other nodes picture of your wallet. If you're on a very slow or expensive link you might decide you want no noise in your filter at all, if you're on a fast wifi connection, you might be OK with downloading a megabyte or two of other peoples transactions just to obscure which ones are yours.

Using these fancy mathematical tools MultiBit and the Android wallet app give us the same nice performance that we can get from a web wallet like blockchain.info or Coinbase, but without the need for any central servers and keeping Bitcoin's P2P nature intact. SPV wallets will always be fast no matter how popular Bitcoin gets. Together with being able to delete old blocks, these are our solutions to the ever-growing size of the chain - which has been Satoshi's plan since the very first day Bitcoin was announced.

I hope it's all clearer now and everyone understands what's going on.

thanks