Bitcoin Forum
December 03, 2016, 04:48:39 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 [2]  All
  Print  
Author Topic: Very few normal people would wait days for the blockchain to download.  (Read 4742 times)
Jan
Legendary
*
Offline Offline

Activity: 1042



View Profile
October 20, 2011, 05:32:16 PM
 #21

unfortunately java is not my favourite language.
I guess I will try to do it from scratch in python, using bitcoin-abe + bitcoind at the server side.

Good luck with your venture.

Mycelium let's you hold your private keys private.
1480740519
Hero Member
*
Offline Offline

Posts: 1480740519

View Profile Personal Message (Offline)

Ignore
1480740519
Reply with quote  #2

1480740519
Report to moderator
1480740519
Hero Member
*
Offline Offline

Posts: 1480740519

View Profile Personal Message (Offline)

Ignore
1480740519
Reply with quote  #2

1480740519
Report to moderator
1480740519
Hero Member
*
Offline Offline

Posts: 1480740519

View Profile Personal Message (Offline)

Ignore
1480740519
Reply with quote  #2

1480740519
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
October 20, 2011, 06:21:13 PM
 #22

I may not understand what's under discussion here, but if we're talking about new users and lightweight clients, what's wrong with downloading only the headers (which should take less than 5 minutes), and then only download the parts of the blockchain that have timestamps after the creation date of each key in the wallet?

A new user who doesn't have any keys, will be creating those keys right now, and actually doesn't need any blocks that were created before his own computer just generated them a minute ago.  He only needs the full set of blockheaders to figure out the longest chain and determine "truth" for when he does need to get blocks.  He stores the parts that are relevant to himself, and will always have a full list of available txouts, without any need to trust anyone else. 

Sure, you can't verify other users' txs easily, unless you see the tx in the blockchain with X confirmations.  This may make some people uncomfortable, but I believe the future will eventually require people to trust the longest chain (and all the Tx's in it) since it will eventually be infeasible for people to store the entire blockchain themselves.

Btw, you mentioned python:  check out my codebase, PyBtcEngine.  Right now, the full suite uses the full blockchain, but I do plan to make a lightweight version of it.  There's no networking yet, but it does handle just about everything else (the last thing I need is knapsack optimization to create a set of txOuts to send to my ECDSA signature code).  You might find the python code alone to be useful without any of the C++/SWIG, you just won't have access to the entire blockchain without the C++ (I found it way too slow to juggle the full chain in python).


Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
kano
Legendary
*
Offline Offline

Activity: 1918


Linux since 1997 RedHat 4


View Profile
October 21, 2011, 12:07:52 AM
 #23

But then how can you accept a payment from a previous transaction?
Each payment has input that is the output of an earlier transaction.
Thus you would need the blockchain back to the earlier transaction to be able to receive it ...

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
October 21, 2011, 12:59:19 AM
 #24

But then how can you accept a payment from a previous transaction?
Each payment has input that is the output of an earlier transaction.
Thus you would need the blockchain back to the earlier transaction to be able to receive it ...

This depends entirely on your use-case.  The fact that miners included it in the blockchain is evidence enough that it's valid, as long as it is sufficiently deep.  The miners are checking the validity for you, and the tx would be DOA on broadcast it if it wasn't.  The fact that the world is still building off the blockchain that includes that transaction confirms that it's a legit tx... as long as you don't mind waiting for a few confirmations. 

Of course, you can't "verify" transactions that have no confirmations yet, because (as you said) you don't have the ability to check the inputs.  But even if you did have that information, there's plenty of reasons not to accept 0- or 1-confirmation transactions anyway (unless they are small and you don't mind eating a couple invalids).  But for most users, transferring money between family/friends, or online customer-to-merchant, you will be waiting for 2+ confirmations regardless of whether you have the entire blockchain -- so why waste the hard-drive space? 

Alternatively, when you receive the Tx, you can request the input transactions/merkle trees from your peers, and verify them against the headers in your longest chain (I'm assuming you're at least holding the headers).  Then you can at least know the tx is potentially valid with 0 confirmations, but again, if it's a significant amount of money, you best wait for 2+ anyway.

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
MoonShadow
Legendary
*
Offline Offline

Activity: 1666



View Profile
October 21, 2011, 01:03:18 AM
 #25

I may not understand what's under discussion here, but if we're talking about new users and lightweight clients, what's wrong with downloading only the headers (which should take less than 5 minutes), and then only download the parts of the blockchain that have timestamps after the creation date of each key in the wallet?


Nothing at all.  This is actually on of the 'lightweight' client models that have been proposed.

"The powers of financial capitalism had another far-reaching aim, nothing less than to create a world system of financial control in private hands able to dominate the political system of each country and the economy of the world as a whole. This system was to be controlled in a feudalist fashion by the central banks of the world acting in concert, by secret agreements arrived at in frequent meetings and conferences. The apex of the systems was to be the Bank for International Settlements in Basel, Switzerland, a private bank owned and controlled by the world's central banks which were themselves private corporations. Each central bank...sought to dominate its government by its ability to control Treasury loans, to manipulate foreign exchanges, to influence the level of economic activity in the country, and to influence cooperative politicians by subsequent economic rewards in the business world."

- Carroll Quigley, CFR member, mentor to Bill Clinton, from 'Tragedy And Hope'
MatthewLM
Legendary
*
Offline Offline

Activity: 1092



View Profile WWW
October 21, 2011, 01:43:02 AM
 #26

Can compression be used on the blocks when transferring? Could they be included in the software packages? Could they be provided online for fast server downloads?

Bitcoin Extra Wallet | Peercoin Android Wallet
BTC: 1D5A1q5d192j5gYuWiP3CSE5fcaaZxe6E9  PPC: PH7fVn1Xs7nkUFmdwCX2ZRYfLPCSwGxAq9
MoonShadow
Legendary
*
Offline Offline

Activity: 1666



View Profile
October 21, 2011, 04:01:58 AM
 #27

Can compression be used on the blocks when transferring?


Yes, but the gain would be small.

Quote

Could they be included in the software packages?


 Could they be provided online for fast server downloads?

Yes to both, but then it becomes a trust issue.  At least on some level.

"The powers of financial capitalism had another far-reaching aim, nothing less than to create a world system of financial control in private hands able to dominate the political system of each country and the economy of the world as a whole. This system was to be controlled in a feudalist fashion by the central banks of the world acting in concert, by secret agreements arrived at in frequent meetings and conferences. The apex of the systems was to be the Bank for International Settlements in Basel, Switzerland, a private bank owned and controlled by the world's central banks which were themselves private corporations. Each central bank...sought to dominate its government by its ability to control Treasury loans, to manipulate foreign exchanges, to influence the level of economic activity in the country, and to influence cooperative politicians by subsequent economic rewards in the business world."

- Carroll Quigley, CFR member, mentor to Bill Clinton, from 'Tragedy And Hope'
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
October 21, 2011, 04:06:46 AM
 #28

Can compression be used on the blocks when transferring?


Yes, but the gain would be small.

Quote

Could they be included in the software packages?


 Could they be provided online for fast server downloads?

Yes to both, but then it becomes a trust issue.  At least on some level.

You can download the headers in less than 5 minutes.  This gives you a definitive map of the data you should receive to fill in the rest.  Therefore, if the client only uses the network to pick out 15 MB of blockheaders with the longest chain/work, then he can get the other 600 MB from anywhere and be confident he's getting the right data.  This doesn't really require any more trust than downloading the data normally.

And compression wouldn't achieve too much:  most of the blockchain is hashes which are, by design, supposed to "random" sequences of bits.  Random data is not very compressible (in fact, you can use compression algorithms to test your encryption/hashing algorithms:  if the output will compress more than 5%-10%, then it's not sufficiently random).

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
Jan
Legendary
*
Offline Offline

Activity: 1042



View Profile
October 21, 2011, 05:04:26 AM
 #29

I may not understand what's under discussion here, but if we're talking about new users and lightweight clients, what's wrong with downloading only the headers (which should take less than 5 minutes), and then only download the parts of the blockchain that have timestamps after the creation date of each key in the wallet?

ThomasV and I are discussing super-light-weight smartphone'ish clients with limited bandwidth that do not need to download/store the block chan at all. They just store the private keys.

Mycelium let's you hold your private keys private.
kano
Legendary
*
Offline Offline

Activity: 1918


Linux since 1997 RedHat 4


View Profile
October 21, 2011, 06:16:35 AM
 #30

...
Could they be provided online for fast server downloads?
Yeah I mentioned that on the previous page.
A download of the block chain (independent of the client) with a standard accepted method to get the hashes from the actual chain would easily ensure it's OK also.
The only issue is the client and well that of course (as I have said) should never be included with a block chain download.

Pool: https://kano.is BTC: 1KanoiBupPiZfkwqB7rfLXAzPnoTshAVmb
CKPool and CGMiner developer, IRC FreeNode #ckpool and #cgminer kanoi
Help keep Bitcoin secure by mining on pools with Stratum, the best protocol to mine Bitcoins with ASIC hardware
finway
Hero Member
*****
Offline Offline

Activity: 714


View Profile
October 21, 2011, 06:34:29 AM
 #31

Yes, and most people close there client, instead, use mtgox.

Jan
Legendary
*
Offline Offline

Activity: 1042



View Profile
October 21, 2011, 06:45:34 AM
 #32

Yes, and most people close there client, instead, use mtgox.

If you want normal people to participate in Bitcoin and you expect them to have a client running at all times that takes up gigabytes of storage/bandwidth and accepts incoming connections, then you are going to utterly fail. We need secure light-weight clients if we want to hit the masses, not bitcoin banks that run away with your money.

Mycelium let's you hold your private keys private.
ShadowOfHarbringer
Legendary
*
Offline Offline

Activity: 1470


Bringing Legendary Har® to you since 1952


View Profile
October 21, 2011, 12:23:51 PM
 #33

Wait, isn't somebody already working on a "Bootstrap" feature ?

Downloading the full blockchain from torrent sites and processing them using client would be definately the fastest option. Torrents also have built-in checksums, so they seem to have enough security.

DiThi
Full Member
***
Offline Offline

Activity: 157

Firstbits: 1dithi


View Profile
October 23, 2011, 10:10:16 PM
 #34

Does anybody thought of this before?

- A method for saving a chain of blocks (or its headers) up to a given point, in a single file/stream. No matter where this file is made, it should be always the same, provided it's old enough (6 hours?). I guess this is like getblocks but unlimited and done locally. (edit: and with a canonical reorganization or whatever it's needed to make sure it's equal everywhere)
- A command for querying the hash of this file.

This way you can download a file from anywhere no matter how untrusty it is, then query a bunch of random peers for the hash that this file should have, in the same way you get and verify the authenticity of individual blocks.

1DiThiTXZpNmmoGF2dTfSku3EWGsWHCjwt
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
October 23, 2011, 10:33:39 PM
 #35

Does anybody thought of this before?

- A method for saving a chain of blocks (or its headers) up to a given point, in a single file/stream. No matter where this file is made, it should be always the same, provided it's old enough (6 hours?). I guess this is like getblocks but unlimited and done locally.
- A command for querying the hash of this file.

This way you can download a file from anywhere no matter how untrusty it is, then query a bunch of random peers for the hash that this file should have, in the same way you get and verify the authenticity of individual blocks.

This is mostly unnecessary.  It's extraordinarily difficult for someone to give you fake blockheaders, because they would need an extraordinary amount of computing power to give you fake headers that match the difficulty of the block (hash has enough leading zero-bits). i.e. proof-of-work.  You get the headers from a few different peers, and you can verify the leading zeros and accumulate all the difficulty values to get the longest chain.  Unless the attacker has more than 50% of the global computation in his control, he won't be able to feed you a chain of headers longer than the "actual, legit" blockchain headers.

I say mostly unnecessary because technically, if the attacker has a lot of computing power and luck, he might be able to feed you a blockheader list with 1 or 2 fake blockheaders at the top even with less than 50% global computation speed.  But unless he has 51%+, the "actual, legit" blockchain will be extended to be longer within few blocks, and your client will correct itself within an hour. 

tl;dr : It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).  I'm sure someone will flame me for this statement...

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
October 24, 2011, 06:20:12 PM
 #36

It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).
True, with a caveat.  Your untrusted block chain must have "pretty good" total difficulty, and you must acquire a recent estimate of "pretty good" from somewhere.

A lightweight client could produce a graph like http://bitcoin.sipa.be/speed-ever.png and let you compare it to a version of the graph obtained from a trusted source.  Perhaps newspapers will publish the graph, or you could go with your recollection of its shape.  Maybe various organisations will sign and publish statistics, and your lightweight client can ship with their public keys, fetch the signed messages, and tell you who agrees with your data as of N hours ago.  If your untrusted chain's implied hash rate starts to underperform the "real" one at some point in the past, you can be sure you have fake blocks from that point on.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
MoonShadow
Legendary
*
Offline Offline

Activity: 1666



View Profile
October 24, 2011, 07:43:37 PM
 #37

It's so difficult to produce fake blockheaders, that, unless you are handling huge amounts of BTC, it is perfectly fine to trust any headers you receive on a lightweight node (as long as its hash has the leading zeros).
True, with a caveat.  Your untrusted block chain must have "pretty good" total difficulty, and you must acquire a recent estimate of "pretty good" from somewhere.

A lightweight client that only uses block headers would simply have to choose three different sources at random, download the block headers from all three sources, and check them against each other to make certain that they agree.  If they don't dump all data collected from that set of three and start over with another set.  Change your set of three every couple thousand blocks, and you're pretty well protected.  This is similar to what the full client does when accepting a new block.

"The powers of financial capitalism had another far-reaching aim, nothing less than to create a world system of financial control in private hands able to dominate the political system of each country and the economy of the world as a whole. This system was to be controlled in a feudalist fashion by the central banks of the world acting in concert, by secret agreements arrived at in frequent meetings and conferences. The apex of the systems was to be the Bank for International Settlements in Basel, Switzerland, a private bank owned and controlled by the world's central banks which were themselves private corporations. Each central bank...sought to dominate its government by its ability to control Treasury loans, to manipulate foreign exchanges, to influence the level of economic activity in the country, and to influence cooperative politicians by subsequent economic rewards in the business world."

- Carroll Quigley, CFR member, mentor to Bill Clinton, from 'Tragedy And Hope'
etotheipi
Legendary
*
expert
Offline Offline

Activity: 1428


Core Armory Developer


View Profile WWW
October 24, 2011, 08:06:53 PM
 #38

Agreed.  I shouldn't have been so hasty to say "accept any blockheaders."  You should download it from a couple different peers and as long as any one of your peers is honest, you'll be able to receive and quickly determine the longest chain.  This is, by definition, the "correct" chain.    Any longer chain that is invalid will soon be outpaced by the correct chain and any one honest peer will set you straight.

Thus, I would argue if someone shows you a tx, you can "quickly" download the entire blockheaders from the network, get the merkle tree with that tx-hash in it, and confirm it matches a blockheader more than 6 blocks deep in the header list.  I would trust that transaction.

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
finway
Hero Member
*****
Offline Offline

Activity: 714


View Profile
October 25, 2011, 12:46:11 AM
 #39

The new QT GUI (in git HEAD) has a nifty block-chain-download-progress indicator.

I'd like to pull together a version 0.5 release candidate and start testing it early next week.

Maybe the big feature for version 0.6 can be fast initial download (I'm thinking the best thing to do for brand-new, starting-with-an-empty-wallet installations is to download only block headers, function as a 'lightweight' client, and 'backfill' full blocks until you're caught up with the full chain-- then function as a 'full' client).


High that feature's priority, downloading is annoying.

ThomasV
Legendary
*
Offline Offline

Activity: 1722



View Profile WWW
November 05, 2011, 10:10:47 AM
 #40

unfortunately java is not my favourite language.
I guess I will try to do it from scratch in python, using bitcoin-abe + bitcoind at the server side.

Good luck with your venture.

alpha version is ready.
see the announcement here: https://bitcointalk.org/index.php?topic=50936.0

Electrum: the convenience of a web wallet, without the risks
Pages: « 1 [2]  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!