Bitcoin Forum

I used to post some constructive threads (https://bitcointalk.org/index.php?topic=287699.0) here, but was fucked off ))

I am a merchant. And bitcoind becomes really asspain. It is usable, but as time comes, it becomes not.

So devs, Bitcoin Foundation or whoever there... Please...

Do something with blockchain size

I mean size and performance of ~/.bitcoin folder. Decentralize, shrink or what we may do with it? We near to times when not every mortal will have ability to accept vanilla Bitcoin payments.

I don't need to hear about Bitpay, Coinbase, electrum or whatever 3rd party service. If I ever want to use middleman - I will sure use bank API.

Thank you!

Can you explain better what limitations you're facing. In other words, "Please explain the problem you're having / trying to solve, and not what you think the solution is".

The current blockchain size is a fraction of a modern commercial _game_ and it stores the whole bitcoin economy. Better understanding the nature and origins of your limitations you're facing avoids thinking we have a solution for you which isn't a solution.

Please also be sure to explain what you expect to do with a node. For example, if you need to be able to add new keys at will and go find their transactions ... there is no way thats going to happen with reduced storage.

Problems:
- Enter to (pure) bitcoin game without super-duper hardware (initializing a client)
- Maintenance. ~/.bitcoind size is not limited in no way. There is no guarantee you run out of space/iops/cpu

I proposed some kind of decentralization of consumed resources. To move some load from disk/cpu into network (DHT).

Crunch. Some time after i will sure need to rent super node at AWS for that with this dynamic. This makes bitcoin unusable for mortals (in future)

AWS nodes, especially smaller ones have horrific IO that is slower than a 4200 rpm laptop drive.  You may be better off looking for other options.

Reducing the blockchain storage will not change the ongoing IO load which you seem to be expressing problems with.  At least from what you're telling me here the size is actually not a problem for you. Am I misunderstanding?

You can't shrink the blockchain size. Its idiotic of the devs to force everybody to download the entire thing as part of Bitcoin QT, it makes it sound like if you're not downloading it all you're somehow using a 3rd party service to store your coins. No, you're not. 99.99% of people will never need the whole chain and there's no point to having it. Just use electrum; its not a 3rd party API, its a superior client.

*facepalm* someone needs to read why the bitcoin-qt client needs the entire blockchain. This allows it to be trustless, that means it relies on nothing and can validate transactions and blocks. Electrum uses a 3rd party server, which if you connect to a cancer server, then you can be fooled to think you have bitcoins you really may not.

to electrum advices: I am merchant and need server software. However electrum servers will expirence same issue. AFAIK, they must store bitcoind's blockchain plus they store blockchain in database back end. So they expirence doubled issue.

I told about AWS just for example. gmaxwell, why don't try to distribute blockchain around all nodes, so nodes will have an option to store full chain or only its part? Just forget about my ask about transaction id lookup in prev topic.

I see it with bitcoind option - how much max full blocks to store locally. I sure if everyone will store even 10k blocks - it will be fine for chain integrity. If node needs to obtain a block not being local - it queries network. Of course block headers should be stored in full locally to not let network give a wrong answer.

So SPV client like MultiBit is what you need

BTW, I am running a full bitcoind on a 80USD ARM baord: http://cubieboard.org/ without any problem. So are we facing a big problem? I would say not yet

Do you read me? SERVER SOFTWARE

How much time did you spent on initializing?

How much time will you spend on restore corrupted DB?

How much time will you spend for sync if your bitcoind will go down?

Initializing: about 36 hours. But remember it's only a ARM board, like your cell phone. It could be much faster if I copy the blockchain files from another full node to the board

Restore: It has been running for 14 days without any problem and is still keeping up with the latest block.


and........... how about my SPV suggestion?

So that's for your online casino? SPV mode should be good enough. I think even SatoshiDice is running on SPV mode with BitcoinJ

What is spv? No matter casino or online store - just merchant.

https://en.bitcoin.it/wiki/Thin_Client_Security#Header-Only_Clients

What would you think you need a bitcoin "server". What functions do you actually need?

Regular merchant functions:
- Create address
- Get notified about incoming TX
- Get TX info, number of confirmations
- Send coins
- Sendmany

BitcoinJ should do everything you need, keeping only the relevant part of blockchain: https://code.google.com/p/bitcoinj/

edit: we have a whole BitcoinJ forum at https://bitcointalk.org/index.php?board=138.0

Except only recent blocks matter for this to be a problem. Sure, you can be "tricked" into thinking that 1000 blocks ago you got a transaction, except nobody really cares as everybody's going to be looking at recent blocks for recent payments which is what 99.99% of people are checking for anyway.

And also, yea, as someone else on this thread has already said, BitcoinJ is much better for OP's purposes than electrum. But both of them share the same advantage of not needing the whole blockchain.

Besides, this mini-PC uses flash disk right? The max flash card size is 32GB right? If so, you will not be able to run bitcoind there soon because out of disk space.

If you can't run bitcoind then yes, a bitcoinj based solution would work, although there isn't any program that's a drop-in compatible replacement that exports the same JSON-RPC API. It would be nice if there was.

But, you get weaker security. I am very skeptical that masterluc has a real problem here. He seems to be worried about hypothetical future problems he didn't actually encounter yet. I wouldn't encourage merchants to use SPV over bitcoind, it's always better to do the latter if you can afford it, and right now it's just not that expensive.

Yes, current problems are solvable. Yes I am worry about Bitcoin future. I am in business since 2011 and I see progress of problems. I see future where killer feature "no middleman" is not usable.

It really depends on which features you need - if you need "Get TX info, number of confirmations" only for recent blocks or would be OK with waiting a bit until your client has fetched the respective data via bloom filters, you just need to prune the block chain to operate on a set that is maybe a few 100 MB large instead of 20 GB+...

The solutions the OP suggests are actually not really usable without trust but he is free to sponsor implementation of a DHT block storage if he thinks that's what is going to solve bitcoind's storage problems.

All his requirements can be fulfilled by bitcoinj today and it requires only to ensure you are not under a sybil attack, which might be a bit harder to detect than on a bitcoin-qt node.

I thought there was already work going on to solve the blockchain size problem?

If its not a problem now, it sure will be if Bitcoin overtook Paypal & the credit card companies.  Just imagine every financial traction in the world being logged on my PC in real time forever.

If the issue is disk space, pruning (when implemented) will solve that problem for nodes that don't wish to support the network with archives.

If the issue is IOPs then it's unlikely we will see significant improvements, however, because of how LevelDB works you're not likely to see significant worsening either, even with large increases in traffic.

At the moment it doesn't seem like there's actually any problem. All good things come with time ...

It supports SATA and USB hard drive so this is not a concern at all

How much weaker (with SPV)? I think this is only weaker in academic sense, especially for low value transactions (relative to block reward)

It's weaker in the sense that it increases the possibility miners might try and change the rules of the system, like by changing the inflation schedule. We need as many people with economic heft (especially merchants) to validate all the rules, to make them harder to change.

It's weaker in a few other ways too, but the above is the main one.

It is trivial for a merchant to run several copies of bitcoind at different locations with cheap internet connection and storage (e.g. at home), and force his bitcoinj (on a server with very limited bandwidth and storage) to connect to the trusted bitcoind.

By the way, the one you mentioned is well known (but I think it's mostly an academic issue as many non-miner, such as early adopter, would have enough incentive to run a full node.) I would like to learn what are the other ways.

maybe some about news this?

A miner could mine a block that contains garbage transactions (like, transactions that spend non-existent money). If it was presented to an SPV client, that client would accept the nonsense tx as legitimate because it appeared in a block on the best chain. Of course, if you didn't keep up the attack, it would quickly re-org back onto the main chain (assuming it's visible) and the bogus tx would go pending forever. But if you're purchase already cleared, by then it may be too late.

Currently the cost of mining an invalid block and losing the reward is so high that you'd need to find a merchant selling something extremely valuable almost immediately, like an exchange, and I don't think any exchange would be stupid enough to not use full nodes. So this attack is somewhat theoretical. It's certainly possible though.

For more info, see here:

https://code.google.com/p/bitcoinj/wiki/SecurityModel

Hey Mike, quick question.

Other than the block reward not being validated, does bitcoinj also not validate if the block difficulty is correct in relation to the previous block? When reading the code, I kind of got that impression.

In other words, would it be possible to fool bitcoinj by sending it a difficulty 1 block?

It does check the difficulty, so you cannot just mine a diff-1 block. Please review the SecurityModel doc to learn exactly what it does or does not do.

Ah sorry, after digging around a bit more I see this check is done in AbstractBlockChain.java. I was initially looking at Block.java (where most of the other checks are done).

Returning after 1 year. How's chain size going (https://blockchain.info/en/charts/blocks-size?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address=)? Probably,  exponential.

Relevant: https://bitcointalk.org/index.php?topic=197810.5 (proposed solution from another nickname).

Even with fast SSDs in RAID1 to store it on, the blockchain costs less than $1 per month in storage media. That is a miniscule cost compared to the Internet connection or dedicated phone line that any business running customer bank transactions would use, or the cost of leasing a credit card terminal from a merchant services provider. One credit card chargeback avoided by using Bitcoin can pay for decades of blockchain storage.

A merchant using Bitcoin for point-of-sale will not have a copy of the blockchain at every cash register. The point-of-sale terminal only needs to show a pay-to Bitcoin address, and a message to the cashier that the payment has been accepted. Everything else will be done in the server room of corporate headquarters or at an outsourced payment processing company.

I don't mind about merchant anymore. I do mind about Bitcoin itself. I remember times when everyone was able to launch full node, even on raspberry. That was ONLY 1-2 years ago! How about 10 years prediction?

Now I see more and more posts where launching full node is kinda unbelievable heroic doing.

Ppl said earlier like you - not a problem. No, it's a problem!  Look at chain size growth in time - it's parabolic! Will we loose full chain eventually? Because nobody cares.

I still run a full node on a RPi, external hard drives are cheap. I expect they will get cheaper per GB/year in the future. I bought a 3 TB the other day for 130.- USD.

And then there are the 8TB and 10TB drives from WD (and others).   http://arstechnica.com/information-technology/2014/09/western-digital-readies-first-10tb-hard-drive-ships-new-8tb-drive/

6TB is about $250 and about 270 times the current block chain.  Even growing at 20 GB/year, that is a long time before it is full.  Even IF it grew at 100GB/year (~10 times last year) that is still many, many years to fill it.  And then there are the 8TB and 10TB drives and that is just what is available now.

You guys say that hard drives are cheap but you still have to scan the block chain at first. It takes a very long time to do but is the only trust less solution, isn't it?

As a merchant, the only way I see things going is to keep the block chain whatever its size becomes, and sync.

It would be good if there was a way to drop the used transactions from the requirement. Last time I checked, there was 27 gb of block data but less than 1 gb worth of active tx.

You could use bitcore as far as I understand it - let's you execute all commands on the chain without the need to download the chain.
Open source and fairly easy to use, shall meet your requirement.

Yes, today. But not at some point in the future. Please read about "UTXO commitments" in https://bitcoinfoundation.org/2014/10/a-scalability-roadmap/

Its exponential, e^x not x^2

It's good to know there is a plan out there to have ultra pruned tree verification. The hashes you mentioned are going to incorporated in Merkle trees? I thought that the Bloom filter mechanism was going to be extended to skip over used tx.

In any case, would it be compatible with deterministic wallets? Without the used txs, they will stop scanning for addresses too early.

A deterministic wallet is simply a way of generating addresses. Bitcoin classically generates new private keys using a true random number generator for each address. When a new address is needed to request a payment, it is generated at random and is not related to any other key. A deterministic wallet instead uses a pseudo-random number formula to create addresses, with only the initial seed state being a generated random number. This prevents the wallet backup from becoming obsolete, because any past or future address you created can be recreated from the seed backup.

Pruning is a technique for reducing blockchain disk space consumption, and is unrelated to your wallet addresses. We assume that the blockchain history has already been thoroughly validated by many miners after it is buried under many blocks and that every transaction in it was properly signed by the owner. Therefore, as long as the hash chain in the headers is correct, we don't need to verify the chain-of-ownership of every single bitcoin amount back to its original creation for our own security. This allows an individual client to remove the spent transactions from the blockchain, potentially freeing hard drive space, and even may enable a 'pruned network' p2p client to exchange a lightweight blockchain between peers.

The "plan out there" for pruning was put forth in the original Bitcoin paper: http://we.lovebitco.in/how-bitcoin-works/bitcoin-paper/#ch7 - it is just that nobody has written a complete client implementation that does this yet.

Thanks. If a node prunes spent transactions, a hierarchical wallet will not 'see' the empty transactions anymore and it will stop scanning after the gap limit is passed.

Spv could work, but youd want to be connected to a few nodes at a time to be confident. Just randomly hop around servers in the electrum server list. You can subscribe to callbacks for events from stratum servers if I recall correctly..

Be careful with assumptions like these.  All (reachable) peers are malicious peers if an attacker controls your network connection. (E.g. compromised your ISP or router, or they are your ISP)

Nodes might prune transactions but maintain a bloom filter of addresses they saw.

This is wrong. Electrum verifies all transactions in its history using SPV.

A bloom filter cannot do that. An invertible bloom filter might but it is not decided yet if it is going to used (cf Gavin's message).

Yes but it could be fooled by omitting some transactions. I switched to Electrum from Armory. I still think it's the best solution at the moment for someone who doesn't want to spend hours downloading the blockchain but it's not suitable for a merchant.

Back to topic,
In my opinion, there should be a client that lets you:
- use a bootstrap file to catchup to a recent checkpoint
- catchup the remaining part of the blockchain by downloading simultaneously from multiple peers (bitcoin core uses 1 peer)
- performs validation in parallel and progressively as the chain gets reconstructed
- prunes the spent transactions locally. It could also prune the blocks if it doesn't care about relaying the blockchain.

If we are allowed to change the protocol, let there be a way to transfer a verifiable list of spent transactions and real transactions. I think this is what Gavin talks about in his post but I need to see more detail to be sure.

false positives only extend the scan. All funds would be found.

BTW, an invertible bloom filter is not a bloom filter.

You seem to be pretty sure of what you say but I really don't see how it would work. The client sends the filter parameters and the server creates a filter per session. The server scans its blocks. If the spent tx are removed from the server, how would it know how to match the filter?
I'm ok being wrong but could you describe the actual process?

The server might prune but accumulate a bloom filter of addresses it ever saw.

I assume the scan knows the master public key. Addresses can be generated sequentially and checked against the bloom filter instead of the block chain until the bloom filter is negative for longer than the gap. Thereafter transactions can be fetched from pruned set by address.

You have it reversed. The client is the one sending the filter (with the filterload message) and the server is scanning the transactions. Of course, what you described could work but it is not the current protocol.

You assumed constraints I did not. The scan does not need to use the P2P protocol if it is about the wallet of the node
or if the client have extended ways to communicate with nodes it trusts to some level.