Bitcoin Forum

I know a future market where people put money where their mouth is would be a better indicator, but for those who would like to speculate without actually risking to lose anything: here you go, you can bet about the exchange rate for next year here!

I say between $20 and $100 by end of 2014. That doesn't mean the price can't go down from here though.

My guess is around 25$

I think Bitcoin will be stable and in the low $17.xx range by the end of 2013.

I think Bitcoin database size will make it totally unusable on regular hardware. GPU miners will quit biz and sell their stuff.
Bitpay will continue to get ppl Bitcoin reserves and sell them on market.

All this will bring single digits or less Bitcoin price.

But the database still fits on a thumb drive?   ???

But causes total hang of system IO scheduler on open, close, sync.

You ppl do not have brains? When DB will reach size of RAM of common computer - customers will better shut themselves rather than use vanilla client.

Have you tried the 0.8 prerelease?  It reduces the working set (the part needed in memory) to around 150 MB with the current blockchain.  It synced on my machine in under four hours without even keeping one core busy and mostly idle disk (it was network bound, which is the next area targeted for speed up).  Someone ran it on an old pentium 4 machine and it synced in under 6 hours (easily done overnight).

In other words, I know you love Satan, but there are enough truths to strike fear in people's hearts.  You don't need to make up lies.

There is no matter, how much data it loads to memory.

They key is index size, which is growing. Another key is number of broadcasted new transactions per time, which require database lookup to verify them.

As index size will run out of ~double avg RAM size of common computer - clients will start goxxxxxing on each incoming unspent trans. Because any disk caching mechanism will not be effective with such data and RAM sizes.

So every client on every incoming unspent trans will perform uncached disk lookup through gigabytes of data.


Hehe.

I mean no software optimizations will be effective on such conditions.

Gavin will have to publish minimum hardware req to run client. And those req will include good amount of RAM and good ssd disk.

I'm no expert in databases but it seems to me you don't know what you're talking about.  Can someone confirm or infirm?

I'm some kind of expert on high load servers maintenance. Including high load databases. I see future bottlenecks when they don't even exists. Project owners pay me good money for my skills.

Growing unspent transactions number mean growing key lookup load on database.

And when index size do not fit in RAM - software will perform key lookup reading whole index from HDD (gigabytes of data on single lookup).

It is obvious that such conditions will require good SCSI or SSD backing array.

Nice bets for 2014 - you guys should buy those cheap coins now, right? ;)

I'm not big expert in Berkeley DB, but it is obvious that one index record takes about 300 bytes in space. It's a sha256 hash + some index data. Remind me true number if you please.

So... To get index size greater than 4G we need only 13m transactions in whole chain. This is nothing for widely used payment system. For Bitcoin it is a 180 days of blocks containing 500 transactions each.

So Bitcoin is about to grow? Hehe.

You know that spent transactions can be ignored, don't you?

With the cost of security, don't you know?

If client receive bcast block/transaction from network it must check its prev outputs in full block chain to know whether its a valid transaction for futher broadcast.

If client burn old transaction - it will not be able to check new transactions for validity.

If it broadcast not valid transaction - it will be banned by vanilla clients.

So burning old block bodies makes client vulnerable to Sybill and DoS attacks, coz he unable to check what is true and what is false comming from network.

There is no security flaw in this.  New transactions are not supposed to try to spend an already spent transaction.  So it makes sense to remove spent transactions (only once buried under a few blocks though) from the index.  They will still be in the database, but accessing to them will take more time.

Though I'm no database expert, I'm pretty sure an index does not have to be exhaustive.

Allright. The client with DB consisting only with block headers receives the transaction. What next he must do with it?

Store? No. It must be sure transaction is valid to store it, but he can not check its validity as he dont have full chain.
Broadcast? No. Same reason. If trx isnt valid, he will be banned as DoS source by vanilla clients.

If network will consist of huge part of such light clients - it will be paralyzed.

I'm not talking about a header-only version of the client here.  I'm talking about which part of the index it is useful to load in memory.

Ah, crunches... All right :) Lets see where it will go.

I see the future of bitcoin chain in some sort of distributed hash table with good redunancy to avoid information drop.

So I see the main load may be put on network layer, instead of SATA bus.

pent, you always break my pessimism  ;D

But this idea does not exist even on draft for today.

I develop DIANNA chain to test DHT chain storage there.

Each bitcoin node does the whole work for serving system: receive, verify, store and broadcast transactions and blocks. This work not divided between participants, but every one doing the same over-redunant work.

So when bitcoin will grow and serve millions transactions per day, every client will serve all theese transactions.

You consider that common user home computer is able to serve the worldwide payment system? And all that we need is to reduce index size loaded into memory?

You dont know what are you talking about. You all here are full of illusions and disconnected from reality.

And reality is disappointing. Bitcoin will meet huge bottlenecks in not far future while Bitcoin Foundation members kidding with their VIP status.

Why not?   Computers are quite powerful and humanity is not that large.

Why not what? Why regular home computer resource not enough to handle load of widely used payment system? Should I explain this for you? Do you think Paypal uses only its CEO desktop to handle system load? I think they have some number of full 42U racks of servers. Even not in one DC. And load is balanced between them.

And we have here that every Bitcoin node replays full system load: receive, verify, store, broadcast.

Are you all really so stupid here?

the only way for bitcoin to scale up and serve to millions of user is to have dozens of "paypal" services and transaction using blockchain to be just occasional

Bitcoin banks could help.

I don't understand -- what's the major difference between Bitcoin and torrents?  Torrents are distributed, and seem to work just fine...

Torrents distributes load all over all nodes. In Bitcoin, each client carries full network load. This load is database specific. Network consists of thousands of nodes and each one verifies and stores incoming transactions and blocks. Each node repeat the same job of another one.

Verification process require multiple lookups in potentially huge database. I don't think regular client can handle this type of load in not far future.

I am tired to repeat this again and again. You all just stupid and full of stereotypes community slugged into your mind.

Open your fucking eyes. we all go to bottleneck apocalypse. Didn't you mind that vanilla client already makes your computer unusable during sync process? Imagine now that Bitcoin has 10k transactions per block instead of 200. Your PCs will just die.

although lots of you don't want to hear it, i am also very concerned and can confirm lucif's view on the database problems. i think it has top priority for devs to focus on that so we don't have only a couple of servers who own the whole chain in a few years, because that scenario would be worst case for bitcoin, since we want a decentraliced network to prevent abuse/fraud.

Regular user will not run bitcoin client 24/7

It will run it while he needs accept ot send coins. The time between those events may span into days or even weeks. And what? He will wait until his PC is stuck for hours in sync to process transaction?

Well, I'm tired to explain this shit.

All we need now is the open gates for two paths:

1. Ability to store database in distributed hash table. This may shift load from HDD and memory to network, as most participants have enough network connection to handle this.

2. Make walletless client and split wallet-related functionality into RPC commands: inject transaction, query for address transactions, etc. This is to run such stuff as Electrum server. So bitcoind will be able to run without wallet as just passive network node, which just stores full chain, doing transaction relay. this one can be run on powerful servers and accept connection for chainless clients.

If people support this i can make a BIP. But I see, nobody gives a fuck. Allright. Lets just trade and go to tha moooo

+1

too many folks here (speculators) see bitcoin as a thing that is just there and works, but forget that is is still in heavy development and far from a state where you could call it a "finished project"

Well, in the whole world I like just one thing - clever people who have ears to hear. They are so rare.

Matthew 7:13-14

All other bubble makers - you will be burned in hell. Hehe.

All is fine as long as your PC can verify those 10k transactions in less than 10 minutes.

From data on bloc 215475, I can read that a transaction is about 528 bytes long.   So 10k transactions will be about 5.28MB of data.  In ten minutes, that's an average of 8 kB per second.   It might require improvement of existing software, but really it does not seem so difficult.  Especially considering that consumer electronics keeps improving with time.

yea, but what if average joe uses his client only once a week, he will then have to verify ca. 1000 blocks, what will take quite a lot of time

Well, if bitcoin becomes so popular that it generates 10k transaction per minute, then I guess most users will make bitcoin transactions more often than once a week.  I mean, supposing that bitcoin meets scale difficulties and yet that users use it very sporadically is a bit contradictory, isn't it?  (it's an interesting question, though...)

my example of once a week was maybe a bit extreme. but just take a look at average banking use today. i don't know many peolpe who use online banking/paypal/etc more often than 2 or 3 times a week.

of course that won't make bitcoin unusable, even if you have to verify 1000 blocks. but if we want to make bitcoin more popular we also have to focus on usability. and i don't think many peolpe would be happy with a payment solution that takes 20min to load before you can use it.


i don't want to make bitcoin sound bad, i love the idea as much as most here do, i just want to point out that there is a lot of work to be done, especially on technical details of the software

I can agree with that.  But there was someone in this thread barking that "your computer will just die"  ::)

He's just spreading FUD to make profit on his short position.

thats inline with the 33% growth, its also what Im expecting.

Too bad for him... even that won't talk the market down right now.

You have no idea what you are talking about. 10k transactions per block is peanuts for even a low end CPU to handle. That is less workload than computing a single frame of any modern PC Video Game. The biggest bottleneck now is the disk read to lookup transactions which is resolved in v0.8 by only storing the unspent output transactions in RAM which is ALL YOU NEED to verify incoming blocks/new transactions. The unspent output set would have to grow by atleast a factor of 30x before storing it in RAM(4GB) becomes an issue. Even if it does, you have on average, 10 mins to process it, something your mobile phone CPU could easily do.

Lets bring it up to 10k/s and assume that the max block size was increased to handle such loads and network bandwidth is not an issue. We could store the unspent output data set in your consumer grade GPU and process them all in realtime.

In short, processing power/harddisk are cheap and plentiful even on low end PCs TODAY.
Memory/Network bandwidth is relatively expensive especially in some parts of the world.

The whole point of having every node process everything is Bitcoin's core philosophy of being trustless.
You dont have to rely on anyone else to verify the validity of any transaction.

Hey, I am talking exactly about disk read. Not about CPU or GPU. Such amount of transactions will cause heavy disk key lookups. Read before write.

Dude, with the latest leveldb ultraprune builds I can sync the complete chain, verify the transactions and block hashes for all blocks, and verify the signatures for all the blocks after the last checkpoint in under 4 hours with mostly idle disk and less than one core of cpu.  It's bottlenecking at the network code (not network speed, just the block download code needs work that is underway or will begin soon).

So how is disk a problem again?

Which as i and others have mentioned is resolved in v0.8. The current RAM required for the unspent set is barely 150MB. We are a long way from having to worry about it exceeding RAM availability. Even if it does, 10minutes is plenty of time to page it from disk.

Off the top of my head, i can already think of some easy ways to optimise this.

1) Only store the most recent unspent outputs in RAM.(Temporal Locality)
This is based on the observation that recent outputs are more likely to be spent again versus those which have been dormant for many years.
This will automatically optimise out lost/saving/cold storage wallets(Which is huge) by leaving them on disk.

2) Do a x pass process to verify the transactions in the block.
Basically x is
Total unspent output DB size / Your RAM

For i = 0 : X
Load each partition(i) in RAM, verify transactions which are in partition

There are probably many other ways you could optimise this but i thought of these 2 in less than a minute.
There are many very experienced and intelligent people involved in the Bitcoin ecosystem, this is the least of my worries.

Slowpokes reserve here.

You will be doing well with this as long as your RAM is enough to cache disk data. When it is not - big problems comming.

Unspent needs to be verified against spent. And spent are growing to infinity.

Btw, disk stunning persists also at syncs (write), but I think its an ugly code.

Why? I believe a transaction only needs to be verified against unspent.

Don't know - don't tell. Search for connectinputs() in source code.

It's not a descending process.

Coinbase transaction A is spent by transaction B which is spent by transaction C.

During database loading, you verify each block and you say:

- ok A is good,
- (you process more blocks)
- oh B spends A so B is good and I can forget about A
- (you keep processing more blocks)
- oh C spends B so C is good and I can forget about B.

Now if a transaction D comes and spends C you don't have to go all the way to A because you just need to remember that C is good.

All you have to do is to put C (and not A nor B) in an index or something so that you don't have to do this everytime you run the client.

PS.  Of course I oversimplify here because there can be chain forking which requires that you don't forget spent transactions too easily.  But even so, "forget" is a bit too strong a word since it just means you don't put it in RAM anymore. It's still on disk.

Yes, but...

http://blockchain.info/charts/bitcoin-days-destroyed-cumulative

well duh it goes up

there are 10 million bitcoin days created every day!

Looks like its out of speculation discussion, but i will continue this chess game.

10k trans per block is a 16 trans per second.

Each trans can have many inputs. And every input needs to be found in huge data set.

Your move.

It's a cumulative graph and it looks quite linear.  What looks linear in a cumulative graph?  A constant!

Also those are "destroyed" bitcoins.  Therefore they represent transactions that could be "forgotten".  So if this is of any relation with this thread (which is not obvious), it really does not seem to prove your point at all.

Plenty of data structures can look up individual transactions in O(log u) where u is the number of unspent transactions.  Even using the lowest base of two, if there are u= 1 trillion unspent transactions, we are looking at about 40*16 lookups per second.  Even a spinning disk can easily hit 2000 seeks per second, and a SSD would handle that many reads trivially.  Only if we assume 1 trillion unspent transactions, a suboptimal data structure, 16 transactions a second and 3 inputs per transaction do we move to SSD as a necessity.  And that's assuming spinning drives don't improve in the decade minimum it takes to get to these levels.  Oh, and we're ignoring the cache in ram that will hold many of the transactions needed.

Your move.

And again, it is not a descending process.

Even if your transaction has an hundred inputs, you'll have to verify that each of these input point to an output of a valid transaction, but you won't go further, for the same reason as described above.  There is no exponential growth of the number of required checks.

Also, the number of inputs and outputs is more or less proportional to the size of the transactions, or the average size of the block.  I've already calculated that for 10k transaction par block it was about  8kB per second, iirc.  However you look at it, processing 8kB of data per second does not seem like such a tough task for a computer.

O(log u) - proof pls.

afaik, spinning drive has about 100 iops performance.

No need for proof, it's common knowledge (http://en.wikipedia.org/wiki/Binary_search_tree).   Just look on the top right.

http://en.wikipedia.org/wiki/B-tree

And I'll give you the point on 100 iops for low end consumer drives.  IOPS is a better measure than seeks.  Good drives can get you close to 200 and SSDs can blow the 2000 I calculated around out of the water: http://en.wikipedia.org/wiki/IOPS#Examples

Not the best example... Binary search trees are only average case log(n), worst case n.  Btrees are worst case log(n).

This is my 7200 seagate stats under running bitcoin. Regular disks have lower rates.


As you can see, its about 100 requests per second with almost busy heads.

And this is cached sequental read, when caches are available as not so much data in DB. And first column displays how many requests were merged.

Guys, cumon :) You are talking with highload architect and unix sysadmin :)

For the third time, quit using the old code.  We know it's shitty.  The new code is already available and barely touches the disk.

Who doesn't believe that optimal search is log(n) ::)

I dont talk about the code - I show you HDD capabilities. Where are 2000 seeks?

 :D

Sorry, I read "stats under running bitcoin" and mistranslated your gibberish as stats with bitcoin running.  I already agreed that 100 IOPS was correct, I thought we were moving on.

So you staff going to announce that good highend SSD needed for using bitcoin? My 7200 is not the cheapest HDD.

Yeah. Haha. I dont too close familiar with BDB, but afaik MySQL uses BTREE in indexes also. I didnt noticed your log(n) formula works there.

I have no staff.

Here is a guy with "Staff" near nickname talking to me.

Oh grondilu... he just helps run the boards.  You should know bitcoin has no staff, and forum staff don't really have any more weight with the bitcoin community than anyone else.

But I imagine if SSD becomes necessary (it isn't even close now) that is what people will recommend.

That is what i said. Regular user will expirence difficulties, including HW incompatibility to run bitcoin. At least vanilla bitcoin.

All other - empty bubble talk.

Right, once we reach the 1 trillion unspent transactions in a decade or 3.  And all advances in storage technology will stop from today forward.

Looks you are completely theoretical, while I'm practical.

With your optimism paypal can run their processing on raspberry PI  ;D

1 trillion? And PC will survive? LOL.

No, you're being theoretical too.  There is currently no problem, and you are imagining usage levels that will take decades to arrive.  I'm being much more practical than you about adoption rate, yet somehow I'm more bullish ???.

Allright  ;D You laugh me a lot here.

I saw databases above 4G with good btree indexes (on integer field!) and know what is it for single SATA HDD.

I'll check your 0.8 version

No problem? You fokin kidding me? If I use Bitcoin for payments once a week or two, and need to wait hours for sync each time I launch client - its not a problem? My PC hangs for that time - its not a problem?

Don't offer me crunch! Imagine I am regular stupid user - not Satan slave. Is it not a problem? Where is log(n) in this formula? Clever blind bulls.

I agree with you that if the unspent output stops fitting in RAM, then we have an issue.
Regardless of algorithm used ,O(1) or O(log n), disk seek is slow.

But as many have pointed out and which you refuse to acknowledge, current unspent outputs will have to grow by atleast 30x before that is an issue with TODAY's RAM sizes.
And unspent output size should grow parallel to bitcoin adoption. So we are looking at a 30x increase in Bitcoin adoption before we even have to worry about today's client.

Also stop basing your arguments with v0.72 which we all know has issues with disk seek.
Go download v0.8, use it, sync it once in 2 weeks before coming here to spread FUD.

http://i45.tinypic.com/2uz8ao3.png

You've seen it here first. :)

As for my prediction, atleast above $100 by end of 2014.

You sound like an IT guy with no understanding of the underlying algorithms and methodologies.  What do you think people did back when each bit of RAM was literally a magnetic torus hand-threaded with wires and data was stored in a big linear magnetic tape that took minutes to "seek" to the proper position?  I didn't live it but have read about it.  There are clever algorithms (btree being one) that minimize uncached key lookup time.  Holding all the database keys in RAM is a very new technique only recently made possible due to how cheap RAM has gotten.  Also, all unspent txn don't need to be held.  Lots of coins out there don't move at all (~90%) which is why some people think a lot of hoarding is going on.  So the RAM need only cache the acts/txns that are "in play".

And if bitcoin becomes incrediably popular, imagine a network that dynamically creates new blockchains among commonly-transacting user groups (regional, for example) spends BTC from the main blockchain into this new chain and then after a few months (or whatever) merges the chain back.  This technique that would make it so that the main blockchain does not see every microtransaction in the entire world, and subchains could have either smaller or longer settling times depending on the requirements for that chain.  Sure it would take a lot of effort to implement.  No point doing it until there really is a problem with the system as it exists today.

January 17th, and we're almost at 15$ already.

Bump.