 Show Posts
|
|
Pages: [1]
|
|
One thing I noticed was that when I ran bitcoind with the data dir on an NFS volume, it performed horrendously - basically maxing out my CPU, even after the blockchain was caught up (which took well over a day). Moving to local disk made an enormous difference, it now only takes a few percent of CPU.
I don't think the problem is that it does so much disk I/O that the network can't keep up; I suspect it's to do with the disk caching behaviour changing. Data that is left in the buffer-cache when running against a local filesystem is being flushed in tiny pieces over NFS, or something like that.
From what I know of EC2, I wouldn't expect it to have that sort of problem, but you never know.
The CPU requirements are so light, my Grand Plan is to try and run bitcoind on a Raspberry Pi (when mine arrives). There are two potential problems; handling the 64-bit integers on a 32-bit core (I assume that a 32-bit Atom is able to use SSE registers, the Pi might suffer more), and secondly, the I/O of an SD card. But it will be interesting to find out.
|
|
|
|
|
I would think the space required is pretty much in proportion to the number of transactions, but I could count that too. Unfortunately running the analysis seems to have messed up by blkindex somehow, so I'm held up while that regenerates.
What I wanted to find out was whether the number of outstanding transaction outputs was still increasing rapidly. If you plot a rolling average of the increase in the number of outstanding outputs per new transaction, it peaked at nearly 1 last September, dropped to 0.25 around January, grew to 0.4 in April and has slowly declined back to 0.25 again.
It's difficult to make sense of, but unless that ratio goes into a steady decline, it means that the number of outstanding balances to track is increasing rapidly along with the number of historical transactions. As I say, the data doesn't really show yet whether that's the case or not. We're adding 20,000 to the number of unspent transaction outputs each week lately, but a year ago, when transaction volume was much lower, we were adding 50,000 a week. If by some mechanism a node is storing only the unspent outputs, figure 70-80 bytes each, that would need 120MB, growing at 1.5MB / week. Meanwhile the blockchain is growing by 100MB / week (that's a rough estimate, I'll try to get real numbers tomorrow).
|
|
|
|
|
Don't know if this is of interest to anyone -- growth in the block chain from the beginning of the year:
Block# Date Tx TxLeft Outputs OutputsLeft
160651 2012-01-05 2141909 537489 4958838 1265195
161701 2012-01-11 2184208 530150 5058211 1256418
162751 2012-01-18 2231239 535105 5167264 1265877
163801 2012-01-25 2277606 544932 5275969 1281789
164851 2012-02-01 2322666 551070 5383353 1296623
165901 2012-02-08 2370911 560104 5497425 1314422
166951 2012-02-15 2430945 570340 5638086 1335083
168001 2012-02-22 2479948 581100 5754867 1355583
169051 2012-02-29 2524194 589006 5866066 1375399
170101 2012-03-07 2573735 597328 5992463 1395833
171151 2012-03-14 2615923 603458 6103448 1410601
172201 2012-03-21 2661845 613033 6217395 1431445
173251 2012-03-28 2706242 624443 6328060 1452971
174301 2012-04-04 2756117 633395 6449913 1471912
175351 2012-04-12 2809748 642169 6579808 1487912
176401 2012-04-20 2872153 650236 6729451 1505063
177451 2012-04-27 2929045 658588 6866321 1521086
178501 2012-05-04 2991166 668820 7014679 1541885
179551 2012-05-10 3087103 676173 7233590 1559496
180601 2012-05-18 3239737 684265 7569078 1576859
181651 2012-05-26 3469896 698220 8067651 1606888
182701 2012-06-02 3654532 708385 8474595 1629500
183751 2012-06-09 3877115 718799 8956077 1648213
184801 2012-06-16 4179254 738515 9623716 1678629
185851 2012-06-23 4407499 758333 10153165 1714365
To clarify: the last four columns are the number of transactions, the number with still unspent outputs, the number of transaction outputs, and the number of transaction outputs still unspent. I hope it's accurate, the numbers look sane to me, and the code was fairly simple, but I could quite possibly have made a stupid mistake.
It's relevant to the various threads on scaling & blockchain efficiency.
|
|
|
|
I'm not talking about the inputs to the transaction that pays me, I'm talking about the transaction that pays me itself. Fred posts a transaction which he says pays me 100 bitcoins. If I have the digested tree, or the whole block chain, I can check the transaction is valid. If I don't, I can't. But either way, it's still unverified. I'm going to wait for confirmations. Once I have confirmations, then I know it's valid too, because if I'm trusting the miners not to include fictional unspent txout digests I might just as well trust them not to include invalid transactions.
The problem this mechanism solves - validating an unverified transaction in a lightweight node - doesn't look to me like a very important one.
Sure, if you aren't mining, like spam, and don't see any value in reliably knowing you have received funds from Fred in under an hour, since you can't tell the difference between a bogus transaction from Fred with fake inputs and a real one. You might be willing and able to wait 6 confirmations before deciding others have paid you, but others won't. Under this proposal, miners can reliably mine using these trees and not the block chain. If you think all miners will want to lug around a block chain whose size tends closer to infinity with each person who starts running a gambling bot, then sure, this isn't important. Being able to validate a transaction instantly is important for spam prevention. Nodes only relay valid transactions. If you can't validate transactions, you have no choice but to blindly spew to your peers anything any of them sends. You'll be a sitting duck for DoS attacks (since for every 1 message coming in you'll nominally send 7 out), and a whole network made of nodes like this would be easy to spam into oblivion. Finally, this tree proposal isn't meant to RUN on a lightweight node. It is meant to make a normal node be able to SERVE another lightweight node, at the same time not having to have the full unabridged block chain. So the scenario in which this helps is where (a) transaction volume is so high that even miners running fancy purpose-built mining rigs can't store the transaction history on a standard-issue 1TB hard drive (b) every Tom, Dick and Harry runs a lightweight node which relays every single transaction on a P2P network. Those two conditions contradict each other. If transaction rate goes up that high (and I think it shouldn't, but that's an entirely different discussion), bandwidth becomes the limiting factor before storage space does. At that transaction rate, inevitably the bitcoin network evolves to a backbone of heavy nodes exchanging everything and lightweight clients which consume only data of interest to them. That's quite independent of how the history is handled. As to unconfirmed transactions, are there really going to be that many people who will accept an unconfirmed transaction, but not be willing to trust anyone to validate it for them? |
|
|
|
Sorry to be slow, but I don't see the gain here. If a lightweight client is going to trust that a metablock that's been merged into the chain is truthful (because it's been built into a block), then it can just as reliably trust that a transaction that's in the chain a few blocks back is valid, because it's been built into a block.
That works if, as a lightweight node, you plan only on receiving funds that have a very small number of confirmations, which eliminates your view of the majority of bitcoins that exist. In USD terms, this would be like limiting yourself to only being able to accept crisp dollar bills that have never been handled more than once or twice. More likely than not, you're going to need to be able to receive funds from anybody, which will have been confirmed anywhere on the block chain between the genesis block and now. You either need the whole block chain to know whether a given incoming transaction is valid, or at least the digested tree of all unspent txouts for the entire block chain. I'm not talking about the inputs to the transaction that pays me, I'm talking about the transaction that pays me itself. Fred posts a transaction which he says pays me 100 bitcoins. If I have the digested tree, or the whole block chain, I can check the transaction is valid. If I don't, I can't. But either way, it's still unverified. I'm going to wait for confirmations. Once I have confirmations, then I know it's valid too, because if I'm trusting the miners not to include fictional unspent txout digests I might just as well trust them not to include invalid transactions. The problem this mechanism solves - validating an unverified transaction in a lightweight node - doesn't look to me like a very important one. |
|
|
|
|
I don't think hard links would work either: the problem is that both processes will think they "own" the block data and try to keep updating it, and be confused and surprised when changes are made that aren't made by it.
I have read that developers are currently working on separating out the blockchain maintenance from the wallet. There is no fundamental reason why you shouldn't be able to run multiple bitcoin clients which share one blockchain, either on one machine or a local network, but the code doesn't yet support it.
I think it's really important, and I would be working on it myself, but, as I said, the developers are working towards it, so I'm still trying to catch up.
|
|
|
|
|
Sorry to be slow, but I don't see the gain here. If a lightweight client is going to trust that a metablock that's been merged into the chain is truthful (because it's been built into a block), then it can just as reliably trust that a transaction that's in the chain a few blocks back is valid, because it's been built into a block. There's no need for it to keep anything. The only real advantage here seems to be that it saves miners from having to have a hard disk, and it seems like a lot of engineering to do that.
Quite possibly I'm missing something, in which case it would probably help for someone to step back and explain the aims and benefits.
|
|
|
|
|
Because most of the data in the block chain is hashes, it's not compressible at all.
|
|
|
|
I don't see that even the complete tree of outstanding balances needs to be validated anyway. To validate a transaction, I need answers to these questions: - What block (number and hash) is the latest?
- In which block (number and hash) ancestral to block aaaaa is transaction xxxxx?
- Is output n of transaction xxxxx spent in any block betweeen number bbbbb (with hash ppppp) and number ccccc (with hash qqqqq)?
A service can give signed answers to all those questions. If it ever gives a wrong answer, it can checked and proved to be wrong by anyone with the complete blockchain. It wouldn't be hard for a client to do random spot-checks on a service at a low rate -- again, if it ever gets a signed wrong answer it has proof for all to see that the service it used was unreliable. Running a service like that is easy: all you need is enough cpu to keep up with the blockchain, which is trivial, and enough disk to store the blockchain and indexes -- currently about 5GB, say 1TB looking forward a few years. Again, that is trivial. The services could be run by ISPs, exchanges, online stores, enthusiasts, and could easily be ad-supported. It could easily be made into an appliance that would cost only about USD200 at current prices, meaning any retailer using bitcoin regularly could have one of their own. The client could check with a few different services: again, any wrong answer can be proved to be wrong by anyone with the blockchain. That surely is the way forward in the long term. |
|
|
|
|
I can't help feeling that the better way is to separate out the wallet-handling code, with an RPC interface between that and the blockchain + network code, and an option of running it the current way with the two modules in one prcess. Still, there may be situations where multiple wallets in one client is better.
|
|
|
|
|
Note that with the GPU hardware most miners use, scanning the whole 2^32 range of nonce values takes a matter of seconds. Mining software will scan the whole range looking for a valid hash for the block and then increment the timestamp or reorder some transactions so it can try again.
|
|
|
|
|
Can you add a little more detail -- exactly what you're running and how, and exactly what is output? Someone might be able to help you.
|
|
|
|
|
Well, the backup is not required, but good surge protection might save you losing a lot of hardware. As someone pointed out above, cheap "surge protection" strips are not to be relied on (though are better than nothing).
On the other hand, if your grid power is generally reliable, it may not be a big deal.
The blog "Dan's Data" has a lot of good information on power supplies / surge protection etc.
|
|
|
|
|
If anyone here works in financial services, do they know how bitcoin activity is looked on by compliance departments / regulators? If your trading activity is restricted for compliance reasons, would that apply to small bitcoin transactions, or would it be seen more like changing currencies to go on holiday, or buying WoW gold or something?
I could just ask them, but that's kind of like asking them to say no.
|
|
|
|
|
I'm tevirk, with some background in software and finance
Key fingerprint: 93B4 D242 C8DD 31F3 FCDF C3ED 65BA 0D01 5E31 5448
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.10 (GNU/Linux)
mQENBE/cOKABCADHcMW761xM7GBF3AHdoNVNGJ5yusygV0xqH6vvEZ5zWs3mdjVS
Ez6N0aE8c9YwfsgsQBAYqznv3DX8/O+oDMkCYcci5aRVjWtPCUYOPiGIaxkwpIb/
iJ8leMszVkIzguHZ1HyF4b7PVDtsBCn0MP2tJedCPLy4dbNkc0MYEGW5/mfgOfQW
oFuK27E/LDSoMwR/Ug8SKGL10Ph+WushQ7BPEu31a33Mz1PQhsqONEaFtVPwoUQN
6ice51w+ZmSwsSpvwAr37BRX1q8AuEcnyRl2/uDX1EY4oMxdziCJOqxSvwDXXAUT
Bbu8ic8xckQDY904nt+eCsZ6siHTFst0MMtlABEBAAG0GHRldmlyayA8dGV2aXJr
QDNkdXAuY29tPokBPgQTAQIAKAUCT9w4oAIbAwUJAeEzgAYLCQgHAwIGFQgCCQoL
BBYCAwECHgECF4AACgkQZboNAV4xVEhl5ggAuYro5uYSo+5ZNinKffElVUpzlMl8
KAoeJw5Pe2RqIbr9j3bDlXN3xEPrwpWGwEZ5f4eQEBKA73sUB1r8QkWz2qc+BgWz
o3pNslI0QEFq6DYVg6YL2FjbNUvvd2VvIl5DLIRTJ2pMCowDWHOgNnTKmM1Lihu6
oZLkj8lqqyte9g1YRBE45q9nwhF6FzunuaPPOKVBTDVj/xZ4lSEWOiv+T3/Zphcc
krJNqWJRMehu5pHdFkpYuu/4UpBFpz1kzZzD6ahGOWMI4Y736zsn5KRo5O8QCU+Q
bUkJTkqc++Pn67NM0IyyqFWqwxXvX17hBl5Z3/ho1qIvWOjAh0yndy1D6LkBDQRP
3DigAQgAzbDKepbh6PTbVSnzIgkXk0sRn0j5HY3+pzLUkKIC7riqR136FN+bSzxB
gGqgU8srAoWB7l9Nzde7dWrOYtDVoUXx2V4PEQrWuRUSluF4lyMJptEmozDuh6aM
5P9hrKHLut+UkLG0VafkqMBB1nAGnMypmihceiSCoibl03xYdNBBHrHCZLYC9k/n
sgzuosQ6ITKVmw/i+7s8b1iQhCFylDmebWjbfw3kxANBYJIJ6rzHoSkTgu5r4Hpg
f2+fkkfkaPJekka6yREjPeT1qQPsrWbFz25UYEQjwQ6+14jGXbNiT7yGJDew/rgk
lIhDtaX1QUoyqX7NHnMck9IzxtgCXwARAQABiQElBBgBAgAPBQJP3DigAhsMBQkB
4TOAAAoJEGW6DQFeMVRIPzcH/1PljPCbM2qZqMI2k14G/7UbqPUHGIUk7/QZKRl8
nXKq6hnEfH7ioPV9NujMKYluZ1//YH+N1rz49I3GWDHHnsSHscIOyoDtzhMytdO8
x7D4wd5Lcd3Qcz3pZtBCV9zpB+c/vMLUCx+kUQOUgjkGv7FS4gfTP6vyCs/lXNCd
epYxG+9g1D4of2gWWbcoNMuCxcRvknAqzMEY/cRJg9O0JrjfzmjwBxGbgV1tDRoH
ujnVH1pvx/B5UQVFjWhBf6TpshtvOrbMYo/uf0TA+iI4V/DKJvQ3vWSOtUcmD3T/
AoSgdxkZEk0pZtHhwqyPGfVVOjsPZepTqewGYWQB8pcnf1g=
=taB5
-----END PGP PUBLIC KEY BLOCK-----
(only 2048 bits, but only valid for a year)
|
|
|
|
|
