I still think a simple solution, like "give me this part of this block/transaction", would have a much better chance of success in a short term. I don't understand your point - that is exactly what Bloom filtering provides. It is deployed and working for SPV clients for some months already. There have been no issues with it. You can't use it as a full node because a full node, by definition, must download full blocks as it must know about all transactions. Well, you have just said it yourself: this Bloom filtering does not help at all, if you want to have a full node. And I do want to have a full node - probably as well as most of you guys out there. So how does it help us? Incidentally, if you're going to make sarcastic comments implying Bitcoin hasn't improved, you should actually know what you're talking about. Bloom filtering launched at the start of this year, it's not something that was originally a part of the protocol - so there have been big improvements quite recently. I think you were the one who did not know what I was talking about.  Improvements are worthless if there is no actual software, which people want to use, that gets advantage of them. What I meant by "give me this part of this block/transaction", is literally "give me X bytes of block Y, starting at offset Z". So, when a new block appears in the network and I need to download it, while being connected to a number of peers, I don't ask each one of them for the same megabyte of data - instead I can just split the work into, let's say 32KB chunks, and this way fetch the entire new block from my peers much quicker. But that would only be useful before making the protocol to support fetching blocks from HTTP servers - which is the ultimate solution, which IMO should be implemented ASAP, if you guys really care about all these small bitcoin users and their internet connections. The mining pools should help here, because it is in their very best interest to propagate the blocks they have mined as quickly as possible across the network - and what could be quicker than a static file, served via http, from the pool's domain, through a clodflare-like infrastructure? For distributing the block chain you can as well use Bittorrent or some other large file distribution mechanism rather than HTTP serving, it's already possible and there are already torrents distributing the chain in this way. They aren't designed for end users because end users should eventually all end up on SPV wallets which already only download partial blocks.
As I said before: I would prefer to focus on improving a behavior of a node that already is synchronized, rather than focusing on making it faster to setup a new one from a scratch. Initial chain download, and the fact that it takes so long, is inconvenient, but it is not really such a big issue for the actual network. Besides when you setup a node from scratch, and so need to re-parse the 236+k blocks, the network communication does not seem to me as much of an issue, as all the hashing and elliptic math that your PC needs to go through.
There is another thing that came to my mind, so I will just add it here, to this post. I believe nodes should not relay transactions that use any inputs which exist only in the memory pool. It should only relay transactions that use inputs from an actually mined blocks. This IMHO would improve the network's traffic pretty much. A regular user never needs (usually is not never able) to spend an output that has not been mined yet - while, at the other hand, relaying of such transactions takes a huge part of his network connection. |
|
|
|
Maybe we should not be focusing so much on the initial blockchain download, but rather on limiting the bandwidth usage of a completely synchronized node. As for relaying transactions, I would even go crazy enough to implement a web of trust - where you don't verify every transaction, but only random ones and then build a trust to the node that sends them to you - then you check them randomly, but less frequent. But also for transactions - the data should be kept on WWW servers. There is no economic reason to fetch them from China |
|
|
|
I agree. But in reality, the logic of what to fetch is only important during the initial chain download. Later you just fetch whatever new there is... So it is not really so important, is it?  |
|
|
|
Using Bloom filtering may not be entirely viable yet, I'll have to check. The big changes is first downloading and validating headers, and then downloading and validating the blocks itself. IMHO, it's the only way to have a sync mechanism that is both fast, stable and understandable (I have no doubt that there are other emchanisms that share two of those three properties...). I still think a simple solution, like "give me this part of this block/transaction", would have a much better chance of success in a short term. And I also think that it would be nice to have something in a short term What protocol is used to actually fetch blocks is pretty much orthogonal to the logic of deciding what to fetch, and how to validate it, IMHO. I disagree. If you are a node behind DSL, and you have a very limited upload bandwidth, you do not want to serve blocks (and maybe even transactions), unless it is really necessary. There are servers out there, connected to the fastest networks in the world - these you should use, as much as you can. Who is going to stop you? |
|
|
|
at least one |
|
|
|
Not sure what you mean by "as deep as possible". We always send getdata starting at whatever block we already know. The reason for starting from early blocks and moving forward is because validation is done is stages, and at each point as much as possible is already validated (as a means to prevent DoS attacks, mostly). As most checks can only be done when you have the entire chain of blocks from genesis to the one being verified, you need them more or less in order. What I mean is when the client has already downloaded the full chain - and is basically waiting for a new block. Why to ask 500 blocks back? That's not true, we only ask for each block once (and retry after a timeout), but it is done to a single peer (not to all, and not balanced across nodes). That's a known badness, but changing isn't trivial, because of how validation is done. OK - than I'm sorry. It only proves how little I know about the bitcoin client, so I should not be changing it There is one strategy however that's pretty much accepted as the way to go, but of course someone still has to implement it, test it, ... and it's a pretty large change. The basic idea is that downloading happens in stages as well, where first only headers are fetched (using getheaders) in a process similar to how getblocks is done now, only much faster of course. However, instead of immediately fetching blocks, wait until a long chain of headers is available and verified. Then you can start fetching individual blocks from individual peers, assemble them, and validate as they are connected to the chain. The advantage is that you already know which chain to fetch blocks from, and don't need to infer that from what others tell you. I saw getheaders and I was thinking about using it. But it would basically only help for the initial chain download. Now I think if you really want to combine the data you got from getheaders, with the parts of blocks acquired from you peers after they have implemented BIP37 (otherwise it won't be much faster) - then good luck with that project, man! I mean, I would rather prefer baby steps - even extreme, like having a central sever from which you can fetch a block, by its hash. I mean: how expensive would be that? But how much bandwidth would it save for these poor people.. BIP37 actually introduced a way to fetch parts of blocks, and it can be used to fetch a block with just the transactions you haven't heard about, so it avoids resending those that have already been transferred as separate transactions (though I don't know of any software that uses this mechanism of block fetching yet; once BIP37 is available on more nodes, I expect it will be). Interesting.. thank you, then maybe that should be the next thing that I will add to my client. There are many ideas about how to improve historic block download. I've been arguing for a separation between archive storage and fresh block relaying, so nodes could be fully verifying active nodes on the network without being required to provide any ancient block to anyone who asks. Regarding moving to other protocols, there is the bootstrap.dat torrent, and there's recently been talk about other mechanism on the bitcoin-development mailinglist. I was talking more about single blocks being available via HTTP - at the very moment when they have been mined. I think all you need is an URL - so it should be up to a peer to choose which URL to give you. As long as the hash of the data you download from there matches what you needed, you have no reason to question his method. Otherwise, just ban the bastard |
|
|
|
Sorry, but I think I can only help by implementing any of these new ideas in my client. I just don't like to have a boss and in my code I am the boss - hope you understand Anyway, if anyone wants to test a solution against a different client - just let me know. I can implement it, if it's not really crazy - and then we can test each other, benefiting everyone |
|
|
|
May I ask, why don't you just release the whole client at once? Do you have it already and testing - or not yet finished, but you think it will be finished soon? Unless it's not 'ask a question' type of topic, in which case: sorry, it will probably pop up my ignore digit |
|
|
|
In what language are you implementing the node? I've been having an idea of also writing one, perhaps in C, but who knows, I don't usually finish projects.
I have implemented it, in Go language: https://bitcointalk.org/index.php?topic=199306.0I thought the moto of the Go language is simple and efficient, will it even perform well? It is simple and efficient and it surely performs well. I would even risk a statement that it performs better than the current satoshi client, though it takes much more RAM, so we are talking about different abstraction layers here. Crypto operation are likely less efficient in Go, then what openssl gives, but my Go client only does them after checking a new highest block against the expected difficulty - so less often... But the only way to be sure how fast it actually is, is to check it out by yourself |
|
|
|
In what language are you implementing the node? I've been having an idea of also writing one, perhaps in C, but who knows, I don't usually finish projects.
I have implemented it, in Go language: https://bitcointalk.org/index.php?topic=199306.0 |
|
|
|
As some of you may have noticed, I have worked on my own implementation of a bitcoin node. During this process I found things, which I would like to point out, so maybe they could be improved in a future... Basically, they are all network bandwidth wasting related. 1. [EDIT: never mind] Maybe I have not verified it well enough, but I have an impression that the original client, whenever it sends "getblocks", it always asks as deep as possible - why to do this? It just creates unnecessary traffic. You can surely optimize it, without much effort. 2. [EDIT: never mind] IMO, you should also optimize the way you do "getdata". Don't just send getdata for all the block that you don't know, to all the peers at the same time - it's crazy. Instead, try to i.e. ask each node for a different block - one at a time, until you collect them all... 3. [EDIT: please, do mind] Last, but not least. The blocks are getting bigger, but there have been no improvements to the protocol, whatsoever. You cannot i.e. ask a peer for a part of a block - you just need to download the whole 1MB of it from a single IP. Moreover, each block has an exact hash, so it is just stupid that in times when even MtGox API goes through CloudFlare to save bandwidth, there is no solution that would allow a node to just download a block from an HTTP server, and so it is forced to leech it from the poor, mostly DSL armed, peers. The way I see it, the solution would be very simple: every mining pool can easily use CloudFlare (or any other cloud service) to serve blocks via HTTP. So if my node says "getdata ...", I do not necessarily mean that I want to have this megabyte of data from the poor node and its thin DSL connection. I would be more than happy to just get a URL, where I can download the block from - it surely would be faster, and would not drain the peer's uplink bandwidth that much. That's about it, but if I recall something more, I will surely append it to this topic. Also, please don't take my criticism personally - it's only meant as a feedback, pointing out areas that I think are important to improve, becasue the official bitcoin client is already eating up my internet connection more efficiently than BitTorrent, and that is very not cool |
|
|
|
First of all, I am pretty sure that the qt allows to import an externally generated private key (with the "importprivkey" RPC), so I don't quite understand why you say that sending from such would not be possible. Besides of that, my gocoin's wallet does such things. All you need to do is to place your (base58 encoded) private key in a text file called others.sec (in the same folder where you run the wallet app from). Then make sure that there is a recent "balance" folder (that contains unspent transactions belonging to your key) - and then you can spend coins from that address by calling: wallet -send 1btcAddressBlahBlaBla=amount |
|
|
|
OK - thanks a lot, man! That's all I needed to know. |
|
|
|
Indeed, it looks cool and simple - and it works! But can you please tell me what the "R*s" (in line 406) actually does? I mean, you have (decimals): Rx = 102145896445573563625240447116654222837109247557536823325858067433615090286321
Ry = 64919894836278270547560110097107560214300342546989031110129938591497073087260
s = 46415740558353013011708862292271156479711188487571029354677187424581448381078 ... and R*s gives you a point having: x = 112793881772482502863430761842017408792441979840968192252645857563994847441261
y = 47321320458075246750488099844078925876574705494449064910511016586200529015312 So how do I multiply a point by a number to get such a result? I mean, not using python.. |
|
|
|
Thanks! I will have a look.. I was hoping that it would actually be less complicated than what I see in bitcoind+openssl |
|
|
|
At some point (in ec_GFp_simple_set_compressed_coordinates function) I do have: x: e1d48751a24f0db604c86a4706d2113ff869808522e8af4a04f253f03ea87ef1
y: 8f875bbc497b680cfe288eeda4bfb1da78af6703afdfaaf580e9e656ad62531c ... but then, the function calls EC_POINT_set_affine_coordinates_GFp to "assign" these x,y values into the R - and after this the R's, x and y are: Rx : 7E677C42747BBDEF47AFAA51001D55A82D8C2B643EF74D6E014F64DCFD31BBA0
Ry : 0D1081A16E20A8839D892ACC384529CA4547D007B58559483991C3EB77DC71BC |
|
|
|
I'm finishing the code, it's coming in the following hours Here's what I get for your test: fb: 1b
recid: 0
r: e1d48751a24f0db604c86a4706d2113ff869808522e8af4a04f253f03ea87ef1
s: 669e60cfad73a7ffd63054dba4d0db88c628633ee90fcf0d620ff40de91c3296
Rx: e1d48751a24f0db604c86a4706d2113ff869808522e8af4a04f253f03ea87ef1
Ry: 8f875bbc497b680cfe288eeda4bfb1da78af6703afdfaaf580e9e656ad62531c Yeah - that was actually my first guess, that the "Rx" should be the same as the "r", while Ry should be calculated in the same way as we calc Y from 02||X, for a compressed public key. But after I put some debugs into bitcoin's key.cpp, it seems that the EC_POINT_set_compressed_coordinates_GFp that is called from there returns completely different X. |
|
|
|
May it be because it is a signature from a testnet address? bitcoind -testnet verifymessage mqMmY5Uc6AgWoemdbRsvkpTes5hF6p5d8w "H+HUh1GiTw22BMhqRwbSET/4aYCFIuivSgTyU/A+qH7xZp5gz61zp//WMFTbpNDbiMYoYz7pD88NYg/0DekcMpY=" test
true And this was a compressed key. Also I was giving the intermediate R - not the Q, that comes at the end. How to calc the Q - this was supposed to be my next question, because is seems to be even more screwed up And then, at the very end, they take the X and Y from the Q (that is already called a "public key"): Q.X: 6130AE7913286EC4D8296AEB7361420C259CD11478FB24FEDA77F81E256059AA
Q.Y: B3A6E7A704FCC0A59AC40BD364336A629483453CF39BB060F92354434B44F636 .. and turn it into an actual bitcoin public key, where: X: 05eb0c4f42ecf74ab8789f2855ef33cad6ac0ec1ba6b7179578cb9f218e7793d So how much more crazy could it be? I spent about 6 hours today trying to figure it out, before finally deciding to seek some help on the forum. |
|
|
|
I'm about to release a code for signing in Armory
If I understood correctly, it should solve your problem
OK - I'll be looking forward for it.. Cheers |
|
|
|
|
Show Posts
