|
Title: HTTP bootstrapping ? Post by: adulau on December 26, 2010, 10:00:38 AM Hi Everyone,
Reading a bit about the bootstrapping[1] of the P2P protocol, I was wondering why the client is not including by default a HTTP bootstrap as an alternative to IRC (often IRC is blocked or some antivirus/anti-malware products tag the Bitcoin application to be suspicious just because of the IRC traffic). For the test, I setup a page where I publish the IP seen by my Bitcoin client: http://btc.fo.vc/ (accessible in IPv4 and IPv6[2]) The IP addresses are collected with a simple script like this: netstat -an | grep 8333 | grep ESTA | awk '{print $5}' | cut -f1,2,3,4 -d"." > /tmp/bitcoin (date | awk '{print "# " $0 " Bitcoin clients seen"}') >>/tmp/bitcoin and push on the remote server. Until now, I didn't dig into the code of Bitcoin but I suppose including an alternative bootstrap to the IRC (and 8333) is not something unrealistic. Let me know what you think. adulau [1] http://www.bitcoin.org/wiki/doku.php?id=network [2] We never know, maybe Bitcoin will support IPv6 in the future Title: Re: HTTP bootstrapping ? Post by: grondilu on December 26, 2010, 10:48:54 AM The IP addresses are collected with a simple script like this: netstat -an | grep 8333 | grep ESTA | awk '{print $5}' | cut -f1,2,3,4 -d"." > /tmp/bitcoin (date | awk '{print "# " $0 " Bitcoin clients seen"}') >>/tmp/bitcoin Oh this is cool. I confess I didn't know the commant netstat. It could be cleaned a bit I think : netstat -an | awk '/8333/ && /ESTA/ { print $5 }' | sed 's/:8333//' | tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.") It's funny : while I was starting bitcoin to test this, I couldn't get any connection even after a few minutes. It's just when I read a post about IRC bootstrapping that I have problems which bootstrapping. How ironic... PS. You might consider adding a "Content-type" line in your script (assuming it's a CGI): Code: #!/bin/sh PS#2. I like this idea a lot. Especially since it's quite easy to install a mini http server such as thttpd for instance. This makes me even dream of a full implementation of bitcoin via pure shell scripting. The http server could also publish his blocks, that could be requested by giving the hash of the block via a simple HTTP GET request. Title: Re: HTTP bootstrapping ? Post by: adulau on December 26, 2010, 12:02:21 PM Oh this is cool. I confess I didn't know the commant netstat. It could be cleaned a bit I think : netstat -an | awk '/8333/ && /ESTA/ { print $5 }' | sed 's/:8333//' | tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.") Thanks for the cleanup. I have also updated the code to get the current remote IP of my Bitcoin client and added a correct Content-Type... Reading a bit the code of the Bitcoin client, the client is using a simple trick to know its remote IP via the IRC server (https://github.com/bitcoin/bitcoin/blob/master/irc.cpp#L333). That's why I added another script http://btc.fo.vc/getip to get your current remote IP and add it into the list and I'm finally sorting the IP addresses. Code: netstat -an | awk '/8333/ && /ESTA/ { print $5 }' The best would be to add an additional addip to merge all the IP announced by each clients with some minimal control to avoid non-Bitcoin client to announce faked IP addresses via HTTP. I think that seems reasonable to add HTTP bootstrapping into Bitcoin. Regarding the port used, it seems that the current client pushing the information on the IRC channel is also pushing the TCP port used. Is there a lot of Bitcoin clients using a different port than the standard one (8333)? PS#2. I like this idea a lot. Especially since it's quite easy to install a mini http server such as thttpd for instance. This makes me even dream of a full implementation of bitcoin via pure shell scripting. The http server could also publish his blocks, that could be requested by giving the hash of the block via a simple HTTP GET request. To have a full-blown Bitcoin client in shell scripting might be a bit difficult especially with the cryptographic aspect required (except if everything could be called from command line using OpenSSL...) but nothing is impossible. Have a nice day, Title: Re: HTTP bootstrapping ? Post by: grondilu on December 26, 2010, 12:20:47 PM To have a full-blown Bitcoin client in shell scripting might be a bit difficult especially with the cryptographic aspect required (except if everything could be called from command line using OpenSSL...) but nothing is impossible. Is it possible to use ECDSA with openssl ? I've looked for this in the openssl manual page, but I haven't found anything apart from DSA and RSA. Anyway indeed most of the parts of the program would be called with command line programs, especially cryptographic stuffs. (PS. you don't have to put a whole pipe in a same line. The '|' character can end a line.) Title: Re: HTTP bootstrapping ? Post by: adulau on December 26, 2010, 01:37:20 PM To have a full-blown Bitcoin client in shell scripting might be a bit difficult especially with the cryptographic aspect required (except if everything could be called from command line using OpenSSL...) but nothing is impossible. Is it possible to use ECDSA with openssl ? I've looked for this in the openssl manual page, but I haven't found anything apart from DSA and RSA. Anyway indeed most of the parts of the program would be called with command line programs, especially cryptographic stuffs. I think so to generate an EC key: Code: openssl ecparam -out ec_key.pem -name sect571k1 -genkey and for signing something like this: Code: openssl dgst -sha1 -sign ec_key.pem -out filetobesigned.txt.signed filetobesigned.txt and to verify the signature: Code: openssl dgst -sha1 -prverify ec_key.pem -signature filetobesigned.txt.signed filetobesigned.txt But I don't know the exact elliptic curves used by Bitcoin. You can get the one supported by OpenSSL by doing an: Code: openssl ecparam -list_curves Is there a table of the EC properties used by Bitcoin somewhere? I suppose the easiest is to read the source code... My reply (being out of scope of the HTTP bootstrapping) should be under your new post about the implementation of a Bitcoin client in shell scripting (http://bitcointalk.org/index.php?topic=2461.0). Hope this helps a little bit, (PS. you don't have to put a whole pipe in a same line. The '|' character can end a line.) I know but I suppose this shows my laziness when doing a paste into the forum ;-) Title: Re: HTTP bootstrapping ? Post by: jgarzik on December 26, 2010, 03:34:18 PM bitcoin maintains a database of P2P addresses. Obtaining addresses via netstat is rather sub-optimal, when you could use bitcointools (https://github.com/gavinandresen/bitcointools) to extract addresses directly from the bitcoin database.
As to the larger point... HTTP and DNS bootstrapping should be pursued. Much more efficient than IRC. Title: Re: HTTP bootstrapping ? Post by: ShadowOfHarbringer on December 26, 2010, 04:12:19 PM bitcoin maintains a database of P2P addresses. Obtaining addresses via netstat is rather sub-optimal, when you could use bitcointools (https://github.com/gavinandresen/bitcointools) to extract addresses directly from the bitcoin database. As to the larger point... HTTP and DNS bootstrapping should be pursued. Much more efficient than IRC. Oh my, dat is soo awsum. +10 to dis idea. Can we has dis idea implemented in mainstream client, plz ? I mean how could anybody refuse this soft fluffy little lolcat ? http://www.threadbombing.com/data/media/19/picdump-19.jpg Title: Re: HTTP bootstrapping ? Post by: SmokeTooMuch on December 26, 2010, 04:19:25 PM I'm not sure how up-to-date they are, but why not adding these IP's to your (or a seperate) list:
http://bitcointalk.org/index.php?topic=59.0 (http://bitcointalk.org/index.php?topic=59.0) (make sure to read these two posts about fallbacl-nodes: http://bitcointalk.org/index.php?topic=59.msg14646#msg14646 (http://bitcointalk.org/index.php?topic=59.msg14646#msg14646), http://bitcointalk.org/index.php?topic=59.msg31133#msg31133 (http://bitcointalk.org/index.php?topic=59.msg31133#msg31133)) Title: Re: HTTP bootstrapping ? Post by: grondilu on December 26, 2010, 06:30:49 PM But I don't know the exact elliptic curves used by Bitcoin. You can get the one supported by OpenSSL by doing an: Code: openssl ecparam -list_curves Is there a table of the EC properties used by Bitcoin somewhere? I suppose the easiest is to read the source code... Indeed you have to look at the source code. I've just check and the EC curve used is : secp256k1, which is in the list given by openssl. I think a scripted implementation is feasable. Title: fallback nodes, and DNS bootstrapping Post by: jgarzik on December 26, 2010, 07:37:04 PM Only a very few fallback nodes are persistent over time, and compiled (hardcoded) into the bitcoin client itself.
https://en.bitcoin.it/wiki/Fallback_Nodes is a viable method of bootstrapping. We'll call that "forum bootstrapping" or "wiki bootstrapping", where one must manually search for a list of nodes, in order to bootstrap onto the network. I think DNS bootstrapping would be the most efficient: a simple DNS lookup to bootstrap.bitcoin.org would work like this:
That would be very, very fast. Much faster than IRC. This is similar to how BitTorrent DHT bootstrapping occurs. The only issue is trust (rogue DNS servers), but this issue also exists with the IRC server, which is a Single Point of Failure (SPOF) for both trust and general reliability. Title: Re: fallback nodes, and DNS bootstrapping Post by: bitcoinex on December 26, 2010, 07:54:24 PM Only a very few fallback nodes are persistent over time, and compiled (hardcoded) into the bitcoin client itself. This is security hole. State can block outgoing 8333 port for all hosts except for hardcoded addresses. On hardcoded addresses they can set up fake bitcoin nodes. Then these nodes will give you the addresses of a dummy nodes to create the illusion of a bitcoin network. So they are force you to spend your own money from a bitcoin client in their favor. Better in the case of network problems ask the user to specify the address for the bootstrap. This address can be obtained from reliable sources, verified by the user. Title: Re: fallback nodes, and DNS bootstrapping Post by: ShadowOfHarbringer on December 26, 2010, 09:14:38 PM This is security hole. Not necessarily. You can place blockchain bootstraps in compressed *.zip or *.tar.gz files, and hardcode just multiple (RMD160, SHA1, SHA256 + Filezize) hashes of backups into mainstream client. OR, updated hashes of blockchain bootstraps can be avaiable for download from main bitcoin server over https, in which case it will be impossible to fake them (but this is a centralized solution, so probably not very good). Possibilities are endless. Title: Re: HTTP bootstrapping ? Post by: adulau on December 26, 2010, 09:19:37 PM bitcoin maintains a database of P2P addresses. Obtaining addresses via netstat is rather sub-optimal, when you could use bitcointools (https://github.com/gavinandresen/bitcointools) to extract addresses directly from the bitcoin database. As to the larger point... HTTP and DNS bootstrapping should be pursued. Much more efficient than IRC. Right, that's why I was pursuing on that way. By the way, I made a test with bitcointools to dump the address out of the database : Code: python2.7 dbdump.py --datadir ~/.bitcoin/ --address Even if the netstat approach could be suboptimal, there is an advantage over relying on the addr dump from the database. If you get the address from the TCP Established session, these are really the active Bitcoin clients and from the database, you are guessing out of the lastseen information where you have already a lot of dead addresses. And picking the appropriate time delta can be tricky except if there is already something in the database structure to just list the active ones. On the other hand, the Berkeley database need to be only accessed by one process at a time and you need to shutdown the existing the current database. Maybe another appropriate way might be to read the addr message passing over the TCP sessions (using pcap) and extract the addresses and publish that stream to the HTTP/DNS directory. What's the most appropriate techniques to get the currently active Bitcoin addresses? Title: Re: fallback nodes, and DNS bootstrapping Post by: theymos on December 27, 2010, 01:59:32 AM So they are force you to spend your own money from a bitcoin client in their favor. They could double-spend transactions to you, but they couldn't redirect your transactions from one Bitcoin address to another one. There are much easier ways to "surround" someone if you control the ISP. Seednode bootstrapping is used in Tor, I2P, GNUnet, and Freenet. Just removing IRC and using the already-implemented seednode system will work fine. Reading a bit the code of the Bitcoin client, the client is using a simple trick to know its remote IP via the IRC server (https://github.com/bitcoin/bitcoin/blob/master/irc.cpp#L333). That's just one method of finding your external IP. There are also two HTTP external IP services. Title: Re: HTTP bootstrapping ? Post by: pj on December 27, 2010, 08:54:39 AM Quote It could be cleaned a bit I think : Good ideas ... I'd like to tweak this a tad more.netstat -an | awk '/8333/ && /ESTA/ { print $5 }' | sed 's/:8333//' | tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.") The ">(process)" construct is not recognized by classic Bourne shells and similar. So far as I know, the redirection to a file, such as to /tmp/bitcoin in the original post, is required for some such shells. One more command, the sed, can be removed by using a little more awk. That (necessary for some shells) tmp file /tmp/bitcoin should be made unique and self-removing The naked search for "8333" would pick up ports 18333, 28333 ... 58333 as well. Prefix with a colon ':' to avoid that. The result is Code: #!/bin/sh Is this correct, that you want both IP addresses coming from remote port 8333 and coming into local port 8333? Or do you just want IP addresses coming into local port 8333? If the later, change the key line above to look for /:8333/ only in field $4, the local address. Code: netstat -an | awk '$6 == "ESTABLISHED" && $4 ~ /:8333/ { split($5, a, ":"); print a[1]}' | tee $tTitle: Re: HTTP bootstrapping ? Post by: grondilu on December 27, 2010, 09:16:08 AM The result is Code: #!/bin/sh Didn't know about the trap command. I doubt we need it though. Being a bit anal : - The standard way to create a temp file is to use the mktemp command. - You can end lines after |. This makes the code clearer. Code: #!/bin/sh Title: Re: HTTP bootstrapping ? Post by: pj on December 27, 2010, 12:57:50 PM Code: t="$(mktemp -t bitcoin)" And I think you really do want the trap - otherwise your /tmp directory will get filled up with these dang files. Yes -- newlines separating each piped command are better (though I prefer to indent all but the first one) -- I was being lazy and just typing as I do at the command prompt. Yes -- mktemp or the more recent tempfile are probably better. I was just being lazy again, and doing it as I have done it for 30 years, long before those commands existed. Sorry. The main problem with my old fashioned method, and even with mktemp, is a security issue -- a hacker can get you to write a file that they have setup, via a symlink that you thought was your file. The main problem with mktemp and tempfile is that not all systems have them (though you have to be on a fairly old, odd, or barebones system not to have them.) You can find more discussion of the temp file issue at: http://www.linuxsecurity.com/content/view/115462/151/ Safely Creating Temporary Files in Shell Scripts So ... all this suggests the following: Code: #!/bin/sh Title: Re: HTTP bootstrapping ? Post by: pj on December 27, 2010, 01:43:27 PM I worry a bit that what might have been my most important question above could have gotten
lost in the code refinement discussion. So I'll ask it again: Quote Is this correct, that you want both IP addresses coming from remote port 8333 and coming into local port 8333? Or do you just want IP addresses coming into local port 8333? If the later, change the key line above to look for /:8333/ only in field $4, the local address. My hunch is that we just want IP addresses coming into our port 8333. Whether or not a connection is coming from port on 8333 on some other system means nothing to us, as best as I can figure. Title: Re: HTTP bootstrapping ? Post by: grondilu on December 27, 2010, 03:03:22 PM Let's just make the final count with awk too...
Code: #!/bin/sh Title: Re: HTTP bootstrapping ? Post by: pj on December 28, 2010, 12:20:37 PM Let's just make the final count with awk too... Duh! Excellent. Thanks.Title: Re: HTTP bootstrapping ? Post by: adulau on December 28, 2010, 02:06:45 PM Let's just make the final count with awk too... Code: #!/bin/sh Just to be a little bit picky, the netstat output is slightly different between BSD-like Unix and GNU/Linux. The port is separated by a dot on the BSD-like Unix. So maybe the pattern matching /:8333/ could be reviewed to include also the other output... but beside that, this is just fine. Title: Re: HTTP bootstrapping ? Post by: pj on December 28, 2010, 02:15:24 PM Just to be a little bit picky, the netstat output is slightly different between BSD-like Unix and GNU/Linux. The port is separated by a dot on the BSD-like Unix. So maybe the pattern matching /:8333/ could be reviewed to include also the other output... but beside that, this is just fine. Could you provide a sample few lines of output from a BSD netstat -an? It would take more than changing the /:8333/ pattern to fix this, if I understand your description correctly. There is also the awk split on the ":" which would have to be fixed as well. This is all doable with a little bit of regular expression hacking (something I do easily.) But I should see the exact BSD netstat -an output first, to be sure I understand it correctly. Title: Re: HTTP bootstrapping ? Post by: adulau on December 28, 2010, 02:35:43 PM Just to be a little bit picky, the netstat output is slightly different between BSD-like Unix and GNU/Linux. The port is separated by a dot on the BSD-like Unix. So maybe the pattern matching /:8333/ could be reviewed to include also the other output... but beside that, this is just fine. Could you provide a sample few lines of output from a BSD netstat -an? It would take more than changing the /:8333/ pattern to fix this, if I understand your description correctly. There is also the awk split on the ":" which would have to be fixed as well. This is all doable with a little bit of regular expression hacking (something I do easily.) But I should see the exact BSD netstat -an output first, to be sure I understand it correctly. I think this could do the trick in awk for the matching : && (/:8333/ || /\.8333/) and for the split, an if block to match the : and another if block to split on dot. It will start to be unreadable for an one-liner ;-) FYI, here is output: Code: tcp4 0 116 192.168.1.2.8333 80.217.82.59.45167 ESTABLISHED Title: Re: HTTP bootstrapping ? Post by: pj on December 28, 2010, 02:53:55 PM Ok - thanks for the BSD sample output.
How about this code then: Code: #!/bin/sh Title: Re: HTTP bootstrapping ? Post by: grondilu on December 28, 2010, 03:14:55 PM Ok - thanks for the BSD sample output. How about this code then: Code: #!/bin/sh omg your regex is ugly. Code: sed -r 's/([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\.([0-9]+)/\1:\2/g' | And I'm pretty sure there is better. Title: Re: HTTP bootstrapping ? Post by: pj on December 28, 2010, 03:34:28 PM Code: ... omg your regex is ugly. Code: sed -r 's/([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\.([0-9]+)/\1:\2/g' | And I'm pretty sure there is better. If one uses sed -r, then I suppose a better (well, shorter anyway) expression would be: Code: sed -r 's/(([0-9]+\.){3}[0-9])\.([0-9])/\1:\3/g' |
