adulau (OP)
Newbie
Offline
Activity: 12
Merit: 0
|
|
December 26, 2010, 10:00:38 AM |
|
Hi Everyone, Reading a bit about the bootstrapping[1] of the P2P protocol, I was wondering why the client is not including by default a HTTP bootstrap as an alternative to IRC (often IRC is blocked or some antivirus/anti-malware products tag the Bitcoin application to be suspicious just because of the IRC traffic). For the test, I setup a page where I publish the IP seen by my Bitcoin client: http://btc.fo.vc/ (accessible in IPv4 and IPv6[2]) The IP addresses are collected with a simple script like this: netstat -an | grep 8333 | grep ESTA | awk '{print $5}' | cut -f1,2,3,4 -d"." > /tmp/bitcoin (date | awk '{print "# " $0 " Bitcoin clients seen"}') >>/tmp/bitcoinand push on the remote server. Until now, I didn't dig into the code of Bitcoin but I suppose including an alternative bootstrap to the IRC (and 8333) is not something unrealistic. Let me know what you think. adulau [1] http://www.bitcoin.org/wiki/doku.php?id=network[2] We never know, maybe Bitcoin will support IPv6 in the future
|
|
|
|
grondilu
Legendary
Offline
Activity: 1288
Merit: 1080
|
|
December 26, 2010, 10:48:54 AM Last edit: December 26, 2010, 11:18:13 AM by grondilu |
|
The IP addresses are collected with a simple script like this:
netstat -an | grep 8333 | grep ESTA | awk '{print $5}' | cut -f1,2,3,4 -d"." > /tmp/bitcoin (date | awk '{print "# " $0 " Bitcoin clients seen"}') >>/tmp/bitcoin
Oh this is cool. I confess I didn't know the commant netstat. It could be cleaned a bit I think : netstat -an | awk '/8333/ && /ESTA/ { print $5 }' | sed 's/:8333//' | tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.")
It's funny : while I was starting bitcoin to test this, I couldn't get any connection even after a few minutes. It's just when I read a post about IRC bootstrapping that I have problems which bootstrapping. How ironic... PS. You might consider adding a "Content-type" line in your script (assuming it's a CGI): #!/bin/sh echo "Content-type: text/plain" echo netstat -an | awk '/8333/ && /ESTA/ { print $5 }' | sed 's/:8333//' | tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.")
PS#2. I like this idea a lot. Especially since it's quite easy to install a mini http server such as thttpd for instance. This makes me even dream of a full implementation of bitcoin via pure shell scripting. The http server could also publish his blocks, that could be requested by giving the hash of the block via a simple HTTP GET request.
|
|
|
|
adulau (OP)
Newbie
Offline
Activity: 12
Merit: 0
|
|
December 26, 2010, 12:02:21 PM |
|
Oh this is cool. I confess I didn't know the commant netstat.
It could be cleaned a bit I think :
netstat -an | awk '/8333/ && /ESTA/ { print $5 }' | sed 's/:8333//' | tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.")
Thanks for the cleanup. I have also updated the code to get the current remote IP of my Bitcoin client and added a correct Content-Type... Reading a bit the code of the Bitcoin client, the client is using a simple trick to know its remote IP via the IRC server ( https://github.com/bitcoin/bitcoin/blob/master/irc.cpp#L333). That's why I added another script http://btc.fo.vc/getip to get your current remote IP and add it into the list and I'm finally sorting the IP addresses. netstat -an | awk '/8333/ && /ESTA/ { print $5 }' | cut -d. -f1,2,3,4 | (tee -a >(curl -s http://btc.fo.vc/getip)) | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n | (tee >(echo "# $(date) $(wc -l) Bitcoin clients seen."))
The best would be to add an additional addip to merge all the IP announced by each clients with some minimal control to avoid non-Bitcoin client to announce faked IP addresses via HTTP. I think that seems reasonable to add HTTP bootstrapping into Bitcoin. Regarding the port used, it seems that the current client pushing the information on the IRC channel is also pushing the TCP port used. Is there a lot of Bitcoin clients using a different port than the standard one (8333)? PS#2. I like this idea a lot. Especially since it's quite easy to install a mini http server such as thttpd for instance. This makes me even dream of a full implementation of bitcoin via pure shell scripting. The http server could also publish his blocks, that could be requested by giving the hash of the block via a simple HTTP GET request.
To have a full-blown Bitcoin client in shell scripting might be a bit difficult especially with the cryptographic aspect required (except if everything could be called from command line using OpenSSL...) but nothing is impossible. Have a nice day,
|
|
|
|
grondilu
Legendary
Offline
Activity: 1288
Merit: 1080
|
|
December 26, 2010, 12:20:47 PM |
|
To have a full-blown Bitcoin client in shell scripting might be a bit difficult especially with the cryptographic aspect required (except if everything could be called from command line using OpenSSL...) but nothing is impossible.
Is it possible to use ECDSA with openssl ? I've looked for this in the openssl manual page, but I haven't found anything apart from DSA and RSA. Anyway indeed most of the parts of the program would be called with command line programs, especially cryptographic stuffs. (PS. you don't have to put a whole pipe in a same line. The '|' character can end a line.)
|
|
|
|
adulau (OP)
Newbie
Offline
Activity: 12
Merit: 0
|
|
December 26, 2010, 01:37:20 PM |
|
To have a full-blown Bitcoin client in shell scripting might be a bit difficult especially with the cryptographic aspect required (except if everything could be called from command line using OpenSSL...) but nothing is impossible.
Is it possible to use ECDSA with openssl ? I've looked for this in the openssl manual page, but I haven't found anything apart from DSA and RSA. Anyway indeed most of the parts of the program would be called with command line programs, especially cryptographic stuffs. I think so to generate an EC key: openssl ecparam -out ec_key.pem -name sect571k1 -genkey
and for signing something like this: openssl dgst -sha1 -sign ec_key.pem -out filetobesigned.txt.signed filetobesigned.txt
and to verify the signature: openssl dgst -sha1 -prverify ec_key.pem -signature filetobesigned.txt.signed filetobesigned.txt
But I don't know the exact elliptic curves used by Bitcoin. You can get the one supported by OpenSSL by doing an: openssl ecparam -list_curves
Is there a table of the EC properties used by Bitcoin somewhere? I suppose the easiest is to read the source code... My reply (being out of scope of the HTTP bootstrapping) should be under your new post about the implementation of a Bitcoin client in shell scripting (http://bitcointalk.org/index.php?topic=2461.0).Hope this helps a little bit, (PS. you don't have to put a whole pipe in a same line. The '|' character can end a line.)
I know but I suppose this shows my laziness when doing a paste into the forum ;-)
|
|
|
|
jgarzik
Legendary
Offline
Activity: 1596
Merit: 1099
|
|
December 26, 2010, 03:34:18 PM |
|
bitcoin maintains a database of P2P addresses. Obtaining addresses via netstat is rather sub-optimal, when you could use bitcointools to extract addresses directly from the bitcoin database. As to the larger point... HTTP and DNS bootstrapping should be pursued. Much more efficient than IRC.
|
Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own. Visit bloq.com / metronome.io Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
|
|
|
ShadowOfHarbringer
Legendary
Offline
Activity: 1470
Merit: 1006
Bringing Legendary Har® to you since 1952
|
|
December 26, 2010, 04:12:19 PM |
|
bitcoin maintains a database of P2P addresses. Obtaining addresses via netstat is rather sub-optimal, when you could use bitcointools to extract addresses directly from the bitcoin database. As to the larger point... HTTP and DNS bootstrapping should be pursued. Much more efficient than IRC. Oh my, dat is soo awsum. +10 to dis idea. Can we has dis idea implemented in mainstream client, plz ? I mean how could anybody refuse this soft fluffy little lolcat ?
|
|
|
|
SmokeTooMuch
Legendary
Offline
Activity: 860
Merit: 1026
|
|
December 26, 2010, 04:19:25 PM |
|
|
|
|
|
grondilu
Legendary
Offline
Activity: 1288
Merit: 1080
|
|
December 26, 2010, 06:30:49 PM |
|
But I don't know the exact elliptic curves used by Bitcoin. You can get the one supported by OpenSSL by doing an: openssl ecparam -list_curves
Is there a table of the EC properties used by Bitcoin somewhere? I suppose the easiest is to read the source code... Indeed you have to look at the source code. I've just check and the EC curve used is : secp256k1, which is in the list given by openssl. I think a scripted implementation is feasable.
|
|
|
|
jgarzik
Legendary
Offline
Activity: 1596
Merit: 1099
|
|
December 26, 2010, 07:37:04 PM |
|
Only a very few fallback nodes are persistent over time, and compiled (hardcoded) into the bitcoin client itself. https://en.bitcoin.it/wiki/Fallback_Nodes is a viable method of bootstrapping. We'll call that "forum bootstrapping" or "wiki bootstrapping", where one must manually search for a list of nodes, in order to bootstrap onto the network. I think DNS bootstrapping would be the most efficient: a simple DNS lookup to bootstrap.bitcoin.org would work like this: - Community members post their nameserver (NS) records for bootstrap.bitcoin.org on the forum. Presumably this list does not change often
- Each member runs a DNS server, independently of anyone else, that retrieves addresses from bitcoin's addr.dat database, randomly selects "fresh" P2P nodes, and stores these in A records or SRV records.
- When bootstrapping, the bitcoin client performs a standard DNS lookup for bootstrap.bitcoin.org
That would be very, very fast. Much faster than IRC. This is similar to how BitTorrent DHT bootstrapping occurs. The only issue is trust (rogue DNS servers), but this issue also exists with the IRC server, which is a Single Point of Failure (SPOF) for both trust and general reliability.
|
Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own. Visit bloq.com / metronome.io Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
|
|
|
bitcoinex
|
|
December 26, 2010, 07:54:24 PM |
|
Only a very few fallback nodes are persistent over time, and compiled (hardcoded) into the bitcoin client itself.
This is security hole. State can block outgoing 8333 port for all hosts except for hardcoded addresses. On hardcoded addresses they can set up fake bitcoin nodes. Then these nodes will give you the addresses of a dummy nodes to create the illusion of a bitcoin network. So they are force you to spend your own money from a bitcoin client in their favor. Better in the case of network problems ask the user to specify the address for the bootstrap. This address can be obtained from reliable sources, verified by the user.
|
New bitcoin lottery: probiwon.com- Moжeт, ты eщё и в Heвидимyю Pyкy Pынкa вepyeшь? - Зaчeм жe вepoвaть в тo, чтo мoжнo нaблюдaть нeпocpeдcтвeннo?
|
|
|
ShadowOfHarbringer
Legendary
Offline
Activity: 1470
Merit: 1006
Bringing Legendary Har® to you since 1952
|
|
December 26, 2010, 09:14:38 PM |
|
This is security hole.
Not necessarily. You can place blockchain bootstraps in compressed *.zip or *.tar.gz files, and hardcode just multiple (RMD160, SHA1, SHA256 + Filezize) hashes of backups into mainstream client. OR, updated hashes of blockchain bootstraps can be avaiable for download from main bitcoin server over https, in which case it will be impossible to fake them (but this is a centralized solution, so probably not very good). Possibilities are endless.
|
|
|
|
adulau (OP)
Newbie
Offline
Activity: 12
Merit: 0
|
|
December 26, 2010, 09:19:37 PM |
|
bitcoin maintains a database of P2P addresses. Obtaining addresses via netstat is rather sub-optimal, when you could use bitcointools to extract addresses directly from the bitcoin database. As to the larger point... HTTP and DNS bootstrapping should be pursued. Much more efficient than IRC. Right, that's why I was pursuing on that way. By the way, I made a test with bitcointools to dump the address out of the database : python2.7 dbdump.py --datadir ~/.bitcoin/ --address
...155.6:36128 (lastseen: Sat Dec 18 21:09:42 2010) 68.52.60.203:36128 (lastseen: Sun Dec 26 15:28:48 2010) 68.53.17.115:36128 (lastseen: Thu Dec 16 18:56:57 2010) 68.56.241.235:36128 (lastseen: Sun Dec 26 17:32:34 2010) 68.62.250.145:36128 (lastseen: Sun Dec 26 15:39:33 2010) ....
Even if the netstat approach could be suboptimal, there is an advantage over relying on the addr dump from the database. If you get the address from the TCP Established session, these are really the active Bitcoin clients and from the database, you are guessing out of the lastseen information where you have already a lot of dead addresses. And picking the appropriate time delta can be tricky except if there is already something in the database structure to just list the active ones. On the other hand, the Berkeley database need to be only accessed by one process at a time and you need to shutdown the existing the current database. Maybe another appropriate way might be to read the addr message passing over the TCP sessions (using pcap) and extract the addresses and publish that stream to the HTTP/DNS directory. What's the most appropriate techniques to get the currently active Bitcoin addresses?
|
|
|
|
theymos
Administrator
Legendary
Offline
Activity: 5334
Merit: 13307
|
|
December 27, 2010, 01:59:32 AM |
|
So they are force you to spend your own money from a bitcoin client in their favor.
They could double-spend transactions to you, but they couldn't redirect your transactions from one Bitcoin address to another one. There are much easier ways to "surround" someone if you control the ISP. Seednode bootstrapping is used in Tor, I2P, GNUnet, and Freenet. Just removing IRC and using the already-implemented seednode system will work fine. That's just one method of finding your external IP. There are also two HTTP external IP services.
|
1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD
|
|
|
pj
Newbie
Offline
Activity: 24
Merit: 0
|
|
December 27, 2010, 08:54:39 AM |
|
It could be cleaned a bit I think :
netstat -an | awk '/8333/ && /ESTA/ { print $5 }' | sed 's/:8333//' | tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.")
Good ideas ... I'd like to tweak this a tad more. The ">(process)" construct is not recognized by classic Bourne shells and similar. So far as I know, the redirection to a file, such as to /tmp/bitcoin in the original post, is required for some such shells. One more command, the sed, can be removed by using a little more awk. That (necessary for some shells) tmp file /tmp/bitcoin should be made unique and self-removing The naked search for "8333" would pick up ports 18333, 28333 ... 58333 as well. Prefix with a colon ':' to avoid that. The result is #!/bin/sh # Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333. # Append line at end with date and count of addresses displayed.
t=/tmp/bitcoin.$$ trap 'rm -f $t; trap 0; exit' 0 1 2 3 15
netstat -an | awk '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1]}' | tee $t echo "# $(date) $(wc -l < $t) Bitcoin clients seen."
Is this correct, that you want both IP addresses coming from remote port 8333 and coming into local port 8333? Or do you just want IP addresses coming into local port 8333? If the later, change the key line above to look for /:8333/ only in field $4, the local address. netstat -an | awk '$6 == "ESTABLISHED" && $4 ~ /:8333/ { split($5, a, ":"); print a[1]}' | tee $t
|
|
|
|
grondilu
Legendary
Offline
Activity: 1288
Merit: 1080
|
|
December 27, 2010, 09:16:08 AM Last edit: December 27, 2010, 01:05:01 PM by grondilu |
|
The result is #!/bin/sh # Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333. # Append line at end with date and count of addresses displayed.
t=/tmp/bitcoin.$$ trap 'rm -f $t; trap 0; exit' 0 1 2 3 15
netstat -an | awk '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1]}' | tee $t echo "# $(date) $(wc -l < $t) Bitcoin clients seen."
Didn't know about the trap command. I doubt we need it though. Being a bit anal : - The standard way to create a temp file is to use the mktemp command. - You can end lines after |. This makes the code clearer. #!/bin/sh # Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333. # Append line at end with date and count of addresses displayed.
t="$(mktemp -t bitcoin)"
netstat -an | awk '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1]}' | tee "$t"
echo "# $(date) $(wc -l < $t) Bitcoin clients seen."
|
|
|
|
pj
Newbie
Offline
Activity: 24
Merit: 0
|
|
December 27, 2010, 12:57:50 PM |
|
That doesn't work so well -- no XXX's in the mktemp -t template. And I think you really do want the trap - otherwise your /tmp directory will get filled up with these dang files. Yes -- newlines separating each piped command are better (though I prefer to indent all but the first one) -- I was being lazy and just typing as I do at the command prompt. Yes -- mktemp or the more recent tempfile are probably better. I was just being lazy again, and doing it as I have done it for 30 years, long before those commands existed. Sorry. The main problem with my old fashioned method, and even with mktemp, is a security issue -- a hacker can get you to write a file that they have setup, via a symlink that you thought was your file. The main problem with mktemp and tempfile is that not all systems have them (though you have to be on a fairly old, odd, or barebones system not to have them.) You can find more discussion of the temp file issue at: http://www.linuxsecurity.com/content/view/115462/151/ Safely Creating Temporary Files in Shell Scripts So ... all this suggests the following: #!/bin/sh # Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333. # Append line at end with date and count of addresses displayed.
t="$(tempfile -p bitcoin)" trap 'rm -f $t; trap 0; exit' 0 1 2 3 15
netstat -an | awk '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1]}' | tee "$t"
echo "# $(date) $(wc -l < $t) Bitcoin clients seen."
|
|
|
|
pj
Newbie
Offline
Activity: 24
Merit: 0
|
|
December 27, 2010, 01:43:27 PM |
|
I worry a bit that what might have been my most important question above could have gotten lost in the code refinement discussion. So I'll ask it again: Is this correct, that you want both IP addresses coming from remote port 8333 and coming into local port 8333? Or do you just want IP addresses coming into local port 8333? If the later, change the key line above to look for /:8333/ only in field $4, the local address. My hunch is that we just want IP addresses coming into our port 8333. Whether or not a connection is coming from port on 8333 on some other system means nothing to us, as best as I can figure.
|
|
|
|
grondilu
Legendary
Offline
Activity: 1288
Merit: 1080
|
|
December 27, 2010, 03:03:22 PM Last edit: December 28, 2010, 09:19:03 AM by grondilu |
|
Let's just make the final count with awk too... #!/bin/sh # Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333. # Append line at end with date and count of addresses displayed.
netstat -an | awk -v date="$(date)" '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1] ; n++ } END { print "# " date " : " n " bitcoin clients seen." }'
|
|
|
|
pj
Newbie
Offline
Activity: 24
Merit: 0
|
|
December 28, 2010, 12:20:37 PM |
|
Let's just make the final count with awk too... Duh! Excellent. Thanks.
|
|
|
|
|