HTTP bootstrapping ?

[adulau]

Hi Everyone,

Reading a bit about the bootstrapping[1] of the P2P protocol, I was wondering
why the client is not including by default a HTTP bootstrap as an alternative to IRC (often
IRC is blocked or some antivirus/anti-malware products tag the Bitcoin application
to be suspicious just because of the IRC traffic).

For the test, I setup a page where I publish the IP seen by my Bitcoin client:

http://btc.fo.vc/ (accessible in IPv4 and IPv6[2])

The IP addresses are collected with a simple script like this:

netstat -an | grep 8333 | grep ESTA | awk '{print $5}' | cut -f1,2,3,4 -d"." > /tmp/bitcoin
(date | awk '{print "# " $0 " Bitcoin clients seen"}') >>/tmp/bitcoin

and push on the remote server.

Until now, I didn't dig into the code of Bitcoin but I suppose including an alternative
bootstrap to the IRC  (and 8333) is not something unrealistic.

Let me know what you think.

adulau

[1] http://www.bitcoin.org/wiki/doku.php?id=network
[2] We never know, maybe Bitcoin will support IPv6 in the future

[1714056872]

1714056872

[grondilu]

The IP addresses are collected with a simple script like this:

netstat -an | grep 8333 | grep ESTA | awk '{print $5}' | cut -f1,2,3,4 -d"." > /tmp/bitcoin
(date | awk '{print "# " $0 " Bitcoin clients seen"}') >>/tmp/bitcoin

Oh this is cool. I confess I didn't know the commant netstat.

It could be cleaned a bit I think :

netstat -an |
awk '/8333/ && /ESTA/ { print $5 }' |
sed 's/:8333//' |
tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.")



It's funny :  while I was starting bitcoin to test this,  I couldn't get any connection even after a few minutes.  It's just when I read a post about IRC bootstrapping that I have problems which bootstrapping.  How ironic...

PS.  You might consider adding a "Content-type" line in your script (assuming it's a CGI):

Code:

#!/bin/sh
echo "Content-type:	text/plain"
echo
netstat -an |
awk '/8333/ && /ESTA/ { print $5 }' |
sed 's/:8333//' |
tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.")

PS#2.  I like this idea a lot.  Especially since it's quite easy to install a mini http server such as thttpd for instance.
This makes me even dream of a full implementation of bitcoin via pure shell scripting.  The http server could also publish his blocks, that could be requested by giving the hash of the block via a simple HTTP GET request.

[adulau]

Oh this is cool. I confess I didn't know the commant netstat.

It could be cleaned a bit I think :

netstat -an |
awk '/8333/ && /ESTA/ { print $5 }' |
sed 's/:8333//' |
tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.")

Thanks for the cleanup.  I have also updated the code to get the current remote IP of my Bitcoin client
and added a correct Content-Type...

Reading a bit the code of the Bitcoin client, the client is using a simple trick to know its remote IP via
the IRC server (https://github.com/bitcoin/bitcoin/blob/master/irc.cpp#L333).

That's why I added another script http://btc.fo.vc/getip to get your current remote IP and add it
into the list and I'm finally sorting the IP addresses.

Code:

netstat -an | awk '/8333/ && /ESTA/ { print $5 }' 
| cut -d. -f1,2,3,4  | (tee -a >(curl -s http://btc.fo.vc/getip))
| sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n | (tee >(echo "# $(date) $(wc -l) Bitcoin clients seen."))

The best would be to add an additional addip to merge all the IP announced by each clients with
some minimal control to avoid non-Bitcoin client to announce faked IP addresses via HTTP.

I think that seems reasonable to add HTTP bootstrapping into Bitcoin. Regarding the port used,
it seems that the current client pushing the information on the IRC channel is also pushing the
TCP port used. Is there a lot of Bitcoin clients using a different port than the standard one (8333)?

PS#2.  I like this idea a lot.  Especially since it's quite easy to install a mini http server such as thttpd for instance.
This makes me even dream of a full implementation of bitcoin via pure shell scripting.  The http server could also publish his blocks, that could be requested by giving the hash of the block via a simple HTTP GET request.

To have a full-blown Bitcoin client in shell scripting might be a bit difficult especially with the cryptographic aspect required (except if everything could be called from command line using OpenSSL...) but nothing is impossible.

Have a nice day,

[grondilu]

To have a full-blown Bitcoin client in shell scripting might be a bit difficult especially with the cryptographic aspect required (except if everything could be called from command line using OpenSSL...) but nothing is impossible.

Is it possible to use ECDSA with openssl ?  I've looked for this in the openssl manual page, but I haven't found anything apart from DSA and RSA.  Anyway indeed most of the parts of the program would be called with command line programs, especially cryptographic stuffs.

(PS.  you don't have to put a whole pipe in a same line.  The '|' character can end a line.)

[adulau]

To have a full-blown Bitcoin client in shell scripting might be a bit difficult especially with the cryptographic aspect required (except if everything could be called from command line using OpenSSL...) but nothing is impossible.

Is it possible to use ECDSA with openssl ?  I've looked for this in the openssl manual page, but I haven't found anything apart from DSA and RSA.  Anyway indeed most of the parts of the program would be called with command line programs, especially cryptographic stuffs.

I think so to generate an EC key:

Code:

openssl ecparam -out ec_key.pem -name sect571k1  -genkey

and for signing something like this:

Code:

openssl dgst -sha1 -sign ec_key.pem -out filetobesigned.txt.signed filetobesigned.txt

and to verify the signature:

Code:

openssl dgst -sha1 -prverify ec_key.pem -signature filetobesigned.txt.signed filetobesigned.txt

But I don't know the exact elliptic curves used by Bitcoin. You can get the one supported
by OpenSSL by doing an:

Code:

openssl ecparam -list_curves

Is there a table of the EC properties used by Bitcoin somewhere? I suppose the easiest is
to read the source code...

My reply (being out of scope of the HTTP bootstrapping) should be under your new post
about the implementation of a Bitcoin client in shell scripting (http://bitcointalk.org/index.php?topic=2461.0).

Hope this helps a little bit,

(PS.  you don't have to put a whole pipe in a same line.  The '|' character can end a line.)

I know but I suppose this shows my laziness when doing a paste into the forum ;-)
 

[jgarzik]

bitcoin maintains a database of P2P addresses.  Obtaining addresses via netstat is rather sub-optimal, when you could use bitcointools to extract addresses directly from the bitcoin database.

As to the larger point...

HTTP and DNS bootstrapping should be pursued.  Much more efficient than IRC.

[ShadowOfHarbringer]

bitcoin maintains a database of P2P addresses.  Obtaining addresses via netstat is rather sub-optimal, when you could use bitcointools to extract addresses directly from the bitcoin database.

As to the larger point...

HTTP and DNS bootstrapping should be pursued.  Much more efficient than IRC.

Oh my, dat is soo awsum. +10 to dis idea.
Can we has dis idea implemented in mainstream client, plz ?

I mean how could anybody refuse this soft fluffy little lolcat ?

[SmokeTooMuch]

I'm not sure how up-to-date they are, but why not adding these IP's to your (or a seperate) list:
http://bitcointalk.org/index.php?topic=59.0
(make sure to read these two posts about fallbacl-nodes:
http://bitcointalk.org/index.php?topic=59.msg14646#msg14646,
http://bitcointalk.org/index.php?topic=59.msg31133#msg31133)

[grondilu]

But I don't know the exact elliptic curves used by Bitcoin. You can get the one supported
by OpenSSL by doing an:

Code:

openssl ecparam -list_curves

Is there a table of the EC properties used by Bitcoin somewhere? I suppose the easiest is
to read the source code...

Indeed you have to look at the source code.  I've just check and the EC curve used is : secp256k1, which is in the list given by openssl.

I think a scripted implementation is feasable.

[jgarzik]

Only a very few fallback nodes are persistent over time, and compiled (hardcoded) into the bitcoin client itself.

https://en.bitcoin.it/wiki/Fallback_Nodes is a viable method of bootstrapping.  We'll call that "forum bootstrapping" or "wiki bootstrapping", where one must manually search for a list of nodes, in order to bootstrap onto the network.

I think DNS bootstrapping would be the most efficient:  a simple DNS lookup to bootstrap.bitcoin.org would work like this:

• Community members post their nameserver (NS) records for bootstrap.bitcoin.org on the forum.  Presumably this list does not change often
• Each member runs a DNS server, independently of anyone else, that retrieves addresses from bitcoin's addr.dat database, randomly selects "fresh" P2P nodes, and stores these in A records or SRV records.
• When bootstrapping, the bitcoin client performs a standard DNS lookup for bootstrap.bitcoin.org

That would be very, very fast.  Much faster than IRC.  This is similar to how BitTorrent DHT bootstrapping occurs.

The only issue is trust (rogue DNS servers), but this issue also exists with the IRC server, which is a Single Point of Failure (SPOF) for both trust and general reliability.

[bitcoinex]

Only a very few fallback nodes are persistent over time, and compiled (hardcoded) into the bitcoin client itself.

This is security hole.

State can block outgoing 8333 port for all hosts except for hardcoded addresses. On hardcoded addresses they can set up fake bitcoin nodes. Then these nodes will give you the addresses of a dummy nodes to create the illusion of a bitcoin network.

So they are force you  to spend your own money from a bitcoin client in their favor.

Better in the case of network problems ask the user to specify the address for the bootstrap. This address can be obtained from reliable sources, verified by the user.

[ShadowOfHarbringer]

This is security hole.

Not necessarily.

You can place blockchain bootstraps in compressed *.zip or *.tar.gz files, and hardcode just multiple (RMD160, SHA1, SHA256 + Filezize) hashes of backups into mainstream client.

OR, updated hashes of blockchain bootstraps can be avaiable for download from main bitcoin server over https, in which case it will be impossible to fake them (but this is a centralized solution, so probably not very good).
Possibilities are endless.

[adulau]

bitcoin maintains a database of P2P addresses.  Obtaining addresses via netstat is rather sub-optimal, when you could use bitcointools to extract addresses directly from the bitcoin database.

As to the larger point...

HTTP and DNS bootstrapping should be pursued.  Much more efficient than IRC.

Right, that's why I was pursuing on that way.

By the way, I made a test with bitcointools to dump the address out of the database :

Code:

 python2.7 dbdump.py --datadir ~/.bitcoin/ --address

...155.6:36128 (lastseen: Sat Dec 18 21:09:42 2010)
68.52.60.203:36128 (lastseen: Sun Dec 26 15:28:48 2010)
68.53.17.115:36128 (lastseen: Thu Dec 16 18:56:57 2010)
68.56.241.235:36128 (lastseen: Sun Dec 26 17:32:34 2010)
68.62.250.145:36128 (lastseen: Sun Dec 26 15:39:33 2010)
....

Even if the netstat approach could be suboptimal, there is an advantage over relying on the addr dump
from the database. If you get the address from the TCP Established session, these are really the active Bitcoin clients
and from the database, you are guessing out of the lastseen information where you have already a lot of dead
addresses. And picking the appropriate time delta can be tricky except if there is already something in the database
structure to just list the active ones. On the other hand, the Berkeley database need to be only accessed by one
process at a time and you need to shutdown the existing the current database.

Maybe another appropriate way might be to read the addr message passing over the TCP sessions (using pcap) and
extract the addresses and publish that stream to the HTTP/DNS directory.

What's the most appropriate techniques to get the currently active Bitcoin addresses?

[theymos]

So they are force you  to spend your own money from a bitcoin client in their favor.

They could double-spend transactions to you, but they couldn't redirect your transactions from one Bitcoin address to another one. There are much easier ways to "surround" someone if you control the ISP.

Seednode bootstrapping is used in Tor, I2P, GNUnet, and Freenet. Just removing IRC and using the already-implemented seednode system will work fine.

Reading a bit the code of the Bitcoin client, the client is using a simple trick to know its remote IP via
the IRC server (https://github.com/bitcoin/bitcoin/blob/master/irc.cpp#L333).

That's just one method of finding your external IP. There are also two HTTP external IP services.

[pj]

It could be cleaned a bit I think :

netstat -an |
awk '/8333/ && /ESTA/ { print $5 }' |
sed 's/:8333//' |
tee >(echo "# $(date) $(wc -l) Bitcoin clients seen.")

Good ideas ... I'd like to tweak this a tad more.

The ">(process)" construct is not recognized by classic Bourne shells and similar.
So far as I know, the redirection to a file, such as to /tmp/bitcoin in the original post,
is required for some such shells.

One more command, the sed, can be removed by using a little more awk.

That (necessary for some shells) tmp file /tmp/bitcoin should be made unique and self-removing

The naked search for "8333" would pick up ports 18333, 28333 ... 58333 as well.  Prefix with a
colon ':' to avoid that.

The result is

Code:

#!/bin/sh
# Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333.
# Append line at end with date and count of addresses displayed.

t=/tmp/bitcoin.$$
trap 'rm -f $t; trap 0; exit' 0 1 2 3 15

netstat -an | awk '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1]}' | tee $t
echo "# $(date) $(wc -l < $t) Bitcoin clients seen."

Is this correct, that you want both IP addresses coming from remote port 8333 and coming
into local port 8333?  Or do you just want IP addresses coming into local port 8333?  If the
later, change the key line above to look for /:8333/ only in field $4, the local address.

Code:

netstat -an | awk '$6 == "ESTABLISHED" && $4 ~ /:8333/ { split($5, a, ":"); print a[1]}' | tee $t

[grondilu]

The result is

Code:

#!/bin/sh
# Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333.
# Append line at end with date and count of addresses displayed.

t=/tmp/bitcoin.$$
trap 'rm -f $t; trap 0; exit' 0 1 2 3 15

netstat -an | awk '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1]}' | tee $t
echo "# $(date) $(wc -l < $t) Bitcoin clients seen."

Didn't know about the trap command.  I doubt we need it though.

Being a bit anal :

- The standard way to create a temp file is to use the mktemp command.
- You can end lines after |.  This makes the code clearer.

Code:

#!/bin/sh
# Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333.
# Append line at end with date and count of addresses displayed.

t="$(mktemp -t bitcoin)"

netstat -an |
awk '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1]}' |
tee "$t"

echo "# $(date) $(wc -l < $t) Bitcoin clients seen."

[pj]

Code:

t="$(mktemp -t bitcoin)"

That doesn't work so well -- no XXX's in the mktemp -t template.

And I think you really do want the trap - otherwise your /tmp directory
will get filled up with these dang files.

Yes -- newlines separating each piped command are better (though
I prefer to indent all but the first one) -- I was being lazy and just
typing as I do at the command prompt.

Yes -- mktemp or the more recent tempfile are probably better.
I was just being lazy again, and doing it as I have done it for 30
years, long before those commands existed.  Sorry.  The main
problem with my old fashioned method, and even with mktemp,
is a security issue -- a hacker can get you to write a file that
they have setup, via a symlink that you thought was your file.
The main problem with mktemp and tempfile is that not all
systems have them (though you have to be on a fairly old,
odd, or barebones system not to have them.)

You can find more discussion of the temp file issue at:
  http://www.linuxsecurity.com/content/view/115462/151/
  Safely Creating Temporary Files in Shell Scripts

So ... all this suggests the following:

Code:

#!/bin/sh
# Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333.
# Append line at end with date and count of addresses displayed.

t="$(tempfile -p bitcoin)"
trap 'rm -f $t; trap 0; exit' 0 1 2 3 15

netstat -an |
  awk '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1]}' |
  tee "$t"

echo "# $(date) $(wc -l < $t) Bitcoin clients seen."

[pj]

I worry a bit that what might have been my most important question above could have gotten
lost in the  code refinement discussion.

So I'll ask it again:

Is this correct, that you want both IP addresses coming from remote port 8333 and coming
into local port 8333?  Or do you just want IP addresses coming into local port 8333?  If the
later, change the key line above to look for /:8333/ only in field $4, the local address.

My hunch is that we just want IP addresses coming into our port 8333.  Whether or not a connection
is coming from port on 8333 on some other system means nothing to us, as best as I can figure.

[grondilu]

Let's just make the final count with awk too...

Code:

#!/bin/sh
# Display foreign IP addresses coming from port 8333 --or-- connected to local port 8333.
# Append line at end with date and count of addresses displayed.

netstat -an |
awk -v date="$(date)" '$6 == "ESTABLISHED" && /:8333/ { split($5, a, ":"); print a[1] ; n++ }
END { print "# " date " : " n " bitcoin clients seen." }'

[pj]

Let's just make the final count with awk too...

Duh!  Excellent.  Thanks.