[ANN] BitEN - bitcoin erlang node

[Matt Corallo]

Matt Corallo, thank you for your kind words.
What if I want not 100, but 1000 incoming connections - can I open 10 (100, 1000) listening ports and announce all of them via addr messages - will it increase likelihood of other peers selecting me as their peer or would it fail due to binning? Or is the only viable alternative getting more IPs from different bins?

Yes, this would fail due to binning.

As for verification, I'm planning to do full verification eventually.

If you just want things to work for now, you can of course use a bitcoind with only two connections to localhost to do the verification for you.  Or if you really hate bitcoind, there are multiple other implemented full verification engines you could use for that.

Actually, the only document I found (wiki) defines it in this manner:

1    NODE_NETWORK    This node can be asked for full blocks instead of just headers.

Hmm...let me rephrase, the documents should clearly define NODE_NETWORK as "does full verification and can relay full blocks to you if you ask."

"Impact of my DDoS tool is very small, as long as only a few people are running my DDoS tool." You see the problem here...

If I wanted to make DDoS tool, I'd make one. I did not ask people to run it, people asked for code. If bitcoin network can be brought down by couple weak PCs, then something needs to be done to "standard client". There I see the problem.

I didnt say a few PCs could bring down the network, but the nature of DDoS is that the more PCs you get, the worse you can make things.  And thanks to some properties of bitcoin that we can't change (eg requiring ECDSA verification on things), Bitcoin could be more susceptible to DDoS than other more standard applications.  That said, even BitTorrent can pretty effectively be taken down by a few attackers in a swarm.

[1714780579]

1714780579

[1714780579]

1714780579

[1714780579]

1714780579

[Matt Corallo]

Do you have some hard data on this? I have some, suggesting otherwise. https://bitcointalk.org/index.php?topic=160326.0

Note that I didnt say propagation speeds are fast.  But that doesn't matter in Bitcoin.  I pointed out that the Bitcoin P2P network propagation speeds for blocks doesnt matter all that much since miners tend to (whether they're aware or not) propagate blocks in other ways too.  For transactions, you have to wait for blocks before your transaction is confirmed anyway, so transaction propagation speed isnt really relevant.  If you are dealing with a merchant which accepts transactions without waiting for blocks, you should be using a payment protocol (coming probably in 0.9) instead anyway. 

[r.willis]

If you are dealing with a merchant which accepts transactions without waiting for blocks, you should be using a payment protocol (coming probably in 0.9) instead anyway. 

Payment protocol? What's this? Is there BIP or at least discussion thread?

[Matt Corallo]

Payment protocol? What's this? Is there BIP or at least discussion thread?

Yes, there have been very extensive discussions of payment protocol work recently, though none of it on bitcointalk (most devs havent tried to use bitcointalk for any real work for....a long time)

[r.willis]

Discussion is good, but don't protocol change/extension needs to go through BIP procedure?

[aantonop]

Yes, there have been very extensive discussions of payment protocol work recently

Where? Is this on the dev mailing list? Or IRC?

[r.willis]

http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg01567.html

[andrewbadr]

I sent you a donation btw, just wanted to make sure you saw it. Cool project.

[r.willis]

I received two donations so far: 2013-03-27 and 2013-03-18.
Thank you for your support. I'll spend coins on additional VPS hosting to help people download blocks and toss tx around, once I have verification working.

[tvbcof]

I received two donations so far: 2013-03-27 and 2013-03-18.
Thank you for your support. I'll spend coins on additional VPS hosting to help people download blocks and toss tx around, once I have verification working.

Make it three.

I see great potential value in the work you are attempting if Bitcoin ever comes under widespread attack, and have seen erlang used to great effect for complex problems.

[wumpus]

I like the idea of using the Erlang platform for bitcoin. It sounds like a perfect fit, a safe language aimed at distributed systems.

[Mike Hearn]

Just wanted to echo Matt's comments here. When he talks about "verifying transactions", that is another way to say "write a full Bitcoin implementation". As many people have discovered, that is a huge amount of very difficult work. If you don't verify transactions then simply connecting to lots of nodes and blasting messages out doesn't actually achieve anything (I know this is unintuitive).

The reason is, as Matt noted, the variance in transaction propagation time doesn't seem to be to do with insufficiently dense connectivity but rather things like not all transactions that are broadcast actually being valid, skew in relay/IsStandard rules, and a few other things (old nodes hurt too but they'll be slowly evicted from the system come May 15th anyway).

Without verifying transactions just echoing them around will result in your nodes eventually getting banned, because connections that send invalid data are assumed to be malicious.

Cool though Erlang is, I'm not sure it has many benefits for something like Bitcoin. Of course you're welcome to disagree. When I look at the causes of inefficiency in the current C++ codebase, it's not clear that the language has anything to do with it. There are various things we could optimise to make propagation of data faster, although it doesn't seem to be Bitcoin's biggest problem at the moment.

[aantonop]

... although it doesn't seem to be Bitcoin's biggest problem at the moment.

Seems to me Bitcoin's biggest problem at the moment is that there is only one implementation. It seems that there is absolutely no desire by the core dev team to ever have more than one client, since that would actually require freezing and documenting the protocol long enough and doing interop testing.

So instead, we will continue to have a mono-culture where there is no bitcoin protocol, other than "whatever some specific version of bitcoind currently speaks".

For that reason alone, I think it is important to back this and other project and if necessary completely ignore the developers who have little interest in standardization or interoperability with more than one implementation. "Bitcoin" the standard, should be a protocol, a money API, and a clean, minimal reference implementation, not a chunk of C++ code that does everything from node to GUI.

Of course, the usual answer is: go build it yourself. I notice those who try get shouted down or thwarted by a moving target, because the dev team is off pursuing a new extension, fad or GUI feature, instead of standardizing the existing protocol.

Yeah, let's disagree.

[jgarzik]

Seems to me Bitcoin's biggest problem at the moment is that there is only one implementation.

Incorrect.  There are several implementations.

It seems that there is absolutely no desire by the core dev team to ever have more than one client,

Incorrect.  Gavin actively encourages alternative implementations, and I've written two alternate implementations.  At least one was linked earlier in this thread.

[Zeilap]

It seems that there is absolutely no desire by the core dev team to ever have more than one client,

Incorrect.  Gavin actively encourages alternative implementations, and I've written two alternate implementations.  At least one was linked earlier in this thread.

Yeah you have written two, but the documentation is the worst no outside developers stands a chance, it takes so much time cause you have to do trial and error to get even the simplest connection.

Not really. If you just want to do something simple like connect, send and receive version and verack messages, then you could have something up and running in an hour. I know because I've done it. It's simply a matter of implementing the code to parse messages which is described very clearly on the wiki, no trial and error required.

[r.willis]

Well, example version message on the wiki has incorrect checksum - was beating my head against the wall for a hour.
Parsing was really easy (thanks for Erlang binary pattern matching), in this respect docs are ok. It's a little lacking in protocol details like state diagrams, timeouts, mempool algorithms, but I think I'll manage to produce something working (maybe not very compatible) from the spec.

[wumpus]

Cool though Erlang is, I'm not sure it has many benefits for something like Bitcoin. Of course you're welcome to disagree. When I look at the causes of inefficiency in the current C++ codebase, it's not clear that the language has anything to do with it. There are various things we could optimise to make propagation of data faster, although it doesn't seem to be Bitcoin's biggest problem at the moment.

I agree that it doesn't solve any efficiency issues or make anything faster (it is hard to beat C++ there). Erlang is very much aimed at reliability, so I'm more hoping for a bulletproof, crashless, seamlessly upgradable implemementation.

[grau]

Cool though Erlang is, I'm not sure it has many benefits for something like Bitcoin. Of course you're welcome to disagree. When I look at the causes of inefficiency in the current C++ codebase, it's not clear that the language has anything to do with it. There are various things we could optimise to make propagation of data faster, although it doesn't seem to be Bitcoin's biggest problem at the moment.

I agree that it doesn't solve any efficiency issues or make anything faster (it is hard to beat C++ there). Erlang is very much aimed at reliability, so I'm more hoping for a bulletproof, crashless, seamlessly upgradable implemementation.

Functional languages like Erlang or Scala are best suited to create robust multithreaded programs since their patterns avoid mutable objects and side effects. Using such language definitely helps writing thread-safe validation and service of lots of parallel requests.

A Bitcoin node performance however is constrained by block storage and EC signature validation, if you use some low level implementations for those, you get best of both worlds.
This is what I also did in bits of proof using LevelDB (native C++) and plan to use for EC.

[Mike Hearn]

Just be aware of the usual warnings. If you produce something that seems to theoretically "work" but doesn't match the behaviour of the C++ client exactly (including bugs), you can get split off the chain or end up being banned by the anti-DoS code.

Over time the documentation has improved a lot, but trying to produce a 100% compatible version of Bitcoin from it is probably still not possible.

[Mike Hearn]

There have been many, many threads on this topic. Look at the bitsofproof thread by grau for some long debates on it.

Basically, everyone who has worked on Bitcoin for a long time believes the code is so complicated that it isn't currently possible to document every weird quirk and edge case that must be supported correctly. Every time we think the spec is complete, someone discovers another piece of unexpected functionality or behaviour that then must be incorporated into the description. This leads to the question of how you know you've got it all written down, and precisely enough for re-implementation.

Eventually, you come to realize - wait. We already have a very precise specification of how Bitcoin behaves. That specification has as much detail as you'd ever need, because it's written in C++ and executed directly by the CPU. So if you have a question about the precise manner in which transactions that have invalid SIGHASH_SINGLE flags are treated, the best way to get an answer is just read the code. Then you know for sure. Reading an English language derivative of it means you can never be sure it was totally correct.