|
Title: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 20, 2013, 06:28:09 PM Hello all!
For some time I was working on bitcoin daemon implementation in erlang. Now it's in the early stage of development. I'm trying to implement protocol not looking into bitcoind source, but using specification from https://en.bitcoin.it/wiki (found one bug so far), original paper and looking at the protocol messages which I get from peers. My goal is to make scalable bitcoin daemon, capable of serving many thousands of peers on relatively low-end servers. At current state it's capable of connecting and parsing messages from 3000+ peers on 256M RAM VPS without swap. It does not do much besides protocol parsing and peer discovery now, but it's only several days old. Chances are that it connected to your machine - check for /ebtcd:0.0.1/ user-agent in logs. Short term pans: accept incoming connections, announce own address via addr message. Mid term plans: add validation and routing (test on testnet), release source under open license (AGPL sounds good). Long term: full-featured daemon. P.S. In case someone wants to encourage me with some BTC, I set up donation adress: 1GVmS56pvVL7YZA7YqMBXmaDedCoputKuJ UPD: git clone git://github.com/r-willis/biten.git Title: Re: [ANN] bitcoin erlang daemon Post by: gweedo on March 20, 2013, 06:30:03 PM Any chance you going to put it on gihub?
Title: Re: [ANN] bitcoin erlang daemon Post by: kokjo on March 20, 2013, 06:33:23 PM please make the source public, og remove [ANN] from the thread subject... i was expecting to see awesome code, but was disappointed.
Title: Re: [ANN] bitcoin erlang daemon Post by: r.willis on March 20, 2013, 06:48:44 PM please make the source public, og remove [ANN] from the thread subject... i was expecting to see awesome code, but was disappointed. You are right, so [ANN] gone for now.Code is in no way awesome now. It's a bunch of loosely connected modules and processes. Current code is proof-of-concept that you can make "very-well-connected" node on low-end virtual server. Right now I'm converting code to OTP (Erlang framework for robust, industrial-grade applications). I'd expect code release in a week or two. Why I (we) want node with many peers? Because it greatly reduces network "diameter", reduces propagation times, helping miners and makes many attacks not feasible. But if it does not verify transactions/blocks it relays, I effectively becomes threat to network, as if there is many such nodes, spammers can use them to amplify their abilities to flood nodes with malformed/fraudulent transactions. Title: Re: [ANN] bitcoin erlang daemon Post by: kokjo on March 20, 2013, 06:56:39 PM But if it does not verify transactions/blocks it relays, I effectively becomes threat to network, as if there is many such nodes, spammers can use them to amplify their abilities to flood nodes with malformed/fraudulent transactions. i don't think so... the satoshi client have some ban system, if your node began to send malformed txs they would just disconnect from your node.Title: Re: bitcoin erlang daemon Post by: grau on March 20, 2013, 07:16:21 PM Now it's in the early stage of development. I'm trying to implement protocol not looking into bitcoind source, ... From this I can tell that this must be really early stage. Keep up, but be prepared for a very long trip. Unfortunately documentation available is very far from being sufficient input for an implementation, and in Bitcoin world there is no half-way compatibility. Title: Re: [ANN] bitcoin erlang daemon Post by: r.willis on March 20, 2013, 07:56:24 PM i don't think so... the satoshi client have some ban system, if your node began to send malformed txs they would just disconnect from your node. Of course it has, but it but before it kicks in, you can spend several megs of some someone's bandwidth. Multiply by some thousands peers, and by number of servers (whatever it is). I just not very comfortable with this.Keep up, but be prepared for a very long trip. Yes, I read your thread about bitsofproof. I hope it will be interesting journey.Title: Re: [ANN] bitcoin erlang daemon Post by: kokjo on March 20, 2013, 07:59:45 PM i don't think so... the satoshi client have some ban system, if your node began to send malformed txs they would just disconnect from your node. Of course it has, but it but before it kicks in, you can spend several megs of some someone's bandwidth. Multiply by some thousands peers, and by number of servers (whatever it is). I just not very comfortable with this.i am also devloping a bitcoin client, pycoin. although i have not done much lately on it. Title: Re: bitcoin erlang daemon Post by: r.willis on March 20, 2013, 08:08:00 PM If you think so, I can implement routing and release code. Let's stress-test network.
I think on 8Gb (right now, without optimisations) machine daemon can connect to every connectable node in network (I estimate it's around 60k peers) + some incoming connections. Title: Re: [ANN] bitcoin erlang daemon Post by: jgarzik on March 21, 2013, 02:14:40 AM i am also devloping a bitcoin client, pycoin. although i have not done much lately on it. If you like python, check out https://github.com/jgarzik/python-bitcoinlib/ and the bitcoin client that uses it, https://github.com/jgarzik/pynode/ I'm trying to encourage everyone to use python-bitcoinlib for their python projects. It fully verifies the blockchain, and passes many reference client tests. Title: Re: [ANN] bitcoin erlang daemon Post by: meowmeowbrowncow on March 21, 2013, 02:17:50 AM i am also devloping a bitcoin client, pycoin. although i have not done much lately on it. If you like python, check out https://github.com/jgarzik/python-bitcoinlib/ and the bitcoin client that uses it, https://github.com/jgarzik/pynode/ I'm trying to encourage everyone to use python-bitcoinlib for their python projects. It fully verifies the blockchain, and passes many reference client tests. And that Python, generally, is cool. ;) Title: Re: bitcoin erlang daemon Post by: aantonop on March 21, 2013, 03:09:27 AM Very interesting.
You should open source even if it is early stage, please! I'd love to I've done some Erlang and OTP and I'm looking at core-daemon (full node) implementations. Don't be shy, open it! Title: Re: [ANN] bitcoin erlang daemon Post by: kokjo on March 21, 2013, 06:45:54 AM i am also devloping a bitcoin client, pycoin. although i have not done much lately on it. If you like python, check out https://github.com/jgarzik/python-bitcoinlib/ and the bitcoin client that uses it, https://github.com/jgarzik/pynode/ I'm trying to encourage everyone to use python-bitcoinlib for their python projects. It fully verifies the blockchain, and passes many reference client tests. Title: Re: bitcoin erlang daemon Post by: r.willis on March 21, 2013, 11:38:21 PM Conversion to OTP mostly done.
Plans for tomorrow: limit rate at which outgoing connections are created. After this, accept incoming connections and advertise own address via addr message. Title: Re: bitcoin erlang daemon Post by: kokjo on March 22, 2013, 06:53:00 AM Conversion to OTP mostly done. Plans for tomorrow: limit rate at which outgoing connections are created. After this, accept incoming connections and advertise own address via addr message. and then: github Title: Re: bitcoin erlang daemon Post by: r.willis on March 22, 2013, 07:41:20 AM Before github, I need cool name for project. Any ideas?
Title: Re: bitcoin erlang daemon Post by: kokjo on March 22, 2013, 10:58:27 AM Before github, I need cool name for project. Any ideas? "The first and only true bitcoin erlang deamon" or TFAOTBED?Title: Re: bitcoin erlang daemon Post by: meowmeowbrowncow on March 22, 2013, 01:14:14 PM Before github, I need cool name for project. Any ideas? "The first and only true bitcoin erlang deamon" or TFAOTBED?How about: The Committee for the Liberation and Integration of Terrifying Organisms and their Rehabilitation Into Society CLITORIS (had to, sorry :D) Title: Re: bitcoin erlang daemon Post by: r.willis on March 22, 2013, 03:00:55 PM It's hilarious... I'll consider it.
Title: Re: bitcoin erlang daemon Post by: r.willis on March 22, 2013, 10:32:20 PM System is live at 192.210.207.147:8333
Accepts connection and does handshake. Nothing fancy yet. Transaction routing and block download/serving are next. Title: Re: bitcoin erlang daemon Post by: aantonop on March 22, 2013, 10:40:32 PM Bite BITcoin Erlang
BED = Bitcoin Erlang Daemon, nice and simple. Usage: "Hey Carla, we're running "BED", are you using bitcoin-js at your startup?" BITTEN : BITcoin Erlang Node without acronyms: The Bitscale Node - Makes Bitcoin Scale CoinOTP Title: Re: bitcoin erlang daemon Post by: uuidman on March 23, 2013, 12:39:07 AM Maybe the name Erlnode
Title: Re: bitcoin erlang daemon Post by: r.willis on March 24, 2013, 07:22:00 AM I really like BITTEN/BITEN. Release is coming.
Can anybody check 198.23.147.4:8333 and tell me if tx relaying works ok? After handshake, host wll notify peers with inv and respond to getdata (type 1 (tx) only). Title: Re: bitcoin erlang daemon Post by: kokjo on March 24, 2013, 10:08:54 AM my log:
Code: __main__ - connecting to 192.210.207.147:8333 you should group your invs to improve anonymity and save bandwidth. only sending a inv once a 5 second. Title: Re: bitcoin erlang daemon Post by: r.willis on March 24, 2013, 01:03:42 PM Thanks for testing. Can you also test if getdata works properly with transasctions?
Also, can you explain about inv and anonymity? For now I send inv to all peers when I get new tx. Title: Re: bitcoin erlang daemon Post by: kokjo on March 24, 2013, 01:48:47 PM Thanks for testing. Can you also test if getdata works properly with transasctions? i don't know, think its only meaningful to do it when you are also sending your own transactions, blending you own in with others.Also, can you explain about inv and anonymity? For now I send inv to all peers when I get new tx. the satoshi client sends them in groups. getdata seems to work fine. Code: __main__ - connecting to 192.210.207.147:8333 Title: Re: bitcoin erlang daemon Post by: r.willis on March 24, 2013, 05:13:16 PM Thank you for your support.
Please check how it works yourself. There is no documentation (and almost no comments), no tests. There are some lurking bugs, but it mostly works. Codebase is really small (~1000 lines + ~500 lines SHA2 by third-party), so do not expect much. Some tunable parameters (build-time) in include/config.erl $ git clone git://github.com/r-willis/biten.git than check README for instructions. UPD: here is one-liner for updating stats every 10 sec: 3> T1 = timer:apply_interval(10*1000, stat, print, []). Title: Re: [ANN] BitEN - bitcoin erlang node Post by: kokjo on March 24, 2013, 06:37:22 PM Awesome! :D code perving now.
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: yoniassia on March 25, 2013, 03:07:02 AM You should check twitcoin.net also implemented a bitcoin wallet in erlang
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 25, 2013, 08:02:55 AM Thanks for the link, I'll check it.
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: kokjo on March 25, 2013, 09:43:28 AM You should check twitcoin.net also implemented a bitcoin wallet in erlang no it does not, in only interfaces with bitcoindTitle: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 25, 2013, 04:32:26 PM So, anyone tried it? Does it even work for you?
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: kokjo on March 25, 2013, 04:45:38 PM So, anyone tried it? Does it even work for you? no, i have exams. if i begin playing with bitten, i gonna fail them...Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 25, 2013, 05:20:22 PM Good luck with your exams.
It seems like there are some malicious nodes on the network, which send thousands of inv messages. As biten have no anti-DoS features, it causes memory exhaustion and erlang VM termination. I hope to push quick-fix today. P.S. I decided to drop extra "T" from name, so it's biten. EDIT: pushed some fixes and improvements. Next: download block headers and organize them into trees. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: kokjo on March 25, 2013, 08:35:42 PM Good luck with your exams. thank you :)Quote It seems like there are some malicious nodes on the network, which send thousands of inv messages. As biten have no anti-DoS features, it causes memory exhaustion and erlang VM termination. I hope to push quick-fix today. they are not malicious, just stress testing the network. ;)Title: Re: bitcoin erlang daemon Post by: aantonop on March 27, 2013, 09:35:08 AM $ git clone git://github.com/r-willis/biten.git ON IT! I'm meeting a real ErlangOTP *expert* (phds, patents, the whole 9) next week who recently got interested in bitcoin. Perfect timing to introduce your code... Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 27, 2013, 07:56:48 PM Do not think he will be impressed - it's my first "real" OTP project. I hope it will be able to run more than 24h without problems - right now after 2-3 hours it terminates.
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: kokjo on March 27, 2013, 08:09:07 PM Do not think he will be impressed - it's my first "real" OTP project. I hope it will be able to run more than 24h without problems - right now after 2-3 hours it terminates. same here, it just dies... :(Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 27, 2013, 08:28:30 PM Quote same here, it just dies... Can you show output ofstat:print() ~5 min and ~30 min after start? Title: Re: [ANN] BitEN - bitcoin erlang node Post by: kokjo on March 27, 2013, 08:46:59 PM Quote same here, it just dies... Can you show output ofstat:print() ~5 min and ~30 min after start? Code: Start time..2013.03.27 20:44:01 UTC Title: Re: [ANN] BitEN - bitcoin erlang node Post by: kokjo on March 27, 2013, 09:17:25 PM Code: 5> Start time..2013.03.27 20:44:01 UTC Title: Re: [ANN] BitEN - bitcoin erlang node Post by: aantonop on March 27, 2013, 09:51:57 PM Do not think he will be impressed - it's my first "real" OTP project. I hope it will be able to run more than 24h without problems - right now after 2-3 hours it terminates. He is impressed when people know about and try to use erlangOTP. Heck, I'm impressed too. Working code. That's all that matters. The rest is only a matter of time. Thank goodness Linus didn't say: It's only a floppy driver a screen driver and a kernel that keeps crashin, I'd better not show it to anyone until it's "ready". I downloaded that "piece of shit" in 1993 on 210 floppy disks. I hear Linux turned out ok. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 27, 2013, 11:01:53 PM I think I fixed it.
Please test. Pull latest version, recompile, then: Code: git clone git://github.com/vinoski/erlsha2.git Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 27, 2013, 11:30:02 PM Start time..2013.03.27 22:50:33 UTC
Cur time....2013.03.27 23:29:55 UTC Connects:...3281 Connected:..1199 Accepted:...24 Answers:....1197 Cur peers...1001 Relayed TX:.204155 Mempool: inv table..49 req table..20 tx table...571 peer table.49 check inv..0.0003 s clean inv..0.0002 s clean req..0.0001 s clean tx...0.0005 s Errors: timeout:...1612 refused:...289 unreach:...167 other:.....5 No signs of lags, etc. This is on my low-end VPS with 256 MB RAM without swap. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: tvbcof on March 28, 2013, 12:11:32 AM Interested. Sorry for the spam, but I would rather have thread updates as replied vs. watchlist.
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Matt Corallo on March 28, 2013, 01:35:06 AM So...this is some kind of way to try to DoS bitcoin?
It connects to tons of nodes, eating listening sockets on them all, without listening to add more to the pool. Does it relay things? It would appear that it relays unchecked objects, so anyone can use a node of this to amplify its bandwidth when attacking the network. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: aantonop on March 28, 2013, 02:21:01 AM So...this is some kind of way to try to DoS bitcoin? It connects to tons of nodes, eating listening sockets on them all, without listening to add more to the pool. Does it relay things? It would appear that it relays unchecked objects, so anyone can use a node of this to amplify its bandwidth when attacking the network. It should be run on testnet, ok. But DoS? Come on! This is working bitcoin code in a language that scales massively and you're looking the gift horse in the mouth. If someone wants to DoS bitcoin, there are far easier ways to do it, all of which won't succeed or do much for long. None that involve this much coding effort. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Matt Corallo on March 28, 2013, 02:27:56 AM Writing clients that are capable of relaying tons of verified messages and helping the network is great. But this doesnt actually do that. Making a few outbound connections and listening for tons of inbound connections is hugely useful for the network, the opposite is just a DoS.
Dont be confused, add some basic anti-DoS verification and change one or two constants to limit the maximum outbound connection count and this would be a really awesome project, as-is, however.... Title: Re: [ANN] BitEN - bitcoin erlang node Post by: aantonop on March 28, 2013, 03:12:06 AM Writing clients that are capable of relaying tons of verified messages and helping the network is great. But this doesnt actually do that. Making a few outbound connections and listening for tons of inbound connections is hugely useful for the network, the opposite is just a DoS. You're the CI/jenkins tester for bitcoind, right? Thanks for that, it's a very important contribution. I am surprised you would discourage code contributions in other languages when the current network is so dangerously mono-culture and susceptible to major problems because of bugs. It seems to me that bitcoind does a fine job DoS-ing itself every now and then, and that fact that there are no alternative clients may have something to do with it. Insular attitudes make insular code which makes fragile and brittle code. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Matt Corallo on March 28, 2013, 03:18:54 AM I am surprised you would discourage code contributions in other languages when the current network is so dangerously mono-culture and susceptible to major problems because of bugs. No, I totally agree with such sentiment, but there is a difference between doing that and creating something that blindly relays any messages to every peer it can findIt seems to me that bitcoind does a fine job DoS-ing itself every now and then No disagreement here.and that fact that there are no alternative clients may have something to do with it. Not sure about this, but I certainly agree it would help prevent some simple attacks. Again, I have to say, I like the idea, it just needs to be modified a bit to avoid being a distributed DOS client. Title: Re: [ANN] bitcoin erlang daemon Post by: Zeilap on March 28, 2013, 03:52:07 AM To be fair, r.willis acknowledged the issue earlier, before he released the code. Perhaps it was the reason he was reluctant to release the source to soon.
But if it does not verify transactions/blocks it relays, I effectively becomes threat to network, as if there is many such nodes, spammers can use them to amplify their abilities to flood nodes with malformed/fraudulent transactions. Or maybe the plan was indeed to have no verification in order to reduce transaction and block propagation time. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: gmaxwell on March 28, 2013, 04:35:51 AM Insular attitudes make insular code which makes fragile and brittle code. This is really orthogonal, as is the code uses 125x network resources compared to normal node software. That should be changed. Otherwise no big deal.Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 28, 2013, 05:55:34 AM Yes, I'm aware of possible misuses of my software, and I told about them earlier in the thread. It does not check TX it relies. This is bad, and out-of spec, or is it? (Oh, wait, spec is the Satoshi client code, need to check what it says about it).
Verifying TX is my next goal. There is connection limit (in config.erl, it's 1000 by default). It does have incoming connection support (not limited, counted towards 1000). For it to work, you must configure your external IP in config.erl. How many nodes does testnet have? I'm interested in maintaining thousands of connections to reduce network diameter and propagation speed. I'm sure network part will work with 10 (or 100) nodes. It does not do more than 1 connection per host:port pair, and tries such pairs only once in lifetime. So, impact on each node is minimal (as long there is not many of nodes running my software). Title: Re: [ANN] BitEN - bitcoin erlang node Post by: aantonop on March 28, 2013, 07:04:27 AM This is really orthogonal, as is the code uses 125x network resources compared to normal node software. That should be changed. Otherwise no big deal. I think it is very much the problem. Look at the attitude displayed from the very get go, on a thread where someone is contributing code. Insular is the perfect description, of that response, the attitude of some of the devs and the resulting code. Where in the spec does it say what the maximum nodes should be? Oh, right - there is no spec. Just the "normal node software". Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 28, 2013, 08:07:58 AM For now, I'v complied with gmaxwell's suggestion and changed default number of peers to 10.
After verification code is in working shape, we will discuss this once more. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: tvbcof on March 28, 2013, 08:25:46 AM This is really orthogonal, as is the code uses 125x network resources compared to normal node software. That should be changed. Otherwise no big deal. I think it is very much the problem. Look at the attitude displayed from the very get go, on a thread where someone is contributing code. Insular is the perfect description, of that response, the attitude of some of the devs and the resulting code. Where in the spec does it say what the maximum nodes should be? Oh, right - there is no spec. Just the "normal node software". Off hand I would suspect that 'reducing network diameter and propagation speed' might have a variety of advantages and dev might want to start thinking about the ramifications if this happened to become the 'new normal'. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Matt Corallo on March 28, 2013, 02:06:40 PM Yes, I'm aware of possible misuses of my software, and I told about them earlier in the thread. It does not check TX it relies. This is bad, and out-of spec, or is it? (Oh, wait, spec is the Satoshi client code, need to check what it says about it). Actually, the client (and other documents) clearly define the nServices bit NODE_NETWORK to indicate verification and full node status. Though you may end up relaying crap and getting banned as a DoS'er by full nodes, if you just did SPV verification and relayed without NODE_NETWORK it would technically be within spec (though non-optimal). Verifying TX is my next goal. Full verification or SPV verification? In any case, that is very good news.There is connection limit (in config.erl, it's 1000 by default). It does have incoming connection support (not limited, counted towards 1000). For it to work, you must configure your external IP in config.erl. OK, I did not see that it could accept incoming connections, in any case, that is very nice. I would, however, like to point out that the two types of connections represent two completely different purposes:Incoming connections allow you to serve data to other peers who need connections. It helps the network by using your bandwidth to provide relaying. That said, we cant purely rely on incoming connections because there is no way to know if they are all connections from one attacker. So we make a few outbound connections to make sure we have diversity. I'd like to note that more outbound connections by everyone does not result in greater propagation speed. It eats /tons/ more bandwidth and you will end up shooting yourself in the foot. How many nodes does testnet have? I'm interested in maintaining thousands of connections to reduce network diameter and propagation speed. I'm sure network part will work with 10 (or 100) nodes. Again, attempting to create more connections to reduce network diameter ends up shooting yourself in the foot, it actually wont end up decreasing propagation speed. Furthermore, propagation speed isnt an issue in today's Bitcoin network. Sure, miners want the best propagation speed they can find so they dont mine stale blocks, but the large pools mostly have private peering with other large miners to ensure that it is never a problem for them. P2Pool also relays the minimum data required to relay blocks found to all p2pool nodes in addition to announcing them on the Bitcoin network at the same time, ensuring the blocks get introduced to the Bitcoin network at many points at once. Finally, if you simply allow incoming connections, you will almost always get around a hundred nodes after a day or two, which results in really damn fast relaying for anything you generate.It does not do more than 1 connection per host:port pair, and tries such pairs only once in lifetime. So, impact on each node is minimal (as long there is not many of nodes running my software). "Impact of my DDoS tool is very small, as long as only a few people are running my DDoS tool." You see the problem here...Again, no one has issues with some of the ideas of this project, its some of its specific implementation details that are broken. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 28, 2013, 06:04:50 PM Matt Corallo, thank you for your kind words.
What if I want not 100, but 1000 incoming connections - can I open 10 (100, 1000) listening ports and announce all of them via addr messages - will it increase likelihood of other peers selecting me as their peer or would it fail due to binning? Or is the only viable alternative getting more IPs from different bins? As for verification, I'm planning to do full verification eventually. UPD: Quote Actually, the client (and other documents) clearly define Which documents?Actually, the only document I found (wiki) defines it in this manner: Quote 1 NODE_NETWORK This node can be asked for full blocks instead of just headers. Quote "Impact of my DDoS tool is very small, as long as only a few people are running my DDoS tool." You see the problem here... If I wanted to make DDoS tool, I'd make one. I did not ask people to run it, people asked for code. If bitcoin network can be brought down by couple weak PCs, then something needs to be done to "standard client". There I see the problem. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Zeilap on March 28, 2013, 07:26:56 PM Quote Clients will not propagate invalid blocks (or invalid transactions) (https://en.bitcoin.it/wiki/Invalid_block) Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 28, 2013, 07:33:15 PM Please define "invalid transaction".
Quote Furthermore, propagation speed isnt an issue in today's Bitcoin network. Do you have some hard data on this? I have some, suggesting otherwise. https://bitcointalk.org/index.php?topic=160326.0Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Matt Corallo on March 28, 2013, 09:41:52 PM Matt Corallo, thank you for your kind words. Yes, this would fail due to binning.What if I want not 100, but 1000 incoming connections - can I open 10 (100, 1000) listening ports and announce all of them via addr messages - will it increase likelihood of other peers selecting me as their peer or would it fail due to binning? Or is the only viable alternative getting more IPs from different bins? As for verification, I'm planning to do full verification eventually. If you just want things to work for now, you can of course use a bitcoind with only two connections to localhost to do the verification for you. Or if you really hate bitcoind, there are multiple other implemented full verification engines you could use for that.Actually, the only document I found (wiki) defines it in this manner: Hmm...let me rephrase, the documents should clearly define NODE_NETWORK as "does full verification and can relay full blocks to you if you ask."Quote 1 NODE_NETWORK This node can be asked for full blocks instead of just headers. Quote "Impact of my DDoS tool is very small, as long as only a few people are running my DDoS tool." You see the problem here... If I wanted to make DDoS tool, I'd make one. I did not ask people to run it, people asked for code. If bitcoin network can be brought down by couple weak PCs, then something needs to be done to "standard client". There I see the problem. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Matt Corallo on March 28, 2013, 09:46:00 PM Do you have some hard data on this? I have some, suggesting otherwise. https://bitcointalk.org/index.php?topic=160326.0 Note that I didnt say propagation speeds are fast. But that doesn't matter in Bitcoin. I pointed out that the Bitcoin P2P network propagation speeds for blocks doesnt matter all that much since miners tend to (whether they're aware or not) propagate blocks in other ways too. For transactions, you have to wait for blocks before your transaction is confirmed anyway, so transaction propagation speed isnt really relevant. If you are dealing with a merchant which accepts transactions without waiting for blocks, you should be using a payment protocol (coming probably in 0.9) instead anyway. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 29, 2013, 06:57:11 AM If you are dealing with a merchant which accepts transactions without waiting for blocks, you should be using a payment protocol (coming probably in 0.9) instead anyway. Payment protocol? What's this? Is there BIP or at least discussion thread?Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Matt Corallo on March 29, 2013, 07:05:54 AM Payment protocol? What's this? Is there BIP or at least discussion thread? Yes, there have been very extensive discussions of payment protocol work recently, though none of it on bitcointalk (most devs havent tried to use bitcointalk for any real work for....a long time)Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 29, 2013, 07:08:31 AM Discussion is good, but don't protocol change/extension needs to go through BIP procedure?
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: aantonop on March 29, 2013, 07:21:52 AM Yes, there have been very extensive discussions of payment protocol work recently Where? Is this on the dev mailing list? Or IRC? Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 29, 2013, 07:44:54 AM http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg01567.html
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: andrewbadr on March 30, 2013, 12:25:57 PM I sent you a donation btw, just wanted to make sure you saw it. Cool project.
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on March 30, 2013, 03:35:26 PM I received two donations so far: 2013-03-27 and 2013-03-18.
Thank you for your support. I'll spend coins on additional VPS hosting to help people download blocks and toss tx around, once I have verification working. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: tvbcof on March 30, 2013, 04:56:49 PM I received two donations so far: 2013-03-27 and 2013-03-18. Thank you for your support. I'll spend coins on additional VPS hosting to help people download blocks and toss tx around, once I have verification working. Make it three. I see great potential value in the work you are attempting if Bitcoin ever comes under widespread attack, and have seen erlang used to great effect for complex problems. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: wumpus on March 31, 2013, 08:05:56 AM I like the idea of using the Erlang platform for bitcoin. It sounds like a perfect fit, a safe language aimed at distributed systems.
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Mike Hearn on March 31, 2013, 11:27:27 PM Just wanted to echo Matt's comments here. When he talks about "verifying transactions", that is another way to say "write a full Bitcoin implementation". As many people have discovered, that is a huge amount of very difficult work. If you don't verify transactions then simply connecting to lots of nodes and blasting messages out doesn't actually achieve anything (I know this is unintuitive).
The reason is, as Matt noted, the variance in transaction propagation time doesn't seem to be to do with insufficiently dense connectivity but rather things like not all transactions that are broadcast actually being valid, skew in relay/IsStandard rules, and a few other things (old nodes hurt too but they'll be slowly evicted from the system come May 15th anyway). Without verifying transactions just echoing them around will result in your nodes eventually getting banned, because connections that send invalid data are assumed to be malicious. Cool though Erlang is, I'm not sure it has many benefits for something like Bitcoin. Of course you're welcome to disagree. When I look at the causes of inefficiency in the current C++ codebase, it's not clear that the language has anything to do with it. There are various things we could optimise to make propagation of data faster, although it doesn't seem to be Bitcoin's biggest problem at the moment. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: aantonop on March 31, 2013, 11:52:47 PM ... although it doesn't seem to be Bitcoin's biggest problem at the moment. Seems to me Bitcoin's biggest problem at the moment is that there is only one implementation. It seems that there is absolutely no desire by the core dev team to ever have more than one client, since that would actually require freezing and documenting the protocol long enough and doing interop testing. So instead, we will continue to have a mono-culture where there is no bitcoin protocol, other than "whatever some specific version of bitcoind currently speaks". For that reason alone, I think it is important to back this and other project and if necessary completely ignore the developers who have little interest in standardization or interoperability with more than one implementation. "Bitcoin" the standard, should be a protocol, a money API, and a clean, minimal reference implementation, not a chunk of C++ code that does everything from node to GUI. Of course, the usual answer is: go build it yourself. I notice those who try get shouted down or thwarted by a moving target, because the dev team is off pursuing a new extension, fad or GUI feature, instead of standardizing the existing protocol. Yeah, let's disagree. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: jgarzik on April 01, 2013, 02:21:03 AM Seems to me Bitcoin's biggest problem at the moment is that there is only one implementation. Incorrect. There are several implementations. Quote It seems that there is absolutely no desire by the core dev team to ever have more than one client, Incorrect. Gavin actively encourages alternative implementations, and I've written two alternate implementations. At least one was linked earlier in this thread. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Zeilap on April 01, 2013, 03:51:14 AM Quote It seems that there is absolutely no desire by the core dev team to ever have more than one client, Incorrect. Gavin actively encourages alternative implementations, and I've written two alternate implementations. At least one was linked earlier in this thread. Yeah you have written two, but the documentation is the worst no outside developers stands a chance, it takes so much time cause you have to do trial and error to get even the simplest connection. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on April 01, 2013, 06:33:00 AM Well, example version message on the wiki has incorrect checksum - was beating my head against the wall for a hour.
Parsing was really easy (thanks for Erlang binary pattern matching), in this respect docs are ok. It's a little lacking in protocol details like state diagrams, timeouts, mempool algorithms, but I think I'll manage to produce something working (maybe not very compatible) from the spec. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: wumpus on April 01, 2013, 10:52:50 AM Cool though Erlang is, I'm not sure it has many benefits for something like Bitcoin. Of course you're welcome to disagree. When I look at the causes of inefficiency in the current C++ codebase, it's not clear that the language has anything to do with it. There are various things we could optimise to make propagation of data faster, although it doesn't seem to be Bitcoin's biggest problem at the moment. I agree that it doesn't solve any efficiency issues or make anything faster (it is hard to beat C++ there). Erlang is very much aimed at reliability, so I'm more hoping for a bulletproof, crashless, seamlessly upgradable implemementation.Title: Re: [ANN] BitEN - bitcoin erlang node Post by: grau on April 01, 2013, 02:28:44 PM Cool though Erlang is, I'm not sure it has many benefits for something like Bitcoin. Of course you're welcome to disagree. When I look at the causes of inefficiency in the current C++ codebase, it's not clear that the language has anything to do with it. There are various things we could optimise to make propagation of data faster, although it doesn't seem to be Bitcoin's biggest problem at the moment. I agree that it doesn't solve any efficiency issues or make anything faster (it is hard to beat C++ there). Erlang is very much aimed at reliability, so I'm more hoping for a bulletproof, crashless, seamlessly upgradable implemementation.Functional languages like Erlang or Scala are best suited to create robust multithreaded programs since their patterns avoid mutable objects and side effects. Using such language definitely helps writing thread-safe validation and service of lots of parallel requests. A Bitcoin node performance however is constrained by block storage and EC signature validation, if you use some low level implementations for those, you get best of both worlds. This is what I also did in bits of proof using LevelDB (native C++) and plan to use for EC. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Mike Hearn on April 01, 2013, 02:44:21 PM Just be aware of the usual warnings. If you produce something that seems to theoretically "work" but doesn't match the behaviour of the C++ client exactly (including bugs), you can get split off the chain or end up being banned by the anti-DoS code.
Over time the documentation has improved a lot, but trying to produce a 100% compatible version of Bitcoin from it is probably still not possible. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Mike Hearn on April 01, 2013, 03:41:03 PM There have been many, many threads on this topic. Look at the bitsofproof thread by grau for some long debates on it.
Basically, everyone who has worked on Bitcoin for a long time believes the code is so complicated that it isn't currently possible to document every weird quirk and edge case that must be supported correctly. Every time we think the spec is complete, someone discovers another piece of unexpected functionality or behaviour that then must be incorporated into the description. This leads to the question of how you know you've got it all written down, and precisely enough for re-implementation. Eventually, you come to realize - wait. We already have a very precise specification of how Bitcoin behaves. That specification has as much detail as you'd ever need, because it's written in C++ and executed directly by the CPU. So if you have a question about the precise manner in which transactions that have invalid SIGHASH_SINGLE flags are treated, the best way to get an answer is just read the code. Then you know for sure. Reading an English language derivative of it means you can never be sure it was totally correct. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on April 01, 2013, 04:27:21 PM And do you personally feel that keeping bugs in original code forever is the right thing? What's BF stance about it, especially "standardization of bitcoin" being major stated goal?
Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Mike Hearn on April 01, 2013, 04:46:34 PM We've talked about cleaning up some of them after a hard fork. At the same time, hard forking is (currently) a fairly tricky thing to do, and there's a desire to avoid dumping "everything and the kitchen sink" into a forking event.
The way we seem to be going is - let's try and make hard forking easier and less intimidating. If we prove it can be done without lots of disruption, then suddenly fixing (known) bugs and quirks seems much less risky and lower cost. But it does have to be low cost because, frankly, there just isn't much point in fixing most of them. They aren't dangerous. Did you know the "Referer" field in HTTP is mis-spelled? Well, nobody ever tried to fix it because it's just not worth the hassle. Even if some of these bugs were resolved and cleaned up, I'm not sure it'd change things much. Like I said, every time we think the "spec" is complete, someone discovers a new and unexpected quirk. So how do you know you got them all? Is C++ really such a terrible language for precisely specifying behaviour? Worse than English? Title: Re: [ANN] BitEN - bitcoin erlang node Post by: grau on April 01, 2013, 05:17:27 PM Eventually, you come to realize - wait. We already have a very precise specification of how Bitcoin behaves. That specification has as much detail as you'd ever need, because it's written in C++ and executed directly by the CPU. the Satoshi implementation is much much more than needed to describe the behavior crucial for consensus, worse it is a Spaghetti and gets extended with any feature the core dev team thinks fancy to ad. Just think of the latest UDP initiative of jgarzik. Satoshi client is a moving target to other implementations and even to itself as it evolves. Although it is standard by its number of installations, this does not seem to be able to hold, as requirements diverge with more advanced use. All major Bitcoin services use modifications of Satoshi, BitcoinJ (soon bitsofproof) or proprietary implementations of the protocol. I plead for a standard defined by text, code examples and extensive set of tests. That standard should be derived and often purposefully extracted from Satoshi code but even future versions of Satoshi code should be measured on that testbed. Wait - this is actually we move toward. The growing list of test cases passed by Satoshi, BitcoinJ and bitsofproof are the building blocks of that new standard! And YES, the BF should use a lion share of its resources to define the standard set as its first goal at inception. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: Mike Hearn on April 01, 2013, 05:25:01 PM Sure, I'd love it if the Bitcoin protocol was specified entirely by comprehensive tests. It's just really hard to get there with confidence.
By the way, UDP support isn't merged, it's just a proposal. So that isn't a great example. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: grau on April 01, 2013, 05:36:25 PM Sure, I'd love it if the Bitcoin protocol was specified entirely by comprehensive tests. It's just really hard to get there with confidence. By the way, UDP support isn't merged, it's just a proposal. So that isn't a great example. Yes it is hard to get the comprehensive tests, but insisting on the entire Satoshi client as reference is not sustainable. UDP is an example of the tremendous misalignment of incentives between "core dev" and commercial use. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: jgarzik on April 01, 2013, 05:53:08 PM UDP is an example of the tremendous misalignment of incentives between "core dev" and commercial use. UDP has the potential to deliver blocks and transactions more rapidly throughout the network. The former benefits miners, and the latter benefits all bitcoin users. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: grau on April 01, 2013, 05:57:52 PM UDP is an example of the tremendous misalignment of incentives between "core dev" and commercial use. UDP has the potential to deliver blocks and transactions more rapidly throughout the network. The former benefits miners, and the latter benefits all bitcoin users.No doubt that it has a potential benefit. It would however have lowest priority on the wish list if ranked by commercial user. If Satoshi client was not the reference but the reference was independent tests, then such extension was permissible. Now it would compound the problem for minuscule upside. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: r.willis on April 18, 2013, 08:33:15 PM Status update:
Implemented parallel blockchain download. For now, only headers are downloaded. Download time ~4 min, CPU-bounded (need to investigate exact cause). Works fine, but not resistant to malicious nodes. Implemented some EC cryptography in pure Erlang. Slow, but easy to understand/verify. Switched from OpenVZ to XEN hosting on development node (still 256 MB Ram, but now with swap). Memory usage <30 MB resident, 100 MB virtual. Plans: download whole blocks and do actual validation. Maintain UTxO set, with rollback capability. Title: Re: [ANN] BitEN - bitcoin erlang node Post by: tvbcof on May 02, 2013, 05:53:31 PM I am now even more interested in BitEN for this reason http://www.paracoin.org/home/depth_l1/network_mesh (http://www.paracoin.org/home/depth_l1/network_mesh) Looking forward to updates, but will also be taking these from the repo pretty soon I hope. |