Mastercoin reference implementation proposal.

[Tachikoma]

Hey guys,

I wanted to give my proposal for an official Mastercoin daemon reference implementation. I think enough valid arguments have already been raised why this is needed but just to confirm.

• It would enable rapid development of products based on Mastercoin without having to know all the ins and outs of the protocol.
• It would serve as a guideline for other implementations on how interpret the rules.
• It would make it easier to get Mastercoin trading on exchanges.

Now when creating a reference implementation the following features are of major importance.

• Cross-platform
• The least amount of dependencies
• Optimised for speed and security
• Readable code and documentation
• Testable (TDD/BDD)
• High level API on top

I've discussed this a lot over the last few weeks with a dear friend of mine, who hopefully be involved with the project soon too, and we came up with the following suggestions.

We would develop a reference implementation in Go using Leveldb as data storage. The application would exist of multiple levels and be developer in the following steps

• Reader module; the reader module would be responsible for reading and listening for messages from different sources (this doesn't have to be Bitcoin perse but would be so initially).
• Parser; retrieves and parses data specifically for Mastercoin and save it to a key value storage (leveldb).
• API Library; the higher level API library which serves as a wrapper around the reader, parser and database interface (so it should be possible to develop upon the specification, rather than the JSON RPC)
• JSON-RPC; A JSON interface wrapper around the library API

The lowest level would be the reader module. This module would read messages from different sources. Initially this would be the Bitcoin block chain. The reader module would be responsible for listing in on a network, gathering messages and then saving these messages to be parsed by the second module. The second module would be the parser itself. It would go through all unparsed messages and retrieve the Mastercoin data. It would then go through all the logic dictated in the spec and save the data in a presentable form in the leveldb database. The last module would be a JSON-RPC interface that gives easy access to all the data saved in the database. There will be no hard dependency on any Bitcoin specific application like Bitcoind. We want it to be as much as stand-alone project as possible carefully weighing development against the dependency cost.

We are considering Go for the following reasons.

• Cross-platform portability and cross-platform compilable
• Easy multithreading
• Secure out of the box. I.e. buffer overflows can't lead to arbitrary code execution
• Build in garbage collection
• Easy dependency management
• Easy syntax, compared to languages like c

Of course there are also downsides to Go.

• Although production-ready it's a relatively young language
• Slower to develop than Python / Ruby

This would be no small project and would probably not make any quick gains. However we believe this might be what is needed for the the future stability of Mastercoin. The idea is that I would keep my curren Ruby libraries up to date with the new features while simultaneously working on the Go implementation. However my friend would have the main role and would be doing the most work on it.

Please voice your opinions and concerns so we can come to a final solution for this problem.

[BitThink]

+1 for Go. Definitely the best language for this purpose. Portable and almost zero dependency. Reasonable speed both in running and developing.

[zbx]

That's an excellent proposal, Tachikoma. You have my full support in this.

[puck2]

I am less familiar with Go, but I trust your judgement in that respect, and do truly believe that this is, overall, a good direction for the Mastercoin project to head in.

[rbdrbd]

This sounds like a solid proposition. At first I was against using some rather obscure language like Go, but I did some research on it and came away impressed.

It's not clear from your proposal, but I would strongly recommend against us writing our own bitcoin block chain parsing logic. Instead, I would recommend we use SWiG to integrate up against libbitcoin (which already can write out to a leveldb backend):

• Go and linking to C++ libraries: http://www.swig.org/Doc2.0/Go.html
• Libbitcoin: https://bitcointalk.org/index.php?topic=30646.0

Writing our own block chain implementation would be a big hassle to both do and maintain, when there's already an excellent library for this. The only hang up here is I don't believe it's been compiled under Windows yet...but that would be a good use of a bounty possibly (i.e. get the dev to spend some time to get a build under windows going...would enhance libbitcoin AND Mastercoin).

Failing that, I would just initially integrate up against bitcoind, to minimize time-to-market as much as possible. Once this thing is out and running up against bitcoind, time can be taken to develop the necessary functionality to remove that dependency. (This guy has some Go code that interacts with bitcoind on Github, which may be a good start for bootstrapping: https://github.com/GeertJohan/go.bitcoin)

I'm encouraged we're finally getting to this point in the conversation. We needed a mastercoind out yesterday.

[Tachikoma]

Actually we will be using libraries wherever possible, as long as they don't bring in hard depdencies, like Bitcoind. I've already talked with Amir about using Libbotcoin for Mastercoin purposes. An other option we came up with is using parts of BTCD which is a pure Go implementation of the Bitcoin reference implementation. It is still alpha but it's already passing block acceptance tests so it looks like we can cut development time by using parts of their libs. The only problem is we haven't had time to really dive into their code and see how useful it would be for us.

We have considered using Bitcoind as an interim solution however we rather cut that out since it would be wasting a lot of resources. We only need a small part of the Blockchain it makes no sense to waste all those resources saving data on transactions we will never use.

[rbdrbd]

Actually we will be using libraries wherever possible, as long as they don't bring in hard depdencies, like Bitcoind. I've already talked with Amir about using Libbotcoin for Mastercoin purposes. An other option we came up with is using parts of BTCD which is a pure Go implementation of the Bitcoin reference implementation. It is still alpha but it's already passing block acceptance tests so it looks like we can cut development time by using parts of their libs. The only problem is we haven't had time to really dive into their code and see how useful it would be for us.

We have considered using Bitcoind as an interim solution however we rather cut that out since it would be wasting a lot of resources. We only need a small part of the Blockchain it makes no sense to waste all those resources saving data on transactions we will never use.

Ok, that's good that the focus is using other people's solid code whereever possible.

Given this, would you be willing to risk a guess as to how long it may take to get v1.0 of this out, given that you have your friend and maybe yourself working full-time on it? Being a developer by trade myself, I know how off these kinds of high level estimates can sometimes be, so if you are not sure, that's fine too.

(I just look at Nxt coming up along side, as well as the success Protoshares is having on the exchanges. Things are ripe for this.)

[Tachikoma]

I can't say anything about that yet. Even if I would take the job I would need a grace period at my old job since with us leaving the whole company might break down. I would probably need at least three months to make sure they stay afloat. I can already start working before that time but it won't be fulltime.

I don't think we need to hurry this thing though. Right now we mostly need new core messages supported. You can't build that much on top of just simple sends. The more value we add to the protocol the more a need for a Mastercoin reference implementation there will be. I rather do this one right then rushed.

I will see if I can do some discovery work to make a estimate about the development time for an early alpha version.

[z0mbie]

Just wanted to jump in and say Hi!

I'm the 'friend' Tachikoma is referring to :-) If everything goes well I'll be dedicating my time developing the Go implementation.

Regarding the time frame for 1.0; it would be too early to tell. I'm sure that once we start plotting out a proper course and thinker in finer detail we could give a rough estimate, for now it's just too early.

Also I don't think we should focus on a quick and small gain and believe that much more support can be found in a proper, solid implementation. I know from experience that building on a poor foundation leads to nothing but trouble and unhappy developers

Anyways, just thought I'd share my thoughts

[zathras]

Hey guys,

Well if I were smart financially I should come out against this and be pro rbdrbd's proposal (since he's using my library & engine = better result for me in the contest).

But... I have to get behind what I think is best for Mastercoin as a whole. Rbdrbd was taking my disparate components (eg lib & engine) and wrapping to come up with a daemon, where as Tachikoma is proposing a ground up build where design intent is a daemon from the get go.

As such fwiw I'm behind this.  Two primary reasons - firstly I completely agree with the benefits a reference client would bring (hence my support of rbdrbd's efforts), secondly Tachikoma's abilities are a 'known commodity' - ie we already know he has the capability to deliver.

Thanks!
Zathras

EDIT: Not to downplay rbdrbd's efforts at all, I'm still of course happy to support you if you end up continuing with your masterdaemon project.

[rbdrbd]

EDIT: Not to downplay rbdrbd's efforts at all, I'm still of course happy to support you if you end up continuing with your masterdaemon project.

I'm down for what's best for the community, given that we can bring something to market quickly and intelligently without sacrificing quality (which I was never advocating doing, BTW ). This is why I put forward the idea of me completing masterdaemon before I actually went off and did it...I do have a decent amount of time already sunk into it, but didn't want to finish it up without getting everyone's feedback for exactly this reason -- namely that something else would pop up.

I put forward masterdaemon and the general concept initially because we needed *something*. As this team becomes more established and the idea gains traction, it makes sense that we would have a more formalized product materialize. Perhaps some of the documentation can end up being picked over, or something (or I can get a few MSC for my troubles, hah!).

Let's definitely have a thought out design and "do it right", but especially if we have 1 or more people working on it full-time and we make maximal use of established libraries where it makes sense, I would think we should be able to put our heads together and come up with something relatively quickly.

I'm excited about the prospects of us all working together to assemble a talented and experienced full-time team to work on this and other core products.

[z0mbie]

The plan so far is me fulltime go while Tachikoma continues his work on his ruby implementation and help out on go when he feels like.

The planning as I see it would be in several stages and will be set up in a modular fashion. Meaning that the daemon would actually exist of more components that can be reused where ever possible. This to ship out individual parts quickly and gaining momentum in the development process. It's easier to hack on individual parts/modules (for 'outsiders') rather than one large project and should (hopefully) gain some support and feedback from other developers.

And because the project would exist of 'small' components planning should be more accurate (though I would never sacrifice quality over making a deadline, if there ever are such things).

[Bitoy]

Go will be used to listen to the exodus address (and probably other addresses)
it parses and saves the transactions (block number, date, reference address, transaction type currency etc ) in LevelDb.

All  developers can then access Go via api or json to get the transactions.

Is this correct?

[Tachikoma]

That could be seen as a summary; yes.

[Bitoy]

Ok.  It's a Go


 z0mbie, looking forward to see your work in Go. 

[rbdrbd]

z0mbie, thanks for stepping up. Do you have a resume or CV you could send to the board and zbx, who has been helping out with hiring for these full time devs? Just to give a bit of a better idea of your background?

(apologies if you've already done this...just wasn't sure)

[Tachikoma]

z0mbie, thanks for stepping up. Do you have a resume or CV you could send to the board and zbx, who has been helping out with hiring for these full time devs? Just to give a bit of a better idea of your background?

(apologies if you've already done this...just wasn't sure)

One of my 'demands' was that if I would be a hire; z0mbie would be as well. I already talked this over with Ron and he had no problems with this. I don't think supplying a resume would be a problem though

[rbdrbd]

z0mbie, thanks for stepping up. Do you have a resume or CV you could send to the board and zbx, who has been helping out with hiring for these full time devs? Just to give a bit of a better idea of your background?

(apologies if you've already done this...just wasn't sure)

One of my 'demands' was that if I would be a hire; z0mbie would be as well. I already talked this over with Ron and he had no problems with this. I don't think supplying a resume would be a problem though

Ok, I wasn't aware of that. Ron had also tasked zbx to help out/take care of the dev hiring as a Tech Lead candidate. I think there is a little bit of confusion from that all, then. (unless zbx was aware of this, then it's just me being out of the loop )

Not trying to be difficult, I'd just like to get a better idea of his experience and background.

[Tachikoma]

I will let him talk about his background, I already have a pm-conversation going with zbx. Let's try to stay on-topic

[rbdrbd]

I will let him talk about his background, I already have a pm-conversation going with zbx. Let's try to stay on-topic

You proposed a technological solution. There didn't seem to be much objection to that (or what there was was cleared up). The next step is determining who will work on it full time (as this is the core product for the full-time team). I asked a question to that extent. How is that not on topic?

Actually, nevermind. If you and zbx are talking that's fine. I'll get the info through him.