mastercoind proposal to speed development: Please read, vote, and comment

[rbdrbd]

ripper234 has asked me to create a thread to investigate the creation of a “mastercoind” program to help lower the bar to entry for the mastercoin development community. This is an idea I put forward on bitcointalk earlier¹.

The Problem

The basic idea is that Mastercoin transaction processing even today is quite involved and is going to get much harder as new features continue to be added to the spec. Currently, we have multiple efforts at creating a parsing framework for the stack, which both must be able to parse the various types of Mastercoin transactions (Class A, B and in the future, C transactions), as well as be able to apply the rules of the spec -– which itself is still quite nebulous in places -- to these transactions.

In the words of killerstorm, "Different implementations might implement it in different ways, and getting it right is very, very tricky."² For proof of this today, we can look at the frequent regressions and incompatibilities we are witnessing between the various implementations (e.g. mastercoin-explorer, mymastercoins, masterchest). For example, there have been several times when I put an MSC-containing address into these three tools, and got two or three different balances reported back. This is absolutely no fault to the developers themselves, who have been doing an excellent and admirable job of building something in quite a short timeframe (working part-time, too). However, it does reflect both the current ambiguous nature of the spec in areas, as well as the base complexity of the Mastercoin concept itself.

This situation will only become orders of magnitude more complex as additional features are layered on, and I believe we need to be proactive now and come up with a way to head this off at the pass, or else we will pay for it painfully later, in dealing with regressions and confusion instead of blazing forward with innovating new feature implementations.

The Solution

To this end, I believe Mastercoin needs a “reference-client” front-end API provider implementation, which, similar to bitcoind, would take care of implementing the “guts” of the spec and take it out to a higher level interface which would focus on things that have actually happened, after all of the Bitcoin and Mastercoin parsing and rule validation have been taken into effect. The daemon would start up, stay resident, and as new bitcoin blocks were processed by bitcoind, would parse and interpret any Mastercoind-transactions and make them available to its clients via an asynchronous (callback-driven) streamed event API, or a request-based synchronous API.

Then, instead of having to write their own parser and spec interpretation logic from scratch, Mastercoin developers could simply hook into mastercoind by spending 10 minutes writing the code to connect to mastercoind, subscribe to events, and decode the JSON-based responses for what they were interested in. Psuedo-code for an example event feed may be similar to:
•   "address A sent address B 50 MSC"
•   "address C put out an offer for 50 MSC @ .50/ea. offer ID is 1234".
•   "address D bid on offer 1234 for 20 of the 50 MSC"
•   "address E created a smart property named 'foobar' with currency ID 4567"

The first example, taken out to the JSON text that mastercoind may actually return, would be:

Code:

{“operation”: “sendmsc”, “from”: “1CE8bBr1dYZRMnpmyYsFEoexa1YoPz2mfB”, “to”: “1EcpemQkcq9Dit7yF9hxEKZzGVkAScwDrP”, “transactionID”: “3f0bcd2930[…]”, “amount”: 50, “block”: 23849202, “when”: “20101117T09:42:00“}

Like bitcoind, the daemon could be made available for Windows, Linux and Mac OS X.

The Benefits
The advantage of this approach is that instead of forcing each new potential Mastercoin developer to essentially write their own parsing and spec-interpretation logic, they could skip that and get started with the actual application they had in mind immediately. Suddenly the bar to Mastercoin development is drastically lowered, and participation by the majority of developers out there is now possible (where before, one would need to have domain-specific knowledge of Bitcoin and Mastercoin inner workings to participate – as most developers are not comfortable with bit twiddling and/or writing implementations to a spec).

This would bring Mastercoin development inline with Bitcoin development, where 90%+ of developers just hang-off of bitcoind and interact with it via its API interface. For the remaining 10% or less of devs that want or need to create their own parsing interface, they are totally free to do so, similar to how Bitcoin has the ‘sx’ toolset as an alternative to bitcoind.

Conclusion and Logistics

The result is an environment where the bar to new Mastercoin development is greatly lowered, and new developers can focus on writing applications that innovate and add value to the effort on day one, not after having to spend 3 months jumping through the hoops of hacking together their own block parsing and spec interpretation logic.

Mastercoind could be created and maintained by a 2 to 4 member development team employed by the foundation, and made available to the community. The community could then be leveraged (via bounties) to create solutions, most of which would be on top of mastercoind (however, alternative mastercoind implementations could still be created as well, as stated earlier). The result is a drastically larger and healthier ecosystem, with a drastically lowered bar to entry, and minimal reproduction of work.

Thanks for your consideration of this proposal. Please feel free to comment on any of these points, and I would be happy to answer any questions. And, if the community favors this, I would be willing to produce the skeleton for a 'mastercoind' in Python, which I can do in a 1 week period.

¹: The original postings where I proposed this:

• https://bitcointalk.org/index.php?topic=265488.msg3666117#msg3666117
• https://bitcointalk.org/index.php?topic=265488.msg3667049#msg3667049

²: See https://bitcointalk.org/index.php?topic=265488.msg3665847#msg3665847

[1714653630]

1714653630

[qwk]

If all you want is some kind of reference implementation to follow or use in real-world-applications, I don't see a need for a mastercoinD, rather, a reference library in the language of choice (I guess that would be Python for most developers right now).

[rbdrbd]

If all you want is some kind of reference implementation to follow or use in real-world-applications, I don't see a need for a mastercoinD, rather, a reference library in the language of choice (I guess that would be Python for most developers right now).

This is what I initially thought as well (i.e. a reference library, either in a single language like python, or in C, which could be taken out to a variety of languages). However, as killerstorm has noted in his reply at https://bitcointalk.org/index.php?topic=265488.msg3665847#msg3665847 , most of the challenge lies in reconstructing the stream of Mastercoin messages from block to block, not in the parsing of the blocks themselves. In this case, I feel a separate resident daemon lends itself better to this (i.e. parsing transactions and maintaining state), and is superior to a reference library for the following reasons:

• one could use any language to interact with it, not just python (or whatever else)
• if other languages were desired, we wouldn't have to write separate implementations in each language (or be forced to use a lower level language and bind it up to the language of choice, if we wanted to share code, for instance)
• It has a very straightforward analogy to bitcoind, and can be quickly picked up by devs that have dealt with bitcoind
• like bitcoind, it can store its state independently of the application that's using it
• is potentially less error prone and "foolproof" in real-world usage, as it's more self-contained
• It can be distributed pre-compiled, with no binding/linking required

qwk: I just reset the poll to account for improving the options, given that I didn't want to try to unfairly re-interpret your vote. Please vote again if you wish.

[qwk]

• like bitcoind, it can store its state independently of the application that's using it
• is potentially less error prone and "foolproof" in real-world usage, as it's more self-contained

Okay, I'll bite. Makes perfect sense

[prophetx]

Could one of the existing projects be repackaged to be this?

https://bitcointalk.org/index.php?topic=292628.msg3695215#msg3695215

[Ola]

+1  Makes all the sense in the world..I am waiting on this

[rbdrbd]

I started development of this yesterday and have made a ton of progress after almost an all-nighter :p. Python using sqlite and cherrypy. Using grazcoin's mastercoin-tools for the mastercoin ops right now (which depends on sx). Eventually I'd like to move to bitcoind for better compatibility under Windows (or get sx compiling under windows -- looking into that), but for now this will work and will allow us to get off the ground much more quickly (grazcoin has done a good job).

The target at first is Ubuntu 13.10 (sx has major build issues with 12.04 LTS), and I have a full build script that will do everything from a stock system install.

At a minimum, even for Windows folks, one could set up a VM with Ubuntu 13.10, install mastercoind to that, and access it from your windows boxes across your LAN. I'll have a step by step how to set it up, once you get the base Ubuntu install on a VM.

[Ola]

I started development of this yesterday and have made a ton of progress after almost an all-nighter :p. Python using sqlite and cherrypy. Using grazcoin's mastercoin-tools for the mastercoin ops right now (which depends on sx). Eventually I'd like to move to bitcoind for better compatibility under Windows (or get sx compiling under windows -- looking into that), but for now this will work and will allow us to get off the ground much more quickly (grazcoin has done a good job).

The target at first is Ubuntu 13.10 (sx has major build issues with 12.04 LTS), and I have a full build script that will do everything from a stock system install.

At a minimum, even for Windows folks, one could set up a VM with Ubuntu 13.10, install mastercoind to that, and access it from your windows boxes across your LAN. I'll have a step by step how to set it up, once you get the base Ubuntu install on a VM.

I too stayed up al night yesterday working on a project, nothing close to mastercoin/bitcoin related though. I am glad you are already leading this charge, i commend you for this. Any upcoming technical difficulties that you foresee on your path? and why not build for Windows and OSX first? the masses...and any particular reason that you choose relational databases? there are a lot of drivers for NoSQL based databases u know ..just curious.

[zbx]

For what it's worth, I fully support this endeavour.

[Tachikoma]

Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

[rbdrbd]

Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

[Ola]

Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

+1000

[ripper234]

+1 for repackaging one or more of the existing implementations and providing various APIs to it.
APIs should be HTTP, RPC, and more ... the more endpoints we have, the more useful an implementation gets.

I'd like to consolidate "development firepower" on one or a few implementations, not to create yet another Mastercoin client.

[ripper234]

Ah sorry I missed that you're wrapping grazcoin's implementation.
That's awesome then

[Tachikoma]

Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

Ok thanks for clarifying you are not actually creating a client but a RPC wrapper on top of a client.

I still think J.R. should be the one who writes the reference client since he can make decisions on how the spec should be interpreted. No offence intended, but who gave you the right to decide your implementation should be the reference one. It's not so much how the clients handles, you can strap an RPC client on top of any implementation, but the decisions you make in the underlying layers.

I think an RPC accessible client on top of any library is a excellent idea, I just think that the word reference implementation should be removed. 

[rbdrbd]

I started development of this yesterday and have made a ton of progress after almost an all-nighter :p. Python using sqlite and cherrypy. Using grazcoin's mastercoin-tools for the mastercoin ops right now (which depends on sx). Eventually I'd like to move to bitcoind for better compatibility under Windows (or get sx compiling under windows -- looking into that), but for now this will work and will allow us to get off the ground much more quickly (grazcoin has done a good job).

The target at first is Ubuntu 13.10 (sx has major build issues with 12.04 LTS), and I have a full build script that will do everything from a stock system install.

At a minimum, even for Windows folks, one could set up a VM with Ubuntu 13.10, install mastercoind to that, and access it from your windows boxes across your LAN. I'll have a step by step how to set it up, once you get the base Ubuntu install on a VM.

I too stayed up al night yesterday working on a project, nothing close to mastercoin/bitcoin related though. I am glad you are already leading this charge, i commend you for this. Any upcoming technical difficulties that you foresee on your path? and why not build for Windows and OSX first? the masses...and any particular reason that you choose relational databases? there are a lot of drivers for NoSQL based databases u know ..just curious.

Actually, I just did see that bitcoind went to leveldb as of 0.8, so I moved us to using leveldb as well for this (I do like NoSQL a lot better than SQL...so much easier to deal with).

I agree with having a Windows and MacOSX build. However, building for Windows is hard right off to bat because of 'sx', which grazcoin's mastercoin library uses. There will need to be a decent amount of work put into creating a build script for windows that will get everything built and setup. Once that is available, then a windows version should be no problem. At first, Ubuntu is the first target because I know it well, and it's much easier to build things like this under it (compiling from source on windows --- especially in an automated fashion -- can be a big pain in the ass).

[rbdrbd]

Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

Ok thanks for clarifying you are not actually creating a client but a RPC wrapper on top of a client.

I still think J.R. should be the one who writes the reference client since he can make decisions on how the spec should be interpreted. No offence intended, but who gave you the right to decide your implementation should be the reference one. It's not so much how the clients handles, you can strap an RPC client on top of any implementation, but the decisions you make in the underlying layers.

I think an RPC accessible client on top of any library is a excellent idea, I just think that the word reference implementation should be removed.  

I agree with that. My initial post was a bit confusing up top, sorry about that, but I wasn't thinking that I was making the "reference library" here.....only that mastercoin needed one.

In my mind, mastercoind is simply a gateway/frontend to whatever the reference library is. At some point, once/if it makes sense, I would like mastercoind to possibly become the "reference client"/"reference front-end" that uses a separate backend reference library, but I have no say in that decision itself. As I said, it can be done if it makes sense, and people start building apps around this thing. Plus, I have no desire or intention of writing any mastercoin parsing library, or having to deal with the low-level details of one (this was the purpose I wanted a mastercoind so badly :p).

As I said earlier, right now I'm using grazcoin's MSC library, which is more utilities for his mastercoin site than a formal library ready for 3rd party use. This requires me to hack it a bit to get it to "fit in" to mastercoind. However, this is 110% better than me writing something from scratch. I see the backend library as taking care of those difficult details, and mastercoind wraps it up real nice and makes it very easy to use (as well as allowing it to be used by a client written in any language, or on a different server, etc ... list the list of advantages I have in my 2nd or 3rd post in this thread).

Maybe a future goal for mastercoind is to allow it to plug into 2 or more backend libraries...i.e. the "reference" one (whatever that is) and the main alternative. That way, you can get the similar front-end API functionality, but swap out the backend library if you need/want (e.g. for easy compatibility testing, or maybe one doesn't compile under windows and another one does, etc). The swap should be as easy as modifying a config file in the program, and making sure the library is installed on the system.

The end goal with this thing is to have something that's as easy to use, or easier, than bitcoind is today.

[djohnston]

rbdrbd,

Thanks for taking the initiative on this effort. I agree that a reference client for development purposes can be very helpful to new developers getting involved in Mastercoin.

I hear where Tachikoma is coming from with his concerns though and lets make sure that what gets developer here is sensitive to consensus from developers and the community. I wonder if the reference client would be another good place for the Proof of Stake voting to be involved or if the community thinks we should keep Proof of Stake at the Protocol Spec level?

I'm sure J.R. will chime in on the effort. But not likely today, as he does the family thing on the weekends and isn't on the computer.

[zathras]

Hey rbdrbd and welcome to Mastercoin development

I thought I'd chime in on a couple of points - I have to agree with Tachikoma on this one, whether or not you state it's the reference implementation, the choice of name (mastercoind) will lead some members of the community to perhaps think it is.  I would love to see any reference implementation written by JR too.

Simply put, the project will likely have a reference mastercoin daemon at some point and I think it would be courteous to leave the 'mastercoind' moniker for that purpose - you could always rename your output binary later if Grazcoin/your work ended up becoming the reference implementation.

Good luck with your wrapper!

P.S. (sorry for plugging my own stuff in your thread ) but masterchest_engine & masterchest_library have been around for a while to offer devs a low barrier to entry - the library already does decoding/encoding/parsing/bitcoind integration etc on behalf of the dev and the engine processes all transactions to arrive at a known/current state so for example web devs can query balances/transactions/exchange state etc without knowing anything other than a bit of SQL.  They're of course works in progress but since your post kind of implies before your wrapper devs had no choice but to start from scratch, I wanted to clear up that misconception.

Thanks!

[rbdrbd]

Hey rbdrbd and welcome to Mastercoin development

I thought I'd chime in on a couple of points - I have to agree with Tachikoma on this one, whether or not you state it's the reference implementation, the choice of name (mastercoind) will lead some members of the community to perhaps think it is.  I would love to see any reference implementation written by JR too.

Simply put, the project will likely have a reference mastercoin daemon at some point and I think it would be courteous to leave the 'mastercoind' moniker for that purpose - you could always rename your output binary later if Grazcoin/your work ended up becoming the reference implementation.

Good luck with your wrapper!

P.S. (sorry for plugging my own stuff in your thread ) but masterchest_engine & masterchest_library have been around for a while to offer devs a low barrier to entry - the library already does decoding/encoding/parsing/bitcoind integration etc on behalf of the dev and the engine processes all transactions to arrive at a known/current state so for example web devs can query balances/transactions/exchange state etc without knowing anything other than a bit of SQL.  They're of course works in progress but since your post kind of implies before your wrapper devs had no choice but to start from scratch, I wanted to clear up that misconception.

Thanks!

Good point on the "mastercoind". I'm good with whatever JR says here. My goal with this thing was to get it to a point and turn it over anyhow.

The optimal library for this purpose would be python, would interface with bitcoind via JSON RPC (so it had good cross-platform compatibility), and would take care of encoding/decoding of transactions, and maybe even processing to a embedded leveldb database (if it didn't do the latter, I could add that easily enough).

We don't have this optimal case today, so I have a few options:

* Zathras' libraries -- Windows-only it appears, but sound like it's the right focus. I just skimmed your code. If I found a way to interface the DLLs well to Python (I think it's possible), I could use them under Windows. Looks like I could use masterchest-library alone if I wanted to use something different than SQL for the serialization (e.g. write something higher level that used that to parse the blocks and serialize to leveldb, for instance), or combine in using masterchest-engine and a SQL backend. I really do like how you're interfacing around bitcoind here.

* grazcoin's stuff looks good for Linux, at least. However, it's still a bit rough around the edges for working with as a 3rd party library... mostly around how it's coupled to the web site interface he has, and how he serializes things. I have a few messages to him, still undecided here (the good news out of this is that I made some setup and init.d scripts that will be included in 'sx' though).

* A clean port of masterchest-library to python is another option here, too, since that interfaces with bitcoind, it could be used on both platforms.

* I looked at Tachikoma's stuff too.... ruby though (same situation).

Optimally, I'd want to use the same library and serialization strategy under both Windows and linux, at least for starters.

It may seem like this is a piece of cake, but my goal is something that is very easy to set up and works across multiple platforms, like bitcoind. With that, things become a lot more complicated - code/script wise. I already have a good amount of work into this (and will share my code once I get a basic version of it up), but if this is something one of you thinks you could do up to this level, and make available for both windows and Linux, just let me know and I'll stop my work on it. I don't want to waste your time or my time, I just want to see this done, which is why I started working on it.

Basically, guys, I'm asking for your advice. I'm willing to write up against something I can use for both Windows and Linux (or possibly take two different approaches). I'm also willing to package this up and make it easy to deploy on both platforms. Just let me know how I can best work with you all to get this done, and what you think the best library choice is for me here, given my parameters.

If this starts becoming political, or if I've ruffled feathers with this, then I'll just dump it on a github repo for folks to potentially pick over and move on. I'm not looking to establish a name or identity in any of this, I just want to see it done in one form or another, to help Mastercoin out, as well as my investment in it. That's all.