And actually, I'm going to stop my work on this until I have a clearer direction of which library/libraries I should be using, and if Mastercoin even wants this. I was a bit premature, sorry. Just let me know.

Good point on the "mastercoind". I'm good with whatever JR says here. My goal with this thing was to get it to a point and turn it over anyhow.

The optimal library for this purpose would be python, would interface with bitcoind via JSON RPC (so it had good cross-platform compatibility), and would take care of encoding/decoding of transactions, and maybe even processing to a embedded leveldb database (if it didn't do the latter, I could add that easily enough).

We don't have this optimal case today, so I have a few options:

* Zathras' libraries -- Windows-only it appears, but sound like it's the right focus. I just skimmed your code. If I found a way to interface the DLLs well to Python (I think it's possible), I could use them under Windows. Looks like I could use masterchest-library alone if I wanted to use something different than SQL for the serialization (e.g. write something higher level that used that to parse the blocks and serialize to leveldb, for instance), or combine in using masterchest-engine and a SQL backend. I really do like how you're interfacing around bitcoind here.

* grazcoin's stuff looks good for Linux, at least. However, it's still a bit rough around the edges for working with as a 3rd party library... mostly around how it's coupled to the web site interface he has, and how he serializes things. I have a few messages to him, still undecided here (the good news out of this is that I made some setup and init.d scripts that will be included in 'sx' though).

* A clean port of masterchest-library to python is another option here, too, since that interfaces with bitcoind, it could be used on both platforms.

* I looked at Tachikoma's stuff too.... ruby though (same situation).

Optimally, I'd want to use the same library and serialization strategy under both Windows and linux, at least for starters.

It may seem like this is a piece of cake, but my goal is something that is very easy to set up and works across multiple platforms, like bitcoind. With that, things become a lot more complicated - code/script wise. I already have a good amount of work into this (and will share my code once I get a basic version of it up), but if this is something one of you thinks you could do up to this level, and make available for both windows and Linux, just let me know and I'll stop my work on it. I don't want to waste your time or my time, I just want to see this done, which is why I started working on it.

Basically, guys, I'm asking for your advice. I'm willing to write up against something I can use for both Windows and Linux (or possibly take two different approaches). I'm also willing to package this up and make it easy to deploy on both platforms. Just let me know how I can best work with you all to get this done, and what you think the best library choice is for me here, given my parameters.

If this starts becoming political, or if I've ruffled feathers with this, then I'll just dump it on a github repo for folks to potentially pick over and move on. I'm not looking to establish a name or identity in any of this, I just want to see it done in one form or another, to help Mastercoin out, as well as my investment in it. That's all.

and building on windows looks like it will be a LOT of work. cygwin may be a good first step, as it has g++ 4.7 I believe...but from what I understand of it, there's still a bunch of work that has to happen to the Makefiles, etc. A native windows port (using vc++) would be even more work.

Yes, absolutely (that was my goal). Modify as needed and go for it! Then I can pull them out of mastercoind.

I really need to get sx working on windows as well... let me spend a bit of time and look into it.

The big challenge for me was to figure out how to use sx in the first place (I would just run it and it would hang)...so then I figured out obworker/obbalancer.

Since my goal was an automated setup for a Mastercoin-related project (mastercoind) that depends on 'sx', I made a script to allow for this (thats why some of the paths may reference mastercoind). You're welcome to use any of this code, if you find it helpful....it sure allows for more of an "out of the box" setup of everything:

setup script for actually configuring obworker and obbalancer (as their own user, etc): http://pastebin.com/MWJsaXKY
obbalancer init.d file: http://pastebin.com/43ivabs5
obworker init.d file: http://pastebin.com/MiFfgA3s
obworker log rotation: http://pastebin.com/udJypGSa

I agree with that. My initial post was a bit confusing up top, sorry about that, but I wasn't thinking that I was making the "reference library" here.....only that mastercoin needed one.

In my mind, mastercoind is simply a gateway/frontend to whatever the reference library is. At some point, once/if it makes sense, I would like mastercoind to possibly become the "reference client"/"reference front-end" that uses a separate backend reference library, but I have no say in that decision itself. As I said, it can be done if it makes sense, and people start building apps around this thing. Plus, I have no desire or intention of writing any mastercoin parsing library, or having to deal with the low-level details of one (this was the purpose I wanted a mastercoind so badly :p).

As I said earlier, right now I'm using grazcoin's MSC library, which is more utilities for his mastercoin site than a formal library ready for 3rd party use. This requires me to hack it a bit to get it to "fit in" to mastercoind. However, this is 110% better than me writing something from scratch. I see the backend library as taking care of those difficult details, and mastercoind wraps it up real nice and makes it very easy to use (as well as allowing it to be used by a client written in any language, or on a different server, etc ... list the list of advantages I have in my 2nd or 3rd post in this thread).

Maybe a future goal for mastercoind is to allow it to plug into 2 or more backend libraries...i.e. the "reference" one (whatever that is) and the main alternative. That way, you can get the similar front-end API functionality, but swap out the backend library if you need/want (e.g. for easy compatibility testing, or maybe one doesn't compile under windows and another one does, etc). The swap should be as easy as modifying a config file in the program, and making sure the library is installed on the system.

The end goal with this thing is to have something that's as easy to use, or easier, than bitcoind is today.

Actually, I just did see that bitcoind went to leveldb as of 0.8, so I moved us to using leveldb as well for this (I do like NoSQL a lot better than SQL...so much easier to deal with).

I agree with having a Windows and MacOSX build. However, building for Windows is hard right off to bat because of 'sx', which grazcoin's mastercoin library uses. There will need to be a decent amount of work put into creating a build script for windows that will get everything built and setup. Once that is available, then a windows version should be no problem. At first, Ubuntu is the first target because I know it well, and it's much easier to build things like this under it (compiling from source on windows --- especially in an automated fashion -- can be a big pain in the ass).

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

Good news, mastercoind is under development, see my last post over at: https://bitcointalk.org/index.php?topic=344229.0

Goal is to make a fully functional version that can parse simple sends (Class A and B). From there it can be extended.

After I get the initial version out, hopefully others can hack on it to and improve it to handle DEx transactions, and more. Would love to see a full time dev maintaining this!

I started development of this yesterday and have made a ton of progress after almost an all-nighter :p. Python using sqlite and cherrypy. Using grazcoin's mastercoin-tools for the mastercoin ops right now (which depends on sx). Eventually I'd like to move to bitcoind for better compatibility under Windows (or get sx compiling under windows -- looking into that), but for now this will work and will allow us to get off the ground much more quickly (grazcoin has done a good job).

The target at first is Ubuntu 13.10 (sx has major build issues with 12.04 LTS), and I have a full build script that will do everything from a stock system install.

At a minimum, even for Windows folks, one could set up a VM with Ubuntu 13.10, install mastercoind to that, and access it from your windows boxes across your LAN. I'll have a step by step how to set it up, once you get the base Ubuntu install on a VM.

Just got sx built under Ubuntu, thanks! Just wondering how portable 'sx' is to windows? Do I need cygwin, or is it possible to compile with VC++? Any build instructions anywhere (or pre-built binaries)?

LOL

Done, thread is at https://bitcointalk.org/index.php?topic=344229.0

This is what I initially thought as well (i.e. a reference library, either in a single language like python, or in C, which could be taken out to a variety of languages). However, as killerstorm has noted in his reply at https://bitcointalk.org/index.php?topic=265488.msg3665847#msg3665847 , most of the challenge lies in reconstructing the stream of Mastercoin messages from block to block, not in the parsing of the blocks themselves. In this case, I feel a separate resident daemon lends itself better to this (i.e. parsing transactions and maintaining state), and is superior to a reference library for the following reasons:

• one could use any language to interact with it, not just python (or whatever else)
• if other languages were desired, we wouldn't have to write separate implementations in each language (or be forced to use a lower level language and bind it up to the language of choice, if we wanted to share code, for instance)
• It has a very straightforward analogy to bitcoind, and can be quickly picked up by devs that have dealt with bitcoind
• like bitcoind, it can store its state independently of the application that's using it
• is potentially less error prone and "foolproof" in real-world usage, as it's more self-contained
• It can be distributed pre-compiled, with no binding/linking required

qwk: I just reset the poll to account for improving the options, given that I didn't want to try to unfairly re-interpret your vote. Please vote again if you wish.

ripper234 has asked me to create a thread to investigate the creation of a “mastercoind” program to help lower the bar to entry for the mastercoin development community. This is an idea I put forward on bitcointalk earlier¹.

The Problem

The basic idea is that Mastercoin transaction processing even today is quite involved and is going to get much harder as new features continue to be added to the spec. Currently, we have multiple efforts at creating a parsing framework for the stack, which both must be able to parse the various types of Mastercoin transactions (Class A, B and in the future, C transactions), as well as be able to apply the rules of the spec -– which itself is still quite nebulous in places -- to these transactions.

In the words of killerstorm, "Different implementations might implement it in different ways, and getting it right is very, very tricky."² For proof of this today, we can look at the frequent regressions and incompatibilities we are witnessing between the various implementations (e.g. mastercoin-explorer, mymastercoins, masterchest). For example, there have been several times when I put an MSC-containing address into these three tools, and got two or three different balances reported back. This is absolutely no fault to the developers themselves, who have been doing an excellent and admirable job of building something in quite a short timeframe (working part-time, too). However, it does reflect both the current ambiguous nature of the spec in areas, as well as the base complexity of the Mastercoin concept itself.

This situation will only become orders of magnitude more complex as additional features are layered on, and I believe we need to be proactive now and come up with a way to head this off at the pass, or else we will pay for it painfully later, in dealing with regressions and confusion instead of blazing forward with innovating new feature implementations.

The Solution

To this end, I believe Mastercoin needs a “reference-client” front-end API provider implementation, which, similar to bitcoind, would take care of implementing the “guts” of the spec and take it out to a higher level interface which would focus on things that have actually happened, after all of the Bitcoin and Mastercoin parsing and rule validation have been taken into effect. The daemon would start up, stay resident, and as new bitcoin blocks were processed by bitcoind, would parse and interpret any Mastercoind-transactions and make them available to its clients via an asynchronous (callback-driven) streamed event API, or a request-based synchronous API.

Then, instead of having to write their own parser and spec interpretation logic from scratch, Mastercoin developers could simply hook into mastercoind by spending 10 minutes writing the code to connect to mastercoind, subscribe to events, and decode the JSON-based responses for what they were interested in. Psuedo-code for an example event feed may be similar to:
•   "address A sent address B 50 MSC"
•   "address C put out an offer for 50 MSC @ .50/ea. offer ID is 1234".
•   "address D bid on offer 1234 for 20 of the 50 MSC"
•   "address E created a smart property named 'foobar' with currency ID 4567"

The first example, taken out to the JSON text that mastercoind may actually return, would be:


Like bitcoind, the daemon could be made available for Windows, Linux and Mac OS X.

The Benefits
The advantage of this approach is that instead of forcing each new potential Mastercoin developer to essentially write their own parsing and spec-interpretation logic, they could skip that and get started with the actual application they had in mind immediately. Suddenly the bar to Mastercoin development is drastically lowered, and participation by the majority of developers out there is now possible (where before, one would need to have domain-specific knowledge of Bitcoin and Mastercoin inner workings to participate – as most developers are not comfortable with bit twiddling and/or writing implementations to a spec).

This would bring Mastercoin development inline with Bitcoin development, where 90%+ of developers just hang-off of bitcoind and interact with it via its API interface. For the remaining 10% or less of devs that want or need to create their own parsing interface, they are totally free to do so, similar to how Bitcoin has the ‘sx’ toolset as an alternative to bitcoind.

Conclusion and Logistics

The result is an environment where the bar to new Mastercoin development is greatly lowered, and new developers can focus on writing applications that innovate and add value to the effort on day one, not after having to spend 3 months jumping through the hoops of hacking together their own block parsing and spec interpretation logic.

Mastercoind could be created and maintained by a 2 to 4 member development team employed by the foundation, and made available to the community. The community could then be leveraged (via bounties) to create solutions, most of which would be on top of mastercoind (however, alternative mastercoind implementations could still be created as well, as stated earlier). The result is a drastically larger and healthier ecosystem, with a drastically lowered bar to entry, and minimal reproduction of work.

Thanks for your consideration of this proposal. Please feel free to comment on any of these points, and I would be happy to answer any questions. And, if the community favors this, I would be willing to produce the skeleton for a 'mastercoind' in Python, which I can do in a 1 week period.

¹: The original postings where I proposed this:

• https://bitcointalk.org/index.php?topic=265488.msg3666117#msg3666117
• https://bitcointalk.org/index.php?topic=265488.msg3667049#msg3667049

²: See https://bitcointalk.org/index.php?topic=265488.msg3665847#msg3665847

Sure thing, do you want me to send the thread text to info@mastercoin.org, or start a thread on bitcointalk and send the link to info@mastercoin.org?

Hmm...to better help folks understand how this may look, would it be possible for you to plot a few graphs of this formula for 3 of so different scenarios over the next 12-24 months, e.g.:
* Linear MSC price rise, Bounty amount in USD stays about the same
* Exponential MSC price rise, Bounty amount in USD increases linearly
* Whatever else may make sense?

I know there are a number of variables expressed here...if you don't think it would be useful due to this, that's fine.

If I may ask, regarding wealth generation, what's your personal goal.... i.e., pay your bills, and then beyond that, stock up on an asset that is risky, but has a huge upside, or, pay your basic bills, and save a much more moderate amount in something that's much more of a "sure thing"? If the latter...what interested you in Bitcoin??

If you were given enough in fiat every month to pay your bills, and then MSC/BTC beyond that at a good level...would that help?

Personally, I'm used to spending my time on risky things, after putting up with insane risk for years...i.e. losing 6 figures in a blown up business, scraping by by a few K in the bank with a company that burns 70K/month and having to have my parents buy my groceries for a few months because I couldn't afford it, etc .... but I am a weird duck I guess...in my case it paid off, but, yes, those hard times lasted years and it took me awhile to adjust to that. Doesn't mean I like that much risk....just saying that because compared to crap like that, to me, this seems pretty safe, hahaha! 

If I didn't have this business that was growing so quick and taking 90% of my time, I'd be here coding for you guys in a heartbeat.

Wow, there are a lot of varying opinions here (as far as the foundation buying up MSC, how to pay devs, JR working full-time, etc). I think any unilateral action on something this big by the mastercoin foundation could risk alienating the community.

For any big issue that requires voting, how about we do this:

* Open a voting period of 1-2 weeks. A thread is started where 3 or 4 voting options are given (along with a 2 paragraph "best justification" for each).
* Each voting option has a corresponding bitcoin address
* To vote, the top 50 or 100 Mastercoin holders will send some small fraction of a mastercoin or bitcoin from their MSC holding address to one of the voting addresses. Some one then tallies up the votes, after removing any duplicate votes, verifying MSC balance, etc.

This way, voting is semi-anonymous, and one can easily verify voter eligibility (as well as perform recounts, if desired, for verification). It also gives the community a voice in this matter.

I do like the "eating your own dogfood" type ethos of this idea, but I share some of Kyune's well-stated concerns as well. This may make sense IF devs are willing to work for MSC (which I doubt, at least at this point, since it's so new and unstable, even compared to BTC).

On the up side, this action would definitely put the price of MSC way up...how much, depends on your buying strategy (e.g. constant, over X months, or managed/strategic where you buy on the dips to act sort of like a market maker of sorts and smooth out volatility, as well as get the best prices for the coins). Depending on the predictability of your acquisition strategy, the other risk is that other traders will front-run you (like exchange traders front run the fed QE/twist operations)...although I'm not sure of the impact in this case, as this is much more of a limited market than the stock market. (Maybe a more knowledgeable trader could comment.)

I think we've established that it's rising prices that attract press (at least with Bitcoin), not the other way around. In that event, it may very well kick off a virtuous cycle in Mastercoin's favor. However, we have to remember that most people are risk adverse, non-strategic momentum-chasers that will get in on a trend near its peak and cash out at the smallest sense of danger...mainstream media caters to that mania because it appeals to, well, the mainstream. This means that any raise may be short lived, similar to the earlier cycles bitcoin went though... Mastercoin will very well have a similar path of peaks and crashes along the way.

The other risk is that Mastercoin could be canned for manipulating their own market by some. Also, if the foundation favored MSC too much to BTC and the value of MSC plummeted in one of these fear-induced reactive sell-offs  (as JR said, it could randomly go down 90% on its way up, like bitcoin has), then the foundation would find itself in that "impotent startup" pickle that Kyune raised.