Bitcoin Forum

Alternate cryptocurrencies => Altcoin Discussion => Topic started by: rbdrbd on November 23, 2013, 06:16:57 PM



Title: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 23, 2013, 06:16:57 PM
ripper234 has asked me to create a thread to investigate the creation of a “mastercoind” program to help lower the bar to entry for the mastercoin development community. This is an idea I put forward on bitcointalk earlier1.

The Problem

The basic idea is that Mastercoin transaction processing even today is quite involved and is going to get much harder as new features continue to be added to the spec. Currently, we have multiple efforts at creating a parsing framework for the stack, which both must be able to parse the various types of Mastercoin transactions (Class A, B and in the future, C transactions), as well as be able to apply the rules of the spec -– which itself is still quite nebulous in places -- to these transactions.

In the words of killerstorm, "Different implementations might implement it in different ways, and getting it right is very, very tricky."2 For proof of this today, we can look at the frequent regressions and incompatibilities we are witnessing between the various implementations (e.g. mastercoin-explorer, mymastercoins, masterchest). For example, there have been several times when I put an MSC-containing address into these three tools, and got two or three different balances reported back. This is absolutely no fault to the developers themselves, who have been doing an excellent and admirable job of building something in quite a short timeframe (working part-time, too). However, it does reflect both the current ambiguous nature of the spec in areas, as well as the base complexity of the Mastercoin concept itself.

This situation will only become orders of magnitude more complex as additional features are layered on, and I believe we need to be proactive now and come up with a way to head this off at the pass, or else we will pay for it painfully later, in dealing with regressions and confusion instead of blazing forward with innovating new feature implementations.

The Solution

To this end, I believe Mastercoin needs a “reference-client” front-end API provider implementation, which, similar to bitcoind, would take care of implementing the “guts” of the spec and take it out to a higher level interface which would focus on things that have actually happened, after all of the Bitcoin and Mastercoin parsing and rule validation have been taken into effect. The daemon would start up, stay resident, and as new bitcoin blocks were processed by bitcoind, would parse and interpret any Mastercoind-transactions and make them available to its clients via an asynchronous (callback-driven) streamed event API, or a request-based synchronous API.

Then, instead of having to write their own parser and spec interpretation logic from scratch, Mastercoin developers could simply hook into mastercoind by spending 10 minutes writing the code to connect to mastercoind, subscribe to events, and decode the JSON-based responses for what they were interested in. Psuedo-code for an example event feed may be similar to:
•   "address A sent address B 50 MSC"
•   "address C put out an offer for 50 MSC @ .50/ea. offer ID is 1234".
•   "address D bid on offer 1234 for 20 of the 50 MSC"
•   "address E created a smart property named 'foobar' with currency ID 4567"

The first example, taken out to the JSON text that mastercoind may actually return, would be:

Code:
{“operation”: “sendmsc”, “from”: “1CE8bBr1dYZRMnpmyYsFEoexa1YoPz2mfB”, “to”: “1EcpemQkcq9Dit7yF9hxEKZzGVkAScwDrP”, “transactionID”: “3f0bcd2930[…]”, “amount”: 50, “block”: 23849202, “when”: “20101117T09:42:00“}

Like bitcoind, the daemon could be made available for Windows, Linux and Mac OS X.

The Benefits
The advantage of this approach is that instead of forcing each new potential Mastercoin developer to essentially write their own parsing and spec-interpretation logic, they could skip that and get started with the actual application they had in mind immediately. Suddenly the bar to Mastercoin development is drastically lowered, and participation by the majority of developers out there is now possible (where before, one would need to have domain-specific knowledge of Bitcoin and Mastercoin inner workings to participate – as most developers are not comfortable with bit twiddling and/or writing implementations to a spec).

This would bring Mastercoin development inline with Bitcoin development, where 90%+ of developers just hang-off of bitcoind and interact with it via its API interface. For the remaining 10% or less of devs that want or need to create their own parsing interface, they are totally free to do so, similar to how Bitcoin has the ‘sx’ toolset as an alternative to bitcoind.

Conclusion and Logistics

The result is an environment where the bar to new Mastercoin development is greatly lowered, and new developers can focus on writing applications that innovate and add value to the effort on day one, not after having to spend 3 months jumping through the hoops of hacking together their own block parsing and spec interpretation logic.

Mastercoind could be created and maintained by a 2 to 4 member development team employed by the foundation, and made available to the community. The community could then be leveraged (via bounties) to create solutions, most of which would be on top of mastercoind (however, alternative mastercoind implementations could still be created as well, as stated earlier). The result is a drastically larger and healthier ecosystem, with a drastically lowered bar to entry, and minimal reproduction of work.

Thanks for your consideration of this proposal. Please feel free to comment on any of these points, and I would be happy to answer any questions. And, if the community favors this, I would be willing to produce the skeleton for a 'mastercoind' in Python, which I can do in a 1 week period.

1: The original postings where I proposed this:
  • https://bitcointalk.org/index.php?topic=265488.msg3666117#msg3666117
  • https://bitcointalk.org/index.php?topic=265488.msg3667049#msg3667049

2: See https://bitcointalk.org/index.php?topic=265488.msg3665847#msg3665847


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: qwk on November 23, 2013, 06:20:45 PM
If all you want is some kind of reference implementation to follow or use in real-world-applications, I don't see a need for a mastercoinD, rather, a reference library in the language of choice (I guess that would be Python for most developers right now).


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 23, 2013, 06:31:50 PM
If all you want is some kind of reference implementation to follow or use in real-world-applications, I don't see a need for a mastercoinD, rather, a reference library in the language of choice (I guess that would be Python for most developers right now).

This is what I initially thought as well (i.e. a reference library, either in a single language like python, or in C, which could be taken out to a variety of languages). However, as killerstorm has noted in his reply at https://bitcointalk.org/index.php?topic=265488.msg3665847#msg3665847 , most of the challenge lies in reconstructing the stream of Mastercoin messages from block to block, not in the parsing of the blocks themselves. In this case, I feel a separate resident daemon lends itself better to this (i.e. parsing transactions and maintaining state), and is superior to a reference library for the following reasons:

  • one could use any language to interact with it, not just python (or whatever else)
  • if other languages were desired, we wouldn't have to write separate implementations in each language (or be forced to use a lower level language and bind it up to the language of choice, if we wanted to share code, for instance)
  • It has a very straightforward analogy to bitcoind, and can be quickly picked up by devs that have dealt with bitcoind
  • like bitcoind, it can store its state independently of the application that's using it
  • is potentially less error prone and "foolproof" in real-world usage, as it's more self-contained
  • It can be distributed pre-compiled, with no binding/linking required

qwk: I just reset the poll to account for improving the options, given that I didn't want to try to unfairly re-interpret your vote. Please vote again if you wish.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: qwk on November 23, 2013, 06:37:29 PM
  • like bitcoind, it can store its state independently of the application that's using it
  • is potentially less error prone and "foolproof" in real-world usage, as it's more self-contained
Okay, I'll bite. Makes perfect sense ;)


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: prophetx on November 24, 2013, 12:21:16 PM
Could one of the existing projects be repackaged to be this?

https://bitcointalk.org/index.php?topic=292628.msg3695215#msg3695215


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: Ola on November 24, 2013, 03:06:53 PM
+1  Makes all the sense in the world..I am waiting on this


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 24, 2013, 03:31:25 PM
I started development of this yesterday and have made a ton of progress after almost an all-nighter :p. Python using sqlite and cherrypy. Using grazcoin's mastercoin-tools for the mastercoin ops right now (which depends on sx). Eventually I'd like to move to bitcoind for better compatibility under Windows (or get sx compiling under windows -- looking into that), but for now this will work and will allow us to get off the ground much more quickly (grazcoin has done a good job).

The target at first is Ubuntu 13.10 (sx has major build issues with 12.04 LTS), and I have a full build script that will do everything from a stock system install.

At a minimum, even for Windows folks, one could set up a VM with Ubuntu 13.10, install mastercoind to that, and access it from your windows boxes across your LAN. I'll have a step by step how to set it up, once you get the base Ubuntu install on a VM.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: Ola on November 24, 2013, 03:53:52 PM
I started development of this yesterday and have made a ton of progress after almost an all-nighter :p. Python using sqlite and cherrypy. Using grazcoin's mastercoin-tools for the mastercoin ops right now (which depends on sx). Eventually I'd like to move to bitcoind for better compatibility under Windows (or get sx compiling under windows -- looking into that), but for now this will work and will allow us to get off the ground much more quickly (grazcoin has done a good job).

The target at first is Ubuntu 13.10 (sx has major build issues with 12.04 LTS), and I have a full build script that will do everything from a stock system install.

At a minimum, even for Windows folks, one could set up a VM with Ubuntu 13.10, install mastercoind to that, and access it from your windows boxes across your LAN. I'll have a step by step how to set it up, once you get the base Ubuntu install on a VM.

I too stayed up al night yesterday working on a project, nothing close to mastercoin/bitcoin related though. I am glad you are already leading this charge, i commend you for this. Any upcoming technical difficulties that you foresee on your path? and why not build for Windows and OSX first? the masses...and any particular reason that you choose relational databases? there are a lot of drivers for NoSQL based databases u know ..just curious.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zbx on November 24, 2013, 04:02:29 PM
For what it's worth, I fully support this endeavour.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: Tachikoma on November 24, 2013, 04:08:56 PM
Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 24, 2013, 04:28:53 PM
Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: Ola on November 24, 2013, 04:34:30 PM
Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

+1000


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: ripper234 on November 24, 2013, 04:40:06 PM
+1 for repackaging one or more of the existing implementations and providing various APIs to it.
APIs should be HTTP, RPC, and more ... the more endpoints we have, the more useful an implementation gets.

I'd like to consolidate "development firepower" on one or a few implementations, not to create yet another Mastercoin client.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: ripper234 on November 24, 2013, 04:41:05 PM
Ah sorry I missed that you're wrapping grazcoin's implementation.
That's awesome then :)


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: Tachikoma on November 24, 2013, 04:49:58 PM
Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

Ok thanks for clarifying you are not actually creating a client but a RPC wrapper on top of a client.

I still think J.R. should be the one who writes the reference client since he can make decisions on how the spec should be interpreted. No offence intended, but who gave you the right to decide your implementation should be the reference one. It's not so much how the clients handles, you can strap an RPC client on top of any implementation, but the decisions you make in the underlying layers.

I think an RPC accessible client on top of any library is a excellent idea, I just think that the word reference implementation should be removed. 


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 24, 2013, 05:41:50 PM
I started development of this yesterday and have made a ton of progress after almost an all-nighter :p. Python using sqlite and cherrypy. Using grazcoin's mastercoin-tools for the mastercoin ops right now (which depends on sx). Eventually I'd like to move to bitcoind for better compatibility under Windows (or get sx compiling under windows -- looking into that), but for now this will work and will allow us to get off the ground much more quickly (grazcoin has done a good job).

The target at first is Ubuntu 13.10 (sx has major build issues with 12.04 LTS), and I have a full build script that will do everything from a stock system install.

At a minimum, even for Windows folks, one could set up a VM with Ubuntu 13.10, install mastercoind to that, and access it from your windows boxes across your LAN. I'll have a step by step how to set it up, once you get the base Ubuntu install on a VM.

I too stayed up al night yesterday working on a project, nothing close to mastercoin/bitcoin related though. I am glad you are already leading this charge, i commend you for this. Any upcoming technical difficulties that you foresee on your path? and why not build for Windows and OSX first? the masses...and any particular reason that you choose relational databases? there are a lot of drivers for NoSQL based databases u know ..just curious.

Actually, I just did see that bitcoind went to leveldb as of 0.8, so I moved us to using leveldb as well for this (I do like NoSQL a lot better than SQL...so much easier to deal with).

I agree with having a Windows and MacOSX build. However, building for Windows is hard right off to bat because of 'sx', which grazcoin's mastercoin library uses. There will need to be a decent amount of work put into creating a build script for windows that will get everything built and setup. Once that is available, then a windows version should be no problem. At first, Ubuntu is the first target because I know it well, and it's much easier to build things like this under it (compiling from source on windows --- especially in an automated fashion -- can be a big pain in the ass).


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 24, 2013, 05:50:24 PM
Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

Ok thanks for clarifying you are not actually creating a client but a RPC wrapper on top of a client.

I still think J.R. should be the one who writes the reference client since he can make decisions on how the spec should be interpreted. No offence intended, but who gave you the right to decide your implementation should be the reference one. It's not so much how the clients handles, you can strap an RPC client on top of any implementation, but the decisions you make in the underlying layers.

I think an RPC accessible client on top of any library is a excellent idea, I just think that the word reference implementation should be removed.  


I agree with that. My initial post was a bit confusing up top, sorry about that, but I wasn't thinking that I was making the "reference library" here.....only that mastercoin needed one.

In my mind, mastercoind is simply a gateway/frontend to whatever the reference library is. At some point, once/if it makes sense, I would like mastercoind to possibly become the "reference client"/"reference front-end" that uses a separate backend reference library, but I have no say in that decision itself. As I said, it can be done if it makes sense, and people start building apps around this thing. Plus, I have no desire or intention of writing any mastercoin parsing library, or having to deal with the low-level details of one (this was the purpose I wanted a mastercoind so badly :p).

As I said earlier, right now I'm using grazcoin's MSC library, which is more utilities for his mastercoin site than a formal library ready for 3rd party use. This requires me to hack it a bit to get it to "fit in" to mastercoind. However, this is 110% better than me writing something from scratch. I see the backend library as taking care of those difficult details, and mastercoind wraps it up real nice and makes it very easy to use (as well as allowing it to be used by a client written in any language, or on a different server, etc ... list the list of advantages I have in my 2nd or 3rd post in this thread).

Maybe a future goal for mastercoind is to allow it to plug into 2 or more backend libraries...i.e. the "reference" one (whatever that is) and the main alternative. That way, you can get the similar front-end API functionality, but swap out the backend library if you need/want (e.g. for easy compatibility testing, or maybe one doesn't compile under windows and another one does, etc). The swap should be as easy as modifying a config file in the program, and making sure the library is installed on the system.

The end goal with this thing is to have something that's as easy to use, or easier, than bitcoind is today.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: djohnston on November 24, 2013, 05:55:06 PM
rbdrbd,

Thanks for taking the initiative on this effort. I agree that a reference client for development purposes can be very helpful to new developers getting involved in Mastercoin.

I hear where Tachikoma is coming from with his concerns though and lets make sure that what gets developer here is sensitive to consensus from developers and the community. I wonder if the reference client would be another good place for the Proof of Stake voting to be involved or if the community thinks we should keep Proof of Stake at the Protocol Spec level?

I'm sure J.R. will chime in on the effort. But not likely today, as he does the family thing on the weekends and isn't on the computer.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zathras on November 24, 2013, 11:04:49 PM
Hey rbdrbd and welcome to Mastercoin development :)

I thought I'd chime in on a couple of points - I have to agree with Tachikoma on this one, whether or not you state it's the reference implementation, the choice of name (mastercoind) will lead some members of the community to perhaps think it is.  I would love to see any reference implementation written by JR too.

Simply put, the project will likely have a reference mastercoin daemon at some point and I think it would be courteous to leave the 'mastercoind' moniker for that purpose - you could always rename your output binary later if Grazcoin/your work ended up becoming the reference implementation.

Good luck with your wrapper! :)

P.S. (sorry for plugging my own stuff in your thread :P) but masterchest_engine & masterchest_library have been around for a while to offer devs a low barrier to entry - the library already does decoding/encoding/parsing/bitcoind integration etc on behalf of the dev and the engine processes all transactions to arrive at a known/current state so for example web devs can query balances/transactions/exchange state etc without knowing anything other than a bit of SQL.  They're of course works in progress but since your post kind of implies before your wrapper devs had no choice but to start from scratch, I wanted to clear up that misconception.

Thanks!


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 02:53:18 AM
Hey rbdrbd and welcome to Mastercoin development :)

I thought I'd chime in on a couple of points - I have to agree with Tachikoma on this one, whether or not you state it's the reference implementation, the choice of name (mastercoind) will lead some members of the community to perhaps think it is.  I would love to see any reference implementation written by JR too.

Simply put, the project will likely have a reference mastercoin daemon at some point and I think it would be courteous to leave the 'mastercoind' moniker for that purpose - you could always rename your output binary later if Grazcoin/your work ended up becoming the reference implementation.

Good luck with your wrapper! :)

P.S. (sorry for plugging my own stuff in your thread :P) but masterchest_engine & masterchest_library have been around for a while to offer devs a low barrier to entry - the library already does decoding/encoding/parsing/bitcoind integration etc on behalf of the dev and the engine processes all transactions to arrive at a known/current state so for example web devs can query balances/transactions/exchange state etc without knowing anything other than a bit of SQL.  They're of course works in progress but since your post kind of implies before your wrapper devs had no choice but to start from scratch, I wanted to clear up that misconception.

Thanks!

Good point on the "mastercoind". I'm good with whatever JR says here. My goal with this thing was to get it to a point and turn it over anyhow.

The optimal library for this purpose would be python, would interface with bitcoind via JSON RPC (so it had good cross-platform compatibility), and would take care of encoding/decoding of transactions, and maybe even processing to a embedded leveldb database (if it didn't do the latter, I could add that easily enough).

We don't have this optimal case today, so I have a few options:

* Zathras' libraries -- Windows-only it appears, but sound like it's the right focus. I just skimmed your code. If I found a way to interface the DLLs well to Python (I think it's possible), I could use them under Windows. Looks like I could use masterchest-library alone if I wanted to use something different than SQL for the serialization (e.g. write something higher level that used that to parse the blocks and serialize to leveldb, for instance), or combine in using masterchest-engine and a SQL backend. I really do like how you're interfacing around bitcoind here.

* grazcoin's stuff looks good for Linux, at least. However, it's still a bit rough around the edges for working with as a 3rd party library... mostly around how it's coupled to the web site interface he has, and how he serializes things. I have a few messages to him, still undecided here (the good news out of this is that I made some setup and init.d scripts that will be included in 'sx' though).

* A clean port of masterchest-library to python is another option here, too, since that interfaces with bitcoind, it could be used on both platforms.

* I looked at Tachikoma's stuff too.... ruby though (same situation).

Optimally, I'd want to use the same library and serialization strategy under both Windows and linux, at least for starters.

It may seem like this is a piece of cake, but my goal is something that is very easy to set up and works across multiple platforms, like bitcoind. With that, things become a lot more complicated - code/script wise. I already have a good amount of work into this (and will share my code once I get a basic version of it up), but if this is something one of you thinks you could do up to this level, and make available for both windows and Linux, just let me know and I'll stop my work on it. I don't want to waste your time or my time, I just want to see this done, which is why I started working on it.

Basically, guys, I'm asking for your advice. I'm willing to write up against something I can use for both Windows and Linux (or possibly take two different approaches). I'm also willing to package this up and make it easy to deploy on both platforms. Just let me know how I can best work with you all to get this done, and what you think the best library choice is for me here, given my parameters.

If this starts becoming political, or if I've ruffled feathers with this, then I'll just dump it on a github repo for folks to potentially pick over and move on. I'm not looking to establish a name or identity in any of this, I just want to see it done in one form or another, to help Mastercoin out, as well as my investment in it. That's all.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 03:06:05 AM
And actually, I'm going to stop my work on this until I have a clearer direction of which library/libraries I should be using, and if Mastercoin even wants this. I was a bit premature, sorry. Just let me know.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zathras on November 25, 2013, 03:49:09 AM
Hey rbdrbd and welcome to Mastercoin development :)

I thought I'd chime in on a couple of points - I have to agree with Tachikoma on this one, whether or not you state it's the reference implementation, the choice of name (mastercoind) will lead some members of the community to perhaps think it is.  I would love to see any reference implementation written by JR too.

Simply put, the project will likely have a reference mastercoin daemon at some point and I think it would be courteous to leave the 'mastercoind' moniker for that purpose - you could always rename your output binary later if Grazcoin/your work ended up becoming the reference implementation.

Good luck with your wrapper! :)

P.S. (sorry for plugging my own stuff in your thread :P) but masterchest_engine & masterchest_library have been around for a while to offer devs a low barrier to entry - the library already does decoding/encoding/parsing/bitcoind integration etc on behalf of the dev and the engine processes all transactions to arrive at a known/current state so for example web devs can query balances/transactions/exchange state etc without knowing anything other than a bit of SQL.  They're of course works in progress but since your post kind of implies before your wrapper devs had no choice but to start from scratch, I wanted to clear up that misconception.

Thanks!

Good point on the "mastercoind". I'm good with whatever JR says here. My goal with this thing was to get it to a point and turn it over anyhow.

The optimal library for this purpose would be python, would interface with bitcoind via JSON RPC (so it had good cross-platform compatibility), and would take care of encoding/decoding of transactions, and maybe even processing to a embedded leveldb database (if it didn't do the latter, I could add that easily enough).

We don't have this optimal case today, so I have a few options:

* Zathras' libraries -- Windows-only it appears, but sound like it's the right focus. I just skimmed your code. If I found a way to interface the DLLs well to Python (I think it's possible), I could use them under Windows. Looks like I could use masterchest-library alone if I wanted to use something different than SQL for the serialization (e.g. write something higher level that used that to parse the blocks and serialize to leveldb, for instance), or combine in using masterchest-engine and a SQL backend. I really do like how you're interfacing around bitcoind here.

* grazcoin's stuff looks good for Linux, at least. However, it's still a bit rough around the edges for working with as a 3rd party library... mostly around how it's coupled to the web site interface he has, and how he serializes things. I have a few messages to him, still undecided here (the good news out of this is that I made some setup and init.d scripts that will be included in 'sx' though).

* A clean port of masterchest-library to python is another option here, too, since that interfaces with bitcoind, it could be used on both platforms.

* I looked at Tachikoma's stuff too.... ruby though (same situation).

Optimally, I'd want to use the same library and serialization strategy under both Windows and linux, at least for starters.

It may seem like this is a piece of cake, but my goal is something that is very easy to set up and works across multiple platforms, like bitcoind. With that, things become a lot more complicated - code/script wise. I already have a good amount of work into this (and will share my code once I get a basic version of it up), but if this is something one of you thinks you could do up to this level, and make available for both windows and Linux, just let me know and I'll stop my work on it. I don't want to waste your time or my time, I just want to see this done, which is why I started working on it.

Basically, guys, I'm asking for your advice. I'm willing to write up against something I can use for both Windows and Linux (or possibly take two different approaches). I'm also willing to package this up and make it easy to deploy on both platforms. Just let me know how I can best work with you all to get this done, and what you think the best library choice is for me here, given my parameters.

If this starts becoming political, or if I've ruffled feathers with this, then I'll just dump it on a github repo for folks to potentially pick over and move on. I'm not looking to establish a name or identity in any of this, I just want to see it done in one form or another, to help Mastercoin out, as well as my investment in it. That's all.


Please don't be discouraged by any of the posts (mine included) - I'm all for having more people & development on board :)

Re the name, that's simply a label but seems to have been adopted by the community at large as the naming for a given projects' daemon - hence my desire to keep it free for when a reference is officially nominated - for my own personal opinion I'd actually like to see 'mastercoind/qt' as a reference client in it's sincerest form - ie without any external dependencies (websites or bitcoind etc).  Mastercoind/qt should be a standalone reference client that connects to the bitcoin network and performs all the necessary bitcoin functions as well as Mastercoin functions without needing anything else.  I see a fork of bitcoind/qt as the simplest (not necessarily best, but simplest) path to this, but that's just my two cents.

Re the libraries, none of them is 'the' correct library at present - for one I can tell you my stuff alone is not ready for stable - the functions do what they're supposed to but we're always coming up with new fringe cases and bugs that can knock things off track.  We currently develop to a consensus where by we discuss & agree areas of the spec that are open to interpretation (or just outright suggested changes/additions) and the multiple different implementations are critical to using RAD (rapid application development) techniques (basically everything can change fluidly with no change control) in these early stages.  

For example only yesterday when I was about to release the new version of my library (and thus broadcast a few example transactions) did I find out my implementation was dropping a byte in the minfee as we crossed packets and not prefixing a leading 0 for the timelimit int if the hex was single digit.  Using my own implementation I would never have known (as I was encoding/decoding the same way), but being able to test against Tachikoma's meant I was able to pick up the error and fix the bug.  This happens A LOT :)  For example Mastercoin-explorer & Masterchest may differ on a transaction so Tachikoma & I will investigate - sometimes he's got a bug, sometimes I have - but it's those differing implementations that actually help us to identify those bugs & correct them while everything is so green.  Tachikoma even has a verification API to automate these checks for differences coming I believe.  EDIT: Sorry, not to downplay Grazcoins & Bitoys work either, I check my errors against their code too :)

TL:DR; contribute what you think will be of value.  If Grazcoin's libraries give you a good foundation for something you want to build & you think it will benefit the community, then go right ahead :)  Just try to keep in mind we're young and there are still multiple ways of doing things - we're not at the stage of saying any one implementation is the 'right' one yet and need to be careful about community confusion.

P.S.  Re Linux support I have actually tested (and rewritten in places) my wallet to work under wine on Linux.  I'm only testing with a couple of distros while I remove the kinks but to date I've worked around most bugs and have it working successfully (just with the odd out of place/badly colored label etc).  I haven't done this with the engine yet but since I'm not using as many of the cryptographic providers or doing any GUI work it should be reasonably easy to get going if that became a requirement.

Thanks! :)


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 04:44:18 AM
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zathras on November 25, 2013, 05:26:28 AM
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs :)  With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.



Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: ofer on November 25, 2013, 06:58:21 AM
Hi,
I think that writing mastercoind  is an excellent idea, and I have an idea for improvement:

I think that the best way to implement this is by making mastercoind use bitcoind as a backend.
Just like HTTP implementations are using Berkeley sockets and not rewriting them.

The benefits we get by choosing this architecture are simplicity and more important security.
We can now assume that bitcoind is a very secured piece of software.

Few days ago I saw a lecture by Gavin Anderssen where he said that the bitcoin-qt had a lot of security holes in it's early days.

We can't afford security holes in mastercoind, it will not be under the radar like bitcoin-qt in it's early days.





Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 12:57:15 PM
For transparency, here's my PM exchange with grazcoin regarding potentially using his library (before I had checked out zathras stuff). grazcoin gave me permission to post:

Hey grazcoin,

I've been working on using your mastercoin-tools with mastercoind that I'm developing and have a few questions:

* Have you considered using an embedded database like leveldb instead of creating a bunch of files on the filesystem? I can think of several advantages of using an embedded db to the raw file level storage, especially as the # and volume of transactions grows.


I considered, but the reasons that kept me in this file based methodology are:
* security: serving static files is much more secure.
* simple + cross platform: each user can take a snapshot and continue parsing while using only python and a connection to an obelisk server. It keeps the system really distributed.
* easy to verify: we humans read text better than leveldb format :)

* Do you have any plans to split the parsing and storage part of mastercoin-tools from the website logic which uses it anytime soon? This would definitely help on the mastercoind front.... the parsing process could store the data into the embedded database (leveldb, sqlite, etc) and your website could just read off of it. likewise, mastercoind would just read off of that same database


I tried to keep it separated. Tools should be only tools. The front end shows that it could look also good.

* I finally got sx semi-working ... in order to get msc-parse.py to really do anything though, do I have to wait for the entire blockchain to download?


unfortunately you have to wait.

Oh yeah... I felt I should mention that after looking at your code more and thinking things through, I am now attempting not to modify your code or monkey patch it at all, trying to just run it as you do, and use the outputs created as-is. This will reduce duplication and headaches in the future, assuming the integration points are set. That's why I had the interest in using leveldb (which bitcoind uses, as well as obelisk)....I could just run your cron-script on a timer and dip into the database to get what I needed.

I think it will be easier for you to take the json results, since my code keep getting developed. Otherwise you will have to constantly merge. I am open for changes/requests in the json format though.
You could also read directly the ready json which is web-exported by any running instance of mastercoin-tools (see http://masterchain.info/API.html )
This introduces a trust issue, but it would be wise for you to include this option for people that do not want to run the python code locally. The trust issue could be partially solved by the code signing the parsing results + ssl (will come soon).

And to clarify a point in my earlier message, by 'splitting up the parsing and website code', I meant having them as separate github repos... there is somewhat of a segmentation now, to a point, but a separate repo for each would be great... just an idea to help you make the library more useful to 3rd party utils (and, have it used by anyone that is using mastercoind, which I hope will be a lot of people).

For now it is simpler for me to maintain a single repo.
Maybe it will change later.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 01:03:30 PM
Hi,
I think that writing mastercoind  is an excellent idea, and I have an idea for improvement:

I think that the best way to implement this is by making mastercoind use bitcoind as a backend.
Just like HTTP implementations are using Berkeley sockets and not rewriting them.

The benefits we get by choosing this architecture are simplicity and more important security.
We can now assume that bitcoind is a very secured piece of software.

Few days ago I saw a lecture by Gavin Anderssen where he said that the bitcoin-qt had a lot of security holes in it's early days.

We can't afford security holes in mastercoind, it will not be under the radar like bitcoin-qt in it's early days.





Good point, I'm aiming for the level of security to at least match bitcoind. From a remote exploit prevention, mastercoind is written in python using cherrypy. This fact makes certain types of exploits less likely (e.g. buffer overflows, and other things that could plague writers in lower level languages like C/C++ if they aren't careful with their string length checking). Other vulnerabilities, such as SQL injection, are still possible, and from there it's good coding practices, and doing extensive cleanup/sanity checking on user-provided input (the lack of which is always what leads to these kinds of attacks, essentially).

Then you have things like stored data security. For now at least, RPC username and password are stored in a plaintext settings file, similar to bitcoind. We can look at changing this in the future, however. If something very sensitive is required (such as private wallet key for signing requests), then I plan to collect that as part of the request and not store it, if at all possible.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: Ola on November 25, 2013, 01:04:56 PM
Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

Ok thanks for clarifying you are not actually creating a client but a RPC wrapper on top of a client.

I still think J.R. should be the one who writes the reference client since he can make decisions on how the spec should be interpreted. No offence intended, but who gave you the right to decide your implementation should be the reference one. It's not so much how the clients handles, you can strap an RPC client on top of any implementation, but the decisions you make in the underlying layers.

I think an RPC accessible client on top of any library is a excellent idea, I just think that the word reference implementation should be removed.  


I agree with that. My initial post was a bit confusing up top, sorry about that, but I wasn't thinking that I was making the "reference library" here.....only that mastercoin needed one.

In my mind, mastercoind is simply a gateway/frontend to whatever the reference library is. At some point, once/if it makes sense, I would like mastercoind to possibly become the "reference client"/"reference front-end" that uses a separate backend reference library, but I have no say in that decision itself. As I said, it can be done if it makes sense, and people start building apps around this thing. Plus, I have no desire or intention of writing any mastercoin parsing library, or having to deal with the low-level details of one (this was the purpose I wanted a mastercoind so badly :p).

As I said earlier, right now I'm using grazcoin's MSC library, which is more utilities for his mastercoin site than a formal library ready for 3rd party use. This requires me to hack it a bit to get it to "fit in" to mastercoind. However, this is 110% better than me writing something from scratch. I see the backend library as taking care of those difficult details, and mastercoind wraps it up real nice and makes it very easy to use (as well as allowing it to be used by a client written in any language, or on a different server, etc ... list the list of advantages I have in my 2nd or 3rd post in this thread).

Maybe a future goal for mastercoind is to allow it to plug into 2 or more backend libraries...i.e. the "reference" one (whatever that is) and the main alternative. That way, you can get the similar front-end API functionality, but swap out the backend library if you need/want (e.g. for easy compatibility testing, or maybe one doesn't compile under windows and another one does, etc). The swap should be as easy as modifying a config file in the program, and making sure the library is installed on the system.

The end goal with this thing is to have something that's as easy to use, or easier, than bitcoind is today.

+1000 If you could design the backend to plug in into more than one of the main libraries that will be super awesome...That way all code from the main and new mastercoin devs will keep being used. A modifiable config file is also definitely the way to go in determining what library to choose...

I urge you to keep on the development path, there is noting stopping you from dong this. More-so it is all open source and WILL DEFINITELY open mastercoin up to main stream developers.  I will be available to aid in testing out the features if you need too because I am seriously waiting on this, I am sure a lot of regular web devs are waiting on this also. I thought this feature would be a year or more out, but you changed all that by taking on this initiative and this could be a reality within a couple of months.


I hope JR/Foundation blesses this initiative so that it is looked upon as a credible point to get involved with mastercoin apps development. There  should be nothing political about this..If the name "Mastercoind" is a problem, which is simple a convention, maybe that can be fixed later by JR or any of the executives of the foundation..Please push on rbdrbd



Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 01:11:35 PM
A few more things:

* This thing is going to be slick, I have plans for packaging the windows version as an exe and using a windows installer to install (using pyinstaller and NSIS or something similar), similar to bitcoind on windows.

* I'm open to naming this "masterdaemon" or something that's not "mastercoind" for the reasons zathras and Tachikoma brought up (mastercoind maybe being a "reserved" name). If JR/Ron would like this, just let me know and I'll change the name.

* Regarding the library, after spending an extensive amount of time looking and working with sx (and even improving its build process, see https://bitcointalk.org/index.php?topic=259999.msg3700259#msg3700259), it's clear to me that Windows support will be a big effort...since it totally lacks support for Windows now, and its not only about compiling sx on Windows, but about getting sx, libbitcoin, obliesk, and leveldb working on windows, along with all other dependencies!

A big goal for mastercoind is to be as multi-platform as possible. If I had to do Linux only, I'd go with grazcoin's stuff (which is written to use sx/libbitcoin for all transaction processing). However, given our multiplatform goal with this, I'm thinking at this point, zathras library sounds like a more straightforward implementation to be the primary/initial library that mastercoind uses. The gotchas are getting it working under mono, and getting it working with sqlite, both of which I think can get done without too much hassle. I'll get into working with it some and have a better idea....(hey, at least we have choices here! :))


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: ofer on November 25, 2013, 01:30:07 PM
Hi,
I think that writing mastercoind  is an excellent idea, and I have an idea for improvement:

I think that the best way to implement this is by making mastercoind use bitcoind as a backend.
Just like HTTP implementations are using Berkeley sockets and not rewriting them.

The benefits we get by choosing this architecture are simplicity and more important security.
We can now assume that bitcoind is a very secured piece of software.

Few days ago I saw a lecture by Gavin Anderssen where he said that the bitcoin-qt had a lot of security holes in it's early days.

We can't afford security holes in mastercoind, it will not be under the radar like bitcoin-qt in it's early days.





Good point, I'm aiming for the level of security to at least match bitcoind. From a remote exploit prevention, mastercoind is written in python using cherrypy. This fact makes certain types of exploits less likely (e.g. buffer overflows, and other things that could plague writers in lower level languages like C/C++ if they aren't careful with their string length checking). Other vulnerabilities, such as SQL injection, are still possible, and from there it's good coding practices, and doing extensive cleanup/sanity checking on user-provided input (the lack of which is always what leads to these kinds of attacks, essentially).

Then you have things like stored data security. For now at least, RPC username and password are stored in a plaintext settings file, similar to bitcoind. We can look at changing this in the future, however. If something very sensitive is required (such as private wallet key for signing requests), then I plan to collect that as part of the request and not store it, if at all possible.

What I was trying to say is that mastercoind should call the API of bitcoind to make the actual low-level bitcoin/mastercoin transactions, this way we reduce the security risk even more.
This is because we can assume that bitcoind is very secure.

Also, good to hear that security is one of your main concerns. Programmers often ignore security issues. In such application that deals with money security is a top priority concern.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 03:09:34 PM
Hi,
I think that writing mastercoind  is an excellent idea, and I have an idea for improvement:

I think that the best way to implement this is by making mastercoind use bitcoind as a backend.
Just like HTTP implementations are using Berkeley sockets and not rewriting them.

The benefits we get by choosing this architecture are simplicity and more important security.
We can now assume that bitcoind is a very secured piece of software.

Few days ago I saw a lecture by Gavin Anderssen where he said that the bitcoin-qt had a lot of security holes in it's early days.

We can't afford security holes in mastercoind, it will not be under the radar like bitcoin-qt in it's early days.





Good point, I'm aiming for the level of security to at least match bitcoind. From a remote exploit prevention, mastercoind is written in python using cherrypy. This fact makes certain types of exploits less likely (e.g. buffer overflows, and other things that could plague writers in lower level languages like C/C++ if they aren't careful with their string length checking). Other vulnerabilities, such as SQL injection, are still possible, and from there it's good coding practices, and doing extensive cleanup/sanity checking on user-provided input (the lack of which is always what leads to these kinds of attacks, essentially).

Then you have things like stored data security. For now at least, RPC username and password are stored in a plaintext settings file, similar to bitcoind. We can look at changing this in the future, however. If something very sensitive is required (such as private wallet key for signing requests), then I plan to collect that as part of the request and not store it, if at all possible.

What I was trying to say is that mastercoind should call the API of bitcoind to make the actual low-level bitcoin/mastercoin transactions, this way we reduce the security risk even more.
This is because we can assume that bitcoind is very secure.

Also, good to hear that security is one of your main concerns. Programmers often ignore security issues. In such application that deals with money security is a top priority concern.

Regarding your first point, I think that eventually, I'd like to see a fully integrated mastercoind. For now, however, I'm planning on probably going with Zathras masterchest-engine to reduce code duplication and greatly speed development time. My hope is that a "reference library" emerges from the Mastercoin community that is written in python (or has python bindings) and can be used by mastercoind directly. Even in that event, I'd still leave in the masterchest-engine backend, and allow switching between them for the reasons I've raised earlier.

Grabbing of masterchest-engine will be done directly from github in the automated setup script, to reduce the chance of error/security. It will also be possible for people to create their own mastercoind windows binaries, if they so desired, or didn't trust the pre-made binaries offered.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 03:25:07 PM
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs :)  With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.



Yeah, I'm thinking your stuff is the best fit considering the multiplatform component. Tachikoma's stuff may work as a backup, but it looks like his whole data storage layer is inside the mastercoin-explorer project (as ruby on rails). Also, I did research calling over to a ruby library from python... didn't find anything conclusive with the short amount of time I spent on it.

masterchest-engine I should be able to us as-is (once it has the sqlite backend support and any necessary mono fixes), without having to touch/fork any of your code.

I'm installing SQL Server Express to be able to test masterchest-engine. The plan is to test it under windows first, then under mono (pushing to a SQL Server instance in Windows still). Once you're able to add the sqlite integration, I can test it all under mono in Linux.

Do you have any .sql files to create the tables and indexes for your schema or should I write my own?


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: dacoinminster on November 25, 2013, 08:41:33 PM
I think it would be cool to have a mastercoin daemon (probably just name it "MasterDaemon" or something -mastercoind sounds too official).

We seem to be doing pretty well with no reference implementation so far. The various devs implement a piece of the spec, find differences and corner cases, discuss them, resolve them, and then repeat.  If I were to contribute code, it would probably just be pull requests against one or more of the existing code-bases.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zathras on November 25, 2013, 09:17:17 PM
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs :)  With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.



Yeah, I'm thinking your stuff is the best fit considering the multiplatform component. Tachikoma's stuff may work as a backup, but it looks like his whole data storage layer is inside the mastercoin-explorer project (as ruby on rails). Also, I did research calling over to a ruby library from python... didn't find anything conclusive with the short amount of time I spent on it.

masterchest-engine I should be able to us as-is (once it has the sqlite backend support and any necessary mono fixes), without having to touch/fork any of your code.

I'm installing SQL Server Express to be able to test masterchest-engine. The plan is to test it under windows first, then under mono (pushing to a SQL Server instance in Windows still). Once you're able to add the sqlite integration, I can test it all under mono in Linux.

Do you have any .sql files to create the tables and indexes for your schema or should I write my own?

It's on the to do list (auto creating tables) just hasn't been prioritized.

I'm just using express for masterchest.info - if you like I'm happy to shoot you a backup of its DB (the engine runs that site) which you can then just restore and test Mono off the bat without needing to manually do the tables?

From a cursory glance SQLite looks an easy enough addition.  Let me know how you get on with Mono (& what distro/kernel/mono versions you're testing with so I can duplicate).

Thanks :)


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 11:07:44 PM
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs :)  With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.



Yeah, I'm thinking your stuff is the best fit considering the multiplatform component. Tachikoma's stuff may work as a backup, but it looks like his whole data storage layer is inside the mastercoin-explorer project (as ruby on rails). Also, I did research calling over to a ruby library from python... didn't find anything conclusive with the short amount of time I spent on it.

masterchest-engine I should be able to us as-is (once it has the sqlite backend support and any necessary mono fixes), without having to touch/fork any of your code.

I'm installing SQL Server Express to be able to test masterchest-engine. The plan is to test it under windows first, then under mono (pushing to a SQL Server instance in Windows still). Once you're able to add the sqlite integration, I can test it all under mono in Linux.

Do you have any .sql files to create the tables and indexes for your schema or should I write my own?

It's on the to do list (auto creating tables) just hasn't been prioritized.

I'm just using express for masterchest.info - if you like I'm happy to shoot you a backup of its DB (the engine runs that site) which you can then just restore and test Mono off the bat without needing to manually do the tables?

From a cursory glance SQLite looks an easy enough addition.  Let me know how you get on with Mono (& what distro/kernel/mono versions you're testing with so I can duplicate).

Thanks :)

A sql schema dump would be great.... I just want to make sure I'm creating the schema to be like you are (a backup is fine too...just throw it up on dropbox or something and PM the link to me).

I'm testing on Ubuntu 13.10 x64, so Mono looks like 2.10, not 3.2.4:

"mono -V" output: "Mono JIT compiler version 2.10.8.1 (Debian 2.10.8.1-5ubuntu2)"
"uname -a" output: "Linux d01b 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux"

Mono-wise, I've installed the "mono-runtime" package. If I need to install any additional packages (e.g. mono-complete, or other specific mono libs) just let me know.

I'm ready to test around sqlite as soon as you can add it, mono too. :) Thanks for your help on this!!


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 25, 2013, 11:38:34 PM
Thanks JR. I'll rename the project to be "masterdaemon" at your advice.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zathras on November 26, 2013, 04:30:12 AM
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs :)  With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.



Yeah, I'm thinking your stuff is the best fit considering the multiplatform component. Tachikoma's stuff may work as a backup, but it looks like his whole data storage layer is inside the mastercoin-explorer project (as ruby on rails). Also, I did research calling over to a ruby library from python... didn't find anything conclusive with the short amount of time I spent on it.

masterchest-engine I should be able to us as-is (once it has the sqlite backend support and any necessary mono fixes), without having to touch/fork any of your code.

I'm installing SQL Server Express to be able to test masterchest-engine. The plan is to test it under windows first, then under mono (pushing to a SQL Server instance in Windows still). Once you're able to add the sqlite integration, I can test it all under mono in Linux.

Do you have any .sql files to create the tables and indexes for your schema or should I write my own?

It's on the to do list (auto creating tables) just hasn't been prioritized.

I'm just using express for masterchest.info - if you like I'm happy to shoot you a backup of its DB (the engine runs that site) which you can then just restore and test Mono off the bat without needing to manually do the tables?

From a cursory glance SQLite looks an easy enough addition.  Let me know how you get on with Mono (& what distro/kernel/mono versions you're testing with so I can duplicate).

Thanks :)

A sql schema dump would be great.... I just want to make sure I'm creating the schema to be like you are (a backup is fine too...just throw it up on dropbox or something and PM the link to me).

I'm testing on Ubuntu 13.10 x64, so Mono looks like 2.10, not 3.2.4:

"mono -V" output: "Mono JIT compiler version 2.10.8.1 (Debian 2.10.8.1-5ubuntu2)"
"uname -a" output: "Linux d01b 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux"

Mono-wise, I've installed the "mono-runtime" package. If I need to install any additional packages (e.g. mono-complete, or other specific mono libs) just let me know.

I'm ready to test around sqlite as soon as you can add it, mono too. :) Thanks for your help on this!!
No probs :) Here you go:

Code:
USE [master]
GO
/****** Object:  Database [chestmaster]    Script Date: 11/26/2013 04:27:36 ******/
CREATE DATABASE [chestmaster] ON  PRIMARY
( NAME = N'chestmaster', FILENAME = N'c:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\chestmaster.mdf' , SIZE = 5120KB , MAXSIZE = UNLIMITED, FILEGROWTH = 1024KB )
 LOG ON
( NAME = N'chestmaster_log', FILENAME = N'c:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\chestmaster.ldf' , SIZE = 10176KB , MAXSIZE = 2048GB , FILEGROWTH = 10%)
GO
ALTER DATABASE [chestmaster] SET COMPATIBILITY_LEVEL = 100
GO
IF (1 = FULLTEXTSERVICEPROPERTY('IsFullTextInstalled'))
begin
EXEC [chestmaster].[dbo].[sp_fulltext_database] @action = 'enable'
end
GO
ALTER DATABASE [chestmaster] SET ANSI_NULL_DEFAULT OFF
GO
ALTER DATABASE [chestmaster] SET ANSI_NULLS OFF
GO
ALTER DATABASE [chestmaster] SET ANSI_PADDING OFF
GO
ALTER DATABASE [chestmaster] SET ANSI_WARNINGS OFF
GO
ALTER DATABASE [chestmaster] SET ARITHABORT OFF
GO
ALTER DATABASE [chestmaster] SET AUTO_CLOSE OFF
GO
ALTER DATABASE [chestmaster] SET AUTO_CREATE_STATISTICS ON
GO
ALTER DATABASE [chestmaster] SET AUTO_SHRINK OFF
GO
ALTER DATABASE [chestmaster] SET AUTO_UPDATE_STATISTICS ON
GO
ALTER DATABASE [chestmaster] SET CURSOR_CLOSE_ON_COMMIT OFF
GO
ALTER DATABASE [chestmaster] SET CURSOR_DEFAULT  GLOBAL
GO
ALTER DATABASE [chestmaster] SET CONCAT_NULL_YIELDS_NULL OFF
GO
ALTER DATABASE [chestmaster] SET NUMERIC_ROUNDABORT OFF
GO
ALTER DATABASE [chestmaster] SET QUOTED_IDENTIFIER OFF
GO
ALTER DATABASE [chestmaster] SET RECURSIVE_TRIGGERS OFF
GO
ALTER DATABASE [chestmaster] SET  DISABLE_BROKER
GO
ALTER DATABASE [chestmaster] SET AUTO_UPDATE_STATISTICS_ASYNC OFF
GO
ALTER DATABASE [chestmaster] SET DATE_CORRELATION_OPTIMIZATION OFF
GO
ALTER DATABASE [chestmaster] SET TRUSTWORTHY OFF
GO
ALTER DATABASE [chestmaster] SET ALLOW_SNAPSHOT_ISOLATION OFF
GO
ALTER DATABASE [chestmaster] SET PARAMETERIZATION SIMPLE
GO
ALTER DATABASE [chestmaster] SET READ_COMMITTED_SNAPSHOT OFF
GO
ALTER DATABASE [chestmaster] SET HONOR_BROKER_PRIORITY OFF
GO
ALTER DATABASE [chestmaster] SET  READ_WRITE
GO
ALTER DATABASE [chestmaster] SET RECOVERY SIMPLE
GO
ALTER DATABASE [chestmaster] SET  MULTI_USER
GO
ALTER DATABASE [chestmaster] SET PAGE_VERIFY CHECKSUM
GO
ALTER DATABASE [chestmaster] SET DB_CHAINING OFF
GO
USE [chestmaster]
GO
/****** Object:  Table [dbo].[transactions_processed]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[transactions_processed](
[TXID] [nvarchar](100) NOT NULL,
[FROMADD] [nvarchar](100) NOT NULL,
[TOADD] [nvarchar](100) NULL,
[VALUE] [bigint] NULL,
[TYPE] [nvarchar](100) NOT NULL,
[BLOCKTIME] [nvarchar](100) NOT NULL,
[BLOCKNUM] [nvarchar](100) NOT NULL,
[VALID] [bit] NOT NULL,
[CURTYPE] [int] NOT NULL,
[ID] [int] IDENTITY(1,1) NOT NULL,
[saleamount] [bigint] NULL,
[offeramount] [bigint] NULL,
[minfee] [bigint] NULL,
[timelimit] [int] NULL,
[purchaseamount] [bigint] NULL,
[matchingtx] [nvarchar](100) NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[transactions]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[transactions](
[TXID] [nvarchar](100) NOT NULL,
[FROMADD] [nvarchar](100) NOT NULL,
[TOADD] [nvarchar](100) NULL,
[VALUE] [bigint] NULL,
[TYPE] [nvarchar](100) NOT NULL,
[BLOCKTIME] [bigint] NOT NULL,
[BLOCKNUM] [int] NOT NULL,
[VALID] [bit] NOT NULL,
[CURTYPE] [int] NOT NULL,
[ID] [int] IDENTITY(1,1) NOT NULL,
[saleamount] [bigint] NULL,
[offeramount] [bigint] NULL,
[minfee] [bigint] NULL,
[timelimit] [int] NULL,
[purchaseamount] [bigint] NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[processedblocks]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[processedblocks](
[BLOCKNUM] [int] NOT NULL,
[BLOCKTIME] [bigint] NOT NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[exotransactions]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[exotransactions](
[txid] [nvarchar](100) NOT NULL,
[blocktime] [bigint] NOT NULL,
[blocknum] [int] NOT NULL,
[vouts] [nvarchar](2048) NOT NULL,
[id] [int] IDENTITY(1,1) NOT NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[exchange]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[exchange](
[txid] [nvarchar](100) NOT NULL,
[fromadd] [nvarchar](100) NOT NULL,
[type] [nvarchar](100) NOT NULL,
[blocktime] [bigint] NOT NULL,
[blocknum] [int] NOT NULL,
[valid] [bit] NOT NULL,
[curtype] [int] NOT NULL,
[id] [int] IDENTITY(1,1) NOT NULL,
[saleamount] [bigint] NULL,
[offeramount] [bigint] NULL,
[minfee] [bigint] NULL,
[timelimit] [int] NULL,
[purchaseamount] [bigint] NULL,
[unitprice] [bigint] NOT NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[balances]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[balances](
[ADDRESS] [nvarchar](100) NOT NULL,
[CBALANCE] [bigint] NOT NULL,
[CBALANCET] [bigint] NOT NULL,
[ubalance] [bigint] NOT NULL,
[ubalancet] [bigint] NOT NULL
) ON [PRIMARY]
GO
/****** Object:  Default [DF__transacti__VALID__0519C6AF]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[transactions] ADD  DEFAULT ('0') FOR [VALID]
GO
/****** Object:  Default [DF__transacti__CURTY__060DEAE8]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[transactions] ADD  DEFAULT ('0') FOR [CURTYPE]
GO
/****** Object:  Default [DF__balances__CBALAN__08EA5793]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[balances] ADD  DEFAULT ((0)) FOR [CBALANCET]
GO
/****** Object:  Default [DF__balances__ubalan__09DE7BCC]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[balances] ADD  DEFAULT ((0)) FOR [ubalance]
GO
/****** Object:  Default [DF__balances__ubalan__0AD2A005]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[balances] ADD  DEFAULT ((0)) FOR [ubalancet]
GO


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 26, 2013, 02:58:18 PM
Great, thanks zathras! I'll start fiddling with masterchest-engine this evening.

One other thing: I looked through your library code and didn't see a way for it to handle simple sends (or DEx user-initiated operations).

In your wallet, I do see some of this functionality though: https://github.com/zathras-crypto/masterchest-wallet/blob/master/Form1.vb#L1142

Optimally, might it be possible for you to abstract this stuff (i.e. functions for simple send and similar DEx user-initiated functions) from the GUI logic and move it to your library or engine?

Then your wallet code could call this directly still. For masterdaemon, this functionality would just need to be wrapped up into a command-line utility, like grazcoin has for simple sends:

https://github.com/grazcoin/mastercoin-tools/blob/master/msc_send.py

If you abstract the functions over from your wallet to the library, I may be able to write this command line wrapper in .NET if you don't have the time (I'm rusty with it).

Let me know what you think about this.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zathras on November 26, 2013, 09:30:13 PM
Great, thanks zathras! I'll start fiddling with masterchest-engine this evening.

One other thing: I looked through your library code and didn't see a way for it to handle simple sends (or DEx user-initiated operations).

In your wallet, I do see some of this functionality though: https://github.com/zathras-crypto/masterchest-wallet/blob/master/Form1.vb#L1142

Optimally, might it be possible for you to abstract this stuff (i.e. functions for simple send and similar DEx user-initiated functions) from the GUI logic and move it to your library or engine?

Then your wallet code could call this directly still. For masterdaemon, this functionality would just need to be wrapped up into a command-line utility, like grazcoin has for simple sends:

https://github.com/grazcoin/mastercoin-tools/blob/master/msc_send.py

If you abstract the functions over from your wallet to the library, I may be able to write this command line wrapper in .NET if you don't have the time (I'm rusty with it).

Let me know what you think about this.

Always happy to help :)

The functions you speak of (creating send & DEx transactions) are already in the library (encodetx, encodeselltx, encodeaccepttx etc).  They're just not used in the engine as its original design intent was just parsing transactions and processing them to arrive at a known state rather than creating new transactions (that's left up to my wallet).

It should be quite easy to spin up a CLI binary to expose these functions though.  I'll throw you something together (it'll be quick & dirty, my primary focus is fixing DEx state processing in the engine & wallet at the mo).

This may also help for some background here (https://bitcointalk.org/index.php?topic=292628.msg3695215#msg3695215)


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zathras on November 27, 2013, 09:38:19 AM
DEVELOPERS ONLY.  THIS IS NOT READY FOR END USERS.  THANKS!

Hey rbdrbd,

Here's the cli wrapper you asked for.  Just provides a binary to expose the library functions for encoding transactions.  Note there is no state or validation here, we're just creating the transaction.  The result can be signed & broadcast if you're happy with the output.

Please use with care, it's completely untested (there could be bad numeric conversions or other bugs for example) so only test against a wallet with fractional amounts :)

Let me know how you get on with it under Mono.  I'll take a look at putting sqlite support into the engine over the weekend, but for the next few days I really need to hunt down these bugs that are messing up my DEx states.

Thanks! :)

Built against .NET4.  Pick it up from here (https://masterchest.info/files/MasterchestCLI_0001.zip).  Note, use the -type=type switch to select between send, sell & accept.

Simple send:
Use the bitcoinrpc switches from the engine and -am (amount) -to (to address) -fr (from address) and -cu (currency type) switches.  For example:
Code:
MasterchestCLI.exe -bitcoinrpcserv=127.0.0.1 -bitcoinrpcport=8332 -bitcoinrpcuser=ooo -bitcoinrpcpass=ooo -type=send -fr=1KUu56RMsafvgwxkiGHmptrLLmBgQ16uQG -to=1MCHESTptvd2LnNp7wmr2sGTpRomteAkq8 -cu=MSC -am=1.234
https://i.imgur.com/CQOentV.png


Sell offer:
Use the bitcoinrpc switches from the engine and -am (sale amount) -fr (from address) -cu (currency type) -of (offer amount) -tl (timelimit) -mf (minfee) switches.  For example:
Code:
-bitcoinrpcserv=127.0.0.1 -bitcoinrpcport=8332 -bitcoinrpcuser=ooo -bitcoinrpcpass=ooo -type=sell -fr=1KUu56RMsafvgwxkiGHmptrLLmBgQ16uQG -cu=MSC -am=1.234 -of=0.5 -tl=6 -mf=0.0001
https://i.imgur.com/sVKGOcf.png


Accept offer:
Use the bitcoinrpc switches from the engine and -am (purchase amount) -fr (from address) -to (to address) -cu (currency type) switches.  For example:
Code:
-bitcoinrpcserv=127.0.0.1 -bitcoinrpcport=8332 -bitcoinrpcuser=ooo -bitcoinrpcpass=ooo -type=accept -fr=1KUu56RMsafvgwxkiGHmptrLLmBgQ16uQG -to=1MCHESTptvd2LnNp7wmr2sGTpRomteAkq8 -cu=MSC -am=1.234

https://i.imgur.com/B9AI4s4.png


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 27, 2013, 02:37:06 PM
Zathras, great! Thank you! I will be integrating this and testing today and tomorrow.

By chance, do you have the source code on github for those CLI utils (perhaps as a part of masterchest-engine repo)? (wanted to check a few things, e.g. like if I could specify a currency ID instead of "MSC", for instance.)

Looking forward to SQLite support being added!

I've converted your schema to SQLite, it's at pastebin here: http://pastebin.com/39juANE1
^ please check and let me know if I missed anything...was pretty straight forward but I'm not super experienced with SQLite itself.

I'm thinking I'm going to build this thing to push through support for both Sqlserver and SQLite...so people can scale up or not depending on their needs.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 27, 2013, 04:41:47 PM
FYI, the MasterchestCLI.exe is showing up in avast! as infected with "Win32: Evo-gen [Susp]". Ran it through Virustotal.com and it looks like a false positive, but it could cause issues for folks with avast! on their computers.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zathras on November 27, 2013, 08:26:11 PM
Zathras, great! Thank you! I will be integrating this and testing today and tomorrow.

By chance, do you have the source code on github for those CLI utils (perhaps as a part of masterchest-engine repo)? (wanted to check a few things, e.g. like if I could specify a currency ID instead of "MSC", for instance.)

Looking forward to SQLite support being added!

I've converted your schema to SQLite, it's at pastebin here: http://pastebin.com/39juANE1
^ please check and let me know if I missed anything...was pretty straight forward but I'm not super experienced with SQLite itself.

I'm thinking I'm going to build this thing to push through support for both Sqlserver and SQLite...so people can scale up or not depending on their needs.

I'll create a MasterchestCLI repo and push the source up to git after work.  FYI you can use either MSC or TMSC (test MSC) for the currency ID for now.  Perhaps I'll drop this straight to an int and whatever currency ID number is specified on the command line is passed straight to the encoding functions (rather than if ucase(cu)=MSC then curtype=1 etc).

I'll take a look at your SQLite schema when I add the support (again sorry but queued behind DEx bug squishing).

FYI, the MasterchestCLI.exe is showing up in avast! as infected with "Win32: Evo-gen [Susp]". Ran it through Virustotal.com and it looks like a false positive, but it could cause issues for folks with avast! on their computers.


You're not kidding - driving me to tears (I run avast as well).  It seems a good portion of the binaries I spit out get flagged with a 'generic' virus match and deleted.  "Build failed" (because Avast deletes the output file) is a daily occurrence for me.

They're safe as houses - Avast have basically said because the EXEs I'm creating have no 'reputation' they are assumed to be bad.  It's an ongoing debate (I think they're well off track with this).  Just google Visual Studio & Avast and you'll see this is a fairly common issue.

In terms of end users, apparently I'm supposed to submit the binaries to Avast first so they an clear them and avoid false positives, though I'm really quite unhappy with that assertion.

To be continued...

Thanks! :)



Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: Ola on November 29, 2013, 02:30:13 PM
rbdrbd this is NXT's api..this should motivate you:    https://bitcointalk.org/index.php?topic=313082.0

They will probably have hundreds of apps at their disposal when the launch their exchange..I hope you are trucking on I am 100% behind you.


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: rbdrbd on November 29, 2013, 05:04:02 PM
zathras,

Got a few Qs/todo items for you:

* with balances, what is the difference between CBALANCE, CBALANCET, ubalance and ubalancet???

* it appears you clear out the transactions_processed table totally every run. If I have an API query going on during this time when it's cleared and being regenerated, I would return bad data. Can you regenerate into a temp table and rename, or come up with another way? I can pause transactions for now when this is being regenerated, but that's kind of ghetto.

* masterchest-engine sql versioning - have a version table or a way for me to deduct the sql schema version....auto schema upgrades in masterchest-engine would be best... worst case, if blow away the DB on schema upgrades, if necessary, and regenerate...

* have a way for me to query the masterchest-engine version

* an abundance of magic numbers -- makes the code confusing to read. e.g. what does "If .Item(6) < 999998 Then" mean??? What is that 999998 ??? lack of comments around these kinds of things. Also, things like "curtype = 1" -- maybe use enums instead? A future task I'm sure...I know you're busy getting Dex working (and my sqlite integration :D)

* what timezone does masterchest-engine store -- we should probably force UTC time. Or, the timezone must be stored with it

* you have no indexes on any rows, columns of your database. This greatly can impact performance. Do you have plans to add indexes?

* from transactions_processed, how do I tell what kind of transaction it is (multi-sig, class A, etc)?? can you add this into the schema?

* also, for transactions_processed, does "VALUE" include the bonus amount for txns in the exodus period, or not?


Title: Re: mastercoind proposal to speed development: Please read, vote, and comment
Post by: zathras on December 02, 2013, 09:38:25 PM
zathras,

Got a few Qs/todo items for you:

* with balances, what is the difference between CBALANCE, CBALANCET, ubalance and ubalancet???
CBALANCE = Confirmed Balance
CBALANCET = Confirmed Balance (Test)
UBALANCE = Unconfirmed Balance
UBALANCET = Unconfirmed Balance (Test)

* it appears you clear out the transactions_processed table totally every run. If I have an API query going on during this time when it's cleared and being regenerated, I would return bad data. Can you regenerate into a temp table and rename, or come up with another way? I can pause transactions for now when this is being regenerated, but that's kind of ghetto.
Yep - doing full state processing on every run is useful in these early stages for hunting bugs.  This was much less of an issue for the first version of the engine as processing only took a few ms.  As we scale out this is no longer viable so temp tables will be used (already are in my wallet).  On the to-do list :)

* masterchest-engine sql versioning - have a version table or a way for me to deduct the sql schema version....auto schema upgrades in masterchest-engine would be best... worst case, if blow away the DB on schema upgrades, if necessary, and regenerate...
Will consider this.  Whilst it was just me using the engine for masterchest.info obviously it didn't matter so much, but if others are going to make use of it I can see how this would be valuable.

* have a way for me to query the masterchest-engine version
No problems here - not difficult to add a -ver switch.

* an abundance of magic numbers -- makes the code confusing to read. e.g. what does "If .Item(6) < 999998 Then" mean??? What is that 999998 ??? lack of comments around these kinds of things. Also, things like "curtype = 1" -- maybe use enums instead? A future task I'm sure...I know you're busy getting Dex working (and my sqlite integration :D)
Haha - the magic 999998 is for my unconfirmed transaction processing.  999999 is a dummy block number for an unconfirmed transaction.  Again as I mentioned below there is some cleanup to be done - this (along with a couple other tricks) was a quick and dirty way of supporting the display of unconfirmed transactions on masterchest.info.

* what timezone does masterchest-engine store -- we should probably force UTC time. Or, the timezone must be stored with it
Doesn't store a timezone at all, it stores the blocktime as recorded in the blockchain.  You should do localization after retrieving transaction times from the db, not while storing them.

* you have no indexes on any rows, columns of your database. This greatly can impact performance. Do you have plans to add indexes?
As needed sure - I haven't run into any performance issues yet (none of my queries take longer than a few ms) which is why I don't have them, but they can be added without to much difficulty if needed.

* from transactions_processed, how do I tell what kind of transaction it is (multi-sig, class A, etc)?? can you add this into the schema?
You can tell what kind of transaction it is from the type column (send, selloffer, acceptoffer etc).  The transaction encoding class isn't stored as I haven't come across a need for it.  Simple sends are the only supported Class A transaction type.  If you throw a simple send to mlib.getmastercointransaction it'll return the correct tx details regardless of whether it used Class A or Class B encoding.

It's not hugely disruptive to add it, but could I ask what you'd be using it for?

* also, for transactions_processed, does "VALUE" include the bonus amount for txns in the exodus period, or not?
Assume you mean for the 'generate' transaction type - and yes, 'generate' transaction types include the early adopter bonus.

Hey rbdrbd,

Sorry for the delayed response, I was away for the weekend & when I got back the forums were down.  Yep the code is quite scrappy but functional and will be cleaned up as time allows.  Please see inline for responses to your Qs.

Thanks! :)