mastercoind proposal to speed development: Please read, vote, and comment

[rbdrbd]

Zathras, great! Thank you! I will be integrating this and testing today and tomorrow.

By chance, do you have the source code on github for those CLI utils (perhaps as a part of masterchest-engine repo)? (wanted to check a few things, e.g. like if I could specify a currency ID instead of "MSC", for instance.)

Looking forward to SQLite support being added!

I've converted your schema to SQLite, it's at pastebin here: http://pastebin.com/39juANE1
^ please check and let me know if I missed anything...was pretty straight forward but I'm not super experienced with SQLite itself.

I'm thinking I'm going to build this thing to push through support for both Sqlserver and SQLite...so people can scale up or not depending on their needs.

[rbdrbd]

FYI, the MasterchestCLI.exe is showing up in avast! as infected with "Win32: Evo-gen [Susp]". Ran it through Virustotal.com and it looks like a false positive, but it could cause issues for folks with avast! on their computers.

[zathras]

Zathras, great! Thank you! I will be integrating this and testing today and tomorrow.

By chance, do you have the source code on github for those CLI utils (perhaps as a part of masterchest-engine repo)? (wanted to check a few things, e.g. like if I could specify a currency ID instead of "MSC", for instance.)

Looking forward to SQLite support being added!

I've converted your schema to SQLite, it's at pastebin here: http://pastebin.com/39juANE1
^ please check and let me know if I missed anything...was pretty straight forward but I'm not super experienced with SQLite itself.

I'm thinking I'm going to build this thing to push through support for both Sqlserver and SQLite...so people can scale up or not depending on their needs.

I'll create a MasterchestCLI repo and push the source up to git after work.  FYI you can use either MSC or TMSC (test MSC) for the currency ID for now.  Perhaps I'll drop this straight to an int and whatever currency ID number is specified on the command line is passed straight to the encoding functions (rather than if ucase(cu)=MSC then curtype=1 etc).

I'll take a look at your SQLite schema when I add the support (again sorry but queued behind DEx bug squishing).

FYI, the MasterchestCLI.exe is showing up in avast! as infected with "Win32: Evo-gen [Susp]". Ran it through Virustotal.com and it looks like a false positive, but it could cause issues for folks with avast! on their computers.

You're not kidding - driving me to tears (I run avast as well).  It seems a good portion of the binaries I spit out get flagged with a 'generic' virus match and deleted.  "Build failed" (because Avast deletes the output file) is a daily occurrence for me.

They're safe as houses - Avast have basically said because the EXEs I'm creating have no 'reputation' they are assumed to be bad.  It's an ongoing debate (I think they're well off track with this).  Just google Visual Studio & Avast and you'll see this is a fairly common issue.

In terms of end users, apparently I'm supposed to submit the binaries to Avast first so they an clear them and avoid false positives, though I'm really quite unhappy with that assertion.

To be continued...

Thanks!

[Ola]

rbdrbd this is NXT's api..this should motivate you:    https://bitcointalk.org/index.php?topic=313082.0

They will probably have hundreds of apps at their disposal when the launch their exchange..I hope you are trucking on I am 100% behind you.

[rbdrbd]

zathras,

Got a few Qs/todo items for you:

* with balances, what is the difference between CBALANCE, CBALANCET, ubalance and ubalancet???

* it appears you clear out the transactions_processed table totally every run. If I have an API query going on during this time when it's cleared and being regenerated, I would return bad data. Can you regenerate into a temp table and rename, or come up with another way? I can pause transactions for now when this is being regenerated, but that's kind of ghetto.

* masterchest-engine sql versioning - have a version table or a way for me to deduct the sql schema version....auto schema upgrades in masterchest-engine would be best... worst case, if blow away the DB on schema upgrades, if necessary, and regenerate...

* have a way for me to query the masterchest-engine version

* an abundance of magic numbers -- makes the code confusing to read. e.g. what does "If .Item(6) < 999998 Then" mean??? What is that 999998 lack of comments around these kinds of things. Also, things like "curtype = 1" -- maybe use enums instead? A future task I'm sure...I know you're busy getting Dex working (and my sqlite integration )

* what timezone does masterchest-engine store -- we should probably force UTC time. Or, the timezone must be stored with it

* you have no indexes on any rows, columns of your database. This greatly can impact performance. Do you have plans to add indexes?

* from transactions_processed, how do I tell what kind of transaction it is (multi-sig, class A, etc)?? can you add this into the schema?

* also, for transactions_processed, does "VALUE" include the bonus amount for txns in the exodus period, or not?

[zathras]

zathras,

Got a few Qs/todo items for you:

* with balances, what is the difference between CBALANCE, CBALANCET, ubalance and ubalancet???
CBALANCE = Confirmed Balance
CBALANCET = Confirmed Balance (Test)
UBALANCE = Unconfirmed Balance
UBALANCET = Unconfirmed Balance (Test)

* it appears you clear out the transactions_processed table totally every run. If I have an API query going on during this time when it's cleared and being regenerated, I would return bad data. Can you regenerate into a temp table and rename, or come up with another way? I can pause transactions for now when this is being regenerated, but that's kind of ghetto.
Yep - doing full state processing on every run is useful in these early stages for hunting bugs.  This was much less of an issue for the first version of the engine as processing only took a few ms.  As we scale out this is no longer viable so temp tables will be used (already are in my wallet).  On the to-do list

* masterchest-engine sql versioning - have a version table or a way for me to deduct the sql schema version....auto schema upgrades in masterchest-engine would be best... worst case, if blow away the DB on schema upgrades, if necessary, and regenerate...
Will consider this.  Whilst it was just me using the engine for masterchest.info obviously it didn't matter so much, but if others are going to make use of it I can see how this would be valuable.

* have a way for me to query the masterchest-engine version
No problems here - not difficult to add a -ver switch.

* an abundance of magic numbers -- makes the code confusing to read. e.g. what does "If .Item(6) < 999998 Then" mean??? What is that 999998 lack of comments around these kinds of things. Also, things like "curtype = 1" -- maybe use enums instead? A future task I'm sure...I know you're busy getting Dex working (and my sqlite integration )
Haha - the magic 999998 is for my unconfirmed transaction processing.  999999 is a dummy block number for an unconfirmed transaction.  Again as I mentioned below there is some cleanup to be done - this (along with a couple other tricks) was a quick and dirty way of supporting the display of unconfirmed transactions on masterchest.info.

* what timezone does masterchest-engine store -- we should probably force UTC time. Or, the timezone must be stored with it
Doesn't store a timezone at all, it stores the blocktime as recorded in the blockchain.  You should do localization after retrieving transaction times from the db, not while storing them.

* you have no indexes on any rows, columns of your database. This greatly can impact performance. Do you have plans to add indexes?
As needed sure - I haven't run into any performance issues yet (none of my queries take longer than a few ms) which is why I don't have them, but they can be added without to much difficulty if needed.

* from transactions_processed, how do I tell what kind of transaction it is (multi-sig, class A, etc)?? can you add this into the schema?
You can tell what kind of transaction it is from the type column (send, selloffer, acceptoffer etc).  The transaction encoding class isn't stored as I haven't come across a need for it.  Simple sends are the only supported Class A transaction type.  If you throw a simple send to mlib.getmastercointransaction it'll return the correct tx details regardless of whether it used Class A or Class B encoding.

It's not hugely disruptive to add it, but could I ask what you'd be using it for?

* also, for transactions_processed, does "VALUE" include the bonus amount for txns in the exodus period, or not?
Assume you mean for the 'generate' transaction type - and yes, 'generate' transaction types include the early adopter bonus.

Hey rbdrbd,

Sorry for the delayed response, I was away for the weekend & when I got back the forums were down.  Yep the code is quite scrappy but functional and will be cleaned up as time allows.  Please see inline for responses to your Qs.

Thanks!