zathras
|
|
January 11, 2014, 07:17:29 PM |
|
Hey guys - ok I'm up (6:15am on a Sunday </yawn> next time can we start a bit later? ). Just jumping in the shower to try and wake myself up then we can get started I have some notes on the posts over the last page or so - are we meeting up on IRC? Thanks Zathras
|
|
|
|
Tachikoma
|
|
January 11, 2014, 07:20:44 PM |
|
I'm on ##mastercoin and #mastercoin, if Grazcoin has a problem with it we can use cryptocat as well but it won't have logs. Bitoy already went to bed, but he never replied to my email so I can he won't be around. I somehow thought it was 8am where you were; next time we can do it two hours later and I will try to stay awake, I'm not great about staying awake though We have set the seconds in a year to 31556926 and I just implemented recalculations on block level so my balance should start syncing up! I will see you on irc and we can talk more about other things
|
|
|
|
zathras
|
|
January 11, 2014, 07:32:24 PM |
|
Figured I'd quickly post just before I hop in the shower - the P&D clarification - Tachikoma - yeah that's exactly what I was worried about (whether the simplification is too aggressive and may result in reference address ambiguity) but to be honest I would also be concerned if the order in which we tested the rules made a difference to how the transaction is interpreted. I actually rewrote the engine already so I could run up a second test instance - I left it running last night but due to a bug with the new code it didn't complete so I don't yet have full comparison results (old way state vs revised way state). I know you're not .NET but hopefully comments will make the logic easy enough to read - this was a massive code & logic simplification for the engine so I am hoping the testing shows no transactions with ambiguity and we can agree to keep it. Just FYI really. See you on IRC in about 10 '/// non-multisig 'straight to p&d 'loop through outputs looking for data address Dim dataaddressfound As Boolean = False Dim dataaddress As String = "" For i = 1 To outputs.Rows.Count Dim rowbarray = ToByteArray(Trim(outputs.Rows(i - 1).Item("address"))) 'test output for simple send bytes If rowbarray(2) = 0 And rowbarray(3) = 0 And rowbarray(4) = 0 And rowbarray(5) = 0 And rowbarray(6) = 0 And rowbarray(7) = 0 And rowbarray(8) = 0 And (rowbarray(9) = 1 Or rowbarray(9) = 2) Then 'check we have not already found the data address If dataaddressfound = True Then 'more than one data address - drop tx Exit Function Else dataaddressfound = True dataaddress = outputs.Rows(i - 1).Item("address") End If End If Next 'identify reference address 'first test for exactly one output with data seqnum+1 Dim dataaddressseqnum As Integer = 9999 For i = 1 To outputs.Rows.Count If outputs.Rows(i - 1).Item("address") = dataaddress Then dataaddressseqnum = outputs.Rows(i - 1).Item("seqnum") Next 'sanity check - ensure we have a dataaddressseqnum If dataaddressseqnum = 9999 Then Exit Function 'calc reference seqnum Dim refseqnum As Integer = dataaddressseqnum + 1 'handle 255 to 0 case If refseqnum = 256 Then refseqnum = 0 'see if we can find a single output with the reference seqnum Dim refaddressfound As Boolean = False Dim refaddress As String For i = 1 To outputs.Rows.Count If outputs.Rows(i - 1).Item("seqnum") = refseqnum Then 'if we've already found one then this doesn't satisy 'single output with a seqnum +1 of the data address' If refaddressfound = True Then 'exit without refaddress refaddressfound = False refaddress = "" Exit For Else refaddress = outputs.Rows(i - 1).Item("address") refaddressfound = True End If End If Next 'refaddress will be populated if we have found a single output with data seqnum+1 'if it's not populated move to second test (output amounts) If refaddressfound = False And refaddress = "" Then 'here we test for a single output with the value the same as the exodus address (excluding data address) For i = 1 To outputs.Rows.Count If outputs.Rows(i - 1).Item("amount") = exoamount And outputs.Rows(i - 1).Item("address") <> dataaddress Then 'if we've already found one then this doesn't satisfy 'exactly two outputs with the same value as the data address' If refaddressfound = True Then 'exit without refaddress refaddressfound = False refaddress = "" Exit For Else refaddress = outputs.Rows(i - 1).Item("address") refaddressfound = True End If End If Next End If 'see if we now have a dataaddress and refaddress, otherwise throw out tx If dataaddress = "" Or refaddress = "" Then Exit Function Else isvalidtx = True End If
'decode transaction ...continues
|
|
|
|
grazcoin
|
|
January 11, 2014, 07:58:00 PM |
|
I'm on ##mastercoin and #mastercoin, if Grazcoin has a problem with it we can use cryptocat as well but it won't have logs.
I am on channel mastercoin in cryptocat
|
|
|
|
Bitoy
|
|
January 12, 2014, 12:31:06 AM |
|
I'm on ##mastercoin and #mastercoin, if Grazcoin has a problem with it we can use cryptocat as well but it won't have logs.
Bitoy already went to bed, but he never replied to my email so I can he won't be around.
Cryptocat doesn't work in ipad. I didn't get your email (in my gmail). I thought we are already synced but when I checked this morning. Any changes in the formula? 1EXoDusjGwvnjZUyKkxZ4UHEf77z6A5S4P MM=11063.84979076 ME=11063.58647044
|
|
|
|
Tachikoma
|
|
January 12, 2014, 12:34:35 AM |
|
Bitoy a little summary for the work Graz, Zath and me have been doing this night/day: - We changed the date from which we started calculating the vesting development coins to the date the last block was accepted for the fundraiser. The new timestamp is: 1377993874
- Our implementations should be 100% on consensus now but your engine appeared to have stopped working so we can't be 100% sure, also Zathras is still updating his blockchain with the new code.
- Two new pull requests: Clarifications on the development Mastercoins and an update on the verification API to make it work for DEx consensus.
|
|
|
|
Bitoy
|
|
January 12, 2014, 10:07:29 AM |
|
Bitoy a little summary for the work Graz, Zath and me have been doing this night/day: - We changed the date from which we started calculating the vesting development coins to the date the last block was accepted for the fundraiser. The new timestamp is: 1377993874
- Our implementations should be 100% on consensus now but your engine appeared to have stopped working so we can't be 100% sure, also Zathras is still updating his blockchain with the new code.
- Two new pull requests: Clarifications on the development Mastercoins and an update on the verification API to make it work for DEx consensus.
Ok will update dev coin calculations today. I update from blockchain.org every 30 minutes. Ill try to update more often ( but I'm very conscious that blockchain might block my IP address .
|
|
|
|
Bitoy
|
|
January 12, 2014, 01:12:51 PM |
|
Masterchain, Mastercoin-Explorer and MyMastercoins are now fully synched (MSC).
Development should move faster now since we don't have any more Class A transaction for DEx.
|
|
|
|
Tachikoma
|
|
January 12, 2014, 07:20:42 PM |
|
Graz: I noticed you fixed the rounding error; what was it in the end?
Zathras: Any details on the new parsing engine? Did it work?
Almost at 100% guys :}
|
|
|
|
grazcoin
|
|
January 12, 2014, 08:55:07 PM |
|
Graz: I noticed you fixed the rounding error; what was it in the end?
Zathras: Any details on the new parsing engine? Did it work?
Almost at 100% guys :}
I was using a different number of total reward mastercoins (ending with ...6245 instead of ...6222). I missed it during the online meeting session as the number is calculated dynamically (not hard coded like in your implementation). A hot fix was to make it hard coded, but I still have to understand why I got a different number. But: I have noticed *another* rounding difference that happens less often. Today at one time you rounded last digit to ....8 and I rounded to ....7. Normally it would mean temporary consensus loss for one block, but if by chance exodus sends all all funds (e.g. the number you suggested that ends with digit 8 ), my implementation will invalidate the tx since not enough funds are there. Then it is a constant consensus loss. The chance for this is close to zero, but it should be fixed. conclusion: we have to define *exactly* the rounding rules.
|
|
|
|
zathras
|
|
January 12, 2014, 10:57:44 PM |
|
Consensus system going offline temporarily while I use it to analyse consensus of the uber-simplified Class A model.
Should be about 15 - 30 mins.
Thanks Zathras
|
|
|
|
zathras
|
|
January 12, 2014, 11:23:01 PM Last edit: January 13, 2014, 06:48:39 AM by zathras |
|
Consensus system going offline temporarily while I use it to analyse consensus of the uber-simplified Class A model.
Should be about 15 - 30 mins.
Thanks Zathras
Typical luck... Bitoy - your verification API is down. I'm bringing consensus back to normal and will re-run testing when Bitoy resolves the issue. P.S. Bitoy - just a suggestion but your security config needs work mate (lock it down in web.config) - for example you're exposing debugging publicly (which enables nefarious parties to look for exploits in your code at runtime - lock it down with customerrors="remoteonly" so you can only see debugging if you're local to the box). If you need any advice please just shout Also brought me to a bug in consensus checking code, if one of you guys is down and doesn't return any data the check is aborted and the state remains as per last consensus check for all implementations which makes it look like we still have consensus when we don't. Will trap those exceptions and enable it so the system can adapt to failures in the various implementations providing consensus data. Tachikoma - what are your views on scoring? For example if I modify the code to adapt to one of the implementations being down, it would then do the comparison against the remaining three implementations. In this scenario if those remaining three agree do you think our consensus score should remain at 100%, or use last-known values for the failed implementation (will definitely cause a drop in consensus score). I see value to both approaches. Thanks Zathras
|
|
|
|
zathras
|
|
January 12, 2014, 11:47:12 PM Last edit: January 13, 2014, 06:43:53 AM by zathras |
|
Zathras: Any details on the new parsing engine? Did it work?
Class A offline analysis looks good I broke multisig though haha Also pushing into the various transaction edge cases I did find one that wasn't satisfied (though has never been used in a transaction) - I'm hoping to exclude if all agree. More details to come, I'd like to give you guys a holistic overview of applying this simplification but without being able to run consensus testing I can only compare against my own original implementation and I'd like to present the whole picture. To be continued! ... Thanks Zathras EDIT: Couple of quick performance wins too: - Removed input checking from the ismastercointx function, as this function really only needs to identify whether it's a Mastercoin transaction and what type, not where it came from. Since this removes recursion looking up transactions for each vin that speeds blockchain scanning up a lot.
- Also found AVG runs a process called AVGNSA (network shield agent) that intercepts and proxies all network traffic (aptly named huh?). This was slowing down my calls to bitcoins RPC server by over 50%. That's something I'm going to have to consider for release.
EDIT2: Grrrr! Lol OK so it wasn't the Class A simplification that broke my multisig code, it was the performance tuning to remove input checking from 'ismastercointx'. That now returns 'none' for multisigs because without the sending address we can't decrypt & decode the data packet(s) to analyze the type of transaction. I'm going to have to roll-back the input recursion performance improvement (no biggy) and come at this a different way. Address watch-only functionality in bitcoind is a perfect use (99% of the work already done by BDB so scanning would only take a few secs) but I've been following that pull since I started developing for Mastercoin several months ago and it's not looking like moving quickly EDIT3: Compromise Hehe. So I've taken middle ground here. ismastercointx has been rewritten to check the inputs only on multisig transactions containing an Exodus output. This gives me most of the performance benefit without breaking multisig. Running another full blockchain scan now while we wait for Bitoy
|
|
|
|
Bitoy
|
|
January 13, 2014, 07:02:13 AM |
|
Bug is fixed.
(I tried to update the api so that it checks for new transaction every time consensus queries it.)
Zathras,
Thanks for the advise regarding customerror. (I created a Generic Error page to catch bugs in the future bugs =)
|
|
|
|
zathras
|
|
January 13, 2014, 07:44:26 AM |
|
Bug is fixed.
(I tried to update the api so that it checks for new transaction every time consensus queries it.)
Zathras,
Thanks for the advise regarding customerror. (I created a Generic Error page to catch bugs in the future bugs =)
Perfect, thanks bud. Seems Masterchain is behind now... What are your scantimes guys? Bitoy I think you mentioned 30 mins - of course it's up to you but since that's (on average) 3 blocks you may spend most of your time a couple of blocks behind. I can't speak for blockchain.info but I doubt they're going to be concerned about you polling their API every 5 mins or so, I'd posit there were many services out there already polling them much quicker than that, but as I say I'm no authority on the matter
|
|
|
|
zathras
|
|
January 13, 2014, 07:49:37 AM |
|
Bug is fixed.
(I tried to update the api so that it checks for new transaction every time consensus queries it.)
Zathras,
Thanks for the advise regarding customerror. (I created a Generic Error page to catch bugs in the future bugs =)
Perfect, thanks bud. Seems Masterchain is behind now... What are your scantimes guys? Bitoy I think you mentioned 30 mins - of course it's up to you but since that's (on average) 3 blocks you may spend most of your time a couple of blocks behind. I can't speak for blockchain.info but I doubt they're going to be concerned about you polling their API every 5 mins or so, I'd posit there were many services out there already polling them much quicker than that, but as I say I'm no authority on the matter Here you go, bottom of https://blockchain.info/api. If you look for new transactions every 5 minutes unless there are 725 Mastercoin transactions you need to grab you should be fine. Also these limits don't apply if you use an API key apparently. Requests in 8 Hours: 4 (Soft Limit = 28800, Hard Limit = 28900) Requests in 5 minutes: 4 (Soft Limit = 700, Hard Limit = 725) Hope that helps
|
|
|
|
Tachikoma
|
|
January 13, 2014, 10:25:20 AM Last edit: January 13, 2014, 10:37:22 AM by Tachikoma |
|
I think we should ignore the consensus for sites who's verification API is down, it makes the most sense to me.
I will be very busy this week so I won't have much time to go into coding a whole lot but if I find some time I will most likely be working on implementing the extended verification API for DEx. I think we need to extend the consensus to transactions soon to make it easier to find discrepancies in the DEx code.
Edit: Wow I missed a page, reading back.
Edit 2: Haha tuning can mess with stuff indeed! I should really create time to do a full parse as well. I havent in a while.... /me adds that to the todo.
|
|
|
|
|
Tachikoma
|
|
January 13, 2014, 11:04:44 AM |
|
No that's MST, this is MSC.
|
|
|
|
|
|