300 BTC Coding Contest: Distributed Exchange (MasterCoin Developer Thread)

[zathras]

Hey guys - ok I'm up (6:15am on a Sunday </yawn> next time can we start a bit later? ). 

Just jumping in the shower to try and wake myself up then we can get started   I have some notes on the posts over the last page or so - are we meeting up on IRC?

Thanks
Zathras

[1715387454]

1715387454

[Tachikoma]

I'm on ##mastercoin and #mastercoin, if Grazcoin has a problem with it we can use cryptocat as well but it won't have logs.

Bitoy already went to bed, but he never replied to my email so I can he won't be around.

I somehow thought it was 8am where you were; next time we can do it two hours later and I will try to stay awake, I'm not great about staying awake though

We have set the seconds in a year to 31556926 and I just implemented recalculations on block level so my balance should start syncing up!

I will see you on irc and we can talk more about other things

[zathras]

Figured I'd quickly post just before I hop in the shower - the P&D clarification - Tachikoma - yeah that's exactly what I was worried about (whether the simplification is too aggressive and may result in reference address ambiguity) but to be honest I would also be concerned if the order in which we tested the rules made a difference to how the transaction is interpreted.

I actually rewrote the engine already so I could run up a second test instance - I left it running last night but due to a bug with the new code it didn't complete so I don't yet have full comparison results (old way state vs revised way state).

I know you're not .NET but hopefully comments will make the logic easy enough to read - this was a massive code & logic simplification for the engine so I am hoping the testing shows no transactions with ambiguity and we can agree to keep it.  Just FYI really.  See you on IRC in about 10

Code:

 '/// non-multisig
                            'straight to p&d
                            'loop through outputs looking for data address
                            Dim dataaddressfound As Boolean = False
                            Dim dataaddress As String = ""
                            For i = 1 To outputs.Rows.Count
                                Dim rowbarray = ToByteArray(Trim(outputs.Rows(i - 1).Item("address")))
                                'test output for simple send bytes
                                If rowbarray(2) = 0 And rowbarray(3) = 0 And rowbarray(4) = 0 And rowbarray(5) = 0 And rowbarray(6) = 0 And rowbarray(7) = 0 And rowbarray(8) = 0 And (rowbarray(9) = 1 Or rowbarray(9) = 2) Then
                                    'check we have not already found the data address
                                    If dataaddressfound = True Then
                                        'more than one data address - drop tx
                                        Exit Function
                                    Else
                                        dataaddressfound = True
                                        dataaddress = outputs.Rows(i - 1).Item("address")
                                    End If
                                End If
                            Next
                            'identify reference address
                            'first test for exactly one output with data seqnum+1
                            Dim dataaddressseqnum As Integer = 9999
                            For i = 1 To outputs.Rows.Count
                                If outputs.Rows(i - 1).Item("address") = dataaddress Then dataaddressseqnum = outputs.Rows(i - 1).Item("seqnum")
                            Next
                            'sanity check - ensure we have a dataaddressseqnum
                            If dataaddressseqnum = 9999 Then Exit Function
                            'calc reference seqnum
                            Dim refseqnum As Integer = dataaddressseqnum + 1
                            'handle 255 to 0 case
                            If refseqnum = 256 Then refseqnum = 0
                            'see if we can find a single output with the reference seqnum
                            Dim refaddressfound As Boolean = False
                            Dim refaddress As String
                            For i = 1 To outputs.Rows.Count
                                If outputs.Rows(i - 1).Item("seqnum") = refseqnum Then
                                    'if we've already found one then this doesn't satisy 'single output with a seqnum +1 of the data address'
                                    If refaddressfound = True Then
                                        'exit without refaddress
                                        refaddressfound = False
                                        refaddress = ""
                                        Exit For
                                    Else
                                        refaddress = outputs.Rows(i - 1).Item("address")
                                        refaddressfound = True
                                    End If
                                End If
                            Next
                            'refaddress will be populated if we have found a single output with data seqnum+1
                            'if it's not populated move to second test (output amounts)
                            If refaddressfound = False And refaddress = "" Then
                                'here we test for a single output with the value the same as the exodus address (excluding data address)
                                For i = 1 To outputs.Rows.Count
                                    If outputs.Rows(i - 1).Item("amount") = exoamount And outputs.Rows(i - 1).Item("address") <> dataaddress Then
                                        'if we've already found one then this doesn't satisfy 'exactly two outputs with the same value as the data address'
                                        If refaddressfound = True Then
                                            'exit without refaddress
                                            refaddressfound = False
                                            refaddress = ""
                                            Exit For
                                        Else
                                            refaddress = outputs.Rows(i - 1).Item("address")
                                            refaddressfound = True
                                        End If
                                    End If
                                Next
                            End If
                            'see if we now have a dataaddress and refaddress, otherwise throw out tx
                            If dataaddress = "" Or refaddress = "" Then
                                Exit Function
                            Else
                                isvalidtx = True
                            End If

                            'decode transaction
...continues

[grazcoin]

I'm on ##mastercoin and #mastercoin, if Grazcoin has a problem with it we can use cryptocat as well but it won't have logs.

I am on channel mastercoin in cryptocat

[Bitoy]

I'm on ##mastercoin and #mastercoin, if Grazcoin has a problem with it we can use cryptocat as well but it won't have logs.

Bitoy already went to bed, but he never replied to my email so I can he won't be around.

Cryptocat doesn't work in ipad.   

I didn't get your email (in my gmail).   

I thought we are already synced but when I checked this morning.   Any changes in the formula?

1EXoDusjGwvnjZUyKkxZ4UHEf77z6A5S4P
MM=11063.84979076 
ME=11063.58647044

[Tachikoma]

Bitoy a little summary for the work Graz, Zath and me have been doing this night/day:

• We changed the date from which we started calculating the vesting development coins to the date the last block was accepted for the fundraiser. The new timestamp is: 1377993874
• Our implementations should be 100% on consensus now but your engine appeared to have stopped working so we can't be 100% sure, also Zathras is still updating his blockchain with the new code.
• Two new pull requests: Clarifications on the development Mastercoins and an update on the verification API to make it work for DEx consensus.

[Bitoy]

Bitoy a little summary for the work Graz, Zath and me have been doing this night/day:

• We changed the date from which we started calculating the vesting development coins to the date the last block was accepted for the fundraiser. The new timestamp is: 1377993874
• Our implementations should be 100% on consensus now but your engine appeared to have stopped working so we can't be 100% sure, also Zathras is still updating his blockchain with the new code.
• Two new pull requests: Clarifications on the development Mastercoins and an update on the verification API to make it work for DEx consensus.

Ok will update dev coin calculations today. 

I update from blockchain.org every 30 minutes. Ill try to update more often ( but I'm very conscious that blockchain might block my IP address  .

[Bitoy]

Masterchain, Mastercoin-Explorer and MyMastercoins are now fully synched (MSC).


Development should move faster now since we don't have any more Class A transaction for DEx.

[Tachikoma]

Graz: I noticed you fixed the rounding error; what was it in the end?

Zathras: Any details on the new parsing engine? Did it work?

Almost at 100% guys :}

[grazcoin]

Graz: I noticed you fixed the rounding error; what was it in the end?

Zathras: Any details on the new parsing engine? Did it work?

Almost at 100% guys :}

I was using a different number of total reward mastercoins (ending with ...6245 instead of ...6222). I missed it during the online meeting session as the number is calculated dynamically (not hard coded like in your implementation).
A hot fix was to make it hard coded, but I still have to understand why I got a different number.

But:

I have noticed *another* rounding difference that happens less often. Today at one time you rounded last digit to ....8 and I rounded to ....7.
Normally it would mean temporary consensus loss for one block, but if by chance exodus sends all all funds (e.g. the number you suggested that ends with digit 8 ), my implementation will invalidate the tx since not enough funds are there. Then it is a constant consensus loss. The chance for this is close to zero, but it should be fixed.

conclusion:
we have to define *exactly* the rounding rules.

[zathras]

Consensus system going offline temporarily while I use it to analyse consensus of the uber-simplified Class A model.

Should be about 15 - 30 mins.

Thanks
Zathras

[zathras]

Consensus system going offline temporarily while I use it to analyse consensus of the uber-simplified Class A model.

Should be about 15 - 30 mins.

Thanks
Zathras

Typical luck...  Bitoy - your verification API is down.  I'm bringing consensus back to normal and will re-run testing when Bitoy resolves the issue.

P.S. Bitoy - just a suggestion but your security config needs work mate (lock it down in web.config) - for example you're exposing debugging publicly (which enables nefarious parties to look for exploits in your code at runtime - lock it down with customerrors="remoteonly" so you can only see debugging if you're local to the box).  If you need any advice please just shout  



Also brought me to a bug in consensus checking code, if one of you guys is down and doesn't return any data the check is aborted and the state remains as per last consensus check for all implementations which makes it look like we still have consensus when we don't.  Will trap those exceptions and enable it so the system can adapt to failures in the various implementations providing consensus data.

Tachikoma - what are your views on scoring?  For example if I modify the code to adapt to one of the implementations being down, it would then do the comparison against the remaining three implementations.  In this scenario if those remaining three agree do you think our consensus score should remain at 100%, or use last-known values for the failed implementation (will definitely cause a drop in consensus score).  I see value to both approaches.

Thanks
Zathras

[zathras]

Zathras: Any details on the new parsing engine? Did it work?

Class A offline analysis looks good  I broke multisig though haha

Also pushing into the various transaction edge cases I did find one that wasn't satisfied (though has never been used in a transaction) - I'm hoping to exclude if all agree.  More details to come, I'd like to give you guys a holistic overview of applying this simplification but without being able to run consensus testing I can only compare against my own original implementation and I'd like to present the whole picture.

To be continued! ...

Thanks
Zathras

EDIT: Couple of quick performance wins too:

• Removed input checking from the ismastercointx function, as this function really only needs to identify whether it's a Mastercoin transaction and what type, not where it came from.  Since this removes recursion looking up transactions for each vin that speeds blockchain scanning up a lot.
• Also found AVG runs a process called AVGNSA (network shield agent) that intercepts and proxies all network traffic (aptly named huh?).  This was slowing down my calls to bitcoins RPC server by over 50%.  That's something I'm going to have to consider for release.

EDIT2: Grrrr! Lol
OK so it wasn't the Class A simplification that broke my multisig code, it was the performance tuning to remove input checking from 'ismastercointx'.  That now returns 'none' for multisigs because without the sending address we can't decrypt & decode the data packet(s) to analyze the type of transaction.  I'm going to have to roll-back the input recursion performance improvement (no biggy) and come at this a different way.  Address watch-only functionality in bitcoind is a perfect use (99% of the work already done by BDB so scanning would only take a few secs) but I've been following that pull since I started developing for Mastercoin several months ago and it's not looking like moving quickly

EDIT3: Compromise
Hehe.  So I've taken middle ground here.  ismastercointx has been rewritten to check the inputs only on multisig transactions containing an Exodus output.  This gives me most of the performance benefit without breaking multisig.  Running another full blockchain scan now while we wait for Bitoy

[Bitoy]

Bug is fixed.   

(I tried to update the api so that it checks for new transaction every time consensus queries it.)


Zathras,

Thanks for the advise regarding customerror.   (I created a Generic Error page to catch bugs in the future bugs =)

[zathras]

Bug is fixed.   

(I tried to update the api so that it checks for new transaction every time consensus queries it.)

Zathras,

Thanks for the advise regarding customerror.   (I created a Generic Error page to catch bugs in the future bugs =)

Perfect, thanks bud.  Seems Masterchain is behind now... 

What are your scantimes guys?  Bitoy I think you mentioned 30 mins - of course it's up to you but since that's (on average) 3 blocks you may spend most of your time a couple of blocks behind.  I can't speak for blockchain.info but I doubt they're going to be concerned about you polling their API every 5 mins or so, I'd posit there were many services out there already polling them much quicker than that, but as I say I'm no authority on the matter

[zathras]

Bug is fixed.  

(I tried to update the api so that it checks for new transaction every time consensus queries it.)

Zathras,

Thanks for the advise regarding customerror.   (I created a Generic Error page to catch bugs in the future bugs =)

Perfect, thanks bud.  Seems Masterchain is behind now...  

What are your scantimes guys?  Bitoy I think you mentioned 30 mins - of course it's up to you but since that's (on average) 3 blocks you may spend most of your time a couple of blocks behind.  I can't speak for blockchain.info but I doubt they're going to be concerned about you polling their API every 5 mins or so, I'd posit there were many services out there already polling them much quicker than that, but as I say I'm no authority on the matter

Here you go, bottom of https://blockchain.info/api.  If you look for new transactions every 5 minutes unless there are 725 Mastercoin transactions you need to grab you should be fine.  Also these limits don't apply if you use an API key apparently.

Requests in 8 Hours: 4 (Soft Limit = 28800, Hard Limit = 28900)
Requests in 5 minutes: 4 (Soft Limit = 700, Hard Limit = 725)

Hope that helps

[Tachikoma]

I think we should ignore the consensus for sites who's verification API is down, it makes the most sense to me.

I will be very busy this week so I won't have much time to go into coding a whole lot but if I find some time I will most likely be working on implementing the extended verification API for DEx. I think we need to extend the consensus to transactions soon to make it easier to find discrepancies in the DEx code.  

Edit: Wow I missed a page, reading back.

Edit 2: Haha tuning can mess with stuff indeed! I should really create time to do a full parse as well. I havent in a while.... /me adds that to the todo.

[Tachikoma]

Devs: Could you +1/-1 this pull so we can quickly merge it: https://github.com/mastercoin-MSC/spec/pull/35#issuecomment-32158885

[Tachikoma]

No that's MST, this is MSC.

[grazcoin]

Devs: Could you +1/-1 this pull so we can quickly merge it: https://github.com/mastercoin-MSC/spec/pull/35#issuecomment-32158885

just commented.