300 BTC Coding Contest: Distributed Exchange (MasterCoin Developer Thread)

[Tachikoma]

OK, so grazcoin posted a MSC address, and tachikoma said to use the one he gave me for the contest. Just need one from zathras now

Sure 1zAtHRASgdHvZDfHs6xJquMghga4eG7gy

No big hurry on this (the distributed exchange is way more important). Sorry for the distraction. The board just wanted to get some concrete evidence that the "dev mastercoins" (I like that name, vokain) would be treated as valid.

No problems - the dev Mastercoins deliverable is likely to be another couple days work and should be quite straight forward to do Still working on adding exchange support to my wallet too - at the moment I'm playing with a bunch of variations on trading by unconfirmed transactions (which greatly increases the speed of the exchange but brings a whole new set of problems) - if I can get it right though our distributed Exchange will be as close to real-time as unconfirmed transaction propagation allows.

Are you at a point where you can confirm that my Selling/Purchase offers are valid? I don't really want to move forward until I know I'm actually creating valid messages.

[1714608641]

1714608641

[1714608641]

1714608641

[1714608641]

1714608641

[1714608641]

1714608641

[1714608641]

1714608641

[zathras]

OK, so grazcoin posted a MSC address, and tachikoma said to use the one he gave me for the contest. Just need one from zathras now

Sure 1zAtHRASgdHvZDfHs6xJquMghga4eG7gy

No big hurry on this (the distributed exchange is way more important). Sorry for the distraction. The board just wanted to get some concrete evidence that the "dev mastercoins" (I like that name, vokain) would be treated as valid.

No problems - the dev Mastercoins deliverable is likely to be another couple days work and should be quite straight forward to do Still working on adding exchange support to my wallet too - at the moment I'm playing with a bunch of variations on trading by unconfirmed transactions (which greatly increases the speed of the exchange but brings a whole new set of problems) - if I can get it right though our distributed Exchange will be as close to real-time as unconfirmed transaction propagation allows.

Are you at a point where you can confirm that my Selling/Purchase offers are valid? I don't really want to move forward until I know I'm actually creating valid messages.

Not yet, sorry (I really only have a mish-mash of half written code at the mo) - give me a day or so

[Tachikoma]

Sure thing! Thanks :}

[superfluouso]

I saw on grazcoins site, the buy and sells created so far show up as Unknown.

[zathras]

While I'm online I thought I'd ask your opinions on scalability?  My biggest concern is that once Mastercoin is big enough to have multiple trades per hour then most of the traders are going to go for the same lowest ask 'sell offers' and by the time the next block comes around (which could be as long as 45 mins plus sometimes) most 'accept offers' are going to be invalidated.

Wallets perhaps hiding sell offers once they see an unconfirmed accept for said sell may go some way towards addressing this.

[Tachikoma]

I wouldn't do that if I were you. The problem is that just because somebody else did a Purchase Offer already does not mean he will end up getting it. The position of your transaction in the block could still be above the one send earlier.

[zathras]

I wouldn't do that if I were you. The problem is that just because somebody else did a Purchase Offer already does not mean he will end up getting it. The position of your transaction in the block could still be above the one send earlier.

It doesn't necessarily have to be a hide, it can be a highlight/color change etc - just something to let the user know that there is already an accept visible on the blockchain for this sell and thus their accept is likely to be invalidated.  

You're correct in that there is nothing stopping the user (protocol wise) from sending their own accept and hoping it gets included earlier in the block than the accept already visible.

That's essentially the crux of the scalability problem - an exchange where the lowest ask sell can only be bought once every (on average) 10 minutes just doesn't scale.  To avoid everyone trying to accept the same sell we need a way to consider that lowest sell 'pending acceptance' (or 'Purchase Offered') and have users look to the next lowest sell intra-block to make it scalable in my opinion.

EDIT: for clarity

[Paleus]

Hey guys,

Just a quick update here, been working on the price tracking chart lately and I have a friend of mine working alongside me so progress is being made. However, I need to fix some hardware in my computer so I may be a bit delayed with a full update.

I will put the ticker on a website when it is finished and will post a full update here when the project is completed.

[Tachikoma]

I wouldn't do that if I were you. The problem is that just because somebody else did a Purchase Offer already does not mean he will end up getting it. The position of your transaction in the block could still be above the one send earlier.

It doesn't necessarily have to be a hide, it can be a highlight/color change etc - just something to let the user know that there is already an accept visible on the blockchain for this sell and thus their accept is likely to be invalidated.  

You're correct in that there is nothing stopping the user (protocol wise) from sending their own accept and hoping it gets included earlier in the block than the accept already visible.

That's essentially the crux of the scalability problem - an exchange where the lowest ask sell can only be bought once every (on average) 10 minutes just doesn't scale. To avoid everyone trying to accept the same sell we need a way to consider that lowest sell 'pending acceptance' (or 'Purchase Offered') and have users look to the next lowest sell intra-block to make it scalable in my opinion.

A color indication would be a great idea. I personally would still try to get the cheapest option since I have just as much chance of getting it then the person who already submitted an offer. All I wanted to convey is that the first message seen by a certain node has no increased chance of being the first one in the block. I would prefer the cheapest option, even if I need to fight for it. But by coloring it you at least give people an option to decide what is important for them.

[dacoinminster]

I wouldn't do that if I were you. The problem is that just because somebody else did a Purchase Offer already does not mean he will end up getting it. The position of your transaction in the block could still be above the one send earlier.

It doesn't necessarily have to be a hide, it can be a highlight/color change etc - just something to let the user know that there is already an accept visible on the blockchain for this sell and thus their accept is likely to be invalidated.  

You're correct in that there is nothing stopping the user (protocol wise) from sending their own accept and hoping it gets included earlier in the block than the accept already visible.

That's essentially the crux of the scalability problem - an exchange where the lowest ask sell can only be bought once every (on average) 10 minutes just doesn't scale. To avoid everyone trying to accept the same sell we need a way to consider that lowest sell 'pending acceptance' (or 'Purchase Offered') and have users look to the next lowest sell intra-block to make it scalable in my opinion.

A color indication would be a great idea. I personally would still try to get the cheapest option since I have just as much chance of getting it then the person who already submitted an offer. All I wanted to convey is that the first message seen by a certain node has no increased chance of being the first one in the block. I would prefer the cheapest option, even if I need to fight for it. But by coloring it you at least give people an option to decide what is important for them.

An interesting dynamic will come out of this, where MSC buyers will have to decide whether they want to go for the best price (and maybe/probably not get it) or pay a bit more for an offer in order to have a better chance of being first to accept.

When we do the distributed eschange messages between MSC and other MSC-derived currencies like smart property and GoldCoins, the buyer doesn't have to specify which seller they are buying from - we can just match up buy/sell offers which happen to match automatically. Currently the spec doesn't state this very well (I need to fix that).

Unfortunately, the scalability problem will always remain when exchanging MSC/BTC. Once someone has purchased MSC and is completely in the MSC ecosystem, everything gets way easier.

[zathras]

I wouldn't do that if I were you. The problem is that just because somebody else did a Purchase Offer already does not mean he will end up getting it. The position of your transaction in the block could still be above the one send earlier.

It doesn't necessarily have to be a hide, it can be a highlight/color change etc - just something to let the user know that there is already an accept visible on the blockchain for this sell and thus their accept is likely to be invalidated.  

You're correct in that there is nothing stopping the user (protocol wise) from sending their own accept and hoping it gets included earlier in the block than the accept already visible.

That's essentially the crux of the scalability problem - an exchange where the lowest ask sell can only be bought once every (on average) 10 minutes just doesn't scale. To avoid everyone trying to accept the same sell we need a way to consider that lowest sell 'pending acceptance' (or 'Purchase Offered') and have users look to the next lowest sell intra-block to make it scalable in my opinion.

A color indication would be a great idea. I personally would still try to get the cheapest option since I have just as much chance of getting it then the person who already submitted an offer. All I wanted to convey is that the first message seen by a certain node has no increased chance of being the first one in the block. I would prefer the cheapest option, even if I need to fight for it. But by coloring it you at least give people an option to decide what is important for them.

You know I may have been operating on a bad assumption re the above; being as an unconfirmed transaction propagated throughout the network it would be added to the bottom of the (currently being mined) block template, thus transactions broadcast earlier would have a better chance of being first in the block.  It sounds like you're saying this is not the case? (Is there some kind of block reorganization etc?)  As I mentioned early on I'm quite new on this cryptocurrency 'scene' and have just been teaching myself by reading as much bitcoin literature (wikis, papers, forum posts etc) as I can get my hands on - so I'm certainly more than happy to be corrected if my interpretation is wrong!

An interesting dynamic will come out of this, where MSC buyers will have to decide whether they want to go for the best price (and maybe/probably not get it) or pay a bit more for an offer in order to have a better chance of being first to accept.

When we do the distributed eschange messages between MSC and other MSC-derived currencies like smart property and GoldCoins, the buyer doesn't have to specify which seller they are buying from - we can just match up buy/sell offers which happen to match automatically. Currently the spec doesn't state this very well (I need to fix that).

Unfortunately, the scalability problem will always remain when exchanging MSC/BTC. Once someone has purchased MSC and is completely in the MSC ecosystem, everything gets way easier.

Thanks for the additional info I'm just going to throw out an idea I've been playing with to see what you guys think (feel free to tear it apart if it's not a good idea!).  What about an additional optional message.  An 'acknowledge' if you will.   MSC/BTC exchange could function as per the discussion so far but have additional support for the user that created the sell to broadcast a message to 'acknowledge' a specific accept/Purchase Offer before it's confirmed in a block.  So:

Wallet broadcasts sell offer > Wallet sees a valid accept/Purchase offer in the block template > Wallet broadcasts an 'acknowledge' including the ID of the accept/Purchase offer it's agreeing to > Trade is considered locked in & waiting to be funded > Other parties see no point trying to accept this sell now and move on to the next

All of that can happen intra-block, as long as the user (specifically their wallet) participating in trading stays online.

Of course a wallet can also simply post a sell, go offline and have it bought by the first accept/Purchase in a block as per the current spec if there is no 'acknowledge' already sent by the seller.

I am concerned that perhaps doing anything at all with unconfirmed messages at a fundamental level regardless of how we structure it is a bad idea because we can't be 100% sure those unconfirmed transactions will be included in the next block.  Appreciate your thoughts

[dacoinminster]

Thanks for the additional info I'm just going to throw out an idea I've been playing with to see what you guys think (feel free to tear it apart if it's not a good idea!).  What about an additional optional message.  An 'acknowledge' if you will.   MSC/BTC exchange could function as per the discussion so far but have additional support for the user that created the sell to broadcast a message to 'acknowledge' a specific accept/Purchase Offer before it's confirmed in a block.  So:

Wallet broadcasts sell offer > Wallet sees a valid accept/Purchase offer in the block template > Wallet broadcasts an 'acknowledge' including the ID of the accept/Purchase offer it's agreeing to > Trade is considered locked in & waiting to be funded > Other parties see no point trying to accept this sell now and move on to the next

All of that can happen intra-block, as long as the user (specifically their wallet) participating in trading stays online.

Of course a wallet can also simply post a sell, go offline and have it bought by the first accept/Purchase in a block as per the current spec if their is no 'acknowledge' in that block sent by the seller.

I am concerned that perhaps doing anything at all with unconfirmed messages at a fundamental level regardless of how we structure it is a bad idea because we can't be 100% sure those unconfirmed transactions will be included in the next block.  Appreciate your thoughts

Ooooo! Interesting idea!

I don't see any problem with doing it this way, as long as it is optional. However, I think this should probably not be a version 1 feature, for a couple reasons:

1) I anticipate that most people will not be online when their sell order clears - they will post the order at some high price and forget about it for a few weeks until the market finally gets there
2) I don't want to encourage high-frequency trading. There is nothing stopping people from creating traditional website-based exchanges for people who want to do a lot of "day trading" of MSC, but I'd rather those short-term speculations not end up in the block chain for all of eternity. My goal is to make it easy on people who are doing a large purchase in or out of MSC, but not too friendly for overly-caffeinated chart monkeys.

I think the most important thing is for the UI on the websites to be really clear when you are confirmed as the first to accept an offer and it is safe to send the BTC. It would be BAD to base that advice on an ACK message that might not be included in the block chain for some reason (that would create an attack vector for fraud). The ACK message (if we create one) should only be to discourage other people from attempting to buy the same MSC.

[Tachikoma]

You know I may have been operating on a bad assumption re the above; being as an unconfirmed transaction propagated throughout the network it would be added to the bottom of the (currently being mined) block template, thus transactions broadcast earlier would have a better chance of being first in the block.  It sounds like you're saying this is not the case? (Is there some kind of block reorganization etc?)  As I mentioned early on I'm quite new on this cryptocurrency 'scene' and have just been teaching myself by reading as much bitcoin literature (wikis, papers, forum posts etc) as I can get my hands on - so I'm certainly more than happy to be corrected if my interpretation is wrong!

I'm not saying I'm an expert yet, so as far as I know it could be that transactions are added 'to the bottom' but I don't think that is the case. And even if it is, there is no 'one block'. Every miner or pool is working on his own version of a block based on the transactions he saw during the last block. There is no 100% propagation per transaction so a lot of time a node only learns about transactions because they have been added to the block. This being the case it does not even matter where the transaction gets added. The only importance is where it got added by the block that wins the race. I would like to learn how transactions get added though. Initial searches didn't give me the answer so I'm afraid I need to dive into the source code.

Thanks for the additional info I'm just going to throw out an idea I've been playing with to see what you guys think (feel free to tear it apart if it's not a good idea!).  What about an additional optional message.  An 'acknowledge' if you will.   MSC/BTC exchange could function as per the discussion so far but have additional support for the user that created the sell to broadcast a message to 'acknowledge' a specific accept/Purchase Offer before it's confirmed in a block.  So:

Wallet broadcasts sell offer > Wallet sees a valid accept/Purchase offer in the block template > Wallet broadcasts an 'acknowledge' including the ID of the accept/Purchase offer it's agreeing to > Trade is considered locked in & waiting to be funded > Other parties see no point trying to accept this sell now and move on to the next

All of that can happen intra-block, as long as the user (specifically their wallet) participating in trading stays online.

Of course a wallet can also simply post a sell, go offline and have it bought by the first accept/Purchase in a block as per the current spec if their is no 'acknowledge' in that block sent by the seller.

I am concerned that perhaps doing anything at all with unconfirmed messages at a fundamental level regardless of how we structure it is a bad idea because we can't be 100% sure those unconfirmed transactions will be included in the next block.  Appreciate your thoughts

I like the idea but I wouldn't want to spend time on it right now. There are so many things to build and it will be some time before we will need such a feature. It might help solve the scalability issues in the long term though.

There is just one major problem with it. Like I explained before there will never be 100% propagation, so there is a huge chance the buyer will never see the acknowledge before it ends up in a block. I really think we should try to steer away from doing messages on unconfirmed transactions.

[zathras]

Thanks for the additional info I'm just going to throw out an idea I've been playing with to see what you guys think (feel free to tear it apart if it's not a good idea!).  What about an additional optional message.  An 'acknowledge' if you will.   MSC/BTC exchange could function as per the discussion so far but have additional support for the user that created the sell to broadcast a message to 'acknowledge' a specific accept/Purchase Offer before it's confirmed in a block.  So:

Wallet broadcasts sell offer > Wallet sees a valid accept/Purchase offer in the block template > Wallet broadcasts an 'acknowledge' including the ID of the accept/Purchase offer it's agreeing to > Trade is considered locked in & waiting to be funded > Other parties see no point trying to accept this sell now and move on to the next

All of that can happen intra-block, as long as the user (specifically their wallet) participating in trading stays online.

Of course a wallet can also simply post a sell, go offline and have it bought by the first accept/Purchase in a block as per the current spec if their is no 'acknowledge' in that block sent by the seller.

I am concerned that perhaps doing anything at all with unconfirmed messages at a fundamental level regardless of how we structure it is a bad idea because we can't be 100% sure those unconfirmed transactions will be included in the next block.  Appreciate your thoughts

Ooooo! Interesting idea!

I don't see any problem with doing it this way, as long as it is optional. However, I think this should probably not be a version 1 feature, for a couple reasons:

1) I anticipate that most people will not be online when their sell order clears - they will post the order at some high price and forget about it for a few weeks until the market finally gets there
2) I don't want to encourage high-frequency trading. There is nothing stopping people from creating traditional website-based exchanges for people who want to do a lot of "day trading" of MSC, but I'd rather those short-term speculations not end up in the block chain for all of eternity. My goal is to make it easy on people who are doing a large purchase in or out of MSC, but not too friendly for overly-caffeinated chart monkeys.

I think the most important thing is for the UI on the websites to be really clear when you are confirmed as the first to accept an offer and it is safe to send the BTC. It would be BAD to base that advice on an ACK message that might not be included in the block chain for some reason (that would create an attack vector for fraud). The ACK message (if we create one) should only be to discourage other people from attempting to buy the same MSC.

Thanks.  Point 2 largely clears it up I think, at least for now - I've been of the mindset that it will be difficult to compete with centralized exchanges if we're substantially slower (users often sacrifice security/decentralization for speed/usability) but if day traders aren't our target audience (for now at least) then speed is much less of an issue

[zathras]

You know I may have been operating on a bad assumption re the above; being as an unconfirmed transaction propagated throughout the network it would be added to the bottom of the (currently being mined) block template, thus transactions broadcast earlier would have a better chance of being first in the block.  It sounds like you're saying this is not the case? (Is there some kind of block reorganization etc?)  As I mentioned early on I'm quite new on this cryptocurrency 'scene' and have just been teaching myself by reading as much bitcoin literature (wikis, papers, forum posts etc) as I can get my hands on - so I'm certainly more than happy to be corrected if my interpretation is wrong!

I'm not saying I'm an expert yet, so as far as I know it could be that transactions are added 'to the bottom' but I don't think that is the case. And even if it is, there is no 'one block'. Every miner or pool is working on his own version of a block based on the transactions he saw during the last block. There is no 100% propagation per transaction so a lot of time a node only learns about transactions because they have been added to the block. This being the case it does not even matter where the transaction gets added. The only importance is where it got added by the block that wins the race. I would like to learn how transactions get added though. Initial searches didn't give me the answer so I'm afraid I need to dive into the source code.

Thanks for the additional info I'm just going to throw out an idea I've been playing with to see what you guys think (feel free to tear it apart if it's not a good idea!).  What about an additional optional message.  An 'acknowledge' if you will.   MSC/BTC exchange could function as per the discussion so far but have additional support for the user that created the sell to broadcast a message to 'acknowledge' a specific accept/Purchase Offer before it's confirmed in a block.  So:

Wallet broadcasts sell offer > Wallet sees a valid accept/Purchase offer in the block template > Wallet broadcasts an 'acknowledge' including the ID of the accept/Purchase offer it's agreeing to > Trade is considered locked in & waiting to be funded > Other parties see no point trying to accept this sell now and move on to the next

All of that can happen intra-block, as long as the user (specifically their wallet) participating in trading stays online.

Of course a wallet can also simply post a sell, go offline and have it bought by the first accept/Purchase in a block as per the current spec if their is no 'acknowledge' in that block sent by the seller.

I am concerned that perhaps doing anything at all with unconfirmed messages at a fundamental level regardless of how we structure it is a bad idea because we can't be 100% sure those unconfirmed transactions will be included in the next block.  Appreciate your thoughts

I like the idea but I wouldn't want to spend time on it right now. There are so many things to build and it will be some time before we will need such a feature. It might help solve the scalability issues in the long term though.

There is just one major problem with it. Like I explained before there will never be 100% propagation, so there is a huge chance the buyer will never see the acknowledge before it ends up in a block. I really think we should try to steer away from doing messages on unconfirmed transactions.

Re 'bottom of the block' - I think that's just a colloquial term I read during research - essentially referring to the bottom of the list of transactions you'll see with getblocktemplate.

Re the acknowledge idea, yeah I'm going to put that on hold for now and just focus on doing this as defined so far.  It was really helpful to know day traders weren't our target audience right now as that means I can stop chasing speed (which I've kind of been fixating on a little if I'm honest!).

[zathras]

Hey Tachikoma - just a quick one while coding the validity rules - I notice there is no output to Exodus when you sent the bitcoins to fund your example trade - in the spec it says:

Update 9/8/2013: In order to make parsing MasterCoin transactions easier, you must also include an output to the Exodus Address when sending the bitcoins to complete a purchase of MasterCoins. The output can be for any amount, but should be above the dust threshold.

Thoughts?  Thanks!

[dacoinminster]

Hey Tachikoma - just a quick one while coding the validity rules - I notice there is no output to Exodus when you sent the bitcoins to fund your example trade - in the spec it says:

Update 9/8/2013: In order to make parsing MasterCoin transactions easier, you must also include an output to the Exodus Address when sending the bitcoins to complete a purchase of MasterCoins. The output can be for any amount, but should be above the dust threshold.

Thoughts?  Thanks!

I assumed this would make parsing easier. Please let me know if that isn't the case. My thought for this was that you could get the complete state of the MasterCoin universe by looking only at transactions which output to 1Exodus, and the rest could be safely ignored by a MasterCoin client.

[zathras]

Hey Tachikoma - just a quick one while coding the validity rules - I notice there is no output to Exodus when you sent the bitcoins to fund your example trade - in the spec it says:

Update 9/8/2013: In order to make parsing MasterCoin transactions easier, you must also include an output to the Exodus Address when sending the bitcoins to complete a purchase of MasterCoins. The output can be for any amount, but should be above the dust threshold.

Thoughts?  Thanks!

I assumed this would make parsing easier. Please let me know if that isn't the case. My thought for this was that you could get the complete state of the MasterCoin universe by looking only at transactions which output to 1Exodus, and the rest could be safely ignored by a MasterCoin client.

I'm of the same view (at the moment for example in my code logic the first thing I do is call 'ismastercointx()' which looks to see if the transaction has an output to Exodus).  It would be good to keep that logic when looking for the transaction that funded the trade.

[Tachikoma]

Wooops! My bad. I will update my verification to reflect this and reparse the transactions.

[zathras]

Are you at a point where you can confirm that my Selling/Purchase offers are valid? I don't really want to move forward until I know I'm actually creating valid messages.

Not yet, sorry (I really only have a mish-mash of half written code at the mo) - give me a day or so

Sure thing! Thanks :}

OK, I've had a bit of time to put my code in order - from a decoding and formatting perspective I think it's all to spec:

Code:

Dim testsell As mastercointx_selloffer = mlib.getmastercointransaction(bitcoin_con, "e3c07141024a392899f63cb1a4b0af3cfc1d4f1c8392f782157260a1ee7cb5c5", "selloffer")
Console.WriteLine("SELLER ADDRESS: " & testsell.fromadd & vbCrLf & "CURRENCY ID: " & testsell.curtype & vbCrLf & "SELL AMOUNT: " & testsell.saleamount & vbCrLf & "SELL PRICE (TOTAL): " & testsell.offeramount & vbCrLf & "SELL PRICE (PER MSC): " & (testsell.offeramount / testsell.saleamount).ToString("0.00######") & vbCrLf & "TIME LIMIT (BLOCKS): " & testsell.timelimit & vbCrLf & "MIN FEE REQUIRED: " & testsell.minfee)

Code:

SELLER ADDRESS: 13NRX88EZbS5q81x6XFrTECzrciPREo821
CURRENCY ID: 2
SELL AMOUNT: 1000000000
SELL PRICE (TOTAL): 10000
SELL PRICE (PER MSC): 0.00001
TIME LIMIT (BLOCKS): 10
MIN FEE REQUIRED: 1000

Code:

Dim testaccept As mastercointx_acceptoffer = mlib.getmastercointransaction(bitcoin_con, "06be47514c46b7e3afd63e98a2c850a9a346cc63f482bec9f7d90d54317b5c49", "acceptoffer")
Console.WriteLine("BUYER ADDRESS: " & testaccept.fromadd & vbCrLf & "CURRENCY ID: " & testaccept.curtype & vbCrLf & "SELLER ADDRESS: " & testaccept.toadd & vbCrLf & "BUY AMOUNT (TOTAL): " & testaccept.purchaseamount)

Code:

BUYER ADDRESS: 13shGFpdbWEjoQ2gMN9MxJY7tQ5tUDNaq7
CURRENCY ID: 2
SELLER ADDRESS: 13NRX88EZbS5q81x6XFrTECzrciPREo821
BUY AMOUNT (TOTAL): 1000000000

The selling address had sufficient funds.  The buying address sent the funds within the time limit.  So far it's all looking good - there's just the question of the missing output to exodus when funding the trade answered above

Awesome!