[ANN][XCP] Counterparty - Pioneering Peer-to-Peer Finance - Official Thread

[led_lcd]

How much of this proposal, and of other similar proposals, can be handled by the asset description field? (Not much space is needed to store a URL.).I love namespaces, etc., but I want to keep the protocol-level as simple and as stupid as possible. We could do this sort of thing is Counterwallet and BootleXCP themselves, for example.

Extremely good point and porqupine has also communicated the same to me.

We would only need a naming convention on how to format the asset description field. The rest can be up to the client side to parse it.

As long as there was some defacto which was perhaps outlined by the Counterparty team but not necessarily enforced, then everyone would be a happy camper.

Eg Lets just use the tilde as a delimiting character in the description. Field 1 = namespace, field 2 = description, field 3 = url

It needs to be a new long asset name field that is enforced by the protocol for long names.  A description will not work because users can put anything in there.

My understanding (a dev can correct me) is that the description field can be modified for an asset.

So lets just say we all agree on a defacto standard for the description field. There are 3 fields delimited by a tilde.

Each of the clients can just invalidate and fail to show any asset in their list where 3 tildes aren't found. The asset owner would have motivation to get it right.

[dexX7]

Could the info needed be compressed to fit into 40 bytes?
What sort of entropy does the typical XCP OP_RETURN data have?

Depending on the data, it might be possible to create a small customized compress/decompress function

If someone can post an example it would help me analyze this a bit more

Edit: due to the importance of this I think it would even make sense to create a custom "dictionary" of common bit patterns (hopefully there are some) in the desired OP_RETURN data. This would have to be put into the counterpartyd, but doing stuff like this might be what is needed to get 2:1 compression, then again maybe standard compression will just work, but I think some sort of custom code would have a good chance. It all depends on the type of data.

James

What do you mean by "dictionaries for common bit patterns"? To my knowledge XCP transactions use some bits to flag a data package as XCP package and some to define the transaction type, but the rest is acutal transaction field data without OP codes or whatsoever.

Code:

1c434e54525052545900000000000000000000000100000045d964b80000000000

20434e5452505254590000001e5329a91040952f00000000000000000033474f4c
20442f55534420505249434520736f757263653a207777772e7175616e646c2e63
106f6d204c6f6e646f6e20466978696e6700000000000000000000000000000000

20434e5452505254590000002800035325c3780000000002faf0800000000002fa
12f08040ae140000000000000013b0000000640000000000000000000000000000

20434e5452505254590000000a00000000000000000000000005f5e10000000000
160000000100000002540be400012c000000000000000000000000000000000000

[jl777]

Could the info needed be compressed to fit into 40 bytes?
What sort of entropy does the typical XCP OP_RETURN data have?

Depending on the data, it might be possible to create a small customized compress/decompress function

If someone can post an example it would help me analyze this a bit more

Edit: due to the importance of this I think it would even make sense to create a custom "dictionary" of common bit patterns (hopefully there are some) in the desired OP_RETURN data. This would have to be put into the counterpartyd, but doing stuff like this might be what is needed to get 2:1 compression, then again maybe standard compression will just work, but I think some sort of custom code would have a good chance. It all depends on the type of data.

James

What do you mean by "dictionaries for common bit patterns"? To my knowledge XCP transactions use some bits to flag a data package as XCP package and some to define the transaction type, but the rest is acutal transaction field data without OP codes or whatsoever.

Code:

1c434e54525052545900000000000000000000000100000045d964b80000000000

20434e5452505254590000001e5329a91040952f00000000000000000033474f4c
20442f55534420505249434520736f757263653a207777772e7175616e646c2e63
106f6d204c6f6e646f6e20466978696e6700000000000000000000000000000000

20434e5452505254590000002800035325c3780000000002faf0800000000002fa
12f08040ae140000000000000013b0000000640000000000000000000000000000

20434e5452505254590000000a00000000000000000000000005f5e10000000000
160000000100000002540be400012c000000000000000000000000000000000000

If this data is representative, 2:1 shouldnt be a problem. "00000000000000000000000" alone saves a lot of space
"20434e545250525459" appears three times, so just using those two as special words in a simple huffman tree you can squeeze this down a lot.

I have no idea what data is in the OP_RETURN, but I am assuming there are things that are globally common, like "0000..", maybe there are other globally common things. Put all those into a dictionary and based on the relative frequency assign a variable length bit pattern to the dictionary words.

However, instead of all this trouble, it seems the trick of storing data in unused multisig outputs will solve it with much less work

James

[Matt Y]

Counterwallet live on testnet

Link: https://testnet.counterwallet.co/

Report bugs here: https://forums.counterparty.co/index.php?topic=188.0
Get testnet XCP here: https://forums.counterparty.co/index.php/topic,184
Get testnet BTC here: http://tpfaucet.appspot.com/
Blog post outlining functionality: https://www.counterparty.co/counterwallet-live-testnet/

xnova has done a wonderful job developing Counterwallet and he deserves a big thanks from all of us. Let's all start testing and get Counterwallet on mainnet as soon as possible!

Thanks xnova. A truly remarkable job you've done on this.

[BitThink]

As far as I know, 80 to 40 does not change much. Previously we divide data into segments of 80 bytes, now 40 bytes. Just the number of outputs doubled. What's the big deal? Only one OP-RETURN is allowed in one transaction?

[dexX7]

If this data is representative, 2:1 shouldnt be a problem. "00000000000000000000000" alone saves a lot of space
"20434e545250525459" appears three times, so just using those two as special words in a simple huffman tree you can squeeze this down a lot.

I have no idea what data is in the OP_RETURN, but I am assuming there are things that are globally common, like "0000..", maybe there are other globally common things. Put all those into a dictionary and based on the relative frequency assign a variable length bit pattern to the dictionary words.

I see. 0x20434e545250525459 ("CNTRPRTY") is actually used to flag outputs as XCP data. There is no room for simple substitution, I think - "0000.." won't alway be "0000..", but the idea about compression is very interesting nevertheless.

[GLaDOS]

How much of this proposal, and of other similar proposals, can be handled by the asset description field? (Not much space is needed to store a URL.).I love namespaces, etc., but I want to keep the protocol-level as simple and as stupid as possible. We could do this sort of thing is Counterwallet and BootleXCP themselves, for example.

Extremely good point and porqupine has also communicated the same to me.

We would only need a naming convention on how to format the asset description field. The rest can be up to the client side to parse it.

As long as there was some defacto which was perhaps outlined by the Counterparty team but not necessarily enforced, then everyone would be a happy camper.

Eg Lets just use the tilde as a delimiting character in the description. Field 1 = namespace, field 2 = description, field 3 = url

It needs to be a new long asset name field that is enforced by the protocol for long names.  A description will not work because users can put anything in there.

My understanding (a dev can correct me) is that the description field can be modified for an asset.

So lets just say we all agree on a defacto standard for the description field. There are 3 fields delimited by a tilde.

Each of the clients can just invalidate and fail to show any asset in their list where 3 tildes aren't found. The asset owner would have motivation to get it right.

Long asset names should not be changeable just like short names.   The long names are enforced at the time of issuance to be valid only if the issuer also owns the top level asset. 

Afterward everything should go through the protocol using only the short names.

[led_lcd]

How much of this proposal, and of other similar proposals, can be handled by the asset description field? (Not much space is needed to store a URL.).I love namespaces, etc., but I want to keep the protocol-level as simple and as stupid as possible. We could do this sort of thing is Counterwallet and BootleXCP themselves, for example.

Extremely good point and porqupine has also communicated the same to me.

We would only need a naming convention on how to format the asset description field. The rest can be up to the client side to parse it.

As long as there was some defacto which was perhaps outlined by the Counterparty team but not necessarily enforced, then everyone would be a happy camper.

Eg Lets just use the tilde as a delimiting character in the description. Field 1 = namespace, field 2 = description, field 3 = url

It needs to be a new long asset name field that is enforced by the protocol for long names.  A description will not work because users can put anything in there.

My understanding (a dev can correct me) is that the description field can be modified for an asset.

So lets just say we all agree on a defacto standard for the description field. There are 3 fields delimited by a tilde.

Each of the clients can just invalidate and fail to show any asset in their list where 3 tildes aren't found. The asset owner would have motivation to get it right.

Long asset names should not be changeable just like short names.   The long names are enforced at the time of issuance to be valid only if the issuer also owns the top level asset. 

Afterward everything should go through the protocol using only the short names.

To clarify, in my proposal the long name wouldn't be changeable.

In the leave things mostly as they are scenerio, which is what I was describing above then the 'name space' would be able to change since it is just part of the asset description. The asset name is still the primary key and has to stay the same.

I can see the disadvantage of this where people can issue spam assets to 'typosquat' or even create duplicates of the same namespace. If it's not enforced by the protocol as you said, it's going to cause confusion.

[dexX7]

As far as I know, 80 to 40 does not change much. Previously we divide data into segments of 80 bytes, now 40 bytes. Just the number of outputs doubled. What's the big deal? Only one OP-RETURN is allowed in one transaction?

Only one allowed.

[xnova]

Counterwallet live on testnet

Link: https://testnet.counterwallet.co/

Report bugs here: https://forums.counterparty.co/index.php?topic=188.0
Get testnet XCP here: https://forums.counterparty.co/index.php/topic,184
Get testnet BTC here: http://tpfaucet.appspot.com/
Blog post outlining functionality: https://www.counterparty.co/counterwallet-live-testnet/

xnova has done a wonderful job developing Counterwallet and he deserves a big thanks from all of us. Let's all start testing and get Counterwallet on mainnet as soon as possible!

Thanks xnova. A truly remarkable job you've done on this.

Thanks guys. There are no doubt still a number of bugs to work through to get the wallet to a mainnet release, but this is a big accomplishment for the team and we're looking forward to bringing the capabilities of Counterparty to a larger user population. We're targeting a mainnet release within the first week of April. Of course this can change depending on how the testnet testing goes, but I'm thinking we should be able to keep that deadline.

[porqupine]

To clarify, in my proposal the long name wouldn't be changeable.

In the leave things mostly as they are scenerio, which is what I was describing above then the 'name space' would be able to change since it is just part of the asset description. The asset name is still the primary key and has to stay the same.

I can see the disadvantage of this where people can issue spam assets to 'typosquat' or even create duplicates of the same namespace. If it's not enforced by the protocol as you said, it's going to cause confusion.

The idea is that it is possible to implement name and category functionality without altering the protocol by making some filtration standards for indexing Assets, so an asset may be DKGCEWU (some unique 8 letter base_58 asset code or 'short name') ~ Gold Troy Ounce ~ ~ (issuer) ~ ~ (comment /pgp link) ~
of course one has to trust the issuer and check the link for the valid pgp signature, but once one has done this all the problems are gui / client problems, you could exclude the ability for someone else to create ~ Gold Troy Ounce ~ on the protocol level but then you are back to the original asset name problem. Instead you link the ~ Gold Troy Ounce ~ to the DKGCEWU that identifies the asset. Sure someone else can make a ~ Gold Troy Ounce ~ but it can easily be distinguished by the 'short name' as different.

[jgarzik]

To date, I've not seen a blockchain data dumping scheme that could not be securely replaced with a simple hash.

You don't need to store data in the blockchain.  That is purely intellectual laziness.  Timestamping hash(data) is just as secure, while more efficient.

Furthermore, a secondary chain can be provably pegged to bitcoin: http://sourceforge.net/p/bitcoin/mailman/message/32108143/

[reader31]

Counterwallet live on testnet

Link: https://testnet.counterwallet.co/

Report bugs here: https://forums.counterparty.co/index.php?topic=188.0
Get testnet XCP here: https://forums.counterparty.co/index.php/topic,184
Get testnet BTC here: http://tpfaucet.appspot.com/
Blog post outlining functionality: https://www.counterparty.co/counterwallet-live-testnet/

xnova has done a wonderful job developing Counterwallet and he deserves a big thanks from all of us. Let's all start testing and get Counterwallet on mainnet as soon as possible!

Thanks xnova. A truly remarkable job you've done on this.

Thanks guys. There are no doubt still a number of bugs to work through to get the wallet to a mainnet release, but this is a big accomplishment for the team and we're looking forward to bringing the capabilities of Counterparty to a larger user population. We're targeting a mainnet release within the first week of April. Of course this can change depending on how the testnet testing goes, but I'm thinking we should be able to keep that deadline.

Awesome Job guys...will give it a whirl...

[BitThink]

As far as I know, 80 to 40 does not change much. Previously we divide data into segments of 80 bytes, now 40 bytes. Just the number of outputs doubled. What's the big deal? Only one OP-RETURN is allowed in one transaction?

Only one allowed.

Oops. Then we have to keep using multisig for long data.
do we know how many percent of current transactions can be squeezed into 40 bytes now?
Could we use asset id instead of using the name everywhere?

[BitThink]

Tried the wallet.

1: "No counterparty servers are currently available. Please try again later. ERROR: undefined"

2: Second, the webpage cannot be scrolled or zoomed in both Safari and Chrome. This is not friendly to small screens especially for mobile phones.

3: In the page for enter words with screen keyboard, the words entered are not masked. Moreover, sometimes the keyboard does not work at all and has to press the screen keyboards and the response is pretty slow. (Chrome) Screen keyboard is really really painful and I doubt anyone will like to use it.

Thanks for the hard working and wish my comment helps.

[LightedLamp]

To date, I've not seen a blockchain data dumping scheme that could not be securely replaced with a simple hash.

You don't need to store data in the blockchain.  That is purely intellectual laziness.  Timestamping hash(data) is just as secure, while more efficient.

Furthermore, a secondary chain can be provably pegged to bitcoin: http://sourceforge.net/p/bitcoin/mailman/message/32108143/

While you are here sir, interested to know why the QT client still has bugs - I never got a response on the newbie forum: https://bitcointalk.org/index.php?topic=518482

[BitcoinTangibleTrust]

To date, I've not seen a blockchain data dumping scheme that could not be securely replaced with a simple hash.

You don't need to store data in the blockchain.  That is purely intellectual laziness.  Timestamping hash(data) is just as secure, while more efficient.

Furthermore, a secondary chain can be provably pegged to bitcoin: http://sourceforge.net/p/bitcoin/mailman/message/32108143/

Greetings Jeff. Thank you for stopping by. I was chatting with Eric and Stephen tonight at Bitpay's new offices about this topic and we had agreed to share an email to explore further, but you beat me to it.

You're absolutely right.
You don't need to store data in the blockchain.
Timestamping hash(data) is just as secure, while more efficient.
A secondary chain can be provably pegged to bitcoin.

However, Counterparty IS storing data in the blockchain using 256 bytes in each one-of-three multi-sig transactions, as per PhantomPhreak's note below. Additionally, all these multisig transactions are processed by the miners.

On the plus side, we can fit a lot more than 80 bytes into 256 one-of-three multi-sig outputs, and now we have no incentive to use less than that except lower fees.

If OP_RETURN was meant to stop/curtail the multisig behavior (Unspent Outputs) and hereby reduce blockchain bloat, then I fear by reducing the size of OP_RETURN from 80 to 40 bytes, you've inadvertently made multisig MORE ATTRACTIVE to all the metaprotocols and you've made OP_RETURN less attractive. The cost of paying the miners fees has not reduced the ability to survive with multisig.

The technical details are well beyond my abilities, but PhantomPhreak's assertion that the economics of time/effort of squeezing into 40bytes should be noted. Here are Phantom's words:

It would take some serious shoe-horning to fit all of the data we need into 40 bytes reliably, and in three months they could lower it to 35 bytes (again, for no reason) and then we'd be screwed.

We (All the metaprotocols) WANT to use OP_RETURN as it is in everyone's interests to do so, but if you hobble the feature's functionality such that the costs to use OP_RETURN are too high to consider, then markets will find other, less costly alternatives to achieve its goals.

We have a mutual interest in reducing blockchain bloat. Is there any reason why 80 bytes will reduce both our abilities to reach that shared objective and is there any reason 40 bytes will make it easier?

Best,
BTT

EDIT & NOTE: Jeff, we're totally okay to take this conversation to the bitcoin core dev mailing list if that's a more appropriate location for this discussion. I will cross-post my follow-up here on the bitcoin core dev mailing list as a new topic.
Thanks!
BTT

[PhantomPhreak]

To date, I've not seen a blockchain data dumping scheme that could not be securely replaced with a simple hash.

You don't need to store data in the blockchain.  That is purely intellectual laziness.  Timestamping hash(data) is just as secure, while more efficient.

Furthermore, a secondary chain can be provably pegged to bitcoin: http://sourceforge.net/p/bitcoin/mailman/message/32108143/

Greetings Jeff. Thank you for stopping by. I was chatting with Eric and Stephen tonight at Bitpay's new offices about this topic and we had agreed to share an email to explore further, but you beat me to it.

You're absolutely right.
You don't need to store data in the blockchain.
Timestamping hash(data) is just as secure, while more efficient.
A secondary chain can be provably pegged to bitcoin.

However, Counterparty IS storing data in the blockchain using 256 bytes in each one-of-three multi-sig transactions, as per PhantomPhreak's note below. Additionally, all these multisig transactions are processed by the miners.

On the plus side, we can fit a lot more than 80 bytes into 256 one-of-three multi-sig outputs, and now we have no incentive to use less than that except lower fees.

If OP_RETURN was meant to stop/curtail the multisig behavior (Unspent Outputs) and hereby reduce blockchain bloat, then I fear by reducing the size of OP_RETURN from 80 to 40 bytes, you've inadvertently made multisig MORE ATTRACTIVE to all the metaprotocols and you've made OP_RETURN less attractive. The cost of paying the miners fees has not reduced the ability to survive with multisig.

The technical details are well beyond my abilities, but PhantomPhreak's assertion that the economics of time/effort of squeezing into 40bytes should be noted. Here are Phantom's words:

It would take some serious shoe-horning to fit all of the data we need into 40 bytes reliably, and in three months they could lower it to 35 bytes (again, for no reason) and then we'd be screwed.

We (All the metaprotocols) WANT to use OP_RETURN as it is in everyone's interests to do so, but if you hobble the feature's functionality such that the costs to use OP_RETURN are too high to consider, then markets will find other, less costly alternatives to achieve its goals.

We have a mutual interest in reducing blockchain bloat. Is there any reason why 80 bytes will reduce both our abilities to reach that shared objective and is there any reason 40 bytes will make it easier?

Best,
BTT

EDIT & NOTE: Jeff, we're totally okay to take this conversation to the bitcoin core dev mailing list if that's a more appropriate location for this discussion. I will cross-post my follow-up here on the bitcoin core dev mailing list as a new topic.
Thanks!
BTT

I think you misunderstood what jgarzik is saying. The idea is that we store the data in a second blockchain, and put hashes of that timestamped data in Bitcoin, which hashes would also be less than 40 bytes. The reason we did not do something like that is not a matter of "intellectual laziness", but rather of implementation complexity. Counterparty is not a project in computer science; it is designed to be a simple as possible, for the benefits in development speed. Even if we have to store our data in multi-sig outputs rather than the too-small OP_RETURN outputs. Worse is definitely better in this space.

In any case, we appreciate the suggestion, jgarzik, and if you think we're still missing anything here, specifically w.r.t. the space of technical possibilities, please let us know. Thanks.

[xnova]

I think you misunderstood what jgarzik is saying. The idea is that we store the data in a second blockchain, and put hashes of that timestamped data in Bitcoin, which hashes would also be less than 40 bytes. The reason we did not do something like that is not a matter of "intellectual laziness", but rather of implementation complexity. Counterparty is not a project in computer science; it is designed to be a simple as possible, for the benefits in development speed. Even if we have to store our data in multi-sig outputs rather than the too-small OP_RETURN outputs. Worse is definitely better in this space.

In any case, we appreciate the suggestion, jgarzik, and if you think we're still missing anything here, specifically w.r.t. the space of technical possibilities, please let us know. Thanks.

Also, with the second blockchain, you have a second blockchain to secure, and due to this, Counterparty could not fully leverage the unparalleled PoW security that Bitcoin offers end-to-end. Moreover, unless I am overlooking something here, we would still need to parse out the data from the blocks in that second blockchain (assuming it's a Bitcoin or Bitcoin derivative implementation, at least) to get our stored data. So:
* it would not enable SPV-type Counterparty clients for what Counterparty offers over Colored Coin functionality (i.e. DEx, Betting, asset callbacks, dividends, CFDs, etc)
* it would lessen security of Counterparty transactions
* it would greatly increase the complexity of the implementation (i.e. increasing the chance of bugs and vulnerabilities)

The only dubious benefit would be the *slight* lessening our storage requirements on the block chain (i.e. maybe 20-40 bytes less per transaction?). I just don't see how this would make any sense here.

One more point: Counterparty can bring immense benefits to Bitcoin, and this will especially become more apparent if/as Ethereum (and other similar non-Bitcoin-meta "2.0" type coins) enter the picture. My personal feeling here at least is that Bitcoin may very well need offerings in the ecosystem with this kind of functionality to effectively compete with Ethereum and (future) crowd's feature list and draws -- or risk becoming obsoleted, at least among investors and financial market operators, which offer the ability to bring billions and even trillions into the Bitcoin ecosystem as it gains more recognition, trust, and mindshare with them. This process is, of course, only beginning, but we feel that the clear benefits that the blockchain can offer to finance (for not only Bitcoin payment operations, but for the more advanced finance operations that the technology can enable) will set off the next great phase of Bitcoin's growth, IF allowed to unfold organically.

[BitcoinTangibleTrust]

I think you misunderstood what jgarzik is saying. The idea is that we store the data in a second blockchain, and put hashes of that timestamped data in Bitcoin, which hashes would also be less than 40 bytes. The reason we did not do something like that is not a matter of "intellectual laziness", but rather of implementation complexity. Counterparty is not a project in computer science; it is designed to be a simple as possible, for the benefits in development speed. Even if we have to store our data in multi-sig outputs rather than the too-small OP_RETURN outputs. Worse is definitely better in this space.

In any case, we appreciate the suggestion, jgarzik, and if you think we're still missing anything here, specifically w.r.t. the space of technical possibilities, please let us know. Thanks.

Thanks for the correction, Phantom. I definitely stepped out of my comfort zone today and was promptly schooled. This stuff is hard.