[ANN] Reality Keys: An oracle letting you use external state in transactions

[edmundedgar]

I'm pleased to announce the release of Reality Keys. We provide information about things happening in the world - everything from exchange rates to election results, drawn from services with public APIs - in a form that can be understood by miners and used to confirm Bitcoin transactions. Our service is a little bit different from the  "external state oracle" Mike Hearn discusses here:
https://en.bitcoin.it/wiki/Contracts
...but we hope people will find it useful for some of the use-cases he describes.

Specifically, you can register an event - "1 USD will be worth more than 1 Euro on Feb 24th", or "Edward Snowden will win the Nobel Peace Prize by the end of 2015" - and we give you a pair of public keys, one representing the event happening and another representing it not happening. When that date comes around, we'll query the relevant API to find out whether the thing happened or not and release the private key for the public key that corresponds to the outcome. You can then use those keys in multi-sig crypto-currency transactions in the same way you might use a normal signature.

Currently our data comes from Coindesk (Bitcoin exchange rates vs USD), the ECB (exchange rates of legacy currencies), blockchain.info (Bitcoin address balances) and Freebase (everything in the known universe). Let us know if anyone has anything else they'd like us to monitor; In principle we can handle any API you can point us to as long as it has reasonably permissive Terms of Service and some other feasible way to double-check the information we get from the it in the event that it misbehaves. Please be patient with us if we run into performance issues over the next week or two, or if you manage to turn up some patterns of information in Freebase that we hadn't bargained for.

Our service can be used free of charge, to the extent that you are happy to abide by the automated results we pull from our data sources. However, we also allow you to pay a fee, currently 10 mBTC, to have a human check what happened, in the event that you think the API-generated result was wrong. For exchange rates etc we expect that this facility will hardly ever be used, but it may be useful for if you've made - say - a high-rolling bet on an election result, and the loser tries to rip you off by tinkering with one of the community-curated databases that feed into Freebase.

Note that although we think this will be useful for people making all kinds of Bitcoin contracts, including some of the scenarios described in this CoinDesk piece, we aren't going to do anything to help you find someone to make a contract with or actually create the transaction. Obviously our service isn't much use without being able to do these things one way or another, but software for brokering advanced transactions and (potentially p2p) order books are big projects in themselves, which we hope to leave to the people already working on them, who are generally much smarter than us. We've provided a simple, authentication-free API that you can use to register facts, grab the resulting keys and check on their status. Please don't hesitate to get in touch if there's anything we can do to make our service more useful to you.

Also note that, since we only issue keys to say what happened in the world rather than actually taking part in contracts, we're not in a position to confirm the legality of using our keys to create any particular contract in your particular jurisdiction. Please make sure you check these things yourself and abide by your local laws and regulations.

Finally, special thanks to a couple of people on these forums: Mike Hearn, who has long been blowing everybody's minds on all things contract-related, and Peter Todd, who seems to come out with at least one important bitcoin-related insight every weekend, and two on bank holidays.

[Peter Todd]

Finally, special thanks to a couple of people on these forums: Mike Hearn, who has long been blowing everybody's minds on all things contract-related, and Peter Todd, who seems to come out with at least one important bitcoin-related insight every weekend, and two on bank holidays.

Ha, glad to help!

You know, I'm really excited to see this service pop up - I was expecting to see the software to actually use this stuff get developed first and see people struggle to actually get the important oracle infrastructure developed, but you've done the exact opposite. I will say, don't be surprised if adoption is slow at first, the infrastructure to actually use the oracle for real transactions isn't really there yet. But hang in there and give it time for use-cases to develop; I'm sure they will in the long run.

[HairyMaclairy]

Very interesting.  Even better if you are able to plug into more authorative sources such as Reuters monitor systems.  The quality of the data is obviously very important.

[darkmatter7]

Edmund, you have no idea how excited I am about your project. 

I've been working on a project for the last month and a half which uses oracles extensively.  Reality Keys may rapidly accelerate development of my project!

[edmundedgar]

I will say, don't be surprised if adoption is slow at first, the infrastructure to actually use the oracle for real transactions isn't really there yet. But hang in there and give it time for use-cases to develop; I'm sure they will in the long run.

Right, that's my assumption too. Like I told the Coindesk guys, it's a chicken-and-egg problem where we need a few different parts, and none of them are much use without the others. Anyhow we have to start somewhere, so here's a chicken. It won't cost much to run and we're not depending on it to pay next month's rent, so we're happy to keep it running for as long as it takes for the ecosystem to develop.

Anyhow I was a little bit taken aback by the amount of attention the CoinDesk piece got last week. It really does look like there's a lot of interest, and all kinds of potential if we can bring all the pieces together.

[edmundedgar]

Edmund, you have no idea how excited I am about your project.  

I've been working on a project for the last month and a half which uses oracles extensively.  Reality Keys may rapidly accelerate development of my project!

Great to hear, let us know if there's anything we can do to help.

[edmundedgar]

Very interesting.  Even better if you are able to plug into more authorative sources such as Reuters monitor systems.  The quality of the data is obviously very important.

Thanks, I'll take a look at the feeds Reuters can provide and see if there's anything we can use. Some of the commercial feeds provide very good quality data, but I'm a little bit nervous about tripping over something in their Terms of Service, or them stumbling on some of the more mind-blowing implications of what we're doing here and suddenly terminating our access, which could suddenly leave people using keys based on that data SOL.

[mmeijeri]

Great stuff! I wonder if you guys could use a blockchain-based trusted timestamping service like virtual-notary.org, proofofexistence.com etc to keep a record of events that you've registered, together with the public keys for the two possible outcomes. Once you've published the correct private key for an event, you could have that fact notarised too. In this way you could build up a reliability record that people can check.

[mmeijeri]

Could you explain why you've chosen to publish keys rather than using Mike Hearn's algorithm? I have to admit I find the writeup on the Bitcoin wiki confusing, but maybe you could shed some light on it.

One advantage your approach appears to have is that you are not signing a transaction, in fact you need not even know what transaction it might unlock if it used P2SH or hasn't been published. This could limit temptations or threats you might be exposed to, including legal liability.

A potential complication is that people could send money to the published public keys. What would you do with such funds?

[edmundedgar]

Great stuff! I wonder if you guys could use a blockchain-based trusted timestamping service like virtual-notary.org, proofofexistence.com etc to keep a record of events that you've registered, together with the public keys for the two possible outcomes. Once you've published the correct private key for an event, you could have that fact notarised too. In this way you could build up a reliability record that people can check.

I like the sound of that, I'll give it some thought.

IIUC there's another way we could be approaching this, which would be that rather using individual random key-pairs as we currently do, we'd use key-pairs derived deterministically from a single key-pair, combined with a hash of the event we've registered. If I've got this right, thanks to the magic of ECC multiplication, people would then be able to independently derive the the public keys used in each case, without being able to derive the private keys. That way we can almost get rid of the concept of "registering" an event altogether; If you've got a record of an event (allowing us to recreate the hash) and our corresponding public keys, you can bring them to us at the due date and we can derive the private keys again to release one. People could actually create events independently, without even using our API.

We didn't spend any time looking into doing things this way because I wasn't quite confident enough that we wouldn't screw it up in some subtle way that somebody would discover halfway through 2015 after people had used these keys for a bunch of contracts. But if it can really be done as I've described it the approach would head off all kinds of nasty potential problems, including our losing data after you register a fact, and somebody hacking our public web server and managing to monkey around with the registered events or public key assignations without us detecting them.

[edmundedgar]

Could you explain why you've chosen to publish keys rather than using Mike Hearn's algorithm? I have to admit I find the writeup on the Bitcoin wiki confusing, but maybe you could shed some light on it.

One advantage your approach appears to have is that you are not signing a transaction, in fact you need not even know what transaction it might unlock if it used P2SH or hasn't been published. This could limit temptations or threats you might be exposed to, including legal liability.

As you'd guessed, we want to stay as far away from any actual contract as possible for the reasons you've given. This approach is also nice and simple and transparent. Everybody can figure out what we're doing, all our events are public, and the results are public too, so everybody can see how we settle them.

Also, although we've been talking about this as a bitcoin service, it's not necessarily bitcoin-specific, or even crypto-currency specific, or particular to the use-cases that have been discussed to date. Our keys could be used for encrypting messages (I think they're really designed for signing, but apparently it's possible to encrypt with them - if people want to do this and need another type of key we can talk) to be released in the event that something happens. For example, "Barack Obama has hidden a fish in a big block of ice somewhere under the floorboards in the Oval Office. He's written a message that will be released if the Democrats win the next election, so they can find it and throw it away before it starts to rot. But if the Republicans win nobody will know about it and they'll be going nuts trying to figure out where that smell is coming from."

A potential complication is that people could send money to the published public keys. What would you do with such funds?

We'd consider it a donation.

[Mike Hearn]

Congrats on the launch! It's looking really great - using Freebase is a fascinating idea.

Probably the nearest term useful app for this would be some kind of currency hedging system. But as you note, it's quite complicated.

[mmeijeri]

We'd consider it a donation.

And of course, it wouldn't bias your decision either way, since you could spend outputs that had been donated to both addresses before destroying the key for the eventuality that didn't happen.

[Peter Todd]

IIUC there's another way we could be approaching this, which would be that rather using individual random key-pairs as we currently do, we'd use key-pairs derived deterministically from a single key-pair, combined with a hash of the event we've registered. If I've got this right, thanks to the magic of ECC multiplication, people would then be able to independently derive the the public keys used in each case, without being able to derive the private keys. That way we can almost get rid of the concept of "registering" an event altogether; If you've got a record of an event (allowing us to recreate the hash) and our corresponding public keys, you can bring them to us at the due date and we can derive the private keys again to release one. People could actually create events independently, without even using our API.

We didn't spend any time looking into doing things this way because I wasn't quite confident enough that we wouldn't screw it up in some subtle way that somebody would discover halfway through 2015 after people had used these keys for a bunch of contracts. But if it can really be done as I've described it the approach would head off all kinds of nasty potential problems, including our losing data after you register a fact, and somebody hacking our public web server and managing to monkey around with the registered events or public key assignations without us detecting them.

You're caution was quite justified in this case! Unfortunately the way ECC math works if you release the secret key for one of these ECC-multiplication-derived pubkeys - as you must in your application - the master secret key can be recovered with some arithmetic if the pubkey key is known - again a must for the public derivation to be useful. Unfortunately there is no known way around this problem with ECC math, and the Bitcoin scripting language is too primitive to check an arbitrary signed message instead. I actually had the same idea when I originally thought of the pubkey derivation, and mentally filed it under "yet another example of BIP32 pubkey derivation is subtly dangerous"

Having said that, if you accept the need for the user to contact you first you can still do this deterministically by just hashing some master seed with the thing the user is asking and using that hash digest to derive a secret key and finally public key. However that version requires you to keep the master secret online on a potentially hackable server - I'd recommend adding at least one more layer of indirection to that system to derive a per-day secret and transferring that secret to the server in question somehow. You probably also want to separate the server holding the secret from the webserver - make the former communicate with the latter (and outside world) by some tightly controlled communication channel and no other means of access.

Finally one curious property of the above is that you can interactively commit to releasing a secret key for a question unknown to you. e.g. you can provide the corresponding pubkey for H(question) without knowing the question itself until the time comes to reveal it. I'm not sure this is necessarily a good thing!

Our keys could be used for encrypting messages (I think they're really designed for signing, but apparently it's possible to encrypt with them - if people want to do this and need another type of key we can talk)

Elliptic Curve Diffie-Hellman key agreement is how that encryption works. Bitmessage implements it here

A potential complication is that people could send money to the published public keys. What would you do with such funds?

We'd consider it a donation.

Note how this is actually an advantage of using revealed secret keys over my earlier proposal of releasing hash pre-images: you can spend such a donation without revealing the secret key.

[edmundedgar]

IIUC there's another way we could be approaching this, which would be that rather using individual random key-pairs as we currently do, we'd use key-pairs derived deterministically from a single key-pair, combined with a hash of the event we've registered. If I've got this right, thanks to the magic of ECC multiplication, people would then be able to independently derive the the public keys used in each case, without being able to derive the private keys. That way we can almost get rid of the concept of "registering" an event altogether; If you've got a record of an event (allowing us to recreate the hash) and our corresponding public keys, you can bring them to us at the due date and we can derive the private keys again to release one. People could actually create events independently, without even using our API.

We didn't spend any time looking into doing things this way because I wasn't quite confident enough that we wouldn't screw it up in some subtle way that somebody would discover halfway through 2015 after people had used these keys for a bunch of contracts. But if it can really be done as I've described it the approach would head off all kinds of nasty potential problems, including our losing data after you register a fact, and somebody hacking our public web server and managing to monkey around with the registered events or public key assignations without us detecting them.

You're caution was quite justified in this case! Unfortunately the way ECC math works if you release the secret key for one of these ECC-multiplication-derived pubkeys - as you must in your application - the master secret key can be recovered with some arithmetic if the pubkey key is known - again a must for the public derivation to be useful. Unfortunately there is no known way around this problem with ECC math, and the Bitcoin scripting language is too primitive to check an arbitrary signed message instead. I actually had the same idea when I originally thought of the pubkey derivation, and mentally filed it under "yet another example of BIP32 pubkey derivation is subtly dangerous"

Ah thanks, that makes sense.

Having said that, if you accept the need for the user to contact you first you can still do this deterministically by just hashing some master seed with the thing the user is asking and using that hash digest to derive a secret key and finally public key. However that version requires you to keep the master secret online on a potentially hackable server - I'd recommend adding at least one more layer of indirection to that system to derive a per-day secret and transferring that secret to the server in question somehow. You probably also want to separate the server holding the secret from the webserver - make the former communicate with the latter (and outside world) by some tightly controlled communication channel and no other means of access.

Finally one curious property of the above is that you can interactively commit to releasing a secret key for a question unknown to you. e.g. you can provide the corresponding pubkey for H(question) without knowing the question itself until the time comes to reveal it. I'm not sure this is necessarily a good thing!

OK, it sounds like for our current purposes we're best sticking with the current thing (a gazillion random key-pairs pre-generated by a plain old vanilla bitcoind with only the public keys accessible to the public web server until an event is settled) and building a bit more verifiability on top. It shouldn't be an impossible problem given that all the data we're talking about is public. We're currently logging all registered events and assigned public keys to an external syslog service, and it should be fairly straightforward to add blockchain notarization or some other form of integrity checking on top of that.

Finally one curious property of the above is that you can interactively commit to releasing a secret key for a question unknown to you. e.g. you can provide the corresponding pubkey for H(question) without knowing the question itself until the time comes to reveal it. I'm not sure this is necessarily a good thing!

Even on the current scheme I guess it would be possible for us to release keys for an event without knowing the contents unless we needed to adjudicate it; If you gave us a regular hash of the event you want keys for rather than the content, you wouldn't have to give us the content until such time as you needed adjudication. Not a feature I'm inclined to offer at the moment, though.

Note how this is actually an advantage of using revealed secret keys over my earlier proposal of releasing hash pre-images: you can spend such a donation without revealing the secret key.

Right, and I suspect there are lots of other little benefits to this approach that we haven't thought of yet.

[mmeijeri]

Note how this is actually an advantage of using revealed secret keys over my earlier proposal of releasing hash pre-images: you can spend such a donation without revealing the secret key.

Then again, if you used a hash, there wouldn't be any addresses to spend to. One advantage I can see is that doing it with an address rather than a hash is that you don't run afoul of IsStandard. Any other pros or cons?

And speaking of IsStandard, what are the prospects of less restrictive rules any time soon, perhaps only for P2SH scripts?

[Peter Todd]

Note how this is actually an advantage of using revealed secret keys over my earlier proposal of releasing hash pre-images: you can spend such a donation without revealing the secret key.

Then again, if you used a hash, there wouldn't be any addresses to spend to. One advantage I can see is that doing it with an address rather than a hash is that you don't run afoul of IsStandard. Any other pros or cons?

I can just as easily construct a spendable txout to tempt the release of a hash-preimage:\

DUP SHA256 <hash> EQUALVERIFY <reality-key's pubkey> CHECKSIG

and send Reality Keys an email informing them of their bounty, if only they'd be dishonest...

And speaking of IsStandard, what are the prospects of less restrictive rules any time soon, perhaps only for P2SH scripts?

Low is my guess among the current set of developers. But if you can come up with a good reason...

[mmeijeri]

It's also nice that your blockchain watching functionality allows us to experiment with some of the things we might want to do in a Script 2.0 without too much risk.

[edmundedgar]

It's also nice that your blockchain watching functionality allows us to experiment with some of the things we might want to do in a Script 2.0 without too much risk.

Yes, I think it could be useful in that area. We're missing a couple of features right now in that respect but they're on the todo list:

1) Watching individual transactions - currently we can tell you about total payments coming into addresses and the total balance in an address, but we don't look at specific transactions. I haven't thought much about the kind of things we'd test for, but let us know if anyone has any specific comparisons they'd like to be able to make against these.

2) Monitoring other chains. We're watching blockchain.info for Bitcoin, but we also built some functionality to watch cryptocoin explorer for alt-chains. This is turned off right now because some of their API calls seem to be misbehaving at the moment, and in any case they seem to be down a lot of the time. People who need this functionality might like to consider donating to them.

[Peter Todd]

Have you seen Bitrated yet? I'd strongly advise you to contact them and see what it'd take to add support for oracle transactions to their escrow service. You could almost even use it as-is right now, except that the system only (currently) lets you set a single pubkey for the escrow agent.