Setting up an oracle machine on Amazon's AWS.

[dansmith]

EDIT 2 May 2015
Updated setup instructions can be found here:
https://github.com/tlsnotary/pagesigner-oracles

---------------------------------------------------------
EDIT 18 Oct 2013
Here is a link to a comprehensive explanation of what must be done to setup an oracle on AWS.
https://github.com/themighty1/ssllog/blob/development/oracle/INSTALL
-----------------------------------------------------------


I'd like to suggest for peer review a method of proving that a certain set of programs (called oracle)
1) is available on a certain IP address hosted on Amazon AWS
2) has not been and cannot be tampered with

This method relies on Amazon Query API and assumes that Amazon itself will not tamper with the oracle.
This can have many applications for the Bitcoin servces, to me this is really useful in connection with the project I'm currently working on: creating a decentralized bank-transfer-to-BTC exchange based on certain SSL logging techniques.
https://bitcointalk.org/index.php?topic=173220.0

Steps needed to setup an oracle:

1. Start an instance from an Amazon-provided AMI with EBS-backed storage and modify all the software acc.to oracle's needs. Make note of the AMI ID.

2. Create a snapshot of the EBS volume and make that snapshot public. Make note of the Snapshot ID.

3. Publically describe all the modifcations made to AMI's EBS volume, so that anyone with an AWS account could take the original AMI's EBS, perform the modifications and compare file tree hashes against the hashes of the public Snapshot.

4. Assuming ec2-api-tools installed. Start the oracle with
ec2-run-instances <AMI id> -k mykeypair -t t1.micro -z us-east-1a -b /dev/sda1=none -b /dev/sda1=<Snapshot ID>::false:standard -O <access ID> -W <secret key>

The trick here is that Amazon starts the AMI, doesn't attach its default volume but attaches a new volume which is automatically created by Amazon from the snapshot we published earlier.

5. Create GET requests using the following Amazon Query API.
   DescribeInstances - shows you launch-time, IP address, block device attach time
   DescribeVolumes - shows you snapshot ID from which the volume was created, and time of creation.
 
Now anyone can use those get requests to compare the times when the instance was started, when the volume was created and when the volume was attached. Usually volume is created and attached on the same second, and the volume gets attached to the instance 2-3 seconds after the instance is launched.


Attacks:
1. If the attacker tries to detach the volume, modify it and then attach it back, the attach time will be changed and the attack will be detected.
2. Amazon only allows instances to use Amazon pre-made kernels and init ramdisks. This rules out the possibility of tampering with the volume durng boot time.                                        
3. All Amazon root partitions are mounted to /dev/sda1. The attacker can attach a rogue volume during the boot to /dev/sda1 and the legit volume as /dev/sda2. He will then modify the legit volume and will try to unmount the rogue volume from /dev/sda1 and re-mount the legit volume to /dev/sda1. Linux doesn't allow the root partition to be unmounted on a live system. Besides, DescribeInstances API will show that the legit volume was initially attached to /dev/sda2 and not /dev/sda1 as it was supposed to.

But assuming the attacker finds a way to game the way Amazon attaches/detaches volumes, there is still another extra layer of protection:

API call GetMetrics returns a list of all volumes of the AWS account that were attached within the last two weeks. Part of the proving process will be that the oracle owner must not have any other volumes attached in his AWS account.


Are there any weaknesses which I might have overlooked?

[jgarzik]

Are there any weaknesses which I might have overlooked?

All weaknesses that are fundamental to virtual machines, versus dedicated servers.

Though Amazon is not on this list AFAIK, other VM providers have been hacked at the cloud management layer, effectively giving anyone 100% root and "physical" access.

[waxwing]

How is the snapshot ID determined? If it's just literally taking hashes of data on disk, wouldn't it vary with account name or something?

[dansmith]

How is the snapshot ID determined? If it's just literally taking hashes of data on disk, wouldn't it vary with account name or something?

When you create a snapshot of your volume, Amazon assigns it a unique-ID. It is unique across all AWS accounts, not just your account.

[jgarzik]

How is the snapshot ID determined? If it's just literally taking hashes of data on disk, wouldn't it vary with account name or something?

Snapshot ID is not hash based, but it is unique to all AWS customers.

Customer A may use a snapshot from Customer B, etc.  (assuming permissions allow that)

[waxwing]

Sorry, I misunderstood what snapshotID was. Thanks for clarification.

3. Publically describe all the modifcations made to AMI's EBS volume, so that anyone with an AWS account could take the original AMI's EBS, perform the modifications and compare file tree hashes against the hashes of the public Snapshot.

This is what I was really asking about. Will a third party be able to verify that the snapshot contains x,y software by comparing file tree hashes (e.g. won't account names on the disk be different or something like that).

[dansmith]

This is what I was really asking about. Will a third party be able to verify that the snapshot contains x,y software by comparing file tree hashes (e.g. won't account names on the disk be different or something like that).

No, they won't be different. The account name that is on the Snapshot will be the same when the verifier mounts a volume created from that snapshot. (When you launch a stock Ubuntu AMI, the default user account name is "ubuntu". It's not like Amazon creates a user account for you automatically based on your AWS account name, no.)

[Mike Hearn]

I know nothing about AWS but if this technique works, it's a really cool hack - sort of a cheaper and much easier version of trusted computing, with Amazon as the root of trust.

This could really jumpstart oracle development and many other things too.

Of course, the real trick is defending the private key. Unless your instance isn't going to run SSH or have any other way for people to reach into it from the outside, the private key can still be taken off the instance and used to sign arbitrary things, so having a reproducible machine environment doesn't necessarily help much.

You also have to be careful about timing attacks. There have been demonstrations of how to extract private keys by colocating VMs on the same host.

If you try it and it works, can you add a description of it to the Bitcoin wiki pages on oracles? I can also do it, if you prefer.

[dansmith]

You also have to be careful about timing attacks. There have been demonstrations of how to extract private keys by colocating VMs on the same host.

Could you please point me to such a demonstration.

If you try it and it works, can you add a description of it to the Bitcoin wiki pages on oracles? I can also do it, if you prefer.

Yes, I could do that, once this thread matures enough, so that I'm confident that there isn't some peculiarity of AWS which I'm overlooking that would render this hack futile.

[dansmith]

Here is a link to a comprehensive explanation of what must be done to setup an oracle on AWS.
https://github.com/themighty1/ssllog/blob/development/oracle/INSTALL

Currently, the entire AWS account must be dedicated to running the oracle. No other volumes or instances must be present (until Amazon comes up with a more fine-grained API)

[dansmith]

@Mike Hearn
You mentioned a specific attack of extractng private keys by colocating VMs on the same host.
Did that attack involve breaking out of the VM?  What if the keys were kept in the memory at all times, would that prevent this specific attack you mentioned.

[Mike Hearn]

No. Those attacks are based on monitoring the L1/L2 dcaches or timing branch mispredictions. Example:

http://err.no/tmp/448.pdf

You defend against them by using very carefully programmed crypto libraries and algorithms that are designed to run in constant time and not leak data through side channels. This is hard! Ed25519/Curve25519 are examples of libraries/algos designed to resist such attacks:

http://ed25519.cr.yp.to/

No secret array indices. The software never reads or writes data from secret addresses in RAM; the pattern of addresses is completely predictable. The software is therefore immune to cache-timing attacks, hyperthreading attacks, and other side-channel attacks that rely on leakage of addresses through the CPU cache.

No secret branch conditions. The software never performs conditional branches based on secret data; the pattern of jumps is completely predictable. The software is therefore immune to side-channel attacks that rely on leakage of information through the branch-prediction unit.

Rather than use low level primitives directly it's often a good idea to use a higher level library, such as NaCL:

http://nacl.cr.yp.to/features.html

NaCL provides high level features like "sign this message" or "encrypt this message". It lets you pick your own ciphers and so on, but by default uses schemes that are designed to resist side channel extraction.

Don't get me wrong, side channel attacks are sophisticated and hard. For prototyping, you wouldn't need to worry about this. For actual production usage where the private key becomes valuable, you'd want to ensure you're using the absolute most modern crypto possible and with tools like NaCL it's easy.

[kactech]

Good project, I had related idea some time ago but sadly no one replied https://bitcointalk.org/index.php?topic=261631
Watching this one

[dansmith]

I updated the OP to include the link to the new setup instruction:
https://github.com/tlsnotary/pagesigner-oracles

PageSigner now uses AWS oracles to enable transferable proofs of https:// pages.
https://tlsnotary.org/pagesigner.html