Standard HTTP Post scheme for bitcoin payment notifications

[MORA]

As discussed briefly in another bitcoinmonitor thread, some form of standard for HTTP Post notifications would be beneficial for all.

The goal with this thread, is to agree on a minimum subset of variables, and signature generation, that all services will implement, so that no matter which service you use, the basic function is the same.
This would allow shops to pick a payment module, and payment notification service(s) independently.

The standard will need to define which fields are mandatory, and which fields are used in the calculation of signature, in which sequence and how the signature is calculated.

To start off, heres my bid for mandatory fields
The names are only suggestions.

• to_address
• amount (in satoshi to avoid any radix point confusion)
• confirmations (Number of confirmations at the time of the notification, not necessary the requested amount)
• txhash (the hash of the transaction that contains the payment)
• block (height of the block that contained the tx, -1 if unconfirmed)
• signature (see below)
• service (name of the service, could be used in shops where 2 of 3 notifications are needed, before its accepted)
• IP (public ip of the service, since this is included in the signature, a replay attack has to be done from the same IP (yes it can be spoofed))

If we can agree on a hashing algorithm, thats cool.
But since we may not, and any algorithm in time could become obsolete, I think a field to specify the used algorithm is useful, also this could allow services to make a easier CRC, for the "internet of things", like xor/crc8.

• algorithm (sha1, md5, crc32, etc.)

To keep it simple to implement, we should only list the ones that are likely to be used, since any payment system will need to implement all, so if possible, we should keep to the most standard ones, that are likely to be available.
I would suggest using PHP as the reference, since its pretty popular, and contains quite a few hashing systems.

I hope we can agree on a sequence, so that the system does not need to load a list of fields to calculate the signature.

After security checks, a signature validation could be performed in PHP

Code:

if ($_SERVER['REMOTE_ADDR'] == $ip &&
$$algorithm($to_address.$amount.$confirmations.$txhash.$block.$ip.$secret) == $signature)

The service name does not need to be included in the calculation, since it would be used to lookup the secret value, so if its changed the validation will fail.

Any service can add extra fields, like btc_amount, from_addresses, datetime, etc.
Since the signature does not include those fields, the same code can still be used.
If those fields contain data, a middle-man would have interest in modifying, a second signature could be added.


This is only my suggestion, please feel free to comment and explain your view on how it can/should be done.

[1715052833]

1715052833

[1715052833]

1715052833

[Herbert]

Thanks Mora,
this is a great proposal, I fully support it!

To start off, heres my bid for mandatory fields
The names are only suggestions.

• to_address
• amount (in satoshi to avoid any radix point confusion)
• confirmations (Number of confirmations at the time of the notification, not necessary the requested amount)
• txhash (the hash of the transaction that contains the payment)
• block (height of the block that contained the tx, -1 if unconfirmed)
• signature (see below)
• service (name of the service, could be used in shops where 2 of 3 notifications are needed, before its accepted)
• IP (public ip of the service, since this is included in the signature, a replay attack has to be done from the same IP (yes it can be spoofed))

I propose an additional fied:
"timestamp" which should be the date/time this transaction was observed by the monitoring system.

I am currently working on extending notification possibilites of bitcoinmonitor.net to support outgoing payments. To support this also for http callback we should keep the wording neutral, so "to_address" would become "address" and "amount" can be positive for incoming and negative for outgoing payments.
About the block number - Is this really necessary? I am not sure if it adds significant information. If not, i would omit it just to keep the list of parameters as short as possible.

If we can agree on a hashing algorithm, thats cool.
But since we may not, and any algorithm in time could become obsolete, I think a field to specify the used algorithm is useful, also this could allow services to make a easier CRC, for the "internet of things", like xor/crc8.

• algorithm (sha1, md5, crc32, etc.)

To keep it simple to implement, we should only list the ones that are likely to be used, since any payment system will need to implement all, so if possible, we should keep to the most standard ones, that are likely to be available.
I would suggest using PHP as the reference, since its pretty popular, and contains quite a few hashing systems.

While this is a very flexible approach it might make adoption of the system a bit more complicated. I think we should be able to agree on one algorithm. TBH I am lacking background knowledge to make an educated decision - basically I don't care which one to use.
There still would be the option to put the algorithm specifier outside the common signed_data part, so special solutions would still be possible.

I hope we can agree on a sequence, so that the system does not need to load a list of fields to calculate the signature.

Very simple proposal which i chose also for bitcoinmonitor.net: Alphabetical ordering ;-)

After security checks, a signature validation could be performed in PHP

Code:

if ($_SERVER['REMOTE_ADDR'] == $ip &&
$$algorithm($to_address.$amount.$confirmations.$txhash.$block.$ip.$secret) == $signature)

The service name does not need to be included in the calculation, since it would be used to lookup the secret value, so if its changed the validation will fail.

Any service can add extra fields, like btc_amount, from_addresses, datetime, etc.
Since the signature does not include those fields, the same code can still be used.
If those fields contain data, a middle-man would have interest in modifying, a second signature could be added.


This is only my suggestion, please feel free to comment and explain your view on how it can/should be done.

Proposal how to store the data in the POST body:
Use a json string e.g.

Code:

{
    "signed_data": {
        "address": "12r9JzPNnyWs2j1s9KLW5keqBr4kbJjxz6",
        "amount": 122678000,
        "confirmations": 2,
        "timestamp": "2012-02-19 10:37:20.203541",
        "txhash": "e0c84120068bfefddab051e751f3df963c4ed29e7b13eadac026e6f17f55fb06",
        "ip": "127.0.0.1"
    },
    "signature": "38a25ec14b9ffcfe98938f3e35ac9cf41de2a971",
    "service": "<service name here>"
    <any additional information specific to service>
}

[MORA]

I propose an additional fied:
"timestamp" which should be the date/time this transaction was observed by the monitoring system.

Good idea, then the receiver can also detect delayed notifications, due to outage of sender/receiver.
I suggest using a a standard timezone so we dont have to include timezone (zulu?)

I am currently working on extending notification possibilites of bitcoinmonitor.net to support outgoing payments. To support this also for http callback we should keep the wording neutral, so "to_address" would become "address" and "amount" can be positive for incoming and negative for outgoing payments.

Agreed, maybe a type field, to designate if the address is sender or receiver ?

About the block number - Is this really necessary? I am not sure if it adds significant information. If not, i would omit it just to keep the list of parameters as short as possible.

I think some gambling sites would like it, to save an extra lookup to find the information, ie. the lotterly ended at block 27 which was the first block in Januar 1970.

While this is a very flexible approach it might make adoption of the system a bit more complicated. I think we should be able to agree on one algorithm. TBH I am lacking background knowledge to make an educated decision - basically I don't care which one to use.
There still would be the option to put the algorithm specifier outside the common signed_data part, so special solutions would still be possible.

As far as I know, it is a bit easyier to find a hash collision with MD5 than sha1, and bitcoin uses some form of sha.
But as a signature, I dont think the difference is all that big, I use sha1 today, but can switch to md5 if someone that knows the systems better, agrees that there is no security problem in it.

I hope we can agree on a sequence, so that the system does not need to load a list of fields to calculate the signature.

Very simple proposal which i chose also for bitcoinmonitor.net: Alphabetical ordering ;-)

Good idea, Agreed.

Proposal how to store the data in the POST body:
Use a json string e.g.

Code:

{
    "signed_data": {
        "address": "12r9JzPNnyWs2j1s9KLW5keqBr4kbJjxz6",
        "amount": 122678000,
        "confirmations": 2,
        "timestamp": "2012-02-19 10:37:20.203541",
        "txhash": "e0c84120068bfefddab051e751f3df963c4ed29e7b13eadac026e6f17f55fb06",
        "ip": "127.0.0.1"
    },
    "signature": "38a25ec14b9ffcfe98938f3e35ac9cf41de2a971",
    "service": "<service name here>"
    <any additional information specific to service>
}

Is there any specific reason you would rather post a json body than use the key=>value system ?
Using key=>value allows PHP scripts to access the data as $_POST['key'], instead of having to go with a json parser, and "internet of things" will have a hard time parseing json.

[Herbert]

I propose an additional fied:
"timestamp" which should be the date/time this transaction was observed by the monitoring system.

Good idea, then the receiver can also detect delayed notifications, due to outage of sender/receiver.
I suggest using a a standard timezone so we dont have to include timezone (zulu?)

Gaaah, i hate this timezone stuff. But indeed this should be un-ambigious a fixed Timezone.

I am currently working on extending notification possibilites of bitcoinmonitor.net to support outgoing payments. To support this also for http callback we should keep the wording neutral, so "to_address" would become "address" and "amount" can be positive for incoming and negative for outgoing payments.

Agreed, maybe a type field, to designate if the address is sender or receiver ?

I don't think this is necessary. It should be quite obvious: Negative amount -> Outgoing payment. Positive amount -> incoming payment.

As far as I know, it is a bit easyier to find a hash collision with MD5 than sha1, and bitcoin uses some form of sha.
But as a signature, I dont think the difference is all that big, I use sha1 today, but can switch to md5 if someone that knows the systems better, agrees that there is no security problem in it.

If sha1 is better for any reason - let's take it :-) No objection from my side.

Proposal how to store the data in the POST body:
Use a json string e.g.

Code:

{
    "signed_data": {
        "address": "12r9JzPNnyWs2j1s9KLW5keqBr4kbJjxz6",
        "amount": 122678000,
        "confirmations": 2,
        "timestamp": "2012-02-19 10:37:20.203541",
        "txhash": "e0c84120068bfefddab051e751f3df963c4ed29e7b13eadac026e6f17f55fb06",
        "ip": "127.0.0.1"
    },
    "signature": "38a25ec14b9ffcfe98938f3e35ac9cf41de2a971",
    "service": "<service name here>"
    <any additional information specific to service>
}

Is there any specific reason you would rather post a json body than use the key=>value system ?
Using key=>value allows PHP scripts to access the data as $_POST['key'], instead of having to go with a json parser, and "internet of things" will have a hard time parseing json.

It has the advantage of allowing nested data structures. So like in my example we can nicely model an array of signed_data and it would be quite obvious which parameters to include for signature checking for everyone.
With a flat key=>value structure it's not so obvious anymore and might lead to errors.
Also json knows basic datatypes (strings, numbers, ...) so there is less risk of misinterpretation/wrong conversion.

A json-parser is available in any language you can think of (see http://json.org/), and some/many popular languages have it already built-in.

[MORA]

It has the advantage of allowing nested data structures. So like in my example we can nicely model an array of signed_data and it would be quite obvious which parameters to include for signature checking for everyone.
With a flat key=>value structure it's not so obvious anymore and might lead to errors.
Also json knows basic datatypes (strings, numbers, ...) so there is less risk of misinterpretation/wrong conversion.

A json-parser is available in any language you can think of (see http://json.org/), and some/many popular languages have it already built-in.

Well, while there is a C parser avaliable, most seem aimed at full blown systems, a sensor platform may have as little as 8MHz, 16KByte flash space and 2KByte ram.

However even those devices can cope with simple json, so its not a "must not have"

[Herbert]

Well, after rethinking the JSON thing I changed my mind. I think Mora is right, we should make this as easy as possible and not require any additional libs/tools.
Also there is another point: For calculating the signature/hash you have to convert all parameters back to string anyway. So json gives you a real (big)Integer for e.g. the Satoshi amount but the first thing you need to to is convert back to string in order to check the signature.

Summary: /me votes for standard key=>value parameters.

[MORA]

Perhaps a (signed) version field would be useful, to allow for future breaking changes.

Not sure how it could be made so a gradual changeover would be possible, a version field would only allow gradual upgrade of clients, once a server is upgraded, everyone gets v2 messenges...

Also do you have an idea about how it should be implemented in existing systems, where users are already using the old system ?

[Herbert]

Perhaps a (signed) version field would be useful, to allow for future breaking changes.

+1 for the version field in the signed data.

Not sure how it could be made so a gradual changeover would be possible, a version field would only allow gradual upgrade of clients, once a server is upgraded, everyone gets v2 messenges...

Also do you have an idea about how it should be implemented in existing systems, where users are already using the old system ?

We have to support both variants, our own and the new standard. I am thinking to add a setting where user can check "old proprietary (obsolete) callback" and "callback according to <name to be found> standard". With the latter being the default for all new clients.
So existing configurations will continue to work while all new users will per default get the new standard callback.

[Herbert]

Hmm. I was thinking today if the whole thing of signing string with user-specific secret is the right thing?

What we want to achieve:
Give the user possibility to verify the provided data is authentic.

What we actually do:
Give the user possibility to verify the provided data is authentic and signed with users secret.

I think the latter part is not necessary - as long as user can make sure the data is authentic everything is fine. So probably we can simplify the setup by not needing user-specific secrets...

[MORA]

Hmm. I was thinking today if the whole thing of signing string with user-specific secret is the right thing?

What we want to achieve:
Give the user possibility to verify the provided data is authentic.

What we actually do:
Give the user possibility to verify the provided data is authentic and signed with users secret.

I think the latter part is not necessary - as long as user can make sure the data is authentic everything is fine. So probably we can simplify the setup by not needing user-specific secrets...

How do you propose to validate the content without a secret ?
-Not only the sender, but also that the contents have not been tampered with.

If the server is compromised, all bets are off, but the user secret means you need to know url, address and secret to fake a call.
If that secret is a user "defined" one, or a ssl key or something else, is maybe the same.

[Herbert]

I was thinking along the lines of GPG signatures.

Our services would need a private and public key. We create a signature on the outgoing data with our private key. User can verify the signature with our public key. (Rough idea: http://www.gnupg.org/gph/en/manual/x135.html)

Pro:
No need to manage user-specific secrets for us and for users.
No need for user to protect his secret/take actions if his secret gets compromised.

Contra:
We need to have high security in place to protect the private "signing" key.
User-side implementation probably not as easy and straightforward as it is now.

Now I really don't know if this is worth the implementation effort. But at least from security standpoint it would be better because the secret is now only needed locally on our servers and does not need to be shared with users.

Anybody know if there exists a complete API/SDK in php/python/... that can accomplish this signature validation in the same one-liner than the current approach we are using?

[MORA]

Well the "secret" vs non secret is really just symmetric or asymmetric encryption.
In symmetric the same key is used to encrypt and decrypt, in asymmetric a public key is used to decrypt data signed by a private key.

We can make asymmetric without having to go the PGP route, but I am not sure if its really needed, the users still need to protect them selves against a server compromise, by using more than 1 service, or looking up the data on a blockexplorer(of cause not everyone will actually do that).