Title: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 03, 2011, 12:49:34 PM This would solve a bunch of problems.
Why not just return the int64 and let the client cast it to a float & divide 10^8 for display? Currently since the JSON RPC returns floats, any library you use will return the values as floats, not strings. So to get the int64 value you need to multiply the float by 10^8 and cast to an int for internal usage. Also by returning the value as int64, it will be enforcing good practice on clients instead of them unwittingly using floats. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 03, 2011, 01:15:26 PM Because JavaScript doesn't have a 64-bit integer type (all Numbers in JavaScript are double-precision floating point).
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ribuck on March 03, 2011, 01:23:16 PM all Numbers in JavaScript are double-precision floating point Fortunately for Bitcoin, double-precision floating point represents integers exactly up to 9,007,199,254,740,992 which is above the number of bitcoin base units i.e. 2,100,000,000,000,000.Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 03, 2011, 01:33:48 PM Can we not beat this dead horse?
I think there are MUCH more important things to work on / worry about than whether or not "send 1 BTC" is expressed as "sendtoaddress FOO 1.00" or "sendtoaddress FOO 100000000" in the JSON-RPC. How about we (I'll start) write a "Proper Money Handling" page for the Wiki that discusses the issue and gives code example of how to convert to/from JSON double-precision floating point and 64-bit integer? Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 03, 2011, 02:17:12 PM Python uses floats for it's JSON library. Herein lies the problems.
$ python >>> import json >>> json.dumps(10.001) '10.000999999999999' >>> json.loads('{"blaa": 0.333331}') {u'blaa': 0.33333099999999999} >>> type(json.loads('{"blaa": 0.333331}')['blaa']) <type 'float'> This is unacceptable. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 03, 2011, 02:33:21 PM Wiki page created: https://en.bitcoin.it/wiki/Proper_Money_Handling_(JSON-RPC)
genjix: You should be calling json.loads(..., parse_float=decimal.Decimal) and use a custom JSON encoder class to convert decimals to JSON strings with no loss of precision... Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 03, 2011, 02:43:17 PM Yep, just noticed that.
But then the Python JSON-RPC library does not and I couldn't find one for the PHP RPC library either. Code: >>> from jsonrpc import ServiceProxy BTW, in that wiki page why did you use those lambdas instead of simply using decimal.Decimal? Multiplying by e8 will cause everything like version numbers or difficulty to be multiplied. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Luke-Jr on March 03, 2011, 02:54:55 PM Currently since the JSON RPC returns floats, any library you use will return the values as floats, not strings. So to get the int64 value you need to multiply the float by 10^8 and cast to an int for internal usage. It's one of the many JSON-RPC design flaws. Instead of trying to fix it, I have moved on to working on a new protocol to address all the problems: https://www.bitcoin.org/smf/index.php?topic=3757.0Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 03, 2011, 03:26:16 PM genjix: here is how to do it right in Python2.6 :
Code:
Code: 10.001 Note that EVEN IF YOU PASSED THE 'WRONG' strings to Bitcoin, Bitcoin would do the right thing. That is, these two are equivalent once they are parsed by bitcoin: Code: sendtoaddress FOO 10.000999999999999 On the bitcoin side, this is a non-issue. And if code on the other end of the JSON-RPC connection does the wrong thing (truncates values like 10.000999999999999 instead of rounding them to the nearest 8'th decimal place) then that's a bug in that code. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 03, 2011, 03:34:56 PM That's code to parse JSONs. There's a Python library to work with JSON-RPC. There's also a PHP library to do JSON-RPC. Both use floats.
Solution A: everybody that wishes to interface with Bitcoin in Python/PHP must write their own (potentially buggy) RPC http code because the default libs for those languages uses floats. Solution B: a small change is made to Bitcoin. B is a much better solution :) Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 03, 2011, 08:06:34 PM That's code to parse JSONs. There's a Python library to work with JSON-RPC. Huh? See that 'import json' statement at the top? That would be the standard (as of python 2.6) JSON parsing library.The code I posted tells the standard JSON parsing library to read JSON Numbers as Decimal. If you are doing monetary calculations in python, then you should be using Decimal. That is what it is for. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 03, 2011, 09:37:28 PM That's code to parse JSONs. There's a Python library to work with JSON-RPC. Huh? See that 'import json' statement at the top? That would be the standard (as of python 2.6) JSON parsing library.The code I posted tells the standard JSON parsing library to read JSON Numbers as Decimal. If you are doing monetary calculations in python, then you should be using Decimal. That is what it is for. Not json but json-rpc as recommended by json-rpc themselves. http://json-rpc.org/wiki/python-json-rpc Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 11, 2011, 08:53:38 PM This is stubborness... Now I'm trying to integrate Bitcoin into a website but the JSON-RPC library only returns floats.
Quote Floating point numbers have limited precision. Although it depends on the system, PHP typically uses the IEEE 754 double precision format, which will give a maximum relative error due to rounding in the order of 1.11e-16. Non elementary arithmetic operations may give larger errors, and, of course, error progragation must be considered when several operations are compounded. Additionally, rational numbers that are exactly representable as floating point numbers in base 10, like 0.1 or 0.7, do not have an exact representation as floating point numbers in base 2, which is used internally, no matter the size of the mantissa. Hence, they cannot be converted into their internal binary counterparts without a small loss of precision. This can lead to confusing results: for example, floor((0.1+0.7)*10) will usually return 7 instead of the expected 8, since the internal representation will be something like 7.9999999999999991118.... So never trust floating number results to the last digit, and never compare floating point numbers for equality. If higher precision is necessary, the arbitrary precision math functions and gmp functions are available. Why can't Bitcoin return strings? You're deliberately breaking with all the languages (python JSON-RPC library, PHP JSON-RPC and Perl JSON-RPC). Using floats anywhere in financial transactions is unacceptable. php's json_decode DOESN'T support anyway to return floats as strings. The option doesn't exist. PHP solution: write/maintain my own JSON parser. Neither does Python's JSON-RPC. Have to write my own JSON-RPC lib using json module instead of using the one that already exists. Same for Perl. Really, why is it such a big deal? Bitcoin is broken and this needs fixing. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 11, 2011, 09:17:22 PM PHP solution: write/maintain my own JSON parser. Why can't you just multiply the numbers by 1.0e8 and then round to the nearest integer? That integer WILL ALWAYS BE EXACTLY RIGHT (assuming you're not running PHP on some really weird hardware). According to the PHP manual: Quote The size of a float is platform-dependent, although a maximum of ~1.8e308 with a precision of roughly 14 decimal digits is a common value (the 64 bit IEEE format). I added a Python JSON-RPC library example on the Proper Money Handling (https://en.bitcoin.it/wiki/Proper_Money_Handling_(JSON-RPC)) wiki page. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 11, 2011, 09:19:46 PM This is stubborness... Now I'm trying to integrate Bitcoin into a website but the JSON-RPC library only returns floats. Why can't Bitcoin return strings? + 1 Floats are a royal pain in the ass. Every bank application programmer will probably tell you that. Especially RPC-like services should operate on strings - it makes a lot of stuff easier and allows infinite precision. According to the PHP manual: Quote The size of a float is platform-dependent, although a maximum of ~1.8e308 with a precision of roughly 14 decimal digits is a common value (the 64 bit IEEE format). For operations of extreme precision, PHP has many sets of mathematical libraries which also operate on strings, not floats. For example, BC-Math or GMP. http://php.net/manual/en/book.bc.php http://www.php.net/manual/en/book.gmp.php Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 11, 2011, 09:33:09 PM One more thing:
Additionally I think that in PHP, result of float <-> integer calculations may differ on 32Bit & 64Bit platforms, however i may be wrong (and I am too lazy to check with Google). There may be some bugs or php-specific "features" involved. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 11, 2011, 10:00:55 PM ShadowOfHarbringer: Do you know how to divide numbers in PHP GMP and obtain a decimal number (not quotient + remainder)?
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 11, 2011, 10:12:47 PM ShadowOfHarbringer: Do you know how to divide numbers in PHP GMP and obtain a decimal number (not quotient + remainder)? Well it should be fairly easy using GMP, however i have never done that before. You will probably need to study usage of following functions: http://www.php.net/manual/en/function.gmp-init.php http://www.php.net/manual/en/function.gmp-div-q.php http://www.php.net/manual/en/function.gmp-strval.php Example from PHP.net: Code: <?php And result: Code: 20 Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 11, 2011, 10:35:33 PM Yeah those are integer values,
var_dump(gmp_strval(gmp_div_q(gmp_init("5"), gmp_init("2")))); output: string(1) "2" Here's my solution, http://codepad.viper-7.com/tbZ9oD Code: <?php Multiply the quotient by 10^p (p = precision), perform the integer division, convert to string, insert the decimal point, convert back to GMP. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 11, 2011, 10:51:31 PM Yeah those are integer values, var_dump(gmp_strval(gmp_div_q(gmp_init("5"), gmp_init("2")))); output: string(1) "2" Here's my solution, http://codepad.viper-7.com/tbZ9oD Code: <?php Multiply the quotient by 10^p (p = precision), perform the integer division, convert to string, insert the decimal point, convert back to GMP. Can't you do everything of that iside GMP (resources) ? It will be much faster than operating on strings. Moving the decimal point should be also possible inside GMP. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: error on March 11, 2011, 10:55:52 PM You can do everything in GMP except decode the JSON, which is what I think OP was complaining about.
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 11, 2011, 10:57:50 PM GMP is an integer only library,
http://codepad.viper-7.com/pj58SK Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 11, 2011, 10:59:19 PM You can do everything in GMP except decode the JSON, which is what I think OP was complaining about. This is a misunderstanding. I only meant do the calculations in GMP, and only convert from/to string on input/output . GMP is an integer only library, http://codepad.viper-7.com/pj58SK Ah sorry then, my mistake. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 11, 2011, 11:00:04 PM You can do everything in GMP except decode the JSON, which is what I think OP was complaining about. No, I was complaining about the fact that the Bitcoin JSON-RPC Api uses numbers (that nearly all libraries convert to floats and have no option to change) when using strings would only be minorly impactful. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 11, 2011, 11:07:10 PM OK, i have found the base for my allegations about dangers of using floats:
Warning Floating point precision Floating point numbers have limited precision. Although it depends on the system, PHP typically uses the IEEE 754 double precision format, which will give a maximum relative error due to rounding in the order of 1.11e-16. Non elementary arithmetic operations may give larger errors, and, of course, error progragation must be considered when several operations are compounded. Additionally, rational numbers that are exactly representable as floating point numbers in base 10, like 0.1 or 0.7, do not have an exact representation as floating point numbers in base 2, which is used internally, no matter the size of the mantissa. Hence, they cannot be converted into their internal binary counterparts without a small loss of precision. This can lead to confusing results: for example, floor((0.1+0.7)*10) will usually return 7 instead of the expected 8, since the internal representation will be something like 7.9999999999999991118.... So never trust floating number results to the last digit, and never compare floating point numbers for equality. If higher precision is necessary, the arbitrary precision math functions and gmp functions are available. If this doesn't shout "floats are bad", then i don't know what else to say about that. So everything i said was true. "Precision depends on the system". So it may be different on 32bit and 64bit systems, and perhaps even on windows / Linux / different types of Unix/BSD. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 11, 2011, 11:15:04 PM GMP is an integer only library, http://codepad.viper-7.com/pj58SK You can still use bcmath, may be faster than working strings: http://www.php.net/manual/en/book.bc.php Also, somebody on the forums told me that there are many other libraries supporting arbitrary precision mathematics for PHP, but i don't know them myself because i never needed that. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Luke-Jr on March 12, 2011, 02:05:17 AM This isn't the only (nor biggest) problem with JSON-RPC. Help create a new standard protocol fixing this and other problems: https://en.bitcoin.it/wiki/Wallet_protocol
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 12, 2011, 04:09:03 AM Code: import urllib This should work in Python is anybody ever needs it. I've updated the wiki, https://en.bitcoin.it/wiki/API_tutorial_%28JSON-RPC%29#Python It's based off the old lib. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 12, 2011, 05:52:18 AM Anyone that prefers a saner API (because they're using PHP) should check out my branch,
https://github.com/genjix/bitcoin/tree/strrpc Values are returned as int64 strings. Code: function numstr_to_internal($numstr) Those are the 2 functions I use to convert from internal values to display/user input values (numstr). I updated the wiki, https://en.bitcoin.it/wiki/API_tutorial_%28JSON-RPC%29#PHP Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: mizerydearia on March 12, 2011, 05:59:10 AM Also see http://meta.witcoin.com/p/348/Transactions---float-conversion-issue
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 12, 2011, 03:24:51 PM Is there a PHP implementation that does not use double-precision floating point?
After doing a little googling I couldn't figure out the answer to that. I will be MUCH more sympathetic to changing the JSON-RPC api if there is. And mizerydearia: re: the witcoin issue: You say: "I see the transaction as 0.94 However, http://json-rpc.org/ retrieves the data as 0.93999999999999994671" So why when you display that value are you truncating it instead of rounding it to 8 decimal places? For example: Quote > php -r "printf('%.8f', 0.94);" #CORRECT 0.94000000 > php -r "printf('%.16f', 0.94);" #WRONG 0.9399999999999999 ... or to convert to an integer-number-of-base-unit: Quote > php -r '$val=0.94; printf("%d", round(1e8*$val));' 94000000 All of that assume that your php support double-precision floating point, which brings me back to my question: are there any php implementations that do not? Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 12, 2011, 03:57:26 PM Code: <?php Output: Code: object(stdClass)#1 (5) { ["a"]=> int(1) ["b"]=> int(2) ["c"]=> int(3) ["d"]=> float(4.08) ["e"]=> int(5) } Why don't why put in a new number like nAPIVersion and call the current API 0. Then we can make nAPIVersion 0.5 supporting the int64 with appended .0 as luke-jr has done in his gitorious branch. Then I want to make an API version 1.0 which puts in a strict naming convention for the methods (getblaa, action instead of the current messy names) and use namespaces. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 12, 2011, 04:12:07 PM Then we can make nAPIVersion 0.5 supporting the int64 with appended .0 as luke-jr has done in his gitorious branch. That's just dumb. If the problem is jsonrpc-supporting-environments that don't support double-precision floats, then multiplying and slapping a zero on the end won't fix the problem-- you'll just either lose precision as it gets squeezed into a 32-bit float or get a 32-bit-integer-overflow error. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 12, 2011, 04:37:03 PM You're right. In that case, it seems that going with strings is the only option.
Still it might be a good idea to have API versions (bitcoin.getinfo()["rpcversion"]) and take an optional -rpcversion command line argument. When you create a new API, then the old one is deprecated for at least one release cycle, allowing compatibility. When a new RPC version is created (lets call new version 1 and old version 0 for example) then: bitcoind The RPC version (0) in this release is DEPRECATED! Re-run bitcoind -rpcversion=1 for the new version. RPC changes can be found at http://bitcoin.org/rpcapi.html bitcoind stop bitcoind -rpcversion=1 bitcoind getinfo ... "rpcversion": 1 ... The policy would be to deprecate only when the API becomes backwards incompatible. Otherwise it's silently upgraded and becomes the default. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 12, 2011, 05:29:31 PM You're right. In that case, it seems that going with strings is the only option. All serious institutions (like banks) never use floats for currency calculations because they know the dangers. If we want to be seen as "serious", float is not an option. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 12, 2011, 05:31:43 PM You're right. In that case, it seems that going with strings is the only option. All serious institutions (like banks) never use floats for currency calculations because they know the dangers. If we want to be seen as "serious", float is not an option. Yep but it's a problem with the json decoding in PHP/Perl/Python. The actual precision for the number in a JSON is accurate AFAIK. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 12, 2011, 05:35:32 PM All serious institutions (like banks) never use floats for currency calculations because they know the dangers. If we want to be seen as "serious", float is not an option. You know, I was looking at the PayPal payment API yesterday, and $1.01 is sent as... 1.01 "transmitting" != "calculations" Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 12, 2011, 06:24:26 PM All serious institutions (like banks) never use floats for currency calculations because they know the dangers. If we want to be seen as "serious", float is not an option. You know, I was looking at the PayPal payment API yesterday, and $1.01 is sent as... 1.01 "transmitting" != "calculations" Of course you're right. I was merely stating the fact. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: j16sdiz on March 14, 2011, 07:50:25 AM .... Quote > php -r "printf('%.8f', 0.94);" #CORRECT ....0.94000000 > php -r "printf('%.16f', 0.94);" #WRONG 0.9399999999999999 It does not work that way .. printf("%.8f") behaviour is inconsistent across platform. If you don't believe, try Code: printf("%.2f", -999.04) -- some platform give -999.03, some give -999.04. Of course, you can fix this.... in your code. (But never expect all 3rd party apps do it right) Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 14, 2011, 11:55:07 AM It does not work that way .. printf("%.8f") behaviour is inconsistent across platform. If you don't believe, try Code: printf("%.2f", -999.04) -- some platform give -999.03, some give -999.04. I tried it... got -999.04 on my Mac and Linux machines. What platform gives the wrong answer? Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 14, 2011, 04:10:18 PM Try this:
Code: <?php Output: Code: object(stdClass)#1 (1) { ["e"]=> float(5.01) } Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 14, 2011, 05:10:25 PM My result:
Code: object(stdClass)#1 (1) { Checked on 2 separate PHP Versions : 5.3.5 and 5.2.14 Different versions, different results ? Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 14, 2011, 05:13:56 PM PHP 5.3.3. on my mac gives:
Code: object(stdClass)#1 (1) { Please be specific about what platform you're running on, what version of PHP are you running, are you running with a standard php.ini or have you tweaked it, etc. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 14, 2011, 05:40:59 PM sorry it was a typo. supposed to say 5.01
The point was that PHP is casting to floats in their JSON decoder and there's no option to use doubles or strings. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: BitterTea on March 14, 2011, 06:15:15 PM The PHP manual says that float=double.
Wouldn't gavin's solution of "float * 1e8" work? Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 14, 2011, 07:26:08 PM sorry it was a typo. supposed to say 5.01 Probably a buggy version of PHP. Anyway, on a lot of hosting providers have old versions of PHP installed (there are older versions in many distros' repositories), so it may be dangerous to use floats. This is one of the reasons why i said initially that floats are dangerous as hell. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Alex Beckenham on March 30, 2011, 04:14:18 PM Anyone that prefers a saner API (because they're using PHP) should check out my branch, https://github.com/genjix/bitcoin/tree/strrpc Values are returned as int64 strings. Code: function numstr_to_internal($numstr) Those are the 2 functions I use to convert from internal values to display/user input values (numstr). I updated the wiki, https://en.bitcoin.it/wiki/API_tutorial_%28JSON-RPC%29#PHP Reading this at the moment... https://en.bitcoin.it/wiki/API_reference_(JSON-RPC) Under 'Precision' you wrote: You will need to get a saner branch and compile it. https://github.com/genjix/bitcoin/tree/strrpc Does 'saner branch' mean get a good version of GMP? I don't quite get what the phrase refers to. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Nefario on March 30, 2011, 04:29:53 PM Saner branch is a genjix fork of bitcoind that returns int64 strings instead of floats or doubles in the json-api
This prevents any rounding errors associated with floats. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Alex Beckenham on March 30, 2011, 04:34:54 PM Saner branch is a genjix fork of bitcoind that returns int64 strings instead of floats or doubles in the json-api This prevents any rounding errors associated with floats. Cheers, still getting used to terms such as 'fork' and 'branch'. So is this still the best way to go about it for a 32-bit PHP install? (Change seems to happen quickly in this community at the moment). Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Nefario on March 30, 2011, 04:47:33 PM I don't know if it's the best way, I'm using ruby, and the whole float thing is a pain in my ass. I'm working with genjix anyway so I use his fork and avoid the entire float thing altogether.
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Alex Beckenham on March 30, 2011, 04:53:42 PM I don't know if it's the best way, I'm using ruby, and the whole float thing is a pain in my ass. I'm working with genjix anyway so I use his fork and avoid the entire float thing altogether. Okay thanks... It looks like my host is able to up my PHP install to 64bit anyway. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 30, 2011, 05:11:31 PM Can you'all educate me about these mythical rounding errors that require using GMP?
I can see, maybe, if you're computing interest down to the penny on a 30-year mortgage you might conceivably be off by a penny if you use 64-bit floats instead of 64-bit integers, although even there you're going to have to think hard about rounding as you get integer remainders. And I can see being really careful if you're writing a bitcoin exchange site or bitcoin bank that deals in thousands of internal transactions that must all balance exactly. But for the typical PHP website that is just going to add up 10 items in a shopping cart using plain-old PHP Numbers will be just fine. I don't see PayPal recommending that PHP users of it's APIs install GMP. Recommending that any website dealing with bitcoins compile genjix' fork and use GMP is a really good way to ensure that nobody accepts bitcoins. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: burtyb on March 30, 2011, 05:36:57 PM I've seen problems when porting C test functions to ARM a few years ago, you definitely can't trust a floats to behave the same on all platforms.
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Nefario on March 30, 2011, 05:55:11 PM Can you'all educate me about these mythical rounding errors that require using GMP? Recommending that any website dealing with bitcoins compile genjix' fork and use GMP is a really good way to ensure that nobody accepts bitcoins. Whoa there cowboy, I wasn't recommending anything, I just ansered a question, what was the genjix fork, and when asked what the best thing to do simply said what I was doing, and why. I'm working with genjix on my project so it's not an issue for me if I use his fork, but I can't say the same for anyone else. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on March 30, 2011, 05:57:26 PM I've seen problems when porting C test functions to ARM a few years ago, you definitely can't trust a floats to behave the same on all platforms. Actually, as long as you make sure all your IEEE compliance options are turned on... Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 30, 2011, 06:16:15 PM I've seen problems when porting C test functions to ARM a few years ago, you definitely can't trust a floats to behave the same on all platforms. Actually, as long as you make sure all your IEEE compliance options are turned on... ...but probably sometimes for some reason they won't be turned on... Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 30, 2011, 06:17:55 PM But for the typical PHP website that is just going to add up 10 items in a shopping cart using plain-old PHP Numbers will be just fine. I don't see PayPal recommending that PHP users of it's APIs install GMP. Recommending that any website dealing with bitcoins compile genjix' fork and use GMP is a really good way to ensure that nobody accepts bitcoins. Can't we just have 2 separate APIs - one float and one string in the official client ? Or maybe perhaps one API with "string mode" and "float mode" ? BTW, are the floats really so necessary and important that you defend them ? No offense, but while reading this topic it seems that almost nobody here likes floats except you. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 30, 2011, 06:22:16 PM Apologies to Nefario, I was reacting to the wiki pages written by genjix on how to use PHP with bitcoind that started with:
+ First, compile my fork. + Next, install the GMP and BCMath libraries... And why do I defend floats: because simple things should be simple. Using GMP/BCMATH is overkill for 98% of what bitcoin JSON-RPC users will be doing. And because certain people keep beating this dead horse. I have said that I am PERFECTLY WILLING to support strings in the JSON-RPC interface if somebody can demonstrate to me someplace where it is actually a real problem (that isn't trivially solved using something like round(value*1e8+0.5) or printf("%.08", value)). Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on March 30, 2011, 06:22:43 PM If we want to be taken remotely seriously by the banking and merchant communities, floats need to go. Whether the end result is "1000000000" or "10.00000000" is largely cosmetic. But I do agree that -- long term -- JSON's "number" and internal floats fall short of what is needed for proper money handling software.
But.... it just hurts to change the API right now, because we will see very little payback for a large amount of pain. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on March 30, 2011, 06:26:13 PM And why do I defend floats: because simple things should be simple. Using GMP/BCMATH is overkill for 98% of what bitcoin JSON-RPC users will be doing. As an aside: Using GMP (or similar) is overkill for two big reasons:
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Nefario on March 30, 2011, 06:28:11 PM The api doesn't need to be changed, we can just add an option to the command line that will specify int64 with float being the default.
Merge genjix's changes to be the ones used when int64 is passed in. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 30, 2011, 06:29:02 PM If we want to be taken remotely seriously by the banking and merchant communities, floats need to go. People keep claiming that, and yet I just did YET ANOTHER google search for "banking apis", clicked through to the Open Financial Exchange standard (from Microsoft and Quicken), and what do you know! Money amounts look like floats: Code: <STMTTRN> Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on March 30, 2011, 06:30:04 PM The api doesn't need to be changed, we can just add an option to the command line that will specify int64 with float being the default. Thus creating a new, second parallel API, conditionally present or absent? Ugh, no thanks. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on March 30, 2011, 06:34:12 PM People keep claiming that, and yet I just did YET ANOTHER google search for "banking apis", clicked through to the Open Financial Exchange standard (from Microsoft and Quicken), and what do you know! Money amounts look like floats: Don't assume a decimal (".") implies floating point. Explanation by example: Remember the difference in SQL between a FLOAT and DECIMAL. Float is Approximate-number data type, which means that not all values in the data type range can be represented exactly. Decimal/Numeric is Fixed-Precision data type, which means that all the values in the data type range can be represented exactly with precision and scale. Anyone who deals with money demands DECIMAL-like behavior. Bitcoin's internal use of int64 for values provides that fixed-precision, DECIMAL-like behavior. Decimals (".") are common in monetary software. Floating point data types are not. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 30, 2011, 06:38:57 PM All righty, I hereby state:
All money values in the bitcoin JSON-RPC interface Are and Shall Be treated as Decimal, with 8 digits of precision after the decimal point. If you're writing a banking application in a language that doesn't support Decimal types from JSON, then you should pack up your bags and go home. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on March 30, 2011, 06:43:41 PM If you're writing a banking application in a language that doesn't support Decimal types from JSON, then you should pack up your bags and go home. Is there a language that does produce fixed-precision from JSON? jansson (C) and python's json both produce float, from a JSON number containing "." Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: error on March 30, 2011, 06:46:15 PM The main problem is that floating point is not sufficient to accurately store a Bitcoin balance or to do simple math and achieve a correct answer. Discussions of this problem are all over the Internet. A slightly oversimplified example from a Python FAQ (http://effbot.org/pyfaq/why-are-floating-point-calculations-so-inaccurate.htm) hints at the real problem.
Ultimately a Bitcoin balance isn't a floating point number, it's a fixed-point number, and such numbers should not be manipulated with floating point functions, ever, lest Bitcoins start showing up in unexpected wallets. Oh, and some people need pictures, so here's one. http://img.thedailywtf.com/images/200902/errord/DSC00669.JPG Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: error on March 30, 2011, 06:47:46 PM If you're writing a banking application in a language that doesn't support Decimal types from JSON, then you should pack up your bags and go home. Is there a language that does produce fixed-precision from JSON? jansson (C) and python's json both produce float, from a JSON number containing "." PHP will soon have an option to json_decode() (http://www.php.net/json_decode) to treat large numbers as strings, though this code hasn't yet been released. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on March 30, 2011, 07:02:04 PM And why do I defend floats: because simple things should be simple. Using GMP/BCMATH is overkill for 98% of what bitcoin JSON-RPC users will be doing. As an aside: Using GMP (or similar) is overkill for two big reasons:
My Bitcoin branch doesn't use GMP. It's simply a change to AmountFromValue, ValueFromAmount and other functions that weren't using those helper methods. They're referring to me using GMP in PHP which I use for a range of reasons. Namely PHP not always having 64 bit ints: http://stackoverflow.com/questions/864058/how-to-have-64-bit-integer-on-php Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 30, 2011, 07:18:46 PM The main problem is that floating point is not sufficient to accurately store a Bitcoin balance An IEEE double-precision floating point number has 53 bits of precision, which IS sufficiently accurate to store a bitcoin balance. Every single possible bitcoin value can be converted to and from an IEEE 64-bit float with no loss of precision. I agree that if you're going to be performing lots of calculations on bitcoin values you need a Decimal type (and ClearCoin stores and uses python's decimal.Decimal(precision=8) for all bitcoin values)-- if you don't, floating point errors can accumulate and eventually cause you to gain or lose .00000001 of a coin. But really the main problem with storing monetary values as any floating point type is you're likely to be embarrassed by mistakes like error's cash register receipt if you truncate values instead of rounding before printing. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Luke-Jr on March 30, 2011, 07:24:27 PM I agree that if you're going to be performing lots of calculations on bitcoin values you need a Decimal type If you're going to be doing *anything* with Bitcoin values, you should be using an int64.Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: error on March 30, 2011, 07:38:04 PM The main problem is that floating point is not sufficient to accurately store a Bitcoin balance An IEEE double-precision floating point number has 53 bits of precision, which IS sufficiently accurate to store a bitcoin balance. Ha. I had to go check this for myself. Indeed, the maximum number of Bitcoins that can currently exist, 21000000.00000000 requires only 51 bits of data to store. Eventually, perhaps not in our lifetimes, Bitcoin will be working with such "tiny" values as 0.00000001, and then it becomes a big issue. So I think it's better to deal with the issue now, so our kids don't have to. :D Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 30, 2011, 08:13:27 PM The api doesn't need to be changed, we can just add an option to the command line that will specify int64 with float being the default. Thus creating a new, second parallel API, conditionally present or absent? Ugh, no thanks. Is adding a "string mode" creating a second API ? I don't think so. This is just a switch, how can you call this a second api ? Also, an option for selecting the format of output data through XMLRPC/Command line/Whatever already exists in many modern applications. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on March 30, 2011, 08:36:14 PM Is adding a "string mode" creating a second API ? I don't think so. This is just a switch, how can you call this a second api ? Because it changes the data format of every number in every RPC call. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 30, 2011, 10:01:18 PM Is adding a "string mode" creating a second API ? I don't think so. This is just a switch, how can you call this a second api ? Because it changes the data format of every number in every RPC call. Well, OK. You've got a point. Still, no big deal. After everybody starts using strings instead of floats (and they will, because floats are seen as VERY unprofessional in the industry), we may remove the old float mode completely. However no hurry there, it may wait a few years. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: BitterTea on March 30, 2011, 10:15:23 PM Well, OK. You've got a point. Still, no big deal. After everybody starts using strings instead of floats (and they will, because floats are seen as VERY unprofessional in the industry), we may remove the old float mode completely. However no hurry there, it may wait a few years. If this change was made, any code using the API must first determine the API version, and then handle numbers in two different ways. No thanks. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Luke-Jr on March 31, 2011, 12:52:35 AM No, they would just have their single "JSON-RPC value to raw bitcoins" function check the type of the argument and react appropriately. However, using strings is even MORE unprofessional than floats.
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: error on March 31, 2011, 12:58:31 AM No, they would just have their single "JSON-RPC value to raw bitcoins" function check the type of the argument and react appropriately. However, using strings is even MORE unprofessional than floats. The language may not give you a choice in the matter. Hello? Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 31, 2011, 01:55:54 AM Well, OK. You've got a point. Still, no big deal. After everybody starts using strings instead of floats (and they will, because floats are seen as VERY unprofessional in the industry), we may remove the old float mode completely. However no hurry there, it may wait a few years. If this change was made, any code using the API must first determine the API version, and then handle numbers in two different ways. No thanks. We can use different names for parameters, like SendCoinsString 100.00000000 Instead of SendCoins 100.00000000 That would eleminate such problems forever. If you want to operate on string, just add suffix "String" to every parameter/attribute name. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 31, 2011, 02:12:05 AM Ok.... so Gavin asked me (many times before and now) for the proof that the banking/financial industry avoids floats.
And i have found it !! http://en.wikipedia.org/wiki/Floating_point#Minimizing_the_effect_of_accuracy_problems Quote Binary floating-point arithmetic is at its best when it is simply being used to measure real-world quantities over a wide range of scales (such as the orbital period of a moon around Saturn or the mass of a proton), and at its worst when it is expected to model the interactions of quantities expressed as decimal strings that are expected to be exact. An example of the latter case is financial calculations. For this reason, financial software tends not to use a binary floating-point number representation.[7] The "decimal" data type of the C# and Python programming languages, and the IEEE 754-2008 decimal floating-point standard, are designed to avoid the problems of binary floating-point representations when applied to human-entered exact decimal values, and make the arithmetic always behave as expected when numbers are printed in decimal. The [7] reference leads here: http://speleotrove.com/decimal/ -------------- I think we can call the topic closed. Let's go string baby ! Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: BitterTea on March 31, 2011, 02:13:17 AM If you want to operate on string, just add suffix "String" to every parameter/attribute name. It's not a matter of what you "want" to operate on, it's a matter of the API version, and the Bitcoin version, of the user. Any code using the API would have to detect that and modify its behavior. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on March 31, 2011, 02:25:15 AM We can use different names for parameters, like SendCoinsString 100.00000000 Instead of SendCoins 100.00000000 That would eleminate such problems forever. If you want to operate on string, just add suffix "String" to every parameter/attribute name. Which is, again, a second API to support on top of the first. Twice bugs, twice the pain. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on March 31, 2011, 12:04:07 PM Ok.... so Gavin asked me (many times before and now) for the proof that the banking/financial industry avoids floats. HOW MANY TIMES DO I HAVE TO YELL THIS?????? Of COURSE you shouldn't use floats internally (unless you are doing something trivial like adding up items in a shopping cart). We are talking about the JSON-RPC api. Which is an api for communicating between bitcoin and other applications, in which all values are turned into strings. So: what are the best practices in the banking world for representing monetary values in strings? As far as I can tell, the answer is "write them out as decimal values and convert them to Decimal() or integer as you read in or write out." Which is exactly what Bitcoin does, and which is what I think we should recommend to people. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: BitterTea on March 31, 2011, 02:54:11 PM So the difference mainly lies in that the banking world isn't using JSON to shuttle data back and forth.
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: ShadowOfHarbringer on March 31, 2011, 03:28:27 PM Ok.... so Gavin asked me (many times before and now) for the proof that the banking/financial industry avoids floats. HOW MANY TIMES DO I HAVE TO YELL THIS?????? Of COURSE you shouldn't use floats internally (unless you are doing something trivial like adding up items in a shopping cart). We are talking about the JSON-RPC api. Which is an api for communicating between bitcoin and other applications, in which all values are turned into strings. Ok, my mistake. So: what are the best practices in the banking world for representing monetary values in strings? Unfortunately, I do not posess this information. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: error on March 31, 2011, 11:55:38 PM So the difference mainly lies in that the banking world isn't using JSON to shuttle data back and forth. This is the heart of the issue. JSON is very weakly typed, while much or most financial software is written in strongly typed languages. When a weakly typed language attempts to handle JSON, that large number, with or without a decimal point, may be converted into a floating-point number by the language. So, really, it's the language losing, but we have to somehow deal with it. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: BitterTea on April 01, 2011, 12:04:07 AM So the difference mainly lies in that the banking world isn't using JSON to shuttle data back and forth. This is the heart of the issue. JSON is very weakly typed, while much or most financial software is written in strongly typed languages. When a weakly typed language attempts to handle JSON, that large number, with or without a decimal point, may be converted into a floating-point number by the language. So, really, it's the language losing, but we have to somehow deal with it. Then I say in the short term, we deal with JSON's quirks and leave the number as a number (which some languages may treat as a float instead of decimal), but take this into account when designing Bitcoin RPC v2. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on April 01, 2011, 01:22:22 AM FWIW, this is the python recipe, straight from Python documentation:
Code: >>> import decimal Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Luke-Jr on April 01, 2011, 06:00:04 AM Then I say in the short term, we deal with JSON's quirks and leave the number as a number (which some languages may treat as a float instead of decimal), but take this into account when designing Bitcoin RPC v2. Exactly: https://en.bitcoin.it/wiki/Wallet_protocolTitle: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on April 01, 2011, 10:56:42 AM FWIW, this is the python recipe, straight from Python documentation: Code: >>> import decimal Except Python's JSON-RPC has no support for that. You have to use my (non-standard) modified version: https://en.bitcoin.it/wiki/API_reference_%28JSON-RPC%29#Python I prefer wherever possible that people stick with official libraries. Bitcoin now is effectively saying that all other languages are broken therefore we won't fix our API. Oh but there's a hack you can use to workaround... It's mostly accurate to the 8th decimal place. Not the point! By exposing an API which nearly all languages interpret as floats, you are expressing a statement of intent that it's fine to use floats for clients (API users). It should be passed as strings (without the decimal point preferably) that users have to explicitly cast to mutable objects. Then they realise that since these values are in INT51 strings (since we can't use large ints in JSON), they'll hopefully understand to deal with the values using ints. You're encouraging users to deal with floats ATM. Bad practice, and shouldn't be encouraged. I am really paranoid about leakage, and for me to even touch a float when dealing with people's money is unacceptable. On Britcoin, there's a whole system of balances and checks at every stage of the various transactions to ensure consistency to all decimal places, and I would not want to throw a spanner in the works because of a tiny rounding error (which would throw up flags and warnings everywhere). Even hearing the words round(...) is like heresy. Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Gavin Andresen on April 01, 2011, 11:58:01 AM I am really paranoid about leakage, and for me to even touch a float when dealing with people's money is unacceptable. On Britcoin, there's a whole system of balances and checks at every stage of the various transactions to ensure consistency to all decimal places, and I would not want to throw a spanner in the works because of a tiny rounding error (which would throw up flags and warnings everywhere). Even hearing the words round(...) is like heresy. Am I the only person here who looks at our documentation for how to use bitcoin from PHP and thinks "people are going to run away screaming" ? See: https://en.bitcoin.it/wiki/API_reference_(JSON-RPC) (https://en.bitcoin.it/wiki/API_reference_(JSON-RPC)) I strongly believe you are making the common cases (simple shopping carts or "hold a bitcoin balance for a customer and let them spend it") much, much more complicated than necessary. What do other people think? I'd especially like to hear from people who have prior experience using PHP to implement shopping carts and other simple applications that deal with money. Did you use BCMATH/GMP? Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: genjix on April 01, 2011, 12:52:59 PM Instead of removing all of this:
https://en.bitcoin.it/w/index.php?title=PHP_developer_intro&action=historysubmit&diff=6404&oldid=5737 Maybe we should add it to the bottom of the page, with an opening paragraph about how it isn't strictly necessary, but can help avoid problems. Because I did take time to write that for people, and doubtless others will find it useful, but not everyone may wish to use it ;) Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: Luke-Jr on April 01, 2011, 03:35:06 PM round(100000000 * amount) works fine in PHP. The new Wallet protocol will use ints properly, so this should be good enough.
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: jgarzik on April 02, 2011, 06:28:21 AM See this thread (http://bitcointalk.org/index.php?topic=5314.0) for a python class that handles bitcoin's JSON-RPC, including proper decoding of JSON float-like numbers into python Decimal.
Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: bytemaster on May 21, 2011, 05:57:02 AM https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_Calls_list says that the sendtoaddress and sendfrom "rounds to the nearest 0.01"
I am guessing that this documentation is out of date? Is the a new, "non original", RPC documentation? Title: Re: why JSON RPC values not use INT64 instead of float string? Post by: laanwj on May 21, 2011, 06:06:31 AM OP is right in that financial software generally uses fixed point instead of floating point (the SQL/Python DECIMAL type). This is because you can guarantee that every picocent is accounted for, when handled correctly there will never be any loss due to precision issues.
Twitter had the same problem with their TweetIDs running out of the 53 bit range that JSON is able to represent with numbers. They opted with using strings: http://blog.programmableweb.com/2010/10/19/the-twitter-id-shuffle-text-vs-numbers/ Bytemaster: yes, it is out of date. No rounding happens in AmountFromValue. |