You have a fair point. But I'm extremely skeptical as to overall software architectural skills in the broad Bitcoin community. My absolute favorite is the inversion of control issue in Satoshi client exhibited by ThreadSafeAskFee(). Such a textbook case of an anti-pattern.

https://bitcointalk.org/index.php?topic=44330.msg538044#msg538044

The rationalization I heard why this "isn't a problem" or the workarounds proposed are quite enlightening and entertaining. Too bad that moderators deleted several flaming comments in that thread.

Hi slush (and other readers)!

I reread your proposal (both here and in Google docs). In my opinion you are proposing either (1) internally inconsistent protocol or (2) family of incompatible protocols supported by a hope of sharing some implementation details.



The resultant Frankenstein monster (or family of monsters) is going to try to strangle its creator.

It is hard to argue with what you proposing point by point because of the contradictory nature of the requirements. I'm going to group my response into the following points.

1) Protocol can be either RPC or bidirectional, but not both. The RPC paradigm (request-response, master-slave) is mutually contradictory with a pair of peers exchanging asynchronous messages. Bitcoin protocol itself is from its origin asynchronous and cannot be squeezed into the master-slave architecture. It demands the bi-directional asynchronous protocol with finite state machines (FSMs) on both ends communicating with a two flows of sequenced frames. The successful design will look very much like TCP on top of IP frames.

2) Your target market (low-end consumer-level devices) demands that there's checksuming and framing at the application layer. Precisely because cheap NAT gateways and cheap DSL/Cable/WLAN modems are known to mangle the transport-level frames due to bugs (in the implementation of NAT) and excessive buffering (to improve one-way file transfer benchmarks).
If you think you can add CRC later you are going to loose by not detecting corruptions early.

3) In my experience JSON is probably the close to being the least resilient encoding possible. I can't disclose proprietary data, but I have over a decade's worth of reliability statistics for various  remote services (RPC-like) that are sold by organizations I consulted. But the rough ranking is
as follows: (from the least errors to the most)

3.1) ultra-lean character-based protocol similar to FIX, designed originally for MNP5 and V.42bis modems, currently used through a simple TCP/IP socket
3.2) SOAP (RPC over XML) with Content-Length, Content-MD5 and DTD verification enabled
3.3) SOAP and plain XML-RPC without the above strenghtening options
3.4) JSON-RPC
3.5) RPC over e-mail with various human-readable encodings

All the above are primarily sold to small businesses and individual traveling salespeople, which seems to be the target market you are planning to serve. We also have enterprise-oriented services using various MQ and *-remoting technologies, but the services offered aren't directly comparable. I thing that JSON could be an acceptable choice if you from the start demand the strengthening by some form of Content-Length and Content-Checksum. JSON is also infamous for letting people easily make byte-endianness mistakes, very much like current "getwork" which is neither big-endian not little-endian. Yes, JSON saves a lot of bandwidth when compared to XML. But I know of no presently available implementation that isn't producing cryptic and hard to troubleshoot failure modes. It is a classic example of a "lean but very mean" design.

4) You somehow read the my earlier suggestions about IPsec imply high-end large-volume target market. The reality is quite opposite: Windows support IPsec since 2000, Linux for a long time, Netgear ProSafe family has several models in $100-$200 range, L2TP and PPTP are available for free on all iPhones,Blackberries,Androids; many Nokias and other smartphones. The real hindrance is the HTTP(S)-uber-alles mindset, not the actual difficulty and cost of the implementation.

In summary I'd like to say that you wrote a very interesting and thought provoking proposal. I just think that the range of the targets you are hoping to cover is way too broad ($3 AVR processors, shared hosting plans, home computers, etc.).

I have my personal litmus test for the technological implementation in the Bitcoin domain: it has to support (and be tested for) chain reorganization. Preferably it should correctly retry the transactions upon the reorg. Absolute minimal implementation should correctly shut down with a clear error message and a defined way to restart the operations. Any implementation that will behave like Tom Williams' Mybitcoin.org blame-it-on-the-chain-reorg is a failure. Thus far your proposals don't seem to encompass this essential feature of Bitcoin. If Electrum (and associated protocols) aren't going to behave properly in the presence of chain reorganization events then I withdraw my earlier opinion that Electrum has more potential than the Satoshi client.

Thank you again for your time and attention.

Well, it depends on how you define "centralized." Currently Bitcoin is 100% owned by the mining cartel. I think that among the "core developer" only Gavin isn't running a mining operation. So if Mt.Gox or other exchange succeeds at breaking into the mining cartel then Bitcoin will actually become more diversified.

In a Bitcoin client unrated blocks could never replace rated blocks. So the 51% attack as it is known now would be a threat only for those who don't trust any rating service: hard-core anarchists, people from the fringes of society, etc. For everybody else the rating services will be the defense. Some will buy the ratings online, some will periodically download the ratings in the public library, very much like the access to financial ratings is happening right now. There is whole spectrum of speed-of-access vs. cost trade-offs that are possible.

Obviously the current hard-coded values in the clients will need to change: 6, 150, 500, probably many others. Right now the trust spectrum is: 100% gavinandresen & fabianhjr, 0% all others. This should change in the future.
 

Per slush's request I moved this post to another thread:

https://bitcointalk.org/index.php?topic=55842.msg664479#msg664479

Try prefixing the pipeline with LC_ALL=C .

I don't know what you & Casascius have in your minds exactly, but my understanding of the "checkpoint services" would be that they will provide ratings for each block: AAA, AA, A and so on. The big exchange like MtGox would run its own pool and its own block rating service. Then instead of the fixed "6 confirmations" to consider the transaction valid people would agree to make the conditions more involved, eg: transaction is valid when MtGox rates the block containing it at least A and Casascius at least AA and slush|TradeHill-conglomerated at least AAA. There wouldn't be a single point to attack. The attacks would result mostly in  delays of the settlement.

Why worry about large banks? Worry about medical imaging centers, even the small ones. Consider the architecture of modern medical imaging machines like CT,MRI,PET,etc. Think of all the FPGAs sitting between the actual image acquisition sensors and the RAID where the images are getting stored.

Now imagine that you could put a bitcoin mining bitstream on an average MRI machine, instead of the usual image filtering bitstream. The power that goes to the pre-processing cabinet is minuscule compared to the power going into the coil and its cooling. The tera-hashes would send shivers down your spine.

Dear Slush!

In this holiday season I'd love to speak to you directly over a small container of a product that made the Czech city of Plzeň world-famous instead of trading jabs on the Internet forum. Thus please don't take the following critique as a personal slight.

JSON-RPC is a hammer & nail combo in search of a wall to pound on.

1) Hammer: RPC over HTTP(S) is a horrible transport for the financial data. The main problem is that a site cannot really distinguish sudden spikes in popularity from distributed denial of service attack. The tools for DDoS defense and normal load balancing are popular and comparatively well understood, but they are create more side-effect problems that they solve. In my experience the transport has to rely on (comparatively) long lasting network sessions and instead of lowest-common-denominator SSL it should use some other form of network security technology like IPSec or any available VPN implementation (but not SSL-VPNs). Please take the advantage of the fact that your design involves two custom programs on both ends of the pipe. You do not need to use the protocol that was designed to support dumb and full-of-holes web browser on the client side.

2) Nail: JSON is essentially a LISP s-expressions sweetened with Javascript syntax. This is a great improvement over XML/SOAP but the lack of transport error-detection and packet boundary weakens all implementations. All the transport-level problems show up as hard to troubleshoot "syntax errors" or just pass undetected. The syntax error recovery with a stack of open-close braces is a really weak point. When the Internet is ubiquitous with "transparent buffers" (that silently overflow) and "transparent proxies" (that silently cache or otherwise mangle) you'll need all you can do do detect and bypass them.

3) Pounding the wall: By committing to a rather primitive tool you also commit to solving many of the well known problems with the RPC paradigm. By the time you correctly implement the garbage collection of stale subscribers, reciprocal authentication of callbacks, etc. you'll understand why Microsoft delivered DCOM (Distributed COM) years after COM (Component Object Model). Please learn from MSFT mistakes and don't undertake an experimental re-verification of their money sinks. I'm not a big fan of DCOM, but it is a great example of how a full and efficient implementation of simple RPC concept needs to be either very complex or very fragile. Also, when speaking of DCOM please disregard ActiveX GUI-overlays on that protocol; they don't pertain here.

I think that Electrum has a much greater potential than the current Satoshi client:

1) from the start it undertakes the correct abstraction-level split between:
1a) wallet-less shared server that knows the block-chain and p2p broadcast network;
1b) blockchain-less private client that concerns itself mostly with security and UI;
2) by not directly supporting mining it will not get involved in the rigmaroles with mining pool operators.

Lastly I want to stress that I don't "personally like such technologies." I simply have a significant experience of fixing the problems for clients who painted themselves into a corner by using some form RPC when a real duplex-communication protocol should have been used in the implementation from the start. Although I'm not directly involved in finance-industry IT anymore I know that the solutions I co-architect-ed and consulted have successfully withstood the test of time for over a decade. I also know that competitors who went for a quick-to-implement solution paid tremendous price in form of the maintenance costs.

Thank you for your time and attention.

Edit: whatever protocol you choose please make sure that it supports block-chain rollback from the start. You don't have to implement it in the initial release, but it has to be there. I'm forever grateful to Gavin Andresen for refusing to remove or tamper with the chain-rollback logic in the Satoshi client.

From my previous conversations with slush I know that he is dead-set on reinventing the wheel, so I'm not going even try to change his opinion.

But if anyone else involved in software architecture is reading this thread, please acquaint yourself with prior art:

http://en.wikipedia.org/wiki/0MQ
http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol

or my other proposition: find some sort of canonical serialization of Bitcoin over FIX:

http://en.wikipedia.org/wiki/Financial_Information_eXchange

The last choice gives you an additional benefit of being compatible with pretty much everything that is used to trade anything fungible.

The satisfaction you'll get from reinventing a working wheel is short lasting, really. I almost guarantee that once some time passes you will be ashamed of your pride of re-inventing the wheel.

use "bitcoind stop" or any equivalent through the JSON-RPC interface.

By the way most of the shutdown time is spent in the database logs cleanup. You can somewhat speed this up by defragmenting all the *.dat files. Significant further speed-up is achieved by the  reorganization of blkindex.dat: use db_dump & db_restore; just be careful to set the DB_CONFIG correctly.

I'm sorry I wasn't clear. It is the sending address that is declared "green", "golden", "disbursal" or whatever. But the logic to recognize it must be on the receiving side. So you cannot just use a single adjective "marked address" to describe the whole mechanism.

The practical examples in the USA are probably various special checks. Funds from the average check are held by the receiving bank for the "hold period". If the check is "Treasury check", "cashier check" or a "check from a corresponding bank" the "hold period" is shortened, sometimes to zero. But the exact rules of which checks deserve fast treatment are specified by the receiver, not by the sender.

Both Meni R. and Death A.T. are 100% correct in their description of network splits.

There is additional wrinke when block_number mod 2016 == 2015. This situation is called "retarget". In those situations the blocks differ not only by the "previous hash" field but also by the "difficulty" field.

The global network will still converge to a dominant, longest chain; just that the convergence could potentially take much longer. But the difference in difficulty creates interesting problems both from the control-theory point of view as well as from the game-theoretical point of view.

For example the dominant mining pools were DDoS-ed just before a retarget, probably in the hope of creating a longer-lasting network split and mutual mistrust between the pools.

The replies from slush and SgtSpike show that they simply don't understand generally accepted accounting principles.

The cannonical use of disbursal account is an exact opposite of what you think:

1) big organisation sets up disbursal account
2) lots of small payees use the "golden" account number as a shortcut through their credit approval process.

This is exactly the case of why a "general purpose" popular client should have a built-in ability to mark some sources of funds as requiring less verification. Since this is 21 century there should be some sort of PKI interface that allows adding and removing "trusted account" numbers. Sort of like every modern web browser has a some sort of weakly hidden tab to set up certification authorities and other PKI miscellanea.

Thanks for the heads up. Good luck in your venture. I'll revise my proposal to flesh out the details and make it more readable to a broader group of people than just persons having oridinary skill in the art.

It isn't really bittorent versus bitcoin. The early bittorrent implementations had the same problem: you had to run one executable per active torrent and open one port per each active torrent. It is more of a "quality of the implementation" issue.

The discussion why DHT is better than IRC or BT trackers has been done so many times that I won't repeat it here.

"Green Address" is just a weird name for the old practice of having a "disbursal account". There are also other names for the similar practices in the field of accounting, with various minor details changed. The general idea is that it allows for savings in the cost and delay of credit check on the receiving end and as such it is widely accepted.

Why people involved in Bitcoin keep giving new names to the old things? I don't know. Do they know?

You are so very naive. The "ufasoft miner" is open source and you can download it yourself and try to build it. This ufasoft guy is a international grand master Philosophy Doctor in software obfuscation for the purpose of padding billable consulting hours.

I'm actually amazed and fascinated:

1) dependence on a precise packaging errors made by the Ubuntu maintainers of libcurl
2) dependence on an open source clone of Microsoft MASM (JWASM)
3) dependence on a poorly documented JWASM extensions to MASM
4) dependence on a very specific release of GNU autotools
5) probably other things I didn't bother to research

The only hope to build it without extensive modification is to use the exact release of 32-bit Ubuntu.

I thought I've seen a lot of bad code. But Bitcoin ecosystem is completely pushing new frontiers. Or maybe Bitcoin is just attracting a special kind of programmers, who previously only created their ambushes far away from the open roads of FOSS?

I wonder if it is possible to produce any theoretical results on the conglomerates:

1) mining pool integrated with an exchanger
2) mining pool integrated with a service provider
3) two or more mining pools (one bitcoin, other alt-chains) joined together through an exchange

Cross-subsidies may be a new frontier. Conglomerates became popular after 1960 to deal with the variance in the dividend income. They still exist in a vastly narrower range, primarily to extract the additional profits from cross-promotion and unified bargaining.

Using your terminology a conglomerate would convert the green line to a green plane: high risk and high variance wouldn't be the opposites but a pair of coordinates on a plane.

My guess is that they just used the estimates from the synthesis tool. It would appear that they didn't even simulate the chip before racing to design the PCB. Pre-simulation power estimates are known to be wildly inaccurate. I would tend to believe their claim that they've been 10 years in this business, which means that there are a lot of opportunities in the FPGA design business for vendors capable of better-organized work-flow.

The time it took for them to develop the software drivers is another piece of circumstantial evidence that they didn't create the simulation testbench.

Another benefit of using LISP (or any similar language) for scripting lies in its transformability. There exist a body of research of ultra-reliable computing that used “SIMD-like” and/or “Hamming distance 3 or higher” coding for error detection and correction. Ultimately no LISP computers were used in the deep space probes because of overall power requirements. For the terrestrial finance transactions the absolute power used by the computer is not really limiting, but the invulnerability to the various side-channel attacks like differential fault analysis becomes a tremendous benefit. Those fault-hardening and SIMD-like transformations could be applied mechanically to the scripts so long as they are represented appropriately.

Obviously Bitcoin stack automaton scripts can be automatically translated to the prefix s-expression notation and undergo the same transformations as above. But I don’t see the benefit it requiring this additional step aside from backward compatibility.

Overall the program implementing the current proposal could be compatible with Bitcoin and all currently existing alternative block-chain currencies, including Litecoin, IxCoin, I0Coin, Tenebrix, and Fairbrix. It would be up to the Bitcoin core development team to commit to the precise rules regarding fees and checkpoints. It could even transact Solidcoin version 2 and would conceivably prevent any closed-source modifications that plague that clone of Bitcoin. The network transport protocols are currently incompatible, but the network adaptation layer would be very simple.

In summary this proposal encompasses three main changes: (1) explicit cryptographically signed and software-executable contract included in the root block, (2) cooperative DHT-based networking protocol that does away with IRC, dedicated ports and 4-byte identifiers, (3) general prefix script notation backed by strong syntax and semantic checkers.
Because of this proposal is very far-reaching I suggest that it will be immediately placed in the dormant state. Initially we can work on clarifying its wording, but the full implementation will require a lot of discussion and research. Hopefully the information included here will stay in public domain and will spread amongst the cryptography research community.

(End.)