Bitcoin Forum
January 24, 2017, 11:28:55 PM
 News: Latest stable version of Bitcoin Core: 0.13.2  [Torrent]. (New!)
 Home Help Search Donate Login Register
 Pages: 1 2 3 4 5 [6] 7 8 9 10 11  All
 Author Topic: [Stratum] Overlay network protocol over Bitcoin  (Read 33574 times)
2112
Legendary

Offline

Activity: 1750

 January 01, 2012, 09:46:32 PM

Hi slush!

I'm fully aware that those requirements are contradictory. It's pretty common that some solution needs to find some compromises.
After waiting a while and reading the replies and comments to your proposal I now understand your approach and my mistakes. Basically I only took a helicopter tour of Klondike, but you are the one who has a ground floor experience of Klondike. I'm kinda living a sheltered life: I'm never on the first line when talking to my customers: before they talk to me they need to jump a trench or two in my organization. I'm not avoiding my customers, it just that the ones I interact with were already filtered. This is completely unlike you: are the one who speaks to every customer.

I now understand that there's no point of trying to require advanced features like pair-of-FSMs design if the potential users are only familiar and comfortable with RPC-style designs. I searched this board for any sign of anyone writing about state-of-the-art financial services technology. There wasn't anyone who would mention things like CICS,Tuxedo or even no-cost MTS & MSDTC.

If the residents of Klondike demand lighter pickaxes there's no point of offering hydraulicking.

Quote
Although  I personally dislike XML, I'm open to change my mind at this point.
I wouldn't recommend switching to XML. What I was trying to suggest is to try re-host some features of XML-based protocols on top of JSON or whatever else you choose. I think you already doing that. The main advantage of SOAP is that first-line tech support person can say things like "Parasoft SOAtest agrees with our diagnosis, the errors are on your side" and avoid the cost of escalated support calls.

Quote
Isn't TCP checksumming and TCP retransmissions on both ends enough to "fix" corrupted information?
Well, NAT breaks the end-to-end principle of IP networking. But the real culprits are usually various marketing-motivated enhancements like SPI (stateful packet inspection); "gaming mode"; "multimedia prioritization", etc.

Quote
I feel like overlay network should be "stateless", which mean that it will "forget" transaction once transaction has been succesfully broadcasted to Bitcoin network.
The "stateless" discussions remind me of "zero configuration" discussions in other areas. After all the late-discovered problems get fixed the "stateless" design is way more complex and fragile than the competing "stateful" designs.

My general suggestions can be summarized as follows:
1) store coins with as key pairs plus the block hash and height in chain
2) client connecting to server send a chain length and tip block-hash of the last connection
3) server responds with positive difference in height and new tip block-hash
4) in case of the reorg server responds with negative difference in height and hash of last block pre-fork followed by the same information as in (3) for the new branch.
5) client can recalculate the wallet and retry the transactions that still have valid inputs after the reorg
6) server takes a proactive retrying approach to the transactions it originated. This is particularly important in case of the reorg and when the originating client already disconnected.

I didn't do a complete http://en.wikipedia.org/wiki/Transaction_processing analysis, but there is a number of possible compromises in the spectrum between a fully ACID-compliant design and a ball-of-rubberbands design.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
1485300535
Hero Member

Offline

Posts: 1485300535

Ignore
 1485300535

1485300535
 Report to moderator
1485300535
Hero Member

Offline

Posts: 1485300535

Ignore
 1485300535

1485300535
 Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1485300535
Hero Member

Offline

Posts: 1485300535

Ignore
 1485300535

1485300535
 Report to moderator
1485300535
Hero Member

Offline

Posts: 1485300535

Ignore
 1485300535

1485300535
 Report to moderator
1485300535
Hero Member

Offline

Posts: 1485300535

Ignore
 1485300535

1485300535
 Report to moderator
marcus_of_augustus
Legendary

Offline

Activity: 2142

 January 02, 2012, 02:39:31 AM

watching

$u=http://latex.codecogs.com/gif.latex?g%282z%29%20%3D%20%5Ccos%20z%20+%20%5Csin%20z&t=573&c=FNhQMwHQwAjvRA$

omar
Jr. Member

Offline

Activity: 35

 January 02, 2012, 05:38:17 AM

A few months ago I wrote a paper about the Bitcoin Gateway, providing the design of a P2P network over the core bitcoin network and a thin client that uses this network to provide a fast, safe and easy to use experience. It essentially makes using bitcoin as easy as using PayPal. I am currently building a team to implement this and hope to release it in 2012 as an open source project. You can read more about it here:
http://arimaa.com/bitcoin/

If you would be interested to join the team to implement this, please let me know.

Red Emerald
Hero Member

Offline

Activity: 742

 January 02, 2012, 05:58:55 AM

Should some kind of secure wallet storage be included in the proposal?  This could be used to keep track of any imported keys.  Of course, deterministically generated keys should never been saved.  Servers could optionally implement it if they don't want to deal with maintaining the database.

I think a rock-solid, secure, possibly distributable key storage could be a very useful feature.

ripper234
Legendary

Offline

Activity: 1260

Ron Gross

 January 02, 2012, 07:34:48 AM

Should some kind of secure wallet storage be included in the proposal?  This could be used to keep track of any imported keys.  Of course, deterministically generated keys should never been saved.  Servers could optionally implement it if they don't want to deal with maintaining the database.

I think a rock-solid, secure, possibly distributable key storage could be a very useful feature.

+1.

If Stratum clients could remember just a single strong password instead of manage keys themselves, that would simplify client implementations even further.

The key storage has to be distributed for this to be useful though. If there are only 1-3 Stratum servers, nobody would dare hold any semi-serious money there, because simply shutting down the servers denies them of money.

A deterministic wallet can be used by clients ... but not all client implementations are deterministic today, and those usually don't store other data like address labels.

The keys+labels could be possibly stored in the blockchain itself (it's a good durable p2p storage device that's always available).

Anyway, this is certainly not in the MVP of Startum.

Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
slush
Legendary

Offline

Activity: 1372

 January 02, 2012, 04:00:15 PM

Anyway, this is certainly not in the MVP of Startum.

This.

Actually I don't care about all potential services which can be done on top of Stratum protocol at this time, but server side storage of seeds/keys can be done. In such case, ultramegalightweight client using such feature don't need to implement any crypto part, just use Stratum server as a standard web wallet, if he wants.

2112
Legendary

Offline

Activity: 1750

 January 02, 2012, 06:24:05 PM

I'm sorry, due to my copy/paste error this fragment didn't get included in my earlier post.
Side note: I'm running json-based protocol on the pool over a year, I had over 3300 rq/s in June peak. [...] But I definitely didn't have any problems with corrupted packets like you're suggesting.
This is probably due to three factors:

1) you collect only one-sided statistics
2) the mining protocol on your end is extraordinarily simple: "getwork" or "getwork <hexstring>". Thus probably some corruptions show as "stales", "invalids" or some such.
3) on your end mining protocol has shorter packets. To really see the errors the amount of data exchanged needs to be larger that TCP/IP MSS (maximum segment size), which in turn depends on MTU (maximum transmission unit)

I only ever solo-mined test coins, so I'm not really familiar with the details of pooled mining. For solo-mining without a pool server the error detection is nonexistent.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
slush
Legendary

Offline

Activity: 1372

 January 03, 2012, 02:56:44 AM

Few moments ago I commited some stuff. Http transport is still under construction.

As a homework I implemented broadcasting transaction to Bitcoin network, it's method "blockchain.transactions.broadcast". It is using ArtForz's halfnode code for talking with trusted Bitcoin node over P2P. However no patch for bitcoind is necessary, which is significant improvement against current Electrum server implementation .

Jan
Legendary

Offline

Activity: 1042

 January 04, 2012, 12:53:56 PM

...I still don't know exactly how to generate the offline TX...

The BCCAPI which BitcoinSpinner is built on top of has all the Java classes you need to do off-line transaction validation/signing and deterministic key derivation etc. Look for Tx.java, DeterministicECKeyManager.java and Account.signSendCoinForm(). This should give you a good head-start.

Regarding the long protocol discussion: Since you are addressing mobile devices that are going to connect over a wide variety of wi-fi networks through god knows how many firewalls you should really reconsider to use your own protocol directly on top of TCP. It is not uncommon for firewalls to block anything but port 80 and 443, and there are many tricks and tools that can come in handy if you use HTTP(S)

This is a really interesting project, and I wish you good luck. Unfortunately I am too tied up to participate actively.

Mycelium let's you hold your private keys private.
ripper234
Legendary

Offline

Activity: 1260

Ron Gross

 January 04, 2012, 12:55:46 PM

...I still don't know exactly how to generate the offline TX...

The BCCAPI which BitcoinSpinner is built on top of has all the Java classes you need to do off-line transaction validation/signing and deterministic key derivation etc. Look for Tx.java, DeterministicECKeyManager.java and Account.signSendCoinForm(). This should give you a good head-start.

Regarding the long protocol discussion: Since you are addressing mobile devices that are going to connect over a wide variety of wi-fi networks through god knows how many firewalls you should really reconsider to use your own protocol directly on top of TCP. It is not uncommon for firewalls to block anything but port 80 and 443, and there are many tricks and tools that can come in handy if you use HTTP(S)

This is a really interesting project, and I wish you good luck. Unfortunately I am too tied up to participate actively.

+1, thanks Jan!
I'll definitely check out bccapi.

Mastercoin Executive Director
Co-founder of the Israeli Bitcoin Association
casascius
Mike Caldwell
VIP
Legendary

Offline

Activity: 1344

The Casascius 1oz 10BTC Silver Round (w/ Gold B)

 January 04, 2012, 04:22:49 PM

I am not sure if this has been discussed, but one important purpose I foresee of any overlay network is a method of communicating recommended transaction fee policies to clients, so that miners have a way of conveying to clients what transaction fees they should expect to pay to get their transactions into blocks.  The whole 0.0005 hardcoded bit isn't going to make sense most of the time.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper wallets instead.
slush
Legendary

Offline

Activity: 1372

 January 04, 2012, 06:21:12 PM

The whole 0.0005 hardcoded bit isn't going to make sense most of the time.

Method blockchain.transaction.guess_fee() is addressing exactly this. At this moment there's 0.0005 constant, but I have an algorithm for calculating recommented fee in my head already...

slush
Legendary

Offline

Activity: 1372

 January 04, 2012, 06:24:10 PM

Regarding the long protocol discussion: Since you are addressing mobile devices that are going to connect over a wide variety of wi-fi networks through god knows how many firewalls you should really reconsider to use your own protocol directly on top of TCP. It is not uncommon for firewalls to block anything but port 80 and 443, and there are many tricks and tools that can come in handy if you use HTTP(S)

There's no need to reconsidering anything. TCP socket is the most elegant and optimized way to communicate over network, so I'm recommending to use TCP socket where it is possible, but I'm implementing HTTP polling, so mobile devices can work with Stratum nicely.

Legendary

Offline

Activity: 1666

 January 04, 2012, 06:34:54 PM

The whole 0.0005 hardcoded bit isn't going to make sense most of the time.

Method blockchain.transaction.guess_fee() is addressing exactly this. At this moment there's 0.0005 constant, but I have an algorithm for calculating recommented fee in my head already...

Could you provide a rough, layman's explaination of that algo please?

"The powers of financial capitalism had another far-reaching aim, nothing less than to create a world system of financial control in private hands able to dominate the political system of each country and the economy of the world as a whole. This system was to be controlled in a feudalist fashion by the central banks of the world acting in concert, by secret agreements arrived at in frequent meetings and conferences. The apex of the systems was to be the Bank for International Settlements in Basel, Switzerland, a private bank owned and controlled by the world's central banks which were themselves private corporations. Each central bank...sought to dominate its government by its ability to control Treasury loans, to manipulate foreign exchanges, to influence the level of economic activity in the country, and to influence cooperative politicians by subsequent economic rewards in the business world."

- Carroll Quigley, CFR member, mentor to Bill Clinton, from 'Tragedy And Hope'
Legendary

Offline

Activity: 1666

 January 04, 2012, 06:38:24 PM

What are the odds that this overlay network could be used to permit two mobile devices to transact directly, in the absence of Internet access for one or both parties?  For example, a direct ad-hoc wifi connection or an indirect connection via a 'piratebox' type device?  Asuuming, of course, that the sending device doesn't need Internet access to produce a transaction from it's own, known address funds.

"The powers of financial capitalism had another far-reaching aim, nothing less than to create a world system of financial control in private hands able to dominate the political system of each country and the economy of the world as a whole. This system was to be controlled in a feudalist fashion by the central banks of the world acting in concert, by secret agreements arrived at in frequent meetings and conferences. The apex of the systems was to be the Bank for International Settlements in Basel, Switzerland, a private bank owned and controlled by the world's central banks which were themselves private corporations. Each central bank...sought to dominate its government by its ability to control Treasury loans, to manipulate foreign exchanges, to influence the level of economic activity in the country, and to influence cooperative politicians by subsequent economic rewards in the business world."

- Carroll Quigley, CFR member, mentor to Bill Clinton, from 'Tragedy And Hope'
kjj
Legendary

Offline

Activity: 1302

 January 04, 2012, 06:52:34 PM

I don't think an algorithm is going to cut it for fees, at least not in the long run.  In the long run, miners will need to provide quotes.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
slush
Legendary

Offline

Activity: 1372

 January 04, 2012, 07:01:13 PM

Could you provide a rough, layman's explaination of that algo please?

Basic idea is to measure the relation between fees in transactions included in few last blocks and fees in pending transactions (assuming that transactions which are pending for some long time and they're not included in the blockchain have lower fees than necessary).

But I agree that some mechanism for negotiating required fees with miners would be interesting, too.

Red Emerald
Hero Member

Offline

Activity: 742

 January 04, 2012, 07:03:56 PM

But I agree that some mechanism for negotiating required fees with miners would be interesting, too.
If only we knew someone with a pool to experiment with

2112
Legendary

Offline

Activity: 1750

 January 04, 2012, 07:26:50 PM

But I agree that some mechanism for negotiating required fees with miners would be interesting, too.
It may be interesting from the anthropological point-of-view: who's gonna fall for it. Technically the negotiation for inclusion in a block doesn't make sense: the blocks are won randomly. I guess from the game-theoretical point-of-view one could come up with an optimal strategy: clairvoyance.

Please comment, critique, criticize or ridicule BIP 2112: https://bitcointalk.org/index.php?topic=54382.0
Long-term mining prognosis: https://bitcointalk.org/index.php?topic=91101.0
slush
Legendary

Offline

Activity: 1372

 January 04, 2012, 07:30:08 PM

I agree that such negotiation can turn into pretty complicated thing. However, at this moment, at least me and deepbit is trying to include as much free transactions as possible, for marketing purposes ("Bitcoin is providing free transactions").

 Pages: 1 2 3 4 5 [6] 7 8 9 10 11  All