Bitcoin Forum

Over the past ~24 hours, the number of satoshidice transactions has increased hugely, leading to transaction memory pools (currently) at around 9000 transactions.  Satoshidice spam is already a huge % of current transactions, but now its just ridiculous.  Is it time to start deprioritizing transactions which use very common addresses?

:o

SD pays a fee on all tx correct? What fee? Whatever it is x30000 seems meaningful.

Paying a meaningful fee to miners has no effect on the thousands and thousands of bitcoin users who have to store the transactions on their disk...

No, it is not time to start deprioritizing anything.  It is time to start prioritizing pruning of the blockchain.

Sadly, chain pruning is by no means easy.  Pruning the index of transactions (blkindex.dat) isnt hard, and is likely to be done in the next release or the release thereafter.  However, pruning blk0001.dat really can't be done without making pretty significant changes to the way bitcoin downloads blocks.

Best get started then!  ;)

Sadly, the most realistic way of doing it involves shipping chain snapshots as a part of the client download.  This introduces more trust in the decentralized network, which really isnt a good thing.  Hence why people are not eager to start working on chain pruning: many users will reject it.  Note that it also makes it very, very difficult for people to run old nodes (they wont sync properly), which in light of recent node statistic generation by luke, is looking like a very, very poor idea for network security.  In any case, chain pruning isnt coming soon as doing it really isnt pretty and could result in some pretty bad things for the network as a whole.

Having the entire chain does more disservice than service to the bitcoin concept.  I would think that only the unconfirmed transactions would be in the chain..and the ones that don't confirm after 96 hours should roll off as well.  IMHO the chain should be used as a means to manage transactions and not as an audit trail.

Sierra

All your sad posts are making me sad now.

I suppose then, the best thing to do is find out how the free market copes with an exponentially increasing blockchain size.  It had to happen eventually anyway...

I suspect we'll see many more users move to a light client in the coming months.  I already removed the full satoshi client from all of my computers except one, simply because it really does take a toll on the machine it is on.

That's the point of chain pruning, sadly chain pruning is only mostly feasible.
So if you dont spend your coins in 96 hours you lose them?
Thats simply not the way bitcoin was designed.

"So if you dont spend your coins in 96 hours you lose them?"

I knew my statement leading up to that was an issue as soon as I walked away from my computer...

Bitcoin was actually doing fairly well and the chain size was only increasing at a moderate rate (largely DeepBit, which still refuses to use multisend, which would cut down on their volume by a ton) up until SatoshiDice.  Realistically, if satoshidice employed more sane methods of operation (multisend would be very easy and would have a pretty large impact on pure transaction volume, but really they should allow users to carry balance and withdraw when they want, the way pretty much every other bitcoin site does it) they would have a relatively tiny impact on the chain compared to what they have now.  The only reason the chain grows the way it does is because people like satoshi dice (and pretty much only satoshidice) refuse to put in an extra 10 minutes of coding time.

Running light nodes is the way the network is going anyway, and thats fine, but forcing everyone to do so purely because people are lazy and stupid is really poor.

Oh, come on, it's not forcing anybody!  It only encourages them.  ;)

Matt, can you comment on the armory dev's proposal for chain pruning facilitated by an balance alt-chain?

I have to admit I havent spent a huge amount of time thinking about it, but its definitely an interesting concept.  It does carry with it some of the issues of alienating old nodes which may have issues finding peers which have a full chain to give them, but I believe that is inherent in any chain-pruning method anyone has come up with so far.  It does complicate bitcoin as a whole very greatly, forcing everyone to follow multiple chains (or...I cant think of a way to switch over to a new chain cleanly without hard-forking) and heavily complicating verifying the initial checking (probably ending up moving the initial download verification to the same trust-model as SPV nodes, which is ok, but not ideal).  These complications only further the effort alt-client devs have to put in, which is already pretty huge and alt-clients are pretty important for the health of the network.  In the end, I dont really see anyone jumping to implement it, so I dont see it happening any time soon, but if it does happen, it will provide some interesting new features.

There is an advantage in using blockchain such as transparency, anyone can verify how the site operates.

EDIT:I never used Satoshi Dice, but I like the idea of honest lottery.

The same result can easily be achieved without forcing users to add at least 2 transactions per bet.

It seems that Deepbit doesn't dynamically increase tx fees like the Satoshi client does. The Satoshi client would keep blocks at around 250 kB.

[For reference:  the idea I proposed is a special blockchain pruning method (https://bitcointalk.org/index.php?topic=52859.msg885838#msg885838) that allows nodes to verifiably query any address balance with only a couple kB download -- and the pruned data would be maintained & enforced on a second/alternate blockchain using merged mining]

Theoretically, the alt-chain created for this purpose is strictly optional.  No one is "forced" to do anything -- they only use it if they want to participate in creating/exchanging/verifying/downloading address-balance information.  One major benefit of the idea is how perfectly non-disruptive it is.

I agree that it would be complex.  But there's a lot of alt-chains already in existence that use merged mining, which probably provides 90% of the template that would be needed.  I don't mean to imply that there's anything easy or quick about it... just that a lot of work has already been done.

Otherwise, there's two options that both have serious downsides (compared to using an alt-chain).  But maybe some brainstorming can resolve it:
(1) Overlay network like what stratum is trying to do.  Issue:  requires some trust of other nodes, and malicious nodes can really muck up the network.
(2) Modify the main blockchain, forcing block-headers to include a valid balance-tree hash to be accepted.  Issue:  requires a hard fork. 

However, there's a dozen other things that could go in as long as we're doing a hard-fork, anyway.  If there's any inclination to believe it will have to be done at some point, earlier is better...

Why do you think so ?

Looking at your statistics, it seems that it's not you. Sorry.

Some big miner is creating blocks near 500 kB without increasing fees.

SatoshiDice is certainly pushing Bitcoin into new territory.

I thought I would mention a data structure I am going to experiment with in MultiBit as it is related to blockchain management.

Currently the MultiBit/ bitcoinj blocks that are stored to disk contain:


i.e 100 bytes per block.

It would be very useful to have a mapping of 'bitcoin address -> blocks the address appears in' so that you can get the transactions for an arbitary address/ private key/ public key quickly.

I was thinking of adding a Bloom filter (http://en.wikipedia.org/wiki/Bloom_filter) for each block (indexed by bitcoin addresses that appear in that block) into the locally stored blockchain.
The idea would be that to get the transactions for an address or set of addresses:
1) You scan the locally stored blockchain bloom filters to get a list of blocks that you know contain relevant transactions.
2) You download just those blocks from a Satoshi client.
3) You then parse the blocks' transactions to get the tx data.

You can store the presence of an object in a Bloom filter with approx 10 bits.
According to this (http://bitcoin.stackexchange.com/questions/3524/how-many-unspent-transaction-outputs-are-there) I estimate there are about 2,000,000 distinct addresses on the current blockchain and each appears, on average, about 4 times. This would mean that the total size of the Bloom filters for bitcoin address indexing would be about:
10 x 2,000,000 x 4 bits = 80,000,000 bits i.e. about 10MB

Not much space for what it gives you. It is small enough that an up-to-date version can be included in the MultiBit installation download (I already include a snapshot of the current 100-byte-per-block blockchain).

Of course this does not solve the disk requirements for the Satoshi bitcoinds. It is only usable for the end-user client as it needs a full client to download the (specific) blocks from.

They don't really have to, AFAIK.

It's up to miners to do something, if they want to do something. Maybe they can handle the volume and are happy with the fees.
EDIT: And when I say miners I mean solo-miners and pool operators, the only ones who really need the entire blockchain to operate.

Bitcoin will die unless this is solved, its not much but I pledge 15 BTC to whoever creates a viable beautiful pruning-client.

Something which can operate with parts of the chain and only get the parts it needs to confirm incoming otherwise.

Should work like a torrent file where the full file could be distributed.


Basically my client should operate as before thinking it has the full chain, only it doesn't - it gets the parts it doesn't when needed from the cloud/peers.

After using such parts they are deleted.

It would be extra cool if parts of blocks could be downloaded instead of full blocks.


Maybe it could send a request to the network to get an incoming transaction verified by the nodes that have that block section?

Say my node has a block with A->B someone sends me a transaction with B->C, I check my block and verify B address has enough then okays it.

To clarify:
1. Blocks are verified as today via their expensive hash/tree root thingie.
2. All the transactions IN that block are verified by the network via the distributed block chain.

Time line:
1. Block is received ->
2. Full prune node (FPN) checks root hash against its own mostly empty chain ->
3. If okay, it checks the transactions it CAN ->
4. If okay it broadcasts the block while assuming the other transactions are fine ->
5. If NOT okay it does NOT broadcast it, but instead reports it including info about the invalid transaction and its block ->
6. Other nodes get the report, check the root hash of that transaction against their empty chain ->
7. If bad report is true they re-broadcast it ->
8. Bad blocks get propagated WAY slower AND are immediately turned down by super nodes ->
9. No major changes to core BTC logic, SMALL additional bandwidth usage for reporting.

If this is not done, BTC is dead.

Another idea:
"Allow transmission of half-blocks."
root hash <- 2nd hashes <- 3rd hashes etc.

You basically treat the second or third hash as THE block - the benefit is that slower connections can keep up with very large blocks as they only look at one branch of the blocks.
They would however need to check their own users addresses via 3rd service.

This is however not critical at present.

So, currently the block size limit is a maximum of 500KiB:

1year/10minutes * 500KiB = 25GiB/year

So yeah, it's not tiny. But seriously, 25GiB/year really isn't all that much storage, and that's the hard maximum, actual is maybe half that. Sure the regular bitcoin-qt client becomes annoying to use for some guy who just wants to move some coin around, but anyone doing anything serious isn't going to have a problem. Myself, I run bitcoin-qt, but my actual wallet which I actualy use is on the excellent blockchain.info

The real issue is if the number of transactions makes it difficult for people to get their transactions verified, but as long as miners do fee-based priority, that's fixed by telling people to increase their fees.

My request for the devs: work on pruning, but no rush.

Exactly I don't see 25GB being an impossible barrier for merchants, miners, exchanges, and hardcore bitcoin supporters. 

Honestly even when pruning happens at least a significant number of need to carry fully blocks so that nodes which wan't/need full block have access to them. 

What a drama, come on...

This will not kill bitcoin. Even without pruning, professional pool operators should be able to handle much more traffic than this. https://en.bitcoin.it/wiki/Scalability

It seems you people are thinking "ordinary users" should be full nodes. That's not the case. Only solo-miners and pool operators have such need.

But anyway, since that's what being debated here, I always wondered if there couldn't be a "relay mode". Something lighter than the full mode of bitcoin-qt, but heavier than the lightweight mode of BitcoinJ.
You store the whole blockchain, but in its raw format. You only validate its headers, not its transactions. You index only the transactions that concern you, plus a simple index to locate the blocks in the big raw file. You relay everything as a full node would.
The greatest charge that a full node imposes today is the indexation of the database. Storing in raw format would allow those that want to contribute their nodes as relays (as me and probably you) to keep doing it for a longer time, I guess.
But, honestly, I'm not even sure we should encourage this. For it to scale, the bitcoin network will have to change to a network where only some full nodes exists and all the others connect to them as lightweight nodes. So maybe SatoshiDice is just encouraging us to take this step. ;)

I'm not even sure developers themselves should worry with this. This is a problem to pool operators mainly. It's up to them to come out with a solution, or eventually finance one by offering bounties.
I find "securing your private key(s)" a much more urgent matter than this.

That is not at all true.  The point of blockchain pruning is that most, if not all nodes on the network no longer need the full chain.  As nodes upgrade to the alt-chain and drop full blocks from the normal chain, nodes which do not want to use the new alt-chain will have a harder and harder time bootstrapping or even getting full blocks from any nodes, thus forcing them all onto the alt-chain.  Even if you argue that old nodes will always exist and be a fair percentage of the network (which they currently are, which is really a security issue for the network) connection mechanisms get complicated as nodes are increasingly trying more and more connections before finding a working one, which could lead to increased sybil attack possibilities and generally higher load on listening nodes.


Absolutely, I wasnt saying it wasnt possible or really /that/ difficult, but merging all of that code into the satoshi client is quite a bit of work, or at least quite a bit of code.  And with the client already so complex and poorly abstracted, it introduces huge possibilities for bugs.

Meh.
You dont have to force anyone to include anything additional in blocks to do pruning.  If you do pruning, you can simply drop transactions and branches in merkle trees.  Of course you have to snapshot a pruned chain and nodes cant do the pruning themselves, but that is true up to checkpoints anyway.

I dont think anyone wants a hardfork, nor do I think its worth doing without serious issues, and I dont even think this is worth it.

See http://sourceforge.net/mailarchive/forum.php?thread_name=CANEZrP0kNZDByHpK2%3DUjP%2Bag0X1KmqHxnJdm%3De_pWMitP4QvvA%40mail.gmail.com&forum_name=bitcoin-development (http://sourceforge.net/mailarchive/forum.php?thread_name=CANEZrP0kNZDByHpK2%3DUjP%2Bag0X1KmqHxnJdm%3De_pWMitP4QvvA%40mail.gmail.com&forum_name=bitcoin-development)

Its entirely true that Bitcoin is designed to operate with a number of SPV clients on the network, and the bloom filter in p2p protocol options makes that more viable.  However, merchants and those who want an better trust model are still encouraged and should still run full nodes.  Also note that a huge increase in SPV clients will starve listening full nodes of resources and connection slots, making it harder and harder for clients to get connected, causing further issues with node-isolation attacks and similar issues.  In the end, though its /possible/ to run the network largely on SPV nodes, its far from ideal, especially when the only reason this is happening is because of one, maybe two sites who insist on being lazy and, frankly, stupid.

Its an interesting idea, but it largely defeats the purpose of running a full node.  If you are gonna relay a full block, you really should check its transactions, or you are running the same trust model as an SPV node while telling other nodes that you are running a standard satoshi-client trust model, which could make it easier for miners to outright lie for short chains of blocks if it becomes to widespread.

I dont think anyone is saying we arent gonna move to more lightweight nodes, but the network needs a significant number of full nodes, or needs a broader base of full nodes with strong connections and are able to manage a ton of connections.

Hi Matt,

Thanks for that headsup. Having custom bloom filters on the server end of the connection would be far more efficient bandwidth wise. Will keep an eye on that discussion.

Jim

The first part of this message is not a constructive critic:

Many people (like me) anticipated it: It need just one disruptive use of Bitcoin (like satoshidice) to break the "all nodes are equal premise".

Now the constructive part: four possible solutions exists:

1- SPV nodes / full nodes distinction and related ideas.
2- Balance sheets (breaks compatibility)
3- Add a new transaction format of much sorter length (e.g.: no scripts, use of firstbits) (breaks compatibility)
4- Add a new (less CPU-demanding) signature standard (e.g. MAVEPAY or Merkle-Wintenitz) (breaks compatibility)

Check out the info at https://en.bitcoin.it/wiki/Hardfork_Wishlist (this page is not well cross-linked).

Best regards!

The all nodes are equal premise was broken more than a year ago when GPU miners started to become popular.  Anyway, the point of this thread is you dont have to break the ability of people to use full clients because one service refuses to work in a reasonable way.  Strongly encouraging miners to deprioritize high-volume addresses and making the satoshi client refuse to forward the full volume of such transactions puts a cap on basic stupidity.  Yes, they can simply switch to using dynamic addresses, but it does force them to spend a bit of extra time coding, and the hope would be that they might add something like multisend and hopefully allow users to carry a balance like virtually all other bitcoin sites.

Its obviously coming, but there are issues with a drastic network-style shift overnight, such changes should happen slowly to give people a chance to watch the shift very closely and make changes where required.
No one wants to hardfork for an issue like this, its simply not worth it.  Hardforking is a huge ordeal.

In any case, this topic has gone far off topic.  The goal was to discuss deprioritizing satoshidice and other similar transactions.

Deprioritizing sounds like a horrible idea. Who gets to decide who to limit?

Maybe 25 GiB/year isn't too much storage-wise, but it is way too much bandwidth-wise.

"Hey have you heard about Bitcoin?" "Yeah, last month" "Why aren't you using it? I find great deals every day on BitMit" "Blockchain's still downloading"

 ::) right...

The size of the blockchain will soon be a huge hindrance to widespread adoption

You're being too harsh on SatoshiDice.
AFAIK, they're paying fees for every transaction they send. In the end, they're contributing to the network security by financing miners. The fact that non-miners are having problems to follow the chain progress is less relevant IMHO, as that will necessarily be the situation in the future if bitcoin "succeeds".

I'd argue they're doing more good than damage with all these paying transactions. They're actually being generous, because if they were to use send-many and reduce their number of transactions, they'd be paying less fees to miners. (and about having a balance, I think they've been so successful precisely because they don't require you to have an account).

And if miners/pools are still accepting free transactions, it just means they don't give a damn to the overhead. (at least not yet).

Everyone deprioritizes every address that is heavily used, thats the point.  
If you think about the use of single address for high tx volume, pretty much all of it can be safely delayed getting into a block.
1. Green Addresses.  A. people shouldnt be using green addresses to begin with. B. the whole point is you trust the person sending, so being late getting into a block is no problem.
2. Poorly-designed sites that use single addresses to receive transactions from everyone (eg SatoshiDice).  uhh...thats the point of this thread.
3. Donation sites.  If you are getting a large volume of donations, I dont think you are worried about being able to spend each new donation now vs in 24 hours.

What? Thats 6kb/s.  Even if your node is only up for one hour/week, you only need 1134 kb/s to download the chain.  I think thats perfectly reasonable.  

Chain download isnt limited by bandwidth, not by a long shot (unless you start downloading from a very slow peer, which is possible, but something that should be worked around soonish).  Chain download is limited by disk speed, but mostly CPU speed at checking signatures.  Hence my statements that sites like Deepbit and SatoshiDice need to implement things like multisend which allow for faster checking of transactions as txindex.dat is able to have fewer keys.

Thats the point of this thread...

Miners/pool operators may prioritize whatever they want.

It is a horrible idea if done by the developers of bitcoind. I'm not comfortable even with the hardcoded fee policy left by Satoshi.

What might be acceptable at "bitcoind level" are configurations. Like, a file where you specify fee policy parameters, if you're a solo-miner/pool operator.


"Normal" users should not be using bitcon-qt.

Please, really please don't make that a rule embedded in bitcoind.


Really? CPU time checking signatures is slower than the database indexation? I always thought it was the other way around. At least my laptop seems to access the disk like crazy when it's getting up to date... I'll check CPU usage.

By the way, I heard you were working on a patch to make the chain update more asynchronous, and thus faster, and that you were obtaining good results. Is that so?

The tiny fees they pay are very, very, very far from paying for the damage they are doing to people's disks/cpus/bandwidth.  Miners (currently) have plenty of finance with 50BTC/block, but even if we switched to 0BTC/block tomorrow, the amount that satoshidice is paying compared to the load that they put on mining bitcoinds is very close to not worth it to the point its almost easier to just drop them and not have to worry about spending too much time generating blocks.

Being generous to miners does not help end-users.  My point here is the incentive structure is really a deal between transactions senders and miners, without thinking about end-users.  Hence why end-user nodes could deprioritize forwarding high-volume address transactions to tweak the incentive structure.  

Agreed, but sadly, in the end, its hugely to the detriment of Bitcoin as a whole.  Hence why Im suggesting we adjust the incentives to discourage bad players like SatoshiDice.  That said, using multisend would not require any user-facing changes while lowering the load on bitcoin clients significantly.  

It means they are interested in supporting Bitcoin's growth.  In fact, because of SatoshiDice's volume, free transactions are being force out of blocks, when it is important for users to be able to use Bitcoin.  The idea is to deprioritize high-volume address transactions so that regular users can get their transactions into blocks in reasonable time, while sites that dont need fast confirmations (like satoshidice) can have their transactions fairly delayed.

Exactly! I am glad someone else gets it.

Right now the BTC network is very small, but the VISION that we have all invested in is HUGE -> "BTC replaces dollar/VISA/Mastercard".

To get there you can't have the network breaking down over something like satoshidice NOR miners starting to ask ridiculous fees.


If you just want BTC to be another small time linden dollar with no future wtf are we all doing here?


Yes a bunch of super computers COULD run the BTC network at VISA size with normal users being light clients, BUT that would leave us wide open to attacks, being shut down or miner-dictatorship = VISA.

Pruning needs to get done or yes bitcoin might not die per say; it will just become pointless.

Well, it's the official client for one.
For two, the more people that stop using a "full" client, the fewer full client nodes we have, and the less secure the network is.

Why not? the point is its to the benifit of the people running bitcoind, so its in their best interest to do such rate-limiting.  In the end, its really less of a deprioritization and more of a "stop-DDoSing me, Im gonna rate limit your transactions"-feature.  The goal is to tweak incentives so that the end-users and those running forwarding nodes can have a say in the incentive structure, as is ultimately fair, as they are putting in way more effort than miners in terms of verifying transactions.

Its a bit of both, and really depends on the disk.  On an SSD/tmpfs its more CPU, on a spinning disk, its more disk.  Note that multisend helps with the database lookups a /TON/.

https://github.com/bitcoin/bitcoin/pull/1233 (https://github.com/bitcoin/bitcoin/pull/1233) Depends on where in the chain you are, but it does help some.

This should be a non-issue in the long term:  If the network can't handle 50k tx per day, Bitcoin was never meant to succeed.  The only reason it's such a big issue right now is because it's a single entity producing the volume -- and thus there's somewhere to point the finger when it was really our lack of preparation to handle it.  It could've just as easily been a piece of some other market that latched onto BTC and produced 50k tx per day in a less-centralized manner.  

It's a downside of having a completely open, decentralized, apolitical, global currency scheme -- people must have the ability to use it as they please, and the rest of the system must find equilibrium.  Talk of deprioritizing certain behavior, especially targeted at a specific service is completely counter-productive and inappropriate:  people who want to repeat the discouraged behavior will find a way around it, and other people who are otherwise legitimate may suffer by being inappropriately affected by the change.

In my mind, the only reason this is an issue is because the community was not prepared to handle the rapid increase in size -- and it will work itself out once users/dev figures out the way to accommodate it.  There just may be some turbulence in the short-term.  

A quick question.

Does downloading of few kilobytes imply a connection to full nodes for that or the same lightweight nodes might also happen to have this info?

If network is mostly populated by nodes who are asking for those small downloads and most of the nodes don't have the data then where is it going to come from?

From trusted nodes.

The point isnt that we cant handle the volume.  We clearly can.  I dont see Bitcoin dying right now.  The issue is that its much better to have the network /slowly/ transition into SPV nodes so that we can closely monitor and make sure there are enough listening full nodes.  Additionally, events like this are important as it lets us reexamine the existing incentive structure and address issues like what is, to most end-user essentially a DoS attack.  The issue isnt that we have a place to point a finger at, its that one user is abusing the network.  Im suggesting a way to discourage people from abusing the network in such obvious ways, hopefully encouraging people to use more sane send-methods.

Actually, its not targeted at a specific service.  Its targeted at all services that have high volume and can reasonably wait for confirmations.  I never said it would be hard for people to work around the deprioritization, quite the opposite.  But it would force people to think about what they are doing.

Thats what Im suggesting - a way to attempt to accommodate the rapid increase - make people who insist on DoSing the network in an obvious way wait longer for confirmations.

+1

I don't get this talk of alternate chains or why it should be so damn difficult.

If there is a block with an invalid tx someone reports it over the peer network. The bandwidth usage, even if you have send the documenting block, would be negligible as it would happen so rarely.

To get your own balance you query your peers as with torrents and if there's an update they send that block to you.

To work you would need more than 8 peers/a smarter protocol, but that's hardly rocket science.


That is like 2-3 updates, what reason NOT to do it?

If it's just a "flood defense" then I'm fine with it, as long as it is configurable. Like, in my config file, I add that a node sending >10tps during >3s is flooding, so I stop downloading some or all of his tx.
Such a thing could finally allow the devs to drop the hardcoded transaction fee policy, which should never have existed IMHO.

Now, where I strongly disagree with you, is that SatoshiDice is flooding the network. It is not. Their transactions are legit. Something should be considered DoS if that's the sole goal of it. Like, someone creating as many transactions to self as he manages and broadcasting them all, only to flood the network. That's bad behavior.
SatoshiDice is not flooding and is legitimately paying for his transactions. Transactions paying something above a certain low limit in fees should never be considered DoS - and this limit should also be configurable.

That would apply to Electrum/Stratum architecture, but there is no such thing as a trusted node in etotheipi's proposal.

That is an interesting change to the trust model...But it is significantly easier to simply have a full node/SPV node distinction and have all full nodes do the checking.

So...SPV clients after the Bloom filter change...

>8 outgoing connections is not really an option.  Last I checked (which was a while ago) the ratio of open incoming connection slots on listening nodes to outgoing connection slots on all nodes wasnt great.  Even if it is, increasing the outgoing connection slots on every node has a huge effect on that ratio, and it would be very easy to blow through the listening slots available on the network.  Even hurting that ratio makes some attacks much easier.

Flood defense against individual nodes doesn't really make sense and is just punishing that node for no reason.  The point is to protect against flood from particular addresses.  Also note that it doesnt really effect the hard-coded fee policy which addresses radically different aspects of a tx's "badness".  That said, I do agree that hardcoded fee policy is bad, but its really there to prevent other trivial DDoSing, as most standard txes should never hit that.

SatoshiDice isnt flooding the network because they have high volume, they are flooding the network because they refuse to use features like multisend which would keep their network load down, while still allowing them to have the same volume.

There's nothing "official" in Bitcoin, and the fact that you think there is shows an understanding issue.
Saying that bitcoin-qt is the "official Bitcoin client" is like saying Microsoft Outlook is the "official POP3 client" or that Firefox is the "official HTTP client".

Granted, it is the "reference implementation", the only one so far which fully implements the protocol, and as consequence, the one which actually defines it. That's different from being "official" though.


Network security is much more related to the amount of computing power behind mining than the amount of full nodes in the network.
It's unlikely that someone manages to DDoS or hack all full nodes at the same time, even if only solo-miners and pool operators were running full nodes.

And, please, understand: if bitcoin succeeds, it is just a matter of time until this happens (few full nodes). If that really makes Bitcoin less secure as you say, then you may say Bitcoin is not secure by design.

Err... for the reason of punishing flooders? If you're relaying the DoS you're a flooder yourself too.


But then the flooder just generates more addresses. What's the point?


But it is avoiding these "trivial DDoS" I'm talking about. But doing so via download limits and all, not via fee policy.


Regardless, they're making a legit use of the technology, not a DoS attack. Specially if they're paying for it.

The idea is to avoid the need for full nodes: I want a 1 tier network.

After re-reading the scalability wiki it seems you could split the block up in to a list of hashes.

This ALSO means you should be able to document an invalid tx with just a few hashes sent along with your report.

Yes except they DO store part of the chain AND they do verify against those parts.

Alone none of the nodes are full, but together they act as a network of full nodes.

Alright my bad.

In that case do "peer switching" if all your 8 peers don't have what you need make them give you one of their peers that DOES.

That way ALL nodes could be storing/sending/verifying >>1% and still function as a full node network with the same safety.

EDIT: Miners would still need to build and send full blocks, however building said blocks could become outsourced to mining pool clients. (from the scalability wiki)

Why are you all so eager to get the network comprised of only "trusted nodes"(trusted by whom?) and "lightweight clients"?

Way to kill decentralization, IMO.

I shall not be paying for your silence ;)

You are the one who insists it be configurable as to how much you forward (and I, mostly, agree with you).  You cant have it both ways or you end up punishing everyone and dropping connections.

As stated above, punishing high-traffic addresses by no means will remove the potential for the problem to occur.  The goal is really because people who do use one address usually dont care about confirmation times, so they can gladly still use them and some might.  Really its to make people think twice before designing their site in an entirely braindead way.

Doing so via download limits removes the possibility of users using Bitcoin for some things that may be very desirable.  Forcing fees before forwarding transactions that we otherwise wouldnt forward at all allows those transactions to exist.

a) they arent making a legitimate use.  Arguing that it works under the current rules is not a reason to not change the rules to make it not possible.
b) they aren't paying for it.  Paying miners does not help or help the high load it inflicts on end-users.

You can't verify against part of the chain.  You can say "this transaction's signature is invalid" but you can't say "this transaction's input doesnt exist".  Also, even if you have a few full nodes to check for this case, they cant prove it unless they send the full chain...

Exactly. What's that Satoshi quote? That governments are good at cutting off the heads of things like Napster? Well... how many trusted nodes are we talking about, who's going to protect their heads, and how do I apply to be a trusted node so that I can mess with the blockchain later on?

EXCELENT post etotheipi! At least there is some self-criticism in the community!

This is what I've been saying in so many posts: we are not prepared.

If just one single entity made such a mess, then think about 10 more satoshidices beginning to operate tomorrow.

That's why I designed MAVEPAY. I'n not saying that MAVEPAY is the only solution, but people should start accepting there is a scalability problem.

SatoshiDice seems to be a great way for core devs to get a taste of what it's like to handle a volume like this when bitcoin network starts to scale up and think of the solutions in advance.

And what is the solution they came up with? Kill the volume maker!
Doesn't sound good to me...

Stop.  Thats not what I am suggesting, nor is this something that is being extensively discussed, I started this thread to see what the community would say in response. 
The goal is not to kill anyone or prevent them from using bitcoin for their use-case.  Read the thread please.

So what's going to happen when BitInstant gets rapid bitcoin->cash conversion working and somebody gets the idea of making a mobile frontend to SatoshiDice that acts as a lightweight wallet and interface to BitInstant behind the scenes? How would something like that affect the Bitcoin network if it became popular?

If its done right and uses multisend, it would be a perfectly manageable load on the network.

More than complainting about SatoshiDice people should be complainting about the biggest pool not using sendmany and only processing around 100 transactions each block, sometimes even less, which I wouldn't be too surprised if they are comprised mostly of it's own transactions to pay miners, even when there are thousands of transactions on the memory pool.

That's the free market at work.  Deepbit can choose to confirm or not confirm whatever transactions it likes.  If you need your transactions to go through quickly, then send it with a fee.  If you are just trying to bring awareness to the issue, then that's probably a good thing - knowledge helps people make better decisions.

My opinions about this so far:

- SatoshiDice is legitimate. It may not try its hardest to make the most efficient transactions, but you will always have clients who aren't perfect. SD just does what might happen in 2-3 years anyway, but by lots of different users. SD can use the system only within the given limits; if it would do something wrong, transactions would get rejected. Basically it boils down to the underlying protocol restrictions: spammers for example are really hated, but they operate fine within the SMTP-RFC.

- Someone mentioned "trusted nodes": that's what SolidCoin advertised so heavily. And what failed. Giving control to few nodes might blow everything up.

- Restricting the heavy users won't help either. That's similar to the ISP who advertises with "unlimited downloads" but later throttles you when you download >1GB/month.

What about changing the way the blockchain is stored? Instead of having everybody store the full chain, they could store only 1-2% of it. 50-100 users then would have the entire chain, and with tens of thousands of active users, there would still be hundreds of sources for each block.

Clients could join a swarm when connecting which sort of represents the entire chain. Although clients would only store fractions of it, the entire blockchain is always available. Pretty much like a Bittorrent swarm is still fully functional as long as every single block is at least on one seeder's machine. You'd only need a small index file (think of blockchain.torrent) which handles the checksums of blocks (note that I'm not talking about bitcoin-blocks, but data-blocks, eg 10MB chunks). Add an option to let users decide how many % they want to store and voila, someone can decide to provide a full copy.

+1.  SD could even partner with DeepBit sending them all their tx and paying a monthly fee for priority service.   The market is wide open.  The days of every pool accepting every tx on every block is likely coming to an end but honestly I find it interesting.

SatoshiDice is the free market at work also. They chose to use the blockchain to make a game following the Bitcoin protocol and paying fees for each and every transaction they send. What was your point again?

And, no, I'm not happy that I'll probably have to take out my bitcoin client from my main laptop and put it in it's own dedicated machine, but what must be done, must be done.
I will not trust you or anyone else to run a "trusted node".

Sorry, but that's the impression I got because you also called them stupid and braindead.

From the quote below it's seems they are also suffering from the same issue you mentioned,
if "use transactions in a more conservative way" means avoiding multisend.
They are a young service and hopefully they will work this out.

I agree.  The market is made up of suppliers (miners) and consumers (tx creators).   No reason to complain.   If you don't like their tx spam require higher fees.  If enough miners do it then their business model will need to change.

The current situation is like having an all you can eat buffet and complaining people are going back for seconds.

I think the point is a free market is made up of both parties.  If SD is free to spam the market then DB is free to ignore those txs.  One can be wrong without the other one being wrong.  Personally I think neither are.

I was responding to your original point, which was "people should be complaining about the biggest pool not using sendmany and only processing around 100 transactions each block."

My point was, no they shouldn't.

Again, the point isnt to restrict, its to encourage them to think twice about what they are doing and encourage them to do simple things like sendmulti which make a huge difference to the load they create across the network.

Read my response to Realpra, who suggested the same thing.

Hi, I am the programmer working on SatoshiDice.

We are absolutely interested in being good bitcoin citizens and doing things in a good and efficient way.  However, changing to handling many bets with a single transaction is a complicated change for us.  It is certainly doable but has to be done carefully because making sure people get paid properly and accountably is our business.  It will also make it confusing for our users.  Right now seeing that they payment was correct is simple.  They see one output to their address of the correct amount and one output to one of our change addresses.  If we combine transactions, they will see a ton of outputs to various addresses.  It gets even more confusing is the same address is getting payments from multiple bets in the same transaction (which will be a common use case, people bet a bunch at once often).

Of course none of that is a deal breaker.  We can do it, but it will likely be a little while before I can get the code in and tested.  I'll also have to think about making improvements to our UI to help users make sense of what they will see in these big transactions.

Guessing from their website that is not whats happening.  AFAICT, their site posts the result of a bet before sending the coins.  Implementing multisend is really dead-simple and would have little effect on payout times if done even close to remotely properly.  Doing multisend is a simple as queuing outputs and running a cronjob to send coins, even if they did it every minute, their total transaction volume would decrease by something like 500-1000%

I am curious. Has anyone checked how much more fees are paid to the network because of Satoshi dice (so both the fees they pay themselves and the fees their clients pay). Fees starting torise is a good an healthy thing and Sathosidice could be a catalyst.

So anyone that can show me some numbers? :)

Incidentally, SatoshiDICE only employs high-traffic addresses (1diceN*) because that is convenient (no need to get a new address for each wager).  But there is no dependency on using those addresses.  The service can provide a unique Bitcoin address for each wager and it would technically function just as well.  If an API for obtaining a wagering address were made available it would be trivial for automated wagering (e.g., martingale bots) and the new user interfaces (e.g., mobile apps for wagering on SatoshiDICE reportedly being developed) to switch over to that.

So punishing high-traffic addresses won't necessarily have much of the expected effect.

I understand that people may see more outputs in their transactions, however in most (sane) clients, they only show the single output to the user in the transaction history.  Thus, though the amount paid back to satoshidice may be harder to determine directly, I dont see any reason why users care much at all about that amount.  If Im betting, its not hard to figure out the amount satoshidice kept by subtracting the amount I gave SatoshiDice from the amount I got back.
This admittedly complicates things, but if you really wanna keep it simple and more verifyable, you can simply not combine to-user payouts and keep those in separate transactions.
Somehow I dont see how throwing payouts in a cronjob and changing the existing payout functions to just put them in a list of to-be-sent takes more than a few hours to implement.

So far what I gathered from this thread.

a) Transactions volume increased dramatically because of a single website, SatoshiDice. Transactions volume size was suppose to reach that level sooner or later, but in decentralized manner. The proposal is a short term solution to the problem, that utilizes the centralized manner of generated transactions, and proposing to discourage a website from generating high volume of transactions. The reasoning is that such behavior is abusive and does more harm then good to bitcoin network as a whole.

b) First, by definition short term solution is bad because the work to implement it would have to be thrown out the window soon enough. Second, it limits the usefulness of Bitcoin Network. SatoshiDice, or any website with simular behavior, is useful because people who uses it find it so.

Any short term solution implies that something it has to be done right now rather then later. Is the situation so dare that the proposal has to be implemented despite all the negatives? And do you think that the current problem is worth getting into regulating of how Bitcoin Network used?

Agreed, but, again its something that will make people think twice about their system-design.  Also, thats why such transactions would be rate-limited and discouraged and not blocked, which allows people to opt into being low-prio.  Someone like satoshidice should be perfectly willing to do this as their users are using unconfirmed transactions anyway and it would let them give other users who are using one-off transactions higher priority and have less of an effect on confirmation times for end-users.

Hmm.. forgot that...


Similar to "my idea", but see other guy above.

Basically your idea would work, but to verify the inputs of transaction existed you would have to go through the entire chain in the cloud = huge bandwidth usage.

So: "Houston we have a problem"?

Hiding behind TOR or something becomes hard with "trusted" super nodes and VISA volumes so unless we solve this, BTC will remain a small network forever and/or get busted up by governments.


Could we change the nature of blocks so that payments are not made up of only transactions, but also balances?

We know the block chain is trustworthy due to work proofs -> Just put address balances in it?

Could we do this without forking? "BIP 34" or something?

That way I could store the balances of maybe 1000 addresses and I simply verify transactions going out from them? IE verifying inputs against a partial chain?

Bad idea.  The power of a 51% attack is limited to only tx denial and double spends.  If balances are updated based on tx in most recent block without back validation a 51% attack (or even a minority attacker which can generate multiple blocks in a row) could mint an unlimited amount of fake coins.

Thank you for your careful and considered estimation of the time it will take to implement a feature in my code, which I am almost certain you have never seen.  I will treasure it always.  

Well, if SD is doing 30k transactions a day, and paying a 0.0005 BTC fee on each one, that'd be an additional 15 BTC of fees for the network each day.

Essentially.

The proposal isnt really that much of a short-term solution.  It is designed to encourage users to not use more transactions than they need to.  Though this is short-term for capping the chain, it should continue to drastically decrease overall chain volume even as that volume increases greatly.

Again, no.  It doesnt prevent satoshidice from existing, it encourages them to be more friendly to the bitcoin network as a whole.

No, but starting a discussion is required to ever move forward ;).

This isnt regulating how bitcoin is used, its another incentive change.  Or, more simply, yes.

Yes but the first two I checked manually were 0.0005 and 0.0001 so I was curious anyone has a programmatic solution to automate the process and come up with non-estimated numbers.

I understand your code may be complicated and it may take time to implement a change to payout methods, but, again, multisend is a very simple change.  In any case, I would argue it is a very important change as well that should be very heavily prioritized.

I don't think it is that dire no.

If it was, fee rates would have shot through the roof and killed satoshidice.

What is needed is a long term and scalable solution to higher BTC volumes. Fees will kill spam just fine until then.

Okay, how about this then:
1. The blockchain stays as is.
2. SPV nodes maintain knowledge of the balance of some addresses. They also maintain the documenting tx/block trail leading to this balance.
3. Invalid transactions can then be reported using this information (inputs).

Basically rather than storing chain links you would store chain strands?

The full block chain could then be constructed using strands, SPV nodes can verify and we have a 1-tier network!

After thinking about this for a while I actually now believe it might be a good idea even long term. Since there is only as many transactions than can be put in the block, the rules might need to be in place for which transaction to include.

The question is do you Matt have a good understanding how to implement this prioritization and how to enforce these rules. I get the impression that miners are the ones deciding what transactions to mine. Can the rules be enforced in the protocol?

What if the blockchain itself gets changed? As in, not only changing how parts/all of it are stored, but also its format.
If a client needs to verify an input, it shouldn't have to scan the entire cloud-chain. Instead it sends off a request for input validation to the network, and e.g. the clients who store those blocks which prove the inputs will reply with an udp-ACK. With enough ACK's from different clients, it's safe to assume that the input indeed exists. Kinda like DNS.

Isolation attack?

DNS isn't anonymous.  It is a poor model to use as a comparison.

The idea suggested is this:
Each user of bitcoin has the option to limit transactions involving a particular address to n/memory pool.  ie I may say "I only want to ever have 1 transaction with any given address in my memory pool at any time."  Thus, I would be rate-limiting each individual address  to n per block, assuming they propagate through the network instantly faster than it takes to create a new block.  The idea here is to keep high-volume address transactions from reaching miners in the first place, making them impossible to mine.  Obviously people who wish to continue to use such volume addresses could peer directly with miners who accept such things but, again, that would require a level of effort much higher than simply using something like multisend.

Then SD would just start using unique addresses for each bet.

Please read the thread.

Didn't understand this.


It doesn't make sense punishing someone for using the same address multiple times. If you're a solo miner or pool operator and want to apply this rule to your blocks, be my guest. But as a network policy, it doesn't make sense. You're just picking on a particular way of spending bitcoins that you consider "braindead" and trying to punish people who don't do as you like.
bitcoind should not contain personal preferences like that.


I don't see a problem in forcing fees per se, I just don't like that it's implemented in bitcoind. A particular fee policy should not be a "implementation reference". At most, it should be configurable.


Please, of course is legit. They're not attacking the network, nor trying to cheat, double-spend, >50% or anything. They're just spending money in a way that's bothering you and making it harder for non-miners to do what they don't need to do (have a full node running).


They are paying for it. I repeat: end-users do not need to run full nodes. Only solo-miners and pool operators need. They are the only ones who should care about SatoshiDice load of transactions, and apply their arbitrary rules if they feel like.

If an user which is not solo-mining nor operating a pool is using a full node, that's his choice. His choosing to dedicate his resources to the network, charitably. He'll handle the load in exchange of no monetary incentive. He should know all that.

Wanting to embed such an arbitrary rule in the reference implementation is really not appropriate. The Bitcoin protocol per se should be totally agnostic of whether people use multiple addresses or the same address every time.

Why does the network need to rely on "trusted nodes" ???
What's wrong with just "nodes"? Anyone can be a node, that's part of the point. Once we get to the point where people mostly use lightweight clients, a client which wants the whole blockchain can keep connecting to nodes and asking for a list of peers until it finds one with the whole blockchain, or half, or whatever. It starts downloading from that one while looking for more peers to verify against. The concept of a "trusted node" is antithetical to Bitcoin's goals.

I already did.  What did I miss?

Other results of such a scheme:
1) Green Addresses: users who are using green addresses a) are usually generating significantly more transactions than they otherwise would, so discouraging green addresses (which this does) is quite nice. b) those who insist on using green addresses are using a different trust model entirely and dont care about transaction confirmation time, so using green addresses would be opting into slower transaction confirmations (which is entirely fair).
2) Forcing users to use non-rotating addresses is already poor for user privacy as it makes it clear what you are using your coins for, so discouraging such usage would help user privacy, in the end.

See? These are your personal preferences.

I agree with item 2, but I don't want to see a "network rule" of some kind forcing people to do so.
The protocol should be agnostic.

My HD jumped off my computer and went to talk with his lawyers.

The reasonable way (and really the only way) to punish users who are DoSing you, is to simply disconnect from them.  If you make it configurable how much "single address spam" transactions you relay, punishing the relaying nodes would mean disconnecting from users who chose to relay more and reject the discouraging of such transactions, which largely removes the choice in the matter.


The point is that miners have no incentive to do so.  The harm is done to the network as a whole, so the goal is to provide the individuals in the network a chance to effect the incentive structure of transaction generators.

There are many disadvantages to users when single addresses are used, but it makes programming a bitcoin site simpler.  Thus, by discouraging such behavior, we can discourage both the behavior itself and hopefully make people designing bitcoin sites put in a bit more effort, in the hope that that effort results in them putting in the effort to do things like multisend as well.

It being configurable I can't really argue with, but the possibility of hordes of people disabling the fee rules without fully understanding what they are doing is what, I believe, many people are scared of, and I think could become a big issue.

Its actually not bothering me at all.  I usually run bitcoin on tmpfs, it takes me no more than an hour to sync fully from 0, and I have plenty of memory left to keep it up.  What does bother me is that they (being a loose term not just for SatoshiDice) refuse to implement very simple technologies that hugely decrease their impact on Bitcoin as a whole.  This, I do call illegitimate.  Refusing to "play by the rules" should be punished, even if its just discouraging them a bit.

Again, a drastic change to a SPV-heavy network overnight is by far not a good thing.  Though end-users do not, by any means, need to run full nodes, for now, they do, and putting heavy load on all of them to essentially make them switch over quickly is not good.  Also, users who use p2pool or other getmemorypool-based mining (which is something that should be very, very, very heavily encouraged) have to deal with the increased load and often do not have the large hardware resources to handle such huge transaction volumes.  You can clearly see the result with the very high orphan rate in p2pool users.

That is fine by me, but we shouldnt make it very, very difficult for people to do so overnight.  Also keep in mind that pool miners (not operators) should be very, very heavily encouraged to use getmemorypool-based mining which requires using a full node.

Whether its the "reference implementation" or not, it is one of the most popular clients, and putting rules in it that are good for the users running it makes sense.  It is entirely up to other nodes whether they want to implement such rules.  Keep in mind that the propose rule here is really a very soft rule, in that it only discourages transactions, there are a ton of workarounds to it and if you happen to get peered with a path to a pool on which this "rule" is not implemented, it will entirely not effect you.

Uhhh...no.

The protocol remains agnostic, again this is not a hard rule.  It is entirely configurable, not all clients have to implement it, and it is just a discourage rule, not a hard rule.

Wouldn't this create more trouble for users/services who are unaware of this change?
They will have one more head ache to figure out why on Earth their transactions are not being propagated.

Would it be easy to follow these rules in a reliable way?
Say if I'm trying to play by the rules and the timing is a bit off my transaction will get stuck somewhere and I will have to use wallet tools to remove it.

By proposing to take measures in reducing transaction volume as a long term strategy you get in the middle of transaction fee versus block reward as incentive to mine. Do have more details of how it would work?


We can only measure usefulness in how many people using it, if the change in how a site operates, not just SatoshiDice but any, would decrease it's user base, e.g. because half of potential users won't go through the hassle of creating account, then it would a detriment to Bitcoin Network usefulness.


Some day Bitcoin will face a problem that could only be solved by introducing regulations, I am just not convinced that SatoshiDice and its high transaction volume is that kind of problem since there viable alternative solution: Special blockchain pruning method by etotheipi (https://bitcointalk.org/index.php?topic=52859.msg885838#msg885838)

Because the rule requires people to upgrade their nodes, it would take a significant amount of time to begin effecting transactions.  Thus there will be plenty of time for people to change off of their previous payment designs.  Obviously such a change would be heavily announced.

The proposed rules do not block any transactions, they only effect the time it takes for transactions using the same address repeatedly to confirm.

The proposed change does not impact all transactions, only a small subset of users and anyone who is in that subset can easily change, and does not consider fees. 


The proposed change will not force any sites to make any major user-facing changes.  It only effects Bitcoin-site programmers, and only in a fairly minor way, but hopefully a meaningful one.

That topic was discussed earlier in this very thread.  Also keep in mind this is not regulation, this offers users of Bitcoin-Qt/bitcoind the option to more directly impact the transactions mine, (really) as satoshi originally intended.

That's probably a bigger issue than the one we are trying to solve. Relying on the fact that people out there are always ready for any changes and announcements is a mistake.


You mentioned that the idea is to prevent these transactions to get to the miners in the first place. So I'm afraid that more transactions will get lost/stuck.
Also discouraging certain behavior by making certain transaction to get stuck/lost doesn't seem right. We already have complaints from users who have their transactions lost due to incorrect fees, this will only add to the problem.

They have "no incentive" because they're earning money with SatoshiDice. No harm is being done to the network. Only those that want to store and validate the chain "for nothing" are having to wait a little more to get up to date.


That's relative. For example, I honestly think that what SatoshiDice is doing is rather positive to the network security, as they're paying fees for their transactions.


What does please me is that they (being a loose term not just for SatoshiDice) choose to implement very simple technologies that hugely increase their donations to Bitcoin as a whole.

See how this is relative? ;)


Illegitimate implies "cheating". They are not cheating. They are perfectly playing by the rules.


You sound like those who are fearful of ASICs.
Define "drastic change", or "switch over quickly"...
I'm still running my full node on my primitive laptop. And I'll probably remain for a while. I am not sure this migration is going that fast. Bitcoin evolution as a whole is even faster.


But they are being payed for it. If what they're earning as fees is not enough, maybe the devs of p2ppool should change the fee policy of the protocol to require an amount in fees that pays off the inclusion of a transaction by the average p2ppool miner.

Or perhaps there should be centralized pools operating as nodes in p2ppool, allowing miners not to have the full chain. The advantage of such is that, by operating in p2ppool, the pool operator clearly shows that he's willing to follow p2ppool rules, and thus cannot "go rogue".

Even if satoshidice used a single sendmany each hour to pay out all bets, they would less than halve the number of SD transactions on the network.

Currently 50k bets per day = 100k transactions per day.

With sendmany, 50k bets per dat = 50024 transactions per day.

"Encouraging" them to use sendmany just to gain a factor of less than 2 reduction in the number of transactions doesn't seem very useful, given the rate they're growing.  It's only a week since their volume last doubled, so by switching to sendmany you're maybe gaining a week until the network can no longer handle their volume (assuming continued exponential growth until the network explodes).

Not true.  The 50k to sd txes are very easily pruned, whereas the 24 from sd txes are not.  In fact, because many of them are int he same block, those transactions never need to be added to the tx index at all, speeding up block download significantly.

It might be worth considering transactions that self-identify into certain traffic classes, similar to the hints provided by IP ToS / DSCP (http://en.wikipedia.org/wiki/Type_of_service).

Yea, the complete apathy towards upgrading in the Bitcoin community is really quite sad and quite bad for the network's overall security.

You dont prevent it, you slow it.  If you (as all existing nodes I know of do) reannounce your transactions, they will eventually get there.

@caveden I think we are going to have to agree to disagree.  In the long term, yea of course the network is going to transition to SPV nodes a ton, and things like satoshidice wont make as big a problem for users, OTOH, we want to encourage getmemorypool-style pool miners, so we really want to keep it as easy as possible for people to run full nodes.  In any case I think the many, many people who come to #bitcoin and #bitcoin-dev and complain about how long it takes for them to sync the chain would disagree that SatoshiDice is causing no harm to the system.

Im torn as to whether or not such a system would work for Bitcoin.  It works for IP (IMHO) because application programmers get to set the flags, not end-users, or, at least its something that is well outside of the effort for end-users to increase their own ToS.  For Bitcoin, what would make sense (IMO) (as stated by gmaxwell recently on IRC) would be to have a ToS which is lower than 0 fee, and then let fees do the rest.  Thus the only users who would set such a flag, would be large firms who have some incentive to make their transactions confirm as fast as or faster than their competitors.  OTOH, they also have some incentive to make sure Bitcoin operates well for end-users and those who want fast-confirming transactions.  I could see a scenario where users complain to such services about how long it is taking them to get their money, encouraging these firms to disable this flag and potentially pay fees to crowd out 0-fee transactions.

Ok that's the crucial part that I misunderstood. So indeed this will only slow but not prevent the transactions from propagating.

However for the end user it might create certain complications under specific circumstances.
Imagine Alice accumulated some amount of coins from her signature address on this forum.
She then decided to make a few payments and go to the party, so she sent a few transactions all from the same address in a short amount of time: one to bitmit, one to her friend and one to her mother she promised long time ago, then she shut down her computer and went partying.

Now with these rules in place (considering 1 tx from 1 address per block) her last two transactions were lost and she wouldn't figure this out until after she starts her computer again to re-broadcast them, which might take a few days or until her mother calls and asks where is the money :)

Yep.  But thats why the actual tx/mempool is variable.  I think 1 is probably too low for the reason you point out here.  But I dont see many people needing more than 2-3 at maximum for any given use-case.  A sane default, IMHO, would probably be around 3.

They seem to be using a % fee.. bigger transactions take bigger fees.  This bet (http://www.satoshidice.com/full.php?tx=c6d4eec1d3f959e6362c7471b0d1583ea3757b8ba5451c23e3e3b4bc0304f39b)  as an example.. 0.01 (http://blockchain.info/tx-index/7573043/c6d4eec1d3f959e6362c7471b0d1583ea3757b8ba5451c23e3e3b4bc0304f39b) on the bet, > 1btc (http://blockchain.info/tx-index/7573083/4ca2f3687f2c978d7eede875daf499808651ea737232c8abfe4e1cfb7e5f5502) on the payout.

It is actually max(0.0005, (ins + outs) * 0.000075) so is fairly stupid but is higher for larger transactions.

Code is here: http://code.google.com/r/fireduck-magicpony/source/browse/core/src/main/java/com/google/bitcoin/core/FeeFinderCounting.java

I'm still concerned this will have more impact on legitimate users or small businesses who process their transactions manually than on high volume makers.

New rules do not discourage the volume itself but using the same address in that volume.
So the change seems to be targeted to the current implementation of SatoshiDice rather than being a more generic solution.

Under new rules SatoshiDice will have to change that's for sure and there are two ways:
1) use sendmulti which is benevolent
2) use new addresses for each bet which is basically bypassing the new rule.
They have already admitted that they want to be a good citizens and they will put efforts to implement sendmulti, so introducing new rules just for them (or rather against them) doesn't make much sense. On the other hand players which choose to be malevolent will simply bypass the new rule by using new addresses if they wish to.

So is it really worth the time and effort, which otherwise could be spent on thinking about proper pruning solutions?

AFAIK SD only pays fees because of limitations in bitcoinj, not because they have to or want to.

How to solve this in 48 hours: release a client that requires a mandatory .01 BTC fee/transaction and get the major pools to use it. SD would be forced to change to sendmany, or fail within 2 days. It would also be a completely generic solution.

And it would also alienate a huge user-base which like bitcoin because of its 0 fees.

Don't do anything. When the system is failing (I doubt it will) miners will start to require ever increasing fees (hopefully proportional to the amount of the transaction, I have always hated minimum fees and still hope that will be removed) until a stable equilibrium is reached.

* In brackets are my personal opinions which are really quite irrelevant

So fuck 'em. 0 fees were never sustainable.

That being said, one option is for miners to offer a webpage where you have to enter a captcha and a transaction id and it'll let that transaction through into that miner's next block for free. Even better, this could be installed into clients and any miner who wishes to trust that the captcha server isn't lying about the captcha being entered successfully can subscribe to that feed.

It seems that the thing we're trying to discourage is automated transfers, and that would do that.

The point of this thread is to get the same result entirely without fucking them.

Not at all.  Im trying to discourage high transaction load generated by sites with low userbase compared to that load.

The userbase for SD seems to be proportional to the load they generate.
Also will it affect MtGox if many users happen to withdraw from a single address in MtGox wallet?

Clear you mind. Return to the point when you discovered Bitcoin with a
fresh mind. Got it? No more minor BIP-x or anything, just Bitcoin.

Now, what would be better for the network? 100 transactions a day, or
100 000 000?

How about 10 transactions?

Bitcoin needs *more* transactions, not less. Last year at the same time
there was even talks about how the block limit was way too large,
resulting in almost non-existent fees.

I agree we can want to find better ways to store the data, but getting
less transactions is not the goal. In fact, if Bitcoin is to be even
mildly successful, there will be a lot more.

I think the only reasonable way to encourage sendmulti is to make it cost less in terms of fees compared to doing it separately, which as I understand how it works already, no?

Enough said. Delaying transactions for high-volume addresses will only cause cascading delays for senders/receivers to those addresses. This then has an adverse affect on user experience, potentially for the very service or cause new users came to bitcoin for. Additionally, we want to support an increasing velocity of money in the bitcoin economy, which currently is moving at a glacial pace, even with SD. Blockchain's cost per transaction (http://blockchain.info/charts/cost-per-transaction) shows that at current USD exchange rates, even with SD volume each transaction is costing the network $.75. Why are we spending so much time complaining about a growth that is truly moderate compared to what the network will need to support? Hell, I hope that within a few years there will be 100+ bitcoiners in every major city, and potentially even a few towns that use bitcoins for market day... If we can't restructure to handle the minor load of satoshi dice, how are we going to support that large but STILL MINOR increase in volume?

Less *oh noes! lets keep us the bandwidths and disk spaces* and more preparing for the future.... we need a roadmap with a defined timeline and approach for blockchain pruning. We may even need a longer term plan for sharding, possibly via a tiered blockchain approach. I'm not sure what the solution is, but I am sure that one will be needed.

-bgc

I'm not trying to flame but what I've been thinking since this thread began is: lazy Bitcoin dev is lazy.
If developers think blockchain  size is turning into a problem maybe the answer is you should start thinking about how to prune it, not how to reduce the amount of transactions.

It's in times like this that I wish I was smarter and could help, but unfortunately that's not the case.

PS:  I'm very thankful to all the developers. If it wasn't for them I wouldn't be able to take part on a movement that will change how we perceive money forever.

Yep, the people complaining that their chain sync is taking too long are all wrong, they should be happy its slow.  Don't get me wrong, I like natural growth, and I think it would be great if bitcoin's huge recent increase in chain size was all because we were getting new users or having increased liquidity in the market.  But, sadly, thats not true.  

Sadly the fees SatoshiDice pays are essentially nothing, so they dont really care.  In fact, AFAIK, the only reason they pay the fees they do is because bitcoinj doesnt (yet) support calculating minimum relay fees the way the satoshi client does.  

The issue is less of "there is a huge volume of transactions, oh no!!!" and more of "there are people in the community who, for whatever reason, are doing actions that are forcing other transactions into long confirmation time when their transactions dont really need confirmed immediately" so the solution is "lets give transaction-creators a way to say that their transactions are really lower priority than other 0-fee transactions and thus dont let their transactions force people who want to send money without paying not suffer as a result" additionally "lets tweak the incentive structure so that we further discourage bad actors in the community."  Im not saying pruning isnt going to eventually be implemented, it probably will (in the form of shipping pruned chain snapshots), but in addition to that, there are things we can do to make it not as rough a transition right now.

OK. I will solve the first time users problem downloading the blockchain...
In the next 2 weeks I'll work on a page where everybody can order the blockchain in a medium of their preference and get it in the mail paying only 2% markup on the chosen medium price+shipping(Worldwide). DvD, HDD,USB flash disk, their choice. I'll get it and send it to them with the most recent blockchain snapshot possible.

This was something I've been thinking about for the last couple of months. Now is the time to do it, I guess.

Any there goes all the excellent decentralized aspects of bitcoin.  Its also easier to point people to eg. tcatm's chain snapshots (see below) and have them use -loadblock (only available in 0.6.99/0.7).

Even if we prevent the blockchain from growing completely right now, it won't make the experience of those users any better, they would still need to download 180000+ blocks.

Any attempts to "improve" things a little bit in the current design would probably be a waste of time.

EDIT: If there are more efficient ways to download and parse the existing blockchain I'm all for that!

That's easier when you are talking about 3 or 4 GB. When you think it will be 50 or 100 GB soon it makes sense to order an HDD with it.
And as far as I know the client will check the blockchain validity when starting, no?
Also, I'm not jealous. Any other guy is free to grab the idea and do it. I don't wish to be the only one doing it. It can get tiresome for almost no reward.

This is all so wrong:

1. The current network is larger than what 5 super computers?
2. You can't handle an absolutely small amount of transactions?
3. Solution: Throttle BTC with fees/blocking/self-prioritizing txs?!?!

NONONONONONO!


This is a serious issue: The BTC network is not scaling AT ALL due to horrible design.


So was my idea with SPV nodes keeping balances documented with input/outputs viable or not?

That way they store a fraction of the chain, but can do everything a full node network can in swarm fashion?

This is not trivial.  Winnings are paid with transactions that redeem the bet that won them so that it can safely operate with zero confirmations.

Apparently the fees they pay are significant.  According to etotheipi's analysis (https://bitcointalk.org/index.php?topic=80312.0):


Fees make up around 25% of their cumulative losses so far.  And make up more than 0.1% of payouts.  Since the house edge is 1.5%, these fees correspond to about 8% of theoretical profits.

 :D

What about, instead of releasing a client with built-in transaction fee policy, you release a client where the fee policy can be easily configured in a text file? You make the "default" text file provided with the executable have the same configuration the current embed fee policy has, in order not to change things abruptly.

I find it better this way.

+1. Well said.
Pruning, storing the chain in raw format without verifying it, and SPVs, these are the ways to go IMHO. Not trying to "punish" a particular way of doing transactions, specially when the sender of these transactions is paying more than the average bitcoin user pays for them.

+1 to that also.

I feel there's something wrong with the block download process. It seems it does everything too synchronously, but I don't know. That's why I liked to hear that you, Matt Corallo, was working on a patch trying to improve that. This is the way to go. Make the system more efficient, without trying to censor users which are donating more to the network than the average user is.

Security = Confidentiality, Integrity, Availability. When the blockchain doesn't fit on a 1TB hard drive anymore (or probably sooner ;D), we'll have an issue with the 2 last points...

It's better that we are confronted *now* with the problem of a lot more transactions with satoshidice than later with a large influx of new users.

This forces the issue now and this gives us more time to solve it before Bitcoin becomes more popular.

My suggestion is solutions that will work on the long-term, i.e. scalability.

I think the responses from a few thread participants have been a bit more... strident and heated than is productive.

The main apparent fact is that SatoshiDice transactions are crowding out and slowing other bitcoin transactions. 

Thus, ignoring all other technical arguments, we can show that SatoshiDice is hurting other bitcoin users' confirmation times.

Well no.  If anything Satoshi Dice is delaying tx which pay less or nothing.  Oh noes.  They paying customer gets to go first.  If you pay more per KB than Satoshi Dice your tx will never be delayed.  SD pays 0.0005 BTC per tx.   Simple experiment.  Add 0.002 BTC tx fee to all your tx.  Yes it will cost you a "whole" US penny.  Your tx will have higher priority.  Problem solved.

Well, worrying about scalability seems legitimate as long as we want BTC to become more mainstream, which necessarily means lots more transactions (and lots more wallets and addresses, too).

Well, yes but I think one of the points of a cryptocurrency is to remain reliable/usable no matter how hostile the environment is. SatoshiDice is a nice "attacker" who simply uses the system a bit much, and pays for it. Bitcoin really has to be able to deal with that nicely... with a better solution than just blocking this usage (or asking SatoshiDice to stop...)

+ 1

I cannot believe that someone thinks blocking a profiting business is MORE productive than discussing (if heated) a software update.

REALLY?

That would be like Google blocking its biggest advertisers because they dare generate so much traffic!

Their brains probably do "work right". They simply do not disclose all of the results of their work. For many Bitcoin is an opportunity to make money off of the "how much per month" crowd, the people for whom long-term thinking means "who's going to win the next Superbowl or the next UEFA games".

To me it seems like you think on the far further time-horizon than the majority of the members of this forum.

Good point.


And good analogy as well. :)


I'm pretty sure Matt Corallo's brain works very well, though.
The debate here is more fundamental. Some people are all angry with SatoshiDice because they've finally increased Bitcoin's transaction volume, making it harder for people that do not have to store and validate  the blockchain to keep storing and validating the blockchain. They believe it would be nice if such unnecessary and unscalable behavior could go on for longer, and want to punish those that are making this difficult.

I don't agree with it, as my messages in this topic make it clear. But my greatest worry here is that such kind of "punishment" should never be part of the protocol. Pool operators and solo-miners are the only ones who should be taking arbitrary decisions concerning which transactions get included or not. End-users should have no say. And, mostly important, the protocol should be neutral. As the "reference implementation", bitciond should remain neutral too.

If Matt wants this so badly, then, well, I guess he's skilled enough to make a patch and provide it to those who agree with them. I just sincerely hope such patch doesn't get included in the main branch of bitcoind. (and I'd find it a pity that he stops to work on this much more important branch he's working on, about making the blockchain update process more efficient, to work on a thing like this.. but well, that's his choice)

Im going to close this thread because I think most of what can be said has been said.  I fully understand that a number of people disagree about how much of a problem satoshidice is, and thats fair.  In any case, there is a discussion about actually implementing it on the mailing list, however I doubt it will be implemented purely because it is so controversial.