Huge increase in satoshidice spam over the past day

[DeepBit]

It seems that Deepbit doesn't dynamically increase tx fees like the Satoshi client does.

Why do you think so ?

[1713290450]

1713290450

[1713290450]

1713290450

[1713290450]

1713290450

[theymos]

Why do you think so ?

Looking at your statistics, it seems that it's not you. Sorry.

Some big miner is creating blocks near 500 kB without increasing fees.

[jim618]

SatoshiDice is certainly pushing Bitcoin into new territory.

I thought I would mention a data structure I am going to experiment with in MultiBit as it is related to blockchain management.

Currently the MultiBit/ bitcoinj blocks that are stored to disk contain:

Code:

        private int height;          // 4 bytes
        private byte[] chainWork;    // 16 bytes
        private byte[] blockHeader   // 80 bytes

i.e 100 bytes per block.

It would be very useful to have a mapping of 'bitcoin address -> blocks the address appears in' so that you can get the transactions for an arbitary address/ private key/ public key quickly.

I was thinking of adding a Bloom filter for each block (indexed by bitcoin addresses that appear in that block) into the locally stored blockchain.
The idea would be that to get the transactions for an address or set of addresses:
1) You scan the locally stored blockchain bloom filters to get a list of blocks that you know contain relevant transactions.
2) You download just those blocks from a Satoshi client.
3) You then parse the blocks' transactions to get the tx data.

You can store the presence of an object in a Bloom filter with approx 10 bits.
According to this I estimate there are about 2,000,000 distinct addresses on the current blockchain and each appears, on average, about 4 times. This would mean that the total size of the Bloom filters for bitcoin address indexing would be about:
10 x 2,000,000 x 4 bits = 80,000,000 bits i.e. about 10MB

Not much space for what it gives you. It is small enough that an up-to-date version can be included in the MultiBit installation download (I already include a snapshot of the current 100-byte-per-block blockchain).

Of course this does not solve the disk requirements for the Satoshi bitcoinds. It is only usable for the end-user client as it needs a full client to download the (specific) blocks from.

[caveden]

Paying a meaningful fee to miners has no effect on the thousands and thousands of bitcoin users who have to store the transactions on their disk...

They don't really have to, AFAIK.

It's up to miners to do something, if they want to do something. Maybe they can handle the volume and are happy with the fees.
EDIT: And when I say miners I mean solo-miners and pool operators, the only ones who really need the entire blockchain to operate.

[Realpra]

Bitcoin will die unless this is solved, its not much but I pledge 15 BTC to whoever creates a viable beautiful pruning-client.

Something which can operate with parts of the chain and only get the parts it needs to confirm incoming otherwise.

Should work like a torrent file where the full file could be distributed.


Basically my client should operate as before thinking it has the full chain, only it doesn't - it gets the parts it doesn't when needed from the cloud/peers.

After using such parts they are deleted.

It would be extra cool if parts of blocks could be downloaded instead of full blocks.


Maybe it could send a request to the network to get an incoming transaction verified by the nodes that have that block section?

Say my node has a block with A->B someone sends me a transaction with B->C, I check my block and verify B address has enough then okays it.

To clarify:
1. Blocks are verified as today via their expensive hash/tree root thingie.
2. All the transactions IN that block are verified by the network via the distributed block chain.

Time line:
1. Block is received ->
2. Full prune node (FPN) checks root hash against its own mostly empty chain ->
3. If okay, it checks the transactions it CAN ->
4. If okay it broadcasts the block while assuming the other transactions are fine ->
5. If NOT okay it does NOT broadcast it, but instead reports it including info about the invalid transaction and its block ->
6. Other nodes get the report, check the root hash of that transaction against their empty chain ->
7. If bad report is true they re-broadcast it ->
8. Bad blocks get propagated WAY slower AND are immediately turned down by super nodes ->
9. No major changes to core BTC logic, SMALL additional bandwidth usage for reporting.

If this is not done, BTC is dead.

Another idea:
"Allow transmission of half-blocks."
root hash <- 2nd hashes <- 3rd hashes etc.

You basically treat the second or third hash as THE block - the benefit is that slower connections can keep up with very large blocks as they only look at one branch of the blocks.
They would however need to check their own users addresses via 3rd service.

This is however not critical at present.

[Peter Todd]

So, currently the block size limit is a maximum of 500KiB:

1year/10minutes * 500KiB = 25GiB/year

So yeah, it's not tiny. But seriously, 25GiB/year really isn't all that much storage, and that's the hard maximum, actual is maybe half that. Sure the regular bitcoin-qt client becomes annoying to use for some guy who just wants to move some coin around, but anyone doing anything serious isn't going to have a problem. Myself, I run bitcoin-qt, but my actual wallet which I actualy use is on the excellent blockchain.info

The real issue is if the number of transactions makes it difficult for people to get their transactions verified, but as long as miners do fee-based priority, that's fixed by telling people to increase their fees.

My request for the devs: work on pruning, but no rush.

[TangibleCryptography]

Exactly I don't see 25GB being an impossible barrier for merchants, miners, exchanges, and hardcore bitcoin supporters. 

Honestly even when pruning happens at least a significant number of need to carry fully blocks so that nodes which wan't/need full block have access to them. 

[caveden]

Bitcoin will die unless this is solved,

What a drama, come on...

This will not kill bitcoin. Even without pruning, professional pool operators should be able to handle much more traffic than this. https://en.bitcoin.it/wiki/Scalability

It seems you people are thinking "ordinary users" should be full nodes. That's not the case. Only solo-miners and pool operators have such need.

But anyway, since that's what being debated here, I always wondered if there couldn't be a "relay mode". Something lighter than the full mode of bitcoin-qt, but heavier than the lightweight mode of BitcoinJ.
You store the whole blockchain, but in its raw format. You only validate its headers, not its transactions. You index only the transactions that concern you, plus a simple index to locate the blocks in the big raw file. You relay everything as a full node would.
The greatest charge that a full node imposes today is the indexation of the database. Storing in raw format would allow those that want to contribute their nodes as relays (as me and probably you) to keep doing it for a longer time, I guess.
But, honestly, I'm not even sure we should encourage this. For it to scale, the bitcoin network will have to change to a network where only some full nodes exists and all the others connect to them as lightweight nodes. So maybe SatoshiDice is just encouraging us to take this step.

[caveden]

My request for the devs: work on pruning, but no rush.

I'm not even sure developers themselves should worry with this. This is a problem to pool operators mainly. It's up to them to come out with a solution, or eventually finance one by offering bounties.
I find "securing your private key(s)" a much more urgent matter than this.

[Matt Corallo]

Theoretically, the alt-chain created for this purpose is strictly optional.  No one is "forced" to do anything -- they only use it if they want to participate in creating/exchanging/verifying/downloading address-balance information.  One major benefit of the idea is how perfectly non-disruptive it is.

That is not at all true.  The point of blockchain pruning is that most, if not all nodes on the network no longer need the full chain.  As nodes upgrade to the alt-chain and drop full blocks from the normal chain, nodes which do not want to use the new alt-chain will have a harder and harder time bootstrapping or even getting full blocks from any nodes, thus forcing them all onto the alt-chain.  Even if you argue that old nodes will always exist and be a fair percentage of the network (which they currently are, which is really a security issue for the network) connection mechanisms get complicated as nodes are increasingly trying more and more connections before finding a working one, which could lead to increased sybil attack possibilities and generally higher load on listening nodes.

I agree that it would be complex.  But there's a lot of alt-chains already in existence that use merged mining, which probably provides 90% of the template that would be needed.  I don't mean to imply that there's anything easy or quick about it... just that a lot of work has already been done.

Absolutely, I wasnt saying it wasnt possible or really /that/ difficult, but merging all of that code into the satoshi client is quite a bit of work, or at least quite a bit of code.  And with the client already so complex and poorly abstracted, it introduces huge possibilities for bugs.

Otherwise, there's two options that both have serious downsides (compared to using an alt-chain).  But maybe some brainstorming can resolve it:
(1) Overlay network like what stratum is trying to do.  Issue:  requires some trust of other nodes, and malicious nodes can really muck up the network.

Meh.

(2) Modify the main blockchain, forcing block-headers to include a valid balance-tree hash to be accepted.  Issue:  requires a hard fork. 

You dont have to force anyone to include anything additional in blocks to do pruning.  If you do pruning, you can simply drop transactions and branches in merkle trees.  Of course you have to snapshot a pruned chain and nodes cant do the pruning themselves, but that is true up to checkpoints anyway.

However, there's a dozen other things that could go in as long as we're doing a hard-fork, anyway.  If there's any inclination to believe it will have to be done at some point, earlier is better...

I dont think anyone wants a hardfork, nor do I think its worth doing without serious issues, and I dont even think this is worth it.

[Matt Corallo]

It would be very useful to have a mapping of 'bitcoin address -> blocks the address appears in' so that you can get the transactions for an arbitary address/ private key/ public key quickly.

See http://sourceforge.net/mailarchive/forum.php?thread_name=CANEZrP0kNZDByHpK2%3DUjP%2Bag0X1KmqHxnJdm%3De_pWMitP4QvvA%40mail.gmail.com&forum_name=bitcoin-development

[Matt Corallo]

They don't really have to, AFAIK.

It's up to miners to do something, if they want to do something. Maybe they can handle the volume and are happy with the fees.
EDIT: And when I say miners I mean solo-miners and pool operators, the only ones who really need the entire blockchain to operate.

Its entirely true that Bitcoin is designed to operate with a number of SPV clients on the network, and the bloom filter in p2p protocol options makes that more viable.  However, merchants and those who want an better trust model are still encouraged and should still run full nodes.  Also note that a huge increase in SPV clients will starve listening full nodes of resources and connection slots, making it harder and harder for clients to get connected, causing further issues with node-isolation attacks and similar issues.  In the end, though its /possible/ to run the network largely on SPV nodes, its far from ideal, especially when the only reason this is happening is because of one, maybe two sites who insist on being lazy and, frankly, stupid.

[Matt Corallo]

But anyway, since that's what being debated here, I always wondered if there couldn't be a "relay mode". Something lighter than the full mode of bitcoin-qt, but heavier than the lightweight mode of BitcoinJ.
You store the whole blockchain, but in its raw format. You only validate its headers, not its transactions. You index only the transactions that concern you, plus a simple index to locate the blocks in the big raw file. You relay everything as a full node would.
The greatest charge that a full node imposes today is the indexation of the database. Storing in raw format would allow those that want to contribute their nodes as relays (as me and probably you) to keep doing it for a longer time, I guess.

Its an interesting idea, but it largely defeats the purpose of running a full node.  If you are gonna relay a full block, you really should check its transactions, or you are running the same trust model as an SPV node while telling other nodes that you are running a standard satoshi-client trust model, which could make it easier for miners to outright lie for short chains of blocks if it becomes to widespread.

But, honestly, I'm not even sure we should encourage this. For it to scale, the bitcoin network will have to change to a network where only some full nodes exists and all the others connect to them as lightweight nodes. So maybe SatoshiDice is just encouraging us to take this step.

I dont think anyone is saying we arent gonna move to more lightweight nodes, but the network needs a significant number of full nodes, or needs a broader base of full nodes with strong connections and are able to manage a ton of connections.

[jim618]

It would be very useful to have a mapping of 'bitcoin address -> blocks the address appears in' so that you can get the transactions for an arbitary address/ private key/ public key quickly.

See http://sourceforge.net/mailarchive/forum.php?thread_name=CANEZrP0kNZDByHpK2%3DUjP%2Bag0X1KmqHxnJdm%3De_pWMitP4QvvA%40mail.gmail.com&forum_name=bitcoin-development

Hi Matt,

Thanks for that headsup. Having custom bloom filters on the server end of the connection would be far more efficient bandwidth wise. Will keep an eye on that discussion.

Jim

[Sergio_Demian_Lerner]

The first part of this message is not a constructive critic:

Many people (like me) anticipated it: It need just one disruptive use of Bitcoin (like satoshidice) to break the "all nodes are equal premise".

Now the constructive part: four possible solutions exists:

1- SPV nodes / full nodes distinction and related ideas.
2- Balance sheets (breaks compatibility)
3- Add a new transaction format of much sorter length (e.g.: no scripts, use of firstbits) (breaks compatibility)
4- Add a new (less CPU-demanding) signature standard (e.g. MAVEPAY or Merkle-Wintenitz) (breaks compatibility)

Check out the info at https://en.bitcoin.it/wiki/Hardfork_Wishlist (this page is not well cross-linked).

Best regards!

[Matt Corallo]

Many people (like me) anticipated it: It need just one disruptive use of Bitcoin (like satoshidice) to break the "all nodes are equal premise".

The all nodes are equal premise was broken more than a year ago when GPU miners started to become popular.  Anyway, the point of this thread is you dont have to break the ability of people to use full clients because one service refuses to work in a reasonable way.  Strongly encouraging miners to deprioritize high-volume addresses and making the satoshi client refuse to forward the full volume of such transactions puts a cap on basic stupidity.  Yes, they can simply switch to using dynamic addresses, but it does force them to spend a bit of extra time coding, and the hope would be that they might add something like multisend and hopefully allow users to carry a balance like virtually all other bitcoin sites.

1- SPV nodes / full nodes distinction and related ideas.

Its obviously coming, but there are issues with a drastic network-style shift overnight, such changes should happen slowly to give people a chance to watch the shift very closely and make changes where required.

2- Balance sheets (breaks compatibility)
3- Add a new transaction format of much sorter length (e.g.: no scripts, use of firstbits) (breaks compatibility)
4- Add a new (less CPU-demanding) signature standard (e.g. MAVEPAY or Merkle-Wintenitz) (breaks compatibility)

Check out the info at https://en.bitcoin.it/wiki/Hardfork_Wishlist (this page is not well cross-linked).

No one wants to hardfork for an issue like this, its simply not worth it.  Hardforking is a huge ordeal.

In any case, this topic has gone far off topic.  The goal was to discuss deprioritizing satoshidice and other similar transactions.

[nimda]

Deprioritizing sounds like a horrible idea. Who gets to decide who to limit?

Maybe 25 GiB/year isn't too much storage-wise, but it is way too much bandwidth-wise.

"Hey have you heard about Bitcoin?" "Yeah, last month" "Why aren't you using it? I find great deals every day on BitMit" "Blockchain's still downloading"

  right...

The size of the blockchain will soon be a huge hindrance to widespread adoption

[caveden]

especially when the only reason this is happening is because of one, maybe two sites who insist on being lazy and, frankly, stupid.

You're being too harsh on SatoshiDice.
AFAIK, they're paying fees for every transaction they send. In the end, they're contributing to the network security by financing miners. The fact that non-miners are having problems to follow the chain progress is less relevant IMHO, as that will necessarily be the situation in the future if bitcoin "succeeds".

I'd argue they're doing more good than damage with all these paying transactions. They're actually being generous, because if they were to use send-many and reduce their number of transactions, they'd be paying less fees to miners. (and about having a balance, I think they've been so successful precisely because they don't require you to have an account).

And if miners/pools are still accepting free transactions, it just means they don't give a damn to the overhead. (at least not yet).

[Matt Corallo]

Deprioritizing sounds like a horrible idea. Who gets to decide who to limit?

Everyone deprioritizes every address that is heavily used, thats the point.  
If you think about the use of single address for high tx volume, pretty much all of it can be safely delayed getting into a block.
1. Green Addresses.  A. people shouldnt be using green addresses to begin with. B. the whole point is you trust the person sending, so being late getting into a block is no problem.
2. Poorly-designed sites that use single addresses to receive transactions from everyone (eg SatoshiDice).  uhh...thats the point of this thread.
3. Donation sites.  If you are getting a large volume of donations, I dont think you are worried about being able to spend each new donation now vs in 24 hours.

Maybe 25 GiB/year isn't too much storage-wise, but it is way too much bandwidth-wise.

What? Thats 6kb/s.  Even if your node is only up for one hour/week, you only need 1134 kb/s to download the chain.  I think thats perfectly reasonable.  

"Hey have you heard about Bitcoin?" "Yeah, last month" "Why aren't you using it? I find great deals every day on BitMit" "Blockchain's still downloading"

Chain download isnt limited by bandwidth, not by a long shot (unless you start downloading from a very slow peer, which is possible, but something that should be worked around soonish).  Chain download is limited by disk speed, but mostly CPU speed at checking signatures.  Hence my statements that sites like Deepbit and SatoshiDice need to implement things like multisend which allow for faster checking of transactions as txindex.dat is able to have fewer keys.

The size of the blockchain will soon be a huge hindrance to widespread adoption

Thats the point of this thread...

[caveden]

Deprioritizing sounds like a horrible idea. Who gets to decide who to limit?

Miners/pool operators may prioritize whatever they want.

It is a horrible idea if done by the developers of bitcoind. I'm not comfortable even with the hardcoded fee policy left by Satoshi.

What might be acceptable at "bitcoind level" are configurations. Like, a file where you specify fee policy parameters, if you're a solo-miner/pool operator.

The size of the blockchain will soon be a huge hindrance to widespread adoption

"Normal" users should not be using bitcon-qt.