Ultraprune merged in mainline

[Peter Todd]

Did the ultra prune patch mean that getrawtransaction and similar can't always retrieve a transaction if it has been spent? I noticed that seems to be the case now.

I mean, obviously that will be the case in the future, but does is it supposed to be doing that already? If so, how do I turn that off?

[1714952981]

1714952981

[1714952981]

1714952981

[1714952981]

1714952981

[Pieter Wuille]

Did the ultra prune patch mean that getrawtransaction and similar can't always retrieve a transaction if it has been spent? I noticed that seems to be the case now.

Indeed. I do plan to add an optional index for people who want such functionality.

It is already the case because the current code does not maintain an index of all transactions anymore - in normal working conditions there is no use for such an index. Since it is still very useful for debugging, I'd like to make it optional to maintain such an index.

[Peter Todd]

Did the ultra prune patch mean that getrawtransaction and similar can't always retrieve a transaction if it has been spent? I noticed that seems to be the case now.

Indeed.

I do plan to an optional index for people who want such functionality.

So just use the version prior? I noticed that pre ultra prune in git seems to be broken on my machine; some sort of database corruption problem. Post leveldb looks ok though.

I really think that index needs to be implemented before the next release with only the ultra prune mode. We don't want the people who do need to be able to get any transaction to start lagging behind on releases. Look at how out of date block explorer is because it was built on an old get block patch. Also, document this limitation somewhere.

[Raoul Duke]

Clearly not everyone in the network can do this, as that would mean new nodes cannot bootstrap anymore.

What will prevent it? If the changes r good then everyone will use new approach.

Why do people today run a full node like Bitcoind or Bitcoin-Qt when they could run an SPV node like MultiBit or not a node at all, like webwallets or Electrum?

I can proudly say that I run a full node and will always run one, even if I have to lease a dedicated server in a datacenter for it, just in case my domestic connection can't keep up anymore. I guess that before leasing a server I will need to have a separate computer running one at home, even if I'll need to run a light client on my laptop to spend and receive bitcoins.
I don't mine and most likely never will, so, it's the only way I can contribute to the network.

[casascius]

I can proudly say that I run a full node and will always run one, even if I have to lease a dedicated server in a datacenter for it, just in case my domestic connection can't keep up anymore. I guess that before leasing a server I will need to have a separate computer running one at home, even if I'll need to run a light client on my laptop to spend and receive bitcoins.
I don't mine and most likely never will, so, it's the only way I can contribute to the network.

I too run a full node, mainly because where I'm running it has effectively unlimited disk space, bandwidth, and CPU time (relative to the needs of the client).  I see no benefit to doing less.

If the bootstrap process were to run in a "minimal" mode where it leeched queries from the network while bootstrapping itself (making itself immediately useful), that would be exciting, not to mention no longer turning off users who download the client expecting it to work today and not tomorrow.  Once bootstrapped, I would have no reason to configure it to contribute less, given that it has no less than a terabyte of disk space at its disposal with no other anticipated purpose in mind.

But of course I would highly value the ability to trim back my node's contribution if, for example, it were running on a cellular connection and being part of the network were too costly.  If I were a chain of mobile food trucks that accepted Bitcoin, I'd put my trucks on the "minimal" setting to keep my costs down, and then run a "full" node at the office to give them a private trusted node to connect to.  I would imagine that any merchant accepting Bitcoin would be in a position and happy to contribute to the network in this way.

[gmaxwell]

Here's how it ought to work in my mind:

This subject has come up a couple times on IRC, here is what I've said:

I'd generally like to avoid throwing more decisions in users faces— especially ones with long explanations.

The major security/burden models I've mostly been thinking around SPV, (eventually) SPV-UTXO (a full node functionally but with only SPV security),  Full but completely pruned, Archive (same security as full but can act as a block explorer and boot strap new nodes).  Partial archive may be possible— but would require some serious p2p protocol effort to make finding the right peers possible and couldn't be used as a block explorer/debug node.

I'd expect that we'd always start in some SPV mode— since it's the only way to get fast startup— and then upgrade to the target mode in the background in order to have fast startups.

What I expect we'll do and would prefer would be to automatically pick the default target mode based on the system type and the available resources. This will get the maximum number of nodes running in the highest contributing state possible without aggregating people by using more resources than their system can comfortably support. Of course, there would be some way to override, but that could possibly be some burred advanced option.

As far as choices go, the amount of bandwidth to use is probably one of the settings we can't mostly hide since it's much harder to gauge whats available and we can't tell when the user is paying for it. 

Though this is all completely orthogonal with ultraprune.

[casascius]

Here's how it ought to work in my mind:

This subject has come up a couple times on IRC, here is what I've said:

I'd generally like to avoid throwing more decisions in users faces— especially ones with long explanations.

If the main decision that the user had to make was a) how much disk space am I willing to contribute to bitcoin [knowing it can be reclaimed anytime if needed], and b) how much bandwidth do I want to contribute, then we could probably get users to make the right decision every time without burdening them with the details.

[gmaxwell]

If the main decision that the user had to make was a) how much disk space am I willing to contribute to bitcoin [knowing it can be reclaimed anytime if needed], and b) how much bandwidth do I want to contribute, then we could probably get users to make the right decision every time without burdening them with the details.

That generally results in "what am I getting out of this? Uh. I don't care. It works with zero right? Zero."  I'd rather express bandwidth settings in terms of rate limiters. Setting yourself down would reduce your contribution, but would also reduce your consumption. Setting it high makes you contribute more but also makes your Bitcoin update faster. So even someone with a very simple understanding of the option knows that he's getting something out of cranking it up. — we have this with listening today, without listening you "only" get three bars. To get your >8 connections and four bars you must listen.  And people do go out of their way to setup the port forwards just to make the gauges go up— not because they're trying to help the network, but because they expect it will make it work better.

The idealized rational participant realizes that running a full node is both good for their own security and that it's good for their financial interests because it's good for the network.   But knowledge isn't frictionless and recognition of the rational choice requires some somewhat subtle thought. Humans tend to reason poorly about the risks of unlikely events (e.g. someone performing a technical attack against thin clients), they only tend to care _after_ everyone has been ripped off. We already see this directly where people are using thin clients (not even SPV) and web wallets and believing it to be just as secure, or modifying their client to make 800 outbound connections and not being _at all_ concerned about the burden they're placing on the network or realizing the doing so is harmful to their own interests in any case.

Bitcoin is secure against attack when the honest participants are rational and selfish.  But if many of the participants are _irrational_ and selfish then they need to be offset by altruistic participants who contribute more resources. Education can help, but there are limits because the lack of time or interest that creates the irrational behavior also keeps education from being effective. As far as I can tell the way to maximize altruism is to make it the default— so that everyone who is indifferent is altruistic— but leave it possible to tone it down if it turns out to be problematic, so that people aren't forced to completely eliminate their contribution if its too burdensome.

[casascius]

If the main decision that the user had to make was a) how much disk space am I willing to contribute to bitcoin [knowing it can be reclaimed anytime if needed], and b) how much bandwidth do I want to contribute, then we could probably get users to make the right decision every time without burdening them with the details.

That generally results in "what am I getting out of this? Uh. I don't care. It works with zero right? Zero."  I'd rather express bandwidth settings in terms of rate limiters. Setting yourself down would reduce your contribution, but would also reduce your consumption. Setting it high makes you contribute more but also makes your Bitcoin update faster. So even someone with a very simple understanding of the option knows that he's getting something out of cranking it up. — we have this with listening today, without listening you "only" get three bars. To get your >8 connections and four bars you must listen.  And people do go out of their way to setup the port forwards just to make the gauges go up— not because they're trying to help the network, but because they expect it will make it work better.

I fully agree with you, it sounds like you're suggesting I've proposed something poorly thought through, when I only provided the simpler explanation in response to your objection that it otherwise was too long for the average user.  As noted previously, I also propose the default being "altruistic", and of course, this setting would be something buried on a settings screen, and not something thrown at the user when they install.  Most casual users won't even see the setting let alone think to slide it to the minimum.

That said, users should have every right to go to a settings screen and choose the minimal contribution for any reason or no reason at all, no questions asked.  Their computer is their property, they have the right to restrict its resource usage as they wish, and software whose developers insist on consuming more resources than welcomed is as annoying as telemarketers who think that nobody will miss the few wasted seconds of their day when they call people.  They are the reason non-developers value "walled gardens" like Apple's App Store, to the befuddlement of many developers.

Allowing a user to get more "bars" in exchange for contributing to the network is very intuitive and demands none of the user's unoffered attention and is consistent with what I have in mind.

[gmaxwell]

That said, users should have every right to go to a settings screen and choose the minimal contribution for any reason or no reason at all, no questions asked.  Their computer is their property, they have the right to restrict its resource usage as they wish, and software whose developers insist on consuming more resources than welcomed is as annoying as telemarketers who think that nobody will miss the few wasted seconds of their day when they call people.  They are the reason non-developers value "walled gardens" like Apple's App Store, to the befuddlement of many developers.

I don't think this is a simple matter of principle as you've stated it.

At the end of the day the developers have given their time, to write freely licensed software, have distributed it with the source, been generally transparent about changes (I'd say very highly transparent, but some security sensitive things have had delayed details), do their work in the open, and generally help people hack on it.  From a principled perspective I think thats where our obligations end... and thats more than enough that even if you're not a hacker you can get other people to code changes for you— perhaps not for free— but it's not generally considered polite to demand what people do with their time if you're not paying them.

Of course, it's _nice_ to be sensitive to people's wishes, but that has it's limits.  We would be fools to add a "[X] DDOS attack bitcoin"  or "[X] Blackhole transactions" settings to the menu and will not do that even though there are some people— "for any reason or no reason at all"— who would want them.  To some extent the ability to exclude "features" like that is part of the payment we get for contributing to the system, and part of the reason people should choose to use and recommend the reference software instead of some other is the consideration of features that where included _and_ excluded, even if some people don't always agree completely with all of them. So features which are dangerous to the user or the network because they're easily misunderstood ought not be included, or ought to be sufficiently burred to prevent injury... and that if it's in there you know that bitcoin experts looked at it and weren't too horrified by it.

Selecting a client mode has obvious important applications, so I'm sure it would be offered.  But if it's possible to maximize the good outcomes without hurting the applications by aggressively burying it— a commandline (-screw-over-bitcoin=more)— then it might not be a bad idea to do that.  I suspect for all the the usecases for switching it the person will _know_ that they want to reduce resources and will go looking, we don't have to suggest the idea that a free lunch is only a click away if they weren't already thinking about it. Although burring it would be less good for user choice it must be balanced against all the other factors.

In any case, sorry for the tangent. We're a long way off from this and it may make sense to do just as you've said. I just wanted to muse a bit on the value of user choice and how I think that fits into feature decisions.

[Atlas]

That said, users should have every right to go to a settings screen and choose the minimal contribution for any reason or no reason at all, no questions asked.  Their computer is their property, they have the right to restrict its resource usage as they wish, and software whose developers insist on consuming more resources than welcomed is as annoying as telemarketers who think that nobody will miss the few wasted seconds of their day when they call people.  They are the reason non-developers value "walled gardens" like Apple's App Store, to the befuddlement of many developers.

I don't think this is a simple matter of principle as you've stated it.

At the end of the day the developers have given their time, to write freely licensed software, have distributed it with the source, been generally transparent about changes (I'd say very highly transparent, but some security sensitive things have had delayed details), do their work in the open, and generally help people hack on it.  From a principled perspective I think thats where our obligations end... and thats more than enough that even if you're not a hacker you can get other people to code changes for you— perhaps not for free— but it's not generally considered polite to demand what people do with their time if you're not paying them.

Of course, it's _nice_ to be sensitive to people's wishes, but that has it's limits.  We would be fools to add a "[X] DDOS attack bitcoin"  or "[X] Blackhole transactions" settings to the menu and will not do that even though there are some people— "for any reason or no reason at all"— who would want them.  To some extent the ability to exclude "features" like that is part of the payment we get for contributing to the system, and part of the reason people should choose to use and recommend the reference software instead of some other is the consideration of features that where included _and_ excluded, even if some people don't always agree completely with all of them. So features which are dangerous to the user or the network because they're easily misunderstood ought not be included, or ought to be sufficiently burred to prevent injury... and that if it's in there you know that bitcoin experts looked at it and weren't too horrified by it.

Selecting a client mode has obvious important applications, so I'm sure it would be offered.  But if it's possible to maximize the good outcomes without hurting the applications by aggressively burying it— a commandline (-screw-over-bitcoin=more)— then it might not be a bad idea to do that.  I suspect for all the the usecases for switching it the person will _know_ that they want to reduce resources and will go looking, we don't have to suggest the idea that a free lunch is only a click away if they weren't already thinking about it. Although burring it would be less good for user choice it must be balanced against all the other factors.

In any case, sorry for the tangent. We're a long way off from this and it may make sense to do just as you've said. I just wanted to muse a bit on the value of user choice and how I think that fits into feature decisions.

Quoting for archival purposes.


...part of the reason people should choose to use and recommend the reference software instead of some other is the consideration of features that where is the consideration of features that where included _and_ excluded, even if some people don't always agree completely with all of them.

This is central planning right here.

[MatthewLM]

Doesn't bittorrent have an algorithm which allows for people to not contribute bandwidth but has incentives for nodes to do so? I just vaguely remember reading something about it.

[Atlas]

Quoting for archival purposes.


...part of the reason people should choose to use and recommend the reference software instead of some other is the consideration of features that where is the consideration of features that where included _and_ excluded, even if some people don't always agree completely with all of them.

This is central planning right here.

Here's another example of central planning: http://www.bitcoin.org/bitcoin.pdf

In other words, not persuasive.

Nakamoto's plan is the canvas in the context of my argument. It functions perfectly fine. Bitcoin has no problems thus far.

All I am arguing against is people who want to single handily plan out its future and shape it to their wishes. Why? Because people fail. Nakamoto hasn't failed and that's great but we should preserve the success we have now and not squander it through radical change that may or may not work nor through a single point of failure that is a single development process.

Hybrid secessions in the Bitcoin development community are a must else we end up with a inbred culture that can be easily corrupted.

No, another currency will not do. Bitcoin has momentum that is very scarce. If destroyed, cryptocurrency might be doomed for another few decades.

[casascius]

it's not generally considered polite to demand what people do with their time if you're not paying them.

The way I see it, by downloading the software, they are paying, just not with money, the same way they "pay" Google by seeing ads.  These people are offering Bitcoin (the idea) a share of their mind and their time.  They are incrementally adding to the entire Bitcoin-based economy by considering accepting Bitcoins in exchange for their goods and services.  They are paying with their willingness to change how they see a BTC as something worth 0.00 to them, to something worth over $10.  Of course, I am also respectfully keeping in mind that you have contributed substantially to bitcoind, and I have not.

That said, I am one of the few who indeed offers bounties for implementing my pet features, which nowadays usually take the form of a 10 BTC silver coin.  I have paid bounties for many of the recent improvements to bitaddress.org, and I have several open bounties for features I'd like to see, including sweepprivkey and a option (that can be enabled by a user, even if command-line only) to create and maintain a disk-based index in bitcoind that allows quickly finding all standard unspent transactions belonging to a given bitcoin address (which is what sweepprivkey would depend upon).

[Atlas]

it's not generally considered polite to demand what people do with their time if you're not paying them.

The way I see it, by downloading the software, they are paying, just not with money, the same way they "pay" Google by seeing ads.  These people are offering Bitcoin (the idea) a share of their mind and their time.  They are incrementally adding to the entire Bitcoin-based economy by considering accepting Bitcoins in exchange for their goods and services.  They are paying with their willingness to change how they see a BTC as something worth 0.00 to them, to something worth over $10.  Of course, I am also respectfully keeping in mind that you have contributed substantially to bitcoind, and I have not.

The fact is Bitcoin would live on without his support. If a feature is truly needed, it will be done anyway by another party or two. There is enough capital and value here to sustain Bitcoin on a as-it-goes basis. Bitcoin as a protocol is sustaining just fine after all. There are no major vulnerabilities. Client innovation is being advanced by various people. The Satoshi client will likely become a relic.

Bitcoin could not live without the users putting their money and time into this. It's the user that is God in the Bitcoin ecosystem.

This isn't Ubuntu Linux or another hippy open-source software project. This is money. Real money. It's beyond the leagues of anything else in the development world.

Yes, I will continue to be a douchebag because I have too much money in this thing.

[Atlas]

Bitcoin has momentum that is very scarce. If destroyed, cryptocurrency might be doomed for another few decades.

The fact is Bitcoin would live on without his support. If a feature is truly needed, it will be done anyway by another party or two.

Which is it?  How many fingers can you chop off your hand before it ceases to be a hand?

The momentum I am referring to is the userbase that continues to grow which is not fully due to the post-2010 work of Bitcoin-Qt/Bitcoind.

I think it would be very hard to argue that it was because of the efforts of Mr. "Bitcoin is just a experiment, don't bet the house." that Bitcoin is trading over $12 a coin.

Satoshi Nakamoto made 98% of what the core Bitcoin protocol is today. That was the Big Bang. Every other software development is a secondary footnote. The real success is in user support of Satoshi's core product.

I recognize people want to be important, they want people to depend on them for Bitcoin to work but that's not how free societies work. We don't indebt ourselves to a supposed leader. We shouldn't have to owe Bitcoin's success to a certain class of people. We all stand on equal ground and stand independently. Bitcoin as a decentralized currency should represent that.  

[Atlas]

The momentum I am referring to is the userbase that continues to grow which is not fully due to the post-2010 work of Bitcoin-Qt/Bitcoind.

I think it would be very hard to argue that it was because of the efforts of Mr. "Bitcoin is just a experiment, don't bet the house." that Bitcoin is trading over $12 a coin.

Satoshi Nakamoto made 98% of what the core Bitcoin protocol is today. That was the Big Bang. Every other software development is a secondary footnote. The real success is in user support of Satoshi's core product.

I recognize people want to be important, they want people to depend on them for Bitcoin to work but that's not how free societies work. We don't indebt ourselves to a supposed leader. We shouldn't have to owe Bitcoin's success to a certain class of people. We all stand on equal ground and stand independently. Bitcoin as a decentralized currency should represent that.  

Satoshi also made 98% of SolidCoin, NameCoin, I0coin, IXCoin, and every other coin out there that is presently deep in the toilet.

Bitcoin trades at over $12 a coin because of the collective actions and frequent releases of the Bitcoin devs, entities like MtGox/BitInstant/BitPay, utilities like BlockChain.info, the many businesses who accept Bitcoin as payment, promoters like Bitcoin Magazine and WeUseCoins.com (not to mention my own coins), and like it or not, things like the Silk Road.  When you say "Hey Gregory, you didn't build that!", that's being a douche and has nothing to do with how much money you have in Bitcoin - it's just plain and simple being a douche.

The frequent releases of the Bitcoin devs play a very small part especially when most users use custom GUIs or a web client to conduct Bitcoin business.

I am not saying other people haven't built anything. I am saying the Bitcoin.org dev team hasn't built everything and that we owe them nothing. They are doing this out of their own interest and their own time, especially when they want to force certain things down our throat.

All I am calling for is no recognition of significant power in this community: Bitcoin.org implying they have great say and clout in this offends that ideal. They imply it when they want to push radical changes. This change may not be too radical but I am waiting for the day a radical push comes.

Also, I have no shame in being a douche. I am just out for the long-time survival of Bitcoin. I have no agenda besides keeping things as they are: Working.

Anyways, I agree, we all built Bitcoin. As for the Bitcoin.org developers, no they don't have exclusive ownership over it.

TLDR: Nobody should be dependent on the work of anybody in the Bitcoin community.

[Atlas]

The frequent releases of the Bitcoin devs play a very small part especially when most users use custom GUIs or a web client to conduct Bitcoin business.

Right, sort of how companies like Cisco, Intel, Apple, Microsoft, and Google, and guys like Linus Torvalds, have little to do with people's ability to read their e-mail, and they owe 98% of the credit to the guys who invented the binary code and the transistor.

Why should we owe anyone anything? The debt has been paid inherently when they released their work. Why should they keep endless authority?

All I am calling for is no authority over the Bitcoin development culture. I am not questioning people doing things. However, I do question the "I am special, worship me, need me, depend on me." mentality.

[Atlas]

gmaxwell: jgarzik: Atlas works at odds with his stated goals, he makes me want to centeralized bitcoin just so we can write him out of it.

This is what he said in response to this thread.

[marcus_of_augustus]

(copy of mailinglist post)

I've just merged my "ultraprune" branch into mainline (including Mike's LevelDB work). This is a very significant change, and all testing is certainly welcome. As a result of this, many pull requests probably don't apply cleanly anymore. If you need help rebasing them on the new structure, ask me.

The idea behind ultraprune is to use an ultra-pruned copy (only unspent transaction outputs in a custom compact format) of the block chain for validation (as opposed to a transaction index into the block chain). It still keeps all blocks around for serving them to other nodes, for rescanning, and for reorganisations. As such, it is still a full node. So, despite the name, it does not implement any actual pruning yet, though pruning would be trivial to implement now. This would have profound effects on the network though, so may still need some discussion first.

A small summary of the changes:

• Instead of blk000?.dat, we have blocks/blk000??.dat files of max 128 MiB, pre-allocated per 16 MiB
• Instead of a Berklely DB blkindex.dat, we have a LevelDB directory blktree/. This only contains a block index, no transaction index.
• A new LevelDB directory coins/, which contains data about the current unspent transaction output set.
• New files blocks/rev000??.dat contain undo data for blocks (necessary for reorganisation).
• More information is kept about blocks and block files, to facilitate pruning in the future, and to prepare for a headers-first mode.
• Two new RPC calls are added: gettxout and gettxoutsetinfo.

The most noticeable change should be performance: LevelDB deals much better with slow I/O than BDB does, and the working set size for validation is an order of magnitude smaller. In the longer run, I think it is an evolution towards separation between validation nodes
and archive nodes, which is needed in my opinion.

What will be the new dependency(s) for a typical Linux box?

Will BDB still remain a dependency also?