To be clear, not anyone could do it.  It would have to be the core developers who make the decision to do that, as they are the only ones that, in general can release/approve any code updates without forking the blockchain.  The problem is, this change would be so dramatic and controversial, it's likely to fork the blockchain anyway, and throw the entire project into chaos.  X% of people would whole-heartedly disagree, and fork the project with the original code and original philosophy.  100%-X% would just follow the core developers and go with it.  I believe that this would cause all sorts of controversy, and frustration, and loss of morale among BTC supporters.  It would probably start a downward spiral of BTC.  I assume X > 10.

Only with an extraordinary level of "consensus," whatever that means, could the developers make such a drastic change and keep the unanimous support of the current user base.  The problem is, there's not really a way to get a consensus.  Even if the software design was truly a democratic process, there's no way to know what users would agree to do.  I think the only way for Bitcoin to continue is with the core philosophical elements intact.  

Other people have decided they don't like the way BTC works, and have made their own forks (one of them, I forget which, uses constant inflation rate).  But to date, none of them have attracted any significant portion of BTC users, and thus, the devs have no reason to believe this is a desirable change (plus, they'd have to believe in it themselves, first).

By "not set" I assume you mean setting NODE_NETWORK to zero.  It sounded like the reference client doesn't really take this into account yet, but that's not a reason to ignore it...


Does this mean that I should be relaying full blocks if this value is set (i.e. other nodes expect me to respond to getdata requests)?  Or they won't give me full blocks unless this is set (i.e. they assume I'm a headers-only kind of node)?

My theory on this is simple:  if SHA256 is "broken," it will very likely not "break" double-SHA256.  If someone were to discover a way to find collisions in 2^50 calculations, then it is still prohibitive (at least, annoying) to find collisions in single SHA256.  However, it's very likely that this vulnerability would reduce double-SHA256 to somwhere between 2^75 and 2^100 calculations.  This is still completely infeasible right now.  I believe the entire BTC network has computed about 2^70 hashes over its entire two years in service.

As I linked in my first post, there is a "services" bitfield in the specification described as "bitfield of features to be enabled for this connection".  I assumed this was the place to identify yourself and your capabilities.  I'm not convinced that the network can really operate with a diverse field of node types without understanding his own peers.  If none of your peers are willing to identify their capabilities, and you aren't receiving any block/tx updates, how do you know that you aren't being attacked, or just connected to bunch of lightweight nodes? 

The fact that such a field exists in the specification page leads me to believe there was intent for nodes to advertise this information.  And it makes sense that the network (and users) would want nodes to specify this information.

And my search-fu sucks.  I don't know why I've always failed at this.  I've tried "lite node" and "lightweight".  I found a page labeled "RFC for lightweight nodes" but it didn't contain this information.  I'm not so concerned with how to manage it, though.  I was more interested with the network protocol around "services" identification. 

If you've ever tried implementing the actual Bitcoin protocols, you'd realize that it is extremely complicated with countless nuances and pitfalls that have the potential to break the entire system.  And it's amusing you used the rocket science phrase with me, as I actually do rocket science at my full-time job.  It's complex programming, math, algorithm/computational optimization, and most importantly--collaboration between lots of developers with a lot of risk of catastrophic failure if you do it wrong.  I consider what I do at my job to share a lot of characteristics with developing for Bitcoin.  I feel very challenged by both.  

As a bunch of part-time hobbyists, I understand the inclination to keep things less formal.  Most open-source projects can get away with this.  But Bitcoin has the goal of wide acceptance outside of the small clique of nerds that currently use it.  A couple months ago I became interested in developing for Bitcoin and was very disappointed to see a lack of documentation and structure.  It feels like someone's pet project (hell, it actually is).  The learning curve is steep, and browsing/participating in forums seems to be the only way to get the information you need to contribute.  This is not a good recipe for wide acceptance.  

This is part of the reason few people, if any, have succeeded in making a usable, alternative BTC client.

This does not seem optimal.  The problem is that if a lot of people are using lightweight nodes that don't identify themselves as such, there's a risk for major network bottlenecks.  You try to propagate information on the network, but none of your peers will actually help you because you accidentally connected to too many lite nodes.  I see a necessity for nodes to be able to query the capabilities of their peers, to make sure that they are connected to a sufficient number of full-nodes that will help propagate important information.




At the moment, I'm working on the first option (full blockchain but no participation), with the intention to fork a branch of the second type (headers only, keeping only block data that is relevant to my wallet).  I currently envision having client software on your computer with the full blockchain and a phone app that only keeps headers.  In the future when the blockchain is big, both pieces would be headers-only, but perhaps with more information stored on the computer.

Perhaps I should search bitcointalk.org through google, I bet that would get me better results.  Regardless,  this probably should've been in the alternative clients subforum, so sorry about that...

I should've stuck to the specific question which is: how should the node identify itself on the network?  It needs to let other nodes know it's not a full node.  But I don't see anywhere on the specification page how this is actually achieved.  I see a 64-bit "services" bitfield in the version message, but I see nothing about how to identify that my node has the capabilities listed there.  Perhaps that specification is just not up to date.

The use-case here is like any other lite-client:   reduced resource utilization, and design focused on the human interface without having to reimplement the entire BTC protocol and risk forking the blockchain when I do it wrong.  Multi-sig transactions will be useless if no one can use them.  Wallet security could greatly benefit from a variety of features that aren't necessarily appropriate for the reference client, and will have questionable success without testing them on the network with real users. 

I whole-heartedly disagree.  Right now Bitcoin -- specification, protocol, documentation, standards - is seriously lacking formalism, and to the detriment of BTC as a whole.  I don't know why people are so afraid of having things well-structured.  This isn't a space-shuttle, but I'd argue it's actually similar in many respects: it's a damned-complicated system with millions of dollars at stake, with the goal to be used by millions of people and businesses, and easy to break with poorly planning/development (look at the endless scenarios that could cause blockchain forks).  I believe a big part of Bitcoin's success will be not just the core developers improving bitcoin, but the capability of other parties to get involved.

If the community is too afraid to add formalism to Bitcoin, it will forever continue to look like an experiment that never got out of the garage, and won't be taken seriously but such parties that would otherwise be interested and enable Bitcoin to grow.  If they can't figure out how to use, implement, and improve Bitcoin, it doesn't have a chance to succeed.

In fact, I think I'll make a BIP, specifically to better improve documentation and protocol specification.  That is a solid BIP.

I've been looking at the BTC Protocol Specification to understand what I would need to do differently (network-wise) if I were to put a "lite" node on the network.  So far, all I see is a "services" bitfield, but it only has one bit defined:  whether the node will relay headers-only, or all block information.  Wouldn't this be the place where you would specify that you are a lite-node?  Is there another way to do this? 

Here's my understanding of how a lite-node will operate:

• Will not verify any information - assumes that headers and tx's it receives (especially those in the blockchain) are valid
• Will not relay/broadcast any information to other nodes because it cannot verify it (is it okay to relay headers if they have the right number of leading 0 bits?)
• Will connect to peers in the way a full node does, but with information identifying its limited capabilities
• Can request information from the nodes, such as headers and block data, but will not forward any of it
• Will receive broadcasted tx and headers without requesting them.
• Can construct and broadcast new tx packets to connected nodes (with tx-fee info exchange)
• Will provide the user payment verification

I guess my question is more generally about what capabilities/behaviors are appropriate for a lite-node and is it inappropriate to be leeching from the network like this without providing the support of a full node.  I can see how peers that are connected to you have less "influence" since your lite-node is filling the slot of a "connected peer" but won't be broadcasting important information, such a new blocks and tx data.  However, we all know that in the far future, all average users will be using lite-nodes, since the blockchain will be too big to store on an average HDD computer... so this isn't unreasonable at all...

That's perfect, thanks!  I'm realizing that, while I don't have the C++ skills to understand much of the client architecture, I can decipher snippets like this (if I know where to find  them).


Not all contracts are executed directly through the blockchain.  Some of them may involve handing another party a partially signed transaction, and letting them sign and broadcast it under whatever conditions are called for.  If the transaction hash can keep changing unpredictably, then it seems then this isn't feasible at all, regardless of whether the network would allow it (loosely speaking).

You're right, I mixed two concepts here.  The client software is designed right now with a lot of flood-defense features, because it is known that the network is not hosting the volume of transactions you're talking about.  ATM, it is assumed that if there is that much traffic, it is someone trying to spam/flood the network, not legitimate users.  However, if volume legitimately increases, then surely the devs will loosen up a bit on those defenses...

I'm not entirely up to date on leading theories for dealing with massive transaction volume (i.e. if BTC were trying to replace all CC transactions), but much of it has to do with converting the standard user to lite nodes (not storing the entire blockchain, only scanning and keeping what is relevant), and only the miners/pools need to store the whole chain. 

Two ways that I'm familiar with are not particularly easy:  the blockchain could be converted into distributed storage, so that very few nodes will have to maintain the entire thing.  Alternatively, the way transactions are executed, allows for pruning of old transactions.  Individual nodes could do this without problem.  But if tx volume really gets big, even expensive nodes might not be able to handle it, and it might be necessary to reach some kind of global agreement to forget the distant past, and "lock in" the current state of the network at a given time.  This would be a major coordinated effort/problem, but if it is "solved" this could allow the network to reset the storage requirements every couple of years...

I'm sure there's other options and theories, but I haven't been following it too closely.  You might consider starting here, where you can see a well-known security researcher asking the same question.  I'm sure there's a lot of discussion on this in the forums.

Btw, you get a very similar vulnerability through timejacking (aka "forcing clock drift").  It's in the highest class of vulnerability according to the Bitcoin weaknesses page.  It does almost exactly what you propose, by forcing the node's clock to be more then 2 hours out of sync, it will reject a perfectly valid block while the rest of the network continues normally.  Since the rest of the network is building off that block, the victim node will not see any new blocks except for "poisoned blocks" sent by the attacker (who needs a bit of computing power to produce them, but nowhere near 50%).

I agree, nodes should "respect" block headers with the correct proof-of-work regardless of who sent them.  Although, this wouldn't benefit a victim of timejacking, I can't think of why it would be bad to accept them.

As for your last point, how does a node actually determine network speed?  Is it averaged over a certain number of previous block timestamps?  If it can measure this reliably and without too much latency, then this could be a very reliable indicator of something abnormal occurring.  In addition to targeted attacks like you suggest, if someone tried to DoS the main mining pools in an effort to boost their own computing power past 50%, there would be a similar drop in global computation speed and the client would go into some sort of safe-mode.

It would take a little bit longer, but it seems it could be checked very reliably through a simple header+merkle root scan.  The blk0001.dat file has the following form:


If there's a bit error in a header, it won't have the leading zeros.  If there's a bit error in a transaction, the merkle root won't match up.  We know the magic bytes in advance, and if the numBytes or numTx values are off, our parser will crash.  With an efficient algorithm for scanning the blockchain, the entire check could be done in less than a minute.

Someone who downloads the blockchain from scratch will get a blk0001.dat file that contains only blocks on the main chain.  However, after the initial download, if you leave your client running, you will invariably pick up the occasional invalid block which will get stored in the blk0001.dat file (because you don't know it's going to be invalid until it's already been added to your file).  This means that your calculated hash will only match between two people if they both just downloaded the blockchain.

That's a topic of great debate these days.  You just described what credit card companies process perpetually, and there's no way the BTC network as it's currently implemented could handle such transaction volume.  But there's lots of options for dealing with it, and sufficient time to figure it out before it's necessary.

Until then, there's a plethora of protocol protections against this.  Most of it falls under the category of "flood defense."  Peers that transmit too much data get disconnected, transaction fees would make this very expensive, and the the tx prioritizaiton scheme does a good job of making sure legitimate transactions can get through first.

Luckily, in the context of bitcoin, this isn't so critical.  There's redundancy everywhere, since there are hashes for everything, and a million other nodes checking your solution before it's accepted.  The only real risk is getting your client dorked up, submitting what you think is a valid transaction and being confused when it's not accepted by the network.  I don't think millions of dollars will be lost in Bitcoins due to this, but certainly, a client that can't recover from a bit error in the blockchain file could cause all sorts of problems for a high-volume user/company that would lose money from downtime.

I have never appreciated the value of ECC RAM, because I've done computation on computers for 10 years without ever noticing an explicit error.  Although, I can see how certain applications need the guarantees against it.  Regular RAM errors are measured in errors/day, whereas ECC RAM errors are measured in errors/century.  

I think we're in agreement.  My own use cases don't require that level of paranoia, but someone with millions of dollars would probably prefer to spend the extra effort to avoid the remaining attack vectors. 

One thing you could do is keep the wallet on a laptop, and when you want to move the money, it will display a QR code on-screen containing the signed transaction (or print it).  Just hold it up to your online computer's camera and it will detect and broadcast automatically.  Perhaps it's more effort than it's worth, but it would probably be easier and less cumbersome for the rich guy that doesn't want to do a lot of work to maintain an offline wallet.

Well, when it comes to Bitcoin, hashing, cryptography, and millions of dollars, I don't have much of a choice but to be extremely thorough.  There is no "partial credit,"   and the cost of mishandling large transactions is not worth the time saved by taking shortcuts.  I want to get this stuff right from the start.

This problem is the kind of thing that happens so rarely, it may not be handled elegantly in the current client.  I'm actually more concerned with the fact that I've made an assumption about blk0001.dat that, if it doesn't hold, will botch my tx scanning code.  If it's an extreme boundary condition, then perhaps I can just work on detecting it and redownload/rescan the chain when it does.  It's annoying, but happens infrequently enough to not cause major problems.

So if the error occurred in a block with less than 2500 confirmations, the client would find it and correct it in-place?  What if the new block ends up being a different size than the original block on disk? 

Last week, I was trying to decide if I could make the assumption that I will always be able to read the blocks in the file order of blk0001.dat and know that all TxIns read will reference a TxOut that I've seen before.  If the block data is corrected by appending to the end of the file, this assumption fails.  But it also seems like it would be "difficult" to correct the block in place if it's a different size than the original, erroneous block data.

Until spending four hours looking for this bit-flip, I had actually ruled out any possibility of the blocks being out of order.  If I can't make this assumption, tx scanning gets more complicated, since TxIns don't always identify the redeeming address.  You have to look at its TxOut to know for sure.   If the correct TxOut is after the TxIn,  then you have no way to tell if it's yours.  Perhaps requiring two scans of the block chain to find all your transactions...

It looks like this question was never answered, and I still want to know how this works.  If a tx uses nLockTime, it allows the tx to be replaced with higher sequence numbers up until the locktime.  But won't these replacement transactions have different hashes than the original?  If so, how can the client/network know which transaction is being replaced other than trying to match up TxOuts being spent?  Presumably the TxOuts could change, which makes this even more unreliable.  And of course, some of the more complex contracts might actually have future transactions based on a previous transaction with a locktime, which would have a changing hash...