The size of the blockchain...

[Peter R]

Trade with caution. User interested in generating double spends and willing to share profits with miners to defraud others.

This is why he was negative trust rating. That doesn't tell you if someone is a troll or not.

I know skooter to be a troll from his past comment history, several in response to my own posts.  The fact that Gerald (DeathAndTaxes) took the time to report him for pursuing double-spends for the purposes of defrauding others tells me that skooter is also a scammer.  

[Massimo80]

Nice idea, but a device like this would need power and an Internet connection. And possibly a wireless connection, if you are unable to get it near an Ethernet cable. And of course a custom-designed board, which has development and production costs.

On the software side, it would require at the very least a TCP/IP stack and network drivers, and probably much more in order to be able to port bitcoind to it.

What I'm saying it's not this is not feasible, but it could get much more complex and costly than you think. And this calls for the main question: why? What would be the purpose of developing, building, selling, buying and running such a device, which can only give a very small benefit to the Bitcoin network and can't perform any useful function at all for his owner?

[Peter R]

I'm not so concerned of the size of the blockchain once it's downloaded. What's more concerning is the bandwidth required to download it. Internet speed (for most) is not keeping pace with the increase in size of harddrives. I personally just got 1MB/s at home and I have cable. That's still a large amount of time to download a 17(?) gig file. Not to mention data caps.

This would be another benefit of low-costs single-purpose plug-and-play bitcoin node hardware.  Rather than downloading the blockchain from a single peer node (as is currently the case), the card could come pre-loaded with the blockchain (or you could choose to download it from a faster torrent).

The microSD card could be partitioned to contain a section for storing the growing blockchain, and another section for storing the bitcoin binary that gets bootloaded by the ARM-core processor.  You would update your bitcoin program by copying the new binary compiled by a trusted source onto the card.   

You could probably do some other useful things too.  If these devices became cheap and simple enough, they could come as part of bitcoin miners with added firmware to run P2P pool by default.  So you just plug your miner in and immediately it begins mining at P2Pool.  The "configuration details" such as where the mined bitcoins get sent can be specified in a simple text file on the SD card.  Alternatively, if these devices become cheap enough, they could be an integral part of bitcoin point-of-sales devices in the future. 

[DeathAndTaxes]

bitcoin QT can sometimes use upwards of 1 GB of RAM.

Well there's your problem skooter up there in bold.  We are talking about a custom implementation of bitcoind to create a tiny and low-cost bitcoin node.  We don't even need a wallet implementation, and we definitely don't need the QT GUI (there's no display lol).

Well most of that memory usage is the db cache.  It probably can be done on a custom ARM based board however it isn't going to be a trivial project and you probably want the board to have a gig of memory.  The UXTO is only going to get larger over time.

[Carlton Banks]

It's been said before, but bears repeating: internet routers make the ideal candidate for a low cost bitcoin node device. The trouble is that both RAM and storage of a typical 2014 device are not in anywhere close to the right kind of league to do the job.

But maybe several development cycles of a mass produced standalone device (as I believe to be what OP suggests) could bring us somewhere close, and perhaps then in ~5 years they could be cheap enough to be packaged together in a slightly premium priced router product.

It certainly makes sense to sell such a device in the interim, as many a local network will be running more than one Bitcoin Core or Armory wallet as and when adoption ramps up. There will be a demand for just such a device that stores, maintains and serves one blockchain copy per building/household. Firing up bitcoin core once every few days, or even only once a week can get tiresome. Better to have an appliance that ensures you're ready to send or receive current transactions at all times (and one that has a dedicated amount of disk space, running out of storage is not an uncommon experience when running a bitcoin node)

[Peter R]

Nice idea, but a device like this would need power

One of the big appeals of ARM-core processors are their low power consumptions.  For example, a cortex-M4 core processor with 1.2V core consumes only 33 uW / MHz.  At 1.25 MIPS/MHz and 150 MIPS operation, this gives a power consumption of ~4 mW.  I believe a large portion of the energy requirements would go towards ECDSA signature verification.  I estimated earlier in this thread that an ARM-core processor running at 150 MHz could execute secp256k1 verifies in 20 ms, which at 4 mW represents 80 microjoules per verify.  Assuming 2 signature checks per TX and 10 TXs / s, this is 1600 uJ/s or 1.6 mW.  So this is very low power.  

Of course there are many other activities that will consume power too (reading and writing to the SD card, TCP/IP communication), but I think it is generally agreed that performing the signature checks is the most work and this process can be made fairly low power.  

and an Internet connection. And possibly a wireless connection, if you are unable to get it near an Ethernet cable.

This is actually quite straight forward, as there are both chips that implement TCP/IP and processors with integrated ethernet modules.

And of course a custom-designed board, which has development and production costs.

Yes, you would need this.  

On the software side, it would require at the very least a TCP/IP stack and network drivers, and probably much more in order to be able to port bitcoind to it.

The TCP/IP stack can come as part of an IC, or as a code library for the processor.  Attaching a microcontroller to the internet is routine nowadays.  However, implementing all the networking features of bitcoin would still certainly be a lot of work.  

What I'm saying it's not this is not feasible, but it could get much more complex and costly than you think.

I'm starting to feel confident that it is feasible too.  But how do you know how complex or costly I think the R&D process would be?  I assume it would be a great deal of work.  But I am interested in the question "how simple, cheap and low-power can a useful bitcoin node by made?"  I think this is an important question.

And this calls for the main question: why? What would be the purpose of developing, building, selling, buying and running such a device, which can only give a very small benefit to the Bitcoin network and can't perform any useful function at all for his owner?

From my experience, once a project like this is complete and working, people find all sorts of applications.  

For example, a lower-power, small and inexpensive bitcoin node could be integrated in certain point-of-sales hardware, or it could come with bitcoin miners and run P2P pool by default (helping to decentralize mining efforts).  

[DeathAndTaxes]

bitcoind requires at least 1GB of RAM and usually more.

Thanks Cypherdoc.  I am trying to make sense of this: is this because bitcoind is loading the unspent outputs into RAM for faster checking?  My intuition is still telling me that for a custom implementation I just need enough RAM to comfortably pool the unconfirmed transactions.

The memory usage is primarily storing the UXTO in memory.  The memory pool is also kept in memory but it is much smaller than the UXTO. You could probably reduce the memory footprint with some "intelligent" caching system scoring each output based on the likelihood it will be in the next transaction (i.e. the unspent output from the coinbase of block2 is probably not going to be used in the next tx so dropping it from the in memory cache won't be an issue).

Remember reads from disk are very slow (compared to memory) and you can't validate a transaction until you lookup the outputs.  If none of the unspent outputs are in memory well you have now made 100% of tx validations requiring a round trip to the disk.  Right now throughput of the network is slow enough that you won't fall behind however for bootstraping a new node it is going to be painfully slow.

[Peter R]

Well most of that memory usage is the db cache.  It probably can be done on a custom ARM based board however it isn't going to be a trivial project and you probably want the board to have a gig of memory.  The UXTO is only going to get larger over time.

Thanks DeathAndTaxes.  If there is a "gotcha," I think it is this.  Earlier I proposed storing the UXTO on the SD card rather than RAM, organized in some efficient manner for fast cross-checking.  The UXTO would be updated each block, and earlier in the thread I showed that erase/write cycles every 10 min should still provide the memory card with several years of life.  

Am I missing something?  Is there a reason I need to store the UXTO in RAM that I cannot see?  It seems to me that even 64 MB or RAM would be sufficient (this is available on a 6x8mm ball-grid array chip for $1.50).  64 MB gives lots of room to pool the unconfirmed transactions, accept new blocks from peers, etc.  


Like you said, this certainly isn't a trivial project.  For the time being, it is a thought experiment: "how simple, cheap, small, and low-power can a useful bitcoin node be made?"

[Peter R]

The memory usage is primarily storing the UXTO in memory.  The memory pool is also kept in memory but it is much smaller than the UXTO. You could probably reduce the memory footprint with some "intelligent" caching system scoring each output based on the likelihood it will be in the next transaction (i.e. the unspent output from the coinbase of block2 is probably not going to be used in the next tx so dropping it from the in memory cache won't be an issue).

Remember reads from disk are very slow (compared to memory) and you can't validate a transaction until you lookup the outputs.  If none of the unspent outputs are in memory well you have now made 100% of tx validations requiring a round trip to the disk.  Right now throughput of the network is slow enough that you won't fall behind however for bootstraping a new node it is going to be painfully slow.

LOL you began answering my next question before I posted it. 

This all makes sense to me then, and explains why Cypherdoc's bitcoind node uses so much RAM: it is mostly to permit fast look-ups from the UXTO.  If the UXTO can be organized and stored efficiently on a SD card, perhaps with the help of some intelligent caching system like you said, this might work.


(I also think people are underestimating how fast SD cards (especially nowadays). SD cards do not force a file-system like FAT, they are basically just an SPI-interface to a huge array of flash.  So I can read small blocks out of flash with an SPI command to address the memory I want and then clock in even short blocks of data at tens of megabytes per second.  There is perhaps less overhead here than it may at first appear.)

[Massimo80]

One of the big appeals of ARM-core processors are their low power consumptions.

I'm not concerned about power usage, but about the necessity to supply power to the device, which requires dedicated hardware. Power supplies nowadays are indeed quite small, but one of them would still probably be bigger than the device itself.

From my experience, once a project like this is complete and working, people find all sorts of applications.  

For example, a lower-power, small and inexpensive bitcoin node could be integrated in certain point-of-sales hardware, or it could come with bitcoin miners and run P2P pool by default (helping to decentralize mining efforts).  

This hypotetical device and a POS have completely different (and actually opposite) design goals: a POS must act as a wallet but doesn't require full node capabilities, while this project wants to create a full node without wallet functions.

Mining is a completely moot point. Given the current difficulty, even a million ARM processors wouldn't even be noticed by the network. They would only waste power.

[DeathAndTaxes]

If the flash (and controller) is sufficiently fast it may be possible to use that as the UXTO lookup storage.  This would drop the memory requirements to near negligible amounts (enough to hold memory pool, current block, and if used as a wallet the user's personal pubkeys).  A lot would come down to how long lookup of a random output from the UXTO would take.  A good place to start would be getting a development board.  Instead of working with a ARM CPU directly you could work with a microcontroller which is based on an ARM CPU.

I am a big fan of STI Micro.  This is their STM32F4 series. 
http://www.st.com/web/en/catalog/mmc/FM141/SC1169/SS1577/LN1806

The 429/439 is the most powerful microcontroller they offer.  It gives you 256KB SRAM, 2MB of flash, ethernet & usb connectivity, plus a hot of other stuff you probably won't need.  The 439 is the 429 with hardware crypto acceleration but sadly does not support the scep256k1 curve (I called to make sure ).  The main differences between the pin packages is the larger ones support more analog and digital IO pins. 

There is a development board available for rapid prototyping
http://www.st.com/web/catalog/tools/FM116/SC959/SS1532/PF259090

[DeathAndTaxes]

Mining is a completely moot point. Given the current difficulty, even a million ARM processors wouldn't even be noticed by the network. They would only waste power.

True but it could be useful to have high power ASIC miners connect to the node which is connected to p2pool.

[Massimo80]

Mining is a completely moot point. Given the current difficulty, even a million ARM processors wouldn't even be noticed by the network. They would only waste power.

True but it could be useful to have high power ASIC miners connect to the node which is connected to p2pool.

This would require USB or serial support, and mining software. This gets increasingly complex as soon as someone starts looking for real use cases, which IMHO means the base design isn't really that useful

[telepatheic]

Bitcoin transactions arrive at up to about 10 per second so checking the inputs are valid by reading flash storage is not too difficult. The problem is you want to avoid DOS attacks by having quick access to transaction hashes There are about 10 million unspent outputs, if you use a modified encoding which hashes transaction ids down to just 7 bytes and 1 byte for the output count this means you need 80MB of memory plus a few MB for the mempool and recent blocks.

[DeathAndTaxes]

This would require USB or serial support, and mining software. This gets increasingly complex as soon as someone starts looking for real use cases, which IMHO means the base design isn't really that useful

No it wouldn't.  Miners connect to the node (just like they do any node running p2pool) over TCP/IP.  The mining software is on the miner.   Instead of pointing it at a pool wallet you point it at your p2pool node, exactly as you do now if p2pool is running on a desktop.

As for not useful well that depends.  Just as a hardware which runs a full node for the network and does nothing for the user well your right that is beyond stupid.  Why would one buy one?  However how about this.   Your wallet is on your computer, you don't want to run bitcoind locally for a couple of reasons such as you have multiple computers running bitcoind on all of them it excessive, or you don't use your wallet everyday and hate the fact that when it syncs up you need to wait.

So instead you use a light wallet which connects to your local bitcoin node (running on that <1W device connected to your local network).  Got 8 wallets spread across multiple computers no problem they can all point to the same local bitcoind.  Now you could use a SPV client but maybe you don't like the security compromise and the fact that it doesn't strengthen the network.

[MegaHustlr]

I don't understand what u mean here won't the memory stick always be the same size and shouldn't change?

He means the size of the blockchain itself.

[Peter R]

If the flash (and controller) is sufficiently fast it may be possible to use that as the UXTO lookup storage.  This would drop the memory requirements to near negligible amounts (enough to hold memory pool, current block, and if used as a wallet the user's personal pubkeys).  A lot would come down to how long lookup of a random output from the UXTO would take.  A good place to start would be getting a development board.  Instead of working with a ARM CPU directly you could work with a microcontroller which is based on an ARM CPU.

Thanks for this.  Do you think it is possible to keep the UXTO stored in some way that I can find a specific output in O(log n) time, where n is the number of unspent outputs?  I am not really qualified to say because I don't even know how the UXTO are currently organized by bitcoind.  But it seems that right now I could sort all the UXTO in canonical order to permit O(log n) look-ups.  I imagine there would be some way to keep the UXTO organized such that this remains the case moving forward in time too...

Personally, this is just a thought experiment for me at the moment, as my hands are full with a much simpler hardware project.  But it is interesting to consider what could be done with existing technology, time, and money.   

[DeathAndTaxes]

Bitcoin transactions arrive at up to about 10 per second so checking the inputs are valid by reading flash storage is not too difficult. The problem is you want to avoid DOS attacks by having quick access to transaction hashes There are about 10 million unspent outputs, if you use a modified encoding which hashes transaction ids down to just 7 bytes and 1 byte for the output count this means you need 80MB of memory plus a few MB for the mempool and recent blocks.

That is a good point.  For a set consisting of 10 million objects 256 bits is not necessary to avoid excessive collisions.  Taking the left x bits should be sufficient as this isn't being used for cryptographic purposes (and NIST certifies SHA-2 to be used for truncated hashes in some applications anyways).  The index on the other hand requires a little more thinking.  The protocol allows more than 256 outputs per transaction and is stored as a VarInt which can potentially be up to 8 bytes long (but realistically I couldn't more than 3 is probably never going to happen).  I guess one option would be to to use x bytes for the lookup hash where x = (8 - VarIntSize).  Worse case scenario you could use a full 8 bytes for the hash and full 8 bytes for the index but that would double the memory requirements.

[fairglu]

SD Flash will die pretty quick if you continuously write to it, like for a blockchain database.

So ok for backup, but not really for a node.

[Peter R]

I'm not concerned about power usage, but about the necessity to supply power to the device, which requires dedicated hardware. Power supplies nowadays are indeed quite small, but one of them would still probably be bigger than the device itself.

The power supply required is intimately related to the device's power consumption.  The kind of device I'm imagining could run from a simple walwart, power-over-ethernet, or directly from a low-voltage outlet if LV wiring was available (as is becoming more common--take for example AC outlets that now include a 5V USB charging port).  

This hypotetical device and a POS have completely different (and actually opposite) design goals: a POS must act as a wallet but doesn't require full node capabilities, while this project wants to create a full node without wallet functions.

You are thinking too specific and perhaps PoS was not the best example.  But imagine a tiny low-power, low-cost bitcoin node with trivial set-up.  I think some people would find innovative uses for these, while others would buy them simply to help support the network.

Mining is a completely moot point. Given the current difficulty, even a million ARM processors wouldn't even be noticed by the network. They would only waste power.

You wouldn't mine with an ARM processor.  You would mine with a bitcoin miner that employs 1 or more SHA256 ASICs.  If the ARM-core bitcoin node was cheap and simple enough, the "hashing device" could be transformed by default into a "plug-and-play P2P miner" by the manufacturer.  This may help to decentralize mining.