Bitcoin Forum
May 21, 2018, 07:16:07 PM *
News: Latest stable version of Bitcoin Core: 0.16.0  [Torrent]. (New!)
 
   Home   Help Search Donate Login Register  
Pages: 1 2 3 4 [All]
  Print  
Author Topic: The size of the blockchain...  (Read 3817 times)
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 01, 2014, 10:43:10 PM
 #1

…is pretty small.  It fits on the tip of a finger with enough space (46 GB free, 18 GB used) to grow for another 1-3 years.  





This got me thinking: does anyone have bitcoind running on an ARM processor?  It seems it should be possible to create a "bitcoin node" based on an ARM processor, a microSD card, and a simple ethernet interface IC.   The entire node would be only slightly larger than the RJ-45 connector on the mating ethernet cable.


(Yes, I know about raspberry-pis, but this would be vastly smaller, cheaper.)  


Run Bitcoin Unlimited (www.bitcoinunlimited.info)
1526930167
Hero Member
*
Offline Offline

Posts: 1526930167

View Profile Personal Message (Offline)

Ignore
1526930167
Reply with quote  #2

1526930167
Report to moderator
1526930167
Hero Member
*
Offline Offline

Posts: 1526930167

View Profile Personal Message (Offline)

Ignore
1526930167
Reply with quote  #2

1526930167
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1526930167
Hero Member
*
Offline Offline

Posts: 1526930167

View Profile Personal Message (Offline)

Ignore
1526930167
Reply with quote  #2

1526930167
Report to moderator
telepatheic
Jr. Member
*
Offline Offline

Activity: 56
Merit: 0

Get off the web. Go do something in meatspace!


View Profile
May 01, 2014, 10:54:13 PM
 #2

Quote
does anyone have bitcoind running on an ARM processor?

People have got it running on raspberry pis. The amount of memory (RAM) you require is the main problem.
skooter
Member
**
Offline Offline

Activity: 70
Merit: 10


View Profile
May 01, 2014, 10:57:54 PM
 #3

Quote
does anyone have bitcoind running on an ARM processor?

People have got it running on raspberry pis. The amount of memory (RAM) you require is the main problem.

Yeah, storing the data is no prob. You can't use it without a lot of RAM though.
durrrr
Sr. Member
****
Offline Offline

Activity: 434
Merit: 250


View Profile
May 01, 2014, 10:58:58 PM
 #4

I don't understand what u mean here won't the memory stick always be the same size and shouldn't change?

Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 01, 2014, 11:12:27 PM
 #5

Quote
does anyone have bitcoind running on an ARM processor?

People have got it running on raspberry pis.

Thanks.  I do know about the raspberry pis, but I'm thinking even smaller: a tiny little custom board with only the bare minimum hardware required to run bitcoind.  Perhaps the ARM processor could still run linux, or perhaps in the future we could write a version of bitcoind that implemented its own simple OS (as is commonly done for DSP projects).  


Quote
The amount of memory (RAM) you require is the main problem.

I suppose you need enough RAM to pool the unconfirmed transactions, right?  A 64 MB chip could hold around 100,000 TXs.   Here's a 64 MB SDRAM chip that is only 6mm x 8mm in size and costs about $1.50: http://www.digikey.com/product-detail/en/AS4C4M16S-6BIN/1450-1077-ND/4531924

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
skooter
Member
**
Offline Offline

Activity: 70
Merit: 10


View Profile
May 01, 2014, 11:17:38 PM
 #6

Quote
does anyone have bitcoind running on an ARM processor?

People have got it running on raspberry pis.

Thanks.  I do know about the raspberry pis, but I'm thinking even smaller: a tiny little custom board with only the bare minimum hardware required to run bitcoind.  Perhaps the ARM processor could still run linux, or perhaps in the future we could write a version of bitcoind that implemented its own simple OS (as is commonly done for DSP projects).  


Quote
The amount of memory (RAM) you require is the main problem.

I suppose you need enough RAM to pool the unconfirmed transactions, right?  A 64 MB chip could hold around 100,000 TXs.   Here's a 64 MB SDRAM chip that is only 6mm x 8mm in size and costs about $1.50: http://www.digikey.com/product-detail/en/AS4C4M16S-6BIN/1450-1077-ND/4531924

bitcoin QT can sometimes use upwards of 1 GB of RAM.
Mobius7
Hero Member
*****
Offline Offline

Activity: 539
Merit: 500



View Profile
May 01, 2014, 11:21:11 PM
 #7

I don't understand what u mean here won't the memory stick always be the same size and shouldn't change?

What memory stick? OP was talking about ARM processor and microSD card.

Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 01, 2014, 11:21:48 PM
 #8

bitcoin QT can sometimes use upwards of 1 GB of RAM.

Well there's your problem skooter up there in bold.  We are talking about a custom implementation of bitcoind to create a tiny and low-cost bitcoin node.  We don't even need a wallet implementation, and we definitely don't need the QT GUI (there's no display lol).

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
telepatheic
Jr. Member
*
Offline Offline

Activity: 56
Merit: 0

Get off the web. Go do something in meatspace!


View Profile
May 01, 2014, 11:34:02 PM
 #9

Quote
bitcoin QT can sometimes use upwards of 1 GB of RAM.

With some custom settings to limit what gets held in memory, you could get this down to less than 10MB, but it would require a fair bit of work.
cypherdoc
Legendary
*
Offline Offline

Activity: 1764
Merit: 1000



View Profile
May 01, 2014, 11:43:36 PM
 #10

bitcoin QT can sometimes use upwards of 1 GB of RAM.

Well there's your problem skooter up there in bold.  We are talking about a custom implementation of bitcoind to create a tiny and low-cost bitcoin node.  We don't even need a wallet implementation, and we definitely don't need the QT GUI (there's no display lol).

bitcoind requires at least 1GB of RAM and usually more.
Soros Shorts
Donator
Legendary
*
Offline Offline

Activity: 1592
Merit: 1000



View Profile
May 01, 2014, 11:47:47 PM
 #11

Quote
bitcoin QT can sometimes use upwards of 1 GB of RAM.

With some custom settings to limit what gets held in memory, you could get this down to less than 10MB, but it would require a fair bit of work.
Whatever work is done must not slow down the speed at which the node works. You would be doing the network a disservice if your node took a long time to validate and relay messages and blocks because of your low memory footprint and under-powered CPU configuration.
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 01, 2014, 11:58:27 PM
 #12

bitcoind requires at least 1GB of RAM and usually more.

Thanks Cypherdoc.  I am trying to make sense of this: is this because bitcoind is loading the unspent outputs into RAM for faster checking?  My intuition is still telling me that for a custom implementation I just need enough RAM to comfortably pool the unconfirmed transactions.

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 12:53:10 AM
 #13

Quote
bitcoin QT can sometimes use upwards of 1 GB of RAM.

With some custom settings to limit what gets held in memory, you could get this down to less than 10MB, but it would require a fair bit of work.
Whatever work is done must not slow down the speed at which the node works. You would be doing the network a disservice if your node took a long time to validate and relay messages and blocks because of your low memory footprint and under-powered CPU configuration.

I think people underestimate how powerful low-cost DSPs have become.  For example, C. Gouvea was able to verify secp256k1 ECDSA signatures in 700 ms using an ultra-lower power 16-bit microcontroller running at a only 8 MHz (and available in a tiny 3 x 3 mm package size).  Even very cheap ARM-core processors ($2 - $4) will typically run at ~150 MHz (and with 32-bit registers).  So let's say signature verification for each received TX takes no more than 20 ms, at which point you can relay the transaction to your other peers.  This seems perfectly acceptable to me.  

The only other issue I can think of is RAM, but I don't see why you can't store the unspent outputs in the SD card itself (new SD cards permit reads in excess of 30 MB/s so I think the time it takes to cross check the outputs in a TX with the blockchain unspent outputs stored in flash will be short compared to the signature verification times [20 ms]).  It seems you just need sufficient RAM to pool the unconfirmed transactions (and some overhead to accept new blocks, etc), so upon first glance the 6x8mm 64 MB IC for $1.50 seems sufficient for this purpose.  

I think such a bitcoin node would be useful if it could be made only a few cm in dimensions, cost less than $10-$15 excluding the SD card, and be trivial to set-up.  

  


Run Bitcoin Unlimited (www.bitcoinunlimited.info)
Meuh6879
Legendary
*
Offline Offline

Activity: 1512
Merit: 1000



View Profile
May 02, 2014, 01:00:56 AM
 #14

Quote
bitcoin QT can sometimes use upwards of 1 GB of RAM.

With some custom settings to limit what gets held in memory, you could get this down to less than 10MB, but it would require a fair bit of work.

yes ... and it's very simple : decrease "cache" setting from 100MB to 10MB for example.
i must do this on my dedicated P2P server (because many others tasks work with bitcoin-QT in mode setting).
cypherdoc
Legendary
*
Offline Offline

Activity: 1764
Merit: 1000



View Profile
May 02, 2014, 01:05:13 AM
 #15

bitcoind requires at least 1GB of RAM and usually more.

Thanks Cypherdoc.  I am trying to make sense of this: is this because bitcoind is loading the unspent outputs into RAM for faster checking?  My intuition is still telling me that in a custom implementation I just need enough RAM to comfortably pool the unconfirmed transactions.

i'm not sure what's filling up the RAM altho UTXO sounds like a reasonable guess.

according to David Schwartz, bitcoind requires around 650MB.  sounds about right.  see here:

http://bitcoin.stackexchange.com/questions/8400/to-what-is-bitcoind-memory-usage-bound

and here:

https://bitcointalk.org/index.php?topic=313321.0

here's a top from one of my vps's:

skooter
Member
**
Offline Offline

Activity: 70
Merit: 10


View Profile
May 02, 2014, 02:00:30 AM
 #16

Quote
bitcoin QT can sometimes use upwards of 1 GB of RAM.

With some custom settings to limit what gets held in memory, you could get this down to less than 10MB, but it would require a fair bit of work.
Whatever work is done must not slow down the speed at which the node works. You would be doing the network a disservice if your node took a long time to validate and relay messages and blocks because of your low memory footprint and under-powered CPU configuration.

I think people underestimate how powerful low-cost DSPs have become.  For example, C. Gouvea was able to verify secp256k1 ECDSA signatures in 700 ms using an ultra-lower power 16-bit microcontroller running at a only 8 MHz (and available as in a tiny 3 x 3 mm package size).  Even very cheap ARM-core processors ($2 - $4) will typically run at ~150 MHz (and with 32-bit registers).  So let's say signature verification for each received TX takes no more than 20 ms, at which point you can relay the transaction to your other peers.  This seems perfectly acceptable to me.  

The only other issue I can think of is RAM, but I don't see why you can't store the unspent outputs in the SD card itself (new SD cards permit reads in excess of 30 MB/s so I think the time it takes to cross check the outputs in a TX with the blockchain unspent outputs stored in flash will be short compared to the signature verification times [20 ms]).  It seems you just need sufficient RAM to pool the unconfirmed transactions (and some overhead to accept new blocks, etc), so upon first glance the 6x8mm 64 MB IC for $1.50 seems sufficient for this purpose.  

I think such a bitcoin node would be useful if it could be made only a few cm in dimensions, cost less than $10-$15 excluding the SD card, and be trivial to set-up.  

  



I think write endurance would be an issue if you're trying to use an SD card as RAM.
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 02:22:00 AM
 #17

I think write endurance would be an issue if you're trying to use an SD card as RAM.

We aren't talking about using a SD card for RAM.  The SD card would be used to store the blockchain, update the blockchain with new blocks, update the unspent outputs each block, and bootload code segments of the bitcoin binary.  Typically, flash memory is rated between 100,000 - 1,000,000 write/erase cycles (2 - 19 years if cycles happen every 10 min) and that's always writing the same segment over and over.  With intelligent disk management to avoid "hot spots," this could be greatly increased.  So I would expect even low-grade memory cards to last many years.

But skooter you don't really care about what we are talking about anyways, as can be verified by your post history as a troll and the glaring red warning in your profile (EDIT: suggesting you are also a scammer).  





Run Bitcoin Unlimited (www.bitcoinunlimited.info)
Lauda
Legendary
*
Offline Offline

Activity: 1848
Merit: 1552


GUNBOT Licenses -15% with ref. code 'GrumpyKitty'


View Profile WWW
May 02, 2014, 11:58:14 AM
 #18

We aren't talking about using a SD card for RAM.  The SD card would be used to store the blockchain, update the blockchain with new blocks, update the unspent outputs each block, and bootload code segments of the bitcoin binary.  Typically, flash memory is rated between 100,000 - 1,000,000 write/erase cycles (2 - 19 years if cycles happen every 10 min) and that's always writing the same segment over and over.  With intelligent disk management to avoid "hot spots," this could be greatly increased.  So I would expect even low-grade memory cards to last many years.

But skooter you don't really care about what we are talking about anyways, as can be verified by your post history as a troll and the glaring red warning in your profile.


Quote
Trade with caution. User interested in generating double spends and willing to share profits with miners to defraud others.
This is why he was negative trust rating. That doesn't tell you if someone is a troll or not.
Also the blockchain has grown over the past years but it will be growing at an increasing rate.

.FORTUNE.JACK.
      ▄▄███████▄▄
   ▄████▀▀ ▄ ██████▄
  ████ ▄▄███ ████████
 █████▌▐███▌ ▀▄ ▀█████
███████▄██▀▀▀▀▄████████
█████▀▄▄▄▄█████████████
████▄▄▄▄ █████████████
 ██████▌ ███▀████████
  ███████▄▀▄████████
   ▀█████▀▀███████▀
      ▀▀██████▀▀
         
         █
...FortuneJack.com                                             
...THE BIGGEST BITCOIN GAMBLING SITE
       ▄▄█████████▄▄
    ▄█████████████████▄
  ▄█████████████████████▄
 ▄██
█████████▀███████████▄
██████████▀   ▀██████████
█████████▀       ▀█████████
████████           ████████
████████▄   ▄ ▄   ▄████████
██████████▀   ▀██████████
 ▀██
█████████████████████▀
  ▀██
███████████████████▀
    ▀█████████████████▀
       ▀▀█████████▀▀
#JACKMATE
WIN 1 BTC
▄█████████████████████████▄
███████████████████████████
███████████████████████████
██████████▀█████▀██████████
███████▀░░▀░░░░░▀░░▀███████
██████▌░░░░░░░░░░░░░▐██████
██████░░░░██░░░██░░░░██████
█████▌░░░░▀▀░░░▀▀░░░░▐█████
██████▄░░▄▄▄░░░▄▄▄░░▄██████
████████▄▄███████▄▄████████

███████████████████████████
███████████████████████████
▀█████████████████████████▀
vpitcher07
Sr. Member
****
Offline Offline

Activity: 342
Merit: 250


View Profile
May 02, 2014, 12:33:14 PM
 #19

I'm not so concerned of the size of the blockchain once it's downloaded. What's more concerning is the bandwidth required to download it. Internet speed (for most) is not keeping pace with the increase in size of harddrives. I personally just got 1MB/s at home and I have cable. That's still a large amount of time to download a 17(?) gig file. Not to mention data caps.

Bitcoin: The currency of liberty
1HBJSf3Lm9i8KxjZ7fuoN9FJ8hniniFbv4
FeedbackLoop
Hero Member
*****
Offline Offline

Activity: 742
Merit: 500



View Profile
May 02, 2014, 12:50:57 PM
 #20


This got me thinking: does anyone have bitcoind running on an ARM processor?  It seems it should be possible to create a "bitcoin node" based on an ARM processor, a microSD card, and a simple ethernet interface IC.   The entire node would be only slightly larger than the RJ-45 connector on the mating ethernet cable.


Would be great if someone mounted and sold those for the lazy people who would like to run a full node at lost cost and low work like me Tongue.

Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 01:19:38 PM
 #21

Quote from: DeathAndTaxes
Trade with caution. User interested in generating double spends and willing to share profits with miners to defraud others.
This is why he was negative trust rating. That doesn't tell you if someone is a troll or not.

I know skooter to be a troll from his past comment history, several in response to my own posts.  The fact that Gerald (DeathAndTaxes) took the time to report him for pursuing double-spends for the purposes of defrauding others tells me that skooter is also a scammer.  

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
Massimo80
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
May 02, 2014, 01:31:58 PM
 #22

Nice idea, but a device like this would need power and an Internet connection. And possibly a wireless connection, if you are unable to get it near an Ethernet cable. And of course a custom-designed board, which has development and production costs.

On the software side, it would require at the very least a TCP/IP stack and network drivers, and probably much more in order to be able to port bitcoind to it.

What I'm saying it's not this is not feasible, but it could get much more complex and costly than you think. And this calls for the main question: why? What would be the purpose of developing, building, selling, buying and running such a device, which can only give a very small benefit to the Bitcoin network and can't perform any useful function at all for his owner?
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 01:41:42 PM
 #23

I'm not so concerned of the size of the blockchain once it's downloaded. What's more concerning is the bandwidth required to download it. Internet speed (for most) is not keeping pace with the increase in size of harddrives. I personally just got 1MB/s at home and I have cable. That's still a large amount of time to download a 17(?) gig file. Not to mention data caps.

This would be another benefit of low-costs single-purpose plug-and-play bitcoin node hardware.  Rather than downloading the blockchain from a single peer node (as is currently the case), the card could come pre-loaded with the blockchain (or you could choose to download it from a faster torrent).

The microSD card could be partitioned to contain a section for storing the growing blockchain, and another section for storing the bitcoin binary that gets bootloaded by the ARM-core processor.  You would update your bitcoin program by copying the new binary compiled by a trusted source onto the card.   

You could probably do some other useful things too.  If these devices became cheap and simple enough, they could come as part of bitcoin miners with added firmware to run P2P pool by default.  So you just plug your miner in and immediately it begins mining at P2Pool.  The "configuration details" such as where the mined bitcoins get sent can be specified in a simple text file on the SD card.  Alternatively, if these devices become cheap enough, they could be an integral part of bitcoin point-of-sales devices in the future. 

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 02, 2014, 01:45:33 PM
 #24

bitcoin QT can sometimes use upwards of 1 GB of RAM.

Well there's your problem skooter up there in bold.  We are talking about a custom implementation of bitcoind to create a tiny and low-cost bitcoin node.  We don't even need a wallet implementation, and we definitely don't need the QT GUI (there's no display lol).

Well most of that memory usage is the db cache.  It probably can be done on a custom ARM based board however it isn't going to be a trivial project and you probably want the board to have a gig of memory.  The UXTO is only going to get larger over time.
Carlton Banks
Legendary
*
Offline Offline

Activity: 2002
Merit: 1168



View Profile
May 02, 2014, 02:18:56 PM
 #25

It's been said before, but bears repeating: internet routers make the ideal candidate for a low cost bitcoin node device. The trouble is that both RAM and storage of a typical 2014 device are not in anywhere close to the right kind of league to do the job.

But maybe several development cycles of a mass produced standalone device (as I believe to be what OP suggests) could bring us somewhere close, and perhaps then in ~5 years they could be cheap enough to be packaged together in a slightly premium priced router product.

It certainly makes sense to sell such a device in the interim, as many a local network will be running more than one Bitcoin Core or Armory wallet as and when adoption ramps up. There will be a demand for just such a device that stores, maintains and serves one blockchain copy per building/household. Firing up bitcoin core once every few days, or even only once a week can get tiresome. Better to have an appliance that ensures you're ready to send or receive current transactions at all times (and one that has a dedicated amount of disk space, running out of storage is not an uncommon experience when running a bitcoin node)

Vires in numeris
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 02:19:28 PM
 #26

Nice idea, but a device like this would need power

One of the big appeals of ARM-core processors are their low power consumptions.  For example, a cortex-M4 core processor with 1.2V core consumes only 33 uW / MHz.  At 1.25 MIPS/MHz and 150 MIPS operation, this gives a power consumption of ~4 mW.  I believe a large portion of the energy requirements would go towards ECDSA signature verification.  I estimated earlier in this thread that an ARM-core processor running at 150 MHz could execute secp256k1 verifies in 20 ms, which at 4 mW represents 80 microjoules per verify.  Assuming 2 signature checks per TX and 10 TXs / s, this is 1600 uJ/s or 1.6 mW.  So this is very low power.  

Of course there are many other activities that will consume power too (reading and writing to the SD card, TCP/IP communication), but I think it is generally agreed that performing the signature checks is the most work and this process can be made fairly low power.  

Quote
and an Internet connection. And possibly a wireless connection, if you are unable to get it near an Ethernet cable.

This is actually quite straight forward, as there are both chips that implement TCP/IP and processors with integrated ethernet modules.

Quote
And of course a custom-designed board, which has development and production costs.

Yes, you would need this.  

Quote
On the software side, it would require at the very least a TCP/IP stack and network drivers, and probably much more in order to be able to port bitcoind to it.

The TCP/IP stack can come as part of an IC, or as a code library for the processor.  Attaching a microcontroller to the internet is routine nowadays.  However, implementing all the networking features of bitcoin would still certainly be a lot of work.  

Quote
What I'm saying it's not this is not feasible, but it could get much more complex and costly than you think.

I'm starting to feel confident that it is feasible too.  But how do you know how complex or costly I think the R&D process would be?  I assume it would be a great deal of work.  But I am interested in the question "how simple, cheap and low-power can a useful bitcoin node by made?"  I think this is an important question.

Quote
And this calls for the main question: why? What would be the purpose of developing, building, selling, buying and running such a device, which can only give a very small benefit to the Bitcoin network and can't perform any useful function at all for his owner?

From my experience, once a project like this is complete and working, people find all sorts of applications.  

For example, a lower-power, small and inexpensive bitcoin node could be integrated in certain point-of-sales hardware, or it could come with bitcoin miners and run P2P pool by default (helping to decentralize mining efforts).  

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 02, 2014, 02:23:02 PM
 #27

bitcoind requires at least 1GB of RAM and usually more.

Thanks Cypherdoc.  I am trying to make sense of this: is this because bitcoind is loading the unspent outputs into RAM for faster checking?  My intuition is still telling me that for a custom implementation I just need enough RAM to comfortably pool the unconfirmed transactions.

The memory usage is primarily storing the UXTO in memory.  The memory pool is also kept in memory but it is much smaller than the UXTO. You could probably reduce the memory footprint with some "intelligent" caching system scoring each output based on the likelihood it will be in the next transaction (i.e. the unspent output from the coinbase of block2 is probably not going to be used in the next tx so dropping it from the in memory cache won't be an issue).

Remember reads from disk are very slow (compared to memory) and you can't validate a transaction until you lookup the outputs.  If none of the unspent outputs are in memory well you have now made 100% of tx validations requiring a round trip to the disk.  Right now throughput of the network is slow enough that you won't fall behind however for bootstraping a new node it is going to be painfully slow.
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 02:29:20 PM
 #28

Well most of that memory usage is the db cache.  It probably can be done on a custom ARM based board however it isn't going to be a trivial project and you probably want the board to have a gig of memory.  The UXTO is only going to get larger over time.

Thanks DeathAndTaxes.  If there is a "gotcha," I think it is this.  Earlier I proposed storing the UXTO on the SD card rather than RAM, organized in some efficient manner for fast cross-checking.  The UXTO would be updated each block, and earlier in the thread I showed that erase/write cycles every 10 min should still provide the memory card with several years of life.  

Am I missing something?  Is there a reason I need to store the UXTO in RAM that I cannot see?  It seems to me that even 64 MB or RAM would be sufficient (this is available on a 6x8mm ball-grid array chip for $1.50).  64 MB gives lots of room to pool the unconfirmed transactions, accept new blocks from peers, etc.  


Like you said, this certainly isn't a trivial project.  For the time being, it is a thought experiment: "how simple, cheap, small, and low-power can a useful bitcoin node be made?"

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 02:50:45 PM
 #29

The memory usage is primarily storing the UXTO in memory.  The memory pool is also kept in memory but it is much smaller than the UXTO. You could probably reduce the memory footprint with some "intelligent" caching system scoring each output based on the likelihood it will be in the next transaction (i.e. the unspent output from the coinbase of block2 is probably not going to be used in the next tx so dropping it from the in memory cache won't be an issue).

Remember reads from disk are very slow (compared to memory) and you can't validate a transaction until you lookup the outputs.  If none of the unspent outputs are in memory well you have now made 100% of tx validations requiring a round trip to the disk.  Right now throughput of the network is slow enough that you won't fall behind however for bootstraping a new node it is going to be painfully slow.

LOL you began answering my next question before I posted it. 

This all makes sense to me then, and explains why Cypherdoc's bitcoind node uses so much RAM: it is mostly to permit fast look-ups from the UXTO.  If the UXTO can be organized and stored efficiently on a SD card, perhaps with the help of some intelligent caching system like you said, this might work.


(I also think people are underestimating how fast SD cards (especially nowadays). SD cards do not force a file-system like FAT, they are basically just an SPI-interface to a huge array of flash.  So I can read small blocks out of flash with an SPI command to address the memory I want and then clock in even short blocks of data at tens of megabytes per second.  There is perhaps less overhead here than it may at first appear.)

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
Massimo80
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
May 02, 2014, 03:02:45 PM
 #30

One of the big appeals of ARM-core processors are their low power consumptions.

I'm not concerned about power usage, but about the necessity to supply power to the device, which requires dedicated hardware. Power supplies nowadays are indeed quite small, but one of them would still probably be bigger than the device itself.

Quote
From my experience, once a project like this is complete and working, people find all sorts of applications.  

For example, a lower-power, small and inexpensive bitcoin node could be integrated in certain point-of-sales hardware, or it could come with bitcoin miners and run P2P pool by default (helping to decentralize mining efforts).  

This hypotetical device and a POS have completely different (and actually opposite) design goals: a POS must act as a wallet but doesn't require full node capabilities, while this project wants to create a full node without wallet functions.

Mining is a completely moot point. Given the current difficulty, even a million ARM processors wouldn't even be noticed by the network. They would only waste power.

DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 02, 2014, 03:03:52 PM
 #31

If the flash (and controller) is sufficiently fast it may be possible to use that as the UXTO lookup storage.  This would drop the memory requirements to near negligible amounts (enough to hold memory pool, current block, and if used as a wallet the user's personal pubkeys).  A lot would come down to how long lookup of a random output from the UXTO would take.  A good place to start would be getting a development board.  Instead of working with a ARM CPU directly you could work with a microcontroller which is based on an ARM CPU.

I am a big fan of STI Micro.  This is their STM32F4 series. 
http://www.st.com/web/en/catalog/mmc/FM141/SC1169/SS1577/LN1806

The 429/439 is the most powerful microcontroller they offer.  It gives you 256KB SRAM, 2MB of flash, ethernet & usb connectivity, plus a hot of other stuff you probably won't need.  The 439 is the 429 with hardware crypto acceleration but sadly does not support the scep256k1 curve (I called to make sure Smiley ).  The main differences between the pin packages is the larger ones support more analog and digital IO pins. 

There is a development board available for rapid prototyping
http://www.st.com/web/catalog/tools/FM116/SC959/SS1532/PF259090
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 02, 2014, 03:05:08 PM
 #32

Mining is a completely moot point. Given the current difficulty, even a million ARM processors wouldn't even be noticed by the network. They would only waste power.

True but it could be useful to have high power ASIC miners connect to the node which is connected to p2pool.
Massimo80
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
May 02, 2014, 03:10:07 PM
 #33

Mining is a completely moot point. Given the current difficulty, even a million ARM processors wouldn't even be noticed by the network. They would only waste power.

True but it could be useful to have high power ASIC miners connect to the node which is connected to p2pool.

This would require USB or serial support, and mining software. This gets increasingly complex as soon as someone starts looking for real use cases, which IMHO means the base design isn't really that useful Roll Eyes
telepatheic
Jr. Member
*
Offline Offline

Activity: 56
Merit: 0

Get off the web. Go do something in meatspace!


View Profile
May 02, 2014, 03:15:23 PM
 #34

Bitcoin transactions arrive at up to about 10 per second so checking the inputs are valid by reading flash storage is not too difficult. The problem is you want to avoid DOS attacks by having quick access to transaction hashes There are about 10 million unspent outputs, if you use a modified encoding which hashes transaction ids down to just 7 bytes and 1 byte for the output count this means you need 80MB of memory plus a few MB for the mempool and recent blocks.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 02, 2014, 03:21:50 PM
 #35

This would require USB or serial support, and mining software. This gets increasingly complex as soon as someone starts looking for real use cases, which IMHO means the base design isn't really that useful Roll Eyes

No it wouldn't.  Miners connect to the node (just like they do any node running p2pool) over TCP/IP.  The mining software is on the miner.   Instead of pointing it at a pool wallet you point it at your p2pool node, exactly as you do now if p2pool is running on a desktop.

As for not useful well that depends.  Just as a hardware which runs a full node for the network and does nothing for the user well your right that is beyond stupid.  Why would one buy one?  However how about this.   Your wallet is on your computer, you don't want to run bitcoind locally for a couple of reasons such as you have multiple computers running bitcoind on all of them it excessive, or you don't use your wallet everyday and hate the fact that when it syncs up you need to wait.

So instead you use a light wallet which connects to your local bitcoin node (running on that <1W device connected to your local network).  Got 8 wallets spread across multiple computers no problem they can all point to the same local bitcoind.  Now you could use a SPV client but maybe you don't like the security compromise and the fact that it doesn't strengthen the network.
MegaHustlr
Hero Member
*****
Offline Offline

Activity: 596
Merit: 500


Vote 4fryn :)


View Profile
May 02, 2014, 03:29:01 PM
 #36

I don't understand what u mean here won't the memory stick always be the same size and shouldn't change?

He means the size of the blockchain itself.




                  ▄  ▀▄▄   ▀▄▄ ▀▄ ▀▄
             ▀█▄▄▄▄███▀▀▀▀▀▀▀█████████ ▄
         ▀████▀▀    ▄▄▄▄▄▄▄▄▄▄▄▄▄    ▀███▄
 ▄▄▄▄▄  ▄▄▀      ██▀▀     ▄██▀▀▀   █     ██
██    ▀█  ▄█▀▀▀▄  ▀█   ▄█▀   ▄▄▄ ▀██  █▀▀▄▀▄

▀▀▀   ▐█ █▌    ▐▌  █  ▐█ ▄█▀▀   █ █  ▄▀ ▄█ ▌
      ▐▌ ▀█ ▄▀▄█  █▀  █  █      ▐▌ █▀ █    ▀▄
      █     ▄█▀  ▀▀▀▀▄█  ▀█▄▄▀ ▄▀  █▄ █  ██ ▐▌
     █   ▄███▄▄███▄▄▄▄       ▄█▄▄▄▄ █▄    ▀  █ ▄
   ▄█▀▀▀▀     █▀      █   ▄█▀▀███    ▀▀▄▄   ▄██
  ▐█         ▐▌       ▐▌▄▀     ██        ▀███ ▐█
  █           ▀▄      ▐█▀       ▀█           ▀▀
  █▄           ██     ██         ██▄
   █▀            ▀     ▀          ▀█▀



.







                 ▄████▄▄    ▄
██             ████████████▀
████▄         █████████████▀
▀████████▄▄   █████████████
▄▄█████████████████████████
██████████████████████████
  ▀██████████████████████
   █████████████████████
    ▀█████████████████▀
      ▄█████████████▀
▄▄███████████████▀
   ▀▀▀▀▀▀▀▀▀▀▀




       ▄▄▄▄▄▄
    ▄████████
    █████▀▀▀▀
   ▐████
   ▐████
████████████
████████████
   ▐████
   ▐████
   ▐████
   ▐████




  ▄██▄▄                ▄▄██▄
  ████████▄▄▄▄▄▄▄▄▄▄▄███████
  ██████████████████████████
  ██████████████████████████
▄████████████████████████████▄
██████████████████████████████▌
█████▀                  ▀█████▌
████    ███▄      ▄███    ████▌
████   ▐████      ████▌   ████
 ███    ▀██▀      ▀██▀    ███▀
  ▀██▄                  ▄██▀
    ▀▀██████████████████▀▀




             ▄██▄
     ▄      ▐████   ▄▄
   █████     ██████████
    █████████████████▀
 ▄████████████▀████▌
██████████     ▀████    
 ▀▀   █████     ██████████
      ▀████▌▄████████████▀
    ▄▄▄███████████████▌
   ██████████▀    ▐████
    ▀▀▀  ████▌     ▀▀▀
         ▀███▀
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 03:31:12 PM
 #37

If the flash (and controller) is sufficiently fast it may be possible to use that as the UXTO lookup storage.  This would drop the memory requirements to near negligible amounts (enough to hold memory pool, current block, and if used as a wallet the user's personal pubkeys).  A lot would come down to how long lookup of a random output from the UXTO would take.  A good place to start would be getting a development board.  Instead of working with a ARM CPU directly you could work with a microcontroller which is based on an ARM CPU.

Thanks for this.  Do you think it is possible to keep the UXTO stored in some way that I can find a specific output in O(log n) time, where n is the number of unspent outputs?  I am not really qualified to say because I don't even know how the UXTO are currently organized by bitcoind.  But it seems that right now I could sort all the UXTO in canonical order to permit O(log n) look-ups.  I imagine there would be some way to keep the UXTO organized such that this remains the case moving forward in time too...

Personally, this is just a thought experiment for me at the moment, as my hands are full with a much simpler hardware project.  But it is interesting to consider what could be done with existing technology, time, and money.   

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 02, 2014, 03:33:21 PM
 #38

Bitcoin transactions arrive at up to about 10 per second so checking the inputs are valid by reading flash storage is not too difficult. The problem is you want to avoid DOS attacks by having quick access to transaction hashes There are about 10 million unspent outputs, if you use a modified encoding which hashes transaction ids down to just 7 bytes and 1 byte for the output count this means you need 80MB of memory plus a few MB for the mempool and recent blocks.

That is a good point.  For a set consisting of 10 million objects 256 bits is not necessary to avoid excessive collisions.  Taking the left x bits should be sufficient as this isn't being used for cryptographic purposes (and NIST certifies SHA-2 to be used for truncated hashes in some applications anyways).  The index on the other hand requires a little more thinking.  The protocol allows more than 256 outputs per transaction and is stored as a VarInt which can potentially be up to 8 bytes long (but realistically I couldn't more than 3 is probably never going to happen).  I guess one option would be to to use x bytes for the lookup hash where x = (8 - VarIntSize).  Worse case scenario you could use a full 8 bytes for the hash and full 8 bytes for the index but that would double the memory requirements.
fairglu
Legendary
*
Offline Offline

Activity: 1092
Merit: 1015


View Profile WWW
May 02, 2014, 03:38:41 PM
 #39

SD Flash will die pretty quick if you continuously write to it, like for a blockchain database.

So ok for backup, but not really for a node.

Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 03:43:59 PM
 #40

I'm not concerned about power usage, but about the necessity to supply power to the device, which requires dedicated hardware. Power supplies nowadays are indeed quite small, but one of them would still probably be bigger than the device itself.

The power supply required is intimately related to the device's power consumption.  The kind of device I'm imagining could run from a simple walwart, power-over-ethernet, or directly from a low-voltage outlet if LV wiring was available (as is becoming more common--take for example AC outlets that now include a 5V USB charging port).  


Quote
This hypotetical device and a POS have completely different (and actually opposite) design goals: a POS must act as a wallet but doesn't require full node capabilities, while this project wants to create a full node without wallet functions.

You are thinking too specific and perhaps PoS was not the best example.  But imagine a tiny low-power, low-cost bitcoin node with trivial set-up.  I think some people would find innovative uses for these, while others would buy them simply to help support the network.


Quote
Mining is a completely moot point. Given the current difficulty, even a million ARM processors wouldn't even be noticed by the network. They would only waste power.

You wouldn't mine with an ARM processor.  You would mine with a bitcoin miner that employs 1 or more SHA256 ASICs.  If the ARM-core bitcoin node was cheap and simple enough, the "hashing device" could be transformed by default into a "plug-and-play P2P miner" by the manufacturer.  This may help to decentralize mining.  

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
Peter R
Legendary
*
Offline Offline

Activity: 1148
Merit: 1005



View Profile
May 02, 2014, 03:45:54 PM
 #41

SD Flash will die pretty quick if you continuously write to it, like for a blockchain database.

So ok for backup, but not really for a node.

As long as you can limit your erase/write cycles, this analysis earlier in the thread suggests it should work:

We aren't talking about using a SD card for RAM.  The SD card would be used to store the blockchain, update the blockchain with new blocks, update the unspent outputs each block, and bootload code segments of the bitcoin binary.  Typically, flash memory is rated between 100,000 - 1,000,000 write/erase cycles (2 - 19 years if cycles happen every 10 min) and that's always writing the same segment over and over.  With intelligent disk management to avoid "hot spots," this could be greatly increased.  So I would expect even low-grade memory cards to last many years.

Run Bitcoin Unlimited (www.bitcoinunlimited.info)
telepatheic
Jr. Member
*
Offline Offline

Activity: 56
Merit: 0

Get off the web. Go do something in meatspace!


View Profile
May 02, 2014, 03:48:50 PM
 #42

Quote
The problem is you want to avoid DOS attacks by having quick access to transaction hashes.

Thinking about this some more, it doesn't really improve your DOS resistance having UTXOs in RAM as connected nodes can just send lots of badly signed (but otherwise valid) transactions instead.

Quote
SD Flash will die pretty quick if you continuously write to it, like for a blockchain database.

As long as you are just writing new data and not writing over old data, they should last well.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 02, 2014, 03:50:23 PM
 #43

SD Flash will die pretty quick if you continuously write to it, like for a blockchain database.

So ok for backup, but not really for a node.

A couple years back I donated a 2GB SD card made by SanDisk to science.  I setup a simple loop which continually wrote to the card with pseudo random sequences until it filled the card.  It would then read it back, verify it was correct, erase the card, and start over.  I got 11TB worth of writes before the card started showing failures.  I have no idea if this is typical but lets assume it is.  A 64GB SD card would be more like 700 TB of writes, which should be sufficient for a decade of blockchain usage even if the tx volume increases by a factor of ten or more.  

Of course one advantage of a dedicated device is you can strike a compromise between storage writes and memory usage.  One example would be to only write confirmed tx to the memory "disk", the memory pool would remain in memory.  Blocks are occasionally orphaned so with enough memory you could delay writing the block to storage until it is x blocks deep in the blockchain (each block deeper becomes exponentially less likely to be orphaned).  Other temporary less critical data like information on current peers, can be batch updated to storage.  If the node goes offline between updates it isn't a critical that the stored information about peers is slightly stale.  The fact that Bitcoin is a decentralized network makes it easier to implement delayed writing.  You can always request recent information from peers again.

The main scenario to avoid with flash is using it as a high throughput "RAM", but even a memory constrained device, will have some memory so there is no need to write everything to flash.
Carlton Banks
Legendary
*
Offline Offline

Activity: 2002
Merit: 1168



View Profile
May 02, 2014, 04:42:44 PM
 #44

A 64GB SD card would be more like 700 TB of writes.  That should be sufficient for a decade of usage for the blockchain even if the tx volume increases by a factor of ten.

And fast storage tech 10 years hence will not have the capacity and cell programming limitations of present-day flash technology. This sort of factor is never accounted for when arguments are made against the scalability of the blockchain. But significant near-term breakthroughs in both mechanical and solid state storage are mature enough to hit the market this year and next, and this is after both technologies have been bumping their heads against a ceiling for a couple of years now.

Vires in numeris
hxtop
Hero Member
*****
Offline Offline

Activity: 735
Merit: 500



View Profile
May 02, 2014, 06:54:46 PM
 #45

 I wish it be vastly smaller, cheaper. as a node is a good idea
Massimo80
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
May 02, 2014, 07:20:56 PM
 #46

No it wouldn't.  Miners connect to the node (just like they do any node running p2pool) over TCP/IP.  The mining software is on the miner.   Instead of pointing it at a pool wallet you point it at your p2pool node, exactly as you do now if p2pool is running on a desktop.

...and the benefit would be?

Quote
As for not useful well that depends.  Just as a hardware which runs a full node for the network and does nothing for the user well your right that is beyond stupid.  Why would one buy one?  However how about this.   Your wallet is on your computer, you don't want to run bitcoind locally for a couple of reasons such as you have multiple computers running bitcoind on all of them it excessive, or you don't use your wallet everyday and hate the fact that when it syncs up you need to wait.

So instead you use a light wallet which connects to your local bitcoin node (running on that <1W device connected to your local network).  Got 8 wallets spread across multiple computers no problem they can all point to the same local bitcoind.  Now you could use a SPV client but maybe you don't like the security compromise and the fact that it doesn't strengthen the network.

This makes very little sense. Running multiple wallets on multiple computers in the same network? I can actually think of a few use cases, but 99% of Bitcoin users would never have such a need. Assuming that they'd want to run a full node at all, which most of them just wouldn't.
Massimo80
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
May 02, 2014, 07:27:27 PM
 #47

You are thinking too specific and perhaps PoS was not the best example.  But imagine a tiny low-power, low-cost bitcoin node with trivial set-up.  I think some people would find innovative uses for these, while others would buy them simply to help support the network.

I'm quite skeptical about this.

Quote
You wouldn't mine with an ARM processor.  You would mine with a bitcoin miner that employs 1 or more SHA256 ASICs.  If the ARM-core bitcoin node was cheap and simple enough, the "hashing device" could be transformed by default into a "plug-and-play P2P miner" by the manufacturer.  This may help to decentralize mining.  

The reason mining is becoming increasingly centralized is related to equipment costs, power costs and equipment delivery times; mining is simply not profitable on a small scale, and nobody with a little common sense would buy an ASIC miner to run it at home. The availability of a cheap stand-alone Bitcoin node is not going to change this at all.
jbreher
Legendary
*
Online Online

Activity: 2016
Merit: 1051


lose: unfind ... loose: untight


View Profile
May 02, 2014, 11:23:18 PM
 #48

Here's a 64 MB SDRAM chip that is only 6mm x 8mm in size and costs about $1.50: http://www.digikey.com/product-detail/en/AS4C4M16S-6BIN/1450-1077-ND/4531924

Well, it's not really 64 MB (MegaBytes), it is 64 Mib (MebiBits). If you've not budgeted this accordingly, your DRAM cost just went up by a factor of almost 8.

Note that I'm not casting aspersions upon the appliance idea.

Anyone with a campaign ad in their signature -- for an organization with which they are not otherwise affiliated -- is automatically deducted credibility points.

I've been convicted of heresy. Convicted by a mere known extortionist. Read my Trust for details.
bountygiver
Member
**
Offline Offline

Activity: 100
Merit: 10


View Profile
May 03, 2014, 12:15:26 AM
 #49

You are thinking too specific and perhaps PoS was not the best example.  But imagine a tiny low-power, low-cost bitcoin node with trivial set-up.  I think some people would find innovative uses for these, while others would buy them simply to help support the network.

I'm quite skeptical about this.

Quote
You wouldn't mine with an ARM processor.  You would mine with a bitcoin miner that employs 1 or more SHA256 ASICs.  If the ARM-core bitcoin node was cheap and simple enough, the "hashing device" could be transformed by default into a "plug-and-play P2P miner" by the manufacturer.  This may help to decentralize mining.  

The reason mining is becoming increasingly centralized is related to equipment costs, power costs and equipment delivery times; mining is simply not profitable on a small scale, and nobody with a little common sense would buy an ASIC miner to run it at home. The availability of a cheap stand-alone Bitcoin node is not going to change this at all.

That's why we need to educate bitcoin users that mining is not about profit.

12dXW87Hhz3gUsXDDCB8rjJPsWdQzjwnm6
renee25
Member
**
Offline Offline

Activity: 76
Merit: 10


View Profile
May 03, 2014, 12:20:58 AM
 #50

a bitcoin wallet the size of a pendrive
this reminds me of the trezor.
 Roll Eyes

Don't donate me: donate to nmc re-base development project.
Carlton Banks
Legendary
*
Offline Offline

Activity: 2002
Merit: 1168



View Profile
May 03, 2014, 12:35:30 AM
 #51

a bitcoin wallet the size of a pendrive
this reminds me of the trezor.
 Roll Eyes

No, not a wallet. A node on the network. Typical wallets are combined with a network node in a single piece of software right now. This would let users run a wallet only app without waiting for blocks to download, the node is doing it for them in the background. Solo miners or p2pool miners could also make use of a low cost network node.

Vires in numeris
cypherdoc
Legendary
*
Offline Offline

Activity: 1764
Merit: 1000



View Profile
May 03, 2014, 12:46:04 AM
 #52

i'm not so sure size of a node is so important at this point.

at $19 per YEAR for a vps full node, it's cheap to run one right now.  that's where my Top image came from above:  

https://bitcointalk.org/index.php?topic=582817.0

of course there's some centralization but i don't see that as a problem.  it's decentralized enough.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 03, 2014, 01:24:34 AM
 #53

It isn't the centralization it is the fact that someone else has control over it.  I would use a SPV client before I accepted data from a bitcoin node under someone elses control.
Massimo80
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
May 03, 2014, 01:40:01 AM
 #54

It isn't the centralization it is the fact that someone else has control over it.  I would use a SPV client before I accepted data from a bitcoin node under someone elses control.

You are always accepting data from nodes under someone else's control. That's what "peer-to-peer network" means.
Where do you think your node will get its data from? Roll Eyes
ninjarobot
Hero Member
*****
Offline Offline

Activity: 761
Merit: 500


Mine Silent, Mine Deep


View Profile
May 03, 2014, 01:53:25 AM
 #55

It's been said before, but bears repeating: internet routers make the ideal candidate for a low cost bitcoin node device. The trouble is that both RAM and storage of a typical 2014 device are not in anywhere close to the right kind of league to do the job.

This is what I would like to see as well. Modern routers nowadays have dual core 1 GHz+ ARM processors and up to 256MB of RAM so they are getting close. It is not too much of a stretch to imagine an open source router firmware like OpenWRT to support bitcoind, bitcore, cgminer, p2pool and other bitcoin goodies. A router might even be a fairly secure place to run a intranet 'web' wallet on. (no malware, viruses or keyloggers)

If someone decides to build a Bitcoin Router and crowdfund it on KickStarter I'd be the fist to back it.
Carlton Banks
Legendary
*
Offline Offline

Activity: 2002
Merit: 1168



View Profile
May 03, 2014, 02:26:27 AM
 #56

It's been said before, but bears repeating: internet routers make the ideal candidate for a low cost bitcoin node device. The trouble is that both RAM and storage of a typical 2014 device are not in anywhere close to the right kind of league to do the job.

This is what I would like to see as well. Modern routers nowadays have dual core 1 GHz+ ARM processors and up to 256MB of RAM so they are getting close.

I'm surprised they're that well spec'ed already. That is probably nearly enough RAM to run a node with no wallet code, I recall that the RPi struggles a little with 512MB while using the wallet code in the 0.8 version of Bitcoin Qt (while running the Debian desktop at the same time, of course).


It is not too much of a stretch to imagine an open source router firmware like OpenWRT to support bitcoind, bitcore, cgminer, p2pool and other bitcoin goodies. A router might even be a fairly secure place to run a intranet 'web' wallet on. (no malware, viruses or keyloggers)

If someone decides to build a Bitcoin Router and crowdfund it on KickStarter I'd be the fist to back it.

p2pool and bitcoind (and internet routing) would be a stretch too far on 2014 hardware, that kind of setup really benefits from multicore processing at faster clockspeeds, as well as the fastest storage tech you can give it. Running bitcoind and cgminer would be well within reach though, and maybe there might be enough resources left over to allow those two applications to run in a solo mining configuration (low overhead solo mining is built into cgminer as of recently). USB stick miners would be happy with that product.

Maybe any new developments in p2pool software and falling prices for fast flash storage might meet each other in the middle within 5 years. The expectation is that stacked flash cell tech should be available in the next few years. A high performance, low cost version of that technology could really help make a device like that viable (the blockchain could easily be 50-100 GB by the end of the decade).

Vires in numeris
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 03, 2014, 03:52:53 AM
 #57

It isn't the centralization it is the fact that someone else has control over it.  I would use a SPV client before I accepted data from a bitcoin node under someone elses control.

You are always accepting data from nodes under someone else's control. That's what "peer-to-peer network" means.
Where do you think your node will get its data from? Roll Eyes

The key difference is that my node (under my control) independently verifies all the data it receives.  It implicitly distrusts anything it gets from other nodes and only acts upon data after it verifies it.   Peers which send it false data are tagged, disconnected, and ignored.  Part of the assumption in the security model for a full node is that the node is actually validating all data and that implicitly requires the node to be running on a system you control.  If your node is running on someone elses system you don't even have assurance they haven't swapped it out for a backdoored version which copies your private keys, logs yours passphrases, or simply feeds you what they want you to see.

If you aren't going to run a full node under your direct control you are better off running a SPV wallet.  Running a full node on hardware you don't control if security theater.
cypherdoc
Legendary
*
Offline Offline

Activity: 1764
Merit: 1000



View Profile
May 03, 2014, 04:57:13 AM
 #58

It isn't the centralization it is the fact that someone else has control over it.  I would use a SPV client before I accepted data from a bitcoin node under someone elses control.

You are always accepting data from nodes under someone else's control. That's what "peer-to-peer network" means.
Where do you think your node will get its data from? Roll Eyes

The key difference is that my node (under my control) independently verifies all the data it receives.  It implicitly distrusts anything it gets from other nodes and only acts upon data after it verifies it.   Peers which send it false data are tagged, disconnected, and ignored.  Part of the assumption in the security model for a full node is that the node is actually validating all data and that implicitly requires the node to be running on a system you control.  If your node is running on someone elses system you don't even have assurance they haven't swapped it out for a backdoored version which copies your private keys, logs yours passphrases, or simply feeds you what they want you to see.

If you aren't going to run a full node under your direct control you are better off running a SPV wallet.  Running a full node on hardware you don't control if security theater.

I don't think that's realistic.  my bet is that a majority of the nodes in the network are running off vps.

First of all, it's highly unlikely a vps provider is going to take the time to tamper with your full node if all you're doing with it is acting as a full relay node and not transacting on it.  Secondly, what would be the point besides just fooling me as to it's valid functioning?

I check my nodes connections constantly via rpc and vnstat and they've regularly got 40-50 connections. Any backdoored version whose purpose is to relay false tx's would immediately have its connections and malicious tx's terminated as invalid by the network.  Sure, i suppose the vps provider could instead feed me false connection data but again what would be the point?   i also routinely ssh in and check statistics and update/upgrade.  


DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 03, 2014, 04:30:08 PM
 #59

It isn't the centralization it is the fact that someone else has control over it.  I would use a SPV client before I accepted data from a bitcoin node under someone elses control.

You are always accepting data from nodes under someone else's control. That's what "peer-to-peer network" means.
Where do you think your node will get its data from? Roll Eyes

The key difference is that my node (under my control) independently verifies all the data it receives.  It implicitly distrusts anything it gets from other nodes and only acts upon data after it verifies it.   Peers which send it false data are tagged, disconnected, and ignored.  Part of the assumption in the security model for a full node is that the node is actually validating all data and that implicitly requires the node to be running on a system you control.  If your node is running on someone elses system you don't even have assurance they haven't swapped it out for a backdoored version which copies your private keys, logs yours passphrases, or simply feeds you what they want you to see.

If you aren't going to run a full node under your direct control you are better off running a SPV wallet.  Running a full node on hardware you don't control if security theater.

I don't think that's realistic.  my bet is that a majority of the nodes in the network are running off vps.

First of all, it's highly unlikely a vps provider is going to take the time to tamper with your full node if all you're doing with it is acting as a full relay node and not transacting on it.

I doubt very few people are running full nodes just to relay tx.  You claim that most full nodes are running on VPS also misses the mark.  The easiest way to run a full node is download bitcoin software and run it on your local machine so I would guess that most full nodes are running on local machines for the purpose of transacting.  If you don't want to run a full node on hardware you control then use a SPV client.  

Quote
I check my nodes connections constantly via rpc and vnstat and they've regularly got 40-50 connections. Any backdoored version whose purpose is to relay false tx's would immediately have its connections and malicious tx's terminated as invalid by the network.

Your missing the point.  If this node you are checking by RPC is running on a VPS you have no idea if it is really connected to anything.  Any "stats" you are getting by RPC can simply be whatever the VPS operator wants you to think it is.  You have absolutely no security.  Worse if you are transacting using that RPC connection you actually have LESS security than you would running a local SPV client.  The VPS could inform you of anything it wants.  Provide you completely bogus transaction notifications.  If you are running a service this way (imagine an exchange) it would be downright trivial to steal from you.

Quote
i also routinely ssh in and check statistics and update/upgrade.

Which means absolutely nothing.  Ok you SSH in to a virtual environment controlled completely by a third party.  Guess what said third party can show you anything they want.  Yup you are "installing" bitcoind.  Here are some stats, oh and here is a transaction paying you.  You have no assurance that any of that is real.  If you want proof I will setup a VPS for you which on the hour will show you that you receive 1,000,000 BTC.

Running a node on a VPS is security theater.  If you haven't been compromised it is simply because you aren't worth enough to be compromised.  You aren't secure you are just hoping nobody every decides they want to compromise you.   Information security begins with physical security.  One would think after all the linode hacks this would be self evident.  If you don't control the hardware you have no assurance that anything you see is "real".
cypherdoc
Legendary
*
Offline Offline

Activity: 1764
Merit: 1000



View Profile
May 03, 2014, 05:15:42 PM
 #60

It isn't the centralization it is the fact that someone else has control over it.  I would use a SPV client before I accepted data from a bitcoin node under someone elses control.

You are always accepting data from nodes under someone else's control. That's what "peer-to-peer network" means.
Where do you think your node will get its data from? Roll Eyes

The key difference is that my node (under my control) independently verifies all the data it receives.  It implicitly distrusts anything it gets from other nodes and only acts upon data after it verifies it.   Peers which send it false data are tagged, disconnected, and ignored.  Part of the assumption in the security model for a full node is that the node is actually validating all data and that implicitly requires the node to be running on a system you control.  If your node is running on someone elses system you don't even have assurance they haven't swapped it out for a backdoored version which copies your private keys, logs yours passphrases, or simply feeds you what they want you to see.

If you aren't going to run a full node under your direct control you are better off running a SPV wallet.  Running a full node on hardware you don't control if security theater.

I don't think that's realistic.  my bet is that a majority of the nodes in the network are running off vps.

First of all, it's highly unlikely a vps provider is going to take the time to tamper with your full node if all you're doing with it is acting as a full relay node and not transacting on it.

I doubt very few people are running full nodes just to relay tx.  You claim that most full nodes are running on VPS also misses the mark.  The easiest way to run a full node is download bitcoin software and run it on your local machine so I would guess that most full nodes are running on local machines for the purpose of transacting.  If you don't want to run a full node on hardware you control then use a SPV client.  

Quote
I check my nodes connections constantly via rpc and vnstat and they've regularly got 40-50 connections. Any backdoored version whose purpose is to relay false tx's would immediately have its connections and malicious tx's terminated as invalid by the network.

Your missing the point.  If this node you are checking by RPC is running on a VPS you have no idea if it is really connected to anything.  Any "stats" you are getting by RPC can simply be whatever the VPS operator wants you to think it is.  You have absolutely no security.  Worse if you are transacting using that RPC connection you actually have LESS security than you would running a local SPV client.  The VPS could inform you of anything it wants.  Provide you completely bogus transaction notifications.  If you are running a service this way (imagine an exchange) it would be downright trivial to steal from you.

Quote
i also routinely ssh in and check statistics and update/upgrade.

Which means absolutely nothing.  Ok you SSH in to a virtual environment controlled completely by a third party.  Guess what said third party can show you anything they want.  Yup you are "installing" bitcoind.  Here are some stats, oh and here is a transaction paying you.  You have no assurance that any of that is real.  If you want proof I will setup a VPS for you which on the hour will show you that you receive 1,000,000 BTC.

Running a node on a VPS is security theater.  If you haven't been compromised it is simply because you aren't worth enough to be compromised.  You aren't secure you are just hoping nobody every decides they want to compromise you.   Information security begins with physical security.  One would think after all the linode hacks this would be self evident.  If you don't control the hardware you have no assurance that anything you see is "real".

of course a vps operator "could" do these things but you've not answered the most important question of all.

what economic motive would drive them to compromise a paying customer's node that is not performing tx's and stores no btc?
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 03, 2014, 09:30:42 PM
 #61

Almost nobody is running nodes just to relay tx.  If they are running a node it is to process transactions.  If you control the node you can subvert the information given to the transaction processor to steal money.

As for a VPS operator compromising a bitcoin node, this isn't academic.  It has happened in the linode hacks (and countless hacks since then).  They did it to STEAL MONEY!  That is a pretty good motivation.  It doesn't have to be the VPS owner, it can be a disgruntled employee, or someone who uses social engineering to gain super admin access to the VPS.
cypherdoc
Legendary
*
Offline Offline

Activity: 1764
Merit: 1000



View Profile
May 03, 2014, 09:40:07 PM
 #62

Almost nobody is running nodes just to relay tx.  If they are running a node it is to process transactions.

As for a VPS operator compromising a bitcoin node, this isn't academic.  It has happened in the linode hacks (and countless hacks since then).  They did it to STEAL MONEY!  That is a pretty good motivation.

that wasn't an answer.  

there are lots of folks running vps full nodes simply to help the network.  go thru the threads linked below.  we're not stupid.  
none of us are holding btc or processing tx's on those vps providers b/c we know what happened with Linode.  i run 4 myself.  Morblias runs 17.  coalacanth just setup 2.  and on and on.  it's a cheap simple way to help.  and there is just no economic incentive for the vps provider to hack the node just to show us inaccurate data.  

there's nothing to steal.

https://bitcointalk.org/index.php?topic=582817.0

http://www.reddit.com/r/Bitcoin/comments/24645i/psa_the_amount_of_full_bitcoin_nodes_is_dropping/

http://www.reddit.com/r/Bitcoin/comments/1se3zd/how_to_create_a_full_bitcoin_node_in_a_5_ubuntu/
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218
Merit: 1003


Gerald Davis


View Profile
May 04, 2014, 12:15:30 AM
 #63

You think that is the majority use case of full nodes?  I never said nobody on the planet is running non-transactional full nodes.  I said it isn't the most common use case.   Everything I said was about the risk to transactions.

So as this point you either lack basic reading comprehension or are just trolling for trolling sake.   Either way I am done. 
cypherdoc
Legendary
*
Offline Offline

Activity: 1764
Merit: 1000



View Profile
May 04, 2014, 08:18:17 AM
 #64

stop being an asshole.

this whole discourse btwn us began with my comment here not even directed at you:

i'm not so sure size of a node is so important at this point.

at $19 per YEAR for a vps full node, it's cheap to run one right now.  that's where my Top image came from above:  

https://bitcointalk.org/index.php?topic=582817.0

of course there's some centralization but i don't see that as a problem.  it's decentralized enough.

it's clear to everyone else here i am talking about a no tx processing , no btc full node.  these nodes are setup voluntarily by ppl like me solely to relay and verify tx's for the network.  and it's a well-known and accepted way to help the network by many, many ppl  as indicated in those 3 links i provided of how to set these nodes up.

then, you engage me by saying this:

It isn't the centralization it is the fact that someone else has control over it.  I would use a SPV client before I accepted data from a bitcoin node under someone elses control.

i don't see any distinction on your part whether we're talking about a no tx, no btc full node or one that does both those things.  all you talk about is that it's bad b/c you don't have control.  even Massimo80 has no idea what you're talking about:

It isn't the centralization it is the fact that someone else has control over it.  I would use a SPV client before I accepted data from a bitcoin node under someone elses control.

You are always accepting data from nodes under someone else's control. That's what "peer-to-peer network" means.
Where do you think your node will get its data from? Roll Eyes

i happen to agree with him.  and then you go off talking about the insecurity of full tx nodes holding btc when that type of full node wasn't even what i was talking about in the first place. Roll Eyes

then this comment made me speak up b/c you're just scaring ppl who want to help the network by setting up a vps no tx, no btc full node:

If you aren't going to run a full node under your direct control you are better off running a SPV wallet.  Running a full node on hardware you don't control if security theater.

again, where's your distinction btwn a no btc, no tx full node and one that does both those things.?  oh, there is none. Roll Eyes  all you're saying is that a full node out of your control is bad.  and that is just wrong.

it is well known the # full nodes is decreasing.  by setting up these vps controlled no tx, no btc full nodes, we are helping the network.   that's b/c the vps operator has no reason to hack that acct b/c there is nothing to steal.  And in that sense it is secure. your mind is stuck on the fact that a hack is technically possible. but like so many things in Bitcoin, it's not just the technical facts that matter in isolation.  it's the economic incentives combined with the technical facts that make it secure.

Even if the vps provider hacked the full node, the worst that can happen is that it will be ignored by the rest of the network. Eventually I would find out by cross checking reality with the network, close my account, stop paying fees, accuse them publicly of being scammers, and they'd lose business.

Pages: 1 2 3 4 [All]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!