Tough problems during running a Full node

[Trojane]

I have had a few difficulties when i was trying to run a FULL NODE using bitcoin core but i also needed lightweight/SPV CLIENT (simplified payment verification client) since i needed a prior VERIFICATION and VALIDATION of all the company's transaction and also to know if there were included in the BLOCK.
I needed to BROADCAST an VERIFY every transaction easily.
I had to initialize SYNCHRONIZATION; So i decided to download the entire  block chain and this was IMPOSSIBLE!!
I was looking for a secured wallet system like  this    
 https://www.researchgate.net/publication/224229795_The_concept_of_Secure_Mobile_Wallet

I also tried to enhance myself on this link too
https://bitcoin.org/en/full-node#other-linux-gui

The question is..
* is there any smart way to download the whole block chain ?
How can i go about it?
Please share your experience 🙏
Regards
Trojane 💎

[n0nce]

So i decided to download the entire  block chain and this was IMPOSSIBLE!!

Why exactly was it impossible? What was the issue? I mean it takes some time, but it's the most secure option for sure.

* is there any smart way to download the whole block chain ?

Yup, using Bitcoin Core. It's quite simple. Download from the link that you see in the forum header (https://bitcoincore.org/en/download), verify the download as explained on that page and then follow https://bitcoin.org/en/full-node for more info about running a full node.

Feel free to read through everything before starting off so you have all the info you need.

[DaveF]

Unless you have very limited bandwidth, less then 4GB RAM or less then a 500GB free on your HDD there is no reason you cannot download the entire blockchain.
Depending on the amount of RAM in your system and speed of your internet connection it can take from a day or so to couple of weeks.

As n0nce asked. What is the issue you are having?

-Dave

[darkv0rt3x]

I bet the problem is that a week or a week and a half to finish the IBD is too much time and the OP is a bit impatient...

But it is what it is. The correct way is to go through that process and bare with the time it takes. If all goes good, you'll only have to do it once!

Just put it downloading an forget it for a few days! Come back later for fine tuning and configuration!

[Trojane]

I was trying an instance of a full node which can be used to verify,make perfect recording and also keep block chain records
Thus,to achieve this ,i guess am to download the whole block chain but its impossible...!
Is there any other way to do that?
What could be the actual problem
That's just the complexity i had...
I haven't gotten over it

[DannyHamilton]

I was trying an instance of a full node which can be used to verify, make perfect recording and also keep block chain records

This doesn't sound like a problem.  This just sounds like a description of what a full node does.

Thus,to achieve this ,i guess am to download the whole block chain

If you want to verify the full blockchain, and you want a perfect recording of the full blockchain, and you want to keep full blockchain records, then yes, you need to download the full blockchain. Again, this doesn't sound like a problem, this just sounds like a description of what you are trying to accomplish.

but its impossible...!

Impossible?  Nah.  Lots of people have done it.  It's less than 375 GB.  All you need is a good broadband internet connection.

Is there any other way to do that?

Another way to obtain the full blockchain without obtaining the full blockchain?  I don't understand the question, that doesn't make any sense.

What could be the actual problem

I don't know. You haven't actually described a problem yet.

That's just the complexity i had...
I haven't gotten over it

You haven't described any complexity yet.  You told us what you wanted to do, then you told us that you didn't want to do what you wanted to do, then you said that it's a problem that you don't want to do what you want to do.

[n0nce]

i guess am to download the whole block chain but its impossible...!

It is possible. I already told you how, and most of us here do run a full Bitcoin Core node with a full blockchain download.

It's quite simple. Download from the link that you see in the forum header (https://bitcoincore.org/en/download), verify the download as explained on that page and then follow https://bitcoin.org/en/full-node for more info about running a full node.

What could be the actual problem

We have no idea what you tried, where it failed or why it failed.  How are we supposed to help you?
What did you try so far? Did you download Bitcoin Core as I mentioned from https://bitcoincore.org/en/download? If not, why? That's the only way I know of to run a full Bitcoin node with full blockchain download. If you did, what happened afterwards? Did it start downloading? If yes, did it stop somewhere?.....

[BlackHatCoiner]

Assuming your problem is that you don't have an answer to this question:

* is there any smart way to download the whole block chain ?

There's no smarter way to receive anything in the internet other than downloading it. There's also not a smarter way to run a full node. You're “doomed” to connect with the peers of the network, receive blocks, verify their validity and repeat until they have no other blocks to give you. Once that happens, you'll have the most recent version of the block chain. As long as there are valid blocks waiting to be verified from your computer, you'll have to let it finish.

With my Raspberry Pi 4 it has taken around a month to finish. It should take less to you if you're having a better CPU than my Pi's. (Which you'll most likely have)

[n0nce]

There's no smarter way to receive anything in the internet other than downloading it.

I mean I guess they could find someone with a torrent of a snapshot or even a zip, download it to the blocks/ folder and then start core; it should find them and verify the whole directory, then get new blocks from its peers at the end.
Not sure it would be any faster or more smart in any way though, after all the verification (computing and disk access) is what needs most time.

[BlackHatCoiner]

I mean I guess they could find someone with a torrent of a snapshot or even a zip, download it to the blocks/ folder and then start core;

But, that wouldn't make the process finish sooner. Once you would have downloaded all the blocks it'd require you to verify them which is the main reason it takes so much time.

You can change the way your node will synchronize with the network, but not the time it'll take. There's no faster way other than running bitcoind from your cmd.

[Trojane]

All thanks to @nonce
I had to read through the link and i got some useful documentaries.
I have gone above my difficulties

[Trojane]

You haven't described any complexity yet.  You told us what you wanted to do, then you told us that you didn't want to do what you wanted to do, then you said that it's a problem that you don't want to do what you want to do.

You know what?
Anything that has to do with TECHNOLOGY and CONNECTIONS might go faulty anytime,thats not because it wasn't working previously
I made this topic according to the the FUNCTION that i needed to perform with the NODE and the output i got during several times of trials ...
Please we are all in the process of learning,nobody knows everything, even in the different fields that we specialise.
I felt very bad for your own opinion. I don't think i should be mocked .,not at all!

I just needed few clips from articles i could read for more details on how i could achieve what i wanted,not being quoted IGNORANT because i am already.
It looks simple to you because you have been doing it OVERTIME
 

[n0nce]

[...] and the output i got during several times of trials ... [...]

I don't think you actually did post any 'output'. You never really posted what actually happened or what you tried, just said that it was 'impossible to download' the blockchain.

Is your issue solved now?

All thanks to @nonce
I had to read through the link and i got some useful documentaries.
I have gone above my difficulties

What do you mean? Why don't you just ask a question similar to the StackOverflow recommended format, which I quite like (https://stackoverflow.com/help/how-to-ask)?
Then we could actually help and not have to guess what your issue is and what you mean by 'ITS IMPOSSIBLE', 'ITS IMPOSSIBLE' etc...

[DaveF]

With my Raspberry Pi 4 it has taken around a month to finish. It should take less to you if you're having a better CPU than my Pi's. (Which you'll most likely have)

That is a REALLY long time. This config: https://bitcointalk.org/index.php?topic=5158920.msg58008085#msg58008085 did it in under a week.
Was this an older 2GB unit? Slow internet? something seems off. Unless you are doing something else with the Pi at the same time.
Or if you don't have good cooling it might throttle a lot. A bad power supply might also cause some issues, but a month still seems long.

-Dave

[DannyHamilton]

You know what?
Anything that has to do with TECHNOLOGY and CONNECTIONS might go faulty anytime,

Yes, it might. But you never described anything faulty.

thats not because it wasn't working previously

Maybe it was, maybe it wasn't.  If you took the time to actually describe what your problem was, rather than complaining about mysterious "tough problems" you probably would have gotten better help a lot faster.

I made this topic according to the the FUNCTION that i needed to perform with the NODE and the output i got during several times of trials ...

You never posted any output. You never described any output. You never explained what happened during any trials.

Please we are all in the process of learning,

And there are many people here willing to help and to teach.  But first, you need to actually ask a question or describe a problem.

nobody knows everything, even in the different fields that we specialise.

I never said anybody knew everything.  I just asked you to actually explain what your problem is.

I felt very bad for your own opinion. I don't think i should be mocked .,not at all!

I'm not mocking you. I'm just asking you to explain what your problem is so that we can help you (and help others that may have a similar problem in the future and may see this discussion.

I just needed few clips from articles i could read for more details on how i could achieve what i wanted,

Glad to hear you figured it out. It would still be good to explain what the problem was and how you resolved it so others can learn from your experience.

not being quoted IGNORANT because i am already.

I never said anyone was ignorant.  Where did you get that nonsense from?

It looks simple to you because you have been doing it OVERTIME
 

Where did I say anything was simple?  I'm just trying to understand what your problem is.

[BlackHatCoiner]

I just needed few clips from articles i could read for more details on how i could achieve what i wanted

But, how can we know those few clips if we don't firstly understand what's your issue? You said you're having some tough problems when you try to run a full node; please elaborate!

Was this an older 2GB unit? Slow internet? something seems off.

Well, no. I'm not having an older 2GB unit neither a slow internet connection. Does it matter if I've set txindex=1? I'm running both main net and test net, but that shouldn't be the issue. I really don't know what went that wrong. The synchronization started in 30st of August and ended yesterday. (Almost a month)

[DaveF]

Was this an older 2GB unit? Slow internet? something seems off.

Well, no. I'm not having an older 2GB unit neither a slow internet connection. Does it matter if I've set txindex=1? I'm running both main net and test net, but that shouldn't be the issue. I really don't know what went that wrong. The synchronization started in 30st of August and ended yesterday. (Almost a month)

The txindex=1 might slow it down a tiny bit like add two or three days onto since you are indexing as you go but not weeks of time.

Running both mainnet and testnet really should not matter much, once again add a day or two onto the sync and once the testnet is fully done it should really make no resource difference.

How is your cooling? Thermal throttle can be a big issue with RPis at times.

What type of USB drive are you using? A slower 5400RPM with limited cache might cause some slowdowns as it's indexing and downloading and verifying 2 blcokchains.

Are you downloading over clearnet or TOR? I have had some painfully slow downloads over TOR for the initial sync at times.

Could also be a combination of things. None of which add a lot of time. But 3 days here and 3 days there and another few days due to something else....

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.
Does not help you figure out why it took so long this time, but if you have to do it again it's one less thing to worry about.

-Dave

[n0nce]

What type of USB drive are you using? A slower 5400RPM with limited cache might cause some slowdowns as it's indexing and downloading and verifying 2 blcokchains.

I had a quite bad experience with 5400 rpm normal USB HDD (RPi 3B+ though). It took a month and then something got corrupted due to a bad powered USB hub..

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.

I don't think it helps much, if it's a drive / verification issue. In my case, I tried downloading the blockchain with PC and then connecting it to the Pi as well, but the verification was what took so long, so it didn't help much. I finally solved my issues by switching to a more powerful machine with more RAM. However, this issue should be solved by using an SSD; people used to run Bitcoin nodes on RPi 3's all the time when that was the latest offering, but they did use solid state.

[BlackHatCoiner]

How is your cooling? Thermal throttle can be a big issue with RPis at times.

The case I have covers the cooling part.

What type of USB drive are you using?

A 1TB external one with USB 3.0. Specifically, this one. (Yes, I plugged it on the 3.0 port)

Are you downloading over clearnet or TOR?

I'm downloading over Tor, but I don't think that's the issue. I believe that what's taking so long is the verification of those blocks.

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.

Well, a backup would require another 1TB disk. Is it that necessary? Have you ever experienced a hard drive issue from a power surge?

[n0nce]

How is your cooling? Thermal throttle can be a big issue with RPis at times.

The case I have covers the cooling part.

Tricky - it might not be enough cooling. Did you check your temps and frequencies at all? Like, if the frequency doesn't stay pegged at its max rating, when under load, it might be throttling.

What type of USB drive are you using?

A 1TB external one with USB 3.0. Specifically, this one. (Yes, I plugged it on the 3.0 port)

Are you downloading over clearnet or TOR?

I'm downloading over Tor, but I don't think that's the issue. I believe that what's taking so long is the verification of those blocks.

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.

Well, a backup would require another 1TB disk. Is it that necessary? Have you ever experienced a hard drive issue from a power surge?

I must say: HDD is not the best choice, because as far as I know for every block, Core has to look up whether the inputs are valid UTXOs, so it needs to 'look back' in the blockchain, requiring random disk reads which are the slowest of disk read operations.^{(I might be wrong about the way the verification works)}

However, I was able to sync quite quickly with a HDD in an old laptop. It only has 4GB of RAM, so maybe less than your Pi, but the computing power is higher and the cooling is good. So in hindsight, I'm not able to tell you 100% if your computing power is limiting you due to cooling constraints or if it's a disk bottleneck. I'd bet on the former though, since my HDD worked fine (around a week) in a laptop node.

Have you ever experienced a hard drive issue from a power surge?

I did! And it needed to re-verify the whoooole damn thing. I don't keep a backup of the blockchain though, since verification takes much longer than the actual downloading part, no matter the setup, so I don't see the need.