The exchange is like others. As far as I understand (I am trying to get to know about it exactly)
- Load balancing : to distribute traffic
- Distributed server : to distribute each program such as mysql and bitcoind to servers to prevent the whole exchange from attacks.
For example, normally the whole exchange is installed in one server. In the distributed server, mysql is installed in one server(ec2) and bitcoind is installed in another server(ec2). When there is an attack the target is only one server (where mysql or bitcoind) the hacker can't take all information but only take data on the targeted server. Am I misunderstanding the concept?
I guess you could call it that, but I have never heard the term "distributed server" in the context of websites. I have only heard it used for distributed computing which is completely different. I suppose you could do that, but there is no "standard" way to set up an exchange like that. Ideally the servers would be behind a firewall and be completely internal so they would not be able to be accessed from outside. That prevents attacks against those internal servers where stuff actually matters. And also if my understanding is wrong what is the difference between load balancing and ddos protection by Cloudflare?
Many thanks for your advice!
Those are two completely different things. Load balancing is to spread the load across multiple servers, ddos protection is a reverse proxy. Instead of your server taking the load of a ddos attack, the reverse proxy server does and it has the necessary setup to deal with such an attack. Reverse proxies also have the benefit of hiding the real ip address of a server. |
|
|
|
|
You will have to install all of the programs required to run the program (e.g. mysql, apache, etc.). Git is simply version control, it doesn't know what you need, it just gets files for you.
Just use git clone to clone the repo and then run it as you normally would as if you had written everything just then and there on that server.
|
|
|
|
What is the difference btw load balancing and distributed server? I think load balancing is related to network traffic and distributed server is about separation of each program that runs an exchange.
In AWS, they provide Elastic Load Balance which is one of productions.
What is distributed server? What and how many programs does an exchange need to run? Exchanges are like other large websites, they only need a few programs: webserver software and database software. There aren't tons of different software that run websites. They only need multiple servers for load balancing so that their servers don't go down when there is too much load. |
|
|
|
True, but it will still be relaying any new blocks and transactions that it receives. Pruning in 0.12 will have block relay enabled.
Can't I disable that as well? (If I really wanted to reduce bandwidth usage) I don't think you can explicitly disable block relaying but you can set the maxuploadtarget to be something really low so that you aren't uploading a lot of data. |
|
|
|
We all know that every address generated on the core wallet or on any other 3rd party wallets like Electrum, Mycellium etc. has a single extended public key (known as xPub) which acts like a master key holding all the addresses like a chain.
Not true. Only some wallets support deterministic keys, and Bitcoin Core is not one of them. Bitcoin Core does not use deterministic keys like Electrum, Mycelium, and Multibit HD do. Not every wallet uses HD (hierarchical deterministic) keys. Wallets like Bitcoin Core still randomly generate all of their keys. In fact we can find our own xPub from the wallet. But how exactly can I find that xPub key for any other arbitrary wallets? I know of walletexplorer.com as it can find the master keys of known companies and exchanges like Coinbase, LocalBitcoins.com, Crypto-Games.net etc. but not an individual's key who is actively using core wallet. Any kind of help is much appreciated  walletexplorer.com does not know the master keys of companies. It determines wallets by looking for spend links, meaning that multiple addresses are involved in the inputs of a transaction. Since that transaction is able to successfully spend those inputs, then that indicates that the creator(s) of that transaction have access to the necessary private keys. This method is messed up by CoinJoin transactions. This means that walletexplorer.com does not know the master public key if one for those wallets even exist. They are simply using spend linking. In fact, it is impossible to know the master key of a wallet unless someone tells you what it is. That is the nature of HD wallets. |
|
|
|
There aren't many nodes or people that are using IPv6. It is likely that there are simply not enough IPv6 nodes that can connect to you.
I am not sure if that is the main reason. Because when I did a quick check with "cat debug.log | grep 'peeraddr=\[' | wc -l" just now, I got 2256 list of peers with IPv6. There are of course duplication of IPv6 addresses. But I expect that my nodes should be able to connect to more than one IPv6 and there should be more IPv6 peers that can connect to my nodes instead of only maximum 3 of them. Just for completeness, with the following command: cat debug.log | grep -oP '(?<=peeraddr=\[).+(\])' | wc -l
I got 2268 IPv6 peers. Out out that, there are actually only 11 unique IPv6 peers which I obtained using the following command: cat debug.log | grep -oP '(?<=peeraddr=\[).*(?=\])' | sort | uniq | wc -l
On the other node, I got 23 unique IPv6 addresses. As I mentioned above, I think my nodes should be able to get more IPv6 peers connected to them than only 3 peers, and they should be able to connect to more than one IPv6 peers. Out of those few nodes, how many are still up or at those addresses? You are checking the debug.log, which only tells you nodes you have connected to in the past, that doesn't mean that all of those nodes are still online. Out of those 34 nodes, it is possible that many of them have gone offline, or that some are the same node just with a different ip address. Of course, your nodes wouldn't know that so they are counted separately. Basically, the probability of getting ipv6 nodes is low compared to ipv4 nodes since there are significantly less of them that exist. According to bitnodes, out of 5703 nodes, only 826 are ipv6 nodes. That isn't a lot compared to the ipv4 nodes. |
|
|
|
Thanks for the link, but "never" is the wrong answer...
There is no plan for 1.0 and there is no definition of what will be 1.0 and given that there is no plan for it, it is very likely that 1.0 will never happen or it will be a long time. |
|
|
|
What about bandwidth? Some people have bandwidth limits and still downloading 60GB of data requires bandwidth. The bandwidth requirement doesn't change, just the storage requirement. It isn't just semantics as there are other things that are still affected by requiring a full download. The pruning doesn't affect the time it takes to download, index, and verify the data. It also doesn't affect the CPU requirements, still need a decently fast CPU to crunch all that data. But in terms of storage, yes it is a big improvement.
The downside is that you can't fully sync someone else and if everyone were to prune, then we would lose a large portion of the blockchain which of course would be bad.
I am less worried about the bandwidth, which will also be significantly reduced... Once the blocks are trimmed, they cant be sent to another user/node... which will reduce bandwidth usage... Sure, you still need to download 60GB worth of blocks, but you wont be uploading those 60GB to anyone... True, but it will still be relaying any new blocks and transactions that it receives. Pruning in 0.12 will have block relay enabled. |
|
|
|
Guess I messed that section on the release notes. I went straight ahead to the pruning part to see if block relaying was enabled or not and it redirects to the release notes of v0.11.0, so I assumed that everything related to pruning was still pretty much the same as it is on 0.11. My bad Regarding BIP 130, all clients 0.10+ support this? That's what I gather from reading it... No, only 0.12+ because the sendHeaders message does not exist prior to 0.12. |
|
|
|
|
There aren't many nodes or people that are using IPv6. It is likely that there are simply not enough IPv6 nodes that can connect to you.
|
|
|
|
|
I think there will be an rc3 that will come out this week. Then the final version will be next week.
|
|
|
|
Semantics...
You download the entire 60GB, then it gets trimmed to 2GB... the 2GB footprint is what I'm referring to (assuming his figure was accurate)...
2GB of disk space costs me $0.20/month as opposed to the 60GB+, which would cost $6+/month
What about bandwidth? Some people have bandwidth limits and still downloading 60GB of data requires bandwidth. The bandwidth requirement doesn't change, just the storage requirement. It isn't just semantics as there are other things that are still affected by requiring a full download. The pruning doesn't affect the time it takes to download, index, and verify the data. It also doesn't affect the CPU requirements, still need a decently fast CPU to crunch all that data. But in terms of storage, yes it is a big improvement. The downside is that you can't fully sync someone else and if everyone were to prune, then we would lose a large portion of the blockchain which of course would be bad. I'm curious if this trimming is done in real-time, or after the entire 60GB is downloaded... i.e., could I get away with a trimmed node on a 5-10GB drive, or would it crash after downloading the first 100k blocks?
It is done in real time. It stores until it reaches that threshold and then starts deleting the old stuff so that you maintain the same amount of space taken up on the disk. |
|
|
|
Of course it is not a strict benevolent dictator type of management, but still those changes are seldom communicated well with miners before they are implemented
Soft fork can cause unprepared hard fork, the July 04 fork is an example. It indicated that miners don't fully understand the consequence of BIP66, that is because they did not took part in the decision making process of BIP 66, thus they blindly follow the recommendation from core devs, and as a result, lost lots of coins
What do you mean they did not take part in the decision making? Of course they did, that was how the fork happened, when most of the miners supported BIP66 by producing the properly versioned blocks. It is just that they were SPV mining so they didn't check the validity of the invalid block they mined on which is what caused the fork. What does that have to do with then not participating the decision making? Besides, Wang Chun, the operator of F2pool, actively participates in development. He posts on the mailing list, sometimes on IRC, and sometimes on Github. He is definitely a part of the decision making process. I'm sure that several other large pool operators do the same, I just don't know their names. |
|
|
|
This is actually possible to run Bitcoin Core on a RPi 2 and to store the blockchain on a NAS ? This could (or maybe not, I d'ont know) be a good idea.
I used to run the blockchain off of my NAS and I never had any issues doing so. However, I did that with a Windows machine running Bitcoin-Qt, so I'm not sure if that would differ from running the node on an RPi (I suspect there would be no difference though - just create a folder, and then mount the network location to it). One thing to note if running the blockchain from a NAS is that the blockchain can only be used by one device at a time. So, if you're wanting to run a node on your PC, as well as your RPi (for whatever reason), you won't be able to. Thank you for your explanation. I don't think that I'll ever need to run it on both my PC and my RPi 2 at the same time, at least with the same folders. Would it be possible to run it on both my PC and my RPi 2, but with two separated blockchain files on the same NAS ? I think that would work as well. IIRC, Bitcoin-qt "locks" the blockchain when it starts up, so that another application won't be able to use it. I suspect having two copies of the blockchain in different locations would bypass that lock restriction. The only downside is that you'd be storing two copies of the blockchain, so you'd be using 120GB instead of 60GB. OK. This was a question. The PCs that I'll be using will have their own hard-drive (that's quite obvious), so I'll surely don't need to put two blockchain copies on a NAS. However, would it be possible to store it on a PC and on a NAS, as a kind of backup. If the blockchain get's corrupted (I don't know if is possible), then the PC will look at the NAS, and thus the full node will never be down. No. The blockchain only becomes corrupted when the node is not running, it shouldn't corrupt while the software is still running. If you ever need to start up Bitcoin Core, it will inform you whether it needs to reindex or not. If it does, then you can simply restart it with the datadir pointed at the NAS copy while you fix the local copy. |
|
|
|
|
Briefly, yes. Transactions confirm when they are included in a block. The difficulty of finding a block of dependant on the hash power, which is partially dependent on the number of miners. The target is to find a block every ten minutes on average. The difficulty to force that target time changes every 2016 blocks which should be roughly 2 weeks. If a number of miners dropped off, then from the time to the next difficulty retarget, the confirmation times will be longer due to a high difficulty and a lower hash rate since blocks are being found less quickly. However after the difficulty retarget confirmation times will go back to normal.
|
|
|
|
Bitcoin has been in "beta testing" for nearly 8 years now... when is it going to officially be version 1.0 instead of this 0.12, etc?
Does anyone know what the holdup is? Transaction Malleability?
Never. This has been discussed before, Google it. |
|
|
|
|
This proposal is similar to Bitcoin Unlimited, but slightly better IMO because the block sizes are announced by the node. However, I don't think that having the block size limit being controlled and determined by the node operator is a good idea. It is consensus critical and could result in nodes being left behind and lots of potential forks. It is bad for miners as it could increase their orphan rate much more than previously.
Another question is the default value. Most node operators are going to be lazy and not change the default value.
Can you provide more specific technical details on how the block size limit of a node is announced and how that announcement will be able to reach miners so that they know what block size they should be producing?
Also, please post this to the mailing list and open a PR to the BIPs repository if you want it to be seriously considered by anyone.
|
|
|
|
Based on a 22 hour catch up, that would be 1MB * 6 * 22 = <132MB average block data.
Running on SSD write enabled cache, catch up took over 2.5 mins, with writes in 25 - 55 MB/s range. Maybe 40 MB/s on average. So as a rough guestimate we are talking over 6GB of disk writes to catch up on potentially 132MB of data. A lot if users with mechanical disks do not want to run a full node now, and it is the catch up time rather than network bandwidth and storage space which is the biggest issue for some.
I do not know why that would be the case but try asking on the github or searching through their issues and pull requests sections to find answers. Also, it is known the the bottleneck with syncing a full node now is usually on the CPU or the hard drive. Does bitcoin core use a full database implementation, or just implement the indexing technology?
Not sure, but I think the indexing is only putting into the database the location on the disk of each block. However, as part of the reindex and a full sync, the blocks need to be reverified, which taxes the CPU. Another issue is during a computer crash, bitcoin core frequently requires a full database reindex. This is bad now, and will become an absolute unmanageable nightmare as the blockchain grows.
So does Bitcoin really need to use a fully fledged database system? If not, a specialised data structure could be the way forward. Period local syncing of last known good blocks would mean in the event of failure, a full reindex would not be required; it would simply redownload blocks from the last sync point on recovery. This would also keep small disk writes to a minimum, using periodic linear disk writes instead.
I think the issue is that it cannot know where the last good block was located because the database with the indices is corrupted so it can't be read. It then has to start from the beginning and reindex all of the blocks so that it can know the location of the blocks and be able to pull data from them when needed. If you were to simply start at the last known good block, then it wouldn't be able to access data earlier in the blockchain. Also, reindexing does not mean that it is redownloading. Reindexing means that Bitcoin Core is reading from the block files and recording in a database the location of those blocks. I think there needs to be improvements made here, and depending on the requirements there could be the potential for significant performance and resilience improvements.
probably, but again, ask it on the github as the developers are active there and also know the ins and outs of Bitcoin Core much better than I do. |
|
|
|
|
Show Posts
! Can you explain here why (this will help all the lazy people to know) ?
