I have decided to publish the source code to the explorers on Github. I do this in the spirit of open source and to encourage others to make suggestions and improve upon them.
Excellent, I was just about to request the same! When I have time (if no one else does it first) I'd like to organize these source trees as clones or branches of jtobey/bitcoin-abe (or a new Abe repo under a shared account). I think that will facilitate merging the code and relegating the differences to config options. Thanks for all your work! |
|
|
|
If there was a way of limiting the work being done network wide if you have an outdated miner/client. Forcing updates to bring hashes back up.
The older version after it expires it works like crippleware. I'm just giving a new perspective on solutions. I'm by no way anyone who understands this as an expert. If I'm understanding this correctly the transition to 0.8 was to slow right?
Well, yes and no. True, if 90% had upgraded before the fork, the decision would probably have gone in their favour. But the real problem was an unexpected incompatibility between versions. There are (rarely) planned forks, and this was an unplanned fork. Slush's pool didn't create the incompatible block because they believed a majority ran 0.8, they did it because they believed a large majority ran a version that would accept it. If that's the case then by crippling the older version and forcing the timely update would prevent older versions from competing with new versions and incentive's people on older versions to update quickly. Preventing a fork.
I could be wrong.
This sort of crippling would be impossible given the circumstances (no central control, open source). The best that you can do is urge users to upgrade... or downgrade, as the case may be. ("rightgrade"?) |
|
|
|
That was my point actually. I was not accusing the developers of negligence I was saying that his platitudes are a form of "I could have done it better then them if I was them and could see the future". Well, he isn't them and they didn't see the future.
Then we are in violent agreement! Hooray! If the network ditches 0.7 then there is no problem.
As long as the network is predominantly 0.7 this would result in you creating a single orphaned block rather then a fork. While certainly a waste, this could happen while using 0.7 to mine as well (for a different reason).
All true, but I suspect you underestimate the, uh, conservatism that makes "the network ditching 0.7" non-trivial. That may end up being the chosen way out, and Deepbit, Eligius, etc., will plan to upgrade at an appointed time. Or another solution may emerge. We've waited months for 0.8, and we can afford to wait a few more days or weeks, though the newly discovered 0.7 bug adds a reason to move. |
|
|
|
@taltamir Let's slow down and think this through, shall we? You claim: Whether you:
a. force 0.8 to reject valid blocks which it predicts 0.7 cannot process - currently cannot be done
b. force 0.8 to not generate blocks which it predicts 0.7 cannot process - currently can be done
The result is the same, you avoid a fork.
Unfortunately, this statement glosses over a crucial detail. In scenario A, as I understand, you mean "force most 0.8 miners to reject valid blocks which it predicts 0.7 cannot process". And you are correct, currently I know of no way to do this. Whereas, in B, you must force all nodes to refrain from generating blocks like the one that caused this incident. Otherwise, in a mixed environment of 0.7, 0.8 unmodified, 0.8 with option B
someone could start a fork the same way as just happened. Do you see? It's the all part that makes us reject your proposal B in the short term. Neither A nor B is workable as of now. Block generators are advised to use 0.7 or earlier. As long as most do, we (probably) avoid this kind of a fork. I don't know why the forum banner still claims that mining on 0.8 is OK. Miners using 0.8 risk wasting hashes by building on a block that the majority (0.7-compatible network) will reject. Perhaps someone who has followed on IRC can comment? The programmers who made 0.8 didn't consider this forking scenario. This platitude is thus entirely without merit
Pipe down, I am sure the developers thought long and hard about ways that 0.8 might cause a fork. One slipped through, because accidents happen. I remember reading not every 0.7 node rejected the block, and the behavior was not even consistent within a same node; some nodes would accept the block after being restarted after initially rejecting it.
That would mean that 0.7 nodes are capable of producing blocks that other nodes of the exact same version may or may not accept, depending on factors which are hard to predict.
That would mean continuing to use 0.7 is an even bigger issue. I agree with you, it is a big issue, and I feel confident that the core developers and stakeholders will find a reasonable solution pretty soon if they haven't already. |
|
|
|
What if the .7 chain never catches the .8 chain?
Correct me if I'm wrong but if many miners abandon the .8 chain won't the difficulty adjust down, so that even though there are much few miners on the .8 chain they will still be able to stay ahead of the .7 chain due to the decreased difficulty?
This could not have happened: "Length" is calculated as total combined difficulty of that chain, not number of blocks...
|
|
|
|
I am not receiving email at John.Tobey@gmail.com since three days ago. If you sent email there in the last 72 hours, it did not reach me, and GMail is not sending delivery failure messages. I've filled out the forms and await a solution in the next couple of days, or I'll be an ex-GMail user. I'll check my personal messages here. Sorry for any inconvenience. |
|
|
|
dreamwatcher, thanks for the free service and for keeping my donation address on the pages! I am curious to see your changes to Abe, it would be nice to integrate them upstream... eventually in my copious spare time.  Also, would you mind sharing the database tweak intended to stop the deadlocks, and does it still appear to work? If I were doing it all over, I would decide on a transaction isolation level at the outset and code to that. I think READ COMMITTED ought to be sufficient for this app, though it would have been worth the effort to retry deadlocks in order to support stricter isolation. Maybe still worth the effort. Given my time constraints, I really don't adequately test Abe but do appreciate bug reports. |
|
|
|
This branch can now TRIM SPENT OUTPUTS, transactions, and addresses from the database. It is not polished code, but it basically seems to work. Loading the blockchain may be a little slower than before, since it does all the same SQL work, plus some deleting. The space reduction appears to be "only" around 50% as of Block 134000 (under PostgreSQL 8.4). This may be due in part to all those early unspent coinbase outputs. To use the new feature, pass --default-trim-depth=6 ( EDIT: make that --default-trim-depth= 40 thanks to the March 12 fork) when creating the database. Replace "6" with a higher number for defence against corruption by long side chains, at the cost of slightly more space. See "default-trim-depth" in abe.conf for details and caveats. |
|
|
|
I've added a branch to the code repository: no-statistics. no-statistics is an experimental branch of Abe that lacks certain features present in the master branch. The features are Percent Coin-Days Destroyed and Chain Age. Removing these features simplifies the code and reduces Abe's storage and processing requirements. I have only lightly tested it, but I consider it the direction of future development. I would like to reimplement the original statistics and others, but their inclusion should be a configurable option so as not to burden applications that do not need them. There is not (yet) an option to convert a database in the original format to one that uses the no-statistics branch or vice versa. |
|
|
|
<sarcasm>
Remove the limit completely, so ppl could spam more useless transactions. Let's make HDD producers to produce disks with 100500 PETABYTES capacity!
</sarcasm>
Removing the limit does not imply useless transactions. It simply means the market will decide what is a useless transaction. So long as clients prefer smaller blocks over larger ones, the miners will decide among themselves what a suitable block size is. If it's too high, small-block miners will earn more and blocks will get smaller. If it's too low, large-block miners will earn more and blocks will get larger. Isn't this what Bitcoin's about? All we need to do is make sure the system cannot be exploited, test it on testnet, and then not need to worry about block sizes ever again. Appendix:
1. What changes are needed?
A: The "longest chain" code needs to increase the length of shorter blocks and decrease the length of longer ones. Clients also need to set a bandwidth quota so that they download and propagate the best currently-downloaded chain.
2. Couldn't a miner just spam the network with a huge block?
A: No. Like transactions, blocks need "confirmations". If other miners think the block is too big, they will refuse to confirm it and a smaller block will replace it (as they orphan large blocks).
3. Wouldn't blocks be too big?
A: No. Miners will take an unacceptable risk if they mine large blocks, as they are likely to get orphaned. Higher demand and higher transaction fees will make taking that risk more attractive, so block size can scale with usage.
4. Wouldn't miners just mine empty blocks?
A: Eventually, no. This is a concern in the short term, but as soon as transaction fees overtake subsidies empty blocks will no longer be attractive.
5. Wouldn't this allow DOS attacks?
A: Yes. However, DOS attack prevention code can be written to give up downloading a particularly large block if a competing chain is running at not far below network hashrate.
6. Isn't this a hard fork?
A: Done properly, not any more than raising the block limit. The length code only acts as a tiebreaker, which we already have in "first block wins".
I have to say Dree12 is making the most logical sense here. I agree. If larger blocks start appearing on the network and become part of the longest chain ( i.e., what would be the longest chain if the limit were removed), I will take it as a sign that the major pools have raised or abolished the limit and that bitcoin exchanges, merchants, etc., will follow. I care more about their idea of what a bitcoin is than about what the devs or the average user thinks, or even my own idea of what would be best. I doubt that the major stakeholders will disagree with each other strongly enough to risk a near-even split. Therefore, I would like the ability to configure my Bitcoin client to ignore the limit. (My client does not generate blocks.) The prospect of my client following the wrong side of a fork because of an oversized block is all risk, no benefit to me. (If the "wrong" side persists due to user sentiment, it will effectively become an alt chain, though some will argue that it is still the true bitcoin.) |
|
|
|
Oh nice!
In readme.txt: "The Abe.reconfigure module turns firstbits on and off once you have upgraded Abe's schema."
How do I updrage Abe's schema?
You already have a schema supporting firstbits, so just ignore that step. Sorry it's unclear, but upgrading is for databases created by earlier versions of Abe, before firstbits support. Anyway, to upgrade, just use the new version of bitcoin-abe and include "--upgrade" after "python -m Abe.abe". |
|
|
|
Am I correct that firstbit is not yet available trough api-calls?
No, there is /q/fb/ADDRESS and /q/addr/FIRSTBITS. However, firstbits is disabled by default, since it requires a big table. (Something like 99% of all addresses are single-use "change addresses" but must be remembered for firstbits.) Also, please note that Abe's algorithm probably differs slightly from firstbits.com and blockchain.info's. See elsewhere in this thread for details. More info, including how to turn on Abe's firstbits support, is in README-FIRSTBITS.txt. |
|
|
|
I was expecting 50*210000 + 25*(height-209999) as well, that was why I was wondering if it added fees or something.
chain/Bitcoin/q/totalbc/100000 : 5000049.98
chain/Bitcoin/q/totalbc/200000 : 10000037.06486171
chain/Bitcoin/q/totalbc/220998 : 10774961.89978955
The numbers are actually below what the formula gives. After block 100000, there had been 100001 blocks (counting block 0), so the expected total was 5000050, 0.02 more than actual. After block 200000, expected 10000050, 12.93513829 more than found. After 220998, expected 10774975, 13.10021045 more than found. It looks okay to me, although we could take off another 100 for at least two cases of duplicate 50BTC coinbase transactions, if I recall correctly. |
|
|
|
A loop that runs it with --no-serve would keep it up to date. It brings it up to date and then exits. And what does /q/totalbc show? Bitcoins created, or bitcoins included in coinbase transaction? Because Local Abe: 10774286.89978955 Blockexplorer: 10736300.00000000 Blockchain.info: 10761275.00000000 Not that it really matters, I don't use it, but just wondering  It gives a statistical estimate of amount in circulation. I am curious which block number you got the Abe value from? You can add the number to the URL like /q/totalbc/220962 to be sure. The number is normally 50*210000 + 25*(height-209999) but is lower due to the following rare events: - coinbase transactions outputting less than they are entitled to,
- coins sent to known dead addresses (Abe recognizes the hash consisting of all 0 bits),
- duplicate coinbase transactions (I don't think Abe accounts for them, but the others may)
and there may be bugs. |
|
|
|
|
I've reimplemented /unspent so it should outperform /address. A couple of things to note. The denial-of-service limit on history rows applies to the total across all addresses given to /unspent, so in some cases, /unspent will return "too many records" when iterated /address calls would work. This limit applies to both spent and unspent coins, unlike in the previous implementation, so you may see "too many records" where my first /unspent worked. In abe.conf, address-history-rows-max controls this limit. For now, I've left the first implementation available by adding "?impl=1" to the URL. It should yield the same results apart from "too many records" errors.
Let me know if this helps.
|
|
|
|
@Jouke, Thanks for the generous tip and the report of slow performance. It should be simple to change /unspent to match the speed of repeated /address calls. Please feel free to remind me if I haven't got to it in a week. Yes, 1PWC7PNHL1SgvZaN7xEtygenKjWobWsCuf is still under my control. |
|
|
|
TypeError: Decimal('4294967295') is not JSON serializable
<ip> - - [26/Jan/2013 20:15:37] "GET /rawtx/38b46ec5330c445d0e6451d305a08e0fd91a660510ba5aa3ed3c9b1f43356905 HTTP/1.1" 500 59Fixed, thanks. And I am willing to put a bounty of 3btc for this: This is done, but with caveats: I do not have a full database to test with, it may require optimization, the output fields are slightly different, and it won't work with your example address because Abe does not yet parse script-to-hash transactions. /unspent/ADDRESS1|ADDRESS2|... http://john-edwin-tobey.org:2752/unspent/1CBXL5tBSqJZyyCTWVkrLiQsKm4KULBfPA|1CfoGajDKynwB1JpaWq2tML4DDsP8VnuPy |
|
|
|
The probability of a block being solved after 10 minutes: 50%, or 1/2
I might be doing something wrong, but it seems my data shows that 65% of block have been solved in less than 10 minutes. The difference seems a bit too large to be explained by our "faster mining", no? No, the probability after 10 minutes at constant hashrate would approach, IIRC, 1 - exp(-1), not 50%. The average time would be 10 minutes, but not the median as 50% would imply. The long tail at the high end pulls the average above the median. 1-exp(-1) is c. 63%, a little less than your number, as one would expect given the history of increasing network speed. |
|
|
|
Hi folks, hello developers (i.e. John Tobey),
Hi! I am playing with the thought of implementing this on my server, since I am already running a full 24/7 BTC-node, LAMP and still have some HDD-space left. Also, since the main fallibility of P2P-Networks is the fact that the truth is determined in a democratic process, you can't have enough block explorers with different blockchain-sources out there.
Nevertheless, I have a problem with understanding the license this was published under: Let's say I want to change the look of the site by rewriting a bit of the CSS-sheet and adding a different logo, I basically have to publish the new sources, even for small derivations, right? But I am allowed to do so, even commercially (aka. linking the logo to my site, which is adversing and therefore commercial...)
Right. One easy way to respect the license is to get an account on GitHub (or another public Git service like Gitorious) and maintain your version there. They give you plenty of handy features. If it is just a logo and CSS changes (branding as opposed to functionality) I don't care. If this becomes a sticking point, let me know and I will try to fix it by adding exceptions to the license. Second question: Is anyone running this on MySQL? How big is the database when finished, and is there a somewhat linear relationship to the size of the blockchain (one might assume so...)? Had it loading for a few hours now and got to block 143000, it's around 2.9 GB so far.
I think it is roughly 4-5 times the sum of the block file sizes. You can check which block file it is on and how far along it is by reading the datadir table as it loads. blkfile_number=1 means it has not finished reading blk0001.dat. blkfile_offset is how far into the current file it has processed. If blkfile_number is greater than 1, add the sizes of the lower blkNNNN.dat files on disk. mysql> select * from datadir;
+------------+------------------------+----------------+----------------+----------+
| datadir_id | dirname | blkfile_number | blkfile_offset | chain_id |
+------------+------------------------+----------------+----------------+----------+
| 1 | /home/bitcoin/.bitcoin | 1 | 699231568 | NULL |
+------------+------------------------+----------------+----------------+----------+
Thanks for the code and all the hard work!
You're welcome and good luck! |
|
|
|
|
Show Posts
