Print Page - Monero no wallet update since 2014, FluffyPony deleting question about it

Quote from: dukey8 on July 31, 2015, 11:16:31 AM

Some github commits and pdf and you want people to believe thats more work than Bytecoin?

Don't worry, I understand it's difficult to grasp. You're outshone, outwitted, and outsmarted, and despite throwing money at the problem you're still disliked by a community that is too smart to fall for your scam. It's a tough world when your audience sees through even a relatively smart scam.

In order to help you understand what I mean by "more work" I've decided to put the contents of the PDFs and the GitHub commits up for you. Let's start with the PDFs.

MRL-0001: A Note on Chain Reactions in Traceability in CryptoNote 2.0

Abstract

This research bulletin describes a plausible attack on a ring-signature based anonymity system. We use as motivation the cryptocurrency protocol CryptoNote 2.0 ostensibly published by Nicolas van Saberhagen in 2012. It has been previously demonstrated that the untraceability obscuring a one-time key pair can be dependent upon the untraceability of all of the keys used in composing that ring signature. This allows for the possibility of chain reactions in traceability between ring signatures, causing a critical loss in untraceability across the whole network if parameters are poorly chosen and if an attacker owns a su.cient percentage of the network. The signatures are still one-time, however, and any such attack will still not necessarily violate the anonymity of users. However, such an attack could plausibly weaken the resistance CryptoNote demonstrates against blockchain analysis. This research bulletin has not undergone peer review, and reﬂects only the results of internal investigation.

1 Introduction

The CryptoNote protocol has a problem: my anonymity depends on your anonymity. If I use 5, 6, or 18 of your outputs when I compose my ring signature then you can see the true signer. If you spend all 5, 6, or 18 outputs with no foreign outputs used as mixins in your ring signatures, you reveal yourself as the spender, and now any observer can also see the true signer of my transaction. This may not be malicious if you spent your outputs non-anonymously for legitimate business or legal reasons. Hence, any party with a large proportion of the UTXO set may gain knowledge of the traceability of others’ transactions and reveal that information to the network at will. One may fancifully interpret this problem as an abstract, cryptocurrency implementation of Gresham’s Law: bad money drives out good. If the unspent transaction output (UTXO) set is ﬁlled with a lot of transactions that aren’t really anonymous, there are fewer ways to make untraceable ring signatures. At this point it must be noted that, even in this scenario, the one-time key pairs (so-called “stealth addresses”) used in CryptoNote protocols are not violated in this scenario, and so the anonymity of users is still not directly violated. Rather, this attack violates the untraceability between one-time ring signatures, but this development is still somewhat worrying. Hence, even non-malicious entities can execute this attack on accident, malicious entities can spam the network to own lots of the UTXO set, and malicious entities can break untraceability for others.

An attacker’s intent may be perhaps in the interest of a pump-n-dump scheme by undermining the credibility of a currency, perhaps in the interest of spying on other users, or perhaps in a disinterest in paying the extra cost of adding more foreign unspent transaction outputs to their ring signatures. In all cases, despite the a priori hope that any particular user should always be interested in using a large number of mixins for a ring signatures as a matter of principle, both in the interest of personal security and the common good, we should have no expectation of this in practice. A version of the tragedy of the commons may take place and cause traceability throughout the network for everyone due to a subset of users. However, we should also keep in mind the forest, not the trees: since any user can always compose a trivial ring signature with no additional outputs, immedi.ately exposing outputs down the line, there is a second-order problem here. A new transaction has been exposed, creating exposure risk for any ring signatures that used the newly exposed transaction as an obfuscating mixin. Is it possible that a chain reaction could occur, even if a malicious entity stops actively attacking early in the history of the coin? This research bulletin was written to analyze this prob.lem, determine the plausibility of a conservative route of such an attack, and assess network parameters to ensure graceful degradation of untraceability.

2 Setup for a Passive Attack

Everyone who has had a college probability class has come across the following, even if you don’t know how to solve it:

An urn contains N marbles of two possible colors. We have NB black marbles and NW white marbles, where NB + NW = N is ﬁxed. We draw M marbles from the urn. What is the probability that all of the marbles we drew from the urn are black? What is the probability that all of the marbles are white? If 0 ≤ m ≤ M, what is the probability that we have m black marbles and M . m white marbles?

When phrased this way, we immediately jump to the hypergeometric distribution. The probability mass function

NB N.NB
BM.B

N
M

can be used to compute the probability that all the marbles in my hand are black (set B = M), and we can use convenient formulae available in any number of free online textbooks, or wikipedia, or whatever, in order to compute the probability of any particular occurrence. It’s one of the nicer distributions.

We play the following game: if all the marbles I draw are black, I’ll toss a new black marble into the urn while I replace my handful. Otherwise, I’ll toss a new white marble into the urn while I replace my handful. As it is with marbles, so it is with CryptoNote coins: It may help to envision each marble as an atomic unit (the CryptoNote analogue to a Satoshi) tracked in the UTXO set of a CryptoNote network. The color denotes who is in control of the marble; black if controlled by a malicious entity, white if otherwise.

Let’s say Lisa wishes to send some money to Milhouse (which would add a marble to the bag). The UTXO set that Lisa draws from in order to make her ring signatures will contain N transaction outputs, of which NB are controlled by Burns (the black marbles; assume the rest are white). Then the above formula is exactly what we need. Lisa composed her ring signature with M ≤ N mixins[1]

(she drew a handful of marbles to decide the color of marble to add to the urn), and if all are controlled by Burns? Well, then, if Burns spends his outputs with no mixins[2], Lisa’s transaction will also be exposed. Furthermore, forget Lisa, any ring signature that used only Burns’ signatures can actually now be considered part of Burns’ controlled set, in terms of untraceability, even if Burns does not control the private keys. Notice that in this scenario, we are having a chain reaction occur: Burns reveals only his own transaction outputs, but he, in turn, reveals Lisa’s output. The four assumptions we have made are

(i) Burns controls an initial proportion of an otherwise unknown UTXO set,
(ii) all new transactions generate their ring signatures drawn from the current UTXO set in a uniform, unordered way without replacement,
(iii) all new transactions use exactly the mandatory minimum mixin level (handful size M ), and
(iv) Burns gains control of new transactions if and only if their ring signatures are solely composed with Burns-controlled transactions (i.e. Burns stops putting new transactions in the system at time t = 0).

We can change our assumptions to strengthen or weaken the attack in our scenario. For example, the attack is weaker if some transactions use more than the mandatory minimum. The attack is stronger if Burns can also gain control of transactions by generating them himself, i.e. he is not some passive actor in the background. However, this seems to be a reasonable, middle-ground scenario, if we interpret it in the following way: Burns initially controls a portion of the UTXO set before a ﬂood of ostensibly honest transactions are composed quite suddenly, so what happens? Protecting the network against a large-scale degradation in untraceability under this scenario is simply a matter of smart engineering. If we can protect the network against large-scale degradation under worse scenarios, all the better, but we shall start here. This could be considered a “Christmas Day” attack, both because our simulations are set up to mimic a sudden inﬂux of economic activity before a truly malicious attacker can respond (for ease of coding) and because the attack can be executed by an ostensibly innocent individual simply spending their money non-anonymously for perfectly legitimate reasons.

Since the hypergeometric distribution is so nice, what can we expect out of this game? Well, let us analyze the results for a single new transaction. Assuming M is the minimum number of mixins across the whole network, what is the probability that all M mixin signatures are controlled by Burns? If this probability is large, then new transactions are very likely to be “controlled” by Burns, even if he has yet

[1]We deﬁne a mixin as a foreign output included in the transaction.
[2]Which, it again must be emphasized, may be an ostensibly innocent act, not an attack!

to reveal the transaction outputs he controls. That is to say, the more CryptoNote transactions controlled by a single malicious party, the weaker the untraceability trait of new transactions becomes. Consider the following numerical example.

Suppose we have N = 103 transaction outputs in our anonymity set (not counting the real output belonging to Lisa), and that Burns controls pN of these transactions for some 0 ≤ p ≤ 1. Denote by Pr(p) the probability that any one ring signature (using three mixins) is signed solely with outputs belonging to Burns. This proba.bility describes, roughly, whether or not Lisa’s new marble will be black or white. Table 1 contains sample values of Pr(p) for this example.

p pN Pr(p)
10.3 1 0
10.2 10 7.22 . 10.7
10.1 100 9.73 . 10.4
0.5 500 0.125

Table 1: If Burns controls pN transaction outputs from an anonymity set of N = 103 outputs, we display the probability Pr(p) that Lisa’s output is traceable from Burns’ point of view, assuming that Lisa uses three mixins in the construction of her ring signature. That is to say, at the beginning of a large scale simulation, if Burns controls half of all transactions, we can expect him to “see” about 12.5% of new transactions.

How do we interpret this table? If Burns controls only a single output in the anonymity set, there is no way three distinct outputs in a ring signature can all be.long to him. If Burns controls ten outputs, there is only a probability of 0.00000722 that Lisa’s ring signature is really controlled by Burns. This is so small that on av.erage, it would take about 13.8 million signatures from this anonymity set to start seeing collisions. If Burns controls 100 outputs, there is a probability of 0.000973 that a signature made by Lisa is controlled by Burns, so we should start seeing col.lisions after around 100, 000 signatures. Finally, if Burns controls half of all outputs, then 12.5% of all new transactions will have a ring signature composed entirely of his outputs, so Burns will only be gaining 1 in every 8 new transactions, and so Burns’ share of the UTXO set will shrink over time unless he takes action.

In Table 2, we produce the same values as for Table 1 but with only two mixins. In this case, if Burns controls half of the outputs, the consequence is that every one in four new outputs will really be controlled by Burns. If Burns controls 10% of the outputs, only 1 out of every 1000 new transactions will be controlled by Burns. Clearly, not as great as 3 mixins, but still not terrible; even if Burns controls half of all transactions, there is only a 25% chance that a new ring signature will belong to Burns. Hence, his share of the UTXO set will shrink over time unless he takes action.

p pN Pr(p)
10.3 1 0
10.2 10 9.01 . 10.5
10.1 100 9.91 . 10.3
0.5 500 0.25

Table 2: If Burns controls pN transaction outputs from an anonymity set of N = 103 outputs, we display the probability Pr(p) that the new output is revealed during an attack, assuming that Lisa uses two mixins in the construction of her ring signature. That is to say, at the beginning of a large scale simulation, if Burns controls half of all transactions, we can expect him to “grab” about 25% of new transactions.

3 Monte Carlo Methods and Results

Hold tight for some notation. We choose parameters NB0 ,NW0 and M, where NB0 denotes the initial number of black marbles owned by Burns, NW0 denotes the initial number of white marbles, and M is the minimum mixin level. We play the game iteratively. We record the initial proportion of unspent transactions owned by burns as P0 = NB0 /(NB0 + NW0 ) before beginning. On the ith iteration, we are adding the Ni =(NB0 + NW0 marble to the bag. We draw a uniform random number u from (0, 1) and compare it to the hypergeometric mass function NBi / Nias described before. If u ≤ MM , then the next marble will be black, so we iterate NBi = NBi.1 + 1. Regardless of the choice of u, we iterate Ni+1 = Ni +1 and if we need to compute NW we can simply subtract. After some iterations, say I iterations, usually enough iterations to double or quadruple the UTXO set, we stop the game and consider the proportion of black marbles, PF = NBI /NI . However, since we are only concerned with how many of the new marbles were black (not how many in total are black), we will watch as our metric of success for Burns’ attack the value PPF =(NBI . NB0 )/I; number of black marbles Burns gained during the game divided by number of marbles added to the bag.

The above process constitutes one simulation. For each choice of parameters (NB0 ,NW0 ,M), we get a result PPF , which was determined as a result of a sequence of random experiments, and is such a random variable. So we run 37 trials and take the sample mean and sample variance from these experiments. We initially looked at the 95% conﬁdence intervals, but they are so narrow that we may as well ignore them, in the end. We simulate a UTXO set growing from 5000 transactions to 20000. Hence, we plot (20000PF . 5000PI )/15000, that is, the proportion of os.tensibly honest transactions that Burns grabs. Figure 1 illustrates our simulation results for NB + NW = 5000 and M =1, 2, 3, and 4 mandatory minimum mixins in the event that the UTXO set quadruples with honest transactions (that is, 15, 000 new transactions need new ring signatures). The horizontal axis displays the ini.tial proportion of the UTXO set controlled by Burns, varying from 0.0 to 1.0, the vertical axis displays the ﬁnal proportion of the new transactions in the UTXO set controlled by Burns, varying from 0.0 to 1.0. Note that any number above zero on the y-axis represents a loss of security.

It’s clear that PPF is monotonically increasing as a function of P0 = NB/(NB + NW ), but that the situation improves for greater numbers of ring signatures. Notice that there are some dynamical systems problems built into our data. If you graph PF vs. PI , (rather than our PPF vs. PI ) then you can interpret this as a discrete-time dynamical system’s evolution function to see that Burns’ share must go to zero over time for any mixin ≥ 2. We can expect, in the 1 mixin scenario, for the urn to stochastically jump up and down the y = x line without appreciably, in expectation, moving in any direction in a random walk. However, as long as we require M ≥ 2 mixins, any chain reaction will burn itself out, so to speak; Burns’ share of the UTXO set will shrink to zero over time unless he takes action. Our results suggest that any mandatory mixin ≥ 2 will allow the system to recover from a passive attack quite quickly. The code we used to generate data can probably be improved, but is located at http://goo.gl/dGj5TZ and we used Google Documents to generate ﬁgures.

Figure 1: This ﬁgure displays, with apologies to the color blind, the “success of an attack” executed by Burns. We see that, for increasing numbers of mixins, success uniformly decreases accross the board. To refresh the reader who probably jumped directly to this graph, here’s the scenario: Evil guy Burns starts with an initial proportion (x-axis) of the UTXO set, which has 5000 transaction outputs inside of it. Then, a ﬂood of 15, 000 new honest transactions hits the network, from Lisa to Milhouse and from Homer to Marge and so on, causing the UTXO set to quadruple in size. After this event, Burns could spend his initial proportion PI of the UTXO set with no mixins and consequently break the untraceability of a proportion of the 15, 000 ostensibly honest transactions. We display this proportion PPF as a function of PI . For 10 mixins, Burns requires 55% of the UTXO set to capture any of a quadrupling event at all, and requires 87% of the UTXO set in order to capture even 1% of a quadrupling event with these parameters.

4 Critical Mass, Slow Chain Reactions, Active Attackers: Further Questions

Recall that we set out to ask the question “can a critical chain reaction occur?” and the related question “what network parameters can we choose to ensure that any plausible chain reaction burns itself out?” So, what do we need for a critical mass? Consider the urn problem again. What happens to the composition of the urn as this game evolves, as we toss new black or white marbles into the urn as we draw more? Remember, black marbles are the ones that will be revealed; if we draw all black marbles, that means the next marble we add to the urn must be black. A critical mass scenario is one in which, if there are enough black marbles in the urn, then Burns starts grabbing more and more and more black marbles. If ever PF >PI , we could expect some sort of critical mass, but this does not occur. Except in numerical edge cases, PF ≤ PI . With 2 and more mixins, our chain reactions become truncated very rapidly.

Chain reactions act slower and slower as mixins increase. Thus, a single mixin is inappropriate for any currency problem, because an anonymous, malicious user may spam the network with transactions they control in the hopes that they even.tually control greater proportions of the UTXO set. This would essentially make all transactions from then on fully traceable (at least from Burns’ point of view), and Burns doesn’t have to take a single action after his initial seed transactions are planted in the UTXO set. This ﬁxed initial cost for the attacker leading to a never-ending stream of information in the form of traceable transactions from other users is, clearly, a catastrophic economic failure. This is so important, we are putting it in italics: any CryptoNote coin that allows for only 1 mixin is vulnerable to a slow chain reaction in which the owner of very few private keys can violate the untraceability of much larger number of other users. Requiring a mixin of at least 2 for all transactions save for transactions that are willfully spent with 0 mixins will keep these chain reactions, probabilistically, to a smaller length. Indeed, any number greater than 1 will do, to force these chain reactions to burn themselves out, rather than to spread to the whole network, and the higher the better. Of course, a protocol-enforced, network-wide mandatory minimimum mixin of M = 10 would, presumably, cause a blockchain bloat, which can hinder adoption, which has it’s own security beneﬁts in terms of network size. Hence, there is likely some optimal size of mandatory minimum mixin. We do no more than to suggest M = 2 as a protocol-enforced mandatory minimum, and to advise users to use as many mixin signatures as their little hearts desire.

Note that our scenario might be di.erent if we change parameters of our model, or if we change our assumptions to include active attackers. Let’s discuss parameters real quick. We can consider the addition of each new marble as nudging Burns’ passive ownership probability one way or another. Unless Burns grabs almost all incoming transactions, his share of the UTXO set will be depleted over time. How do our parameters change this process? Recall that our parameters are:

1 NB0 = Number of initial atomic units controlled by Burns in UTXO set,
2 NW0 = the remaining atomic units in the UTXO set, and
3 M = mandatory minimum mixin enforced by network protocol. However, we are really interested in PI = NB0 /(NB0 + NW0 ) and N0 = NB0 + NW0 . We have seen above how PI and M, in general, change the outcome of the experiment. As N0 gets big, Burns’ share will start to slow down. Meaning even if Burns is grabbing all of the incoming transactions, he’s getting a reduced rate of returns in terms of overall proportion of the UTXO set that he owns. We can think of the UTXO set size as a mass. As it gets bigger, it gets harder to change its state.

Another “hidden” parameter here is “how many new transactions are being added?” We swept this under the rug by simply saying “okay, the UTXO set is going to quadruple from it’s initial size,” reducing our parameter space a little bit, and in so doing, we have reduced the importance of N0 as a parameter. Indeed, we know how N changes, so we can easily compute slope of any statistic, T (N), with respect to “change in population size” and call that the sensitivity:

T (Nf ) . T (No) T (4 · No) . T (No)
S ==
Nf . No 3No

which, when close to zero, suggests no relationship, and when far from zero suggests a strong relationship (positive or negative). We could just as easily compute the slope of the log of the statistic with respect to the log of the population, and call that the sensitivity:

log(T (Nf )) . log(T (N0)) 1 T (4 · No)
S = = log
log(Nf ) . log(No)4 T (No)

in this version of sensitivity, we don’t even see Nf as part of the formula, simply the multiplicative factor Nf /No = 4. However, we feel it is unnecessary to present a full-scale parameter-space analysis of this model.

Now let’s discuss changing assumptions. One apparently small tweak we could add to this simulation is to change assumption (iv). In order to model a more aggressive attacker rather than a “Christmas Day” accidental reveal sort of attack, we could implement a probabilistic assumption like

(v) Burns gains control of a new transaction as the result of a sequence of Bernoulli trials. With a ﬁxed probability q, Burns generates a transaction that he knows is his, and with probability 1.q, a foreign transaction appears and generates a new ring signature that may be solely composed with Burns-controlled transactions

(i.e. Burns still participates in the economy) or we could implement a dynamical model where a malicious Burns tries to maintain a minimum proportion of the UTXO set by spamming transactions to the network whenever his proportion drops his secretly determined percentage:

(v) Burns gains control of a new transaction as the result of a sequence of non-autonomous, Bernoulli-like experiments. With a probability q(B, N) dependent both on the number of UTXOs controlled by Burns, B, and the total UTXO set, N, Burns generates a transaction that he knows is his, and with probability 1 . q(B, N), a foreign transaction appears and generates a new ring signature that may be solely composed with Burns controlled transactions.

Any function q(B, N) that Burns can try to make close to 1 when B/N is smaller than Burns’ preference will allow Burns to grow his share of the UTXOset. Of course, the easiest, simplest strategy for Burns would be to just spam the network as fast as possible with his own transactions. But with fees, this isn’t free.

We bring up parameters and assumptions because the work is not really done on this problem. We have discovered what we, as network engineers, need to know in order to ensure graceful degradation: pick big mixins. However, there are really interesting questions still lurking under the surface. One of us solved this problem with graph theory, one of us with probability. Note that, as time goes on, a UTXO is more likely to be chosen for mixins, but is also more likely to have been exposed from previous users revealing their transactions by spending them with 0 mixins, complicating the wisdom of choosing UTXOs for the ring signature from a uniform distribution. Technically, for any sort of “critical mass” sort of problem, there’s going to be a steady state problem lurking under the surface, and we deﬁnitely have some dynamical systems stu. ﬂying[3] about in this problem. Maybe some ambitious undergraduate wants to pick up where we leave o. and contribute to the cryptocurrency community by expanding on all of this.

[3]See what I did there?

MRL-0002: Counterfeiting via Merkle Tree Exploits within Virtual Currencies Employing the CryptoNote Protocol

Abstract
On 4 September 2014, an unusual and novel attack was executed against the Monero cryptocurrency network. This attack partitioned the network into two distinct subsets which refused to accept the legitimacy of the other subset. This had myriad e.ects, not all of which are yet known. The attacker had a short window of time during which a sort of counterfeiting could occur, for example. This research bulletin describes deﬁciencies in the CryptoNote reference code allowing for this attack, describes the solution initially put forth by Rafal Freeman from Tigusoft.pl and subsequently by the CryptoNote team, describes the current ﬁx in the Monero code base, and elaborates upon exactly what the o.ending blcok did to the network. This research bulletin has not undergone peer review, and reﬂects only the results of internal investigation.

1 Introduction

On 4 September 2014, a novel attack was executed against the Monero cryptocur.rency network, leading to a never-before-seen phenomenon in the network. The attacker must have had extensive knowledge of the Monero code, had good knowl.edge of Merkle Trees, and basic knowledge of cryptographic hash functions.

The Monero code was forked from the Bytecoin reference code in April 2014 before the CryptoNote reference code was released. The Bytecoin code appeared to be somewhat obfuscated and poorly commented, so any given segment of code must necessarily be considered with skepticism. Hence, any attacker with an extensive knowledge of the Monero code also, presumably, has an extensive knowledge of the Bytecoin and CryptoNote reference codes.

A Merkle Tree is a data structure in which every non-leaf node in the tree gains a label, and the label is the hash of the child nodes[3]. That is to say, to build a Merkle Tree, we take some blocks of data (transactions), and we hash them. Then we hash those, two at a time. Then we hash those, two at a time. And we repeat this process until we are done. You can think of a Merkle Tree as a football bracket of cryptographic hashes. A natural question arises: what if we do not have a nice, even, power-of-two, divisible number of transactions? This is where the exploit came in, as we will see later.

CryptoNote currencies, and indeed, most cryptocurrencies, use Merkle Trees to construct the hash of a block of transactions (which is subsequently packed into the header of the block). The attacker noticed that a mis-implementation of a commonly used power-of-two rounding algorithm could be exploited while computing the hash of a block in order to generate two distinct blocks with the same hash. This should be all but impossible, of course, as collisions of cryptographic hash functions are rare. When discussing the rarity of these collisions, comparisons like “how many fundamental particles exist in the universe?” start to pop up, and usually those numbers are not big enough to describe collision rarity. In general, hash collisions typically imply mis-implementation rather than a failure of the hash functions, presuming we are using a good hash function [2].

To the authors’ best knowledge, this attack has only occurred once. Although Monero was the target, any coin utilizing CryptoNote reference code from before 4 September 2014 su.ered the mis-implementations allowing for such exploitation, with two exceptions. Fantomcoin and Moneta Verde apparently ﬁxed the relevant exploit in secret several months ago. Again, only to the authors’ best knowledge, all popular CryptoNote currencies have implemented the changes described by R. Freeman and the CryptoNote team, although we have not performed an extensive code review. The purpose of this document is to describe the section of code that allowed this attack and to elaborate on the attack’s e.ects. For a slightly di.erent take on the attack, see, for example, [1].

2 Exploitation of the CryptoNote Reference Code

Throughout this section, we abuse terminology slightly when we say CryptoNote reference code, as Monero was forked from ByteCoin before the CryptoNote refer.ence code was released; we do not fear confusion in the reader. The exploitation of the CryptoNote reference code can be understood using the football bracket anal.ogy described before: what happens when the data we wish to hash does not come in nice, even powers of two? We simply take any overﬂow above a power of two and play a smaller, ﬁrst-round bracket, where all other, pre-overﬂow data gets a bye[1] to the next round. Of course, this just kicks the can down the road to the next smallest power of two, but eventually this process will stop at two teams and we can proceed with our bracket like usual. In order to perform this strategy, we wish to ﬁrst ﬁgure out how many teams we really do have playing.

Of course, in order to do this, the size, length, or measure of the current data set needs to be rounded down to the greatest power of two strictly less than the current data size; all data above this amount (in terms of indices) needs to be folded into the data below this amount with cryptographic hash functions. This is where the exploit came in. The cuto. index splitting above versus below was being miscomputed. The attacker noticed that this would allow some data to be ignored when computing the hash of a block, and so two distinct blocks could obtain the same hash. This was not due to any sort of failure of the hashing functions used, but, as described above, a failure in a measurement of the size of the “real” data to be used in construction of the Merkle Tree.

[1]Not byte.

The following code in src/crypto/tree-hash.c was exploited:

src/crypto/tree -hash.c:

46 size _t cnt = count -1;
48 for (i = 1; i < sizeof(size _t); i <<= 1) {
49 cnt |= cnt >> i;
50 }
51 cnt &= ~(cnt >> 1);

The goal of this code, a version of a commonly used algorithm, is to round the number count to largest power of two that is strictly smaller than count, setting this value as cnt[4].

Input Output
100 64
255 128
512 256
513 512
1205 1024

To understand how this segment of code works, we’ll ﬁrst consider a toy example and then compare it to how the algorithm actually works. Suppose we want to determine the largest power of two strictly less than the integer 1205. Deﬁne x to be the binary number formed by decrementing the integer 1205 and setting every bit to the right of the most signiﬁcant bit to one:

120410 = 100101101002 . x = 111111111112

We next set y to be the binary number formed by performing a single right-shift on x and taking its negative:

y = x > 1 = 011111111112 = 100000000002

The result is the number xy, using the binary AND operation:

xy = 111111111112 · 100000000002 = 100000000002 = 102410

We now compare this toy example to the algorithm in the above-referenced block of code from src/crypto/tree-hash.c. Assume, as is true for most mod.ern architectures, that sizeof(size t) = 8; in this case, we have i .{1, 2, 4}, so the loop executes three times. Consider the operation of the code from src/crypto/tree-hash.c using the sample data count = 51310:

cnt = 513 . 1 = 51210 = 10000000002
i =1: cnt = 10000000002 + 01000000002 = 11000000002
i =2: cnt = 11000000002 + 00110000002 = 11110000002
i =4: cnt = 11110000002 + 00001111002 = 11111111002

Observe that because the loop executes only three times, we did not change the state of the two least signiﬁcant bits. The algorithm proceeds:

cnt = 11111111002 · 1000000012 = 10000000002 = 51210

We obtain the correct result. Suppose, however, that we instead use the sample data count = 51410:

cnt = 514 . 1 = 51310 = 10000000012

i =1: cnt = 10000000012 + 01000000002 = 11000000012
i =2: cnt = 11000000012 + 00110000002 = 11110000012
i =4: cnt = 11110000012 + 00001111002 = 11111111012

Similar to before, the last two bits remain unchanged. The algorithm proceeds:

cnt = 11111111012 · 10000000012 = 10000000012 = 51310

This is clearly incorrect, as we expect the algorithm to output the integer 512. This mis-implementation of the previously described rounding algorithm was the root of the exploitation of the CryptoNote reference code. Let us consider the ramiﬁcations of the mis-implementation of this rounding down by considering the a.ected code further down the line:

src/crypto/tree -hash.c:

src/crypto/tree -hash.c:
47 char (*ints)[HASH _SIZE];
52 ints = alloca(cnt * HASH _SIZE);
53 memcpy(ints, hashes, (2 * cnt -count) * HASH _SIZE);
54 for (i =2*cnt -count, j=2*cnt -count; j<cnt; i+= 2, ++j) {
55 cn_fast_ hash(hashes[i\], 64, ints[j]);
56 }
57 assert(i == count);
58 while (cnt > 2) {
59 cnt >>= 1;
60 for (i =0, j=0; j<cnt; i+= 2, ++j) {
61 cn _fast _hash(ints[i\], 64, ints[j]);
62 }
63 }
64 cn _fast_hash(ints[0], 64, root _hash);

In this code block, lines 47 through 56 perform the following tasks. We allocate an array of hashes to ints. We take what we think is supposed to be the data from the original hashes that has not yet overﬂowed our desired power-of-two structure, and we use memcpy to copy the data into ints. However, the quantity 2·cnt.count is, as previously described, an overestimate of this amount of data. For example, the known exploit described above will round down the number 514 to 513, not 512. Hence, setting cnt= 513 and count= 514, we see that we copy 2·cnt.count= 512 elements, ﬁlling all of the array that is used in subsequent hashing rounds. However, we should have computed cnt= 512, copying 2·cnt.count= 510 pieces of data. Thus, the bug previously reported leads to two pieces of data being wholly ignored. That is to say, the ﬁrst 2·cnt.count hashes should receive a bye in the sports analogy, advancing to the next round without actually playing any games (or rather, being hashed with any other data). Since cnt is being overestimated, too much data is being passed forth without being properly handled, leaving the remaining data unused. Of course, we still hash the remaining data together, but these hashes are not used.

Note the stunning degree of technicality required to ﬁnd and exploit this bug. Someone out there is very, very, very smart to not only ﬁnd, but also to exploit, this deep bug. They know algorithms extremely well, and they appear to be malicious.

3 Fixes Applied

On 4 September 2014, the same date as the attack, the ﬁrst solution to the exploit was publicly announced by Rafal Freeman from Tigusoft.pl[2]. The solution is not a unique one, as the CryptoNote reference code implements a di.erent solution[3]. The CryptoNote solution establishes a “quick ﬁx” in src/crypto/tree-hash.c, by using the ending condition 8·sizeof(size t), causing their loop to iterate through all bits in cnt. However, without adequate checks on the size of cnt, it may be possible to choose a suitably large value to conduct the same attack. Furthermore, using magic numbers is, simply, a generally frowned-upon practice in computer science and cryptographic protocols. Then again, so is code obfuscation, and it is considered good form to comment your code, both of which the CryptoNote team is ostensibly guilty.

The Monero codebase, by contrast, establishes checks on the size of cnt and uses the following rounding algorithm:

src/crypto/tree -hash.c:

47 assert ( count >= 3) ; // cases for 0,1,2 are handled elsewhere
49 size _t tmp = count -1;
50 size _t jj = 1;

51 for (jj=1 ;tmp != 0;++jj) {

52 tmp /= 2;

53 }

54 size _t cnt = 1 << (jj -2);

56 assert( cnt > 0 ); assert( cnt >= count/2 ); assert( cnt <= count
);
57 assert( ispowerof2_ size _t( cnt ));
58 return cnt ;

This algorithm determines the number of powers of two less than cnt and performs an appropriate bitshift to recover the correct power of two. This algorithm does not su.er from the same size limitations as the CryptoNote ﬁx.

[2]see Monero commit 2ef0aee81d20c002ed50d6dec4baceee1ac40b44
[3]see CryptoNote commit 6be8153a8bddf7be43aca1efb829ba719409787a

4 Effects of the Attack

Once two blocks with the same hash were published nearly-simultaneously, part of the network had one block and the rest of the network had the other block (ignoring nodes that had not yet received the block). Simply checking transaction hashes on the opposing half of the network would cause a fault. This partitioned the network into two distinct subsets which refused to accept the legitimacy of the other subset. For awhile, the network was split into two. A reader may be tempted to call this a fork in the blockchain. Be wary. Calling this a fork would be inappropriate. Indeed, a fork occurs when two competing chains of transactions in the block-tree, both of which are ostensibly legitimate, compete for network hash rate. Eventually, one of the two chains will win since the network uses the “longest chain” decision method proposed by Satoshi Nakamoto. Furthermore, although it’s extremely di.cult, it’s theoretically possible that, after a time, the network switches from one chain to another if it grows long enough.

The two parts of the network refused to accept the legitimacy of the blockchain upon which the other part was operating. It was as if, suddenly, half the Monero network switched to a brand new CryptoNote coin’s blockchain. It still appeared to, say, currency exchanges or vendors, that they were both one network still working on Monero.

This had myriad e.ects, not all of which are yet known. For one thing, balances were essentially doubled. If you had 1.0 XMR before the attack, you now had 1.0 XMR on each version of the network during the doubling. One could fancifully interpret this as a counterfeiting scenario: the attacker published two blocks with the same hash, and now instead of one network, there are two and the attacker has double the balance. The attacker had a short window of time during which they could sell “counterfeit” Monero from the new network on an exchange such as Poloniex for Bitcoin and immediately withdraw. The attacker still has his original balance on the “old” network, of course. With the major exception of MintPal, most major exchanges dealing with Monero were able to freeze those transactions during the attack.

For currently unknown reasons, most major Monero mining pools ended up to.gether on one side of the network; this could have simply been a result of demo.graphic stochasticity as the network pulled itself apart, or it could be a deterministic result of the speed at which blocks propagate. Indeed, if two blocks with the same block hash hit the network simultaneously, and if one block is heard by a large pool before the other due to stochasticity, it is almost certain that block will be heard by more of the network than the other. On the other hand, if two blocks with the same block hash hit the network nearly simultaneously but randomness is excluded, the ﬁrst block will be the dominant block.

No matter what, the much smaller side of the network, of course, had signiﬁcantly lower hashing power, and began experiencing terrifying errors. The di.culty was inherited from before the split, as well. Due to the protocol within CryptoNote to ignore outliers in block duration while computing di.culty, it would take somewhere between three days and one week for the B network to adjust di.culty so as to compensate. Any node stuck on the B side of the network saw a dramatic drop in their proﬁtability; this, as well as the error spam they have received, has caused most nodes to resync their blockchain, or to reboot their computer, or something along those lines. The B side has all but vanished at this point, but the e.ects of the Block 202612 attack still linger on the Monero network. The Monero development team released patch 0.8.8.3 on 6 September 2014 to ensure that this attack can never occur via this route again, to allow miners to identify the A side of the network (that is, the A-side version of block 202612), and to identify foreign nodes with the B-side version of block 202612.

There may be other, as-yet-unknown, longer-term e.ects of having block 202612 stored in the blockchain. There may be other, as-yet-unknown, longer-term e.ects of using the transactions validated in that block as obfuscating elements in new ring signatures, if the owners of those transactions (presumably the attackers) proceed to spend those transactions with zero mix-ins.

References
1.    Werner Albert. Monero Network Exploit Post-Mortem, 2014 (accessed Sept 15, 2014). https://forum.cryptonote.org/viewtopic.php?f=7&t=270.
2.    Ross Anderson. The classiﬁcation of hash functions. 1993.
3.    Ralph C Merkle. A digital signature based on a conventional encryption function. In Advances in Cryptology—CRYPTO’87, pages 369–378. Springer, 1988.
4.    John Viega and Matt Messier. Secure Programming Cookbook for C and C++: Recipes for Cryptography, Authentication, Input Validation & More. O’Reilly Media, Inc., 2003.

MRL-0003: Monero is Not That Mysterious

1 Introduction

Recently, there have been some vague fears about the CryptoNote source code and protocol ﬂoating around the internet based on the fact that it is a more complicated protocol than, for instance, Bitcoin. The purpose of this note is to try and clear up some misconceptions, and hopefully remove some of the mystery surrounding Monero Ring Signatures. I will start by comparing the mathematics involved in CryptoNote ring signatures (as described in [CN]) to the mathematics in [FS], on which CryptoNote is based. After this, I will compare the mathematics of the ring signature to what is actually in the CryptoNote codebase.

2 CryptoNote Origins

As noted in ([CN], 4.1) by Saberhagen, group ring signatures have a history starting as early as 1991 [CH], and various ring signature schemes have been studied by a number of researchers throughout the past two decades. As claimed in ([CN] 4.1), the main ring signature used in CryptoNote is based on [FS], with some changes to accomodate blockchain technology.

2.1 Traceable Ring Signatures

In [FS], Fujisaki and Suzuki introduce a scheme for a ring signature designed to “leak secrets anonymously, without the risk of identity escrow.” This lack of a need for identity escrow allows users of the ring signature to hide themselves in a group with an inherently higher level of distrust compared to schemes relying on a group manager.

In ring-signature schemes relying on a group manager, such as the original ring signatures described in [CH], a designated trusted person guards the secrets of the group participants. While anonymous, such schemes rely, of course, on the manager not being compromised. The result of having a group-manager, in terms of currencies, is essentially the same as having a trusted organization or node to mix your coins.

In contrast, the traceable ring signature scheme given in [FS] has no group manager. According to [FS], there are four formal security requirements to their traceable ring signature scheme:

•    Public Traceability -Anyone who creates two signatures for di.erent mes.sages with respect to the same tag can be traced. (In CryptoNote, if the user opts not to use a one-time key for each transaction, then they will be traceable, however if they desire anonymity, then they will use the one-time key. Thus as stated on page 5 of [CN], the traceability property is weakened in CryptoNote.)
•    Tag-Linkability -Every two signatures generated by the same signer with respect to the same tag are linked. (This aspect in CryptoNote refers to each transaction having a key image which prevents double spending.)
•    Anonymity -As long as a signer does not sign on two di.erent messages with respect to the same tag, the identity of the signer is indistinguishable from any of the possible ring members. In addition, any two signatures gen.erated with respect to two distinct tags are always unlinkable. (In terms of CryptoNote, if the signer attempts to use the same key-image more than once, then they can be identiﬁed out of the group. The unlinkability aspect is retained and is a key part of CryptoNote.)
•    Exculpability -An honest ring member cannot be accused of signing twice with respect to the same tag. In other words, it should be infeasible to counterfeit a tag corresponding to another person’s secret key. (In terms of CryptoNote, this says that key images cannot be faked.)

In addition, [FS] provide a ring signature protocol on page 10 of their paper, which is equivalent to the CryptoNote ring signature algorithm, as described on page 9-10 of [CN]. It is worthwhile to note that [FS] is a publicly peer-reviewed publication appearing in Lecture Notes in Computer Science, as opposed to typical crypto.currency protocol descriptions, where it is unclear whether or not they have been reviewed or not.

2.2 Traceability vs CryptoNote

In the original traceable ring signature algorithm described in [FS], it is possible to use the tag corresponding to a signature multiple times. However, multiple uses of the tag allow the user to be traced; in other words, the signer’s index can be determined out of the group of users signing. It is worthwhile to note that, due to the exculpability feature of the protocol ([FS] 5.6, [CN], A2), keys cannot be stolen this way, unless an attacker is able to solve the Elliptic Curve Discrete Logarithm Problem (ECDLP) upon which a large portion of modern cryptography is based ([Si] XI.4).

The process to trace a tag used more than once is described on ([FS], page 10). In the CryptoNote protocol, however, key images (tags) used more than once are rejected by the blockchain as double-spends, and hence traceability is not an aspect of CryptoNote.

2.3 Tag-Linkability vs CryptoNote

In essence, the tag-linkability aspect of the traceable ring signature protocol is what prevents CryptoNote transactions from being double-spends. The relevant protocols are referred to as “Trace” in ([FS], 5) and “LNK” in the CryptoNote paper. Essentially all that is required is to be able to keep track of the key images which have been used before, and to verify that a key image is not used again.

If one key-image is detected on the blockchain before another key-image, then the second key image is detected as a double-spend transaction. As key-images cannot be forged, being exculpable, the double-spender must in fact be the same person, and not another person trying to steal a wallet.

3 One-Time Ring Signatures (mathematics)

The security of the ring signature scheme as described in ([FS] 10, [CN] 10) and implemented in the CryptoNote source relies on the known security properties of Curve25519. Note that this is the same curve used in OpenSSH 6.5, Tor, Apple iOS, and many other[1] security systems.

3.1 Twisted Edwards Curves

The basic security in the CryptoNote Ring Signature algorithm is guaranteed by the ECDLP ([Si], XI.4) on the Twisted Edwards curve ed25519. The security properties of curve ed25519 are described in [Bern], by noted cryptographer Daniel Bernstein, andin([BCPM])byateamfromMicrosoftResearch.Bernsteinnotesabouted25519 the “every known attack is more expensive than performing a brute-force search on a typical 128-bit secret-key cipher.”

The curve ed25519 is a singular curve of genus 1 with a group law, and described

2 .121665 22
by .x+ y2 =1+xy. This curve is considered over the ﬁnite ﬁeld Fq,
121666q =2255 . 19. For those readers unfamiliar with algebraic geometry, an algebraic curve is considered as a one dimensional sort of space, consisting of all points (x, y) satisfying the above equation. All points are also considered modulo q. By virtue of its genus, ed25519 has a “group structure” which, for the purpose of this discussion, means if P =(x1,y1) is a point on the curve, and Q =(x2,y2) is another point on the curve, then these points can be added (or subtracted) and the sum (or di.erence), P + Q (or P . Q) will also be on the curve. The addition is not the naive adding of x1 + x2 and y1 + y2, but instead points are added using the rule

x1y2 + y1x2 y1y2 + x1x2
P + Q = ,
1+ dx1x2y1y2 1 . dx1x2y1y2

.121665

[1]http://ianix.com/pub/curve25519-deployment.html

where d =([BBJLP] 6, [BCPM]). The mathematics of curves of genus one are explained in great detail in [Si] for the interested reader.

Based on the above, we can compute P +P for any such point. In order to shorten notation, we rely on our algebraic intuition and denote 2P = P + P . If n . Z, then nP denotes the repeated sum

P + P + ··· + P
P+_ P
n times

using the above nonlinear addition law. As an example of how this di.ers from ordinary addition, consider the following system of equations:

aP + bQ = X
aP ' + bQ' = Y

where a, b, c, d are integers and P, Q, X are points. If this were a standard system of linear equations then one could use linear algebraic techniques to easily solve for a and b, assuming that P, Q, X, Y, P ', and Q ' are known. However, even if a, b are very small the above system is extremely di.cult to solve using the ed25519 addition law. For example, if a =1 and b =1, we have

xP yQ + yP xQ yP yQ + xP xQ
, =(xX ,yX )
1+ dxP xQyP yQ 1 . dxP xQyP yQ
xP 1 yQ1 + yP 1 xQ1 yP 1 yQ1 + xP 1 xQ1
, =(xY ,yY )
1+ dxP 1 xQ1 yP1 yQ1 1 . dxP 1 xQ1 yP 1 yQ1

So in reality, this is a system of 4 nonlinear equations. To convince yourself that it is in fact di.cult to ﬁgure out a and b, try writing the above systems assuming a =2, b =1. It should become clear that the problem is extremely di.cult when a, b are chosen to be very large. As of yet, there are no known methods available to e.ciently solve this system for large values of a and b.

Consider the following problem. Suppose your friend has a random integer q, and computes qP using the above form of addition. Your friend then tells you the x and y coordinates qP =(x, y), but not what q itself is. Without asking, how do you ﬁnd out what q is? A naive approach might be to start with P and keep adding P + P + P... until you reach qP (which you will know because you will end up at (x, y)). But if q is very large then this naive approach might take billions of years using modern supercomputers. Based on what mathematicians currently know about the problem and the number of possible q, none of the currently known attacking techniques can, as a general rule, do better in any practical sense than brute force.

In CryptoNote, your secret key is essentially just a very, very large number x (for other considerations, see section 4.3.3, we choose x to be a multiple of 8). There is a special point G on the curve ed25519 called “the base point” of the curve which is used as the starting point to get xG. Your public key is just xG, and you are protected by the above problem from someone using known information to determine the private key.

3.2 Relation to Diffie Helman

Included in a ring signature are the following equations involving your secret key x:

P = xG
I = xHp (P )

rs = qs . csx.

Here s is a number giving the index in the group signature to your public key, and Hp (P ) is a hash function which deterministically takes the point P to another point

''
P = x ' G, where x is another very large uniformly chosen number. The value qs is chosen uniformly at random, and cs is computed using another equation involving random values. The particular hash function used in CryptoNote is Keccak1600, used in other applications such as SHA-3; it is currently considered to be secure ([FIPS]). The CryptoNote use of a single hash function is consistent with the standard procedure of consolidating distinct random oracles (in proofs of security in [FS], for example) into a single strong hash function.

The above equations can be written as follows:

P = xG
' ' G '
P = xx
rs = qs . csx

Solving the top two equations is equivalent to the ECDH (as outlined in a previ.ous note ([SN])) and is the same practical di.culty as the ECDLP. Although the equations appear linear, they are in fact highly non-linear, as they use the addition described in 3.1 and above. The third equation (with unknowns qs and x), has the di.culty of ﬁnding a random number (either qs or x) in Fq, a very large ﬁnite ﬁeld; this is not feasible. Note that as the third equation has two unknowns, combining it with the previous two equations does not help; an attacker needs to determine at least one of the random numbers qs or x.

3.3 Time Cost to Guess q or x

Since q and x are assumed to be random very large numbers in Fq , with q = 2255 . 19 (generated as 32-byte integers), this is equivalent to a 128-bit security level ([BCPM]), which is known to take billions of years to compute with current supercomputers.

3.4 Review of Proofs in Appendix

In the CryptoNote appendix, there are four proofs of the four basic properties required for security of the one-time ring-signature scheme:

• Linkability (protection against double-spending)
• Exculpability (protection of your secret key)
• Unforgeability (protection against forged ring signatures)
• Anonymity (ability to hide a transaction within other transactions) These theorems are essentially identical to those in [FS] and show that the ring signature protocol satisﬁes the above traits. The ﬁrst theorem shows that only the secret keys corresponding to the public keys included in a group can produce a signature for that group. This relies on the ECDLP for the solution of two simulta.neous (non-linear) elliptic curve equations, which, as explained in 3.2, is practically unsolvable. The second theorem uses the same reasoning, but shows that in order to create a fake signature that passes veriﬁcation, one would need to be able to solve the ECDLP. The third and fourth theorems are taken directly from [FS].

4 One-Time Ring Signatures (Application)

To understand how CryptoNote is implementing the One-Time Ring signatures, I built a model in Python of Crypto-ops.cpp and Crypto.cpp from the Monero source code using naive Twisted Edwards Curve operations (taken from code by Bernstein), rather than the apparently reasonably optimized operations existing in the CryptoNote code. Functions are explained in the code comments below. Using themodelwillproduceaworkingringsignaturethatdi.ersslightlyfromtheMonero ring signatures only because of hashing and packing di.erences between the used libraries. The full code is hosted at the following address: https://github.com/monero-project/mininero

Note that most of the important helper functions in crypto-ops.cpp in the CryptoNote source are pulled from the reference implementation of Curve25519. This reference implentation was coded by Matthew Dempsky (Mochi Media, now Google)[2].

In addition, after comparing the python code to the paper, and in turn comparing the python code to the actual Monero source, it is fairly easy to see that functions like generate_ring_sig are all doing what they are supposed to based on the pro.tocol describedinthewhitepaper. For example,here is the ring signature generation algorithm used in the CryptoNote source:

Algorithm 1 Ring Signatures

i . 0
while i < numkeys do

if i = s then
k . random Fq element
Li . k · G
Ri . k ·Hp(Pi)

else
k1 . random Fq element
k2 . random Fq element
Li . k1Pi + k2G
Ri . k1I + k2Hp(Pi)
ci . k1
ri . k2

end if
i . i +1
end while
h .Hs(prefix + L.is + R.is))
p
cs . h . i ci
=s
rs . k . xcs
return (I, {ci}, {ri})

Comparing this with [CN] shows that it agrees with the whitepaper. Similarly, here is the algorithm used in the CryptoNote source to verify ring signatures:

Algorithm 2 VER
i =0
while i < numkeys do L.. ciPi + riG
i R.. riHp(Pi)+ ciI
i
i . i +1
end while
h .Hs(prefix + L.is + R.is))
p
h . h .
i
=s ci return (h == 0(mod q)) == 0

4.1 Important Crypto-ops Functions

Descriptions of important functions in Crypto-ops.cpp. Even more references and information is given in the comments in the MiniNero.py code linked above.

[2]http://nacl.cr.yp.to/

4.1.1 ge_frombytes_vartime

Takes as input some data and converts to a point on ed25519. For a reference of (q.5)/837 the equation used, . = uvuv, see ([BBJLP], section 5).

4.1.2 ge_fromfe_frombytesvartime

Similar to the above, but compressed in another form.

4.1.3 ge_double_scalarmult_base_vartime

Takes as inputs two integers a and b and a point A on ed25519 and returns the point aA + bG, where G is the ed25519 base point. Used for the ring signatures when computing, for example, Li with i = s as in ([CN], 4.4)

4.1.4 ge_double_scalarmult_vartime

Takes as inputs two integers a and b and two points A and B on ed25519 and outputs aA + bB. Used, for example, when computing the Ri in the ring signatures with i = s ([CN], 4.4)

4.1.5 ge_scalarmult

Given a point A on ed25519 and an integer a, this computes the point aA. Used for example when computing Li and Ri when i = s.

4.1.6 ge_scalarmult_base

Takes as input an integer a and computes aG, where G is the ed25519 base point.

4.1.7 ge_p1p1_to_p2

There are different representations of curve points for ed25519, this converts between them. See MiniNero for more reference.

4.1.8 ge_p2_dbl

This takes a point in the “p2” representation and doubles it.

4.1.9 ge_p3_to_p2

Takes a point in the “p3” representation on ed25519 and turns it into a point in the “p2” representation.

4.1.10 ge_mul8

This takes a point A on ed25519 and returns 8A.

4.1.11 sc_reduce

Takes a 64-byte integer and outputs the lowest 32 bytes modulo the prime q. This is not a CryptoNote-speciﬁc function, but comes from the standard ed25519 library.

4.1.12 sc_reduce32

Takes a 32-byte integer and outputs the integer modulo q. Same code as above, except skipping the 64.32 byte step.

4.1.13 sc_mulsub

Takes three integers a, b, c in Fq and returns c . ab modulo q.

4.2 Important Hashing Functions

4.2.1 cn_fast_hash

Takes data and returns the Keccak1600 hash of the data.

4.3 Crypto.cpp Functions

4.3.1 random_scalar

Generates a 64-byte integer and then reduces it to a 32 byte integer modulo q for 128-bit security as described in section 3.3.

4.3.2 hash_to_scalar

Inputs data (for example, a point P on ed25519) and outputs Hs (P ), which is the Keccak1600 hash of the data. The function then converts the hashed data to a 32-byte integer modulo q.

4.3.3 generate_keys

Returns a secret key and public key pair, using random_scalar (asdescribed above) to get the secret key. Note that, as described in [Bern], the key set for ed25519 actually is only multiples of 8 in Fq, and hence ge_scalarmult_base includes a ge_mul8 to ensure the secret key is in the key set. This prevents transaction malleability at.tacks as described in ([Bern], c.f. section on “small subgroup attacks”). This is part of the GEN algorithm as described in ([CN], 4.4).

4.3.4 check_key

Inputs a public key and outputs if the point is on the curve.

4.3.5 secret_key_to_public_key

Inputs a secret key, checks it for some uniformity conditions, and outputs the cor.responding public key, which is essentially just 8 times the base point times the point.

4.3.6 hash_to_ec

Inputs a key, hashes it, and then does the equivalent in bitwise operations of mul.tiplying the resulting integer by the base point and then by 8.

4.3.7 generate_key_image

Takes as input a secret key x and public key P , and returns I = xHp (P ), the key image. This is part of the GEN algorithm as described in ([CN], 4.4).

4.3.8 generate_ring_signature

Computes a ring signature, performing SIG as in ([CN], 4.4) given a key image I, a list of n public keys Pi, and a secret index. Essentially there is a loop on i, and if the secret-index is reached, then an if-statement controls the special computation of Li,Ri when i is equal to the secret index. The values ci and ri for the signature are computed throughout the loop and returned along with the image to create the total signature (I,c1, ..., cn,r1, ..., rn) .

4.3.9 check_ring_signature

Runs the VER algorithm in ([CN], 4.4). The veriﬁer uses a given ring signature n
to compute L ' i = riGi, Ri ' = riHp (Pi)+ ciI, and ﬁnally to check ifi=0 ci = Hs (m, L 0' , ..., L ' ,R 0' , ..., R ' ) mod l.nn

4.3.10 generate_key_derivation

Takes a secretkey b,andapublic key P , and outputs 8·bP . (The 8 being for thepur.pose of the secret key set, as described in 4.3.3). This is used in derive_public_key as part of creating one-time addresses.

4.3.11 derivation_to_scalar

Performs Hs (.) as part of generating keys in ([CN], 4.3, 4.4). It hashes an output index together with the point.

4.3.12 derive_public_key

Takes a derivation rA (computed via generate_key_derivation), a point B, and an output index, computes a scalar via derivation_to_scalar, and then computes Hs (rA)+ B.

4.3.13 generate_signature

This takes a preﬁx, a public key, and a secret key, and generates a standard (not ring) transaction signature (similar to a Bitcoin transaction signature).

4.3.14 check_signature

This checks if a standard (not ring) signature is a valid signature.

5 Conclusion

Despite the ring signature functions in the original CryptoNote source being poorly commented, the code can be traced back to established and used sources, and is rel.atively straightfoward. The Python implementation provided with this review gives further indication of the code’s correctness. Furthermore, the elliptic curve mathe.matics underlying the ring signature scheme has been extremely well-studied; the concept of ring signatures is not novel, even if their application to cryptocurrencies is.

References

BBJLP. Bernstein, Daniel J., et al. "Twisted edwards curves." Progress in Cryptology–AFRICACRYPT 2008. Springer Berlin Heidelberg, 2008. 389-405. BCPM. Bos, Joppe W., et al. "Selecting Elliptic Curves for Cryptography: An E.ciency and Security Analysis." IACR Cryptology ePrint Archive 2014 (2014): 130. Bern. Bernstein, Daniel J. "Curve25519: new Di.e-Hellman speed records." Public Key Cryptography-PKC 2006. Springer Berlin Heidelberg, 2006. 207-228. CH. Chaum, David, and Eugene Van Heyst. "Group signatures." Advances in Cryptology—EUROCRYPT’91. Springer Berlin Heidelberg, 1991. CN. van Saberhagen, Nicolas. "CryptoNote v 2.0." (2013). FIPS. SHA, NIST DRAFT. "standard: Permutation-based hash and extendable-output functions." DRAFT FIPS 202 (2014).
Fu.    Fujisaki, Eiichiro. "Sub-linear size traceable ring signatures without random oracles." IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences 95.1 (2012): 151-166.
FS.    Fujisaki, Eiichiro, and Koutarou Suzuki. "Traceable ring signature." Public Key Cryptography–PKC 2007. Springer Berlin Heidelberg, 2007. 181-200.
IAN. IANIX http://ianix.com/pub/curve25519-deployment.html Si. Silverman, Joseph H. The arithmetic of elliptic curves. Vol. 106. Dordrecht: Springer, 2009.
SN. http://lab.monero.cc/pubs/multiple_equations_attack.pdf

MRL-0004: Improving Obfuscation in the CryptoNote Protocol

Abstract

We identify several blockchain analysis attacks available to degrade the untraceability of the CryptoNote 2.0 protocol. We analyze possible solutions, discuss the relative merits and drawbakcs to those solutions, and recommend improvements to the Monero protocol that will hopefully provide long-term resistance of the cryptocurrency against blockchain analysis. Our recommended improvements to Monero include a protocol-level network-wide minimum mix-in policy of n =2 foreign outputs per ring signature, a protocol-level increase of this value to n =4 after two years, and a wallet-level default value of n =4 in the interim. We also recommend a torrent-style method of sending Monero output. We also discuss a non-uniform, age-dependent mix-in selection method to mitigate the other forms of blockchain analysis identiﬁed herein, but we make no formal recommendations on implementation for a variety of reasons. The ramiﬁcations following these improvements are also discussed in some detail.

This research bulletin has not undergone peer review, and reﬂects only the results of internal investigation.

1 Overview

We address several security concerns in traceability when using a ring signature approach to ensure the privacy of digital currency transactions. Recall that the CryptoNote protocolproposalproves unconditionalunlinkability, but untraceability remained unproven; indeed, CryptoNote is very traceable, as we demonstrate in this document.

We remind the reader that we say that two transactions are unlinkable when an observercannotprovethatthosetwotransactionsweresenttothesameuser.Wesay that a transaction is considered untraceable if all possible senders of a transaction are equiprobable. Hence, the problems with untraceability in CryptoNote suggest that, while users can receive CryptoNote-based cryptocurrencies with no concern for their privacy, they cannot necessarily spend those currencies without releasing someinformationabouttheirpasttransactions.Ofcourse,withoutanon-interactive zero-knowledge (NIZK) approach (see, for example, [2]), traceability is inevitable. However, NIZK approaches to date have had limited applications and su.er the disadvantage of computational cost. Addressing some of the security issues related totraceability withoutzero-knowledgetechniquesis themain goal ofthisdocument. Critically, we ask the reader to observe that the techniques presented do not violate users’ capabilities to provide auditable histories of their transactions in order to comply with local laws (see Section 6).

In Section 2, we review how spending in the clear, or zero-mix spending leads to traceability, as detailed in [3]. In Section 3, we identify three further routes of attack that may be used to reduce untraceability in CryptoNote, although we make no claims that this list is exhaustive. In particular, Section 3.1 details how the age of transaction outputs can be used to remove untraceability from CryptoNote. Section 3.3 details how observable data about the transactions used in a ring sig.nature can be used in a combinatorial analysis to associate transaction outputs that are otherwise not associated together. Section 3.2 details how careless usage of transaction outputs together in a new transaction can lead an attacker to draw conclusions about the ownership of the original transaction outputs; we call this an association-by-use attack.

In Section 4, we detail some recommendations to mitigate these avenues of attack, and for other recommendations we merely make a sketch. In Section 4.1, we recommend a mandatory, network-wide minimum mix-in of n =2, and we justify this choice. In Sections 4.2 and 4.3, we detail our recommendations for mitigating the combinatorial and association-by-use attacks described in Sections 3.3 and 3.2, respectively.

In Section 5 we detail our roadmap to implementing our policy suggestions. Fi.nally, in Section 6, we discuss a method that a user can use to comply with local laws without sacriﬁcing the privacy we have worked so hard to attain.

2 Traceability with Zero Mix-In Spending

In this section,we list several problems with untraceability in CryptoNote and other ring signature schemes. The issues detailed in [3] are not comprehensive, of course, and we describe other issues in this document. We make no claims that the issues presented in this document are comprehensive, either. In Section 3.1, we describe age-related attacks on transaction outputs, in Section 3.3, we describe certain combinatorial attacks that can be used to force traceability using data external to the ring signatures such as block height, and in Section 3.2, we describe how careless usage of transaction outputs together can reveal to an eavesdropper further infor.mation about transaction ownership. The chain reactions caused by transactions spent in the clear, described in [3], form a good starting point of the discussion.

When using a CryptoNote-style ring signature protocol in a digital currency to obscure traceability and linkability, a malicious party with a large number of trans.action outputs can cause a chain reaction in traceability by sacriﬁcing their own anonymity, allowable through so-called zero mix transactions which have no foreign outputs used as mix-ins for the associated ring-signature. Before we proceed, we wish to establish some terminology for the reader. First, recall that in transparent cryptocurrencies such as Bitcoin, transaction outputs can be inspected to determine whether they have been spent or not. Indeed, the acronym UTXO is often used, meaning unspent transaction output. In CryptoNote, this acryonym would be inap.propriate, as it is di.cult to determine whether a transaction output has been spent or not without using some form of blockchain analysis or other form of traceability attack, such as the attacks described herein. As a consequence, we will not refer to transaction outputs as spent or unspent within this document. Second, we shall refer to the practice of spending a transaction output with a ring signature that was generated using no foreign transaction outputs used as mix-ins by spending in the clear or as zero-mix spending. Spending a transaction in the clear means generating a ring signature with no mix-ins; the ring signature is equivalent to a usual digital signature in this case.

Our initial security concerns can be visualized quite simply. Just as in the original ring signature paper, [4], consider the situation in which an employee of a government leaks a secret by collecting the public keys of n other employees (mix-ins). The leaker then uses those keys and her own private key to generate a ring signature to sign the document that they are leaking. An observer will, apparently, only have a 1 in n +1 chance at guessing which employee leaked the secret. What happens if the employees who owned the mix-in keys begin vigorously denying that they leaked that secret? What happens as, one by one, those employees willfully reveal their private keys to prove to an authority ﬁgure, or simply to one another, that they aren’t the leak? At ﬁrst, your obfuscatory power drops from 1 of n +1 to 1 of n after the ﬁrst employee reveals their key. After the second employee reveals their key, your obfuscatory power drops from 1 of n to 1 of n . 1, and so on. In this scenario, ring signatures become signiﬁcantly weaker. As we identiﬁed in [3], when ring signatures are iteratively used, transaction after transaction, the problem compounds. Through chain reactions, the security of one transaction can depend on the security of a transaction with no directly related parties. Of course, ring signatures are still valid as digital signatures go, and so their ability to act as the backbone of a cryptocurrency network is not in question. The only question is to address this degradation in untraceability.

To see an example of this, say that Alice fashions a ring signature using the mix-ins {Bob, Cynthia, Doug}. For any observer, then, the list of possible senders of the transaction is clearly {Alice, Bob, Cynthia, Doug}. If Bob, Cynthia, and Doug then each spend their transaction outputs with zero mix-ins, then it is obvious any any observer that they no longer own those transaction outputs. Hence, they can be excluded from the list of possible senders of Alice’s transaction, and Alice is now exposed as the sender of her transaction. It is not as if only three users, Bob, Cynthia,andDoug,spenttheirtransactionsintheclear,butasifallfourusersspent their transactions in the clear. Hence, any signatures using Alice’s transactions as a mix-in are now also compromised. If enough transactions are a.ected, there can be second-order e.ects, leading to a chain reaction.

We modeled the creation of ring signatures in [3] using a variant of the Pólya Urn Process described in depth in [1], and we used this model to demonstrate that chain reactions in traceability are possible. Consequently, if all of the mix-ins used to generate a new ring signature are controlled by a malicious party, then that malicious party has gained control over the untraceability of the new ring signature. The Pólya Urn Process su.ers from the rich get richer e.ect; an attacker in control of many outputs at the beginning of the experiment will probabilistically control more and more as time goes on unless parameters are chosen carefully.

As a saving grace, within the CryptoNote protocol, users cannot be directly as.sociated with transaction outputs. All that can be said is whether a key image is associated with a transaction output. Indeed, the so-called stealth address element of the CryptoNote protocol ensures that this does not lead to a complete failure in anonymity. To be precise, when Bob, Cynthia, and Doug spend their transaction outputs in the clear, they are revealing to the world via the blockchain that their outputs belong to their associated key images. When this occurs, and when it is clear that Alice’s key image was used in none of those transactions, her key image is now linked to the relevant output, not Alice’s identity.

2.1 Change and Dust Force Zero Mix Spending

We now show how users are regularly forced to make transactions in the clear, regardless of their intent.

One may have reasonably assumed that the original CryptoNote developers would have implemented a standardized denomination requirement for all transaction outputs, but this is not so. For example, whereas 102.5 XMR would reasonably be represented as 100 XMR, 2 XMR and 0.5 XMR, the protocol does not disal.low any strange denominations such as 100 XMR, 2 XMR, 0.499999999 XMR and 0.000000001 XMR. An output is unmixable either because it has been denominated unusually, such as our example with 0.499999999 XMR (and hence the transaction output set associated with the denomination is empty) or its amount is smaller than the minimum transaction fee, such as our example with 0.000000001 XMR.

Below is a simple example of dust generation using the current denomination code. In this example, Bob has 103.32111111 in his wallet and decides to send Alice 53.32111111 XMRwith 2 foreigntransactionoutputsusedasmix-inswhilereceiving 50 XMR in change. Let us presume that Bob’s XMR is split up by denomination such that his 103.32111111 XMR consists of an output of 100 XMR, an output of 3 XMR, an output of 0.3 XMR, an output of 0.02 XMR, and an output of dust, 0.00111111 XMR. The current denomination code will take the output of 100 XMR that Bob owns, split it into two outputs, each for 50 XMR, and send one to Alice and one to Bob. It will then take all of the remaining outputs in Bob’s wallet (3, 0.3, 0.02, 0.00111111) and send them to Alice, each signed with their own ring signature with the minimum mix-in level wherever possible. Unfortunately, the dust output, .00111111 is in a unique category in the sense that the transaction output set responsible for the denomination .00111111 is empty. Hence, no mix-ins are possible. We get a diagram like this:

100 XMR (Bob) (mix-in=2) - 50 XMR (Alice), 50 XMR (Bob)
3 XMR (Bob) (mix-in=2) - 3 XMR (Alice)
0.3 XMR (Bob) (mix-in=2) - 0.3 XMR (Alice)
0.02 XMR (Bob) (mix-in=2) - 0.02 XMR (Alice)
0.00111111 XMR (Bob) (mix-in=0) - 0.00111111 XMR (Alice)

Some privacy issues with this transaction may have become obvious already; since CryptoNote does not enforce transaction outputs to adhere to a strict form of A . 10B for an integer 1 , A , 9, we have a so-called dust output of 0.00111111 XMR, and this output cannot be mixed unless many other dust outputs happen to have such a strange denomination. Despite the privacy allegedly a.orded by ring signatures with all other outputs, Bob has e.ectively stripped himself of privacy by including the dust input that is unmixable. Indeed, any observer can see this strange, unmixed output bundled with this transaction, and included in the same block; the observer will then draw obvious conclusions about funds ownership.

To worsen matters, Alice now also has a dust input that can not be spent using ring signatures without compounding the previous problem. While no observer can yet determine who received the transaction due to the uconditional unlinkability of the CryptoNote protocol, Alice will remove privacy from her transactions if she later decides to use that dust as an input due to the aforementioned problems with untraceability.

3 More Issues with Traceability

In this section, we identify other security concerns related to traceability in CryptoNote. This is not intended to be a comprehensive list of all routes of at.tack toward violating untraceability in CryptoNote, but represents a list of the most pressing issues identiﬁed by the Monero Core Team and the Monero Research Lab as routes of attack that may erode untraceability in the event of blockchain analysis.

3.1 Temporal Associations

This section describes how the age of a transaction output causes a circumstantial traceability among transactions. There are at least two distinct problems with the age of transactions. As before, we start with chain reactions. In expectation, the older outputs have been used in more ring signatures than younger outputs; they have simply been extant in the transaction output set for longer, and have had more opportunities to be chosen from ring signatures. Any ring signature generation scheme that ignores this will provide a disproportionate amount of chain reaction potential to the over-used transaction output. Further, the earlier an attacker gains control of a transaction output set, the greater their capability to execute a chain reaction attack.

The second problem we identify is related to traceability, but not to chain reac.tions. Each transaction spends a transaction output using a ring signature, which uses an associated list of transaction outputs as mix-ins. The genuine transaction output being spent is a member of this list. If n foreign transaction outputs have been used in this ring signature, then this list has n +1 elements on it. An observer cannot distinguish which transaction output in the list associated with the ring sig.nature is the output actually being spent without some form of blockchain analysis. However, in practice, given a certain transaction output, an attacker may model the cumulative probability that the output has already been spent as an increasing function of time. An output that has been in the blockchain for 202612 blocks is much more likely to have already been spent than an output that has been in the blockchain for 202 blocks. Hence, in the absence of any other external information, with any given output, the youngest output used to fashion the ring signature is the most likely output to be the genuine output being spent. Conversely, the oldest output is the one least likely to be the genuine output being spent.

In this paper, we do not directly address how to mitigate these observations for a variety of reasons. If it were possible to determine whether outputs had been spent, we would simply choose outputs that had remained unspent. However, this is not a simply tractable problem without blockchain analysis. One solution to this problem is to determine a non-uniform method of choosing transaction outputs for ring signatures; choose transaction outputs based on their age such that the probability that these outputs are chosen for a ring signature is inversely related to the probability that they have been spent already. This would suggest that we, the developers of Monero, must estimate the probability distribution governing the age of transaction outputs.

This, too, is problematic. When an exchange rate is experiencing a strong long.term decline (inﬂation), rational users are more likely to spend their transaction outputs, for tomorrow their outputs will be worth less in terms of goods and services than today and hoarding is not economically rational. When an exchange rate is experiencing a strong long-term increase (deﬂation), rational users are more likely tohoardtheirtransactionoutputsfortheoppositereason. Hence,thedistributionof transaction output ages will at least vary over time, and, presuming any proportion ofusersarerationalwillcertainlydependsensitivelyontheeconomicperformanceof the currency. It is unwise to design security recommendations around the economic performance of our protocol.

An astute observer may recommend mining Bitcoin data, which is transparent, and developing a marginal lifespan distribution from that information. However, in this instance, we are making several problematic assumptions. Assuming that the Bitcoin economy is equivalent to the Monero economy, or even a decent proxy, is problematic due to the comparative sizes of the economies. Further, choosing any one particular distribution that does not vary over time will allow an attacker to exploit the di.erence between the ﬁxed distribution of choice and the time-varying distribution in the wallet software. One would be inclined to choose distributions that vary over time, perhaps with some iterative updating method, but this requires some form of observing data and responding to those observations; as we’ve previ.ously mentioned, it’s not easily tractable to observe the lifespan behavior within a CryptoNote protocol.

Hence, choosing a ﬁxed distribution in order to generate random numbers to pick outputs used in a ring signature is a suboptimal solution. Another alternative to choosing a ﬁxed distribution is to probabilistically choose from a family of distri.butions; to put it simply, we could roll dice twice, ﬁrst to pick a distribution, and then second to pick outputs using that previously chosen distribution for the ring signature. However, these mixed distribution approaches are, in the end, equivalent to choosing a single ﬁxed distribution and the problems with this approach reduce to the problems in the ﬁrst, ﬁxed distribution scenario.

Paradoxically, keeping wallet software intact without modiﬁcation is equivalent to choosing the uniform distribution for each ring signature! Which is to say, we are stuckbetweenarockandahardplacewithtransactionoutputlifespandistributions, for any choice we make is certain to be a sub-optimal decision. The Monero Core TeamandtheMoneroResearchLabwouldliketofollowthedevelopmentphilosophy that it is wise to start with smaller changes at ﬁrst and then ramp those changes up over time, rather than start with drastic changes and try to scale them back. Hence, although we have identiﬁed this security issue, we are not making formal recommendations yet until we have further data to inform our choices.

3.2 Association by Use of Outputs Within a Transaction

In this Section, we describe how careless usage of outputs together can be used to link those transactions as belonging to a common user. In other words, transaction outputs that are used together are probably owned together. Using the principle from Section 3.3, we see that this is a form of the combinatorial attack. Although the ring signature a.ords observers equiprobability across all possible signers, data exterior to the ring signature may allow for extra information.

Consider the following generalized example. If Bob sends Alice some Monero, it is broken up into several transaction outputs as in Example 2.1. When Alice goes on to spend that money, she may use two or more outputs from that same initial transaction in her ring signature. If so, an observer may conclude that the outputs stemming from a common root transaction are more likely to belong to the true signer of the transaction. This can lead to a probabilistic issue with traceability. This may be hard to visualize, so here is a speciﬁc example beginning with 1 XMR in Bob’s wallet.

Bob wishes to send 0.75 XMRtoAlice,andwillpay 0.01 XMRinfees.TheMonero that Alice receives, 0.75 XMR, will be delivered as two new unspent transaction outputs with amounts 0.7 XMR and 0.05 XMR. Further, an output of 0.01 XMR must be delivered as a fee. This leaves 0.24 XMR as change, which will be delivered to Bob in two unspent transaction outputs of amount 0.2 and 0.04 XMR each. At some point in time later, Alice realizes she has 0.75 XMR and decides to go spend it some place. When she does so, both of her outputs, 0.7 XMR and 0.05 XMR, are included in her ring signature. An observer could then look at her ring signature and draw a conclusion that whoever signed that ring signature probably is the owner of both the 0.7 XMR and the 0.05 XMR. Even if Alice used a ring signature with many mix-ins, since she used two outputs that originated from the same transaction, those outputs are more likely to be identiﬁable as owned by the true signer. The transaction could look something like Table 1.

Table 1: Linking transaction outputs by association
(a) Transaction 1 by Bob (b) Transaction 2 by Bob
In Out
1.0 XMR (Bob)
0.7 XMR (Alice)
0.05 XMR (Alice)
0.2 XMR (Bob)
0.04 XMR (Bob)
0.01 XMR (Fee)
In Out
0.7 XMR (Alice) · · ·
0.05 XMR (Alice) · · ·

Notice a few things about this example. First, this example is almost identical to the combinatorial attack using block height as will be described in Section 3.3. However, in this example, these outputs do not just share block height, but in fact they share a common root transaction. Second, this problem is symmetric with respect to Bob and Alice. Bob encounters a hazard also when he tries to spend his change output of 0.2 or 0.04 in a transaction. His ring signatures specify his originating transaction, as well, so if he decides to spend both his 0.2 and 0.04 XMR in the same transaction, they can be linked to him.

3.3 Combinatorial Attacks to Reveal Outputs

In this section, we generalize the attack described in Section 3.2 to describe a so-called combinatorial attack, during which a combinatorial analysis can be executed in order to draw a conclusion about output ownership. The primary conclusion of this section is simply that transactions should be broken up into several smaller transactions across periods of time as in a torrent, and users should re-send them.selves their funds periodically.

Any data available to an observer about the transactions used to fashion a ring signature can be used in a combinatorial analysis to draw a conclusion about funds ownership. Consider the following example using block height. Lets say Cynthia received 10, 20, and 30 XMR at block height 39, 859 in separate transactions. In a much later block, an observer, Eve, sees three transactions of 10, 20, and 30 XMR, respectively. Eve sees that the 10 XMR transaction uses a mix-in from block 39, 859, and so does the 20 XMR transaction, and so does the 30 XMR transaction. By checking all combinations available, Eve concludes that it is more likely that whoever received 60 XMR at block height 39, 859 is spending those funds than any other user, although Cynthia and Eve do not necessarily know each others’ identities.

This information can be used in tandem with other routes of attack.Tobespeciﬁc, if a set of distinct transactions has some common data (such as a common block height) and if all transactions in this set use mix-ins with some other common data, then it is likely that these transactions are related by a common user. This provides some information to attackers about traceability. Our deﬁnition of a combinatorial attack seems strikingly general, and, in fact, includes the association-by-use attack described in Section 3.2. A combinatorial analysis of a set of transactions with some common data will lead to very few conclusions using this principle. That is to say, although this sort of combinatorial attack may not work most of the time, when it does work, it works very well.

4 Our Recommendations

In this Section, we specify our recommendations to the Monero Core Team for developing the Monero cryptocurrency. Our recommendations, while not perfect solutions to each of the above problems, mitigate the exposure users experience to traceability attacks. We recommend any and all CryptoNote developers to imple.ment at least the network-wide minimum mix-in requirement of n . 3, and the larger the better.

4.1 Requiring Minimum Mix-Ins

In this section, we attempt to mitigate the chain reaction phenomenon by recom.mending a change to the Monero protocol requiring a minimum n . 2 mix-ins per transaction whenever at least 2 foreign outputs are available for mixing, and setting the default in wallet software to n . 4 mix-ins per transaction. We also justify this recommendation.

Speciﬁcally, we recommend that the Monero protocol be modiﬁed such that a node refuses to relay a transaction containing a ring signature without the requisite number of mix-ins, unless that transaction is denominated such that it lies in a transaction output set without a su.cient number of transaction outputs. We also recommend that the Monero protocol be modiﬁed such that a block is rejected if it contains any transactions containing ring signatures without the requisite number of mix-ins.

Of course, any mandatory minimum mix-in n . 2 allows for a probabilistic guar.antee that any traceability chain reaction will terminate. Speciﬁcally, an attacker will have to spend more funds than they will reveal, as we demonstrated in [3]. The only question is “what value of n should be enforced?” This has been a source of heated debate within the Monero Research Lab and the Monero Core Team. Large values of n will lead to greater security at the expense of an unreasonably large blockchain and greater transaction fees. As previously mentioned, we are not using NIZK techniques and no ring signature method can match that level of security, and so any blockchain size exceeding the size of an NIZK blockchain (e.g. Zerocoin) is unacceptable. On the other hand, smaller values of n may save blockchain space, but could possibly compromise security. Luckily, the size, in terms of bytes, of a ring signature grows approximately linearly with the number of foreign keys used in the ring signature, and increasing n uniformly improves system resilience to chain reaction attacks as described in [3]. This resilience improvement is not linear, how.ever, and there is no optimal point. In terms of security, n =3 is superior to n =2, and n =4 is superior to n =3, and so on.

Using Monte Carlo simulations of chain reactions at [5], we have concluded that individual users can be protected against chain reaction attacks by using at least n =3 mix-ins for their ring signatures. Quantiﬁable security metrics can verify these results in at least two ways. The ﬁrst metric we used to verify these results demanded that the success of a chain reaction attack depend on the output set size (yielding a choice of n 3). Indeed, using this ﬁrst metric, we see that, regardless of the size of an output set, a 50% attacker can be 95% conﬁdent of controlling one in ten new transactions when n =2, but when n 3, the attacker’s success depends sensitively on output set size. The second metric we used demanded that an attacker controlling 50% of an output set of arbitrary size must wait for tens of thousands of transactions before they may be 95% conﬁdent of gaining control over a single new transaction (yielding a choice of at least n 4).

However, it is still true that, for n =2, chain reactions are still probabilistically guaranteed to burn themselves out, ensuring that degradation of system integrity is somewhat graceful and that any chain reaction attacker must spend more capital than they reveal. After much debate, we have decided that setting the default mix-in value within wallet software to n =4 but allowing users who are less concerned about anonymity and more concerned about transaction fees to lower their mix-ins to n =2 seems to be prudent. As previously mentioned, the Monero Core Team and the Monero Research Lab would like to follow the development philosophy that it is wise to start with smaller changes at ﬁrst and then ramp those changes up over time, rather than start with drastic changes and try to scale them back. We are also formally recommending that the protocol be modiﬁed such that, after the blockchain has gotten su.ciently long, the network-wide minimum mix-in policy be increased to n =4. After 2 years, Monero will automatically become more secure.

As an aside, notice that this is not the only way to address the chain reaction problem. An alternative way to address the chain reaction problem induced by allowing transactions to be spent in the clear is a tag-based solution. This solution recommends tagging some transaction outputs as safe to mix, and some as unsafe to mix; equivalently, this would involve pruning mix-in sets. Before validating a block, a miner could verify that no ring signature in the block uses a zero mix-in transaction output as a signature input by checking tags.

However, such tag-based solutions have some problems associated with them. For example, while a single block may not contain any ring signatures which use zero mix-in transaction outputs as signature inputs, due to chain reactions as described in [3], the block may still contain traceable transactions. Indeed, the transactions present in a block may be separated from dangerous “in-the-clear” transactions by several other transactions. Further, in a tag-based solution that still allows for zero mix-in transactions, users may still opt-out of beneﬁting from the privacy that ring signatures provide. It would be preferable to ﬁnd a solution that maintains privacy as a uniform policy across the network, allows users to opt-in to greater privacy, and allows for compliance with local laws as described in Section 6.

4.2 Mitigating Combination Attacks

The easiest way to obfuscate ownership of the funds from eavesdropping by Eve during a combinatorial attack would be for Bob to simply send outputs owned by himself to himself separately every few random periods of time. This skews the blockchain analysis performed by Eve in Section 3.3 and, in fact, in Section 3.1. In this section, we merely specify that no user resend all of her outputs to herself at the same time. Furthermore, any receiver of funds, by contrast, should request that the sender break the transaction up into pieces in a torrent and send the required amount over a period of time . This way, if Eve is anticipating a certain transaction amount within a certain window of time, she can not readily ascertain if some combination of outputs from all transactions in a given block might correspond to the exact amount which she expects the recipient to be sent.

By re-sending transactions to oneself iteratively over intervals of time with ran.dom length, and by breaking all transactions (including a resend transaction) into multiple smaller transactions, also sent over an interval of time, we dramatically weaken the ability for an eavesdropper to glean information from the blockchain based solely on block height. Notice that this recommendation is a wallet-level rec.ommendation, not a protocol-level recommendation.

4.3 Mitigating Association by Use

To prevent a so-called “Assocation by Use” attack, an alternative transaction gen.eration scheme could be used. Rather than generating a single transaction with as few unspent transaction outputs as possible, we can generate a separate transaction for each output. Hence, if Bob wants to send Alice 0.75 XMR out of a wallet with 1 XMR, then he will actually send two transactions. One transaction will send 0.7 XMR and one transaction will send 0.05 XMR. First, the usual denomination and change-splitting method will be applied before sending 0.7 XMR out of a wallet with 1 XMR within, as in Table 2. Then, with his change, Bob will complete the transaction; after his ﬁrst transaction, this will leave Bob with a spare 0.09 XMR output, from which he can execute the usual denomination and change-splitting method before sending 0.05 XMR to Alice as in Table 3.

Table 2: Transaction 1
(a) Transaction 1A by Bob (b) Transaction 1B by Alice
In Out
1.0 XMR (Bob)
0.7 XMR (Alice)
0.2 XMR (Bob)
0.09 XMR (Bob)
In Out
0.7 XMR (Alice) · · ·

0.01 XMR (Fee)
Table 3: Transaction 2
(a) Transaction 2A by Bob
(b) Transaction 2B by Alice

In Out
0.05 XMR (Alice) · · ·

In the above scheme, Bob generates only generates one extra output, but in doing so is able to protect Alice from being uncovered in the future by transactional association of outputs used in ring signatures. Mining fees are increased according to the number of transaction outputs received, but anonymity is preserved.

Notice, however, that there is a subtlety under this alternative scheme. If Bob uses his own change output in order to send the 0.05 XMR in Transaction 2A, this can lead to the same circumstantial likelihood analysis that we were trying to avoid in the ﬁrst place. Outputs may be associated as being spent from a common transaction, or from a common tree of transactions, or spent from a common block, or spent from within a short span of time. This list may not be comprehensive. Of course, these scenarios are harder and harder for Eve to analyze compared to the simple scenario initially presented in Section 3.2.

Two outputs from disjoint transactions with two di.erent block heights are less likely to come from a common owner than two outputs from a common transaction, or than two outputs from two transactions linked by a single step, or than two outputs from a similar block height. Hence, since Bob can also use any other output he owns in order to send the remaining 0.05 XMR, not just his change output, he should endeavor to do so if possible. This leads us to the natural solution, which is simple to implement in its basic form: use one and only one input for every transaction, and output one and only one output to the recipient. This is repeated until all desired funds are transferred to the recipient.

This is, of course, an ine.cient method of generating transaction outputs. To increase e.ciency, multiple inputs may also be used to generate a transaction if and only if the outputs referenced do not lie within a few blocks of each other.

5 Roadmap to our Recommendations

Implementing a network-wide minimum mix-in policy n =2 to the Monero archi.tecture can be done in several ways.

(1) Within a wallet, users may be required to send via wallet-to-wallet RPC with a minimum mix-in, unless those transactions lie in transaction output sets with.out enough mix-ins.
(2) Miners may be requested or required to only include mixed transactions satis.fying the minimum mix-in, unless those transactions lie in transaction output sets without enough mix-ins.
(3) The peer-to-peer protocol may be modiﬁed such that transactions that do not satisfy the minimum mix-in for all their ring signatures are not relayed, unless those transactions lie in transaction output sets without enough mix-ins.

Furthermore, if a transaction is included in a block with height greater than a certain critical value, n =2 shall not be su.cient, but in fact we will require n =4. We recommend this block height to correspond to about two to three years after publication of this document, but the speciﬁcs are unimportant. Our roadmap to implement a network-wide minimum mix-in policy is to implement (1) soon, skip (2), and plan (3) for a later date after vigorous testing on our testnet. The primary obstacletoimmediateimplementationistransaction dust.Toovercometheobstacle, we deﬁne a legal transaction output form as any number A . 10B where A is an integer 1 , A , 9 and B is any integer .12 , B. Hence, a transaction output with amount 0.00111111 XMR does not follow the legal transaction output form, but the transaction outputs 0.001, 0.0001, 0.00001, 0.000001, 0.0000001, and 0.00000001 XMR are each legal. We propose the following policy:

(i) Within a wallet, users may be required to produce only transaction outputs adhering to the legal transaction output form.
(ii) Miners may be requested or required to only include transactions with all outputs satisfying the legal transaction output form.
(iii) The peer-to-peer protocol may be modiﬁed such that transactions that do not satisfy the legal transaction output form for all of their transaction outputs are not relayed.

Implementing these will allow a user to use a non-legal transaction output as a transaction input, so long as all outputs from that transaction adhere to this strict form. No new dust will be created in the system, and dust will only be removed from the system over time. Again, our roadmap to implement a legal transaction output form is to attack (i) ﬁrst, skip (ii), and plan (iii) for a later date after vigorous testing.

In order to encourage users to re-send transactions to themselves iteratively over intervalsoftimewithrandomlength,werecommendthat,withinawallet,awarning be displayed if a random but minimum number of blocks have been added to the blockchain since an output has been received in the wallet. After 6 to 8 weeks have transpired, for example, all outputs in a wallet should be re-sent.

Furthermore, wallet software should specify that

(a) one and only one transaction output should be sent to a recipient at a time;
(b) a transaction that must consist of several outputs should be broken up into several smaller transactions spread over time as in a torrent; and
(c) each transaction output received by a recipient should use transaction inputs with dissimilar block heights and dissimilar root transactions.

6 Auditability for Compliance Purposes

Businesses using ring signatures and stealth addressing through Monero will have the reasonable desire to comply with the corresponding local laws within their jurisdictions. Typically, these laws require the business to prove ownership of their own funds, and demonstrate where funds may have been sent. To prove ownership of any unspent funds on the block chain, a Monero account holder may publish a zero mix-in ring signature of their public key. Please be aware that a zero-mix-in ring signature is just an ordinary digital signature. An auditor can then scan the block chain for the presence of the output public key, and verify that the key image produced from the zero mix-in ring signature has not yet been used on the block chain. To protect the privacy of the business and its customers, the auditor should then destroy the signature and the key image.

7 Conclusion

We identified several routes an attacker could use to execute a blockchain analysis to degrade the untraceability of the CryptoNote 2.0 protocol. The zero mix spending problem identiﬁed in [3], the age-based association problem, the association-byusage problem, and a more generalized combinatorial analysis problem are each identiﬁed. We recommend a network-wide minimum mix-in of n =2 to address zero mix chain reaction problems, and we discuss how a mix-in of n =4 is likely to be larger than necessary for most practical purposes. We recommend a non.uniform transaction output selection method for ring generation to mitigate age-based association problems. We recommend partitioning all transactions into many smaller sub-transactions such that they occur over an interval of time, as in a torrent, and such that each sub-transaction has one and only one output, in order to mitigate the combinatorial analysis problem.

References

1. Norman Lloyd Johnson and Samuel Kotz. Urn models and their application: an approach to modern discrete probability theory. Wiley New York, 1977.
2. Ian Miers, Christina Garman, Matthew Green, and Aviel D Rubin. Zerocoin: Anonymous distributed e-cash from bitcoin. In Security and Privacy (SP), 2013 IEEE Symposium on, pages 397–411. IEEE, 2013.
3. Surae Noether, Sarang Noether, and Adam Mackenzie. Mrl-0001: A note on chain reactions in traceability in cryptonote 2.0. Technical report.
4. Ronald L Rivest, Adi Shamir, and Yael Tauman. How to leak a secret. In Advances in Cryptology-ASIACRYPT 2001, pages 552–565. Springer, 2001.
5. Tacotime. Saturation of unspent transaction outputs, September 2014.

Git diff from commit e940386f9a8765423ab3dd9e3aabe19a68cba9f9 (thankful_for_today's last commit) to current HEAD, excluding changes in /external so as to exclude libraries and submoduled code, and excluding contrib/ so as to exclude changes made to epee, excluding most comments because apparently nobody that writes code for BCN knows what a comment is anyway, excluding removed lines, excluding empty lines

You can use this line to generate the diff: git diff e940386f9a8765423ab3dd9e3aabe19a68cba9f9 HEAD --name-only | grep -v '^external/' | grep -v '^contrib/' | xargs git diff e940386f9a8765423ab3dd9e3aabe19a68cba9f9 HEAD -- | grep -v "^#" | grep -v "^+#" | grep -v "^\/\/" | grep -v "^+\/\/" | grep -v "^-" | grep -v "^[[:space:]]*$" | grep -v "^+[[:space:]]*$"

diff --git a/.gitattributes b/.gitattributes index 6afd357..f75097e 100644 +++ b/.gitattributes @@ -1,2 +1,2 @@ .git* export-ignore \ No newline at end of file +version.cmake export-subst diff --git a/.gitignore b/.gitignore index 2a68f64..18653c2 100644 +++ b/.gitignore @@ -1,2 +1,104 @@ +.DS_Store +/doc /build \ No newline at end of file +/tags +log/ +*.swp +*.swo +TAGS +!TAGS/ +tags +!tags/ +gtags.files +GTAGS +GRTAGS +GPATH +cscope.files +cscope.out +cscope.in.out +cscope.po.out +external/miniupnpc/Makefile +miniupnpcstrings.h +version/ +*.slo +*.lo +*.o +*.obj +*.gch +*.pch +*.so +*.dylib +*.dll +*.mod +*.lai +*.la +*.a +*.lib +*.exe +*.out +*.app +CMakeCache.txt +CMakeFiles +cmake_install.cmake +install_manifest.txt +*.cmake +*~ +.directory +*.pydevproject +.metadata +.gradle +bin/ +tmp/ +*.tmp +*.bak +*.swp +*~.nib +local.properties +.settings/ +.loadpath +.externalToolBuilders/ +*.launch +.cproject +.buildpath +.target +.texlipse diff --git a/CMakeLists.txt b/CMakeLists.txt index 6b293cc..f9e2160 100644 +++ b/CMakeLists.txt @@ -1,15 +1,263 @@ +cmake_minimum_required(VERSION 2.8.7) +project(bitmonero) +function (die msg) + if (NOT WIN32) + string(ASCII 27 Esc) + set(ColourReset "${Esc}[m") + set(BoldRed "${Esc}[1;31m") + else () + set(ColourReset "") + set(BoldRed "") + endif () + message(FATAL_ERROR "${BoldRed}${msg}${ColourReset}") +endfunction () +if (NOT ${ARCH} STREQUAL "") + string(SUBSTRING ${ARCH} 0 3 IS_ARM) + string(TOLOWER ${IS_ARM} IS_ARM) + if (${IS_ARM} STREQUAL "arm") + string(SUBSTRING ${ARCH} 0 5 ARM_TEST) + string(TOLOWER ${ARM_TEST} ARM_TEST) + if (${ARM_TEST} STREQUAL "armv6") + set(ARM6 1) + else() + set(ARM6 0) + endif() + if (${ARM_TEST} STREQUAL "armv7") + set(ARM7 1) + else() + set(ARM7 0) + endif() + endif() +endif() +if(WIN32 OR ARM7 OR ARM6) + set(CMAKE_C_FLAGS_RELEASE "-O2 -DNDEBUG") + set(CMAKE_CXX_FLAGS_RELEASE "-O2 -DNDEBUG") +endif() +set(PER_BLOCK_CHECKPOINT 1) +if(PER_BLOCK_CHECKPOINT) + add_definitions("-DPER_BLOCK_CHECKPOINT") +endif() +list(INSERT CMAKE_MODULE_PATH 0 + "${CMAKE_SOURCE_DIR}/cmake") +if (NOT DEFINED ENV{DEVELOPER_LOCAL_TOOLS}) + message(STATUS "Could not find DEVELOPER_LOCAL_TOOLS in env (not required)") + set(BOOST_IGNORE_SYSTEM_PATHS_DEFAULT OFF) +elseif ("$ENV{DEVELOPER_LOCAL_TOOLS}" EQUAL 1) + message(STATUS "Found: env DEVELOPER_LOCAL_TOOLS = 1") + set(BOOST_IGNORE_SYSTEM_PATHS_DEFAULT ON) +else() + message(STATUS "Found: env DEVELOPER_LOCAL_TOOLS = 0") + set(BOOST_IGNORE_SYSTEM_PATHS_DEFAULT OFF) +endif() +message(STATUS "BOOST_IGNORE_SYSTEM_PATHS defaults to ${BOOST_IGNORE_SYSTEM_PATHS_DEFAULT}") +option(BOOST_IGNORE_SYSTEM_PATHS "Ignore boost system paths for local boost installation" ${BOOST_IGNORE_SYSTEM_PATHS_DEFAULT}) +if (NOT DEFINED ENV{DEVELOPER_LIBUNBOUND_OLD}) + message(STATUS "Could not find DEVELOPER_LIBUNBOUND_OLD in env (not required)") +elseif ("$ENV{DEVELOPER_LIBUNBOUND_OLD}" EQUAL 1) + message(STATUS "Found: env DEVELOPER_LIBUNBOUND_OLD = 1, will use the work around") + add_definitions(-DDEVELOPER_LIBUNBOUND_OLD) +elseif ("$ENV{DEVELOPER_LIBUNBOUND_OLD}" EQUAL 0) + message(STATUS "Found: env DEVELOPER_LIBUNBOUND_OLD = 0") +else() + message(STATUS "Found: env DEVELOPER_LIBUNBOUND_OLD with bad value. Will NOT use the work around") +endif() set_property(GLOBAL PROPERTY USE_FOLDERS ON) enable_testing() +option(BUILD_DOCUMENTATION "Build the Doxygen documentation." ON) +if(CMAKE_SIZEOF_VOID_P EQUAL "8") + set(DEFAULT_BUILD_64 ON) +else() + set(DEFAULT_BUILD_64 OFF) +endif() +option(BUILD_64 "Build for 64-bit? 'OFF' builds for 32-bit." ${DEFAULT_BUILD_64}) +if(BUILD_64) + set(ARCH_WIDTH "64") +else() + set(ARCH_WIDTH "32") +endif() +message(STATUS "Building for a ${ARCH_WIDTH}-bit system") +if(CMAKE_SYSTEM_NAME MATCHES "kFreeBSD.*") + set(FREEBSD TRUE) +elseif(CMAKE_SYSTEM_NAME MATCHES "DragonFly.*|FreeBSD") + set(FREEBSD TRUE) +endif() include_directories(src contrib/epee/include external "${CMAKE_BINARY_DIR}/version") +if(APPLE) + include_directories(SYSTEM /usr/include/malloc) +endif() +if(MSVC OR MINGW) + set(DEFAULT_STATIC true) +else() + set(DEFAULT_STATIC false) +endif() +option(STATIC "Link libraries statically" ${DEFAULT_STATIC}) +if(MINGW) + string(REGEX MATCH "^[^/]:/[^/]*" msys2_install_path "${CMAKE_C_COMPILER}") + message(STATUS "MSYS location: ${msys2_install_path}") + set(CMAKE_INCLUDE_PATH "${msys2_install_path}/mingw${ARCH_WIDTH}/include") + # This is necessary because otherwise CMake will make Boost libraries -lfoo + # rather than a full path. Unfortunately, this makes the shared libraries get + # linked due to a bug in CMake which misses putting -static flags around the + # -lfoo arguments. + set(DEFLIB ${msys2_install_path}/mingw${ARCH_WIDTH}/lib) + list(REMOVE_ITEM CMAKE_C_IMPLICIT_LINK_DIRECTORIES ${DEFLIB}) + list(REMOVE_ITEM CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES ${DEFLIB}) +endif() +if(STATIC) + if(MSVC) + set(CMAKE_FIND_LIBRARY_SUFFIXES .lib .dll.a .a ${CMAKE_FIND_LIBRARY_SUFFIXES}) + else() + set(CMAKE_FIND_LIBRARY_SUFFIXES .a ${CMAKE_FIND_LIBRARY_SUFFIXES}) + endif() +endif() +set(DATABASE lmdb) +if (DEFINED ENV{DATABASE}) + set(DATABASE $ENV{DATABASE}) + message(STATUS "DATABASE set: ${DATABASE}") +else() + message(STATUS "Could not find DATABASE in env (not required unless you want to change database type from default: ${DATABASE})") +endif() +set(BERKELEY_DB 0) +if (DATABASE STREQUAL "lmdb") + set(BLOCKCHAIN_DB DB_LMDB) + # temporarily allow mingw to compile with berkeley_db, + # regardless if building static or not + if(NOT STATIC OR MINGW) + find_package(BerkeleyDB) + if(NOT BERKELEY_DB_LIBRARIES) + message(STATUS "BerkeleyDB not found and has been disabled.") + else() + message(STATUS "Found BerkeleyDB include (db.h) in ${BERKELEY_DB_INCLUDE_DIR}") + if(BERKELEY_DB_LIBRARIES) + message(STATUS "Found BerkeleyDB shared library") + set(BDB_STATIC false CACHE BOOL "BDB Static flag") + set(BDB_INCLUDE ${BERKELEY_DB_INCLUDE_DIR} CACHE STRING "BDB include path") + set(BDB_LIBRARY ${BERKELEY_DB_LIBRARIES} CACHE STRING "BDB library name") + set(BDB_LIBRARY_DIRS "" CACHE STRING "BDB Library dirs") + set(BERKELEY_DB 1) + else() + message(STATUS "Found BerkeleyDB includes, but could not find BerkeleyDB library. Please make sure you have installed libdb and libdb-dev or the equivalent") + endif() + endif() + endif() +elseif (DATABASE STREQUAL "memory") + set(BLOCKCHAIN_DB DB_MEMORY) +else() + die("Invalid database type: ${DATABASE}") +endif() +if(BERKELEY_DB) + add_definitions("-DBERKELEY_DB") +endif() +add_definitions("-DBLOCKCHAIN_DB=${BLOCKCHAIN_DB}") +if (UNIX AND NOT APPLE) + # Note that at the time of this writing the -Wstrict-prototypes flag added below will make this fail + set(THREADS_PREFER_PTHREAD_FLAG ON) + find_package(Threads) +endif() +add_subdirectory(external) +if(UPNP_STATIC) + add_definitions("-DUPNP_STATIC") +else() + add_definitions("-DUPNP_DYNAMIC") + include_directories(${UPNP_INCLUDE}) +endif() +include_directories(${UNBOUND_INCLUDE}) +link_directories(${UNBOUND_LIBRARY_DIRS}) +include_directories(external/rapidjson) +include_directories(${LMDB_INCLUDE}) +if (BERKELEY_DB) + include_directories(${BDB_INCLUDE}) +endif() if(MSVC) add_definitions("/bigobj /MP /W3 /GS- /D_CRT_SECURE_NO_WARNINGS /wd4996 /wd4345 /D_WIN32_WINNT=0x0600 /DWIN32_LEAN_AND_MEAN /DGTEST_HAS_TR1_TUPLE=0 /FIinline_c.h /D__SSE4_1__") @@ -23,93 +271,179 @@ if(MSVC) include_directories(SYSTEM src/platform/msc) else() set(ARCH native CACHE STRING "CPU to build for: -march value or default") + # -march=armv7-a conflicts with -mcpu=cortex-a7 + if(ARCH STREQUAL "default" OR ARM7) set(ARCH_FLAG "") else() + if(ARCH STREQUAL "x86_64") + set(ARCH_FLAG "-march=x86-64") + else() + set(ARCH_FLAG "-march=${ARCH}") + endif() + endif() + set(WARNINGS "-Wall -Wextra -Wpointer-arith -Wundef -Wvla -Wwrite-strings -Wno-error=extra -Wno-error=deprecated-declarations -Wno-error=sign-compare -Wno-error=strict-aliasing -Wno-error=type-limits -Wno-unused-parameter -Wno-error=unused-variable -Wno-error=undef -Wno-error=uninitialized") + if(CMAKE_C_COMPILER_ID STREQUAL "Clang") + set(WARNINGS "${WARNINGS} -Wno-deprecated-register") + endif() + if(NOT MINGW) + set(WARNINGS "${WARNINGS} -Werror") # to allow pedantic but not stop compilation endif() if(CMAKE_C_COMPILER_ID STREQUAL "Clang") set(WARNINGS "${WARNINGS} -Wno-error=mismatched-tags -Wno-error=null-conversion -Wno-overloaded-shift-op-parentheses -Wno-error=shift-count-overflow -Wno-error=tautological-constant-out-of-range-compare -Wno-error=unused-private-field -Wno-error=unneeded-internal-declaration") else() set(WARNINGS "${WARNINGS} -Wlogical-op -Wno-error=maybe-uninitialized") endif() if(MINGW) + set(WARNINGS "${WARNINGS} -Wno-error=unused-value -Wno-error=unused-but-set-variable") + set(MINGW_FLAG "${MINGW_FLAG} -DWIN32_LEAN_AND_MEAN") + set(Boost_THREADAPI win32) include_directories(SYSTEM src/platform/mingw) + # mingw doesn't support LTO (multiple definition errors at link time) + set(USE_LTO_DEFAULT false) endif() set(C_WARNINGS "-Waggregate-return -Wnested-externs -Wold-style-definition -Wstrict-prototypes") set(CXX_WARNINGS "-Wno-reorder -Wno-missing-field-initializers") + try_compile(STATIC_ASSERT_RES "${CMAKE_CURRENT_BINARY_DIR}/static-assert" "${CMAKE_CURRENT_SOURCE_DIR}/cmake/test-static-assert.c" COMPILE_DEFINITIONS "-std=c11") if(STATIC_ASSERT_RES) set(STATIC_ASSERT_FLAG "") else() set(STATIC_ASSERT_FLAG "-Dstatic_assert=_Static_assert") endif() + option(NO_AES "Explicitly disable AES support" ${NO_AES}) + if (NO_AES) + message(STATUS "Disabling AES support") + set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c11 -D_GNU_SOURCE ${MINGW_FLAG} ${STATIC_ASSERT_FLAG} ${WARNINGS} ${C_WARNINGS} ${ARCH_FLAG}") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -D_GNU_SOURCE ${MINGW_FLAG} ${WARNINGS} ${CXX_WARNINGS} ${ARCH_FLAG}") + else() + message(STATUS "Enabling AES support") + set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c11 -D_GNU_SOURCE ${MINGW_FLAG} ${STATIC_ASSERT_FLAG} ${WARNINGS} ${C_WARNINGS} ${ARCH_FLAG} -maes") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -D_GNU_SOURCE ${MINGW_FLAG} ${WARNINGS} ${CXX_WARNINGS} ${ARCH_FLAG} -maes") + endif() + if(ARM6) + message(STATUS "Setting ARM6 C and C++ flags") + set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mfpu=vfp -mfloat-abi=hard") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mfpu=vfp -mfloat-abi=hard") + endif() + if(ARM7) + message(STATUS "Setting ARM7 C and C++ flags") + set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O2 -mcpu=cortex-a7 -mfloat-abi=hard -mfpu=vfpv4 -funsafe-math-optimizations -mtune=cortex-a7") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O2 -mcpu=cortex-a7 -mfloat-abi=hard -mfpu=vfpv4 -funsafe-math-optimizations -mtune=cortex-a7") + endif() + if(APPLE) + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DGTEST_HAS_TR1_TUPLE=0") + endif() if(CMAKE_C_COMPILER_ID STREQUAL "GNU" AND NOT (CMAKE_C_COMPILER_VERSION VERSION_LESS 4.8)) set(DEBUG_FLAGS "-g3 -Og") else() set(DEBUG_FLAGS "-g3 -O0") endif() + set(RELEASE_FLAGS "-Ofast -DNDEBUG -Wno-unused-variable") + if(NOT DEFINED USE_LTO_DEFAULT) + set(USE_LTO_DEFAULT true) + endif() + set(USE_LTO ${USE_LTO_DEFAULT} CACHE BOOL "Use Link-Time Optimization (Release mode only)") + if(CMAKE_CXX_COMPILER_ID STREQUAL "Clang") + # There is a clang bug that does not allow to compile code that uses AES-NI intrinsics if -flto is enabled, so explicitly disable + set(USE_LTO false) + # explicitly define stdlib for older versions of clang + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -stdlib=libc++") + set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -stdlib=libc++") + endif() + if(USE_LTO) + set(RELEASE_FLAGS "${RELEASE_FLAGS} -flto") + if(STATIC) + set(RELEASE_FLAGS "${RELEASE_FLAGS} -ffat-lto-objects") + endif() + # Since gcc 4.9 the LTO format is non-standard (slim), so we need the gcc-specific ar and ranlib binaries + if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND NOT (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 4.9.0)) + set(CMAKE_AR "gcc-ar") + set(CMAKE_RANLIB "gcc-ranlib") + endif() + endif() set(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} ${DEBUG_FLAGS}") set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} ${DEBUG_FLAGS}") set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE} ${RELEASE_FLAGS}") set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} ${RELEASE_FLAGS}") + if(STATIC AND NOT APPLE AND NOT FREEBSD) set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static-libgcc -static-libstdc++") endif() endif() +if (${BOOST_IGNORE_SYSTEM_PATHS} STREQUAL "ON") + set(Boost_NO_SYSTEM_PATHS TRUE) +endif() +set(OLD_LIB_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES}) if(STATIC) + if(MINGW) + set(CMAKE_FIND_LIBRARY_SUFFIXES .a) + endif() set(Boost_USE_STATIC_LIBS ON) set(Boost_USE_STATIC_RUNTIME ON) endif() +find_package(Boost 1.53 QUIET REQUIRED COMPONENTS system filesystem thread date_time chrono regex serialization program_options) +set(CMAKE_FIND_LIBRARY_SUFFIXES ${OLD_LIB_SUFFIXES}) +if(NOT Boost_FOUND) + die("Could not find Boost libraries, please make sure you have installed Boost or libboost-all-dev (1.53 or 1.55+) or the equivalent") +endif() +if((Boost_MAJOR_VERSION EQUAL 1) AND (Boost_MINOR_VERSION EQUAL 54)) + die("Boost version 1.54 is unsupported due to a bug (see: http://goo.gl/RrCFmA), please install Boost 1.53 or 1.55 and above") endif() include_directories(SYSTEM ${Boost_INCLUDE_DIRS}) if(MINGW) + set(EXTRA_LIBRARIES mswsock;ws2_32;iphlpapi) +elseif(APPLE OR FREEBSD) + set(EXTRA_LIBRARIES "") elseif(NOT MSVC) + find_library(RT rt) + set(EXTRA_LIBRARIES ${RT} ${DL}) endif() +include(version.cmake) +add_subdirectory(contrib) add_subdirectory(src) +if(BUILD_TESTS) + add_subdirectory(tests) +endif() +if(BUILD_DOCUMENTATION) + set(DOC_GRAPHS "YES" CACHE STRING "Create dependency graphs (needs graphviz)") + set(DOC_FULLGRAPHS "NO" CACHE STRING "Create call/callee graphs (large)") + find_program(DOT_PATH dot) + if (DOT_PATH STREQUAL "DOT_PATH-NOTFOUND") + message("Doxygen: graphviz not found - graphs disabled") + set(DOC_GRAPHS "NO") + endif() + find_package(Doxygen) + if(DOXYGEN_FOUND) + configure_file("cmake/Doxyfile.in" "Doxyfile" @ONLY) + configure_file("cmake/Doxygen.extra.css.in" "Doxygen.extra.css" @ONLY) + add_custom_target(doc + ${DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} + COMMENT "Generating API documentation with Doxygen.." VERBATIM) + endif() +endif() diff --git a/Doxyfile b/Doxyfile new file mode 100644 index 0000000..a70ef81 +++ b/Doxyfile @@ -0,0 +1,2310 @@ +DOXYFILE_ENCODING = UTF-8 +PROJECT_NAME = "Bitmonero" +PROJECT_NUMBER = +PROJECT_BRIEF = +PROJECT_LOGO = +OUTPUT_DIRECTORY = doc +CREATE_SUBDIRS = NO +ALLOW_UNICODE_NAMES = NO +OUTPUT_LANGUAGE = English +BRIEF_MEMBER_DESC = YES +REPEAT_BRIEF = YES +ABBREVIATE_BRIEF = +ALWAYS_DETAILED_SEC = NO +INLINE_INHERITED_MEMB = NO +FULL_PATH_NAMES = YES +STRIP_FROM_PATH = +STRIP_FROM_INC_PATH = +SHORT_NAMES = NO +JAVADOC_AUTOBRIEF = NO +QT_AUTOBRIEF = NO +MULTILINE_CPP_IS_BRIEF = NO +INHERIT_DOCS = YES +SEPARATE_MEMBER_PAGES = NO +TAB_SIZE = 4 +ALIASES = +TCL_SUBST = +OPTIMIZE_OUTPUT_FOR_C = NO +OPTIMIZE_OUTPUT_JAVA = NO +OPTIMIZE_FOR_FORTRAN = NO +OPTIMIZE_OUTPUT_VHDL = NO +EXTENSION_MAPPING = +MARKDOWN_SUPPORT = YES +AUTOLINK_SUPPORT = YES +BUILTIN_STL_SUPPORT = NO +CPP_CLI_SUPPORT = NO +SIP_SUPPORT = NO +IDL_PROPERTY_SUPPORT = YES +DISTRIBUTE_GROUP_DOC = NO +SUBGROUPING = YES +INLINE_GROUPED_CLASSES = NO +INLINE_SIMPLE_STRUCTS = NO +TYPEDEF_HIDES_STRUCT = NO +LOOKUP_CACHE_SIZE = 0 +EXTRACT_ALL = NO +EXTRACT_PRIVATE = NO +EXTRACT_PACKAGE = NO +EXTRACT_STATIC = NO +EXTRACT_LOCAL_CLASSES = YES +EXTRACT_LOCAL_METHODS = NO +EXTRACT_ANON_NSPACES = NO +HIDE_UNDOC_MEMBERS = NO +HIDE_UNDOC_CLASSES = NO +HIDE_FRIEND_COMPOUNDS = NO +HIDE_IN_BODY_DOCS = NO +INTERNAL_DOCS = NO +CASE_SENSE_NAMES = YES +HIDE_SCOPE_NAMES = NO +SHOW_INCLUDE_FILES = YES +SHOW_GROUPED_MEMB_INC = NO +FORCE_LOCAL_INCLUDES = NO +INLINE_INFO = YES +SORT_MEMBER_DOCS = YES +SORT_BRIEF_DOCS = NO +SORT_MEMBERS_CTORS_1ST = NO +SORT_GROUP_NAMES = NO +SORT_BY_SCOPE_NAME = NO +STRICT_PROTO_MATCHING = NO +GENERATE_TODOLIST = YES +GENERATE_TESTLIST = YES +GENERATE_BUGLIST = YES +GENERATE_DEPRECATEDLIST= YES +ENABLED_SECTIONS = +MAX_INITIALIZER_LINES = 30 +SHOW_USED_FILES = YES +SHOW_FILES = YES +SHOW_NAMESPACES = YES +FILE_VERSION_FILTER = +LAYOUT_FILE = +CITE_BIB_FILES = +QUIET = NO +WARNINGS = YES +WARN_IF_UNDOCUMENTED = YES +WARN_IF_DOC_ERROR = YES +WARN_NO_PARAMDOC = NO +WARN_FORMAT = "$file:$line: $text" +WARN_LOGFILE = +INPUT = src +INPUT_ENCODING = UTF-8 +FILE_PATTERNS = +RECURSIVE = YES +EXCLUDE = +EXCLUDE_SYMLINKS = NO +EXCLUDE_PATTERNS = +EXCLUDE_SYMBOLS = +EXAMPLE_PATH = +EXAMPLE_PATTERNS = +EXAMPLE_RECURSIVE = NO +IMAGE_PATH = +INPUT_FILTER = +FILTER_PATTERNS = +FILTER_SOURCE_FILES = NO +FILTER_SOURCE_PATTERNS = +USE_MDFILE_AS_MAINPAGE = +SOURCE_BROWSER = NO +INLINE_SOURCES = NO +STRIP_CODE_COMMENTS = YES +REFERENCED_BY_RELATION = NO +REFERENCES_RELATION = NO +REFERENCES_LINK_SOURCE = YES +SOURCE_TOOLTIPS = YES +USE_HTAGS = NO +VERBATIM_HEADERS = YES +ALPHABETICAL_INDEX = YES +COLS_IN_ALPHA_INDEX = 5 +IGNORE_PREFIX = +GENERATE_HTML = YES +HTML_OUTPUT = html +HTML_FILE_EXTENSION = .html +HTML_HEADER = +HTML_FOOTER = +HTML_STYLESHEET = +HTML_EXTRA_STYLESHEET = +HTML_EXTRA_FILES = +HTML_COLORSTYLE_HUE = 220 +HTML_COLORSTYLE_SAT = 100 +HTML_COLORSTYLE_GAMMA = 80 +HTML_TIMESTAMP = YES +HTML_DYNAMIC_SECTIONS = NO +HTML_INDEX_NUM_ENTRIES = 100 +GENERATE_DOCSET = NO +DOCSET_FEEDNAME = "Doxygen generated docs" +DOCSET_BUNDLE_ID = org.doxygen.Project +DOCSET_PUBLISHER_ID = org.doxygen.Publisher +DOCSET_PUBLISHER_NAME = Publisher +GENERATE_HTMLHELP = NO +CHM_FILE = +HHC_LOCATION = +GENERATE_CHI = NO +CHM_INDEX_ENCODING = +BINARY_TOC = NO +TOC_EXPAND = NO +GENERATE_QHP = NO +QCH_FILE = +QHP_NAMESPACE = org.doxygen.Project +QHP_VIRTUAL_FOLDER = doc +QHP_CUST_FILTER_NAME = +QHP_CUST_FILTER_ATTRS = +QHP_SECT_FILTER_ATTRS = +QHG_LOCATION = +GENERATE_ECLIPSEHELP = NO +ECLIPSE_DOC_ID = org.doxygen.Project +DISABLE_INDEX = NO +GENERATE_TREEVIEW = NO +ENUM_VALUES_PER_LINE = 4 +TREEVIEW_WIDTH = 250 +EXT_LINKS_IN_WINDOW = NO +FORMULA_FONTSIZE = 10 +FORMULA_TRANSPARENT = YES +USE_MATHJAX = NO +MATHJAX_FORMAT = HTML-CSS +MATHJAX_RELPATH = http://cdn.mathjax.org/mathjax/latest +MATHJAX_EXTENSIONS = +MATHJAX_CODEFILE = +SEARCHENGINE = YES +SERVER_BASED_SEARCH = NO +EXTERNAL_SEARCH = NO +SEARCHENGINE_URL = +SEARCHDATA_FILE = searchdata.xml +EXTERNAL_SEARCH_ID = +EXTRA_SEARCH_MAPPINGS = +GENERATE_LATEX = YES +LATEX_OUTPUT = latex +LATEX_CMD_NAME = latex +MAKEINDEX_CMD_NAME = makeindex +COMPACT_LATEX = NO +PAPER_TYPE = a4 +EXTRA_PACKAGES = +LATEX_HEADER = +LATEX_FOOTER = +LATEX_EXTRA_FILES = +PDF_HYPERLINKS = YES +USE_PDFLATEX = YES +LATEX_BATCHMODE = NO +LATEX_HIDE_INDICES = NO +LATEX_SOURCE_CODE = NO +LATEX_BIB_STYLE = plain +GENERATE_RTF = NO +RTF_OUTPUT = rtf +COMPACT_RTF = NO +RTF_HYPERLINKS = NO +RTF_STYLESHEET_FILE = +RTF_EXTENSIONS_FILE = +GENERATE_MAN = NO +MAN_OUTPUT = man +MAN_EXTENSION = .3 +MAN_SUBDIR = +MAN_LINKS = NO +GENERATE_XML = NO +XML_OUTPUT = xml +XML_PROGRAMLISTING = YES +GENERATE_DOCBOOK = NO +DOCBOOK_OUTPUT = docbook +GENERATE_AUTOGEN_DEF = NO +GENERATE_PERLMOD = NO +PERLMOD_LATEX = NO +PERLMOD_PRETTY = YES +PERLMOD_MAKEVAR_PREFIX = +ENABLE_PREPROCESSING = YES +MACRO_EXPANSION = NO +EXPAND_ONLY_PREDEF = NO +SEARCH_INCLUDES = YES +INCLUDE_PATH = +INCLUDE_FILE_PATTERNS = +PREDEFINED = +EXPAND_AS_DEFINED = +SKIP_FUNCTION_MACROS = YES +TAGFILES = +GENERATE_TAGFILE = +ALLEXTERNALS = NO +EXTERNAL_GROUPS = YES +EXTERNAL_PAGES = YES +PERL_PATH = /usr/bin/perl +CLASS_DIAGRAMS = YES +MSCGEN_PATH = +DIA_PATH = +HIDE_UNDOC_RELATIONS = YES +HAVE_DOT = $(HAVE_DOT) +DOT_NUM_THREADS = 0 +DOT_FONTNAME = Helvetica +DOT_FONTSIZE = 10 +DOT_FONTPATH = +CLASS_GRAPH = YES +COLLABORATION_GRAPH = YES +GROUP_GRAPHS = YES +UML_LOOK = NO +UML_LIMIT_NUM_FIELDS = 10 +TEMPLATE_RELATIONS = NO +INCLUDE_GRAPH = YES +INCLUDED_BY_GRAPH = YES +CALL_GRAPH = NO +CALLER_GRAPH = NO +GRAPHICAL_HIERARCHY = YES +DIRECTORY_GRAPH = YES +DOT_IMAGE_FORMAT = png +INTERACTIVE_SVG = NO +DOT_PATH = +DOTFILE_DIRS = +MSCFILE_DIRS = +DIAFILE_DIRS = +DOT_GRAPH_MAX_NODES = 50 +MAX_DOT_GRAPH_DEPTH = 0 +DOT_TRANSPARENT = NO +DOT_MULTI_TARGETS = NO +GENERATE_LEGEND = YES +DOT_CLEANUP = YES diff --git a/Makefile b/Makefile index 451319b..dda8663 100644 +++ b/Makefile @@ -1,33 +1,96 @@ +all: release-all cmake-debug: mkdir -p build/debug cd build/debug && cmake -D CMAKE_BUILD_TYPE=Debug ../.. +debug: cmake-debug cd build/debug && $(MAKE) +debug-test: debug + mkdir -p build/debug + cd build/debug && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=Debug ../.. && $(MAKE) test +debug-all: + mkdir -p build/debug + cd build/debug && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=Debug ../.. && $(MAKE) cmake-release: mkdir -p build/release cd build/release && cmake -D CMAKE_BUILD_TYPE=Release ../.. +release: cmake-release cd build/release && $(MAKE) +release-test: release + mkdir -p build/release + cd build/release && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) test +release-all: + mkdir -p build/release + cd build/release && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-arm6: + mkdir -p build/release + cd build/release && cmake -D BUILD_TESTS=OFF -D ARCH="armv6zk" -D BUILD_64=OFF -D NO_AES=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-arm7: + mkdir -p build/release + cd build/release && cmake -D BUILD_TESTS=OFF -D ARCH="armv7-a" -D BUILD_64=OFF -D NO_AES=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-static: release-static-64 +release-static-64: + mkdir -p build/release + cd build/release && cmake -D STATIC=ON -D ARCH="x86-64" -D BUILD_64=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-static-32: + mkdir -p build/release + cd build/release && cmake -D STATIC=ON -D ARCH="i686" -D BUILD_64=OFF -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-static-win64: + mkdir -p build/release + cd build/release && cmake -G "MSYS Makefiles" -D STATIC=ON -D ARCH="x86-64" -D BUILD_64=ON -D CMAKE_BUILD_TYPE=Release -D CMAKE_TOOLCHAIN_FILE=../../cmake/64-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys64 ../.. && $(MAKE) +release-static-win32: + mkdir -p build/release + cd build/release && cmake -G "MSYS Makefiles" -D STATIC=ON -D ARCH="i686" -D BUILD_64=OFF -D CMAKE_BUILD_TYPE=Release -D CMAKE_TOOLCHAIN_FILE=../../cmake/32-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys32 ../.. && $(MAKE) clean: + @echo "WARNING: Back-up your wallet if it exists within ./build!" ; \ + read -r -p "This will destroy the build directory, continue (y/N)?: " CONTINUE; \ + [ $$CONTINUE = "y" ] || [ $$CONTINUE = "Y" ] || (echo "Exiting."; exit 1;) rm -rf build tags: ctags -R --sort=1 --c++-kinds=+p --fields=+iaS --extra=+q --language-force=C++ src contrib tests/gtest +.PHONY: all cmake-debug debug debug-test debug-all cmake-release release release-test release-all clean tags diff --git a/README b/README deleted file mode 100644 index f63a4cc..0000000 +++ /dev/null @@ -1,32 +0,0 @@ diff --git a/README.i18n b/README.i18n new file mode 100644 index 0000000..2b913f7 +++ b/README.i18n @@ -0,0 +1,45 @@ +The Monero command line tools can be translated in various languages. +In order to use the same translation workflow as the future GUI, they +use Qt Linguist translation files. However, to avoid the dependencies +on Qt this normally implies, they use a custom loader to read those +files at runtime. In order to update, or build translations files, you +do need to have Qt tools installed, however. For translating, you need +either the Qt Linguist GUI, or another tool that supports Qt ts files, +such as Transifex. To run, you do not need anything Qt. +To update ts files after changing source code: + ./utils/translations/update-translations.sh +To add a new language, eg Spanish (ISO code es): + cp transations/monero.ts transations/monero_es.ts +To edit translations for Spanish: + linguist translations/monero_es.ts +To build translations after modiying them: + ./utils/translations/build-translations.sh +To test a translation: + LANG=es ./build/release/bin/simplewallet +To add new translatable sources in the source: + Use the tr(string) function if possible. If the code is in a class, + and this class doesn't already have a tr() static function, add one, + which uses a context named after what lupdate uses for the context, + usually the fully qualified class name (eg, cryptonote::simple_wallet). + If you need to use tr in code that's not in a class, you can use the + fully qualified version (eg, simple_wallet::tr) of the one matching + the context you want. + Use QT_TRANSLATE_NOOP(string) if you want to specify a context manually. +If you're getting messages of the form: + Class 'cryptonote::simple_wallet' lacks Q_OBJECT macro +all is fine, we don't actually need that here. diff --git a/README.md b/README.md new file mode 100644 index 0000000..e2ba191 +++ b/README.md @@ -0,0 +1,174 @@ +Copyright (c) 2014-2015, The Monero Project +Web: [getmonero.org](https://getmonero.org) +Forum: [forum.getmonero.org](https://forum.getmonero.org) +Mail: [dev@getmonero.org](mailto:dev@getmonero.org) +Github (staging): [https://github.com/monero-project/bitmonero](https://github.com/monero-project/bitmonero) +Github (development): [http://github.com/monero-project/bitmonero/tree/development](http://github.com/monero-project/bitmonero/tree/development) +IRC: [#monero-dev on Freenode](irc://chat.freenode.net/#monero-dev) +Monero is a private, secure, untraceable currency. You are your bank, you control your funds, and nobody can trace your transfers unless you decide so. +**Privacy:** Monero uses a cryptographically sound system to allow you to send and receive funds without your transactions being easily revealed on the blockchain (the ledger of transactions that everyone has). This ensures that your purchases, receipts, and all transfers remain absolutely private by default. +**Security:** Using the power of a distributed peer-to-peer consensus network, every transaction on the network is cryptographically secured. Individual wallets have a 24 word mnemonic seed that is only displayed once, and can be written down to backup the wallet. Wallet files are encrypted with a passphrase to ensure they are useless if stolen. +**Untraceability:** By taking advantage of ring signatures, a special property of a certain type of cryptography, Monero is able to ensure that transactions are not only untraceable, but have an optional measure of ambiguity that ensures that transactions cannot easily be tied back to an individual user or computer. +This is the core implementation of Monero. It is open source and completely free to use without restrictions, except for those specified in the license agreement below. There are no restrictions on anyone creating an alternative implementation of Monero that uses the protocol and network in a compatible manner. +As with many development projects, the repository on Github is considered to be the "staging" area for the latest changes. Before changes are merged into that branch on the main repository, they are tested by individual developers, committed to the "development" branch, and then subsequently tested by contributors who focus on thorough testing and code reviews. That having been said, the repository should be carefully considered before using it in a production environment, unless there is a patch in the repository for a particular show-stopping issue you are experiencing. It is generally a better idea to use a tagged release for stability. +Anyone is welcome to contribute to Monero. If you have a fix or code change, feel free to submit is as a pull request directly to the "development" branch. In cases where the change is relatively small or does not affect other parts of the codebase it may be merged in immediately by any one of the collaborators. On the other hand, if the change is particularly large or complex, it is expected that it will be discussed at length either well in advance of the pull request being submitted, or even directly on the pull request. +Monero development can be supported directly through donations. +Both Monero and Bitcoin donations can be made to donate.getmonero.org if using a client that supports the [OpenAlias](https://openalias.org) standard +The Monero donation address is: 46BeWrHpwXmHDpDEUmZBWZfoQpdc6HaERCNmx1pEYL2rAcuwufPN9rXHHtyUA4QVy66qeFQkn6sfK8a HYjA3jk3o1Bv16em (viewkey: e422831985c9205238ef84daf6805526c14d96fd7b059fe68c7ab98e495e5703) +The Bitcoin donation address is: 1FhnVJi2V1k4MqXm2nHoEbY5LV7FPai7bb +Core development funding and/or some supporting services are also graciously provided by sponsors: +[![MyMonero](https://static.getmonero.org/images/sponsors/mymonero.png)](https://mymonero.com) [![Kitware](https://static.getmonero.org/images/sponsors/kitware.png?1)](http://kitware.com) [![Dome9](https://static.getmonero.org/images/sponsors/dome9.png)](http://dome9.com) [![Araxis](https://static.getmonero.org/images/sponsors/araxis.png)](http://araxis.com) [![JetBrains](https://static.getmonero.org/images/sponsors/jetbrains.png)](http://www.jetbrains.com/) [![Navicat](https://static.getmonero.org/images/sponsors/navicat.png)](http://www.navicat.com/) +There are also several mining pools that kindly donate a portion of their fees, [a list of them can be found on our Bitcointalk post](https://bitcointalk.org/index.php?topic=583449.0). +Copyright (c) 2014-2015, The Monero Project +All rights reserved. +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: +1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. +3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Parts of the project are originally copyright (c) 2012-2013 The Cryptonote developers +Dependencies: GCC 4.7.3 or later, CMake 2.8.6 or later, libunbound 1.4.16 or later (note: Unbound is not a dependency, libunbound is), libevent 2.0 or later, libgtest 1.5 or later, and Boost 1.53 or later (except 1.54, [more details here](http://goo.gl/RrCFmA)). +Static Build Additional Dependencies: ldns 1.6.17 or later, expat 1.1 or later, bison or yacc +**Basic Process:** +* Install the dependencies (see below for more detailed instructions for your OS) +* To build, change to the root of the source code directory, and run `make`. Please note that Windows systems follow a slightly different process, outlined below. +* The resulting executables can be found in `build/release/bin` or `build/debug/bin`, depending on what you're building. +**Advanced options:** +* Parallel build: run `make -j<number of threads>` instead of `make`. +* Statically linked release build: run `make release-static`. +* Debug build: run `make debug`. +* Test suite: run `make release-test` to run tests in addition to building. Running `make debug-test` will do the same to the debug version. +**Makefile Targets for Static Builds:** +For static builds there are a number of Makefile targets to make the build process easier. +* ```make release-static-win64``` builds statically for 64-bit Windows systems +* ```make release-static-win32``` builds statically for 32-bit Windows systems +* ```make release-static-64``` the default, builds statically for 64-bit non-Windows systems +* ```make release-static-32``` builds statically for 32-bit non-Windows systems +* ```make release-static-arm6``` builds statically for ARMv6 devices, such as the Raspberry Pi +The instructions above should provide enough detail. +The project can be built from scratch by following instructions for Unix and Linux above. +Alternatively, it can be built in an easier and more automated fashion using Homebrew: +* Ensure Homebrew is installed, it can be found at http://brew.sh +* Add the repository to brew: `brew tap sammy007/cryptonight` +* Build Monero: `brew install bitmonero --build-from-source` +Dependencies: mingw-w64, msys2, CMake 2.8.6 or later, libunbound 1.4.16 or later (note: Unbound is not a dependency, libunbound is), and Boost 1.53 or 1.55 (except 1.54, [more details here](http://goo.gl/RrCFmA)), BerkeleyDB 4.8 or later (note: on Ubuntu this means installing libdb-dev and libdb++-dev). +**Preparing the Build Environment** +* Download the [MSYS2 installer](http://msys2.github.io), 64-bit or 32-bit as needed, and run it. +* Use the shortcut associated with your architecture to launch the MSYS2 environment. On 64-bit systems that would be the MinGW-w64 Win64 Shell shortcut. Note that if you are running 64-bit Windows, you will have both 64-bit and 32-bit environments. +* Update the packages in your MSYS2 install: +``` +pacman -Sy +pacman -Su --ignoregroup base +pacman -Su +``` +* For those of you already familiar with pacman, you can run the normal `pacman -Syu` to update, but you may get errors and need to restart MSYS2 if pacman's dependencies are updated. +* Install dependencies: `pacman -S mingw-w64-x86_64-gcc make mingw-w64-x86_64-cmake mingw-w64-x86_64-unbound mingw-w64-x86_64-boost` +* If you are planning to build statically you will also need to install: `pacman -S mingw-w64-x86_64-ldns mingw-w64-x86_64-expat` (note that these are likely already installed by the unbound dependency installation above) +**Building** +* From the root of the source code directory run: +``` +mkdir build +cd build +``` +* If you are on a 64-bit system, run: +``` +cmake -G "MSYS Makefiles" -D CMAKE_BUILD_TYPE=Release -D ARCH="x86-64" -D BUILD_64=ON -D CMAKE_TOOLCHAIN_FILE=../cmake/64-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys64 .. +``` +* If you are on a 32-bit system, run: +``` +cmake -G "MSYS Makefiles" -D CMAKE_BUILD_TYPE=Release -D ARCH="i686" -D BUILD_64=OFF -D CMAKE_TOOLCHAIN_FILE=../cmake/32-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys32 .. +``` +* You can now run `make` to have it build +* The resulting executables can be found in `build/release/bin` or `build/debug/bin`, depending on what you're building. +If you installed MSYS2 in a folder other than c:/msys64, make the appropriate substitution above. +**Advanced options:** +* Parallel build: run `make -j<number of threads>` instead of `make`. +* Statically linked release build: run `make release-static`. +* Debug build: run `make debug`. +* Test suite: run `make release-test` to run tests in addition to building. Running `make debug-test` will do the same to the debug version. +The project can be built from scratch by following instructions for Unix and Linux above. +We expect to add Monero into the ports tree in the near future, which will aid in managing installations using ports or packages. +Monero developer documentation uses Doxygen, and is currently a work-in-progress. +Dependencies: Doxygen 1.8.0 or later, Graphviz 2.28 or later (optional). +* To build, change to the root of the source code directory, and run `doxygen Doxyfile` +* If you have installed Graphviz, you can also generate in-doc diagrams by instead running `HAVE_DOT=YES doxygen Doxyfile` +* The output will be built in doc/html/ +See README.i18n diff --git a/ReleaseNotes.txt b/ReleaseNotes.txt deleted file mode 100644 index 7d6f152..0000000 +++ /dev/null @@ -1,31 +0,0 @@ diff --git a/cmake/32-bit-toolchain.cmake b/cmake/32-bit-toolchain.cmake new file mode 100644 index 0000000..ce98362 +++ b/cmake/32-bit-toolchain.cmake @@ -0,0 +1,47 @@ +set (CMAKE_SYSTEM_NAME Windows) +set (GCC_PREFIX i686-w64-mingw32) +set (CMAKE_C_COMPILER ${GCC_PREFIX}-gcc) +set (CMAKE_CXX_COMPILER ${GCC_PREFIX}-g++) +set (CMAKE_AR ar CACHE FILEPATH "" FORCE) +set (CMAKE_NM nm CACHE FILEPATH "" FORCE) +set (CMAKE_RC_COMPILER windres) +set (CMAKE_FIND_ROOT_PATH "${MSYS2_FOLDER}/mingw32") +set (CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) # Find programs on host +set (CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) # Find libs in target +set (CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) # Find includes in target +set (MINGW_FLAG "-m32") +set (USE_LTO_DEFAULT false) diff --git a/cmake/64-bit-toolchain.cmake b/cmake/64-bit-toolchain.cmake new file mode 100644 index 0000000..e2d07e5 +++ b/cmake/64-bit-toolchain.cmake @@ -0,0 +1,47 @@ +set (CMAKE_SYSTEM_NAME Windows) +set (GCC_PREFIX x86_64-w64-mingw32) +set (CMAKE_C_COMPILER ${GCC_PREFIX}-gcc) +set (CMAKE_CXX_COMPILER ${GCC_PREFIX}-g++) +set (CMAKE_AR ar CACHE FILEPATH "" FORCE) +set (CMAKE_NM nm CACHE FILEPATH "" FORCE) +set (CMAKE_RC_COMPILER windres) +set (CMAKE_FIND_ROOT_PATH "${MSYS2_FOLDER}/mingw64") +set (CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) # Find programs on host +set (CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) # Find libs in target +set (CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) # Find includes in target +set (MINGW_FLAG "-m64") +set (USE_LTO_DEFAULT false) diff --git a/cmake/Doxyfile.in b/cmake/Doxyfile.in new file mode 100644 index 0000000..35a4911 +++ b/cmake/Doxyfile.in @@ -0,0 +1,1803 @@ +DOXYFILE_ENCODING = UTF-8 +PROJECT_NAME = Monero +PROJECT_NUMBER = @VERSION_STRING@ +PROJECT_BRIEF = @CPACK_PACKAGE_DESCRIPTION_SUMMARY@ +PROJECT_LOGO = +OUTPUT_DIRECTORY = ./docs +CREATE_SUBDIRS = YES +OUTPUT_LANGUAGE = English +BRIEF_MEMBER_DESC = YES +REPEAT_BRIEF = YES +ABBREVIATE_BRIEF = +ALWAYS_DETAILED_SEC = NO +INLINE_INHERITED_MEMB = NO +FULL_PATH_NAMES = YES +STRIP_FROM_PATH = +STRIP_FROM_INC_PATH = +SHORT_NAMES = NO +JAVADOC_AUTOBRIEF = NO +QT_AUTOBRIEF = NO +MULTILINE_CPP_IS_BRIEF = NO +INHERIT_DOCS = YES +SEPARATE_MEMBER_PAGES = NO +TAB_SIZE = 4 +ALIASES = +TCL_SUBST = +OPTIMIZE_OUTPUT_FOR_C = NO +OPTIMIZE_OUTPUT_JAVA = NO +OPTIMIZE_FOR_FORTRAN = NO +OPTIMIZE_OUTPUT_VHDL = NO +EXTENSION_MAPPING = +MARKDOWN_SUPPORT = YES +BUILTIN_STL_SUPPORT = YES +CPP_CLI_SUPPORT = NO +SIP_SUPPORT = NO +IDL_PROPERTY_SUPPORT = YES +DISTRIBUTE_GROUP_DOC = NO +SUBGROUPING = YES +INLINE_GROUPED_CLASSES = NO +INLINE_SIMPLE_STRUCTS = NO +TYPEDEF_HIDES_STRUCT = NO +SYMBOL_CACHE_SIZE = 0 +LOOKUP_CACHE_SIZE = 0 +EXTRACT_ALL = YES +EXTRACT_PRIVATE = NO +EXTRACT_PACKAGE = NO +EXTRACT_STATIC = NO +EXTRACT_LOCAL_CLASSES = YES +EXTRACT_LOCAL_METHODS = NO +EXTRACT_ANON_NSPACES = NO +HIDE_UNDOC_MEMBERS = NO +HIDE_UNDOC_CLASSES = NO +HIDE_FRIEND_COMPOUNDS = NO +HIDE_IN_BODY_DOCS = NO +INTERNAL_DOCS = NO +CASE_SENSE_NAMES = NO +HIDE_SCOPE_NAMES = NO +SHOW_INCLUDE_FILES = YES +FORCE_LOCAL_INCLUDES = NO +INLINE_INFO = YES +SORT_MEMBER_DOCS = YES +SORT_BRIEF_DOCS = NO +SORT_MEMBERS_CTORS_1ST = NO +SORT_GROUP_NAMES = NO +SORT_BY_SCOPE_NAME = NO +STRICT_PROTO_MATCHING = NO +GENERATE_TODOLIST = YES +GENERATE_TESTLIST = YES +GENERATE_BUGLIST = YES +GENERATE_DEPRECATEDLIST= YES +ENABLED_SECTIONS = +MAX_INITIALIZER_LINES = 30 +SHOW_USED_FILES = YES +SHOW_FILES = YES +SHOW_NAMESPACES = YES +FILE_VERSION_FILTER = +LAYOUT_FILE = +CITE_BIB_FILES = +QUIET = NO +WARNINGS = YES +WARN_IF_UNDOCUMENTED = YES +WARN_IF_DOC_ERROR = YES +WARN_NO_PARAMDOC = NO +WARN_FORMAT = "$file:$line: $text" +WARN_LOGFILE = +INPUT = ../README.md \ + ../contrib/ \ + ../external/ \ + ../include/ \ + ../tests/ \ + ../src/ +INPUT_ENCODING = UTF-8 +FILE_PATTERNS = *.cpp \ + *.h \ + *.hpp +RECURSIVE = YES +EXCLUDE = +EXCLUDE_SYMLINKS = NO +EXCLUDE_PATTERNS = *.pb.* \ + tinythread.* \ + fast_mutex.* \ + anyoption.* \ + stacktrace.* \ + */simpleini/* \ + ExportWrapper.* \ + WinsockWrapper.* \ + otapicli.* \ + test*.* \ + irrXML.* \ + */chaiscript/* \ + */zmq/* +EXCLUDE_SYMBOLS = tinythread +EXAMPLE_PATH = +EXAMPLE_PATTERNS = +EXAMPLE_RECURSIVE = NO +IMAGE_PATH = +INPUT_FILTER = +FILTER_PATTERNS = +FILTER_SOURCE_FILES = NO +FILTER_SOURCE_PATTERNS = +SOURCE_BROWSER = YES +INLINE_SOURCES = YES +STRIP_CODE_COMMENTS = YES +REFERENCED_BY_RELATION = NO +REFERENCES_RELATION = NO +REFERENCES_LINK_SOURCE = YES +USE_HTAGS = NO +VERBATIM_HEADERS = YES +ALPHABETICAL_INDEX = YES +COLS_IN_ALPHA_INDEX = 5 +IGNORE_PREFIX = +GENERATE_HTML = YES +HTML_OUTPUT = html +HTML_FILE_EXTENSION = .html +HTML_HEADER = +HTML_FOOTER = +HTML_STYLESHEET = +HTML_EXTRA_FILES = +HTML_COLORSTYLE_HUE = 220 +HTML_COLORSTYLE_SAT = 100 +HTML_COLORSTYLE_GAMMA = 80 +HTML_TIMESTAMP = YES +HTML_DYNAMIC_SECTIONS = NO +HTML_INDEX_NUM_ENTRIES = 100 +GENERATE_DOCSET = NO +DOCSET_FEEDNAME = "Doxygen generated docs" +DOCSET_BUNDLE_ID = org.doxygen.Project +DOCSET_PUBLISHER_ID = org.doxygen.Publisher +DOCSET_PUBLISHER_NAME = Publisher +GENERATE_HTMLHELP = NO +CHM_FILE = +HHC_LOCATION = +GENERATE_CHI = NO +CHM_INDEX_ENCODING = +BINARY_TOC = NO +TOC_EXPAND = NO +GENERATE_QHP = NO +QCH_FILE = +QHP_NAMESPACE = org.doxygen.Project +QHP_VIRTUAL_FOLDER = doc +QHP_CUST_FILTER_NAME = +QHP_CUST_FILTER_ATTRS = +QHP_SECT_FILTER_ATTRS = +QHG_LOCATION = +GENERATE_ECLIPSEHELP = NO +ECLIPSE_DOC_ID = org.doxygen.Project +DISABLE_INDEX = NO +GENERATE_TREEVIEW = YES +ENUM_VALUES_PER_LINE = 4 +TREEVIEW_WIDTH = 250 +EXT_LINKS_IN_WINDOW = YES +FORMULA_FONTSIZE = 10 +FORMULA_TRANSPARENT = YES +USE_MATHJAX = NO +MATHJAX_RELPATH = http://cdn.mathjax.org/mathjax/latest +MATHJAX_EXTENSIONS = +SEARCHENGINE = YES +SERVER_BASED_SEARCH = NO +GENERATE_LATEX = NO +LATEX_OUTPUT = latex +LATEX_CMD_NAME = latex +MAKEINDEX_CMD_NAME = makeindex +COMPACT_LATEX = NO +PAPER_TYPE = a4 +EXTRA_PACKAGES = +LATEX_HEADER = +LATEX_FOOTER = +PDF_HYPERLINKS = YES +USE_PDFLATEX = YES +LATEX_BATCHMODE = NO +LATEX_HIDE_INDICES = NO +LATEX_SOURCE_CODE = NO +LATEX_BIB_STYLE = plain +GENERATE_RTF = NO +RTF_OUTPUT = rtf +COMPACT_RTF = NO +RTF_HYPERLINKS = NO +RTF_STYLESHEET_FILE = +RTF_EXTENSIONS_FILE = +GENERATE_MAN = NO +MAN_OUTPUT = man +MAN_EXTENSION = .3 +MAN_LINKS = NO +GENERATE_XML = NO

Continuation of: Git diff from commit e940386f9a8765423ab3dd9e3aabe19a68cba9f9 (thankful_for_today's last commit) to current HEAD, excluding changes in /external so as to exclude libraries and submoduled code, and excluding contrib/ so as to exclude changes made to epee, excluding most comments because apparently nobody that writes code for BCN knows what a comment is anyway, excluding removed lines, excluding empty lines

+XML_OUTPUT = xml +XML_PROGRAMLISTING = YES +GENERATE_DOCBOOK = NO +DOCBOOK_OUTPUT = docbook +GENERATE_AUTOGEN_DEF = NO +GENERATE_PERLMOD = NO +PERLMOD_LATEX = NO +PERLMOD_PRETTY = YES +PERLMOD_MAKEVAR_PREFIX = +ENABLE_PREPROCESSING = YES +MACRO_EXPANSION = NO +EXPAND_ONLY_PREDEF = NO +SEARCH_INCLUDES = YES +INCLUDE_PATH = +INCLUDE_FILE_PATTERNS = +PREDEFINED = +EXPAND_AS_DEFINED = +SKIP_FUNCTION_MACROS = YES +TAGFILES = +GENERATE_TAGFILE = +ALLEXTERNALS = NO +EXTERNAL_GROUPS = YES +EXTERNAL_PAGES = YES +PERL_PATH = /usr/bin/perl +CLASS_DIAGRAMS = YES +MSCGEN_PATH = +DIA_PATH = +HIDE_UNDOC_RELATIONS = YES +HAVE_DOT = $(HAVE_DOT) +DOT_NUM_THREADS = 0 +DOT_FONTNAME = Helvetica +DOT_FONTSIZE = 10 +DOT_FONTPATH = +CLASS_GRAPH = YES +COLLABORATION_GRAPH = YES +GROUP_GRAPHS = YES +UML_LOOK = NO +UML_LIMIT_NUM_FIELDS = 10 +TEMPLATE_RELATIONS = NO +INCLUDE_GRAPH = YES +INCLUDED_BY_GRAPH = YES +CALL_GRAPH = NO +CALLER_GRAPH = NO +GRAPHICAL_HIERARCHY = YES +DIRECTORY_GRAPH = YES +DOT_IMAGE_FORMAT = png +INTERACTIVE_SVG = NO +DOT_PATH = +DOTFILE_DIRS = +MSCFILE_DIRS = +DIAFILE_DIRS = +DOT_GRAPH_MAX_NODES = 50 +MAX_DOT_GRAPH_DEPTH = 0 +DOT_TRANSPARENT = NO +DOT_MULTI_TARGETS = NO +GENERATE_LEGEND = YES +DOT_CLEANUP = YES diff --git a/Makefile b/Makefile index 451319b..dda8663 100644 +++ b/Makefile @@ -1,33 +1,96 @@ +all: release-all cmake-debug: mkdir -p build/debug cd build/debug && cmake -D CMAKE_BUILD_TYPE=Debug ../.. +debug: cmake-debug cd build/debug && $(MAKE) +debug-test: debug + mkdir -p build/debug + cd build/debug && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=Debug ../.. && $(MAKE) test +debug-all: + mkdir -p build/debug + cd build/debug && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=Debug ../.. && $(MAKE) cmake-release: mkdir -p build/release cd build/release && cmake -D CMAKE_BUILD_TYPE=Release ../.. +release: cmake-release cd build/release && $(MAKE) +release-test: release + mkdir -p build/release + cd build/release && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) test +release-all: + mkdir -p build/release + cd build/release && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-arm6: + mkdir -p build/release + cd build/release && cmake -D BUILD_TESTS=OFF -D ARCH="armv6zk" -D BUILD_64=OFF -D NO_AES=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-arm7: + mkdir -p build/release + cd build/release && cmake -D BUILD_TESTS=OFF -D ARCH="armv7-a" -D BUILD_64=OFF -D NO_AES=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-static: release-static-64 +release-static-64: + mkdir -p build/release + cd build/release && cmake -D STATIC=ON -D ARCH="x86-64" -D BUILD_64=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-static-32: + mkdir -p build/release + cd build/release && cmake -D STATIC=ON -D ARCH="i686" -D BUILD_64=OFF -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE) +release-static-win64: + mkdir -p build/release + cd build/release && cmake -G "MSYS Makefiles" -D STATIC=ON -D ARCH="x86-64" -D BUILD_64=ON -D CMAKE_BUILD_TYPE=Release -D CMAKE_TOOLCHAIN_FILE=../../cmake/64-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys64 ../.. && $(MAKE) +release-static-win32: + mkdir -p build/release + cd build/release && cmake -G "MSYS Makefiles" -D STATIC=ON -D ARCH="i686" -D BUILD_64=OFF -D CMAKE_BUILD_TYPE=Release -D CMAKE_TOOLCHAIN_FILE=../../cmake/32-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys32 ../.. && $(MAKE) clean: + @echo "WARNING: Back-up your wallet if it exists within ./build!" ; \ + read -r -p "This will destroy the build directory, continue (y/N)?: " CONTINUE; \ + [ $$CONTINUE = "y" ] || [ $$CONTINUE = "Y" ] || (echo "Exiting."; exit 1;) rm -rf build tags: ctags -R --sort=1 --c++-kinds=+p --fields=+iaS --extra=+q --language-force=C++ src contrib tests/gtest +.PHONY: all cmake-debug debug debug-test debug-all cmake-release release release-test release-all clean tags diff --git a/README b/README deleted file mode 100644 index f63a4cc..0000000 +++ /dev/null @@ -1,32 +0,0 @@ diff --git a/README.i18n b/README.i18n new file mode 100644 index 0000000..2b913f7 +++ b/README.i18n @@ -0,0 +1,45 @@ +The Monero command line tools can be translated in various languages. +In order to use the same translation workflow as the future GUI, they +use Qt Linguist translation files. However, to avoid the dependencies +on Qt this normally implies, they use a custom loader to read those +files at runtime. In order to update, or build translations files, you +do need to have Qt tools installed, however. For translating, you need +either the Qt Linguist GUI, or another tool that supports Qt ts files, +such as Transifex. To run, you do not need anything Qt. +To update ts files after changing source code: + ./utils/translations/update-translations.sh +To add a new language, eg Spanish (ISO code es): + cp transations/monero.ts transations/monero_es.ts +To edit translations for Spanish: + linguist translations/monero_es.ts +To build translations after modiying them: + ./utils/translations/build-translations.sh +To test a translation: + LANG=es ./build/release/bin/simplewallet +To add new translatable sources in the source: + Use the tr(string) function if possible. If the code is in a class, + and this class doesn't already have a tr() static function, add one, + which uses a context named after what lupdate uses for the context, + usually the fully qualified class name (eg, cryptonote::simple_wallet). + If you need to use tr in code that's not in a class, you can use the + fully qualified version (eg, simple_wallet::tr) of the one matching + the context you want. + Use QT_TRANSLATE_NOOP(string) if you want to specify a context manually. +If you're getting messages of the form: + Class 'cryptonote::simple_wallet' lacks Q_OBJECT macro +all is fine, we don't actually need that here. diff --git a/README.md b/README.md new file mode 100644 index 0000000..e2ba191 +++ b/README.md @@ -0,0 +1,174 @@ +Copyright (c) 2014-2015, The Monero Project +Web: [getmonero.org](https://getmonero.org) +Forum: [forum.getmonero.org](https://forum.getmonero.org) +Mail: [dev@getmonero.org](mailto:dev@getmonero.org) +Github (staging): [https://github.com/monero-project/bitmonero](https://github.com/monero-project/bitmonero) +Github (development): [http://github.com/monero-project/bitmonero/tree/development](http://github.com/monero-project/bitmonero/tree/development) +IRC: [#monero-dev on Freenode](irc://chat.freenode.net/#monero-dev) +Monero is a private, secure, untraceable currency. You are your bank, you control your funds, and nobody can trace your transfers unless you decide so. +**Privacy:** Monero uses a cryptographically sound system to allow you to send and receive funds without your transactions being easily revealed on the blockchain (the ledger of transactions that everyone has). This ensures that your purchases, receipts, and all transfers remain absolutely private by default. +**Security:** Using the power of a distributed peer-to-peer consensus network, every transaction on the network is cryptographically secured. Individual wallets have a 24 word mnemonic seed that is only displayed once, and can be written down to backup the wallet. Wallet files are encrypted with a passphrase to ensure they are useless if stolen. +**Untraceability:** By taking advantage of ring signatures, a special property of a certain type of cryptography, Monero is able to ensure that transactions are not only untraceable, but have an optional measure of ambiguity that ensures that transactions cannot easily be tied back to an individual user or computer. +This is the core implementation of Monero. It is open source and completely free to use without restrictions, except for those specified in the license agreement below. There are no restrictions on anyone creating an alternative implementation of Monero that uses the protocol and network in a compatible manner. +As with many development projects, the repository on Github is considered to be the "staging" area for the latest changes. Before changes are merged into that branch on the main repository, they are tested by individual developers, committed to the "development" branch, and then subsequently tested by contributors who focus on thorough testing and code reviews. That having been said, the repository should be carefully considered before using it in a production environment, unless there is a patch in the repository for a particular show-stopping issue you are experiencing. It is generally a better idea to use a tagged release for stability. +Anyone is welcome to contribute to Monero. If you have a fix or code change, feel free to submit is as a pull request directly to the "development" branch. In cases where the change is relatively small or does not affect other parts of the codebase it may be merged in immediately by any one of the collaborators. On the other hand, if the change is particularly large or complex, it is expected that it will be discussed at length either well in advance of the pull request being submitted, or even directly on the pull request. +Monero development can be supported directly through donations. +Both Monero and Bitcoin donations can be made to donate.getmonero.org if using a client that supports the [OpenAlias](https://openalias.org) standard +The Monero donation address is: 46BeWrHpwXmHDpDEUmZBWZfoQpdc6HaERCNmx1pEYL2rAcuwufPN9rXHHtyUA4QVy66qeFQkn6sfK8a HYjA3jk3o1Bv16em (viewkey: e422831985c9205238ef84daf6805526c14d96fd7b059fe68c7ab98e495e5703) +The Bitcoin donation address is: 1FhnVJi2V1k4MqXm2nHoEbY5LV7FPai7bb +Core development funding and/or some supporting services are also graciously provided by sponsors: +[![MyMonero](https://static.getmonero.org/images/sponsors/mymonero.png)](https://mymonero.com) [![Kitware](https://static.getmonero.org/images/sponsors/kitware.png?1)](http://kitware.com) [![Dome9](https://static.getmonero.org/images/sponsors/dome9.png)](http://dome9.com) [![Araxis](https://static.getmonero.org/images/sponsors/araxis.png)](http://araxis.com) [![JetBrains](https://static.getmonero.org/images/sponsors/jetbrains.png)](http://www.jetbrains.com/) [![Navicat](https://static.getmonero.org/images/sponsors/navicat.png)](http://www.navicat.com/) +There are also several mining pools that kindly donate a portion of their fees, [a list of them can be found on our Bitcointalk post](https://bitcointalk.org/index.php?topic=583449.0). +Copyright (c) 2014-2015, The Monero Project +All rights reserved. +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: +1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. +3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Parts of the project are originally copyright (c) 2012-2013 The Cryptonote developers +Dependencies: GCC 4.7.3 or later, CMake 2.8.6 or later, libunbound 1.4.16 or later (note: Unbound is not a dependency, libunbound is), libevent 2.0 or later, libgtest 1.5 or later, and Boost 1.53 or later (except 1.54, [more details here](http://goo.gl/RrCFmA)). +Static Build Additional Dependencies: ldns 1.6.17 or later, expat 1.1 or later, bison or yacc +**Basic Process:** +* Install the dependencies (see below for more detailed instructions for your OS) +* To build, change to the root of the source code directory, and run `make`. Please note that Windows systems follow a slightly different process, outlined below. +* The resulting executables can be found in `build/release/bin` or `build/debug/bin`, depending on what you're building. +**Advanced options:** +* Parallel build: run `make -j<number of threads>` instead of `make`. +* Statically linked release build: run `make release-static`. +* Debug build: run `make debug`. +* Test suite: run `make release-test` to run tests in addition to building. Running `make debug-test` will do the same to the debug version. +**Makefile Targets for Static Builds:** +For static builds there are a number of Makefile targets to make the build process easier. +* ```make release-static-win64``` builds statically for 64-bit Windows systems +* ```make release-static-win32``` builds statically for 32-bit Windows systems +* ```make release-static-64``` the default, builds statically for 64-bit non-Windows systems +* ```make release-static-32``` builds statically for 32-bit non-Windows systems +* ```make release-static-arm6``` builds statically for ARMv6 devices, such as the Raspberry Pi +The instructions above should provide enough detail. +The project can be built from scratch by following instructions for Unix and Linux above. +Alternatively, it can be built in an easier and more automated fashion using Homebrew: +* Ensure Homebrew is installed, it can be found at http://brew.sh +* Add the repository to brew: `brew tap sammy007/cryptonight` +* Build Monero: `brew install bitmonero --build-from-source` +Dependencies: mingw-w64, msys2, CMake 2.8.6 or later, libunbound 1.4.16 or later (note: Unbound is not a dependency, libunbound is), and Boost 1.53 or 1.55 (except 1.54, [more details here](http://goo.gl/RrCFmA)), BerkeleyDB 4.8 or later (note: on Ubuntu this means installing libdb-dev and libdb++-dev). +**Preparing the Build Environment** +* Download the [MSYS2 installer](http://msys2.github.io), 64-bit or 32-bit as needed, and run it. +* Use the shortcut associated with your architecture to launch the MSYS2 environment. On 64-bit systems that would be the MinGW-w64 Win64 Shell shortcut. Note that if you are running 64-bit Windows, you will have both 64-bit and 32-bit environments. +* Update the packages in your MSYS2 install: +``` +pacman -Sy +pacman -Su --ignoregroup base +pacman -Su +``` +* For those of you already familiar with pacman, you can run the normal `pacman -Syu` to update, but you may get errors and need to restart MSYS2 if pacman's dependencies are updated. +* Install dependencies: `pacman -S mingw-w64-x86_64-gcc make mingw-w64-x86_64-cmake mingw-w64-x86_64-unbound mingw-w64-x86_64-boost` +* If you are planning to build statically you will also need to install: `pacman -S mingw-w64-x86_64-ldns mingw-w64-x86_64-expat` (note that these are likely already installed by the unbound dependency installation above) +**Building** +* From the root of the source code directory run: +``` +mkdir build +cd build +``` +* If you are on a 64-bit system, run: +``` +cmake -G "MSYS Makefiles" -D CMAKE_BUILD_TYPE=Release -D ARCH="x86-64" -D BUILD_64=ON -D CMAKE_TOOLCHAIN_FILE=../cmake/64-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys64 .. +``` +* If you are on a 32-bit system, run: +``` +cmake -G "MSYS Makefiles" -D CMAKE_BUILD_TYPE=Release -D ARCH="i686" -D BUILD_64=OFF -D CMAKE_TOOLCHAIN_FILE=../cmake/32-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys32 .. +``` +* You can now run `make` to have it build +* The resulting executables can be found in `build/release/bin` or `build/debug/bin`, depending on what you're building. +If you installed MSYS2 in a folder other than c:/msys64, make the appropriate substitution above. +**Advanced options:** +* Parallel build: run `make -j<number of threads>` instead of `make`. +* Statically linked release build: run `make release-static`. +* Debug build: run `make debug`. +* Test suite: run `make release-test` to run tests in addition to building. Running `make debug-test` will do the same to the debug version. +The project can be built from scratch by following instructions for Unix and Linux above. +We expect to add Monero into the ports tree in the near future, which will aid in managing installations using ports or packages. +Monero developer documentation uses Doxygen, and is currently a work-in-progress. +Dependencies: Doxygen 1.8.0 or later, Graphviz 2.28 or later (optional). +* To build, change to the root of the source code directory, and run `doxygen Doxyfile` +* If you have installed Graphviz, you can also generate in-doc diagrams by instead running `HAVE_DOT=YES doxygen Doxyfile` +* The output will be built in doc/html/ +See README.i18n diff --git a/ReleaseNotes.txt b/ReleaseNotes.txt deleted file mode 100644 index 7d6f152..0000000 +++ /dev/null @@ -1,31 +0,0 @@ diff --git a/cmake/32-bit-toolchain.cmake b/cmake/32-bit-toolchain.cmake new file mode 100644 index 0000000..ce98362 +++ b/cmake/32-bit-toolchain.cmake @@ -0,0 +1,47 @@ +set (CMAKE_SYSTEM_NAME Windows) +set (GCC_PREFIX i686-w64-mingw32) +set (CMAKE_C_COMPILER ${GCC_PREFIX}-gcc) +set (CMAKE_CXX_COMPILER ${GCC_PREFIX}-g++) +set (CMAKE_AR ar CACHE FILEPATH "" FORCE) +set (CMAKE_NM nm CACHE FILEPATH "" FORCE) +set (CMAKE_RC_COMPILER windres) +set (CMAKE_FIND_ROOT_PATH "${MSYS2_FOLDER}/mingw32") +set (CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) # Find programs on host +set (CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) # Find libs in target +set (CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) # Find includes in target +set (MINGW_FLAG "-m32") +set (USE_LTO_DEFAULT false) diff --git a/cmake/64-bit-toolchain.cmake b/cmake/64-bit-toolchain.cmake new file mode 100644 index 0000000..e2d07e5 +++ b/cmake/64-bit-toolchain.cmake @@ -0,0 +1,47 @@ +set (CMAKE_SYSTEM_NAME Windows) +set (GCC_PREFIX x86_64-w64-mingw32) +set (CMAKE_C_COMPILER ${GCC_PREFIX}-gcc) +set (CMAKE_CXX_COMPILER ${GCC_PREFIX}-g++) +set (CMAKE_AR ar CACHE FILEPATH "" FORCE) +set (CMAKE_NM nm CACHE FILEPATH "" FORCE) +set (CMAKE_RC_COMPILER windres) +set (CMAKE_FIND_ROOT_PATH "${MSYS2_FOLDER}/mingw64") +set (CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) # Find programs on host +set (CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) # Find libs in target +set (CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) # Find includes in target +set (MINGW_FLAG "-m64") +set (USE_LTO_DEFAULT false) diff --git a/cmake/Doxyfile.in b/cmake/Doxyfile.in new file mode 100644 index 0000000..35a4911 +++ b/cmake/Doxyfile.in @@ -0,0 +1,1803 @@ +DOXYFILE_ENCODING = UTF-8 +PROJECT_NAME = Monero +PROJECT_NUMBER = @VERSION_STRING@ +PROJECT_BRIEF = @CPACK_PACKAGE_DESCRIPTION_SUMMARY@ +PROJECT_LOGO = +OUTPUT_DIRECTORY = ./docs +CREATE_SUBDIRS = YES +OUTPUT_LANGUAGE = English +BRIEF_MEMBER_DESC = YES +REPEAT_BRIEF = YES +ABBREVIATE_BRIEF = +ALWAYS_DETAILED_SEC = NO +INLINE_INHERITED_MEMB = NO +FULL_PATH_NAMES = YES +STRIP_FROM_PATH = +STRIP_FROM_INC_PATH = +SHORT_NAMES = NO +JAVADOC_AUTOBRIEF = NO +QT_AUTOBRIEF = NO +MULTILINE_CPP_IS_BRIEF = NO +INHERIT_DOCS = YES +SEPARATE_MEMBER_PAGES = NO +TAB_SIZE = 4 +ALIASES = +TCL_SUBST = +OPTIMIZE_OUTPUT_FOR_C = NO +OPTIMIZE_OUTPUT_JAVA = NO +OPTIMIZE_FOR_FORTRAN = NO +OPTIMIZE_OUTPUT_VHDL = NO +EXTENSION_MAPPING = +MARKDOWN_SUPPORT = YES +BUILTIN_STL_SUPPORT = YES +CPP_CLI_SUPPORT = NO +SIP_SUPPORT = NO +IDL_PROPERTY_SUPPORT = YES +DISTRIBUTE_GROUP_DOC = NO +SUBGROUPING = YES +INLINE_GROUPED_CLASSES = NO +INLINE_SIMPLE_STRUCTS = NO +TYPEDEF_HIDES_STRUCT = NO +SYMBOL_CACHE_SIZE = 0 +LOOKUP_CACHE_SIZE = 0 +EXTRACT_ALL = YES +EXTRACT_PRIVATE = NO +EXTRACT_PACKAGE = NO +EXTRACT_STATIC = NO +EXTRACT_LOCAL_CLASSES = YES +EXTRACT_LOCAL_METHODS = NO +EXTRACT_ANON_NSPACES = NO +HIDE_UNDOC_MEMBERS = NO +HIDE_UNDOC_CLASSES = NO +HIDE_FRIEND_COMPOUNDS = NO +HIDE_IN_BODY_DOCS = NO +INTERNAL_DOCS = NO +CASE_SENSE_NAMES = NO +HIDE_SCOPE_NAMES = NO +SHOW_INCLUDE_FILES = YES +FORCE_LOCAL_INCLUDES = NO +INLINE_INFO = YES +SORT_MEMBER_DOCS = YES +SORT_BRIEF_DOCS = NO +SORT_MEMBERS_CTORS_1ST = NO +SORT_GROUP_NAMES = NO +SORT_BY_SCOPE_NAME = NO +STRICT_PROTO_MATCHING = NO +GENERATE_TODOLIST = YES +GENERATE_TESTLIST = YES +GENERATE_BUGLIST = YES +GENERATE_DEPRECATEDLIST= YES +ENABLED_SECTIONS = +MAX_INITIALIZER_LINES = 30 +SHOW_USED_FILES = YES +SHOW_FILES = YES +SHOW_NAMESPACES = YES +FILE_VERSION_FILTER = +LAYOUT_FILE = +CITE_BIB_FILES = +QUIET = NO +WARNINGS = YES +WARN_IF_UNDOCUMENTED = YES +WARN_IF_DOC_ERROR = YES +WARN_NO_PARAMDOC = NO +WARN_FORMAT = "$file:$line: $text" +WARN_LOGFILE = +INPUT = ../README.md \ + ../contrib/ \ + ../external/ \ + ../include/ \ + ../tests/ \ + ../src/ +INPUT_ENCODING = UTF-8 +FILE_PATTERNS = *.cpp \ + *.h \ + *.hpp +RECURSIVE = YES +EXCLUDE = +EXCLUDE_SYMLINKS = NO +EXCLUDE_PATTERNS = *.pb.* \ + tinythread.* \ + fast_mutex.* \ + anyoption.* \ + stacktrace.* \ + */simpleini/* \ + ExportWrapper.* \ + WinsockWrapper.* \ + otapicli.* \ + test*.* \ + irrXML.* \ + */chaiscript/* \ + */zmq/* +EXCLUDE_SYMBOLS = tinythread +EXAMPLE_PATH = +EXAMPLE_PATTERNS = +EXAMPLE_RECURSIVE = NO +IMAGE_PATH = +INPUT_FILTER = +FILTER_PATTERNS = +FILTER_SOURCE_FILES = NO +FILTER_SOURCE_PATTERNS = +SOURCE_BROWSER = YES +INLINE_SOURCES = YES +STRIP_CODE_COMMENTS = YES +REFERENCED_BY_RELATION = NO +REFERENCES_RELATION = NO +REFERENCES_LINK_SOURCE = YES +USE_HTAGS = NO +VERBATIM_HEADERS = YES +ALPHABETICAL_INDEX = YES +COLS_IN_ALPHA_INDEX = 5 +IGNORE_PREFIX = +GENERATE_HTML = YES +HTML_OUTPUT = html +HTML_FILE_EXTENSION = .html +HTML_HEADER = +HTML_FOOTER = +HTML_STYLESHEET = +HTML_EXTRA_FILES = +HTML_COLORSTYLE_HUE = 220 +HTML_COLORSTYLE_SAT = 100 +HTML_COLORSTYLE_GAMMA = 80 +HTML_TIMESTAMP = YES +HTML_DYNAMIC_SECTIONS = NO +HTML_INDEX_NUM_ENTRIES = 100 +GENERATE_DOCSET = NO +DOCSET_FEEDNAME = "Doxygen generated docs" +DOCSET_BUNDLE_ID = org.doxygen.Project +DOCSET_PUBLISHER_ID = org.doxygen.Publisher +DOCSET_PUBLISHER_NAME = Publisher +GENERATE_HTMLHELP = NO +CHM_FILE = +HHC_LOCATION = +GENERATE_CHI = NO +CHM_INDEX_ENCODING = +BINARY_TOC = NO +TOC_EXPAND = NO +GENERATE_QHP = NO +QCH_FILE = +QHP_NAMESPACE = org.doxygen.Project +QHP_VIRTUAL_FOLDER = doc +QHP_CUST_FILTER_NAME = +QHP_CUST_FILTER_ATTRS = +QHP_SECT_FILTER_ATTRS = +QHG_LOCATION = +GENERATE_ECLIPSEHELP = NO +ECLIPSE_DOC_ID = org.doxygen.Project +DISABLE_INDEX = NO +GENERATE_TREEVIEW = YES +ENUM_VALUES_PER_LINE = 4 +TREEVIEW_WIDTH = 250 +EXT_LINKS_IN_WINDOW = YES +FORMULA_FONTSIZE = 10 +FORMULA_TRANSPARENT = YES +USE_MATHJAX = NO +MATHJAX_RELPATH = http://cdn.mathjax.org/mathjax/latest +MATHJAX_EXTENSIONS = +SEARCHENGINE = YES +SERVER_BASED_SEARCH = NO +GENERATE_LATEX = NO +LATEX_OUTPUT = latex +LATEX_CMD_NAME = latex +MAKEINDEX_CMD_NAME = makeindex +COMPACT_LATEX = NO +PAPER_TYPE = a4 +EXTRA_PACKAGES = +LATEX_HEADER = +LATEX_FOOTER = +PDF_HYPERLINKS = YES +USE_PDFLATEX = YES +LATEX_BATCHMODE = NO +LATEX_HIDE_INDICES = NO +LATEX_SOURCE_CODE = NO +LATEX_BIB_STYLE = plain +GENERATE_RTF = NO +RTF_OUTPUT = rtf +COMPACT_RTF = NO +RTF_HYPERLINKS = NO +RTF_STYLESHEET_FILE = +RTF_EXTENSIONS_FILE = +GENERATE_MAN = NO +MAN_OUTPUT = man +MAN_EXTENSION = .3 +MAN_LINKS = NO +GENERATE_XML = NO +XML_OUTPUT = xml +XML_SCHEMA = +XML_DTD = +XML_PROGRAMLISTING = YES +GENERATE_AUTOGEN_DEF = NO +GENERATE_PERLMOD = NO +PERLMOD_LATEX = NO +PERLMOD_PRETTY = YES +PERLMOD_MAKEVAR_PREFIX = +ENABLE_PREPROCESSING = YES +MACRO_EXPANSION = NO +EXPAND_ONLY_PREDEF = NO +SEARCH_INCLUDES = YES +INCLUDE_PATH = +INCLUDE_FILE_PATTERNS = +PREDEFINED = OT_CRYPTO_USING_OPENSSL \ + OT_CASH_USING_LUCRE +EXPAND_AS_DEFINED = +SKIP_FUNCTION_MACROS = YES +TAGFILES = +GENERATE_TAGFILE = +ALLEXTERNALS = NO +EXTERNAL_GROUPS = YES +PERL_PATH = /usr/bin/perl +CLASS_DIAGRAMS = NO +MSCGEN_PATH = +HIDE_UNDOC_RELATIONS = YES +HAVE_DOT = YES +DOT_NUM_THREADS = 0 +DOT_FONTNAME = Helvetica +DOT_FONTSIZE = 10 +DOT_FONTPATH = +CLASS_GRAPH = YES +COLLABORATION_GRAPH = YES +GROUP_GRAPHS = YES +UML_LOOK = NO +UML_LIMIT_NUM_FIELDS = 10 +TEMPLATE_RELATIONS = NO +INCLUDE_GRAPH = YES +INCLUDED_BY_GRAPH = YES +CALL_GRAPH = YES +CALLER_GRAPH = YES +GRAPHICAL_HIERARCHY = YES +DIRECTORY_GRAPH = YES +DOT_IMAGE_FORMAT = png +INTERACTIVE_SVG = NO +DOT_PATH = +DOTFILE_DIRS = +MSCFILE_DIRS = +DOT_GRAPH_MAX_NODES = 50 +MAX_DOT_GRAPH_DEPTH = 0 +DOT_TRANSPARENT = NO +DOT_MULTI_TARGETS = NO +GENERATE_LEGEND = YES +DOT_CLEANUP = YES diff --git a/cmake/Doxygen.extra.css.in b/cmake/Doxygen.extra.css.in new file mode 100644 index 0000000..022974f +++ b/cmake/Doxygen.extra.css.in @@ -0,0 +1,14 @@ +/* increase vertical space */ + display: none; + height: 0px; +} +/* uncomment these lines for some extra vertical space */ +/* +.tablist li { + line-height: 26px; +} +*/ diff --git a/cmake/FindBerkeleyDB.cmake b/cmake/FindBerkeleyDB.cmake new file mode 100644 index 0000000..916d8f9 +++ b/cmake/FindBerkeleyDB.cmake @@ -0,0 +1,25 @@ +find_path(BERKELEY_DB_INCLUDE_DIR db_cxx.h + /usr/include/db4 + /usr/local/include/db4 +) +find_library(BERKELEY_DB_LIBRARIES NAMES db_cxx ) +include(FindPackageHandleStandardArgs) +find_package_handle_standard_args(Berkeley "Could not find Berkeley DB >= 4.1" BERKELEY_DB_INCLUDE_DIR BERKELEY_DB_LIBRARIES) +mark_as_advanced(BERKELEY_DB_INCLUDE_DIR BERKELEY_DB_LIBRARIES ) diff --git a/cmake/FindMiniupnpc.cmake b/cmake/FindMiniupnpc.cmake new file mode 100644 index 0000000..ab525b3 +++ b/cmake/FindMiniupnpc.cmake @@ -0,0 +1,185 @@ +if (MINIUPNP_INCLUDE_DIR AND MINIUPNP_LIBRARY) + # Already in cache, be silent + set(MINIUPNP_FIND_QUIETLY TRUE) +endif () +find_path(MINIUPNP_INCLUDE_DIR miniupnpc.h + PATH_SUFFIXES miniupnpc) +find_library(MINIUPNP_LIBRARY miniupnpc) +if (MINIUPNP_INCLUDE_DIR AND MINIUPNP_LIBRARY) + set (MINIUPNP_FOUND TRUE) +endif () +if (MINIUPNP_FOUND) + include(CheckCXXSourceRuns) + if (NOT MINIUPNP_FIND_QUIETLY) + message (STATUS "Found the miniupnpc libraries at ${MINIUPNP_LIBRARY}") + message (STATUS "Found the miniupnpc headers at ${MINIUPNP_INCLUDE_DIR}") + endif () + message(STATUS "Detecting version of miniupnpc in path: ${MINIUPNP_INCLUDE_DIR}") + set(CMAKE_REQUIRED_INCLUDES ${MINIUPNP_INCLUDE_DIR}) + set(CMAKE_REQUIRED_LIBRARIES ${MINIUPNP_LIBRARY}) + check_cxx_source_runs(" + #include <miniwget.h> + #include <miniupnpc.h> + #include <upnpcommands.h> + #include <stdio.h> + int main() + { + static struct UPNPUrls urls; + static struct IGDdatas data; + GetUPNPUrls (&urls, &data, \"myurl\",0); + return 0; + }" + MINIUPNPC_VERSION_1_7_OR_HIGHER) +IF (NOT MINIUPNPC_VERSION_1_7_OR_HIGHER) + set(CMAKE_REQUIRED_INCLUDES ${MINIUPNP_INCLUDE_DIR}) + set(CMAKE_REQUIRED_LIBRARIES ${MINIUPNP_LIBRARY}) + check_cxx_source_runs(" + #include <miniwget.h> + #include <miniupnpc.h> + #include <upnpcommands.h> + #include <stdio.h> + int main() + { + struct UPNPDev *devlist = NULL; + int upnp_delay = 5000; + const char *upnp_multicastif = NULL; + const char *upnp_minissdpdsock = NULL; + int upnp_sameport = 0; + int upnp_ipv6 = 0; + int upnp_error = 0; + devlist = upnpDiscover(upnp_delay, upnp_multicastif, upnp_minissdpdsock, upnp_sameport, upnp_ipv6, &upnp_error); + return 0; + }" + MINIUPNPC_VERSION_PRE1_7) + ENDIF() + IF (NOT MINIUPNPC_VERSION_PRE1_7 AND NOT MINIUPNPC_VERSION_1_7_OR_HIGHER) + set(CMAKE_REQUIRED_INCLUDES ${MINIUPNP_INCLUDE_DIR}) + set(CMAKE_REQUIRED_LIBRARIES ${MINIUPNP_LIBRARY}) + check_cxx_source_runs(" + #include <miniwget.h> + #include <miniupnpc.h> + #include <upnpcommands.h> + #include <stdio.h> + int main() + { + struct UPNPDev *devlist = NULL; + int upnp_delay = 5000; + const char *upnp_multicastif = NULL; + const char *upnp_minissdpdsock = NULL; + int upnp_sameport = 0; + int upnp_ipv6 = 0; + int upnp_error = 0; + devlist = upnpDiscover(upnp_delay, upnp_multicastif, upnp_minissdpdsock, upnp_sameport); + return 0; + }" + MINIUPNPC_VERSION_PRE1_6) + ENDIF() + IF (NOT MINIUPNPC_VERSION_PRE1_6 AND NOT MINIUPNPC_VERSION_PRE1_7 AND NOT MINIUPNPC_VERSION_1_7_OR_HIGHER) + set(CMAKE_REQUIRED_INCLUDES ${MINIUPNP_INCLUDE_DIR}) + set(CMAKE_REQUIRED_LIBRARIES ${MINIUPNP_LIBRARY}) + check_cxx_source_runs(" + #include <miniwget.h> + #include <miniupnpc.h> + #include <upnpcommands.h> + #include <stdio.h> + static struct UPNPUrls urls; + static struct IGDdatas data; + int main() + { + char externalIP[16] = \"\"; + UPNP_GetExternalIPAddress(urls.controlURL, data.first.servicetype, externalIP); + return 0; + }" + MINIUPNPC_VERSION_1_5_OR_HIGHER) + ENDIF() + IF (NOT MINIUPNPC_VERSION_1_5_OR_HIGHER AND NOT MINIUPNPC_VERSION_PRE1_6 AND NOT MINIUPNPC_VERSION_PRE1_7 AND NOT MINIUPNPC_VERSION_1_7_OR_HIGHER) + set(CMAKE_REQUIRED_INCLUDES ${MINIUPNP_INCLUDE_DIR}) + set(CMAKE_REQUIRED_LIBRARIES ${MINIUPNP_LIBRARY}) + check_cxx_source_runs(" + #include <miniwget.h> + #include <miniupnpc.h> + #include <upnpcommands.h> + #include <stdio.h> + static struct UPNPUrls urls; + static struct IGDdatas data; + int main() + { + char externalIP[16] = \"\"; + UPNP_GetExternalIPAddress(urls.controlURL, data.servicetype, externalIP); + return 0; + }" + MINIUPNPC_VERSION_PRE1_5) +ENDIF() +IF(MINIUPNPC_VERSION_PRE1_5) + message(STATUS "Found miniupnpc version is pre v1.5") +ENDIF() +IF(MINIUPNPC_VERSION_PRE1_6) + message(STATUS "Found miniupnpc version is pre v1.6") +ENDIF() +IF(MINIUPNPC_VERSION_PRE1_7) + message(STATUS "Found miniupnpc version is pre v1.7") +ENDIF() +IF(NOT MINIUPNPC_VERSION_PRE1_5 AND NOT MINIUPNPC_VERSION_PRE1_6 AND NOT MINIUPNPC_VERSION_PRE1_7) + IF(MINIUPNPC_VERSION_1_5_OR_HIGHER) + message(STATUS "Found miniupnpc version is v1.5 or higher") + ELSE() + message(STATUS "Found miniupnpc version is v1.7 or higher") + ENDIF() +ENDIF() +else () + message (STATUS "Could not find miniupnp") +endif () +MARK_AS_ADVANCED(MINIUPNP_INCLUDE_DIR MINIUPNP_LIBRARY) diff --git a/cmake/FindUnbound.cmake b/cmake/FindUnbound.cmake new file mode 100644 index 0000000..ff6de8d +++ b/cmake/FindUnbound.cmake @@ -0,0 +1,40 @@ +MESSAGE(STATUS "Looking for libunbound") +FIND_PATH(UNBOUND_INCLUDE_DIR + NAMES unbound.h + PATH_SUFFIXES include/ include/unbound/ + PATHS "${PROJECT_SOURCE_DIR}" + ${UNBOUND_ROOT} + $ENV{UNBOUND_ROOT} + /usr/local/ + /usr/ +) +find_library(UNBOUND_LIBRARIES unbound) diff --git a/cmake/test-static-assert.c b/cmake/test-static-assert.c new file mode 100644 index 0000000..8f4cb6e +++ b/cmake/test-static-assert.c @@ -0,0 +1,34 @@ +static_assert(1, "FAIL"); +int main(int argc, char *argv[]) { + return 0; +} \ No newline at end of file diff --git a/include/INode.h b/include/INode.h index 624d4ac..c7c559d 100644 +++ b/include/INode.h @@ -1,6 +1,32 @@ #pragma once diff --git a/include/IWallet.h b/include/IWallet.h index 32d5cad..bb8ac06 100644 +++ b/include/IWallet.h @@ -1,6 +1,32 @@ #pragma once diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 61c0471..34c5291 100644 +++ b/src/CMakeLists.txt @@ -1,46 +1,109 @@ +if (WIN32 OR STATIC) + add_definitions(-DSTATICLIB) + # miniupnp changed their static define + add_definitions(-DMINIUPNP_STATICLIB) +endif () +function (bitmonero_private_headers group) + source_group("${group}\\Private" + FILES + ${ARGN}) +endfunction () +function (bitmonero_install_headers subdir) + install( + FILES ${ARGN} + DESTINATION "include/${subdir}" + COMPONENT development) +endfunction () +function (bitmonero_add_executable name) + source_group("${name}" + FILES + ${ARGN}) + add_executable("${name}" + ${ARGN}) + target_link_libraries("${name}" + LINK_PRIVATE + ${EXTRA_LIBRARIES}) + set_property(TARGET "${name}" + PROPERTY + FOLDER "prog") + set_property(TARGET "${name}" + PROPERTY + RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin") + if (STATIC) + set_property(TARGET "${name}" + PROPERTY + LINK_SEARCH_START_STATIC 1) + set_property(TARGET "${name}" + PROPERTY + LINK_SEARCH_END_STATIC 1) + endif () +endfunction () +function (bitmonero_add_library name) + source_group("${name}" + FILES + ${ARGN}) + add_library("${name}" + ${ARGN}) + set_property(TARGET "${name}" + PROPERTY + FOLDER "libs") +endfunction () +add_subdirectory(common) +add_subdirectory(crypto) +add_subdirectory(cryptonote_core) +add_subdirectory(blockchain_db) +add_subdirectory(mnemonics) +add_subdirectory(rpc) +add_subdirectory(wallet) +add_subdirectory(p2p) +add_subdirectory(cryptonote_protocol) +add_subdirectory(connectivity_tool) +add_subdirectory(miner) +add_subdirectory(simplewallet) +add_subdirectory(daemonizer) +add_subdirectory(daemon) +add_subdirectory(blockchain_utilities) +if(PER_BLOCK_CHECKPOINT) + add_subdirectory(blocks) +endif() diff --git a/src/blockchain_db/CMakeLists.txt b/src/blockchain_db/CMakeLists.txt new file mode 100644 index 0000000..7301cab +++ b/src/blockchain_db/CMakeLists.txt @@ -0,0 +1,76 @@ +set(blockchain_db_sources + blockchain_db.cpp + lmdb/db_lmdb.cpp + ) +if (BERKELEY_DB) + set(blockchain_db_sources + ${blockchain_db_sources} + berkeleydb/db_bdb.cpp + ) +endif() +set(blockchain_db_headers) +set(blockchain_db_private_headers + blockchain_db.h + lmdb/db_lmdb.h + ) +if (BERKELEY_DB) + set(blockchain_db_private_headers + ${blockchain_db_private_headers} + berkeleydb/db_bdb.h + ) +endif() +bitmonero_private_headers(blockchain_db + ${crypto_private_headers}) +bitmonero_add_library(blockchain_db + ${blockchain_db_sources} + ${blockchain_db_headers} + ${blockchain_db_private_headers}) +target_link_libraries(blockchain_db + LINK_PUBLIC + common + crypto + cryptonote_core + ${Boost_DATE_TIME_LIBRARY} + ${Boost_PROGRAM_OPTIONS_LIBRARY} + ${Boost_SERIALIZATION_LIBRARY} + ${LMDB_LIBRARY} + ${BDB_LIBRARY} + LINK_PRIVATE + ${Boost_FILESYSTEM_LIBRARY} + ${Boost_SYSTEM_LIBRARY} + ${Boost_THREAD_LIBRARY} + ${EXTRA_LIBRARIES}) diff --git a/src/blockchain_db/berkeleydb/db_bdb.cpp b/src/blockchain_db/berkeleydb/db_bdb.cpp new file mode 100644 index 0000000..ad3febb +++ b/src/blockchain_db/berkeleydb/db_bdb.cpp @@ -0,0 +1,1877 @@ +using epee::string_tools::pod_to_hex; +namespace +{ +template <typename T> +inline void throw0(const T &e) +{ + LOG_PRINT_L0(e.what()); + throw e; +} +template <typename T> +inline void throw1(const T &e) +{ + LOG_PRINT_L1(e.what()); + throw e; +} +struct bdb_cur +{ + bdb_cur(DbTxn* txn, Db* dbi) + { + if (dbi->cursor(txn, &m_cur, 0)) + throw0(cryptonote::DB_ERROR("Error opening db cursor")); + done = false; + } + ~bdb_cur() + { + close(); + } + operator Dbc*() + { + return m_cur; + } + operator Dbc**() + { + return &m_cur; + } + Dbc* operator->() + { + return m_cur; + } + void close() + { + if (!done) + { + m_cur->close(); + done = true; + } + } +private: + Dbc* m_cur; + bool done; +}; +const char* const BDB_BLOCKS = "blocks"; +const char* const BDB_BLOCK_TIMESTAMPS = "block_timestamps"; +const char* const BDB_BLOCK_HEIGHTS = "block_heights"; +const char* const BDB_BLOCK_HASHES = "block_hashes"; +const char* const BDB_BLOCK_SIZES = "block_sizes"; +const char* const BDB_BLOCK_DIFFS = "block_diffs"; +const char* const BDB_BLOCK_COINS = "block_coins"; +const char* const BDB_TXS = "txs"; +const char* const BDB_TX_UNLOCKS = "tx_unlocks"; +const char* const BDB_TX_HEIGHTS = "tx_heights"; +const char* const BDB_TX_OUTPUTS = "tx_outputs"; +const char* const BDB_OUTPUT_TXS = "output_txs"; +const char* const BDB_OUTPUT_INDICES = "output_indices"; +const char* const BDB_OUTPUT_AMOUNTS = "output_amounts"; +const char* const BDB_OUTPUT_KEYS = "output_keys"; +const char* const BDB_SPENT_KEYS = "spent_keys"; +const unsigned int MB = 1024 * 1024; +const unsigned int DB_MAX_LOCKS = 5000; +const unsigned int DB_BUFFER_LENGTH = 32 * MB; +const unsigned int DB_DEF_CACHESIZE = 256 * MB; +const unsigned int DB_BUFFER_COUNT = std::thread::hardware_concurrency(); +const unsigned int DB_BUFFER_COUNT = 1; +template<typename T> +struct Dbt_copy: public Dbt +{ + Dbt_copy(const T &t) : + t_copy(t) + { + init(); + } + Dbt_copy() + { + init(); + } + void init() + { + set_data(&t_copy); + set_size(sizeof(T)); + set_ulen(sizeof(T)); + set_flags(DB_DBT_USERMEM); + } + operator T() + { + return t_copy; + } +private: + T t_copy; +}; +template<> +struct Dbt_copy<cryptonote::blobdata>: public Dbt +{ + Dbt_copy(const cryptonote::blobdata &bd) : + m_data(new char[bd.size()]) + { + memcpy(m_data.get(), bd.data(), bd.size()); + set_data(m_data.get()); + set_size(bd.size()); + set_ulen(bd.size()); + set_flags(DB_DBT_USERMEM); + } +private: + std::unique_ptr<char[]> m_data; +}; +struct Dbt_safe : public Dbt +{ + Dbt_safe() + { + set_data(NULL); + set_flags(DB_DBT_MALLOC); + } + ~Dbt_safe() + { + void* buf = get_data(); + if (buf != NULL) + { + free(buf); + } + } +}; +} // anonymous namespace +namespace cryptonote +{ +void BlockchainBDB::add_block(const block& blk, const size_t& block_size, const difficulty_type& cumulative_difficulty, const uint64_t& coins_generated, const crypto::hash& blk_hash) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::hash> val_h(blk_hash); + if (m_block_heights->exists(DB_DEFAULT_TX, &val_h, 0) == 0) + throw1(BLOCK_EXISTS("Attempting to add block that's already in the db")); + if (m_height > 0) + { + Dbt_copy<crypto::hash> parent_key(blk.prev_id); + Dbt_copy<uint32_t> parent_h; + if (m_block_heights->get(DB_DEFAULT_TX, &parent_key, &parent_h, 0)) + { + LOG_PRINT_L3("m_height: " << m_height); + LOG_PRINT_L3("parent_key: " << blk.prev_id); + throw0(DB_ERROR("Failed to get top block hash to check for new block's parent")); + } + uint32_t parent_height = parent_h; + if (parent_height != m_height) + throw0(BLOCK_PARENT_DNE("Top block is not new block's parent")); + } + Dbt_copy<uint32_t> key(m_height + 1); + Dbt_copy<blobdata> blob(block_to_blob(blk)); + auto res = m_blocks->put(DB_DEFAULT_TX, &key, &blob, 0); + if (res) + throw0(DB_ERROR("Failed to add block blob to db transaction.")); + Dbt_copy<size_t> sz(block_size); + if (m_block_sizes->put(DB_DEFAULT_TX, &key, &sz, 0)) + throw0(DB_ERROR("Failed to add block size to db transaction.")); + Dbt_copy<uint64_t> ts(blk.timestamp); + if (m_block_timestamps->put(DB_DEFAULT_TX, &key, &ts, 0)) + throw0(DB_ERROR("Failed to add block timestamp to db transaction.")); + Dbt_copy<difficulty_type> diff(cumulative_difficulty); + if (m_block_diffs->put(DB_DEFAULT_TX, &key, &diff, 0)) + throw0(DB_ERROR("Failed to add block cumulative difficulty to db transaction.")); + Dbt_copy<uint64_t> coinsgen(coins_generated); + if (m_block_coins->put(DB_DEFAULT_TX, &key, &coinsgen, 0)) + throw0(DB_ERROR("Failed to add block total generated coins to db transaction.")); + if (m_block_heights->put(DB_DEFAULT_TX, &val_h, &key, 0)) + throw0(DB_ERROR("Failed to add block height by hash to db transaction.")); + if (m_block_hashes->put(DB_DEFAULT_TX, &key, &val_h, 0)) + throw0(DB_ERROR("Failed to add block hash to db transaction.")); +} +void BlockchainBDB::remove_block() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + if (m_height == 0) + throw0(BLOCK_DNE ("Attempting to remove block from an empty blockchain")); + Dbt_copy<uint32_t> k(m_height); + Dbt_copy<crypto::hash> h; + if (m_block_hashes->get(DB_DEFAULT_TX, &k, &h, 0)) + throw1(BLOCK_DNE("Attempting to remove block that's not in the db")); + if (m_blocks->del(DB_DEFAULT_TX, &k, 0)) + throw1(DB_ERROR("Failed to add removal of block to db transaction")); + if (m_block_sizes->del(DB_DEFAULT_TX, &k, 0)) + throw1(DB_ERROR("Failed to add removal of block size to db transaction")); + if (m_block_diffs->del(DB_DEFAULT_TX, &k, 0)) + throw1(DB_ERROR("Failed to add removal of block cumulative difficulty to db transaction")); + if (m_block_coins->del(DB_DEFAULT_TX, &k, 0)) + throw1(DB_ERROR("Failed to add removal of block total generated coins to db transaction")); + if (m_block_timestamps->del(DB_DEFAULT_TX, &k, 0)) + throw1(DB_ERROR("Failed to add removal of block timestamp to db transaction")); + if (m_block_heights->del(DB_DEFAULT_TX, &h, 0)) + throw1(DB_ERROR("Failed to add removal of block height by hash to db transaction")); + if (m_block_hashes->del(DB_DEFAULT_TX, &k, 0)) + throw1(DB_ERROR("Failed to add removal of block hash to db transaction")); +} +void BlockchainBDB::add_transaction_data(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash& tx_hash) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::hash> val_h(tx_hash); + if (m_txs->exists(DB_DEFAULT_TX, &val_h, 0) == 0) + throw1(TX_EXISTS("Attempting to add transaction that's already in the db")); + Dbt_copy<blobdata> blob(tx_to_blob(tx)); + if (m_txs->put(DB_DEFAULT_TX, &val_h, &blob, 0)) + throw0(DB_ERROR("Failed to add tx blob to db transaction")); + Dbt_copy<uint64_t> height(m_height + 1); + if (m_tx_heights->put(DB_DEFAULT_TX, &val_h, &height, 0)) + throw0(DB_ERROR("Failed to add tx block height to db transaction")); + Dbt_copy<uint64_t> unlock_time(tx.unlock_time); + if (m_tx_unlocks->put(DB_DEFAULT_TX, &val_h, &unlock_time, 0)) + throw0(DB_ERROR("Failed to add tx unlock time to db transaction")); +} +void BlockchainBDB::remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::hash> val_h(tx_hash); + if (m_txs->exists(DB_DEFAULT_TX, &val_h, 0)) + throw1(TX_DNE("Attempting to remove transaction that isn't in the db")); + if (m_txs->del(DB_DEFAULT_TX, &val_h, 0)) + throw1(DB_ERROR("Failed to add removal of tx to db transaction")); + if (m_tx_unlocks->del(DB_DEFAULT_TX, &val_h, 0)) + throw1(DB_ERROR("Failed to add removal of tx unlock time to db transaction")); + if (m_tx_heights->del(DB_DEFAULT_TX, &val_h, 0)) + throw1(DB_ERROR("Failed to add removal of tx block height to db transaction")); + remove_tx_outputs(tx_hash, tx); + if (m_tx_outputs->del(DB_DEFAULT_TX, &val_h, 0)) + throw1(DB_ERROR("Failed to add removal of tx outputs to db transaction")); +}

Continuation of: Git diff from commit e940386f9a8765423ab3dd9e3aabe19a68cba9f9 (thankful_for_today's last commit) to current HEAD, excluding changes in /external so as to exclude libraries and submoduled code, and excluding contrib/ so as to exclude changes made to epee, excluding most comments because apparently nobody that writes code for BCN knows what a comment is anyway, excluding removed lines, excluding empty lines

+void BlockchainBDB::add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<uint32_t> k(m_num_outputs + 1); + Dbt_copy<crypto::hash> v(tx_hash); + if (m_output_txs->put(DB_DEFAULT_TX, &k, &v, 0)) + throw0(DB_ERROR("Failed to add output tx hash to db transaction")); + if (m_tx_outputs->put(DB_DEFAULT_TX, &v, &k, 0)) + throw0(DB_ERROR("Failed to add tx output index to db transaction")); + Dbt_copy<uint64_t> val_local_index(local_index); + if (m_output_indices->put(DB_DEFAULT_TX, &k, &val_local_index, 0)) + throw0(DB_ERROR("Failed to add tx output index to db transaction")); + Dbt_copy<uint64_t> val_amount(tx_output.amount); + if (m_output_amounts->put(DB_DEFAULT_TX, &val_amount, &k, 0)) + throw0(DB_ERROR("Failed to add output amount to db transaction.")); + if (tx_output.target.type() == typeid(txout_to_key)) + { + output_data_t od; + od.pubkey = boost::get < txout_to_key > (tx_output.target).key; + od.unlock_time = unlock_time; + od.height = m_height; + Dbt_copy<output_data_t> data(od); + if (m_output_keys->put(DB_DEFAULT_TX, &k, &data, 0)) + throw0(DB_ERROR("Failed to add output pubkey to db transaction")); + } + m_num_outputs++; +} +void BlockchainBDB::remove_tx_outputs(const crypto::hash& tx_hash, const transaction& tx) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + bdb_cur cur(DB_DEFAULT_TX, m_tx_outputs); + Dbt_copy<crypto::hash> k(tx_hash); + Dbt_copy<uint32_t> v; + auto result = cur->get(&k, &v, DB_SET); + if (result == DB_NOTFOUND) + { + throw0(DB_ERROR("Attempting to remove a tx's outputs, but none found.")); + } + else if (result) + { + throw0(DB_ERROR("DB error attempting to get an output")); + } + else + { + db_recno_t num_elems = 0; + cur->count(&num_elems, 0); + for (uint64_t i = 0; i < num_elems; ++i) + { + const tx_out tx_output = tx.vout; + remove_output(v, tx_output.amount); + if (i < num_elems - 1) + { + cur->get(&k, &v, DB_NEXT_DUP); + } + } + } + cur.close(); +} +void BlockchainBDB::remove_output(const tx_out& tx_output) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__ << " (unused version - does nothing)"); + return; +} +void BlockchainBDB::remove_output(const uint64_t& out_index, const uint64_t amount) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<uint32_t> k(out_index); + auto result = m_output_indices->del(DB_DEFAULT_TX, &k, 0); + if (result == DB_NOTFOUND) + { + LOG_PRINT_L0("Unexpected: global output index not found in m_output_indices"); + } + else if (result) + { + throw1(DB_ERROR("Error adding removal of output tx index to db transaction")); + } + result = m_output_txs->del(DB_DEFAULT_TX, &k, 0); + // if (result != 0 && result != DB_NOTFOUND) + // throw1(DB_ERROR("Error adding removal of output tx hash to db transaction")); + if (result == DB_NOTFOUND) + { + LOG_PRINT_L0("Unexpected: global output index not found in m_output_txs"); + } + else if (result) + { + throw1(DB_ERROR("Error adding removal of output tx hash to db transaction")); + } + result = m_output_keys->del(DB_DEFAULT_TX, &k, 0); + if (result == DB_NOTFOUND) + { + LOG_PRINT_L0("Unexpected: global output index not found in m_output_keys"); + } + else if (result) + throw1(DB_ERROR("Error adding removal of output pubkey to db transaction")); + remove_amount_output_index(amount, out_index); + m_num_outputs--; +} +void BlockchainBDB::remove_amount_output_index(const uint64_t amount, const uint64_t global_output_index) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + bdb_cur cur(DB_DEFAULT_TX, m_output_amounts); + Dbt_copy<uint64_t> k(amount); + Dbt_copy<uint32_t> v; + auto result = cur->get(&k, &v, DB_SET); + if (result == DB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + db_recno_t num_elems = 0; + cur->count(&num_elems, 0); + uint64_t amount_output_index = 0; + uint64_t goi = 0; + bool found_index = false; + for (uint64_t i = 0; i < num_elems; ++i) + { + goi = v; + if (goi == global_output_index) + { + amount_output_index = i; + found_index = true; + break; + } + cur->get(&k, &v, DB_NEXT_DUP); + } + if (found_index) + { + // found the amount output index + // now delete it + result = cur->del(0); + if (result) + throw0(DB_ERROR(std::string("Error deleting amount output index ").append(boost::lexical_cast<std::string>(amount_output_index)).c_str())); + } + else + { + // not found + throw1(OUTPUT_DNE("Failed to find amount output index")); + } + cur.close(); +} +void BlockchainBDB::add_spent_key(const crypto::key_image& k_image) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::key_image> val_key(k_image); + if (m_spent_keys->exists(DB_DEFAULT_TX, &val_key, 0) == 0) + throw1(KEY_IMAGE_EXISTS("Attempting to add spent key image that's already in the db")); + Dbt_copy<char> val('\0'); + if (m_spent_keys->put(DB_DEFAULT_TX, &val_key, &val, 0)) + throw1(DB_ERROR("Error adding spent key image to db transaction.")); +} +void BlockchainBDB::remove_spent_key(const crypto::key_image& k_image) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::key_image> k(k_image); + auto result = m_spent_keys->del(DB_DEFAULT_TX, &k, 0); + if (result != 0 && result != DB_NOTFOUND) + throw1(DB_ERROR("Error adding removal of key image to db transaction")); +} +blobdata BlockchainBDB::output_to_blob(const tx_out& output) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + blobdata b; + if (!t_serializable_object_to_blob(output, b)) + throw1(DB_ERROR("Error serializing output to blob")); + return b; +} +tx_out BlockchainBDB::output_from_blob(const blobdata& blob) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + std::stringstream ss; + ss << blob; + binary_archive<false> ba(ss); + tx_out o; + if (!(::serialization::serialize(ba, o))) + throw1(DB_ERROR("Error deserializing tx output blob")); + return o; +} +uint64_t BlockchainBDB::get_output_global_index(const uint64_t& amount, const uint64_t& index) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + std::vector < uint64_t > offsets; + std::vector < uint64_t > global_indices; + offsets.push_back(index); + get_output_global_indices(amount, offsets, global_indices); + if (!global_indices.size()) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + return global_indices[0]; +} +void BlockchainBDB::check_open() const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + if (!m_open) + throw0(DB_ERROR("DB operation attempted on a not-open DB instance")); +} +BlockchainBDB::~BlockchainBDB() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + if (m_open) + { + close(); + } +} +BlockchainBDB::BlockchainBDB(bool batch_transactions) : + m_buffer(DB_BUFFER_COUNT, DB_BUFFER_LENGTH) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + // initialize folder to something "safe" just in case + // someone accidentally misuses this class... + m_folder = "thishsouldnotexistbecauseitisgibberish"; + m_open = false; + m_run_checkpoint = 0; + m_batch_transactions = batch_transactions; + m_write_txn = nullptr; + m_height = 0; +} +void BlockchainBDB::open(const std::string& filename, const int db_flags) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + if (m_open) + throw0(DB_OPEN_FAILURE("Attempted to open db, but it's already open")); + boost::filesystem::path direc(filename); + if (boost::filesystem::exists(direc)) + { + if (!boost::filesystem::is_directory(direc)) + throw0(DB_OPEN_FAILURE("DB needs a directory path, but a file was passed")); + } + else + { + if (!boost::filesystem::create_directory(direc)) + throw0(DB_OPEN_FAILURE(std::string("Failed to create directory ").append(filename).c_str())); + } + m_folder = filename; + try + { + //Create BerkeleyDB environment + m_env = new DbEnv(0); // no flags needed for DbEnv + uint32_t db_env_open_flags = DB_CREATE | DB_INIT_MPOOL | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_TXN | DB_RECOVER | DB_THREAD; + // Set some default values for these parameters. + // They can be overridden using the DB_CONFIG file. + m_env->set_cachesize(0, DB_DEF_CACHESIZE, 1); + m_env->set_lk_max_locks(DB_MAX_LOCKS); + m_env->set_lk_max_lockers(DB_MAX_LOCKS); + m_env->set_lk_max_objects(DB_MAX_LOCKS); + // last parameter left 0, files will be created with default rw access + m_env->open(filename.c_str(), db_env_open_flags, 0); + m_env->set_flags(db_flags, 1); + if(m_auto_remove_logs) + m_env->log_set_config(DB_LOG_AUTO_REMOVE, 1); + // begin transaction to init dbs + bdb_txn_safe txn; + m_env->txn_begin(NULL, txn, 0); + // create Dbs in the environment + m_blocks = new Db(m_env, 0); + m_block_heights = new Db(m_env, 0); + m_block_hashes = new Db(m_env, 0); + m_block_timestamps = new Db(m_env, 0); + m_block_sizes = new Db(m_env, 0); + m_block_diffs = new Db(m_env, 0); + m_block_coins = new Db(m_env, 0); + m_txs = new Db(m_env, 0); + m_tx_unlocks = new Db(m_env, 0); + m_tx_heights = new Db(m_env, 0); + m_tx_outputs = new Db(m_env, 0); + m_output_txs = new Db(m_env, 0); + m_output_indices = new Db(m_env, 0); + m_output_amounts = new Db(m_env, 0); + m_output_keys = new Db(m_env, 0); + m_spent_keys = new Db(m_env, 0); + // Tell DB about Dbs that need duplicate support + // Note: no need to tell about sorting, + // as the default is insertion order, which we want + m_tx_outputs->set_flags(DB_DUP); + m_output_amounts->set_flags(DB_DUP); + // Tell DB about fixed-size values. + m_block_hashes->set_re_len(sizeof(crypto::hash)); + m_block_timestamps->set_re_len(sizeof(uint64_t)); + m_block_sizes->set_re_len(sizeof(size_t)); // should really store block size as uint64_t... + m_block_diffs->set_re_len(sizeof(difficulty_type)); + m_block_coins->set_re_len(sizeof(uint64_t)); + m_output_txs->set_re_len(sizeof(crypto::hash)); + m_output_indices->set_re_len(sizeof(uint64_t)); + m_output_keys->set_re_len(sizeof(output_data_t)); + //TODO: Find out if we need to do Db::set_flags(DB_RENUMBER) + // for the RECNO databases. We shouldn't as we're only + // inserting/removing from the end, but we'll see. + // open Dbs in the environment + // m_tx_outputs and m_output_amounts must be DB_HASH or DB_BTREE + // because they need duplicate entry support. The rest are DB_RECNO, + // as it seems that will be the most performant choice. + m_blocks->open(txn, BDB_BLOCKS, NULL, DB_RECNO, DB_CREATE, 0); + m_block_timestamps->open(txn, BDB_BLOCK_TIMESTAMPS, NULL, DB_RECNO, DB_CREATE, 0); + m_block_heights->open(txn, BDB_BLOCK_HEIGHTS, NULL, DB_HASH, DB_CREATE, 0); + m_block_hashes->open(txn, BDB_BLOCK_HASHES, NULL, DB_RECNO, DB_CREATE, 0); + m_block_sizes->open(txn, BDB_BLOCK_SIZES, NULL, DB_RECNO, DB_CREATE, 0); + m_block_diffs->open(txn, BDB_BLOCK_DIFFS, NULL, DB_RECNO, DB_CREATE, 0); + m_block_coins->open(txn, BDB_BLOCK_COINS, NULL, DB_RECNO, DB_CREATE, 0); + m_txs->open(txn, BDB_TXS, NULL, DB_HASH, DB_CREATE, 0); + m_tx_unlocks->open(txn, BDB_TX_UNLOCKS, NULL, DB_HASH, DB_CREATE, 0); + m_tx_heights->open(txn, BDB_TX_HEIGHTS, NULL, DB_HASH, DB_CREATE, 0); + m_tx_outputs->open(txn, BDB_TX_OUTPUTS, NULL, DB_HASH, DB_CREATE, 0); + m_output_txs->open(txn, BDB_OUTPUT_TXS, NULL, DB_RECNO, DB_CREATE, 0); + m_output_indices->open(txn, BDB_OUTPUT_INDICES, NULL, DB_RECNO, DB_CREATE, 0); + m_output_amounts->open(txn, BDB_OUTPUT_AMOUNTS, NULL, DB_HASH, DB_CREATE, 0); + m_output_keys->open(txn, BDB_OUTPUT_KEYS, NULL, DB_RECNO, DB_CREATE, 0); + m_spent_keys->open(txn, BDB_SPENT_KEYS, NULL, DB_HASH, DB_CREATE, 0); + txn.commit(); + DB_BTREE_STAT* stats; + // DB_FAST_STAT can apparently cause an incorrect number of records + // to be returned. The flag should be set to 0 instead if this proves + // to be the case. + // ND: The bug above can occur when a block is popped and the application + // exits without pushing a new block to the db. Set txn to NULL and DB_FAST_STAT + // to zero (0) for reliability. + m_blocks->stat(NULL, &stats, 0); + m_height = stats->bt_nkeys; + delete stats; + // see above comment about DB_FAST_STAT + m_output_indices->stat(NULL, &stats, 0); + m_num_outputs = stats->bt_nkeys; + delete stats; + // run checkpoint thread + m_run_checkpoint = true; + m_checkpoint_thread.reset(new boost::thread(&BlockchainBDB::checkpoint_worker, this)); + } + catch (const std::exception& e) + { + throw0(DB_OPEN_FAILURE(e.what())); + } + m_open = true; +} +void BlockchainBDB::close() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + this->sync(); + m_run_checkpoint = false; + m_checkpoint_thread->join(); + m_checkpoint_thread.reset(); + // FIXME: not yet thread safe!!! Use with care. + m_open = false; + m_env->close(DB_FORCESYNC); +} +void BlockchainBDB::sync() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + try + { + m_blocks->sync(0); + m_block_heights->sync(0); + m_block_hashes->sync(0); + m_block_timestamps->sync(0); + m_block_sizes->sync(0); + m_block_diffs->sync(0); + m_block_coins->sync(0); + m_txs->sync(0); + m_tx_unlocks->sync(0); + m_tx_heights->sync(0); + m_tx_outputs->sync(0); + m_output_txs->sync(0); + m_output_indices->sync(0); + m_output_amounts->sync(0); + m_output_keys->sync(0); + m_spent_keys->sync(0); + } + catch (const std::exception& e) + { + throw0(DB_ERROR(std::string("Failed to sync database: ").append(e.what()).c_str())); + } +} +void BlockchainBDB::reset() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + // TODO: this +} +std::vector<std::string> BlockchainBDB::get_filenames() const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + std::vector<std::string> filenames; + char *fname, *dbname; + const char **pfname, **pdbname; + pfname = (const char **)&fname; + pdbname = (const char **)&dbname; + m_blocks->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_block_heights->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_block_hashes->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_block_timestamps->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_block_sizes->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_block_diffs->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_block_coins->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_txs->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_tx_unlocks->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_tx_heights->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_tx_outputs->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_output_txs->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_output_indices->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_output_amounts->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_output_keys->get_dbname(pfname, pdbname); + filenames.push_back(fname); + m_spent_keys->get_dbname(pfname, pdbname); + filenames.push_back(fname); + std::vector<std::string> full_paths; + for (auto& filename : filenames) + { + boost::filesystem::path p(m_folder); + p /= filename; + full_paths.push_back(p.string()); + } + return full_paths; +} +std::string BlockchainBDB::get_db_name() const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + return std::string("BerkeleyDB"); +} +bool BlockchainBDB::lock() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + return false; +} +void BlockchainBDB::unlock() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); +} +bool BlockchainBDB::block_exists(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::hash> key(h); + auto get_result = m_block_heights->exists(DB_DEFAULT_TX, &key, 0); + if (get_result == DB_NOTFOUND) + { + LOG_PRINT_L3("Block with hash " << epee::string_tools::pod_to_hex(h) << " not found in db"); + return false; + } + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch block index from hash")); + return true; +} +block BlockchainBDB::get_block(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + return get_block_from_height(get_block_height(h)); +} +uint64_t BlockchainBDB::get_block_height(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::hash> key(h); + Dbt_copy<uint32_t> result; + auto get_result = m_block_heights->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + throw1(BLOCK_DNE("Attempted to retrieve non-existent block height")); + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a block height from the db")); + return result - 1; +} +block_header BlockchainBDB::get_block_header(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + // block_header object is automatically cast from block object + return get_block(h); +} +block BlockchainBDB::get_block_from_height(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<uint32_t> key(height + 1); + Dbt_safe result; + auto get_result = m_blocks->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get block from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- block not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a block from the db")); + blobdata bd; + bd.assign(reinterpret_cast<char*>(result.get_data()), result.get_size()); + block b; + if (!parse_and_validate_block_from_blob(bd, b)) + throw0(DB_ERROR("Failed to parse block from blob retrieved from the db")); + return b; +} +uint64_t BlockchainBDB::get_block_timestamp(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<uint32_t> key(height + 1); + Dbt_copy<uint64_t> result; + auto get_result = m_block_timestamps->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get timestamp from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- timestamp not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a timestamp from the db")); + return result; +} +uint64_t BlockchainBDB::get_top_block_timestamp() const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + // if no blocks, return 0 + if (m_height == 0) + { + return 0; + } + return get_block_timestamp(m_height - 1); +} +size_t BlockchainBDB::get_block_size(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<uint32_t> key(height + 1); + Dbt_copy<size_t> result; + auto get_result = m_block_sizes->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get block size from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- block size not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a block size from the db")); + return result; +} +difficulty_type BlockchainBDB::get_block_cumulative_difficulty(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__ << " height: " << height); + check_open(); + Dbt_copy<uint32_t> key(height + 1); + Dbt_copy<difficulty_type> result; + auto get_result = m_block_diffs->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get cumulative difficulty from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- difficulty not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a cumulative difficulty from the db")); + return result; +} +difficulty_type BlockchainBDB::get_block_difficulty(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + difficulty_type diff1 = 0; + difficulty_type diff2 = 0; + diff1 = get_block_cumulative_difficulty(height); + if (height != 0) + { + diff2 = get_block_cumulative_difficulty(height - 1); + } + return diff1 - diff2; +} +uint64_t BlockchainBDB::get_block_already_generated_coins(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<uint32_t> key(height + 1); + Dbt_copy<uint64_t> result; + auto get_result = m_block_coins->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get generated coins from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- block size not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a total generated coins from the db")); + return result; +} +crypto::hash BlockchainBDB::get_block_hash_from_height(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<uint32_t> key(height + 1); + Dbt_copy<crypto::hash> result; + auto get_result = m_block_hashes->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + { + throw0(BLOCK_DNE(std::string("Attempt to get hash from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- hash not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a block hash from the db.")); + return result; +} +std::vector<block> BlockchainBDB::get_blocks_range(const uint64_t& h1, const uint64_t& h2) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + std::vector<block> v; + for (uint64_t height = h1; height <= h2; ++height) + { + v.push_back(get_block_from_height(height)); + } + return v; +} +std::vector<crypto::hash> BlockchainBDB::get_hashes_range(const uint64_t& h1, const uint64_t& h2) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + std::vector<crypto::hash> v; + for (uint64_t height = h1; height <= h2; ++height) + { + v.push_back(get_block_hash_from_height(height)); + } + return v; +} +crypto::hash BlockchainBDB::top_block_hash() const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + if (m_height > 0) + { + return get_block_hash_from_height(m_height - 1); + } + return null_hash; +} +block BlockchainBDB::get_top_block() const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + if (m_height > 0) + { + return get_block_from_height(m_height - 1); + } + block b; + return b; +} +uint64_t BlockchainBDB::height() const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + return m_height; +} +bool BlockchainBDB::tx_exists(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::hash> key(h); + TIME_MEASURE_START(time1); + auto get_result = m_txs->exists(DB_DEFAULT_TX, &key, 0); + TIME_MEASURE_FINISH(time1); + time_tx_exists += time1; + if (get_result == DB_NOTFOUND) + { + LOG_PRINT_L1("transaction with hash " << epee::string_tools::pod_to_hex(h) << " not found in db"); + return false; + } + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch transaction from hash")); + return true; +} +uint64_t BlockchainBDB::get_tx_unlock_time(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::hash> key(h); + Dbt_copy<uint64_t> result; + auto get_result = m_tx_unlocks->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + throw1(TX_DNE(std::string("tx unlock time with hash ").append(epee::string_tools::pod_to_hex(h)).append(" not found in db").c_str())); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch tx unlock time from hash")); + return result; +} +transaction BlockchainBDB::get_tx(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::hash> key(h); + Dbt_safe result; + auto get_result = m_txs->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + throw1(TX_DNE(std::string("tx with hash ").append(epee::string_tools::pod_to_hex(h)).append(" not found in db").c_str())); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch tx from hash")); + blobdata bd; + bd.assign(reinterpret_cast<char*>(result.get_data()), result.get_size()); + transaction tx; + if (!parse_and_validate_tx_from_blob(bd, tx)) + throw0(DB_ERROR("Failed to parse tx from blob retrieved from the db")); + return tx; +} +uint64_t BlockchainBDB::get_tx_count() const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + DB_BTREE_STAT* stats; + // DB_FAST_STAT can apparently cause an incorrect number of records + // to be returned. The flag should be set to 0 instead if this proves + // to be the case. + m_txs->stat(DB_DEFAULT_TX, &stats, DB_FAST_STAT); + auto num_txs = stats->bt_nkeys; + delete stats; + return num_txs; +} +std::vector<transaction> BlockchainBDB::get_tx_list(const std::vector<crypto::hash>& hlist) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + std::vector<transaction> v; +for (auto& h : hlist) + { + v.push_back(get_tx(h)); + } + return v; +} +uint64_t BlockchainBDB::get_tx_block_height(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::hash> key(h); + Dbt_copy<uint64_t> result; + auto get_result = m_tx_heights->get(DB_DEFAULT_TX, &key, &result, 0); + if (get_result == DB_NOTFOUND) + { + throw1(TX_DNE(std::string("tx height with hash ").append(epee::string_tools::pod_to_hex(h)).append(" not found in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch tx height from hash")); + return (uint64_t)result - 1; +} +uint64_t BlockchainBDB::get_random_output(const uint64_t& amount) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + uint64_t num_outputs = get_num_outputs(amount); + if (num_outputs == 0) + throw1(OUTPUT_DNE("Attempting to get a random output for an amount, but none exist")); + return crypto::rand<uint64_t>() % num_outputs; +} +uint64_t BlockchainBDB::get_num_outputs(const uint64_t& amount) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + bdb_cur cur(DB_DEFAULT_TX, m_output_amounts); + Dbt_copy<uint64_t> k(amount); + Dbt_copy<uint32_t> v; + auto result = cur->get(&k, &v, DB_SET); + if (result == DB_NOTFOUND) + { + return 0; + } + else if (result) + throw0(DB_ERROR("DB error attempting to get number of outputs of an amount")); + db_recno_t num_elems = 0; + cur->count(&num_elems, 0); + cur.close(); + return num_elems; +} +output_data_t BlockchainBDB::get_output_key(const uint64_t& global_index) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<uint32_t> k(global_index); + Dbt_copy<output_data_t> v; + auto get_result = m_output_keys->get(DB_DEFAULT_TX, &k, &v, 0); + if (get_result == DB_NOTFOUND) + throw0(DB_ERROR("Attempting to get output pubkey by global index, but key does not exist")); + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve an output pubkey from the db")); + return v; +} +output_data_t BlockchainBDB::get_output_key(const uint64_t& amount, const uint64_t& index) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + uint64_t glob_index = get_output_global_index(amount, index); + return get_output_key(glob_index); +} +tx_out BlockchainBDB::get_output(const crypto::hash& h, const uint64_t& index) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + return tx_out(); +} +tx_out BlockchainBDB::get_output(const uint64_t& index) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + return tx_out(); +} +tx_out_index BlockchainBDB::get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + std::vector < uint64_t > offsets; + std::vector<tx_out_index> indices; + offsets.push_back(index); + get_output_tx_and_index(amount, offsets, indices); + if (!indices.size()) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + return indices[0]; +} +std::vector<uint64_t> BlockchainBDB::get_tx_output_indices(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + std::vector<uint64_t> index_vec; + bdb_cur cur(DB_DEFAULT_TX, m_tx_outputs); + Dbt_copy<crypto::hash> k(h); + Dbt_copy<uint32_t> v; + auto result = cur->get(&k, &v, DB_SET); + if (result == DB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output by tx hash and tx index, but output not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + db_recno_t num_elems = 0; + cur->count(&num_elems, 0); + for (uint64_t i = 0; i < num_elems; ++i) + { + index_vec.push_back(v); + cur->get(&k, &v, DB_NEXT_DUP); + } + cur.close(); + return index_vec; +} +std::vector<uint64_t> BlockchainBDB::get_tx_amount_output_indices(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + std::vector<uint64_t> index_vec; + std::vector<uint64_t> index_vec2; + // get the transaction's global output indices first + index_vec = get_tx_output_indices(h); + // these are next used to obtain the amount output indices + transaction tx = get_tx(h); + uint64_t i = 0; + uint64_t global_index; + for (const auto& vout : tx.vout) + { + uint64_t amount = vout.amount; + global_index = index_vec; + bdb_cur cur(DB_DEFAULT_TX, m_output_amounts); + Dbt_copy<uint64_t> k(amount); + Dbt_copy<uint32_t> v; + auto result = cur->get(&k, &v, DB_SET); + if (result == DB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + db_recno_t num_elems = 0; + cur->count(&num_elems, 0); + uint64_t amount_output_index = 0; + uint64_t output_index = 0; + bool found_index = false; + for (uint64_t j = 0; j < num_elems; ++j) + { + output_index = v; + if (output_index == global_index) + { + amount_output_index = j; + found_index = true; + break; + } + cur->get(&k, &v, DB_NEXT_DUP); + } + if (found_index) + { + index_vec2.push_back(amount_output_index); + } + else + { + // not found + cur.close(); + throw1(OUTPUT_DNE("specified output not found in db")); + } + cur.close(); + ++i; + } + return index_vec2; +} +tx_out_index BlockchainBDB::get_output_tx_and_index_from_global(const uint64_t& index) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<uint32_t> k(index); + Dbt_copy<crypto::hash > v; + auto get_result = m_output_txs->get(DB_DEFAULT_TX, &k, &v, 0); + if (get_result == DB_NOTFOUND) + throw1(OUTPUT_DNE("output with given index not in db")); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch output tx hash")); + crypto::hash tx_hash = v; + Dbt_copy<uint64_t> result; + get_result = m_output_indices->get(DB_DEFAULT_TX, &k, &result, 0); + if (get_result == DB_NOTFOUND) + throw1(OUTPUT_DNE("output with given index not in db")); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch output tx index")); + return tx_out_index(tx_hash, result); +} +bool BlockchainBDB::has_key_image(const crypto::key_image& img) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + Dbt_copy<crypto::key_image> val_key(img); + if (m_spent_keys->exists(DB_DEFAULT_TX, &val_key, 0) == 0) + { + return true; + } + return false; +} +void BlockchainBDB::batch_start(uint64_t batch_num_blocks) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); +} +void BlockchainBDB::batch_commit() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); +} +void BlockchainBDB::batch_stop() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); +} +void BlockchainBDB::batch_abort() +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); +} +void BlockchainBDB::set_batch_transactions(bool batch_transactions) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + m_batch_transactions = batch_transactions; + LOG_PRINT_L3("batch transactions " << (m_batch_transactions ? "enabled" : "disabled")); +} +uint64_t BlockchainBDB::add_block(const block& blk, const size_t& block_size, const difficulty_type& cumulative_difficulty, const uint64_t& coins_generated, const std::vector<transaction>& txs) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + bdb_txn_safe txn; + if (m_env->txn_begin(NULL, txn, 0)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + m_write_txn = &txn; + uint64_t num_outputs = m_num_outputs; + try + { + BlockchainDB::add_block(blk, block_size, cumulative_difficulty, coins_generated, txs); + m_write_txn = NULL; + TIME_MEASURE_START(time1); + txn.commit(); + TIME_MEASURE_FINISH(time1); + time_commit1 += time1; + } + catch (const std::exception& e) + { + m_num_outputs = num_outputs; + m_write_txn = NULL; + throw; + } + return ++m_height; +} +void BlockchainBDB::pop_block(block& blk, std::vector<transaction>& txs) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + bdb_txn_safe txn; + if (m_env->txn_begin(NULL, txn, 0)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + m_write_txn = &txn; + uint64_t num_outputs = m_num_outputs; + try + { + BlockchainDB::pop_block(blk, txs); + m_write_txn = NULL; + txn.commit(); + } + catch (...) + { + m_num_outputs = num_outputs; + m_write_txn = NULL; + throw; + } + --m_height; +} +void BlockchainBDB::get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices, std::vector<tx_out_index> &tx_out_indices) const +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + tx_out_indices.clear(); + for (const uint64_t &index : global_indices) + { + Dbt_copy<uint32_t> k(index); + Dbt_copy<crypto::hash> v; + auto get_result = m_output_txs->get(DB_DEFAULT_TX, &k, &v, 0); + if (get_result == DB_NOTFOUND) + throw1(OUTPUT_DNE("output with given index not in db")); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch output tx hash")); + crypto::hash tx_hash = v; + Dbt_copy<uint64_t> result; + get_result = m_output_indices->get(DB_DEFAULT_TX, &k, &result, 0); + if (get_result == DB_NOTFOUND) + throw1(OUTPUT_DNE("output with given index not in db")); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch output tx index")); + auto hashindex = tx_out_index(tx_hash, result); + tx_out_indices.push_back(hashindex); + } +} +void BlockchainBDB::get_output_global_indices(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<uint64_t> &global_indices) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + TIME_MEASURE_START(txx); + check_open(); + bdb_cur cur(DB_DEFAULT_TX, m_output_amounts); + uint64_t max = 0; + for (const uint64_t& index : offsets) + { + if (index > max) + max = index; + } + // get returned keypairs count + do { \ + uint32_t *_p = (uint32_t *) ((uint8_t *)(dbt)->data + \ + (dbt)->ulen - sizeof(uint32_t)); \ + cnt = 0; \ + while(*_p != (uint32_t) -1) { \ + _p -= 2; \ + ++cnt; \ + } \ + } while(0); \ + Dbt_copy<uint64_t> k(amount); + Dbt_copy<uint32_t> v; + uint64_t buflen = 0; + uint64_t t_dbmul = 0; + uint64_t t_dbscan = 0; + auto result = cur->get(&k, &v, DB_SET); + if (result == DB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + db_recno_t num_elems = 0; + cur->count(&num_elems, 0); + if (max <= 1 && num_elems <= max) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but output not found")); + TIME_MEASURE_START(db2); + if (max <= 1) + { + for (const uint64_t& index : offsets) + { + TIME_MEASURE_START(t_seek); + auto result = cur->get(&k, &v, DB_SET); + if (result == DB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + for (uint64_t i = 0; i < index; ++i) + cur->get(&k, &v, DB_NEXT_DUP); + uint64_t glob_index = v; + LOG_PRINT_L3("L0->v: " << glob_index); + global_indices.push_back(glob_index); + TIME_MEASURE_FINISH(t_seek); + } + } + else + { + // setup a 256KB minimum buffer size + uint32_t pagesize = 256 * 1024; + // Retrieve only a suitable portion of the kvp data, up to somewhere near + // the maximum offset value being retrieved + buflen = (max + 1) * 4 * sizeof(uint64_t); + buflen = ((buflen / pagesize) + ((buflen % pagesize) > 0 ? 1 : 0)) * pagesize; + bool nomem = false; + Dbt data; + bool singlebuff = buflen <= m_buffer.get_buffer_size(); + buflen = buflen < m_buffer.get_buffer_size() ? buflen : m_buffer.get_buffer_size(); + bdb_safe_buffer_t::type buffer = nullptr; + bdb_safe_buffer_autolock<bdb_safe_buffer_t> lock(m_buffer, buffer); + data.set_data(buffer); + data.set_ulen(buflen); + data.set_size(buflen); + data.set_flags(DB_DBT_USERMEM); + uint32_t curcount = 0; + uint32_t blockstart = 0; + for (const uint64_t& index : offsets) + { + if (index >= num_elems) + { + LOG_PRINT_L1("Index: " << index << " Elems: " << num_elems << " partial results found for get_output_tx_and_index"); + break; + } + // fixme! for whatever reason, the first call to DB_MULTIPLE | DB_SET does not + // retrieve the first value.

Continuation of: Git diff from commit e940386f9a8765423ab3dd9e3aabe19a68cba9f9 (thankful_for_today's last commit) to current HEAD, excluding changes in /external so as to exclude libraries and submoduled code, and excluding contrib/ so as to exclude changes made to epee, excluding most comments because apparently nobody that writes code for BCN knows what a comment is anyway, excluding removed lines, excluding empty lines

+ if (index <= 1 || nomem) + { + auto result = cur->get(&k, &v, DB_SET); + if (result == DB_NOTFOUND) + { + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + } + else if (result) + { + throw0(DB_ERROR("DB error attempting to get an output")); + } + for (uint64_t i = 0; i < index; ++i) + cur->get(&k, &v, DB_NEXT_DUP); + } + else + { + while (index >= curcount) + { + TIME_MEASURE_START(t_db1); + try + { + cur->get(&k, &data, DB_MULTIPLE | (curcount == 0 ? DB_SET : DB_NEXT_DUP)); + blockstart = curcount; + int count = 0; + DB_COUNT_RECORDS((DBT * ) &data, count); + curcount += count; + } + catch (const std::exception &e) + { + cur.close(); + throw0(DB_ERROR(std::string("Failed on DB_MULTIPLE: ").append(e.what()).c_str())); + } + TIME_MEASURE_FINISH(t_db1); + t_dbmul += t_db1; + if (singlebuff) + break; + } + LOG_PRINT_L3("Records returned: " << curcount << " Index: " << index); + TIME_MEASURE_START(t_db2); + DBT *pdata = (DBT *) &data; + uint8_t *value; + uint64_t dlen = 0; + void *pbase = ((uint8_t *) (pdata->data)) + pdata->ulen - sizeof(uint32_t); + uint32_t *p = (uint32_t *) pbase; + if (*p == (uint32_t) -1) + { + value = NULL; + } + else + { + p -= (index - blockstart) * 2; // index * 4 + 2; <- if DB_MULTIPLE_KEY + value = (uint8_t *) pdata->data + *p--; + dlen = *p--; + if (value == (uint8_t *) pdata->data) + value = NULL; + } + if (value != NULL) + { + v = dlen == sizeof(uint64_t) ? *((uint64_t *) value) : *((uint32_t *) value); + } + TIME_MEASURE_FINISH(t_db2); + t_dbscan += t_db2; + } + uint64_t glob_index = v; + LOG_PRINT_L3("L1->v: " << glob_index); + global_indices.push_back(glob_index); + } + } + TIME_MEASURE_FINISH(db2); + cur.close(); + TIME_MEASURE_FINISH(txx); + LOG_PRINT_L3("blen: " << buflen << " txx: " << txx << " db1: " << t_dbmul << " db2: " << t_dbscan); +} +void BlockchainBDB::get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + TIME_MEASURE_START(txx); + outputs.clear(); + std::vector < uint64_t > global_indices; + get_output_global_indices(amount, offsets, global_indices); + TIME_MEASURE_START(db3); + if (global_indices.size() > 0) + { + for (const uint64_t &index : global_indices) + { + Dbt_copy<uint32_t> k(index); + Dbt_copy<output_data_t> v; + auto get_result = m_output_keys->get(DB_DEFAULT_TX, &k, &v, 0); + if (get_result == DB_NOTFOUND) + throw1(OUTPUT_DNE("output with given index not in db")); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch output tx hash")); + output_data_t data = *(output_data_t *) v.get_data(); + outputs.push_back(data); + } + } + TIME_MEASURE_FINISH(txx); + LOG_PRINT_L3("db3: " << db3); +} +void BlockchainBDB::get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) +{ + LOG_PRINT_L3("BlockchainBDB::" << __func__); + check_open(); + std::vector < uint64_t > global_indices; + get_output_global_indices(amount, offsets, global_indices); + TIME_MEASURE_START(db3); + if (global_indices.size() > 0) + get_output_tx_and_index_from_global(global_indices, indices); + TIME_MEASURE_FINISH(db3); + LOG_PRINT_L3("db3: " << db3); +} +void BlockchainBDB::checkpoint_worker() const +{ + LOG_PRINT_L0("Entering BDB checkpoint thread.") + int count = 0; + while(m_run_checkpoint && m_open) + { + // sleep every second, so we don't delay exit condition m_run_checkpoint = false + sleep(1); + // checkpoint every 5 minutes + if(count++ >= 300) + { + count = 0; + if(m_env->txn_checkpoint(0, 0, 0) != 0) + { + LOG_PRINT_L0("BDB txn_checkpoint failed.") + break; + } + } + } + LOG_PRINT_L0("Leaving BDB checkpoint thread.") +} +} // namespace cryptonote diff --git a/src/blockchain_db/berkeleydb/db_bdb.h b/src/blockchain_db/berkeleydb/db_bdb.h new file mode 100644 index 0000000..41f4bcb +++ b/src/blockchain_db/berkeleydb/db_bdb.h @@ -0,0 +1,443 @@ +namespace cryptonote +{ +struct bdb_txn_safe +{ + bdb_txn_safe() : m_txn(NULL) { } + ~bdb_txn_safe() + { + LOG_PRINT_L3("bdb_txn_safe: destructor"); + if (m_txn != NULL) + abort(); + } + void commit(std::string message = "") + { + if (message.size() == 0) + { + message = "Failed to commit a transaction to the db"; + } + if (m_txn->commit(0)) + { + m_txn = NULL; + LOG_PRINT_L0(message); + throw DB_ERROR(message.c_str()); + } + m_txn = NULL; + } + void abort() + { + LOG_PRINT_L3("bdb_txn_safe: abort()"); + if(m_txn != NULL) + { + m_txn->abort(); + m_txn = NULL; + } + else + { + LOG_PRINT_L0("WARNING: bdb_txn_safe: abort() called, but m_txn is NULL"); + } + } + operator DbTxn*() + { + return m_txn; + } + operator DbTxn**() + { + return &m_txn; + } +private: + DbTxn* m_txn; +}; +template <typename T> +class bdb_safe_buffer +{ + // limit the number of buffers to 8 + const size_t MaxAllowedBuffers = 8; +public: + bdb_safe_buffer(size_t num_buffers, size_t count) + { + if(num_buffers > MaxAllowedBuffers) + num_buffers = MaxAllowedBuffers; + set_count(num_buffers); + for (size_t i = 0; i < num_buffers; i++) + m_buffers.push_back((T) malloc(sizeof(T) * count)); + m_buffer_count = count; + } + ~bdb_safe_buffer() + { + for (size_t i = 0; i < m_buffers.size(); i++) + { + if (m_buffers) + { + free(m_buffers); + m_buffers = nullptr; + } + } + m_buffers.resize(0); + } + T acquire_buffer() + { + std::unique_lock<std::mutex> lock(m_lock); + m_cv.wait(lock, [&]{ return m_count > 0; }); + --m_count; + size_t index = -1; + for (size_t i = 0; i < m_open_slot.size(); i++) + { + if (m_open_slot) + { + m_open_slot = false; + index = i; + break; + } + } + assert(index >= 0); + T buffer = m_buffers[index]; + m_buffer_map.emplace(buffer, index); + return buffer; + } + void release_buffer(T buffer) + { + std::unique_lock<std::mutex> lock(m_lock); + assert(buffer != nullptr); + auto it = m_buffer_map.find(buffer); + if (it != m_buffer_map.end()) + { + auto index = it->second; + assert(index < m_open_slot.size()); + assert(m_open_slot[index] == false); + assert(m_count < m_open_slot.size()); + ++m_count; + m_open_slot[index] = true; + m_buffer_map.erase(it); + m_cv.notify_one(); + } + } + size_t get_buffer_size() const + { + return m_buffer_count * sizeof(T); + } + size_t get_buffer_count() const + { + return m_buffer_count; + } + typedef T type; +private: + void set_count(size_t count) + { + assert(count > 0); + m_open_slot.resize(count, true); + m_count = count; + } + std::vector<T> m_buffers; + std::unordered_map<T, size_t> m_buffer_map; + std::condition_variable m_cv; + std::vector<bool> m_open_slot; + size_t m_count; + std::mutex m_lock; + size_t m_buffer_count; +}; +template <typename T> +class bdb_safe_buffer_autolock +{ +public: + bdb_safe_buffer_autolock(T &safe_buffer, typename T::type &buffer) : + m_safe_buffer(safe_buffer), m_buffer(nullptr) + { + m_buffer = m_safe_buffer.acquire_buffer(); + buffer = m_buffer; + } + ~bdb_safe_buffer_autolock() + { + if (m_buffer != nullptr) + { + m_safe_buffer.release_buffer(m_buffer); + m_buffer = nullptr; + } + } +private: + T &m_safe_buffer; + typename T::type m_buffer; +}; +class BlockchainBDB : public BlockchainDB +{ +public: + BlockchainBDB(bool batch_transactions=false); + ~BlockchainBDB(); + virtual void open(const std::string& filename, const int db_flags); + virtual void close(); + virtual void sync(); + virtual void reset(); + virtual std::vector<std::string> get_filenames() const; + virtual std::string get_db_name() const; + virtual bool lock(); + virtual void unlock(); + virtual bool block_exists(const crypto::hash& h) const; + virtual block get_block(const crypto::hash& h) const; + virtual uint64_t get_block_height(const crypto::hash& h) const; + virtual block_header get_block_header(const crypto::hash& h) const; + virtual block get_block_from_height(const uint64_t& height) const; + virtual uint64_t get_block_timestamp(const uint64_t& height) const; + virtual uint64_t get_top_block_timestamp() const; + virtual size_t get_block_size(const uint64_t& height) const; + virtual difficulty_type get_block_cumulative_difficulty(const uint64_t& height) const; + virtual difficulty_type get_block_difficulty(const uint64_t& height) const; + virtual uint64_t get_block_already_generated_coins(const uint64_t& height) const; + virtual crypto::hash get_block_hash_from_height(const uint64_t& height) const; + virtual std::vector<block> get_blocks_range(const uint64_t& h1, const uint64_t& h2) const; + virtual std::vector<crypto::hash> get_hashes_range(const uint64_t& h1, const uint64_t& h2) const; + virtual crypto::hash top_block_hash() const; + virtual block get_top_block() const; + virtual uint64_t height() const; + virtual bool tx_exists(const crypto::hash& h) const; + virtual uint64_t get_tx_unlock_time(const crypto::hash& h) const; + virtual transaction get_tx(const crypto::hash& h) const; + virtual uint64_t get_tx_count() const; + virtual std::vector<transaction> get_tx_list(const std::vector<crypto::hash>& hlist) const; + virtual uint64_t get_tx_block_height(const crypto::hash& h) const; + virtual uint64_t get_random_output(const uint64_t& amount) const; + virtual uint64_t get_num_outputs(const uint64_t& amount) const; + virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index); + virtual output_data_t get_output_key(const uint64_t& global_index) const; + virtual void get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs); + virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const; + /** + * @brief get an output from its global index + * + * @param index global index of the output desired + * + * @return the output associated with the index. + * Will throw OUTPUT_DNE if not output has that global index. + * Will throw DB_ERROR if there is a non-specific LMDB error in fetching + */ + tx_out get_output(const uint64_t& index) const; + virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const; + virtual void get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices, + std::vector<tx_out_index> &tx_out_indices) const; + virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index); + virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices); + virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const; + virtual std::vector<uint64_t> get_tx_amount_output_indices(const crypto::hash& h) const; + virtual bool has_key_image(const crypto::key_image& img) const; + virtual uint64_t add_block( const block& blk + , const size_t& block_size + , const difficulty_type& cumulative_difficulty + , const uint64_t& coins_generated + , const std::vector<transaction>& txs + ); + virtual void set_batch_transactions(bool batch_transactions); + virtual void batch_start(uint64_t batch_num_blocks=0); + virtual void batch_commit(); + virtual void batch_stop(); + virtual void batch_abort(); + virtual void pop_block(block& blk, std::vector<transaction>& txs); + virtual bool can_thread_bulk_indices() const { return true; } + virtual bool can_thread_bulk_indices() const { return false; } +private: + virtual void add_block( const block& blk + , const size_t& block_size + , const difficulty_type& cumulative_difficulty + , const uint64_t& coins_generated + , const crypto::hash& block_hash + ); + virtual void remove_block(); + virtual void add_transaction_data(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash& tx_hash); + virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx); + virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time); + virtual void remove_output(const tx_out& tx_output); + void remove_tx_outputs(const crypto::hash& tx_hash, const transaction& tx); + void remove_output(const uint64_t& out_index, const uint64_t amount); + void remove_amount_output_index(const uint64_t amount, const uint64_t global_output_index); + virtual void add_spent_key(const crypto::key_image& k_image); + virtual void remove_spent_key(const crypto::key_image& k_image); + void get_output_global_indices(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<uint64_t> &global_indices); + /** + * @brief convert a tx output to a blob for storage + * + * @param output the output to convert + * + * @return the resultant blob + */ + blobdata output_to_blob(const tx_out& output) const; + /** + * @brief convert a tx output blob to a tx output + * + * @param blob the blob to convert + * + * @return the resultant tx output + */ + tx_out output_from_blob(const blobdata& blob) const; + /** + * @brief get the global index of the index-th output of the given amount + * + * @param amount the output amount + * @param index the index into the set of outputs of that amount + * + * @return the global index of the desired output + */ + uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index); + void checkpoint_worker() const; + void check_open() const; + bool m_run_checkpoint; + std::unique_ptr<boost::thread> m_checkpoint_thread; + typedef bdb_safe_buffer<void *> bdb_safe_buffer_t; + bdb_safe_buffer_t m_buffer; + DbEnv* m_env; + Db* m_blocks; + Db* m_block_heights; + Db* m_block_hashes; + Db* m_block_timestamps; + Db* m_block_sizes; + Db* m_block_diffs; + Db* m_block_coins; + Db* m_txs; + Db* m_tx_unlocks; + Db* m_tx_heights; + Db* m_tx_outputs; + Db* m_output_txs; + Db* m_output_indices; + Db* m_output_amounts; + Db* m_output_keys; + Db* m_spent_keys; + uint64_t m_height; + uint64_t m_num_outputs; + std::string m_folder; + bdb_txn_safe *m_write_txn; + bool m_batch_transactions; // support for batch transactions +}; +} // namespace cryptonote diff --git a/src/blockchain_db/blockchain_db.cpp b/src/blockchain_db/blockchain_db.cpp new file mode 100644 index 0000000..f07f300 +++ b/src/blockchain_db/blockchain_db.cpp @@ -0,0 +1,185 @@ +using epee::string_tools::pod_to_hex; +namespace cryptonote +{ +void BlockchainDB::pop_block() +{ + block blk; + std::vector<transaction> txs; + pop_block(blk, txs); +} +void BlockchainDB::add_transaction(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash* tx_hash_ptr) +{ + crypto::hash tx_hash; + if (!tx_hash_ptr) + { + // should only need to compute hash for miner transactions + tx_hash = get_transaction_hash(tx); + LOG_PRINT_L3("null tx_hash_ptr - needed to compute: " << tx_hash); + } + else + { + tx_hash = *tx_hash_ptr; + } + add_transaction_data(blk_hash, tx, tx_hash); + // iterate tx.vout using indices instead of C++11 foreach syntax because + // we need the index + if (tx.vout.size() != 0) // it may be technically possible for a tx to have no outputs + { + for (uint64_t i = 0; i < tx.vout.size(); ++i) + { + add_output(tx_hash, tx.vout, i, tx.unlock_time); + } + for (const txin_v& tx_input : tx.vin) + { + if (tx_input.type() == typeid(txin_to_key)) + { + add_spent_key(boost::get<txin_to_key>(tx_input).k_image); + } + } + } +} +uint64_t BlockchainDB::add_block( const block& blk + , const size_t& block_size + , const difficulty_type& cumulative_difficulty + , const uint64_t& coins_generated + , const std::vector<transaction>& txs + ) +{ + TIME_MEASURE_START(time1); + crypto::hash blk_hash = get_block_hash(blk); + TIME_MEASURE_FINISH(time1); + time_blk_hash += time1; + // call out to subclass implementation to add the block & metadata + time1 = epee::misc_utils::get_tick_count(); + add_block(blk, block_size, cumulative_difficulty, coins_generated, blk_hash); + TIME_MEASURE_FINISH(time1); + time_add_block1 += time1; + // call out to add the transactions + time1 = epee::misc_utils::get_tick_count(); + add_transaction(blk_hash, blk.miner_tx); + int tx_i = 0; + crypto::hash tx_hash = null_hash; + for (const transaction& tx : txs) + { + tx_hash = blk.tx_hashes[tx_i]; + add_transaction(blk_hash, tx, &tx_hash); + ++tx_i; + } + TIME_MEASURE_FINISH(time1); + time_add_transaction += time1; + ++num_calls; + return height(); +} +void BlockchainDB::pop_block(block& blk, std::vector<transaction>& txs) +{ + blk = get_top_block(); + remove_block(); + remove_transaction(get_transaction_hash(blk.miner_tx)); + for (const auto& h : blk.tx_hashes) + { + txs.push_back(get_tx(h)); + remove_transaction(h); + } +} +bool BlockchainDB::is_open() const +{ + return m_open; +} +void BlockchainDB::remove_transaction(const crypto::hash& tx_hash) +{ + transaction tx = get_tx(tx_hash); + for (const txin_v& tx_input : tx.vin) + { + if (tx_input.type() == typeid(txin_to_key)) + { + remove_spent_key(boost::get<txin_to_key>(tx_input).k_image); + } + } + // need tx as tx.vout has the tx outputs, and the output amounts are needed + remove_transaction_data(tx_hash, tx); +} +void BlockchainDB::reset_stats() +{ + num_calls = 0; + time_blk_hash = 0; + time_tx_exists = 0; + time_add_block1 = 0; + time_add_transaction = 0; + time_commit1 = 0; +} +void BlockchainDB::show_stats() +{ + LOG_PRINT_L1(ENDL + << "*********************************" + << ENDL + << "num_calls: " << num_calls + << ENDL + << "time_blk_hash: " << time_blk_hash << "ms" + << ENDL + << "time_tx_exists: " << time_tx_exists << "ms" + << ENDL + << "time_add_block1: " << time_add_block1 << "ms" + << ENDL + << "time_add_transaction: " << time_add_transaction << "ms" + << ENDL + << "time_commit1: " << time_commit1 << "ms" + << ENDL + << "*********************************" + << ENDL + ); +} +} // namespace cryptonote diff --git a/src/blockchain_db/blockchain_db.h b/src/blockchain_db/blockchain_db.h new file mode 100644 index 0000000..ff15109 +++ b/src/blockchain_db/blockchain_db.h @@ -0,0 +1,509 @@ +/* DB Driver Interface + * + * The DB interface is a store for the canonical block chain. + * It serves as a persistent storage for the blockchain. + * + * For the sake of efficiency, the reference implementation will also + * store some blockchain data outside of the blocks, such as spent + * transfer key images, unspent transaction outputs, etc. + * + * Transactions are duplicated so that we don't have to fetch a whole block + * in order to fetch a transaction from that block. If this is deemed + * unnecessary later, this can change. + * + * Spent key images are duplicated outside of the blocks so it is quick + * to verify an output hasn't already been spent + * + * Unspent transaction outputs are duplicated to quickly gather random + * outputs to use for mixins + * + * IMPORTANT: + * A concrete implementation of this interface should populate these + * duplicated members! It is possible to have a partial implementation + * of this interface call to private members of the interface to be added + * later that will then populate as needed. + * + * General: + * open() + * is_open() + * close() + * sync() + * reset() + * + * Lock and unlock provided for reorg externally, and for block + * additions internally, this way threaded reads are completely fine + * unless the blockchain is changing. + * bool lock() + * unlock() + * + * vector<str> get_filenames() + * + * Blocks: + * bool block_exists(hash) + * height add_block(block, block_size, cumulative_difficulty, coins_generated, transactions) + * block get_block(hash) + * height get_block_height(hash) + * header get_block_header(hash) + * block get_block_from_height(height) + * size_t get_block_size(height) + * difficulty get_block_cumulative_difficulty(height) + * uint64_t get_block_already_generated_coins(height) + * uint64_t get_block_timestamp(height) + * uint64_t get_top_block_timestamp() + * hash get_block_hash_from_height(height) + * blocks get_blocks_range(height1, height2) + * hashes get_hashes_range(height1, height2) + * hash top_block_hash() + * block get_top_block() + * height height() + * void pop_block(block&, tx_list&) + * + * Transactions: + * bool tx_exists(hash) + * uint64_t get_tx_unlock_time(hash) + * tx get_tx(hash) + * uint64_t get_tx_count() + * tx_list get_tx_list(hash_list) + * height get_tx_block_height(hash) + * + * Outputs: + * index get_random_output(amount) + * uint64_t get_num_outputs(amount) + * pub_key get_output_key(amount, index) + * tx_out get_output(tx_hash, index) + * hash,index get_output_tx_and_index_from_global(index) + * hash,index get_output_tx_and_index(amount, index) + * vec<uint64> get_tx_output_indices(tx_hash) + * + * + * Spent Output Key Images: + * bool has_key_image(key_image) + * + * Exceptions: + * DB_ERROR -- generic + * DB_OPEN_FAILURE + * DB_CREATE_FAILURE + * DB_SYNC_FAILURE + * BLOCK_DNE + * BLOCK_PARENT_DNE + * BLOCK_EXISTS + * BLOCK_INVALID -- considering making this multiple errors + * TX_DNE + * TX_EXISTS + * OUTPUT_DNE + * OUTPUT_EXISTS + * KEY_IMAGE_EXISTS + */ +namespace cryptonote +{ +typedef std::pair<crypto::hash, uint64_t> tx_out_index; +struct output_data_t +{ + crypto::public_key pubkey; + uint64_t unlock_time; + uint64_t height; +}; +/*********************************** + * Exception Definitions + ***********************************/ +class DB_EXCEPTION : public std::exception +{ + private: + std::string m; + protected: + DB_EXCEPTION(const char *s) : m(s) { } + public: + virtual ~DB_EXCEPTION() { } + const char* what() const throw() + { + return m.c_str(); + } +}; +class DB_ERROR : public DB_EXCEPTION +{ + public: + DB_ERROR() : DB_EXCEPTION("Generic DB Error") { } + DB_ERROR(const char* s) : DB_EXCEPTION(s) { } +}; +class DB_OPEN_FAILURE : public DB_EXCEPTION +{ + public: + DB_OPEN_FAILURE() : DB_EXCEPTION("Failed to open the db") { } + DB_OPEN_FAILURE(const char* s) : DB_EXCEPTION(s) { } +}; +class DB_CREATE_FAILURE : public DB_EXCEPTION +{ + public: + DB_CREATE_FAILURE() : DB_EXCEPTION("Failed to create the db") { } + DB_CREATE_FAILURE(const char* s) : DB_EXCEPTION(s) { } +}; +class DB_SYNC_FAILURE : public DB_EXCEPTION +{ + public: + DB_SYNC_FAILURE() : DB_EXCEPTION("Failed to sync the db") { } + DB_SYNC_FAILURE(const char* s) : DB_EXCEPTION(s) { } +}; +class BLOCK_DNE : public DB_EXCEPTION +{ + public: + BLOCK_DNE() : DB_EXCEPTION("The block requested does not exist") { } + BLOCK_DNE(const char* s) : DB_EXCEPTION(s) { } +}; +class BLOCK_PARENT_DNE : public DB_EXCEPTION +{ + public: + BLOCK_PARENT_DNE() : DB_EXCEPTION("The parent of the block does not exist") { } + BLOCK_PARENT_DNE(const char* s) : DB_EXCEPTION(s) { } +}; +class BLOCK_EXISTS : public DB_EXCEPTION +{ + public: + BLOCK_EXISTS() : DB_EXCEPTION("The block to be added already exists!") { } + BLOCK_EXISTS(const char* s) : DB_EXCEPTION(s) { } +}; +class BLOCK_INVALID : public DB_EXCEPTION +{ + public: + BLOCK_INVALID() : DB_EXCEPTION("The block to be added did not pass validation!") { } + BLOCK_INVALID(const char* s) : DB_EXCEPTION(s) { } +}; +class TX_DNE : public DB_EXCEPTION +{ + public: + TX_DNE() : DB_EXCEPTION("The transaction requested does not exist") { } + TX_DNE(const char* s) : DB_EXCEPTION(s) { } +}; +class TX_EXISTS : public DB_EXCEPTION +{ + public: + TX_EXISTS() : DB_EXCEPTION("The transaction to be added already exists!") { } + TX_EXISTS(const char* s) : DB_EXCEPTION(s) { } +}; +class OUTPUT_DNE : public DB_EXCEPTION +{ + public: + OUTPUT_DNE() : DB_EXCEPTION("The output requested does not exist!") { } + OUTPUT_DNE(const char* s) : DB_EXCEPTION(s) { } +}; +class OUTPUT_EXISTS : public DB_EXCEPTION +{ + public: + OUTPUT_EXISTS() : DB_EXCEPTION("The output to be added already exists!") { } + OUTPUT_EXISTS(const char* s) : DB_EXCEPTION(s) { } +}; +class KEY_IMAGE_EXISTS : public DB_EXCEPTION +{ + public: + KEY_IMAGE_EXISTS() : DB_EXCEPTION("The spent key image to be added already exists!") { } + KEY_IMAGE_EXISTS(const char* s) : DB_EXCEPTION(s) { } +}; +/*********************************** + * End of Exception Definitions + ***********************************/ +class BlockchainDB +{ +private: + /********************************************************************* + * private virtual members + *********************************************************************/ + // tells the subclass to add the block and metadata to storage + virtual void add_block( const block& blk + , const size_t& block_size + , const difficulty_type& cumulative_difficulty + , const uint64_t& coins_generated + , const crypto::hash& blk_hash + ) = 0; + // tells the subclass to remove data about the top block + virtual void remove_block() = 0; + // tells the subclass to store the transaction and its metadata + virtual void add_transaction_data(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash& tx_hash) = 0; + // tells the subclass to remove data about a transaction + virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx) = 0; + // tells the subclass to store an output + virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time) = 0; + // tells the subclass to remove an output + virtual void remove_output(const tx_out& tx_output) = 0; + // tells the subclass to store a spent key + virtual void add_spent_key(const crypto::key_image& k_image) = 0; + // tells the subclass to remove a spent key + virtual void remove_spent_key(const crypto::key_image& k_image) = 0; + /********************************************************************* + * private concrete members + *********************************************************************/ + // private version of pop_block, for undoing if an add_block goes tits up + void pop_block(); + // helper function for add_transactions, to add each individual tx + void add_transaction(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash* tx_hash_ptr = NULL); + // helper function to remove transaction from blockchain + void remove_transaction(const crypto::hash& tx_hash); + uint64_t num_calls = 0; + uint64_t time_blk_hash = 0; + uint64_t time_add_block1 = 0; + uint64_t time_add_transaction = 0; +protected: + mutable uint64_t time_tx_exists = 0; + uint64_t time_commit1 = 0; + bool m_auto_remove_logs = true; +public: + // virtual dtor + virtual ~BlockchainDB() { }; + // reset profiling stats + void reset_stats(); + // show profiling stats + void show_stats(); + // open the db at location <filename>, or create it if there isn't one. + virtual void open(const std::string& filename, const int db_flags = 0) = 0; + // returns true of the db is open/ready, else false + bool is_open() const; + // close and sync the db + virtual void close() = 0; + // sync the db + virtual void sync() = 0; + // reset the db -- USE WITH CARE + virtual void reset() = 0; + // get all files used by this db (if any) + virtual std::vector<std::string> get_filenames() const = 0; + // return the name of the folder the db's file(s) should reside in + virtual std::string get_db_name() const = 0; + // FIXME: these are just for functionality mocking, need to implement + // RAII-friendly and multi-read one-write friendly locking mechanism + // + // acquire db lock + virtual bool lock() = 0; + // release db lock + virtual void unlock() = 0; + virtual void batch_start(uint64_t batch_num_blocks=0) = 0; + virtual void batch_stop() = 0; + virtual void set_batch_transactions(bool) = 0; + // adds a block with the given metadata to the top of the blockchain, returns the new height + // NOTE: subclass implementations of this (or the functions it calls) need + // to handle undoing any partially-added blocks in the event of a failure. + virtual uint64_t add_block( const block& blk + , const size_t& block_size + , const difficulty_type& cumulative_difficulty + , const uint64_t& coins_generated + , const std::vector<transaction>& txs + ); + // return true if a block with hash <h> exists in the blockchain + virtual bool block_exists(const crypto::hash& h) const = 0; + // return block with hash <h> + virtual block get_block(const crypto::hash& h) const = 0; + // return the height of the block with hash <h> on the blockchain, + // throw if it doesn't exist + virtual uint64_t get_block_height(const crypto::hash& h) const = 0; + // return header for block with hash <h> + virtual block_header get_block_header(const crypto::hash& h) const = 0; + // return block at height <height> + virtual block get_block_from_height(const uint64_t& height) const = 0; + // return timestamp of block at height <height> + virtual uint64_t get_block_timestamp(const uint64_t& height) const = 0; + // return timestamp of most recent block + virtual uint64_t get_top_block_timestamp() const = 0; + // return block size of block at height <height> + virtual size_t get_block_size(const uint64_t& height) const = 0; + // return cumulative difficulty up to and including block at height <height> + virtual difficulty_type get_block_cumulative_difficulty(const uint64_t& height) const = 0; + // return difficulty of block at height <height> + virtual difficulty_type get_block_difficulty(const uint64_t& height) const = 0; + // return number of coins generated up to and including block at height <height> + virtual uint64_t get_block_already_generated_coins(const uint64_t& height) const = 0; + // return hash of block at height <height> + virtual crypto::hash get_block_hash_from_height(const uint64_t& height) const = 0; + // return vector of blocks in range <h1,h2> of height (inclusively) + virtual std::vector<block> get_blocks_range(const uint64_t& h1, const uint64_t& h2) const = 0; + // return vector of block hashes in range <h1, h2> of height (inclusively) + virtual std::vector<crypto::hash> get_hashes_range(const uint64_t& h1, const uint64_t& h2) const = 0; + // return the hash of the top block on the chain + virtual crypto::hash top_block_hash() const = 0; + // return the block at the top of the blockchain + virtual block get_top_block() const = 0; + // return the height of the chain + virtual uint64_t height() const = 0; + // pops the top block off the blockchain. + // Returns by reference the popped block and its associated transactions + // + // IMPORTANT: + // When a block is popped, the transactions associated with it need to be + // removed, as well as outputs and spent key images associated with + // those transactions. + virtual void pop_block(block& blk, std::vector<transaction>& txs); + // return true if a transaction with hash <h> exists + virtual bool tx_exists(const crypto::hash& h) const = 0; + // return unlock time of tx with hash <h> + virtual uint64_t get_tx_unlock_time(const crypto::hash& h) const = 0; + // return tx with hash <h> + // throw if no such tx exists + virtual transaction get_tx(const crypto::hash& h) const = 0; + // returns the total number of transactions in all blocks + virtual uint64_t get_tx_count() const = 0; + // return list of tx with hashes <hlist>. + // TODO: decide if a missing hash means return empty list + // or just skip that hash + virtual std::vector<transaction> get_tx_list(const std::vector<crypto::hash>& hlist) const = 0; + // returns height of block that contains transaction with hash <h> + virtual uint64_t get_tx_block_height(const crypto::hash& h) const = 0; + // return global output index of a random output of amount <amount> + virtual uint64_t get_random_output(const uint64_t& amount) const = 0; + // returns the total number of outputs of amount <amount> + virtual uint64_t get_num_outputs(const uint64_t& amount) const = 0; + // return public key for output with global output amount <amount> and index <index> + virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index) = 0; + virtual output_data_t get_output_key(const uint64_t& global_index) const = 0; + // returns the output indexed by <index> in the transaction with hash <h> + virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const = 0; + // returns the tx hash associated with an output, referenced by global output index + virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const = 0; + // returns the transaction-local reference for the output with <amount> at <index> + // return type is pair of tx hash and index + virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) = 0; + virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) = 0; + virtual void get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs) = 0; + virtual bool can_thread_bulk_indices() const = 0; + // return a vector of indices corresponding to the global output index for + // each output in the transaction with hash <h> + virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const = 0; + // return a vector of indices corresponding to the amount output index for + // each output in the transaction with hash <h> + virtual std::vector<uint64_t> get_tx_amount_output_indices(const crypto::hash& h) const = 0; + // returns true if key image <img> is present in spent key images storage + virtual bool has_key_image(const crypto::key_image& img) const = 0; + void set_auto_remove_logs(bool auto_remove) { m_auto_remove_logs = auto_remove; } + bool m_open; + mutable epee::critical_section m_synchronization_lock; +}; // class BlockchainDB +} // namespace cryptonote diff --git a/src/blockchain_db/db_types.h b/src/blockchain_db/db_types.h new file mode 100644 index 0000000..b13007d +++ b/src/blockchain_db/db_types.h @@ -0,0 +1,40 @@ +namespace cryptonote +{ + const std::unordered_set<std::string> blockchain_db_types = + { "lmdb" + , "berkeley" + }; +} // namespace cryptonote diff --git a/src/blockchain_db/lmdb/db_lmdb.cpp b/src/blockchain_db/lmdb/db_lmdb.cpp new file mode 100644 index 0000000..dd829f3 +++ b/src/blockchain_db/lmdb/db_lmdb.cpp @@ -0,0 +1,2346 @@ +using epee::string_tools::pod_to_hex; +namespace +{ +template <typename T> +inline void throw0(const T &e) +{ + LOG_PRINT_L0(e.what()); + throw e; +} +template <typename T> +inline void throw1(const T &e) +{ + LOG_PRINT_L1(e.what()); + throw e; +} +struct lmdb_cur +{ + lmdb_cur(MDB_txn* txn, MDB_dbi dbi) + { + if (mdb_cursor_open(txn, dbi, &m_cur)) + throw0(cryptonote::DB_ERROR("Error opening db cursor")); + done = false; + } + ~lmdb_cur() + { + close(); + } + operator MDB_cursor*() + { + return m_cur; + } + operator MDB_cursor**() + { + return &m_cur; + } + void close() + { + if (!done) + { + mdb_cursor_close(m_cur); + done = true; + } + } +private: + MDB_cursor* m_cur; + bool done; +}; +template<typename T> +struct MDB_val_copy: public MDB_val +{ + MDB_val_copy(const T &t) : + t_copy(t) + { + mv_size = sizeof (T); + mv_data = &t_copy; + } +private: + T t_copy; +}; +template<> +struct MDB_val_copy<cryptonote::blobdata>: public MDB_val +{ + MDB_val_copy(const cryptonote::blobdata &bd) : + data(new char[bd.size()]) + { + memcpy(data.get(), bd.data(), bd.size()); + mv_size = bd.size(); + mv_data = data.get(); + } +private: + std::unique_ptr<char[]> data; +}; +auto compare_uint64 = [](const MDB_val *a, const MDB_val *b) +{ + const uint64_t va = *(const uint64_t*)a->mv_data; + const uint64_t vb = *(const uint64_t*)b->mv_data; + if (va < vb) return -1; + else if (va == vb) return 0; + else return 1; +}; +int compare_hash32(const MDB_val *a, const MDB_val *b) +{ + uint32_t *va = (uint32_t*) a->mv_data; + uint32_t *vb = (uint32_t*) b->mv_data; + for (int n = 7; n >= 0; n--) + { + if (va[n] == vb[n]) + continue; + return va[n] < vb[n] ? -1 : 1; + } + return 0; +} +const char* const LMDB_BLOCKS = "blocks"; +const char* const LMDB_BLOCK_TIMESTAMPS = "block_timestamps"; +const char* const LMDB_BLOCK_HEIGHTS = "block_heights"; +const char* const LMDB_BLOCK_HASHES = "block_hashes"; +const char* const LMDB_BLOCK_SIZES = "block_sizes"; +const char* const LMDB_BLOCK_DIFFS = "block_diffs"; +const char* const LMDB_BLOCK_COINS = "block_coins"; +const char* const LMDB_TXS = "txs"; +const char* const LMDB_TX_UNLOCKS = "tx_unlocks"; +const char* const LMDB_TX_HEIGHTS = "tx_heights"; +const char* const LMDB_TX_OUTPUTS = "tx_outputs"; +const char* const LMDB_OUTPUT_TXS = "output_txs"; +const char* const LMDB_OUTPUT_INDICES = "output_indices"; +const char* const LMDB_OUTPUT_AMOUNTS = "output_amounts"; +const char* const LMDB_OUTPUT_KEYS = "output_keys"; +const char* const LMDB_OUTPUTS = "outputs"; +const char* const LMDB_OUTPUT_GINDICES = "output_gindices"; +const char* const LMDB_SPENT_KEYS = "spent_keys";

Continuation of: Git diff from commit e940386f9a8765423ab3dd9e3aabe19a68cba9f9 (thankful_for_today's last commit) to current HEAD, excluding changes in /external so as to exclude libraries and submoduled code, and excluding contrib/ so as to exclude changes made to epee, excluding most comments because apparently nobody that writes code for BCN knows what a comment is anyway, excluding removed lines, excluding empty lines

+inline void lmdb_db_open(MDB_txn* txn, const char* name, int flags, MDB_dbi& dbi, const std::string& error_string) +{ + if (mdb_dbi_open(txn, name, flags, &dbi)) + throw0(cryptonote::DB_OPEN_FAILURE(error_string.c_str())); +} +} // anonymous namespace +namespace cryptonote +{ +std::atomic<uint64_t> mdb_txn_safe::num_active_txns{0}; +std::atomic_flag mdb_txn_safe::creation_gate = ATOMIC_FLAG_INIT; +mdb_txn_safe::mdb_txn_safe() : m_txn(NULL) +{ + while (creation_gate.test_and_set()); + num_active_txns++; + creation_gate.clear(); +} +mdb_txn_safe::~mdb_txn_safe() +{ + LOG_PRINT_L3("mdb_txn_safe: destructor"); + if (m_txn != NULL) + { + if (m_batch_txn) // this is a batch txn and should have been handled before this point for safety + { + LOG_PRINT_L0("WARNING: mdb_txn_safe: m_txn is a batch txn and it's not NULL in destructor - calling mdb_txn_abort()"); + } + else + { + // Example of when this occurs: a lookup fails, so a read-only txn is + // aborted through this destructor. However, successful read-only txns + // ideally should have been committed when done and not end up here. + // + // NOTE: not sure if this is ever reached for a non-batch write + // transaction, but it's probably not ideal if it did. + LOG_PRINT_L3("mdb_txn_safe: m_txn not NULL in destructor - calling mdb_txn_abort()"); + } + mdb_txn_abort(m_txn); + } + num_active_txns--; +} +void mdb_txn_safe::commit(std::string message) +{ + if (message.size() == 0) + { + message = "Failed to commit a transaction to the db"; + } + if (auto result = mdb_txn_commit(m_txn)) + { + m_txn = NULL; + throw0(DB_ERROR((message + ": ").append(mdb_strerror(result)).c_str())); + } + m_txn = NULL; +} +void mdb_txn_safe::abort() +{ + LOG_PRINT_L3("mdb_txn_safe: abort()"); + if(m_txn != NULL) + { + mdb_txn_abort(m_txn); + m_txn = NULL; + } + else + { + LOG_PRINT_L0("WARNING: mdb_txn_safe: abort() called, but m_txn is NULL"); + } +} +uint64_t mdb_txn_safe::num_active_tx() const +{ + return num_active_txns; +} +void mdb_txn_safe::prevent_new_txns() +{ + while (creation_gate.test_and_set()); +} +void mdb_txn_safe::wait_no_active_txns() +{ + while (num_active_txns > 0); +} +void mdb_txn_safe::allow_new_txns() +{ + creation_gate.clear(); +} +void BlockchainLMDB::do_resize(uint64_t increase_size) +{ + CRITICAL_REGION_LOCAL(m_synchronization_lock); + const uint64_t add_size = 1LL << 30; + // check disk capacity + try + { + boost::filesystem::path path(m_folder); + boost::filesystem::space_info si = boost::filesystem::space(path); + if(si.available < add_size) + { + LOG_PRINT_RED_L0("!! WARNING: Insufficient free space to extend database !!: " << si.available / 1LL << 20L); + return; + } + } + catch(...) + { + // print something but proceed. + LOG_PRINT_YELLOW("Unable to query free disk space.", LOG_LEVEL_0); + } + MDB_envinfo mei; + mdb_env_info(m_env, &mei); + MDB_stat mst; + mdb_env_stat(m_env, &mst); + // add 1Gb per resize, instead of doing a percentage increase + uint64_t new_mapsize = (double) mei.me_mapsize + add_size; + // If given, use increase_size intead of above way of resizing. + // This is currently used for increasing by an estimated size at start of new + // batch txn. + if (increase_size > 0) + new_mapsize = mei.me_mapsize + increase_size; + new_mapsize += (new_mapsize % mst.ms_psize); + mdb_txn_safe::prevent_new_txns(); + if (m_write_txn != nullptr) + { + if (m_batch_active) + { + throw0(DB_ERROR("lmdb resizing not yet supported when batch transactions enabled!")); + } + else + { + throw0(DB_ERROR("attempting resize with write transaction in progress, this should not happen!")); + } + } + mdb_txn_safe::wait_no_active_txns(); + mdb_env_set_mapsize(m_env, new_mapsize); + LOG_PRINT_GREEN("LMDB Mapsize increased." << " Old: " << mei.me_mapsize / (1024 * 1024) << "MiB" << ", New: " << new_mapsize / (1024 * 1024) << "MiB", LOG_LEVEL_0); + mdb_txn_safe::allow_new_txns(); +} +bool BlockchainLMDB::need_resize(uint64_t threshold_size) const +{ + MDB_envinfo mei; + mdb_env_info(m_env, &mei); + MDB_stat mst; + mdb_env_stat(m_env, &mst); + // size_used doesn't include data yet to be committed, which can be + // significant size during batch transactions. For that, we estimate the size + // needed at the beginning of the batch transaction and pass in the + // additional size needed. + uint64_t size_used = mst.ms_psize * mei.me_last_pgno; + LOG_PRINT_L1("DB map size: " << mei.me_mapsize); + LOG_PRINT_L1("Space used: " << size_used); + LOG_PRINT_L1("Space remaining: " << mei.me_mapsize - size_used); + LOG_PRINT_L1("Size threshold: " << threshold_size); + LOG_PRINT_L1("Percent used: " << (double)size_used/mei.me_mapsize << " Percent threshold: " << RESIZE_PERCENT); + if (threshold_size > 0) + { + if (mei.me_mapsize - size_used < threshold_size) + { + LOG_PRINT_L1("Threshold met (size-based)"); + return true; + } + else + return false; + } + std::mt19937 engine(std::random_device{}()); + std::uniform_real_distribution<double> fdis(0.6, 0.9); + double resize_percent = fdis(engine); + if ((double)size_used / mei.me_mapsize > resize_percent) + { + LOG_PRINT_L1("Threshold met (percent-based)"); + return true; + } + return false; + return false; +} +void BlockchainLMDB::check_and_resize_for_batch(uint64_t batch_num_blocks) +{ + LOG_PRINT_L1("[batch] checking DB size"); + const uint64_t min_increase_size = 128 * (1 << 20); + uint64_t threshold_size = 0; + uint64_t increase_size = 0; + if (batch_num_blocks > 0) + { + threshold_size = get_estimated_batch_size(batch_num_blocks); + LOG_PRINT_L1("calculated batch size: " << threshold_size); + // The increased DB size could be a multiple of threshold_size, a fixed + // size increase (> threshold_size), or other variations. + // + // Currently we use the greater of threshold size and a minimum size. The + // minimum size increase is used to avoid frequent resizes when the batch + // size is set to a very small numbers of blocks. + increase_size = (threshold_size > min_increase_size) ? threshold_size : min_increase_size; + LOG_PRINT_L1("increase size: " << increase_size); + } + // if threshold_size is 0 (i.e. number of blocks for batch not passed in), it + // will fall back to the percent-based threshold check instead of the + // size-based check + if (need_resize(threshold_size)) + { + LOG_PRINT_L0("[batch] DB resize needed"); + do_resize(increase_size); + } +} +uint64_t BlockchainLMDB::get_estimated_batch_size(uint64_t batch_num_blocks) const +{ + uint64_t threshold_size = 0; + // batch size estimate * batch safety factor = final size estimate + // Takes into account "reasonable" block size increases in batch. + float batch_safety_factor = 1.7f; + // estimate of stored block expanded from raw block, including denormalization and db overhead. + // Note that this probably doesn't grow linearly with block size. + float db_expand_factor = 4.5f; + uint64_t num_prev_blocks = 500; + // For resizing purposes, allow for at least 4k average block size. + uint64_t min_block_size = 4 * 1024; + uint64_t block_stop = m_height - 1; + uint64_t block_start = 0; + if (block_stop >= num_prev_blocks) + block_start = block_stop - num_prev_blocks + 1; + uint32_t num_blocks_used = 0; + uint64_t total_block_size = 0; + for (uint64_t block_num = block_start; block_num <= block_stop; ++block_num) + { + uint32_t block_size = get_block_size(block_num); + total_block_size += block_size; + // Track number of blocks being totalled here instead of assuming, in case + // some blocks were to be skipped for being outliers. + ++num_blocks_used; + } + size_t avg_block_size = total_block_size / num_blocks_used; + LOG_PRINT_L1("average block size across recent " << num_blocks_used << " blocks: " << avg_block_size); + if (avg_block_size < min_block_size) + avg_block_size = min_block_size; + LOG_PRINT_L1("estimated average block size for batch: " << avg_block_size); + threshold_size = avg_block_size * db_expand_factor * batch_num_blocks; + threshold_size = threshold_size * batch_safety_factor; + return threshold_size; +} +void BlockchainLMDB::add_block(const block& blk, const size_t& block_size, const difficulty_type& cumulative_difficulty, const uint64_t& coins_generated, + const crypto::hash& blk_hash) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + MDB_val_copy<crypto::hash> val_h(blk_hash); + MDB_val unused; + if (mdb_get(*m_write_txn, m_block_heights, &val_h, &unused) == 0) + throw1(BLOCK_EXISTS("Attempting to add block that's already in the db")); + if (m_height > 0) + { + MDB_val_copy<crypto::hash> parent_key(blk.prev_id); + MDB_val parent_h; + if (mdb_get(*m_write_txn, m_block_heights, &parent_key, &parent_h)) + { + LOG_PRINT_L3("m_height: " << m_height); + LOG_PRINT_L3("parent_key: " << blk.prev_id); + throw0(DB_ERROR("Failed to get top block hash to check for new block's parent")); + } + uint64_t parent_height = *(const uint64_t *)parent_h.mv_data; + if (parent_height != m_height - 1) + throw0(BLOCK_PARENT_DNE("Top block is not new block's parent")); + } + int result = 0; + MDB_val_copy<uint64_t> key(m_height); + MDB_val_copy<blobdata> blob(block_to_blob(blk)); + result = mdb_put(*m_write_txn, m_blocks, &key, &blob, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add block blob to db transaction: ").append(mdb_strerror(result)).c_str())); + MDB_val_copy<size_t> sz(block_size); + result = mdb_put(*m_write_txn, m_block_sizes, &key, &sz, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add block size to db transaction: ").append(mdb_strerror(result)).c_str())); + MDB_val_copy<uint64_t> ts(blk.timestamp); + result = mdb_put(*m_write_txn, m_block_timestamps, &key, &ts, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add block timestamp to db transaction: ").append(mdb_strerror(result)).c_str())); + MDB_val_copy<difficulty_type> diff(cumulative_difficulty); + result = mdb_put(*m_write_txn, m_block_diffs, &key, &diff, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add block cumulative difficulty to db transaction: ").append(mdb_strerror(result)).c_str())); + MDB_val_copy<uint64_t> coinsgen(coins_generated); + result = mdb_put(*m_write_txn, m_block_coins, &key, &coinsgen, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add block total generated coins to db transaction: ").append(mdb_strerror(result)).c_str())); + result = mdb_put(*m_write_txn, m_block_heights, &val_h, &key, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add block height by hash to db transaction: ").append(mdb_strerror(result)).c_str())); + result = mdb_put(*m_write_txn, m_block_hashes, &key, &val_h, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add block hash to db transaction: ").append(mdb_strerror(result)).c_str())); +} +void BlockchainLMDB::remove_block() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + if (m_height == 0) + throw0(BLOCK_DNE ("Attempting to remove block from an empty blockchain")); + MDB_val_copy<uint64_t> k(m_height - 1); + MDB_val h; + if (mdb_get(*m_write_txn, m_block_hashes, &k, &h)) + throw1(BLOCK_DNE("Attempting to remove block that's not in the db")); + if (mdb_del(*m_write_txn, m_blocks, &k, NULL)) + throw1(DB_ERROR("Failed to add removal of block to db transaction")); + if (mdb_del(*m_write_txn, m_block_sizes, &k, NULL)) + throw1(DB_ERROR("Failed to add removal of block size to db transaction")); + if (mdb_del(*m_write_txn, m_block_diffs, &k, NULL)) + throw1(DB_ERROR("Failed to add removal of block cumulative difficulty to db transaction")); + if (mdb_del(*m_write_txn, m_block_coins, &k, NULL)) + throw1(DB_ERROR("Failed to add removal of block total generated coins to db transaction")); + if (mdb_del(*m_write_txn, m_block_timestamps, &k, NULL)) + throw1(DB_ERROR("Failed to add removal of block timestamp to db transaction")); + if (mdb_del(*m_write_txn, m_block_heights, &h, NULL)) + throw1(DB_ERROR("Failed to add removal of block height by hash to db transaction")); + if (mdb_del(*m_write_txn, m_block_hashes, &k, NULL)) + throw1(DB_ERROR("Failed to add removal of block hash to db transaction")); +} +void BlockchainLMDB::add_transaction_data(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash& tx_hash) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + int result = 0; + MDB_val_copy<crypto::hash> val_h(tx_hash); + MDB_val unused; + if (mdb_get(*m_write_txn, m_txs, &val_h, &unused) == 0) + throw1(TX_EXISTS("Attempting to add transaction that's already in the db")); + MDB_val_copy<blobdata> blob(tx_to_blob(tx)); + result = mdb_put(*m_write_txn, m_txs, &val_h, &blob, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add tx blob to db transaction: ").append(mdb_strerror(result)).c_str())); + MDB_val_copy<uint64_t> height(m_height); + result = mdb_put(*m_write_txn, m_tx_heights, &val_h, &height, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add tx block height to db transaction: ").append(mdb_strerror(result)).c_str())); + MDB_val_copy<uint64_t> unlock_time(tx.unlock_time); + result = mdb_put(*m_write_txn, m_tx_unlocks, &val_h, &unlock_time, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add tx unlock time to db transaction: ").append(mdb_strerror(result)).c_str())); +} +void BlockchainLMDB::remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + MDB_val_copy<crypto::hash> val_h(tx_hash); + MDB_val unused; + if (mdb_get(*m_write_txn, m_txs, &val_h, &unused)) + throw1(TX_DNE("Attempting to remove transaction that isn't in the db")); + if (mdb_del(*m_write_txn, m_txs, &val_h, NULL)) + throw1(DB_ERROR("Failed to add removal of tx to db transaction")); + if (mdb_del(*m_write_txn, m_tx_unlocks, &val_h, NULL)) + throw1(DB_ERROR("Failed to add removal of tx unlock time to db transaction")); + if (mdb_del(*m_write_txn, m_tx_heights, &val_h, NULL)) + throw1(DB_ERROR("Failed to add removal of tx block height to db transaction")); + remove_tx_outputs(tx_hash, tx); + if (mdb_del(*m_write_txn, m_tx_outputs, &val_h, NULL)) + throw1(DB_ERROR("Failed to add removal of tx outputs to db transaction")); +} +void BlockchainLMDB::add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + int result = 0; + MDB_val_copy<uint64_t> k(m_num_outputs); + MDB_val_copy<crypto::hash> v(tx_hash); + result = mdb_put(*m_write_txn, m_output_txs, &k, &v, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add output tx hash to db transaction: ").append(mdb_strerror(result)).c_str())); + result = mdb_put(*m_write_txn, m_tx_outputs, &v, &k, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add <tx hash, global output index> to db transaction: ").append(mdb_strerror(result)).c_str())); + MDB_val_copy<uint64_t> val_local_index(local_index); + result = mdb_put(*m_write_txn, m_output_indices, &k, &val_local_index, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add tx output index to db transaction: ").append(mdb_strerror(result)).c_str())); + MDB_val_copy<uint64_t> val_amount(tx_output.amount); + result = mdb_put(*m_write_txn, m_output_amounts, &val_amount, &k, 0); + if (result) + throw0(DB_ERROR(std::string("Failed to add output amount to db transaction: ").append(mdb_strerror(result)).c_str())); + if (tx_output.target.type() == typeid(txout_to_key)) + { + output_data_t od; + od.pubkey = boost::get < txout_to_key > (tx_output.target).key; + od.unlock_time = unlock_time; + od.height = m_height; + MDB_val_copy<output_data_t> data(od); + //MDB_val_copy<crypto::public_key> val_pubkey(boost::get<txout_to_key>(tx_output.target).key); + if (mdb_put(*m_write_txn, m_output_keys, &k, &data, 0)) + throw0(DB_ERROR("Failed to add output pubkey to db transaction")); + } +/****** Uncomment if ever outputs actually need to be stored in this manner + * + blobdata b = output_to_blob(tx_output); + v.mv_size = b.size(); + v.mv_data = &b; + if (mdb_put(*m_write_txn, m_outputs, &k, &v, 0)) + throw0(DB_ERROR("Failed to add output to db transaction")); + if (mdb_put(*m_write_txn, m_output_gindices, &v, &k, 0)) + throw0(DB_ERROR("Failed to add output global index to db transaction")); +************************************************************************/ + m_num_outputs++; +} +void BlockchainLMDB::remove_tx_outputs(const crypto::hash& tx_hash, const transaction& tx) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + lmdb_cur cur(*m_write_txn, m_tx_outputs); + MDB_val_copy<crypto::hash> k(tx_hash); + MDB_val v; + auto result = mdb_cursor_get(cur, &k, &v, MDB_SET); + if (result == MDB_NOTFOUND) + { + LOG_ERROR("Attempting to remove a tx's outputs, but none found. Continuing, but...be wary, because that's weird."); + } + else if (result) + { + throw0(DB_ERROR("DB error attempting to get an output")); + } + else + { + size_t num_elems = 0; + mdb_cursor_count(cur, &num_elems); + mdb_cursor_get(cur, &k, &v, MDB_FIRST_DUP); + for (uint64_t i = 0; i < num_elems; ++i) + { + const tx_out tx_output = tx.vout; + remove_output(*(const uint64_t*)v.mv_data, tx_output.amount); + if (i < num_elems - 1) + { + mdb_cursor_get(cur, &k, &v, MDB_NEXT_DUP); + } + } + } + cur.close(); +} +void BlockchainLMDB::remove_output(const tx_out& tx_output) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__ << " (unused version - does nothing)"); + return; +} +void BlockchainLMDB::remove_output(const uint64_t& out_index, const uint64_t amount) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + MDB_val_copy<uint64_t> k(out_index); +/****** Uncomment if ever outputs actually need to be stored in this manner + blobdata b; + t_serializable_object_to_blob(tx_output, b); + k.mv_size = b.size(); + k.mv_data = &b; + if (mdb_get(*m_write_txn, m_output_gindices, &k, &v)) + throw1(OUTPUT_DNE("Attempting to remove output that does not exist")); + uint64_t gindex = *(uint64_t*)v.mv_data; + auto result = mdb_del(*m_write_txn, m_output_gindices, &k, NULL); + if (result != 0 && result != MDB_NOTFOUND) + throw1(DB_ERROR("Error adding removal of output global index to db transaction")); + result = mdb_del(*m_write_txn, m_outputs, &v, NULL); + if (result != 0 && result != MDB_NOTFOUND) + throw1(DB_ERROR("Error adding removal of output to db transaction")); +*********************************************************************/ + auto result = mdb_del(*m_write_txn, m_output_indices, &k, NULL); + if (result == MDB_NOTFOUND) + { + LOG_PRINT_L0("Unexpected: global output index not found in m_output_indices"); + } + else if (result) + { + throw1(DB_ERROR("Error adding removal of output tx index to db transaction")); + } + result = mdb_del(*m_write_txn, m_output_txs, &k, NULL); + // if (result != 0 && result != MDB_NOTFOUND) + // throw1(DB_ERROR("Error adding removal of output tx hash to db transaction")); + if (result == MDB_NOTFOUND) + { + LOG_PRINT_L0("Unexpected: global output index not found in m_output_txs"); + } + else if (result) + { + throw1(DB_ERROR("Error adding removal of output tx hash to db transaction")); + } + result = mdb_del(*m_write_txn, m_output_keys, &k, NULL); + if (result == MDB_NOTFOUND) + { + LOG_PRINT_L0("Unexpected: global output index not found in m_output_keys"); + } + else if (result) + throw1(DB_ERROR("Error adding removal of output pubkey to db transaction")); + remove_amount_output_index(amount, out_index); + m_num_outputs--; +} +void BlockchainLMDB::remove_amount_output_index(const uint64_t amount, const uint64_t global_output_index) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + lmdb_cur cur(*m_write_txn, m_output_amounts); + MDB_val_copy<uint64_t> k(amount); + MDB_val v; + auto result = mdb_cursor_get(cur, &k, &v, MDB_SET); + if (result == MDB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + size_t num_elems = 0; + mdb_cursor_count(cur, &num_elems); + mdb_cursor_get(cur, &k, &v, MDB_FIRST_DUP); + uint64_t amount_output_index = 0; + uint64_t goi = 0; + bool found_index = false; + for (uint64_t i = 0; i < num_elems; ++i) + { + mdb_cursor_get(cur, &k, &v, MDB_GET_CURRENT); + goi = *(const uint64_t *)v.mv_data; + if (goi == global_output_index) + { + amount_output_index = i; + found_index = true; + break; + } + mdb_cursor_get(cur, &k, &v, MDB_NEXT_DUP); + } + if (found_index) + { + // found the amount output index + // now delete it + result = mdb_cursor_del(cur, 0); + if (result) + throw0(DB_ERROR(std::string("Error deleting amount output index ").append(boost::lexical_cast<std::string>(amount_output_index)).c_str())); + } + else + { + // not found + cur.close(); + throw1(OUTPUT_DNE("Failed to find amount output index")); + } + cur.close(); +} +void BlockchainLMDB::add_spent_key(const crypto::key_image& k_image) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + MDB_val_copy<crypto::key_image> val_key(k_image); + MDB_val unused; + if (mdb_get(*m_write_txn, m_spent_keys, &val_key, &unused) == 0) + throw1(KEY_IMAGE_EXISTS("Attempting to add spent key image that's already in the db")); + char anything = '\0'; + unused.mv_size = sizeof(char); + unused.mv_data = &anything; + if (auto result = mdb_put(*m_write_txn, m_spent_keys, &val_key, &unused, 0)) + throw1(DB_ERROR(std::string("Error adding spent key image to db transaction: ").append(mdb_strerror(result)).c_str())); +} +void BlockchainLMDB::remove_spent_key(const crypto::key_image& k_image) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + MDB_val_copy<crypto::key_image> k(k_image); + auto result = mdb_del(*m_write_txn, m_spent_keys, &k, NULL); + if (result != 0 && result != MDB_NOTFOUND) + throw1(DB_ERROR("Error adding removal of key image to db transaction")); +} +blobdata BlockchainLMDB::output_to_blob(const tx_out& output) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + blobdata b; + if (!t_serializable_object_to_blob(output, b)) + throw1(DB_ERROR("Error serializing output to blob")); + return b; +} +tx_out BlockchainLMDB::output_from_blob(const blobdata& blob) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + std::stringstream ss; + ss << blob; + binary_archive<false> ba(ss); + tx_out o; + if (!(::serialization::serialize(ba, o))) + throw1(DB_ERROR("Error deserializing tx output blob")); + return o; +} +uint64_t BlockchainLMDB::get_output_global_index(const uint64_t& amount, const uint64_t& index) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + std::vector <uint64_t> offsets; + std::vector <uint64_t> global_indices; + offsets.push_back(index); + get_output_global_indices(amount, offsets, global_indices); + if (!global_indices.size()) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + return global_indices[0]; +} +void BlockchainLMDB::check_open() const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + if (!m_open) + throw0(DB_ERROR("DB operation attempted on a not-open DB instance")); +} +BlockchainLMDB::~BlockchainLMDB() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + // batch transaction shouldn't be active at this point. If it is, consider it aborted. + if (m_batch_active) + batch_abort(); +} +BlockchainLMDB::BlockchainLMDB(bool batch_transactions) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + // initialize folder to something "safe" just in case + // someone accidentally misuses this class... + m_folder = "thishsouldnotexistbecauseitisgibberish"; + m_open = false; + m_batch_transactions = batch_transactions; + m_write_txn = nullptr; + m_write_batch_txn = nullptr; + m_batch_active = false; + m_height = 0; +} +void BlockchainLMDB::open(const std::string& filename, const int mdb_flags) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + if (m_open) + throw0(DB_OPEN_FAILURE("Attempted to open db, but it's already open")); + boost::filesystem::path direc(filename); + if (boost::filesystem::exists(direc)) + { + if (!boost::filesystem::is_directory(direc)) + throw0(DB_OPEN_FAILURE("LMDB needs a directory path, but a file was passed")); + } + else + { + if (!boost::filesystem::create_directory(direc)) + throw0(DB_OPEN_FAILURE(std::string("Failed to create directory ").append(filename).c_str())); + } + // check for existing LMDB files in base directory + boost::filesystem::path old_files = direc.parent_path(); + if (boost::filesystem::exists(old_files / "data.mdb") || boost::filesystem::exists(old_files / "lock.mdb")) + { + LOG_PRINT_L0("Found existing LMDB files in " << old_files.string()); + LOG_PRINT_L0("Move data.mdb and/or lock.mdb to " << filename << ", or delete them, and then restart"); + throw DB_ERROR("Database could not be opened"); + } + m_folder = filename; + // set up lmdb environment + if (mdb_env_create(&m_env)) + throw0(DB_ERROR("Failed to create lmdb environment")); + if (mdb_env_set_maxdbs(m_env, 20)) + throw0(DB_ERROR("Failed to set max number of dbs")); + size_t mapsize = DEFAULT_MAPSIZE; + if (auto result = mdb_env_open(m_env, filename.c_str(), mdb_flags, 0644)) + throw0(DB_ERROR(std::string("Failed to open lmdb environment: ").append(mdb_strerror(result)).c_str())); + MDB_envinfo mei; + mdb_env_info(m_env, &mei); + uint64_t cur_mapsize = (double)mei.me_mapsize; + if (cur_mapsize < mapsize) + { + if (auto result = mdb_env_set_mapsize(m_env, mapsize)) + throw0(DB_ERROR(std::string("Failed to set max memory map size: ").append(mdb_strerror(result)).c_str())); + mdb_env_info(m_env, &mei); + cur_mapsize = (double)mei.me_mapsize; + LOG_PRINT_L1("LMDB memory map size: " << cur_mapsize); + } + if (need_resize()) + { + LOG_PRINT_L0("LMDB memory map needs resized, doing that now."); + do_resize(); + } + int txn_flags = 0; + if (mdb_flags & MDB_RDONLY) + txn_flags |= MDB_RDONLY; + // get a read/write MDB_txn, depending on mdb_flags + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, txn_flags, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + // open necessary databases, and set properties as needed + // uses macros to avoid having to change things too many places + lmdb_db_open(txn, LMDB_BLOCKS, MDB_INTEGERKEY | MDB_CREATE, m_blocks, "Failed to open db handle for m_blocks"); + lmdb_db_open(txn, LMDB_BLOCK_TIMESTAMPS, MDB_INTEGERKEY | MDB_CREATE, m_block_timestamps, "Failed to open db handle for m_block_timestamps"); + lmdb_db_open(txn, LMDB_BLOCK_HEIGHTS, MDB_CREATE, m_block_heights, "Failed to open db handle for m_block_heights"); + lmdb_db_open(txn, LMDB_BLOCK_HASHES, MDB_INTEGERKEY | MDB_CREATE, m_block_hashes, "Failed to open db handle for m_block_hashes"); + lmdb_db_open(txn, LMDB_BLOCK_SIZES, MDB_INTEGERKEY | MDB_CREATE, m_block_sizes, "Failed to open db handle for m_block_sizes"); + lmdb_db_open(txn, LMDB_BLOCK_DIFFS, MDB_INTEGERKEY | MDB_CREATE, m_block_diffs, "Failed to open db handle for m_block_diffs"); + lmdb_db_open(txn, LMDB_BLOCK_COINS, MDB_INTEGERKEY | MDB_CREATE, m_block_coins, "Failed to open db handle for m_block_coins"); + lmdb_db_open(txn, LMDB_TXS, MDB_CREATE, m_txs, "Failed to open db handle for m_txs"); + lmdb_db_open(txn, LMDB_TX_UNLOCKS, MDB_CREATE, m_tx_unlocks, "Failed to open db handle for m_tx_unlocks"); + lmdb_db_open(txn, LMDB_TX_HEIGHTS, MDB_CREATE, m_tx_heights, "Failed to open db handle for m_tx_heights"); + lmdb_db_open(txn, LMDB_TX_OUTPUTS, MDB_DUPSORT | MDB_CREATE, m_tx_outputs, "Failed to open db handle for m_tx_outputs"); + lmdb_db_open(txn, LMDB_OUTPUT_TXS, MDB_INTEGERKEY | MDB_CREATE, m_output_txs, "Failed to open db handle for m_output_txs"); + lmdb_db_open(txn, LMDB_OUTPUT_INDICES, MDB_INTEGERKEY | MDB_CREATE, m_output_indices, "Failed to open db handle for m_output_indices"); + lmdb_db_open(txn, LMDB_OUTPUT_AMOUNTS, MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED | MDB_CREATE, m_output_amounts, "Failed to open db handle for m_output_amounts"); + lmdb_db_open(txn, LMDB_OUTPUT_KEYS, MDB_INTEGERKEY | MDB_CREATE, m_output_keys, "Failed to open db handle for m_output_keys"); +/*************** not used, but kept for posterity + lmdb_db_open(txn, LMDB_OUTPUTS, MDB_INTEGERKEY | MDB_CREATE, m_outputs, "Failed to open db handle for m_outputs"); + lmdb_db_open(txn, LMDB_OUTPUT_GINDICES, MDB_CREATE, m_output_gindices, "Failed to open db handle for m_output_gindices"); +*************************************************/ + lmdb_db_open(txn, LMDB_SPENT_KEYS, MDB_CREATE, m_spent_keys, "Failed to open db handle for m_spent_keys"); + mdb_set_dupsort(txn, m_output_amounts, compare_uint64); + mdb_set_dupsort(txn, m_tx_outputs, compare_uint64); + mdb_set_compare(txn, m_spent_keys, compare_hash32); + mdb_set_compare(txn, m_block_heights, compare_hash32); + mdb_set_compare(txn, m_txs, compare_hash32); + mdb_set_compare(txn, m_tx_unlocks, compare_hash32); + mdb_set_compare(txn, m_tx_heights, compare_hash32); + // get and keep current height + MDB_stat db_stats; + if (mdb_stat(txn, m_blocks, &db_stats)) + throw0(DB_ERROR("Failed to query m_blocks")); + LOG_PRINT_L2("Setting m_height to: " << db_stats.ms_entries); + m_height = db_stats.ms_entries; + // get and keep current number of outputs + if (mdb_stat(txn, m_output_indices, &db_stats)) + throw0(DB_ERROR("Failed to query m_output_indices")); + m_num_outputs = db_stats.ms_entries; + // ND: This "new" version of the lmdb database is incompatible with + // the previous version. Ensure that the output_keys database is + // sizeof(output_data_t) in length. Otherwise, inform user and + // terminate. + if(m_height > 0) + { + MDB_val_copy<uint64_t> k(0); + MDB_val v; + auto get_result = mdb_get(txn, m_output_keys, &k, &v); + if(get_result != MDB_SUCCESS) + { + txn.abort(); + m_open = false; + return; + } + // LOG_PRINT_L0("Output keys size: " << v.mv_size); + if(v.mv_size != sizeof(output_data_t)) + { + txn.abort(); + mdb_env_close(m_env); + m_open = false; + LOG_PRINT_RED_L0("Existing lmdb database is incompatible with this version."); + LOG_PRINT_RED_L0("Please delete the existing database and resync."); + return; + } + } + // commit the transaction + txn.commit(); + m_open = true; + // from here, init should be finished +} +void BlockchainLMDB::close() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + if (m_batch_active) + { + LOG_PRINT_L3("close() first calling batch_abort() due to active batch transaction"); + batch_abort(); + } + this->sync(); + // FIXME: not yet thread safe!!! Use with care. + mdb_env_close(m_env); +} +void BlockchainLMDB::sync() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + // Does nothing unless LMDB environment was opened with MDB_NOSYNC or in part + // MDB_NOMETASYNC. Force flush to be synchronous. + if (auto result = mdb_env_sync(m_env, true)) + { + throw0(DB_ERROR(std::string("Failed to sync database: ").append(mdb_strerror(result)).c_str())); + } +} +void BlockchainLMDB::reset() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + // TODO: this +} +std::vector<std::string> BlockchainLMDB::get_filenames() const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + std::vector<std::string> filenames; + boost::filesystem::path datafile(m_folder); + datafile /= "data.mdb"; + boost::filesystem::path lockfile(m_folder); + lockfile /= "lock.mdb"; + filenames.push_back(datafile.string()); + filenames.push_back(lockfile.string()); + return filenames; +} +std::string BlockchainLMDB::get_db_name() const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + return std::string("lmdb"); +} +bool BlockchainLMDB::lock() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + return false; +} +void BlockchainLMDB::unlock() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); +} +bool BlockchainLMDB::block_exists(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + MDB_val_copy<crypto::hash> key(h); + MDB_val result; + auto get_result = mdb_get(txn, m_block_heights, &key, &result); + if (get_result == MDB_NOTFOUND) + { + txn.commit(); + LOG_PRINT_L3("Block with hash " << epee::string_tools::pod_to_hex(h) << " not found in db"); + return false; + } + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch block index from hash")); + txn.commit(); + return true; +} +block BlockchainLMDB::get_block(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + return get_block_from_height(get_block_height(h)); +} +uint64_t BlockchainLMDB::get_block_height(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + MDB_val_copy<crypto::hash> key(h); + MDB_val result; + auto get_result = mdb_get(txn, m_block_heights, &key, &result); + if (get_result == MDB_NOTFOUND) + throw1(BLOCK_DNE("Attempted to retrieve non-existent block height")); + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a block height from the db")); + txn.commit(); + return *(const uint64_t*)result.mv_data; +} +block_header BlockchainLMDB::get_block_header(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + // block_header object is automatically cast from block object + return get_block(h); +} +block BlockchainLMDB::get_block_from_height(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + MDB_val_copy<uint64_t> key(height); + MDB_val result; + auto get_result = mdb_get(txn, m_blocks, &key, &result); + if (get_result == MDB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get block from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- block not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a block from the db")); + txn.commit(); + blobdata bd; + bd.assign(reinterpret_cast<char*>(result.mv_data), result.mv_size); + block b; + if (!parse_and_validate_block_from_blob(bd, b)) + throw0(DB_ERROR("Failed to parse block from blob retrieved from the db")); + return b; +} +uint64_t BlockchainLMDB::get_block_timestamp(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + MDB_val_copy<uint64_t> key(height); + MDB_val result; + auto get_result = mdb_get(*txn_ptr, m_block_timestamps, &key, &result); + if (get_result == MDB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get timestamp from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- timestamp not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a timestamp from the db")); + if (! m_batch_active) + txn.commit(); + return *(const uint64_t*)result.mv_data; +} +uint64_t BlockchainLMDB::get_top_block_timestamp() const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + // if no blocks, return 0 + if (m_height == 0) + { + return 0; + } + return get_block_timestamp(m_height - 1); +} +size_t BlockchainLMDB::get_block_size(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + MDB_val_copy<uint64_t> key(height); + MDB_val result; + auto get_result = mdb_get(*txn_ptr, m_block_sizes, &key, &result); + if (get_result == MDB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get block size from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- block size not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a block size from the db")); + if (! m_batch_active) + txn.commit(); + return *(const size_t*)result.mv_data; +} +difficulty_type BlockchainLMDB::get_block_cumulative_difficulty(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__ << " height: " << height); + check_open(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + MDB_val_copy<uint64_t> key(height); + MDB_val result; + auto get_result = mdb_get(*txn_ptr, m_block_diffs, &key, &result); + if (get_result == MDB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get cumulative difficulty from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- difficulty not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a cumulative difficulty from the db")); + if (! m_batch_active) + txn.commit(); + return *(difficulty_type*)result.mv_data; +} +difficulty_type BlockchainLMDB::get_block_difficulty(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + difficulty_type diff1 = 0; + difficulty_type diff2 = 0; + diff1 = get_block_cumulative_difficulty(height); + if (height != 0) + { + diff2 = get_block_cumulative_difficulty(height - 1); + } + return diff1 - diff2; +}

Continuation of: Git diff from commit e940386f9a8765423ab3dd9e3aabe19a68cba9f9 (thankful_for_today's last commit) to current HEAD, excluding changes in /external so as to exclude libraries and submoduled code, and excluding contrib/ so as to exclude changes made to epee, excluding most comments because apparently nobody that writes code for BCN knows what a comment is anyway, excluding removed lines, excluding empty lines

+uint64_t BlockchainLMDB::get_block_already_generated_coins(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + MDB_val_copy<uint64_t> key(height); + MDB_val result; + auto get_result = mdb_get(*txn_ptr, m_block_coins, &key, &result); + if (get_result == MDB_NOTFOUND) + { + throw0(DB_ERROR(std::string("Attempt to get generated coins from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- block size not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve a total generated coins from the db")); + if (! m_batch_active) + txn.commit(); + return *(const uint64_t*)result.mv_data; +} +crypto::hash BlockchainLMDB::get_block_hash_from_height(const uint64_t& height) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + MDB_val_copy<uint64_t> key(height); + MDB_val result; + auto get_result = mdb_get(*txn_ptr, m_block_hashes, &key, &result); + if (get_result == MDB_NOTFOUND) + { + throw0(BLOCK_DNE(std::string("Attempt to get hash from height ").append(boost::lexical_cast<std::string>(height)).append(" failed -- hash not in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR(std::string("Error attempting to retrieve a block hash from the db: "). + append(mdb_strerror(get_result)).c_str())); + if (! m_batch_active) + txn.commit(); + return *(crypto::hash*)result.mv_data; +} +std::vector<block> BlockchainLMDB::get_blocks_range(const uint64_t& h1, const uint64_t& h2) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + std::vector<block> v; + for (uint64_t height = h1; height <= h2; ++height) + { + v.push_back(get_block_from_height(height)); + } + return v; +} +std::vector<crypto::hash> BlockchainLMDB::get_hashes_range(const uint64_t& h1, const uint64_t& h2) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + std::vector<crypto::hash> v; + for (uint64_t height = h1; height <= h2; ++height) + { + v.push_back(get_block_hash_from_height(height)); + } + return v; +} +crypto::hash BlockchainLMDB::top_block_hash() const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + if (m_height != 0) + { + return get_block_hash_from_height(m_height - 1); + } + return null_hash; +} +block BlockchainLMDB::get_top_block() const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + if (m_height != 0) + { + return get_block_from_height(m_height - 1); + } + block b; + return b; +} +uint64_t BlockchainLMDB::height() const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + return m_height; +} +bool BlockchainLMDB::tx_exists(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + MDB_val_copy<crypto::hash> key(h); + MDB_val result; + TIME_MEASURE_START(time1); + auto get_result = mdb_get(*txn_ptr, m_txs, &key, &result); + TIME_MEASURE_FINISH(time1); + time_tx_exists += time1; + if (get_result == MDB_NOTFOUND) + { + if (! m_batch_active) + txn.commit(); + LOG_PRINT_L1("transaction with hash " << epee::string_tools::pod_to_hex(h) << " not found in db"); + return false; + } + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch transaction from hash")); + return true; +} +uint64_t BlockchainLMDB::get_tx_unlock_time(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + MDB_val_copy<crypto::hash> key(h); + MDB_val result; + auto get_result = mdb_get(txn, m_tx_unlocks, &key, &result); + if (get_result == MDB_NOTFOUND) + throw1(TX_DNE(std::string("tx unlock time with hash ").append(epee::string_tools::pod_to_hex(h)).append(" not found in db").c_str())); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch tx unlock time from hash")); + return *(const uint64_t*)result.mv_data; +} +transaction BlockchainLMDB::get_tx(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + MDB_val_copy<crypto::hash> key(h); + MDB_val result; + auto get_result = mdb_get(*txn_ptr, m_txs, &key, &result); + if (get_result == MDB_NOTFOUND) + throw1(TX_DNE(std::string("tx with hash ").append(epee::string_tools::pod_to_hex(h)).append(" not found in db").c_str())); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch tx from hash")); + blobdata bd; + bd.assign(reinterpret_cast<char*>(result.mv_data), result.mv_size); + transaction tx; + if (!parse_and_validate_tx_from_blob(bd, tx)) + throw0(DB_ERROR("Failed to parse tx from blob retrieved from the db")); + if (! m_batch_active) + txn.commit(); + return tx; +} +uint64_t BlockchainLMDB::get_tx_count() const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + MDB_stat db_stats; + if (mdb_stat(txn, m_txs, &db_stats)) + throw0(DB_ERROR("Failed to query m_txs")); + txn.commit(); + return db_stats.ms_entries; +} +std::vector<transaction> BlockchainLMDB::get_tx_list(const std::vector<crypto::hash>& hlist) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + std::vector<transaction> v; + for (auto& h : hlist) + { + v.push_back(get_tx(h)); + } + return v; +} +uint64_t BlockchainLMDB::get_tx_block_height(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + MDB_val_copy<crypto::hash> key(h); + MDB_val result; + auto get_result = mdb_get(*txn_ptr, m_tx_heights, &key, &result); + if (get_result == MDB_NOTFOUND) + { + throw1(TX_DNE(std::string("tx height with hash ").append(epee::string_tools::pod_to_hex(h)).append(" not found in db").c_str())); + } + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch tx height from hash")); + if (! m_batch_active) + txn.commit(); + return *(const uint64_t*)result.mv_data; +} +uint64_t BlockchainLMDB::get_random_output(const uint64_t& amount) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + uint64_t num_outputs = get_num_outputs(amount); + if (num_outputs == 0) + throw1(OUTPUT_DNE("Attempting to get a random output for an amount, but none exist")); + return crypto::rand<uint64_t>() % num_outputs; +} +uint64_t BlockchainLMDB::get_num_outputs(const uint64_t& amount) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + lmdb_cur cur(txn, m_output_amounts); + MDB_val_copy<uint64_t> k(amount); + MDB_val v; + auto result = mdb_cursor_get(cur, &k, &v, MDB_SET); + if (result == MDB_NOTFOUND) + { + return 0; + } + else if (result) + throw0(DB_ERROR("DB error attempting to get number of outputs of an amount")); + size_t num_elems = 0; + mdb_cursor_count(cur, &num_elems); + txn.commit(); + return num_elems; +} +output_data_t BlockchainLMDB::get_output_key(const uint64_t &global_index) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + MDB_val_copy<uint64_t> k(global_index); + MDB_val v; + auto get_result = mdb_get(txn, m_output_keys, &k, &v); + if (get_result == MDB_NOTFOUND) + throw0(DB_ERROR("Attempting to get output pubkey by global index, but key does not exist")); + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve an output pubkey from the db")); + txn.commit(); + return *(output_data_t *) v.mv_data; +} +output_data_t BlockchainLMDB::get_output_key(const uint64_t& amount, const uint64_t& index) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + uint64_t glob_index = get_output_global_index(amount, index); + return get_output_key(glob_index); +} +tx_out BlockchainLMDB::get_output(const crypto::hash& h, const uint64_t& index) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + lmdb_cur cur(txn, m_tx_outputs); + MDB_val_copy<crypto::hash> k(h); + MDB_val v; + auto result = mdb_cursor_get(cur, &k, &v, MDB_SET); + if (result == MDB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output by tx hash and tx index, but output not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + size_t num_elems = 0; + mdb_cursor_count(cur, &num_elems); + if (num_elems <= index) + throw1(OUTPUT_DNE("Attempting to get an output by tx hash and tx index, but output not found")); + mdb_cursor_get(cur, &k, &v, MDB_FIRST_DUP); + for (uint64_t i = 0; i < index; ++i) + { + mdb_cursor_get(cur, &k, &v, MDB_NEXT_DUP); + } + mdb_cursor_get(cur, &k, &v, MDB_GET_CURRENT); + blobdata b; + b = *(blobdata*)v.mv_data; + cur.close(); + txn.commit(); + return output_from_blob(b); +} +tx_out BlockchainLMDB::get_output(const uint64_t& index) const +{ + return tx_out(); + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + MDB_val_copy<uint64_t> k(index); + MDB_val v; + auto get_result = mdb_get(txn, m_outputs, &k, &v); + if (get_result == MDB_NOTFOUND) + { + throw OUTPUT_DNE("Attempting to get output by global index, but output does not exist"); + } + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve an output from the db")); + blobdata b = *(blobdata*)v.mv_data; + return output_from_blob(b); +} +tx_out_index BlockchainLMDB::get_output_tx_and_index_from_global(const uint64_t& index) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + MDB_val_copy<uint64_t> k(index); + MDB_val v; + auto get_result = mdb_get(*txn_ptr, m_output_txs, &k, &v); + if (get_result == MDB_NOTFOUND) + throw1(OUTPUT_DNE("output with given index not in db")); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch output tx hash")); + crypto::hash tx_hash = *(crypto::hash*)v.mv_data; + get_result = mdb_get(*txn_ptr, m_output_indices, &k, &v); + if (get_result == MDB_NOTFOUND) + throw1(OUTPUT_DNE("output with given index not in db")); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch output tx index")); + if (! m_batch_active) + txn.commit(); + return tx_out_index(tx_hash, *(const uint64_t *)v.mv_data); +} +tx_out_index BlockchainLMDB::get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + std::vector < uint64_t > offsets; + std::vector<tx_out_index> indices; + offsets.push_back(index); + get_output_tx_and_index(amount, offsets, indices); + if (!indices.size()) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + return indices[0]; +} +std::vector<uint64_t> BlockchainLMDB::get_tx_output_indices(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + std::vector<uint64_t> index_vec; + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + lmdb_cur cur(txn, m_tx_outputs); + MDB_val_copy<crypto::hash> k(h); + MDB_val v; + auto result = mdb_cursor_get(cur, &k, &v, MDB_SET); + if (result == MDB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output by tx hash and tx index, but output not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + size_t num_elems = 0; + mdb_cursor_count(cur, &num_elems); + mdb_cursor_get(cur, &k, &v, MDB_FIRST_DUP); + for (uint64_t i = 0; i < num_elems; ++i) + { + mdb_cursor_get(cur, &k, &v, MDB_GET_CURRENT); + index_vec.push_back(*(const uint64_t *)v.mv_data); + mdb_cursor_get(cur, &k, &v, MDB_NEXT_DUP); + } + cur.close(); + txn.commit(); + return index_vec; +} +std::vector<uint64_t> BlockchainLMDB::get_tx_amount_output_indices(const crypto::hash& h) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + std::vector<uint64_t> index_vec; + std::vector<uint64_t> index_vec2; + // get the transaction's global output indices first + index_vec = get_tx_output_indices(h); + // these are next used to obtain the amount output indices + transaction tx = get_tx(h); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + uint64_t i = 0; + uint64_t global_index; + BOOST_FOREACH(const auto& vout, tx.vout) + { + uint64_t amount = vout.amount; + global_index = index_vec; + lmdb_cur cur(txn, m_output_amounts); + MDB_val_copy<uint64_t> k(amount); + MDB_val v; + auto result = mdb_cursor_get(cur, &k, &v, MDB_SET); + if (result == MDB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + size_t num_elems = 0; + mdb_cursor_count(cur, &num_elems); + mdb_cursor_get(cur, &k, &v, MDB_FIRST_DUP); + uint64_t amount_output_index = 0; + uint64_t output_index = 0; + bool found_index = false; + for (uint64_t j = 0; j < num_elems; ++j) + { + mdb_cursor_get(cur, &k, &v, MDB_GET_CURRENT); + output_index = *(const uint64_t *)v.mv_data; + if (output_index == global_index) + { + amount_output_index = j; + found_index = true; + break; + } + mdb_cursor_get(cur, &k, &v, MDB_NEXT_DUP); + } + if (found_index) + { + index_vec2.push_back(amount_output_index); + } + else + { + // not found + cur.close(); + txn.commit(); + throw1(OUTPUT_DNE("specified output not found in db")); + } + cur.close(); + ++i; + } + txn.commit(); + return index_vec2; +} +bool BlockchainLMDB::has_key_image(const crypto::key_image& img) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + MDB_val_copy<crypto::key_image> val_key(img); + MDB_val unused; + if (mdb_get(txn, m_spent_keys, &val_key, &unused) == 0) + { + txn.commit(); + return true; + } + txn.commit(); + return false; +} +void BlockchainLMDB::batch_start(uint64_t batch_num_blocks) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + if (! m_batch_transactions) + throw0(DB_ERROR("batch transactions not enabled")); + if (m_batch_active) + throw0(DB_ERROR("batch transaction already in progress")); + if (m_write_batch_txn != nullptr) + throw0(DB_ERROR("batch transaction already in progress")); + if (m_write_txn) + throw0(DB_ERROR("batch transaction attempted, but m_write_txn already in use")); + check_open(); + check_and_resize_for_batch(batch_num_blocks); + m_write_batch_txn = new mdb_txn_safe(); + // NOTE: need to make sure it's destroyed properly when done + if (mdb_txn_begin(m_env, NULL, 0, *m_write_batch_txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + // indicates this transaction is for batch transactions, but not whether it's + // active + m_write_batch_txn->m_batch_txn = true; + m_write_txn = m_write_batch_txn; + m_batch_active = true; + LOG_PRINT_L3("batch transaction: begin"); +} +void BlockchainLMDB::batch_commit() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + if (! m_batch_transactions) + throw0(DB_ERROR("batch transactions not enabled")); + if (! m_batch_active) + throw0(DB_ERROR("batch transaction not in progress")); + if (m_write_batch_txn == nullptr) + throw0(DB_ERROR("batch transaction not in progress")); + check_open(); + LOG_PRINT_L3("batch transaction: committing..."); + TIME_MEASURE_START(time1); + m_write_txn->commit(); + TIME_MEASURE_FINISH(time1); + time_commit1 += time1; + LOG_PRINT_L3("batch transaction: committed"); + m_write_txn = nullptr; + delete m_write_batch_txn; +} +void BlockchainLMDB::batch_stop() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + if (! m_batch_transactions) + throw0(DB_ERROR("batch transactions not enabled")); + if (! m_batch_active) + throw0(DB_ERROR("batch transaction not in progress")); + if (m_write_batch_txn == nullptr) + throw0(DB_ERROR("batch transaction not in progress")); + check_open(); + LOG_PRINT_L3("batch transaction: committing..."); + TIME_MEASURE_START(time1); + m_write_txn->commit(); + TIME_MEASURE_FINISH(time1); + time_commit1 += time1; + // for destruction of batch transaction + m_write_txn = nullptr; + delete m_write_batch_txn; + m_write_batch_txn = nullptr; + m_batch_active = false; + LOG_PRINT_L3("batch transaction: end"); +} +void BlockchainLMDB::batch_abort() +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + if (! m_batch_transactions) + throw0(DB_ERROR("batch transactions not enabled")); + if (! m_batch_active) + throw0(DB_ERROR("batch transaction not in progress")); + check_open(); + // for destruction of batch transaction + m_write_txn = nullptr; + // explicitly call in case mdb_env_close() (BlockchainLMDB::close()) called before BlockchainLMDB destructor called. + m_write_batch_txn->abort(); + m_batch_active = false; + m_write_batch_txn = nullptr; + LOG_PRINT_L3("batch transaction: aborted"); +} +void BlockchainLMDB::set_batch_transactions(bool batch_transactions) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + m_batch_transactions = batch_transactions; + LOG_PRINT_L3("batch transactions " << (m_batch_transactions ? "enabled" : "disabled")); +} +uint64_t BlockchainLMDB::add_block(const block& blk, const size_t& block_size, const difficulty_type& cumulative_difficulty, const uint64_t& coins_generated, + const std::vector<transaction>& txs) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + if (m_height % 1000 == 0) + { + // for batch mode, DB resize check is done at start of batch transaction + if (! m_batch_active && need_resize()) + { + LOG_PRINT_L0("LMDB memory map needs resized, doing that now."); + do_resize(); + } + } + mdb_txn_safe txn; + if (! m_batch_active) + { + if (mdb_txn_begin(m_env, NULL, 0, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + m_write_txn = &txn; + } + uint64_t num_outputs = m_num_outputs; + try + { + BlockchainDB::add_block(blk, block_size, cumulative_difficulty, coins_generated, txs); + if (! m_batch_active) + { + m_write_txn = NULL; + TIME_MEASURE_START(time1); + txn.commit(); + TIME_MEASURE_FINISH(time1); + time_commit1 += time1; + } + } + catch (...) + { + m_num_outputs = num_outputs; + if (! m_batch_active) + m_write_txn = NULL; + throw; + } + return ++m_height; +} +void BlockchainLMDB::pop_block(block& blk, std::vector<transaction>& txs) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + mdb_txn_safe txn; + if (! m_batch_active) + { + if (mdb_txn_begin(m_env, NULL, 0, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + m_write_txn = &txn; + } + uint64_t num_outputs = m_num_outputs; + try + { + BlockchainDB::pop_block(blk, txs); + if (! m_batch_active) + { + m_write_txn = NULL; + txn.commit(); + } + } + catch (...) + { + m_num_outputs = num_outputs; + m_write_txn = NULL; + throw; + } + --m_height; +} +void BlockchainLMDB::get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices, + std::vector<tx_out_index> &tx_out_indices) const +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + tx_out_indices.clear(); + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + for (const uint64_t &index : global_indices) + { + MDB_val_copy<uint64_t> k(index); + MDB_val v; + auto get_result = mdb_get(*txn_ptr, m_output_txs, &k, &v); + if (get_result == MDB_NOTFOUND) + throw1(OUTPUT_DNE("output with given index not in db")); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch output tx hash")); + crypto::hash tx_hash = *(crypto::hash*) v.mv_data; + get_result = mdb_get(*txn_ptr, m_output_indices, &k, &v); + if (get_result == MDB_NOTFOUND) + throw1(OUTPUT_DNE("output with given index not in db")); + else if (get_result) + throw0(DB_ERROR("DB error attempting to fetch output tx index")); + auto result = tx_out_index(tx_hash, *(const uint64_t *) v.mv_data); + tx_out_indices.push_back(result); + } + if (!m_batch_active) + txn.commit(); +} +void BlockchainLMDB::get_output_global_indices(const uint64_t& amount, const std::vector<uint64_t> &offsets, + std::vector<uint64_t> &global_indices) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + TIME_MEASURE_START(txx); + check_open(); + global_indices.clear(); + uint64_t max = 0; + for (const uint64_t &index : offsets) + { + if (index > max) + max = index; + } + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if(m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + lmdb_cur cur(*txn_ptr, m_output_amounts); + MDB_val_copy<uint64_t> k(amount); + MDB_val v; + auto result = mdb_cursor_get(cur, &k, &v, MDB_SET); + if (result == MDB_NOTFOUND) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found")); + else if (result) + throw0(DB_ERROR("DB error attempting to get an output")); + size_t num_elems = 0; + mdb_cursor_count(cur, &num_elems); + if (max <= 1 && num_elems <= max) + throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but output not found")); + uint64_t t_dbmul = 0; + uint64_t t_dbscan = 0; + if (max <= 1) + { + for (const uint64_t& index : offsets) + { + mdb_cursor_get(cur, &k, &v, MDB_FIRST_DUP); + for (uint64_t i = 0; i < index; ++i) + { + mdb_cursor_get(cur, &k, &v, MDB_NEXT_DUP); + } + mdb_cursor_get(cur, &k, &v, MDB_GET_CURRENT); + uint64_t glob_index = *(const uint64_t*) v.mv_data; + LOG_PRINT_L3("Amount: " << amount << " M0->v: " << glob_index); + global_indices.push_back(glob_index); + } + } + else + { + uint32_t curcount = 0; + uint32_t blockstart = 0; + for (const uint64_t& index : offsets) + { + if (index >= num_elems) + { + LOG_PRINT_L1("Index: " << index << " Elems: " << num_elems << " partial results found for get_output_tx_and_index"); + break; + } + while (index >= curcount) + { + TIME_MEASURE_START(db1); + if (mdb_cursor_get(cur, &k, &v, curcount == 0 ? MDB_GET_MULTIPLE : MDB_NEXT_MULTIPLE) != 0) + { + // allow partial results + result = false; + break; + } + int count = v.mv_size / sizeof(uint64_t); + blockstart = curcount; + curcount += count; + TIME_MEASURE_FINISH(db1); + t_dbmul += db1; + } + LOG_PRINT_L3("Records returned: " << curcount << " Index: " << index); + TIME_MEASURE_START(db2); + uint64_t actual_index = index - blockstart; + uint64_t glob_index = ((const uint64_t*) v.mv_data)[actual_index]; + LOG_PRINT_L3("Amount: " << amount << " M1->v: " << glob_index); + global_indices.push_back(glob_index); + TIME_MEASURE_FINISH(db2); + t_dbscan += db2; + } + } + cur.close(); + if(!m_batch_active) + txn.commit(); + TIME_MEASURE_FINISH(txx); + LOG_PRINT_L3("txx: " << txx << " db1: " << t_dbmul << " db2: " << t_dbscan); +} +void BlockchainLMDB::get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + TIME_MEASURE_START(db3); + check_open(); + outputs.clear(); + std::vector <uint64_t> global_indices; + get_output_global_indices(amount, offsets, global_indices); + if (global_indices.size() > 0) + { + mdb_txn_safe txn; + mdb_txn_safe* txn_ptr = &txn; + if (m_batch_active) + txn_ptr = m_write_txn; + else + { + if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn)) + throw0(DB_ERROR("Failed to create a transaction for the db")); + } + for (const uint64_t &index : global_indices) + { + MDB_val_copy<uint64_t> k(index); + MDB_val v; + auto get_result = mdb_get(*txn_ptr, m_output_keys, &k, &v); + if (get_result != 0) + throw0(DB_ERROR("Attempting to get output pubkey by global index, but key does not exist")); + else if (get_result) + throw0(DB_ERROR("Error attempting to retrieve an output pubkey from the db")); + output_data_t data = *(output_data_t *) v.mv_data; + outputs.push_back(data); + } + if (!m_batch_active) + txn.commit(); + } + TIME_MEASURE_FINISH(db3); + LOG_PRINT_L3("db3: " << db3); +} +void BlockchainLMDB::get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) +{ + LOG_PRINT_L3("BlockchainLMDB::" << __func__); + check_open(); + indices.clear(); + std::vector <uint64_t> global_indices; + get_output_global_indices(amount, offsets, global_indices); + TIME_MEASURE_START(db3); + if(global_indices.size() > 0) + { + get_output_tx_and_index_from_global(global_indices, indices); + } + TIME_MEASURE_FINISH(db3); + LOG_PRINT_L3("db3: " << db3); +} +} // namespace cryptonote diff --git a/src/blockchain_db/lmdb/db_lmdb.h b/src/blockchain_db/lmdb/db_lmdb.h new file mode 100644 index 0000000..0facb87 +++ b/src/blockchain_db/lmdb/db_lmdb.h @@ -0,0 +1,320 @@ +namespace cryptonote +{ +struct mdb_txn_safe +{ + mdb_txn_safe(); + ~mdb_txn_safe(); + void commit(std::string message = ""); + // This should only be needed for batch transaction which must be ensured to + // be aborted before mdb_env_close, not after. So we can't rely on + // BlockchainLMDB destructor to call mdb_txn_safe destructor, as that's too late + // to properly abort, since mdb_env_close would have been called earlier. + void abort(); + operator MDB_txn*() + { + return m_txn; + } + operator MDB_txn**() + { + return &m_txn; + } + uint64_t num_active_tx() const; + static void prevent_new_txns(); + static void wait_no_active_txns(); + static void allow_new_txns(); + MDB_txn* m_txn; + bool m_batch_txn = false; + static std::atomic<uint64_t> num_active_txns; + // could use a mutex here, but this should be sufficient. + static std::atomic_flag creation_gate; +}; +class BlockchainLMDB : public BlockchainDB +{ +public: + BlockchainLMDB(bool batch_transactions=false); + ~BlockchainLMDB(); + virtual void open(const std::string& filename, const int mdb_flags=0); + virtual void close(); + virtual void sync(); + virtual void reset(); + virtual std::vector<std::string> get_filenames() const; + virtual std::string get_db_name() const; + virtual bool lock(); + virtual void unlock(); + virtual bool block_exists(const crypto::hash& h) const; + virtual block get_block(const crypto::hash& h) const; + virtual uint64_t get_block_height(const crypto::hash& h) const; + virtual block_header get_block_header(const crypto::hash& h) const; + virtual block get_block_from_height(const uint64_t& height) const; + virtual uint64_t get_block_timestamp(const uint64_t& height) const; + virtual uint64_t get_top_block_timestamp() const; + virtual size_t get_block_size(const uint64_t& height) const; + virtual difficulty_type get_block_cumulative_difficulty(const uint64_t& height) const; + virtual difficulty_type get_block_difficulty(const uint64_t& height) const; + virtual uint64_t get_block_already_generated_coins(const uint64_t& height) const; + virtual crypto::hash get_block_hash_from_height(const uint64_t& height) const; + virtual std::vector<block> get_blocks_range(const uint64_t& h1, const uint64_t& h2) const; + virtual std::vector<crypto::hash> get_hashes_range(const uint64_t& h1, const uint64_t& h2) const; + virtual crypto::hash top_block_hash() const; + virtual block get_top_block() const; + virtual uint64_t height() const; + virtual bool tx_exists(const crypto::hash& h) const; + virtual uint64_t get_tx_unlock_time(const crypto::hash& h) const; + virtual transaction get_tx(const crypto::hash& h) const; + virtual uint64_t get_tx_count() const; + virtual std::vector<transaction> get_tx_list(const std::vector<crypto::hash>& hlist) const; + virtual uint64_t get_tx_block_height(const crypto::hash& h) const; + virtual uint64_t get_random_output(const uint64_t& amount) const; + virtual uint64_t get_num_outputs(const uint64_t& amount) const; + virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index); + virtual output_data_t get_output_key(const uint64_t& global_index) const; + virtual void get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs); + virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const; + /** + * @brief get an output from its global index + * + * @param index global index of the output desired + * + * @return the output associated with the index. + * Will throw OUTPUT_DNE if not output has that global index. + * Will throw DB_ERROR if there is a non-specific LMDB error in fetching + */ + tx_out get_output(const uint64_t& index) const; + virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const; + virtual void get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices, + std::vector<tx_out_index> &tx_out_indices) const; + virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index); + virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices); + virtual void get_output_global_indices(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<uint64_t> &indices); + virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const; + virtual std::vector<uint64_t> get_tx_amount_output_indices(const crypto::hash& h) const; + virtual bool has_key_image(const crypto::key_image& img) const; + virtual uint64_t add_block( const block& blk + , const size_t& block_size + , const difficulty_type& cumulative_difficulty + , const uint64_t& coins_generated + , const std::vector<transaction>& txs + ); + virtual void set_batch_transactions(bool batch_transactions); + virtual void batch_start(uint64_t batch_num_blocks=0); + virtual void batch_commit(); + virtual void batch_stop(); + virtual void batch_abort(); + virtual void pop_block(block& blk, std::vector<transaction>& txs); + virtual bool can_thread_bulk_indices() const { return true; } +private: + void do_resize(uint64_t size_increase=0); + bool need_resize(uint64_t threshold_size=0) const; + void check_and_resize_for_batch(uint64_t batch_num_blocks); + uint64_t get_estimated_batch_size(uint64_t batch_num_blocks) const; + virtual void add_block( const block& blk + , const size_t& block_size + , const difficulty_type& cumulative_difficulty + , const uint64_t& coins_generated + , const crypto::hash& block_hash + ); + virtual void remove_block(); + virtual void add_transaction_data(const crypto::hash& blk_hash, const transaction& tx, const crypto::hash& tx_hash); + virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx); + virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time); + virtual void remove_output(const tx_out& tx_output); + void remove_tx_outputs(const crypto::hash& tx_hash, const transaction& tx); + void remove_output(const uint64_t& out_index, const uint64_t amount); + void remove_amount_output_index(const uint64_t amount, const uint64_t global_output_index); + virtual void add_spent_key(const crypto::key_image& k_image); + virtual void remove_spent_key(const crypto::key_image& k_image); + /** + * @brief convert a tx output to a blob for storage + * + * @param output the output to convert + * + * @return the resultant blob + */ + blobdata output_to_blob(const tx_out& output) const; + /** + * @brief convert a tx output blob to a tx output + * + * @param blob the blob to convert + * + * @return the resultant tx output + */ + tx_out output_from_blob(const blobdata& blob) const; + /** + * @brief get the global index of the index-th output of the given amount + * + * @param amount the output amount + * @param index the index into the set of outputs of that amount + * + * @return the global index of the desired output + */ + uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index); + void check_open() const; + MDB_env* m_env; + MDB_dbi m_blocks; + MDB_dbi m_block_heights; + MDB_dbi m_block_hashes; + MDB_dbi m_block_timestamps; + MDB_dbi m_block_sizes; + MDB_dbi m_block_diffs; + MDB_dbi m_block_coins; + MDB_dbi m_txs; + MDB_dbi m_tx_unlocks; + MDB_dbi m_tx_heights; + MDB_dbi m_tx_outputs; + MDB_dbi m_output_txs; + MDB_dbi m_output_indices; + MDB_dbi m_output_gindices; + MDB_dbi m_output_amounts; + MDB_dbi m_output_keys; + MDB_dbi m_outputs; + MDB_dbi m_spent_keys; + uint64_t m_height; + uint64_t m_num_outputs; + std::string m_folder; + mdb_txn_safe* m_write_txn; // may point to either a short-lived txn or a batch txn + mdb_txn_safe* m_write_batch_txn; // persist batch txn outside of BlockchainLMDB + bool m_batch_transactions; // support for batch transactions + bool m_batch_active; // whether batch transaction is in progress + // force a value so it can compile with 32-bit ARM + constexpr static uint64_t DEFAULT_MAPSIZE = 1LL << 31; + constexpr static uint64_t DEFAULT_MAPSIZE = 1LL << 30; + constexpr static uint64_t DEFAULT_MAPSIZE = 1LL << 33; + constexpr static float RESIZE_PERCENT = 0.8f; +}; +} // namespace cryptonote diff --git a/src/blockchain_utilities/CMakeLists.txt b/src/blockchain_utilities/CMakeLists.txt new file mode 100644 index 0000000..5be37c4 +++ b/src/blockchain_utilities/CMakeLists.txt @@ -0,0 +1,118 @@ +set(blockchain_converter_sources + blockchain_converter.cpp + ) +set(blockchain_converter_private_headers) +bitmonero_private_headers(blockchain_converter + ${blockchain_converter_private_headers}) +set(blockchain_import_sources + blockchain_import.cpp + bootstrap_file.cpp + ) +set(blockchain_import_private_headers + fake_core.h + bootstrap_file.h + bootstrap_serialization.h + ) +bitmonero_private_headers(blockchain_import + ${blockchain_import_private_headers}) +set(blockchain_export_sources + blockchain_export.cpp + bootstrap_file.cpp + ) +set(blockchain_export_private_headers + bootstrap_file.h + bootstrap_serialization.h + ) +bitmonero_private_headers(blockchain_export + ${blockchain_export_private_headers}) +if (BLOCKCHAIN_DB STREQUAL DB_LMDB) +bitmonero_add_executable(blockchain_converter + ${blockchain_converter_sources} + ${blockchain_converter_private_headers}) +target_link_libraries(blockchain_converter + LINK_PRIVATE + cryptonote_core + p2p + blockchain_db + ${CMAKE_THREAD_LIBS_INIT}) +add_dependencies(blockchain_converter + version) +set_property(TARGET blockchain_converter + PROPERTY + OUTPUT_NAME "blockchain_converter") +endif () +bitmonero_add_executable(blockchain_import + ${blockchain_import_sources} + ${blockchain_import_private_headers}) +target_link_libraries(blockchain_import + LINK_PRIVATE + cryptonote_core + blockchain_db + p2p + ${CMAKE_THREAD_LIBS_INIT}) +add_dependencies(blockchain_import + version) +set_property(TARGET blockchain_import + PROPERTY + OUTPUT_NAME "blockchain_import") +bitmonero_add_executable(blockchain_export + ${blockchain_export_sources} + ${blockchain_export_private_headers}) +target_link_libraries(blockchain_export + LINK_PRIVATE + cryptonote_core + p2p + blockchain_db + ${CMAKE_THREAD_LIBS_INIT}) +add_dependencies(blockchain_export + version) +set_property(TARGET blockchain_export + PROPERTY + OUTPUT_NAME "blockchain_export") diff --git a/src/blockchain_utilities/README.md b/src/blockchain_utilities/README.md new file mode 100644 index 0000000..9d24b9a +++ b/src/blockchain_utilities/README.md @@ -0,0 +1,68 @@ +Copyright (c) 2014-2015, The Monero Project +The blockchain utilities allow one to convert an old style blockchain.bin file +to a new style database. There are two ways to upgrade an old style blockchain: +The recommended way is to run a `blockchain_export`, then `blockchain_import`. +The other way is to run `blockchain_converter`. In both cases, you will be left +with a new style blockchain. +For importing into the LMDB database, compile with `DATABASE=lmdb` +e.g. +`DATABASE=lmdb make release` +This is also the default compile setting on the master branch. +By default, the exporter will use the original in-memory database (blockchain.bin) as its source. +This default is to make migrating to an LMDB database easy, without having to recompile anything. +To change the source, adjust `SOURCE_DB` in `src/blockchain_utilities/bootstrap_file.h` according to the comments. +See also each utility's "--help" option. +`$ blockchain_export` +This loads the existing blockchain, for whichever database type it was compiled for, and exports it to `$MONERO_DATA_DIR/export/blockchain.raw` +`$ blockchain_import`

Continuation of: Git diff from commit e940386f9a8765423ab3dd9e3aabe19a68cba9f9 (thankful_for_today's last commit) to current HEAD, excluding changes in /external so as to exclude libraries and submoduled code, and excluding contrib/ so as to exclude changes made to epee, excluding most comments because apparently nobody that writes code for BCN knows what a comment is anyway, excluding removed lines, excluding empty lines

+This imports blocks from `$MONERO_DATA_DIR/export/blockchain.raw` into the current database. +Defaults: `--batch on`, `--batch size 20000`, `--verify on` +Batch size refers to number of blocks and can be adjusted for performance based on available RAM. +Verification should only be turned off if importing from a trusted blockchain. +If you encounter an error like "resizing not supported in batch mode", you can just re-run +the `blockchain_import` command again, and it will restart from where it left off. +```bash +$ blockchain_import +$ blockchain_import --batch-size 100000 --verify off +$ blockchain_import --database lmdb#nosync +$ blockchain_import --database lmdb#nosync,nometasync +``` +`blockchain_converter` has also been updated and includes batching for faster writes. However, on lower RAM systems, this will be slower than using the exporter and importer utilities. The converter needs to keep the blockchain in memory for the duration of the conversion, like the original bitmonerod, thus leaving less memory available to the destination database to operate. +```bash +$ blockchain_converter --batch on --batch-size 20000 +``` diff --git a/src/blockchain_utilities/blockchain_converter.cpp b/src/blockchain_utilities/blockchain_converter.cpp new file mode 100644 index 0000000..d18ce87 +++ b/src/blockchain_utilities/blockchain_converter.cpp @@ -0,0 +1,308 @@ +unsigned int epee::g_test_dbg_lock_sleep = 0; +namespace +{ +bool opt_batch = true; +bool opt_resume = true; +bool opt_testnet = false; +uint64_t db_batch_size_verify = 5000; +uint64_t db_batch_size_verify = 1000; +uint64_t db_batch_size = db_batch_size_verify; +} +namespace po = boost::program_options; +using namespace cryptonote; +using namespace epee; +struct fake_core +{ + Blockchain dummy; + tx_memory_pool m_pool; + blockchain_storage m_storage; + // for multi_db_runtime: + fake_core(const boost::filesystem::path &path, const bool use_testnet) : dummy(m_pool), m_pool(&dummy), m_storage(m_pool) + // for multi_db_compile: + fake_core(const boost::filesystem::path &path, const bool use_testnet) : dummy(m_pool), m_pool(dummy), m_storage(&m_pool) + { + m_pool.init(path.string()); + m_storage.init(path.string(), use_testnet); + } +}; +int main(int argc, char* argv[]) +{ + uint64_t height = 0; + uint64_t start_block = 0; + uint64_t end_block = 0; + uint64_t num_blocks = 0; + boost::filesystem::path default_data_path {tools::get_default_data_dir()}; + boost::filesystem::path default_testnet_data_path {default_data_path / "testnet"}; + po::options_description desc_cmd_only("Command line options"); + po::options_description desc_cmd_sett("Command line options and settings options"); + const command_line::arg_descriptor<uint32_t> arg_log_level = {"log-level", "", LOG_LEVEL_0}; + const command_line::arg_descriptor<uint64_t> arg_batch_size = {"batch-size", "", db_batch_size}; + const command_line::arg_descriptor<bool> arg_testnet_on = { + "testnet" + , "Run on testnet." + , opt_testnet + }; + const command_line::arg_descriptor<uint64_t> arg_block_number = + {"block-number", "Number of blocks (default: use entire source blockchain)", + 0}; + command_line::add_arg(desc_cmd_sett, command_line::arg_data_dir, default_data_path.string()); + command_line::add_arg(desc_cmd_sett, command_line::arg_testnet_data_dir, default_testnet_data_path.string()); + command_line::add_arg(desc_cmd_sett, arg_log_level); + command_line::add_arg(desc_cmd_sett, arg_batch_size); + command_line::add_arg(desc_cmd_sett, arg_testnet_on); + command_line::add_arg(desc_cmd_sett, arg_block_number); + command_line::add_arg(desc_cmd_only, command_line::arg_help); + const command_line::arg_descriptor<bool> arg_batch = {"batch", + "Batch transactions for faster import", true}; + const command_line::arg_descriptor<bool> arg_resume = {"resume", + "Resume from current height if output database already exists", true}; + // call add_options() directly for these arguments since command_line helpers + // support only boolean switch, not boolean argument + desc_cmd_sett.add_options() + (arg_batch.name, make_semantic(arg_batch), arg_batch.description) + (arg_resume.name, make_semantic(arg_resume), arg_resume.description) + ; + po::options_description desc_options("Allowed options"); + desc_options.add(desc_cmd_only).add(desc_cmd_sett); + po::variables_map vm; + bool r = command_line::handle_error_helper(desc_options, [&]() + { + po::store(po::parse_command_line(argc, argv, desc_options), vm); + po::notify(vm); + return true; + }); + if (!r) + return 1; + int log_level = command_line::get_arg(vm, arg_log_level); + opt_batch = command_line::get_arg(vm, arg_batch); + opt_resume = command_line::get_arg(vm, arg_resume); + db_batch_size = command_line::get_arg(vm, arg_batch_size); + if (command_line::get_arg(vm, command_line::arg_help)) + { + std::cout << CRYPTONOTE_NAME << " v" << MONERO_VERSION_FULL << ENDL << ENDL; + std::cout << desc_options << std::endl; + return 1; + } + if (! opt_batch && ! vm["batch-size"].defaulted()) + { + std::cerr << "Error: batch-size set, but batch option not enabled" << ENDL; + return 1; + } + if (! db_batch_size) + { + std::cerr << "Error: batch-size must be > 0" << ENDL; + return 1; + } + log_space::get_set_log_detalisation_level(true, log_level); + log_space::log_singletone::add_logger(LOGGER_CONSOLE, NULL, NULL); + LOG_PRINT_L0("Starting..."); + std::string src_folder; + opt_testnet = command_line::get_arg(vm, arg_testnet_on); + auto data_dir_arg = opt_testnet ? command_line::arg_testnet_data_dir : command_line::arg_data_dir; + src_folder = command_line::get_arg(vm, data_dir_arg); + boost::filesystem::path dest_folder(src_folder); + num_blocks = command_line::get_arg(vm, arg_block_number); + if (opt_batch) + { + LOG_PRINT_L0("batch: " << std::boolalpha << opt_batch << std::noboolalpha + << " batch size: " << db_batch_size); + } + else + { + LOG_PRINT_L0("batch: " << std::boolalpha << opt_batch << std::noboolalpha); + } + LOG_PRINT_L0("resume: " << std::boolalpha << opt_resume << std::noboolalpha); + LOG_PRINT_L0("testnet: " << std::boolalpha << opt_testnet << std::noboolalpha); + fake_core c(src_folder, opt_testnet); + height = c.m_storage.get_current_blockchain_height(); + BlockchainDB *blockchain; + blockchain = new BlockchainLMDB(opt_batch); + dest_folder /= blockchain->get_db_name(); + LOG_PRINT_L0("Source blockchain: " << src_folder); + LOG_PRINT_L0("Dest blockchain: " << dest_folder.string()); + LOG_PRINT_L0("Opening dest blockchain (BlockchainDB " << blockchain->get_db_name() << ")"); + blockchain->open(dest_folder.string()); + LOG_PRINT_L0("Source blockchain height: " << height); + LOG_PRINT_L0("Dest blockchain height: " << blockchain->height()); + if (opt_resume) + // next block number to add is same as current height + start_block = blockchain->height(); + if (! num_blocks || (start_block + num_blocks > height)) + end_block = height - 1; + else + end_block = start_block + num_blocks - 1; + LOG_PRINT_L0("start height: " << start_block+1 << " stop height: " << + end_block+1); + if (start_block > end_block) + { + LOG_PRINT_L0("Finished: no blocks to add"); + delete blockchain; + return 0; + } + if (opt_batch) + blockchain->batch_start(db_batch_size); + uint64_t i = 0; + for (i = start_block; i < end_block + 1; ++i) + { + // block: i height: i+1 end height: end_block + 1 + if ((i+1) % 10 == 0) + { + std::cout << "\r \r" << "height " << i+1 << "/" << + end_block+1 << " (" << (i+1)*100/(end_block+1)<< "%)" << std::flush; + } + // for debugging: + // std::cout << "height " << i+1 << "/" << end_block+1 + // << " ((" << i+1 << ")*100/(end_block+1))" << "%)" << ENDL; + block b = c.m_storage.get_block(i); + size_t bsize = c.m_storage.get_block_size(i); + difficulty_type bdiff = c.m_storage.get_block_cumulative_difficulty(i); + uint64_t bcoins = c.m_storage.get_block_coins_generated(i); + std::vector<transaction> txs; + std::vector<crypto::hash> missed; + c.m_storage.get_transactions(b.tx_hashes, txs, missed); + if (missed.size()) + { + std::cout << ENDL; + std::cerr << "Missed transaction(s) for block at height " << i + 1 << ", exiting" << ENDL; + delete blockchain; + return 1; + } + try + { + blockchain->add_block(b, bsize, bdiff, bcoins, txs); + if (opt_batch) + { + if ((i < end_block) && ((i + 1) % db_batch_size == 0)) + { + std::cout << "\r \r"; + std::cout << "[- batch commit at height " << i + 1 << " -]" << ENDL; + blockchain->batch_stop(); + blockchain->batch_start(db_batch_size); + std::cout << ENDL; + blockchain->show_stats(); + } + } + } + catch (const std::exception& e) + { + std::cout << ENDL; + std::cerr << "Error adding block " << i << " to new blockchain: " << e.what() << ENDL; + delete blockchain; + return 2; + } + } + if (opt_batch) + { + std::cout << "\r \r" << "height " << i << "/" << + end_block+1 << " (" << (i)*100/(end_block+1)<< "%)" << std::flush; + std::cout << ENDL; + std::cout << "[- batch commit at height " << i << " -]" << ENDL; + blockchain->batch_stop(); + } + std::cout << ENDL; + blockchain->show_stats(); + std::cout << "Finished at height: " << i << " block: " << i-1 << ENDL; + delete blockchain; + return 0; +} diff --git a/src/blockchain_utilities/blockchain_export.cpp b/src/blockchain_utilities/blockchain_export.cpp new file mode 100644 index 0000000..ec885ea +++ b/src/blockchain_utilities/blockchain_export.cpp @@ -0,0 +1,157 @@ +unsigned int epee::g_test_dbg_lock_sleep = 0; +namespace po = boost::program_options; +using namespace epee; // log_space +int main(int argc, char* argv[]) +{ + uint32_t log_level = 0; + uint64_t block_stop = 0; + std::string import_filename = BLOCKCHAIN_RAW; + boost::filesystem::path default_data_path {tools::get_default_data_dir()}; + boost::filesystem::path default_testnet_data_path {default_data_path / "testnet"}; + po::options_description desc_cmd_only("Command line options"); + po::options_description desc_cmd_sett("Command line options and settings options"); + const command_line::arg_descriptor<uint32_t> arg_log_level = {"log-level", "", log_level}; + const command_line::arg_descriptor<uint64_t> arg_block_stop = {"block-stop", "Stop at block number", block_stop}; + const command_line::arg_descriptor<bool> arg_testnet_on = { + "testnet" + , "Run on testnet." + , false + }; + command_line::add_arg(desc_cmd_sett, command_line::arg_data_dir, default_data_path.string()); + command_line::add_arg(desc_cmd_sett, command_line::arg_testnet_data_dir, default_testnet_data_path.string()); + command_line::add_arg(desc_cmd_sett, arg_testnet_on); + command_line::add_arg(desc_cmd_sett, arg_log_level); + command_line::add_arg(desc_cmd_sett, arg_block_stop); + command_line::add_arg(desc_cmd_only, command_line::arg_help); + po::options_description desc_options("Allowed options"); + desc_options.add(desc_cmd_only).add(desc_cmd_sett); + po::variables_map vm; + bool r = command_line::handle_error_helper(desc_options, [&]() + { + po::store(po::parse_command_line(argc, argv, desc_options), vm); + po::notify(vm); + return true; + }); + if (! r) + return 1; + if (command_line::get_arg(vm, command_line::arg_help)) + { + std::cout << CRYPTONOTE_NAME << " v" << MONERO_VERSION_FULL << ENDL << ENDL; + std::cout << desc_options << std::endl; + return 1; + } + log_level = command_line::get_arg(vm, arg_log_level); + block_stop = command_line::get_arg(vm, arg_block_stop); + log_space::get_set_log_detalisation_level(true, log_level); + log_space::log_singletone::add_logger(LOGGER_CONSOLE, NULL, NULL); + LOG_PRINT_L0("Starting..."); + LOG_PRINT_L0("Setting log level = " << log_level); + bool opt_testnet = command_line::get_arg(vm, arg_testnet_on); + std::string m_config_folder; + auto data_dir_arg = opt_testnet ? command_line::arg_testnet_data_dir : command_line::arg_data_dir; + m_config_folder = command_line::get_arg(vm, data_dir_arg); + boost::filesystem::path output_dir {m_config_folder}; + output_dir /= "export"; + LOG_PRINT_L0("Export directory: " << output_dir.string()); + // If we wanted to use the memory pool, we would set up a fake_core. + // blockchain_storage* core_storage = NULL; + // tx_memory_pool m_mempool(*core_storage); // is this fake anyway? just passing in NULL! so m_mempool can't be used anyway, right? + // core_storage = new blockchain_storage(&m_mempool); + blockchain_storage* core_storage = new blockchain_storage(NULL); + LOG_PRINT_L0("Initializing source blockchain (in-memory database)"); + r = core_storage->init(m_config_folder, opt_testnet); + // Use Blockchain instead of lower-level BlockchainDB for two reasons: + // 1. Blockchain has the init() method for easy setup + // 2. exporter needs to use get_current_blockchain_height(), get_block_id_by_height(), get_block_by_hash() + // + // cannot match blockchain_storage setup above with just one line, + // e.g. + // Blockchain* core_storage = new Blockchain(NULL); + // because unlike blockchain_storage constructor, which takes a pointer to + // tx_memory_pool, Blockchain's constructor takes tx_memory_pool object. + LOG_PRINT_L0("Initializing source blockchain (BlockchainDB)"); + Blockchain* core_storage = NULL; + tx_memory_pool m_mempool(*core_storage); + core_storage = new Blockchain(m_mempool); + BlockchainDB* db = new BlockchainLMDB(); + boost::filesystem::path folder(m_config_folder); + folder /= db->get_db_name(); + int lmdb_flags = 0; + lmdb_flags |= MDB_RDONLY; + const std::string filename = folder.string(); + LOG_PRINT_L0("Loading blockchain from folder " << filename << " ..."); + try + { + db->open(filename, lmdb_flags); + } + catch (const std::exception& e) + { + LOG_PRINT_L0("Error opening database: " << e.what()); + throw; + } + r = core_storage->init(db, opt_testnet); + CHECK_AND_ASSERT_MES(r, false, "Failed to initialize source blockchain storage"); + LOG_PRINT_L0("Source blockchain storage initialized OK"); + LOG_PRINT_L0("Exporting blockchain raw data..."); + BootstrapFile bootstrap; + r = bootstrap.store_blockchain_raw(core_storage, NULL, output_dir, block_stop); + CHECK_AND_ASSERT_MES(r, false, "Failed to export blockchain raw data"); + LOG_PRINT_L0("Blockchain raw data exported OK"); +} diff --git a/src/blockchain_utilities/blockchain_import.cpp b/src/blockchain_utilities/blockchain_import.cpp new file mode 100644 index 0000000..924b46d +++ b/src/blockchain_utilities/blockchain_import.cpp @@ -0,0 +1,770 @@ +unsigned int epee::g_test_dbg_lock_sleep = 0; +namespace +{ +bool opt_batch = true; +bool opt_verify = true; // use add_new_block, which does verification before calling add_block +bool opt_resume = true; +bool opt_testnet = true; +uint64_t db_batch_size = 20000; +uint64_t db_batch_size = 1000; +uint64_t db_batch_size_verify = 5000; +std::string refresh_string = "\r \r"; +} +namespace po = boost::program_options; +using namespace cryptonote; +using namespace epee; +int parse_db_arguments(const std::string& db_arg_str, std::string& db_engine, int& mdb_flags) +{ + std::vector<std::string> db_args; + boost::split(db_args, db_arg_str, boost::is_any_of("#")); + db_engine = db_args.front(); + boost::algorithm::trim(db_engine); + if (db_args.size() == 1) + { + return 0; + } + else if (db_args.size() > 2) + { + std::cerr << "unrecognized database argument format: " << db_arg_str << ENDL; + return 1; + } + std::string db_arg_str2 = db_args[1]; + boost::split(db_args, db_arg_str2, boost::is_any_of(",")); + for (auto& it : db_args) + { + boost::algorithm::trim(it); + if (it.empty()) + continue; + LOG_PRINT_L1("LMDB flag: " << it); + if (it == "nosync") + mdb_flags |= MDB_NOSYNC; + else if (it == "nometasync") + mdb_flags |= MDB_NOMETASYNC; + else if (it == "writemap") + mdb_flags |= MDB_WRITEMAP; + else if (it == "mapasync") + mdb_flags |= MDB_MAPASYNC; + else if (it == "nordahead") + mdb_flags |= MDB_NORDAHEAD; + else + { + std::cerr << "unrecognized database flag: " << it << ENDL; + return 1; + } + } + return 0; +} +template <typename FakeCore> +int pop_blocks(FakeCore& simple_core, int num_blocks) +{ + bool use_batch = false; + if (opt_batch) + { + if (simple_core.support_batch) + use_batch = true; + else + LOG_PRINT_L0("WARNING: batch transactions enabled but unsupported or unnecessary for this database engine - ignoring"); + } + if (use_batch) + simple_core.batch_start(); + int quit = 0; + block popped_block; + std::vector<transaction> popped_txs; + for (int i=0; i < num_blocks; ++i) + { + simple_core.m_storage.debug_pop_block_from_blockchain(); + // simple_core.m_storage.pop_block_from_blockchain() is private, so call directly through db + simple_core.m_storage.get_db().pop_block(popped_block, popped_txs); + quit = 1; + } + if (use_batch) + { + if (quit > 1) + { + // There was an error, so don't commit pending data. + // Destructor will abort write txn. + } + else + { + simple_core.batch_stop(); + } + simple_core.m_storage.get_db().show_stats(); + } + return num_blocks; +} +template <typename FakeCore> +int import_from_file(FakeCore& simple_core, std::string& import_file_path, uint64_t block_stop=0) +{ + static_assert(std::is_same<fake_core_memory, FakeCore>::value || std::is_same<fake_core_lmdb, FakeCore>::value, + "FakeCore constraint error"); + if (std::is_same<fake_core_lmdb, FakeCore>::value) + { + // Reset stats, in case we're using newly created db, accumulating stats + // from addition of genesis block. + // This aligns internal db counts with importer counts. + simple_core.m_storage.get_db().reset_stats(); + } + boost::filesystem::path raw_file_path(import_file_path); + boost::system::error_code ec; + if (!boost::filesystem::exists(raw_file_path, ec)) + { + LOG_PRINT_L0("bootstrap file not found: " << raw_file_path); + return false; + } + BootstrapFile bootstrap; + // BootstrapFile bootstrap(import_file_path); + uint64_t total_source_blocks = bootstrap.count_blocks(import_file_path); + LOG_PRINT_L0("bootstrap file last block number: " << total_source_blocks-1 << " (zero-based height) total blocks: " << total_source_blocks); + std::cout << ENDL; + std::cout << "Preparing to read blocks..." << ENDL; + std::cout << ENDL; + std::ifstream import_file; + import_file.open(import_file_path, std::ios_base::binary | std::ifstream::in); + uint64_t h = 0; + uint64_t num_imported = 0; + if (import_file.fail()) + { + LOG_PRINT_L0("import_file.open() fail"); + return false; + } + // 4 byte magic + (currently) 1024 byte header structures + bootstrap.seek_to_first_chunk(import_file); + std::string str1; + char buffer1[1024]; + char buffer_block[BUFFER_SIZE]; + block b; + transaction tx; + int quit = 0; + uint64_t bytes_read = 0; + uint64_t start_height = 1; + if (opt_resume) + start_height = simple_core.m_storage.get_current_blockchain_height(); + // Note that a new blockchain will start with block number 0 (total blocks: 1) + // due to genesis block being added at initialization. + if (! block_stop) + { + block_stop = total_source_blocks - 1; + } + // These are what we'll try to use, and they don't have to be a determination + // from source and destination blockchains, but those are the defaults. + LOG_PRINT_L0("start block: " << start_height << " stop block: " << + block_stop); + bool use_batch = false; + if (opt_batch) + { + if (simple_core.support_batch) + use_batch = true; + else + LOG_PRINT_L0("WARNING: batch transactions enabled but unsupported or unnecessary for this database engine - ignoring"); + } + if (use_batch) + simple_core.batch_start(db_batch_size); + LOG_PRINT_L0("Reading blockchain from bootstrap file..."); + std::cout << ENDL; + // Within the loop, we skip to start_height before we start adding. + // TODO: Not a bottleneck, but we can use what's done in count_blocks() and + // only do the chunk size reads, skipping the chunk content reads until we're + // at start_height. + while (! quit) + { + uint32_t chunk_size; + import_file.read(buffer1, sizeof(chunk_size)); + // TODO: bootstrap.read_chunk(); + if (! import_file) { + std::cout << refresh_string; + LOG_PRINT_L0("End of file reached"); + quit = 1; + break; + } + bytes_read += sizeof(chunk_size); + str1.assign(buffer1, sizeof(chunk_size)); + if (! ::serialization::parse_binary(str1, chunk_size)) + { + throw std::runtime_error("Error in deserialization of chunk size"); + } + LOG_PRINT_L3("chunk_size: " << chunk_size); + if (chunk_size > BUFFER_SIZE) + { + LOG_PRINT_L0("WARNING: chunk_size " << chunk_size << " > BUFFER_SIZE " << BUFFER_SIZE); + throw std::runtime_error("Aborting: chunk size exceeds buffer size"); + } + if (chunk_size > 100000) + { + LOG_PRINT_L0("NOTE: chunk_size " << chunk_size << " > 100000"); + } + else if (chunk_size == 0) { + LOG_PRINT_L0("ERROR: chunk_size == 0"); + return 2; + } + import_file.read(buffer_block, chunk_size); + if (! import_file) { + LOG_PRINT_L0("ERROR: unexpected end of file: bytes read before error: " + << import_file.gcount() << " of chunk_size " << chunk_size); + return 2; + } + bytes_read += chunk_size; + LOG_PRINT_L3("Total bytes read: " << bytes_read); + if (h + NUM_BLOCKS_PER_CHUNK < start_height + 1) + { + h += NUM_BLOCKS_PER_CHUNK; + continue; + } + if (h > block_stop) + { + std::cout << refresh_string << "block " << h-1 + << " / " << block_stop + << std::flush; + std::cout << ENDL << ENDL; + LOG_PRINT_L0("Specified block number reached - stopping. block: " << h-1 << " total blocks: " << h); + quit = 1; + break; + } + try + { + str1.assign(buffer_block, chunk_size); + bootstrap::block_package bp; + if (! ::serialization::parse_binary(str1, bp)) + throw std::runtime_error("Error in deserialization of chunk"); + int display_interval = 1000; + int progress_interval = 10; + // NOTE: use of NUM_BLOCKS_PER_CHUNK is a placeholder in case multi-block chunks are later supported. + for (int chunk_ind = 0; chunk_ind < NUM_BLOCKS_PER_CHUNK; ++chunk_ind) + { + ++h; + if ((h-1) % display_interval == 0) + { + std::cout << refresh_string; + LOG_PRINT_L0("loading block number " << h-1); + } + else + { + LOG_PRINT_L3("loading block number " << h-1); + } + b = bp.block; + LOG_PRINT_L2("block prev_id: " << b.prev_id << ENDL); + if ((h-1) % progress_interval == 0) + { + std::cout << refresh_string << "block " << h-1 + << " / " << block_stop + << std::flush; + } + std::vector<transaction> txs; + std::vector<transaction> archived_txs; + archived_txs = bp.txs; + // std::cout << refresh_string; + // LOG_PRINT_L1("txs: " << archived_txs.size()); + // if archived_txs is invalid + // { + // std::cout << refresh_string; + // LOG_PRINT_RED_L0("exception while de-archiving txs, height=" << h); + // quit = 1; + // break; + // } + // tx number 1: coinbase tx + // tx number 2 onwards: archived_txs + unsigned int tx_num = 1; + for (const transaction& tx : archived_txs) + { + ++tx_num; + // if tx is invalid + // { + // LOG_PRINT_RED_L0("exception while indexing tx from txs, height=" << h <<", tx_num=" << tx_num); + // quit = 1; + // break; + // } + // std::cout << refresh_string; + // LOG_PRINT_L1("tx hash: " << get_transaction_hash(tx)); + // crypto::hash hsh = null_hash; + // size_t blob_size = 0; + // NOTE: all tx hashes except for coinbase tx are available in the block data + // get_transaction_hash(tx, hsh, blob_size); + // LOG_PRINT_L0("tx " << tx_num << " " << hsh << " : " << ENDL); + // LOG_PRINT_L0(obj_to_json_str(tx) << ENDL); + // add blocks with verification. + // for Blockchain and blockchain_storage add_new_block(). + if (opt_verify) + { + // crypto::hash hsh = null_hash; + // size_t blob_size = 0; + // get_transaction_hash(tx, hsh, blob_size); + tx_verification_context tvc = AUTO_VAL_INIT(tvc); + bool r = true; + r = simple_core.m_pool.add_tx(tx, tvc, true); + if (!r) + { + LOG_PRINT_RED_L0("failed to add transaction to transaction pool, height=" << h <<", tx_num=" << tx_num); + quit = 1; + break; + } + } + else + { + // for add_block() method, without (much) processing. + // don't add coinbase transaction to txs. + // + // because add_block() calls + // add_transaction(blk_hash, blk.miner_tx) first, and + // then a for loop for the transactions in txs. + txs.push_back(tx); + } + } + if (opt_verify) + { + block_verification_context bvc = boost::value_initialized<block_verification_context>(); + simple_core.m_storage.add_new_block(b, bvc); + if (bvc.m_verifivation_failed) + { + LOG_PRINT_L0("Failed to add block to blockchain, verification failed, height = " << h); + LOG_PRINT_L0("skipping rest of file"); + // ok to commit previously batched data because it failed only in + // verification of potential new block with nothing added to batch + // yet + quit = 1; + break; + } + if (! bvc.m_added_to_main_chain) + { + LOG_PRINT_L0("Failed to add block to blockchain, height = " << h); + LOG_PRINT_L0("skipping rest of file"); + // make sure we don't commit partial block data + quit = 2; + break; + } + } + else + { + size_t block_size; + difficulty_type cumulative_difficulty; + uint64_t coins_generated; + block_size = bp.block_size; + cumulative_difficulty = bp.cumulative_difficulty; + coins_generated = bp.coins_generated; + // std::cout << refresh_string; + // LOG_PRINT_L2("block_size: " << block_size); + // LOG_PRINT_L2("cumulative_difficulty: " << cumulative_difficulty); + // LOG_PRINT_L2("coins_generated: " << coins_generated); + try + { + simple_core.add_block(b, block_size, cumulative_difficulty, coins_generated, txs); + } + catch (const std::exception& e) + { + std::cout << refresh_string; + LOG_PRINT_RED_L0("Error adding block to blockchain: " << e.what()); + quit = 2; // make sure we don't commit partial block data + break; + } + } + ++num_imported; + if (use_batch) + { + if ((h-1) % db_batch_size == 0) + { + std::cout << refresh_string; + // zero-based height + std::cout << ENDL << "[- batch commit at height " << h-1 << " -]" << ENDL; + simple_core.batch_stop(); + simple_core.batch_start(db_batch_size); + std::cout << ENDL; + simple_core.m_storage.get_db().show_stats(); + } + } + } + } + catch (const std::exception& e) + { + std::cout << refresh_string; + LOG_PRINT_RED_L0("exception while reading from file, height=" << h); + return 2; + } + } // while + import_file.close(); + if (use_batch) + { + if (quit > 1) + { + // There was an error, so don't commit pending data. + // Destructor will abort write txn. + } + else + { + simple_core.batch_stop(); + } + simple_core.m_storage.get_db().show_stats(); + LOG_PRINT_L0("Number of blocks imported: " << num_imported) + if (h > 0) + // TODO: if there was an error, the last added block is probably at zero-based height h-2 + LOG_PRINT_L0("Finished at block: " << h-1 << " total blocks: " << h); + } + std::cout << ENDL; + return 0; +} +int main(int argc, char* argv[]) +{ + std::string import_filename = BLOCKCHAIN_RAW; + std::string default_db_engine = "memory"; + std::string default_db_engine = "lmdb"; + uint32_t log_level = LOG_LEVEL_0; + uint64_t num_blocks = 0; + uint64_t block_stop = 0; + std::string dirname; + std::string db_arg_str; + boost::filesystem::path default_data_path {tools::get_default_data_dir()}; + boost::filesystem::path default_testnet_data_path {default_data_path / "testnet"}; + po::options_description desc_cmd_only("Command line options"); + po::options_description desc_cmd_sett("Command line options and settings options"); + const command_line::arg_descriptor<uint32_t> arg_log_level = {"log-level", "", log_level}; + const command_line::arg_descriptor<uint64_t> arg_block_stop = {"block-stop", "Stop at block number", block_stop}; + const command_line::arg_descriptor<uint64_t> arg_batch_size = {"batch-size", "", db_batch_size}; + const command_line::arg_descriptor<uint64_t> arg_pop_blocks = {"pop-blocks", "Remove blocks from end of blockchain", num_blocks}; + const command_line::arg_descriptor<bool> arg_testnet_on = { + "testnet" + , "Run on testnet." + , false + }; + const command_line::arg_descriptor<bool> arg_count_blocks = { + "count-blocks" + , "Count blocks in bootstrap file and exit" + , false + }; + const command_line::arg_descriptor<std::string> arg_database = { + "database", "available: memory, lmdb" + , default_db_engine + }; + const command_line::arg_descriptor<bool> arg_verify = {"verify", + "Verify blocks and transactions during import", true}; + const command_line::arg_descriptor<bool> arg_batch = {"batch", + "Batch transactions for faster import", true}; + const command_line::arg_descriptor<bool> arg_resume = {"resume", + "Resume from current height if output database already exists", true}; + command_line::add_arg(desc_cmd_sett, command_line::arg_data_dir, default_data_path.string()); + command_line::add_arg(desc_cmd_sett, command_line::arg_testnet_data_dir, default_testnet_data_path.string()); + command_line::add_arg(desc_cmd_sett, arg_testnet_on); + command_line::add_arg(desc_cmd_sett, arg_log_level); + command_line::add_arg(desc_cmd_sett, arg_database); + command_line::add_arg(desc_cmd_sett, arg_batch_size); + command_line::add_arg(desc_cmd_sett, arg_block_stop); + command_line::add_arg(desc_cmd_only, arg_count_blocks); + command_line::add_arg(desc_cmd_only, arg_pop_blocks); + command_line::add_arg(desc_cmd_only, command_line::arg_help); + // call add_options() directly for these arguments since + // command_line helpers support only boolean switch, not boolean argument + desc_cmd_sett.add_options() + (arg_verify.name, make_semantic(arg_verify), arg_verify.description) + (arg_batch.name, make_semantic(arg_batch), arg_batch.description) + (arg_resume.name, make_semantic(arg_resume), arg_resume.description) + ; + po::options_description desc_options("Allowed options"); + desc_options.add(desc_cmd_only).add(desc_cmd_sett); + po::variables_map vm; + bool r = command_line::handle_error_helper(desc_options, [&]() + { + po::store(po::parse_command_line(argc, argv, desc_options), vm); + po::notify(vm); + return true; + }); + if (! r) + return 1; + log_level = command_line::get_arg(vm, arg_log_level); + opt_verify = command_line::get_arg(vm, arg_verify); + opt_batch = command_line::get_arg(vm, arg_batch); + opt_resume = command_line::get_arg(vm, arg_resume); + block_stop = command_line::get_arg(vm, arg_block_stop); + db_batch_size = command_line::get_arg(vm, arg_batch_size); + if (command_line::get_arg(vm, command_line::arg_help)) + { + std::cout << CRYPTONOTE_NAME << " v" << MONERO_VERSION_FULL << ENDL << ENDL; + std::cout << desc_options << std::endl; + return 1; + } + if (! opt_batch && ! vm["batch-size"].defaulted()) + { + std::cerr << "Error: batch-size set, but batch option not enabled" << ENDL; + return 1; + } + if (! db_batch_size) + { + std::cerr << "Error: batch-size must be > 0" << ENDL; + return 1; + } + if (opt_verify && vm["batch-size"].defaulted()) + { + // usually want batch size default lower if verify on, so progress can be + // frequently saved. + // + // currently, with Windows, default batch size is low, so ignore + // default db_batch_size_verify unless it's even lower + if (db_batch_size > db_batch_size_verify) + { + db_batch_size = db_batch_size_verify; + } + } + std::vector<std::string> db_engines {"memory", "lmdb"}; + opt_testnet = command_line::get_arg(vm, arg_testnet_on); + auto data_dir_arg = opt_testnet ? command_line::arg_testnet_data_dir : command_line::arg_data_dir; + dirname = command_line::get_arg(vm, data_dir_arg); + db_arg_str = command_line::get_arg(vm, arg_database); + log_space::get_set_log_detalisation_level(true, log_level); + log_space::log_singletone::add_logger(LOGGER_CONSOLE, NULL, NULL); + LOG_PRINT_L0("Starting..."); + LOG_PRINT_L0("Setting log level = " << log_level); + boost::filesystem::path file_path {dirname}; + std::string import_file_path; + import_file_path = (file_path / "export" / import_filename).string(); + if (command_line::has_arg(vm, arg_count_blocks)) + { + BootstrapFile bootstrap; + bootstrap.count_blocks(import_file_path); + return 0; + } + std::string db_engine; + int mdb_flags = 0; + int res = 0; + res = parse_db_arguments(db_arg_str, db_engine, mdb_flags); + if (res) + { + std::cerr << "Error parsing database argument(s)" << ENDL; + return 1; + } + if (std::find(db_engines.begin(), db_engines.end(), db_engine) == db_engines.end()) + { + std::cerr << "Invalid database engine: " << db_engine << std::endl; + return 1; + } + LOG_PRINT_L0("database: " << db_engine); + LOG_PRINT_L0("verify: " << std::boolalpha << opt_verify << std::noboolalpha); + if (opt_batch) + { + LOG_PRINT_L0("batch: " << std::boolalpha << opt_batch << std::noboolalpha + << " batch size: " << db_batch_size); + } + else + { + LOG_PRINT_L0("batch: " << std::boolalpha << opt_batch << std::noboolalpha); + } + LOG_PRINT_L0("resume: " << std::boolalpha << opt_resume << std::noboolalpha); + LOG_PRINT_L0("testnet: " << std::boolalpha << opt_testnet << std::noboolalpha); + LOG_PRINT_L0("bootstrap file path: " << import_file_path); + LOG_PRINT_L0("database path: " << file_path.string()); + try + { + // fake_core needed for verification to work when enabled. + // + // NOTE: don't need fake_core method of doing things when we're going to call + // BlockchainDB add_block() directly and have available the 3 block + // properties to do so. Both ways work, but fake core isn't necessary in that + // circumstance. + // for multi_db_runtime: + if (db_engine == "lmdb") + { + fake_core_lmdb simple_core(dirname, opt_testnet, opt_batch, mdb_flags); + import_from_file(simple_core, import_file_path, block_stop); + } + else if (db_engine == "memory") + { + fake_core_memory simple_core(dirname, opt_testnet); + import_from_file(simple_core, import_file_path, block_stop); + } + else + { + std::cerr << "database engine unrecognized" << ENDL; + return 1; + } + // for multi_db_compile: + if (db_engine != default_db_engine) + { + std::cerr << "Invalid database engine for compiled version: " << db_engine << std::endl; + return 1; + } + fake_core_lmdb simple_core(dirname, opt_testnet, opt_batch, mdb_flags); + fake_core_memory simple_core(dirname, opt_testnet); + if (! vm["pop-blocks"].defaulted()) + { + num_blocks = command_line::get_arg(vm, arg_pop_blocks); + LOG_PRINT_L0("height: " << simple_core.m_storage.get_current_blockchain_height()); + pop_blocks(simple_core, num_blocks); + LOG_PRINT_L0("height: " << simple_core.m_storage.get_current_blockchain_height()); + return 0; + } + import_from_file(simple_core, import_file_path, block_stop); + } + catch (const DB_ERROR& e) + { + std::cout << std::string("Error loading blockchain db: ") + e.what() + " -- shutting down now" << ENDL; + return 1; + } + // destructors called at exit: + // + // ensure db closed + // - transactions properly checked and handled + // - disk sync if needed + // + // fake_core object's destructor is called when it goes out of scope. For an + // LMDB fake_core, it calls Blockchain::deinit() on its object, which in turn + // calls delete on its BlockchainDB derived class' object, which closes its + // files. + return 0; +} diff --git a/src/blockchain_utilities/bootstrap_file.cpp b/src/blockchain_utilities/bootstrap_file.cpp new file mode 100644 index 0000000..573cb15 +++ b/src/blockchain_utilities/bootstrap_file.cpp @@ -0,0 +1,505 @@ +namespace po = boost::program_options; +using namespace cryptonote; +using namespace epee; +namespace +{ + // This number was picked by taking the leading 4 bytes from this output: + // echo Monero bootstrap file | sha1sum + const uint32_t blockchain_raw_magic = 0x28721586; + const uint32_t header_size = 1024; + std::string refresh_string = "\r \r"; +} +bool BootstrapFile::open_writer(const boost::filesystem::path& dir_path) +{ + if (boost::filesystem::exists(dir_path)) + { + if (!boost::filesystem::is_directory(dir_path)) + { + LOG_PRINT_RED_L0("export directory path is a file: " << dir_path); + return false; + } + } + else + { + if (!boost::filesystem::create_directory(dir_path)) + { + LOG_PRINT_RED_L0("Failed to create directory " << dir_path); + return false; + } + } + std::string file_path = (dir_path / BLOCKCHAIN_RAW).string(); + m_raw_data_file = new std::ofstream(); + bool do_initialize_file = false; + uint64_t num_blocks = 0; + if (! boost::filesystem::exists(file_path)) + { + LOG_PRINT_L0("creating file"); + do_initialize_file = true; + num_blocks = 0; + } + else + { + num_blocks = count_blocks(file_path); + LOG_PRINT_L0("appending to existing file with height: " << num_blocks-1 << " total blocks: " << num_blocks); + } + m_height = num_blocks; + if (do_initialize_file) + m_raw_data_file->open(file_path, std::ios_base::binary | std::ios_base::out | std::ios::trunc); + else + m_raw_data_file->open(file_path, std::ios_base::binary | std::ios_base::out | std::ios::app | std::ios::ate); + if (m_raw_data_file->fail()) + return false; + m_output_stream = new boost::iostreams::stream<boost::iostreams::back_insert_device<buffer_type>>(m_buffer); + if (m_output_stream == nullptr) + return false; + if (do_initialize_file) + initialize_file(); + return true; +}

Continuation of: Git diff from commit e940386f9a8765423ab3dd9e3aabe19a68cba9f9 (thankful_for_today's last commit) to current HEAD, excluding changes in /external so as to exclude libraries and submoduled code, and excluding contrib/ so as to exclude changes made to epee, excluding most comments because apparently nobody that writes code for BCN knows what a comment is anyway, excluding removed lines, excluding empty lines

+bool BootstrapFile::initialize_file() +{ + const uint32_t file_magic = blockchain_raw_magic; + std::string blob; + if (! ::serialization::dump_binary(file_magic, blob)) + { + throw std::runtime_error("Error in serialization of file magic"); + } + *m_raw_data_file << blob; + bootstrap::file_info bfi; + bfi.major_version = 0; + bfi.minor_version = 1; + bfi.header_size = header_size; + bootstrap::blocks_info bbi; + bbi.block_first = 0; + bbi.block_last = 0; + bbi.block_last_pos = 0; + buffer_type buffer2; + boost::iostreams::stream<boost::iostreams::back_insert_device<buffer_type>>* output_stream_header; + output_stream_header = new boost::iostreams::stream<boost::iostreams::back_insert_device<buffer_type>>(buffer2); + uint32_t bd_size = 0; + blobdata bd = t_serializable_object_to_blob(bfi); + LOG_PRINT_L1("bootstrap::file_info size: " << bd.size()); + bd_size = bd.size(); + if (! ::serialization::dump_binary(bd_size, blob)) + { + throw std::runtime_error("Error in serialization of bootstrap::file_info size"); + } + *output_stream_header << blob; + *output_stream_header << bd; + bd = t_serializable_object_to_blob(bbi); + LOG_PRINT_L1("bootstrap::blocks_info size: " << bd.size()); + bd_size = bd.size(); + if (! ::serialization::dump_binary(bd_size, blob)) + { + throw std::runtime_error("Error in serialization of bootstrap::blocks_info size"); + } + *output_stream_header << blob; + *output_stream_header << bd; + output_stream_header->flush(); + *output_stream_header << std::string(header_size-buffer2.size(), 0); // fill in rest with null bytes + output_stream_header->flush(); + std::copy(buffer2.begin(), buffer2.end(), std::ostreambuf_iterator<char>(*m_raw_data_file)); + return true; +} +void BootstrapFile::flush_chunk() +{ + m_output_stream->flush(); + uint32_t chunk_size = m_buffer.size(); + // LOG_PRINT_L0("chunk_size " << chunk_size); + if (chunk_size > BUFFER_SIZE) + { + LOG_PRINT_L0("WARNING: chunk_size " << chunk_size << " > BUFFER_SIZE " << BUFFER_SIZE); + } + std::string blob; + if (! ::serialization::dump_binary(chunk_size, blob)) + { + throw std::runtime_error("Error in serialization of chunk size"); + } + *m_raw_data_file << blob; + if (m_max_chunk < chunk_size) + { + m_max_chunk = chunk_size; + } + long pos_before = m_raw_data_file->tellp(); + std::copy(m_buffer.begin(), m_buffer.end(), std::ostreambuf_iterator<char>(*m_raw_data_file)); + m_raw_data_file->flush(); + long pos_after = m_raw_data_file->tellp(); + long num_chars_written = pos_after - pos_before; + if (static_cast<unsigned long>(num_chars_written) != chunk_size) + { + LOG_PRINT_RED_L0("Error writing chunk: height: " << m_cur_height << " chunk_size: " << chunk_size << " num chars written: " << num_chars_written); + throw std::runtime_error("Error writing chunk"); + } + m_buffer.clear(); + delete m_output_stream; + m_output_stream = new boost::iostreams::stream<boost::iostreams::back_insert_device<buffer_type>>(m_buffer); + LOG_PRINT_L1("flushed chunk: chunk_size: " << chunk_size); +} +void BootstrapFile::write_block(block& block) +{ + bootstrap::block_package bp; + bp.block = block; + std::vector<transaction> txs; + uint64_t block_height = boost::get<txin_gen>(block.miner_tx.vin.front()).height; + // now add all regular transactions + for (const auto& tx_id : block.tx_hashes) + { + if (tx_id == null_hash) + { + throw std::runtime_error("Aborting: tx == null_hash"); + } + const transaction* tx = m_blockchain_storage->get_tx(tx_id); + transaction tx = m_blockchain_storage->get_db().get_tx(tx_id); + if(tx == NULL) + { + if (! m_tx_pool) + throw std::runtime_error("Aborting: tx == NULL, so memory pool required to get tx, but memory pool isn't enabled"); + else + { + transaction tx; + if(m_tx_pool->get_transaction(tx_id, tx)) + txs.push_back(tx); + else + throw std::runtime_error("Aborting: tx not found in pool"); + } + } + else + txs.push_back(*tx); + txs.push_back(tx); + } + // these non-coinbase txs will be serialized using this structure + bp.txs = txs; + // These three attributes are currently necessary for a fast import that adds blocks without verification. + bool include_extra_block_data = true; + if (include_extra_block_data) + { + size_t block_size = m_blockchain_storage->get_block_size(block_height); + difficulty_type cumulative_difficulty = m_blockchain_storage->get_block_cumulative_difficulty(block_height); + uint64_t coins_generated = m_blockchain_storage->get_block_coins_generated(block_height); + size_t block_size = m_blockchain_storage->get_db().get_block_size(block_height); + difficulty_type cumulative_difficulty = m_blockchain_storage->get_db().get_block_cumulative_difficulty(block_height); + uint64_t coins_generated = m_blockchain_storage->get_db().get_block_already_generated_coins(block_height); + bp.block_size = block_size; + bp.cumulative_difficulty = cumulative_difficulty; + bp.coins_generated = coins_generated; + } + blobdata bd = t_serializable_object_to_blob(bp); + m_output_stream->write((const char*)bd.data(), bd.size()); +} +bool BootstrapFile::close() +{ + if (m_raw_data_file->fail()) + return false; + m_raw_data_file->flush(); + delete m_output_stream; + delete m_raw_data_file; + return true; +} +bool BootstrapFile::store_blockchain_raw(blockchain_storage* _blockchain_storage, tx_memory_pool* _tx_pool, boost::filesystem::path& output_dir, uint64_t requested_block_stop) +bool BootstrapFile::store_blockchain_raw(Blockchain* _blockchain_storage, tx_memory_pool* _tx_pool, boost::filesystem::path& output_dir, uint64_t requested_block_stop) +{ + uint64_t num_blocks_written = 0; + m_max_chunk = 0; + m_blockchain_storage = _blockchain_storage; + m_tx_pool = _tx_pool; + uint64_t progress_interval = 100; + LOG_PRINT_L0("Storing blocks raw data..."); + if (!BootstrapFile::open_writer(output_dir)) + { + LOG_PRINT_RED_L0("failed to open raw file for write"); + return false; + } + block b; + // block_start, block_stop use 0-based height. m_height uses 1-based height. So to resume export + // from last exported block, block_start doesn't need to add 1 here, as it's already at the next + // height. + uint64_t block_start = m_height; + uint64_t block_stop = 0; + LOG_PRINT_L0("source blockchain height: " << m_blockchain_storage->get_current_blockchain_height()-1); + if ((requested_block_stop > 0) && (requested_block_stop < m_blockchain_storage->get_current_blockchain_height())) + { + LOG_PRINT_L0("Using requested block height: " << requested_block_stop); + block_stop = requested_block_stop; + } + else + { + block_stop = m_blockchain_storage->get_current_blockchain_height() - 1; + LOG_PRINT_L0("Using block height of source blockchain: " << block_stop); + } + for (m_cur_height = block_start; m_cur_height <= block_stop; ++m_cur_height) + { + // this method's height refers to 0-based height (genesis block = height 0) + crypto::hash hash = m_blockchain_storage->get_block_id_by_height(m_cur_height); + m_blockchain_storage->get_block_by_hash(hash, b); + write_block(b); + if (m_cur_height % NUM_BLOCKS_PER_CHUNK == 0) { + flush_chunk(); + num_blocks_written += NUM_BLOCKS_PER_CHUNK; + } + if (m_cur_height % progress_interval == 0) { + std::cout << refresh_string; + std::cout << "block " << m_cur_height << "/" << block_stop << std::flush; + } + } + // NOTE: use of NUM_BLOCKS_PER_CHUNK is a placeholder in case multi-block chunks are later supported. + if (m_cur_height % NUM_BLOCKS_PER_CHUNK != 0) + { + flush_chunk(); + } + // print message for last block, which may not have been printed yet due to progress_interval + std::cout << refresh_string; + std::cout << "block " << m_cur_height-1 << "/" << block_stop << ENDL; + LOG_PRINT_L0("Number of blocks exported: " << num_blocks_written); + if (num_blocks_written > 0) + LOG_PRINT_L0("Largest chunk: " << m_max_chunk << " bytes"); + return BootstrapFile::close(); +} +uint64_t BootstrapFile::seek_to_first_chunk(std::ifstream& import_file) +{ + uint32_t file_magic; + std::string str1; + char buf1[2048]; + import_file.read(buf1, sizeof(file_magic)); + if (! import_file) + throw std::runtime_error("Error reading expected number of bytes"); + str1.assign(buf1, sizeof(file_magic)); + if (! ::serialization::parse_binary(str1, file_magic)) + throw std::runtime_error("Error in deserialization of file_magic"); + if (file_magic != blockchain_raw_magic) + { + LOG_PRINT_RED_L0("bootstrap file not recognized"); + throw std::runtime_error("Aborting"); + } + else + LOG_PRINT_L0("bootstrap file recognized"); + uint32_t buflen_file_info; + import_file.read(buf1, sizeof(buflen_file_info)); + str1.assign(buf1, sizeof(buflen_file_info)); + if (! import_file) + throw std::runtime_error("Error reading expected number of bytes"); + if (! ::serialization::parse_binary(str1, buflen_file_info)) + throw std::runtime_error("Error in deserialization of buflen_file_info"); + LOG_PRINT_L1("bootstrap::file_info size: " << buflen_file_info); + if (buflen_file_info > sizeof(buf1)) + throw std::runtime_error("Error: bootstrap::file_info size exceeds buffer size"); + import_file.read(buf1, buflen_file_info); + if (! import_file) + throw std::runtime_error("Error reading expected number of bytes"); + str1.assign(buf1, buflen_file_info); + bootstrap::file_info bfi; + if (! ::serialization::parse_binary(str1, bfi)) + throw std::runtime_error("Error in deserialization of bootstrap::file_info"); + LOG_PRINT_L0("bootstrap file v" << unsigned(bfi.major_version) << "." << unsigned(bfi.minor_version)); + LOG_PRINT_L0("bootstrap magic size: " << sizeof(file_magic)); + LOG_PRINT_L0("bootstrap header size: " << bfi.header_size) + uint64_t full_header_size = sizeof(file_magic) + bfi.header_size; + import_file.seekg(full_header_size); + return full_header_size; +} +uint64_t BootstrapFile::count_blocks(const std::string& import_file_path) +{ + boost::filesystem::path raw_file_path(import_file_path); + boost::system::error_code ec; + if (!boost::filesystem::exists(raw_file_path, ec)) + { + LOG_PRINT_L0("bootstrap file not found: " << raw_file_path); + throw std::runtime_error("Aborting"); + } + std::ifstream import_file; + import_file.open(import_file_path, std::ios_base::binary | std::ifstream::in); + uint64_t h = 0; + if (import_file.fail()) + { + LOG_PRINT_L0("import_file.open() fail"); + throw std::runtime_error("Aborting"); + } + uint64_t full_header_size; // 4 byte magic + length of header structures + full_header_size = seek_to_first_chunk(import_file); + LOG_PRINT_L0("Scanning blockchain from bootstrap file..."); + block b; + bool quit = false; + uint64_t bytes_read = 0; + int progress_interval = 10; + std::string str1; + char buf1[2048]; + while (! quit) + { + uint32_t chunk_size; + import_file.read(buf1, sizeof(chunk_size)); + if (!import_file) { + std::cout << refresh_string; + LOG_PRINT_L1("End of file reached"); + quit = true; + break; + } + h += NUM_BLOCKS_PER_CHUNK; + if ((h-1) % progress_interval == 0) + { + std::cout << "\r" << "block height: " << h-1 << + " " << + std::flush; + } + bytes_read += sizeof(chunk_size); + str1.assign(buf1, sizeof(chunk_size)); + if (! ::serialization::parse_binary(str1, chunk_size)) + throw std::runtime_error("Error in deserialization of chunk_size"); + LOG_PRINT_L3("chunk_size: " << chunk_size); + if (chunk_size > BUFFER_SIZE) + { + std::cout << refresh_string; + LOG_PRINT_L0("WARNING: chunk_size " << chunk_size << " > BUFFER_SIZE " << BUFFER_SIZE + << " height: " << h-1); + throw std::runtime_error("Aborting: chunk size exceeds buffer size"); + } + if (chunk_size > 100000) + { + std::cout << refresh_string; + LOG_PRINT_L0("NOTE: chunk_size " << chunk_size << " > 100000" << " height: " + << h-1); + } + else if (chunk_size <= 0) { + std::cout << refresh_string; + LOG_PRINT_L0("ERROR: chunk_size " << chunk_size << " <= 0" << " height: " << h-1); + throw std::runtime_error("Aborting"); + } + // skip to next expected block size value + import_file.seekg(chunk_size, std::ios_base::cur); + if (! import_file) { + std::cout << refresh_string; + LOG_PRINT_L0("ERROR: unexpected end of file: bytes read before error: " + << import_file.gcount() << " of chunk_size " << chunk_size); + throw std::runtime_error("Aborting"); + } + bytes_read += chunk_size; + // std::cout << refresh_string; + LOG_PRINT_L3("Number bytes scanned: " << bytes_read); + } + import_file.close(); + std::cout << ENDL; + std::cout << "Done scanning bootstrap file" << ENDL; + std::cout << "Full header length: " << full_header_size << " bytes" << ENDL; + std::cout << "Scanned for blocks: " << bytes_read << " bytes" << ENDL; + std::cout << "Total: " << full_header_size + bytes_read << " bytes" << ENDL; + std::cout << "Number of blocks: " << h << ENDL; + std::cout << ENDL; + // NOTE: h is the number of blocks. + // Note that a block's stored height is zero-based, but parts of the code use + // one-based height. + return h; +} diff --git a/src/blockchain_utilities/bootstrap_file.h b/src/blockchain_utilities/bootstrap_file.h new file mode 100644 index 0000000..5fb8a1d +++ b/src/blockchain_utilities/bootstrap_file.h @@ -0,0 +1,116 @@ +using namespace cryptonote; +class BootstrapFile +{ +public: + uint64_t count_blocks(const std::string& dir_path); + uint64_t seek_to_first_chunk(std::ifstream& import_file); + bool store_blockchain_raw(cryptonote::blockchain_storage* cs, cryptonote::tx_memory_pool* txp, + boost::filesystem::path& output_dir, uint64_t use_block_height=0); + bool store_blockchain_raw(cryptonote::Blockchain* cs, cryptonote::tx_memory_pool* txp, + boost::filesystem::path& output_dir, uint64_t use_block_height=0); +protected: + blockchain_storage* m_blockchain_storage; + Blockchain* m_blockchain_storage; + tx_memory_pool* m_tx_pool; + typedef std::vector<char> buffer_type; + std::ofstream * m_raw_data_file; + buffer_type m_buffer; + boost::iostreams::stream<boost::iostreams::back_insert_device<buffer_type>>* m_output_stream; + // open export file for write + bool open_writer(const boost::filesystem::path& dir_path); + bool initialize_file(); + bool close(); + void write_block(block& block); + void flush_chunk(); +private: + uint64_t m_height; + uint64_t m_cur_height; // tracks current height during export + uint32_t m_max_chunk; +}; diff --git a/src/blockchain_utilities/bootstrap_serialization.h b/src/blockchain_utilities/bootstrap_serialization.h new file mode 100644 index 0000000..6fa9493 +++ b/src/blockchain_utilities/bootstrap_serialization.h @@ -0,0 +1,88 @@ +namespace cryptonote +{ + namespace bootstrap + { + struct file_info + { + uint8_t major_version; + uint8_t minor_version; + uint32_t header_size; + BEGIN_SERIALIZE_OBJECT() + FIELD(major_version); + FIELD(minor_version); + VARINT_FIELD(header_size); + END_SERIALIZE() + }; + struct blocks_info + { + // block heights of file's first and last blocks, zero-based indexes + uint64_t block_first; + uint64_t block_last; + // file position, for directly reading last block + uint64_t block_last_pos; + BEGIN_SERIALIZE_OBJECT() + VARINT_FIELD(block_first); + VARINT_FIELD(block_last); + VARINT_FIELD(block_last_pos); + END_SERIALIZE() + }; + struct block_package + { + cryptonote::block block; + std::vector<transaction> txs; + size_t block_size; + difficulty_type cumulative_difficulty; + uint64_t coins_generated; + BEGIN_SERIALIZE() + FIELD(block) + FIELD(txs) + VARINT_FIELD(block_size) + VARINT_FIELD(cumulative_difficulty) + VARINT_FIELD(coins_generated) + END_SERIALIZE() + }; + } +} diff --git a/src/blockchain_utilities/fake_core.h b/src/blockchain_utilities/fake_core.h new file mode 100644 index 0000000..5eda504 +++ b/src/blockchain_utilities/fake_core.h @@ -0,0 +1,165 @@ +using namespace cryptonote; +struct fake_core_lmdb +{ + Blockchain m_storage; + tx_memory_pool m_pool; + bool support_batch; + bool support_add_block; + // for multi_db_runtime: + fake_core_lmdb(const boost::filesystem::path &path, const bool use_testnet=false, const bool do_batch=true, const int mdb_flags=0) : m_pool(&m_storage), m_storage(m_pool) + // for multi_db_compile: + fake_core_lmdb(const boost::filesystem::path &path, const bool use_testnet=false, const bool do_batch=true, const int mdb_flags=0) : m_pool(m_storage), m_storage(m_pool) + { + m_pool.init(path.string()); + BlockchainDB* db = new BlockchainLMDB(); + boost::filesystem::path folder(path); + folder /= db->get_db_name(); + LOG_PRINT_L0("Loading blockchain from folder " << folder.string() << " ..."); + const std::string filename = folder.string(); + try + { + db->open(filename, mdb_flags); + } + catch (const std::exception& e) + { + LOG_PRINT_L0("Error opening database: " << e.what()); + throw; + } + m_storage.init(db, use_testnet); + if (do_batch) + m_storage.get_db().set_batch_transactions(do_batch); + support_batch = true; + support_add_block = true; + } + ~fake_core_lmdb() + { + m_storage.deinit(); + } + uint64_t add_block(const block& blk + , const size_t& block_size + , const difficulty_type& cumulative_difficulty + , const uint64_t& coins_generated + , const std::vector<transaction>& txs + ) + { + return m_storage.get_db().add_block(blk, block_size, cumulative_difficulty, coins_generated, txs); + } + void batch_start(uint64_t batch_num_blocks = 0) + { + m_storage.get_db().batch_start(batch_num_blocks); + } + void batch_stop() + { + m_storage.get_db().batch_stop(); + } +}; +struct fake_core_memory +{ + blockchain_storage m_storage; + tx_memory_pool m_pool; + bool support_batch; + bool support_add_block; + // for multi_db_runtime: + fake_core_memory(const boost::filesystem::path &path, const bool use_testnet=false) : m_pool(&m_storage), m_storage(m_pool) + // for multi_db_compile: + fake_core_memory(const boost::filesystem::path &path, const bool use_testnet=false) : m_pool(m_storage), m_storage(&m_pool) + { + m_pool.init(path.string()); + m_storage.init(path.string(), use_testnet); + support_batch = false; + support_add_block = false; + } + ~fake_core_memory() + { + LOG_PRINT_L3("fake_core_memory() destructor called - want to see it ripple down"); + m_storage.deinit(); + } + uint64_t add_block(const block& blk + , const size_t& block_size + , const difficulty_type& cumulative_difficulty + , const uint64_t& coins_generated + , const std::vector<transaction>& txs + ) + { + // TODO: + // would need to refactor handle_block_to_main_chain() to have a direct add_block() method like Blockchain class + throw std::runtime_error("direct add_block() method not implemented for in-memory db"); + return 2; + } + void batch_start(uint64_t batch_num_blocks = 0) + { + LOG_PRINT_L0("WARNING: [batch_start] opt_batch set, but this database doesn't support/need transactions - ignoring"); + } + void batch_stop() + { + LOG_PRINT_L0("WARNING: [batch_stop] opt_batch set, but this database doesn't support/need transactions - ignoring"); + } +}; diff --git a/src/blocks/CMakeLists.txt b/src/blocks/CMakeLists.txt new file mode 100644 index 0000000..4020132 +++ b/src/blocks/CMakeLists.txt @@ -0,0 +1,38 @@ +if(APPLE) + add_library(blocks STATIC blockexports.c) + set_target_properties(blocks PROPERTIES LINKER_LANGUAGE C) +else() + add_custom_command(OUTPUT blocks.o COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && ld -r -b binary -o ${CMAKE_CURRENT_BINARY_DIR}/blocks.o blocks.dat) + add_library(blocks STATIC blocks.o blockexports.c) + set_target_properties(blocks PROPERTIES LINKER_LANGUAGE C) +endif() diff --git a/src/blocks/blockexports.c b/src/blocks/blockexports.c new file mode 100644 index 0000000..cea72b2 +++ b/src/blocks/blockexports.c @@ -0,0 +1,48 @@ +extern const struct mach_header _mh_execute_header; +extern const struct mach_header_64 _mh_execute_header; +const unsigned char *get_blocks_dat_start() +{ + size_t size; + return getsectiondata(&_mh_execute_header, "__DATA", "__blocks_dat", &size); +} +size_t get_blocks_dat_size() +{ + size_t size; + getsectiondata(&_mh_execute_header, "__DATA", "__blocks_dat", &size); + return size; +} +extern const unsigned char _binary_blocks_start[]; +extern const unsigned char _binary_blocks_end[]; +const unsigned char *get_blocks_dat_start(void) +{ + return _binary_blocks_start; +} +size_t get_blocks_dat_size(void) +{ + return (size_t) (_binary_blocks_end - _binary_blocks_start); +} diff --git a/src/blocks/blocks.dat b/src/blocks/blocks.dat new file mode 100644 index 0000000..e69de29 diff --git a/src/blocks/blocks.h b/src/blocks/blocks.h new file mode 100644 index 0000000..76a08c8 +++ b/src/blocks/blocks.h @@ -0,0 +1,16 @@ +extern "C" { +const unsigned char *get_blocks_dat_start(); +size_t get_blocks_dat_size(); +} diff --git a/src/blocks/checkpoints.dat b/src/blocks/checkpoints.dat new file mode 100644 index 0000000..249956e Binary files /dev/null and b/src/blocks/checkpoints.dat differ diff --git a/src/common/CMakeLists.txt b/src/common/CMakeLists.txt new file mode 100644 index 0000000..b1db006 +++ b/src/common/CMakeLists.txt @@ -0,0 +1,69 @@ +set(common_sources + base58.cpp + command_line.cpp + dns_utils.cpp + util.cpp + i18n.cpp) +set(common_headers) +set(common_private_headers + base58.h + boost_serialization_helper.h + command_line.h + dns_utils.h + http_connection.h + int-util.h + pod-class.h + rpc_client.h + scoped_message_writer.h + unordered_containers_boost_serialization.h + util.h + varint.h + i18n.h) +bitmonero_private_headers(common + ${common_private_headers}) +bitmonero_add_library(common + ${common_sources} + ${common_headers} + ${common_private_headers}) +target_link_libraries(common + LINK_PRIVATE + crypto + ${UNBOUND_LIBRARY} + ${Boost_DATE_TIME_LIBRARY} + ${Boost_FILESYSTEM_LIBRARY} + ${Boost_SYSTEM_LIBRARY} + ${EXTRA_LIBRARIES}) diff --git a/src/common/base58.cpp b/src/common/base58.cpp index 454c0db..6adc46c 100644 +++ b/src/common/base58.cpp @@ -1,7 +1,32 @@ #include "base58.h" @@ -110,7 +135,7 @@ namespace tools void encode_block(const char* block, size_t size, char* res) { + assert(1 <= size && size <= full_block_size); uint64_t num = uint_8be_to_64(reinterpret_cast<const uint8_t*>(block), size); int i = static_cast<int>(encoded_block_sizes[size]) - 1; diff --git a/src/common/base58.h b/src/common/base58.h index 4055f62..1a9ad1a 100644 +++ b/src/common/base58.h @@ -1,6 +1,32 @@ #pragma once diff --git a/src/common/boost_serialization_helper.h b/src/common/boost_serialization_helper.h index 74016ae..c108c52 100644 +++ b/src/common/boost_serialization_helper.h @@ -1,6 +1,32 @@ #pragma once @@ -14,15 +40,55 @@ namespace tools bool serialize_obj_to_file(t_object& obj, const std::string& file_path) { TRY_ENTRY(); + // Need to know HANDLE of file to call FlushFileBuffers + HANDLE data_file_handle = ::CreateFile(file_path.c_str(), GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL); + if (INVALID_HANDLE_VALUE == data_file_handle) + return false; + int data_file_descriptor = _open_osfhandle((intptr_t)data_file_handle, 0); + if (-1 == data_file_descriptor) + { + ::CloseHandle(data_file_handle); + return false; + } + FILE* data_file_file = _fdopen(data_file_descriptor, "wb"); + if (0 == data_file_file) + { + // Call CloseHandle is not necessary + _close(data_file_descriptor); + return false; + } + // HACK: undocumented constructor, this code may not compile + std::ofstream data_file(data_file_file); + if (data_file.fail()) + { + // Call CloseHandle and _close are not necessary + fclose(data_file_file); + return false; + } std::ofstream data_file; + data_file.open(file_path , std::ios_base::binary | std::ios_base::out| std::ios::trunc); + if (data_file.fail()) return false; boost::archive::binary_oarchive a(data_file); a << obj; + if (data_file.fail()) + return false; + data_file.flush(); + // To make sure the file is fully stored on disk + ::FlushFileBuffers(data_file_handle); + fclose(data_file_file); + return true; CATCH_ENTRY_L0("serialize_obj_to_file", false); } diff --git a/src/common/command_line.cpp b/src/common/command_line.cpp index 0b90345..d2cd75e 100644 +++ b/src/common/command_line.cpp @@ -1,12 +1,54 @@ #include "command_line.h" namespace command_line { + std::string input_line(const std::string& prompt) + { + std::cout << prompt; + std::string buf; + std::getline(std::cin, buf); + return epee::string_tools::trim(buf); + } const arg_descriptor<bool> arg_help = {"help", "Produce help message"}; const arg_descriptor<bool> arg_version = {"version", "Output version information"}; const arg_descriptor<std::string> arg_data_dir = {"data-dir", "Specify data directory"}; + const arg_descriptor<std::string> arg_testnet_data_dir = {"testnet-data-dir", "Specify testnet data directory"}; + const arg_descriptor<bool> arg_test_drop_download = {"test-drop-download", "For net tests: in download, discard ALL blocks instead checking/saving them (very fast)"}; + const arg_descriptor<uint64_t> arg_test_drop_download_height = {"test-drop-download-height", "Like test-drop-download but disards only after around certain height", 0}; + const arg_descriptor<int> arg_test_dbg_lock_sleep = {"test-dbg-lock-sleep", "Sleep time in ms, defaults to 0 (off), used to debug before/after locking mutex. Values 100 to 1000 are good for tests."}; } diff --git a/src/common/command_line.h b/src/common/command_line.h index 8606537..ae79f0a 100644 +++ b/src/common/command_line.h @@ -1,6 +1,32 @@ #pragma once @@ -14,6 +40,9 @@ namespace command_line { + std::string input_line(const std::string& prompt); template<typename T, bool required = false> struct arg_descriptor; @@ -174,4 +203,8 @@ namespace command_line extern const arg_descriptor<bool> arg_help; extern const arg_descriptor<bool> arg_version; extern const arg_descriptor<std::string> arg_data_dir; + extern const arg_descriptor<std::string> arg_testnet_data_dir; + extern const arg_descriptor<bool> arg_test_drop_download; + extern const arg_descriptor<uint64_t> arg_test_drop_download_height; + extern const arg_descriptor<int> arg_test_dbg_lock_sleep; } diff --git a/src/common/dns_utils.cpp b/src/common/dns_utils.cpp new file mode 100644 index 0000000..e442d3d +++ b/src/common/dns_utils.cpp @@ -0,0 +1,347 @@ +using namespace epee; +namespace bf = boost::filesystem; +namespace +{ +/* + * The following two functions were taken from unbound-anchor.c, from + * the unbound library packaged with this source. The license and source + * can be found in $PROJECT_ROOT/external/unbound + */ +/* Cert builtin commented out until it's used, as the compiler complains +static const char* +get_builtin_cert(void) +{ + return +"-----BEGIN CERTIFICATE-----\n" +"MIIDdzCCAl+gAwIBAgIBATANBgkqhkiG9w0BAQsFADBdMQ4wDAYDVQQKEwVJQ0FO\n" +"TjEmMCQGA1UECxMdSUNBTk4gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxFjAUBgNV\n" +"BAMTDUlDQU5OIFJvb3QgQ0ExCzAJBgNVBAYTAlVTMB4XDTA5MTIyMzA0MTkxMloX\n" +"DTI5MTIxODA0MTkxMlowXTEOMAwGA1UEChMFSUNBTk4xJjAkBgNVBAsTHUlDQU5O\n" +"IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MRYwFAYDVQQDEw1JQ0FOTiBSb290IENB\n" +"MQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKDb\n" +"cLhPNNqc1NB+u+oVvOnJESofYS9qub0/PXagmgr37pNublVThIzyLPGCJ8gPms9S\n" +"G1TaKNIsMI7d+5IgMy3WyPEOECGIcfqEIktdR1YWfJufXcMReZwU4v/AdKzdOdfg\n" +"ONiwc6r70duEr1IiqPbVm5T05l1e6D+HkAvHGnf1LtOPGs4CHQdpIUcy2kauAEy2\n" +"paKcOcHASvbTHK7TbbvHGPB+7faAztABLoneErruEcumetcNfPMIjXKdv1V1E3C7\n" +"MSJKy+jAqqQJqjZoQGB0necZgUMiUv7JK1IPQRM2CXJllcyJrm9WFxY0c1KjBO29\n" +"iIKK69fcglKcBuFShUECAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8B\n" +"Af8EBAMCAf4wHQYDVR0OBBYEFLpS6UmDJIZSL8eZzfyNa2kITcBQMA0GCSqGSIb3\n" +"DQEBCwUAA4IBAQAP8emCogqHny2UYFqywEuhLys7R9UKmYY4suzGO4nkbgfPFMfH\n" +"6M+Zj6owwxlwueZt1j/IaCayoKU3QsrYYoDRolpILh+FPwx7wseUEV8ZKpWsoDoD\n" +"2JFbLg2cfB8u/OlE4RYmcxxFSmXBg0yQ8/IoQt/bxOcEEhhiQ168H2yE5rxJMt9h\n" +"15nu5JBSewrCkYqYYmaxyOC3WrVGfHZxVI7MpIFcGdvSb2a1uyuua8l0BKgk3ujF\n" +"0/wsHNeP22qNyVO+XVBzrM8fk8BSUFuiT/6tZTYXRtEt5aKQZgXbKU5dUF3jT9qg\n" +"j/Br5BZw3X/zd325TvnswzMC1+ljLzHnQGGk\n" +"-----END CERTIFICATE-----\n" + ; +} +*/ +/** return the built in root DS trust anchor */ +static const char* +get_builtin_ds(void) +{ + return +". IN DS 19036 8 2 49AAC11D7B6F6446702E54A1607371607A1A41855200FD2CE1CDDE32F24E8FB5\n"; +} +/************************************************************ + ************************************************************ + ***********************************************************/ +} // anonymous namespace +namespace tools +{ +std::string ipv4_to_string(const char* src) +{ + std::stringstream ss; + unsigned int bytes[4]; + for (int i = 0; i < 4; i++) + { + unsigned char a = src; + bytes = a; + } + ss << bytes[0] << "." + << bytes[1] << "." + << bytes[2] << "." + << bytes[3]; + return ss.str(); +} +std::string ipv6_to_string(const char* src) +{ + std::stringstream ss; + unsigned int bytes[8]; + for (int i = 0; i < 8; i++) + { + unsigned char a = src; + bytes = a; + } + ss << bytes[0] << ":" + << bytes[1] << ":" + << bytes[2] << ":" + << bytes[3] << ":" + << bytes[4] << ":" + << bytes[5] << ":" + << bytes[6] << ":" + << bytes[7]; + return ss.str(); +} +template<typename type, void (*freefunc)(type*)> +class scoped_ptr +{ +public: + scoped_ptr(): + ptr(nullptr) + { + } + scoped_ptr(type *p): + ptr(p) + { + } + ~scoped_ptr() + { + freefunc(ptr); + } + operator type *() { return ptr; } + type **operator &() { return &ptr; } + type *operator->() { return ptr; } + operator const type*() const { return &ptr; } +private: + type* ptr; +}; +typedef class scoped_ptr<ub_result,ub_resolve_free> ub_result_ptr; +static void freestring(char *ptr) { free(ptr); } +typedef class scoped_ptr<char,freestring> string_ptr; +struct DNSResolverData +{ + ub_ctx* m_ub_context; +}; +DNSResolver::DNSResolver() : m_data(new DNSResolverData()) +{ + // init libunbound context + m_data->m_ub_context = ub_ctx_create(); + // look for "/etc/resolv.conf" and "/etc/hosts" or platform equivalent + ub_ctx_resolvconf(m_data->m_ub_context, NULL); + ub_ctx_hosts(m_data->m_ub_context, NULL); + #ifdef DEVELOPER_LIBUNBOUND_OLD + #pragma message "Using the work around for old libunbound" + { // work around for bug https://www.nlnetlabs.nl/bugs-script/show_bug.cgi?id=515 needed for it to compile on e.g. Debian 7 + char * ds_copy = NULL; // this will be the writable copy of string that bugged version of libunbound requires + try { + char * ds_copy = strdup( ::get_builtin_ds() ); + ub_ctx_add_ta(m_data->m_ub_context, ds_copy); + } catch(...) { // probably not needed but to work correctly in every case... + if (ds_copy) { free(ds_copy); ds_copy=NULL; } // for the strdup + throw ; + } + if (ds_copy) { free(ds_copy); ds_copy=NULL; } // for the strdup + } + #else + // normal version for fixed libunbound + ub_ctx_add_ta(m_data->m_ub_context, ::get_builtin_ds() ); + #endif +} +DNSResolver::~DNSResolver() +{ + if (m_data) + { + if (m_data->m_ub_context != NULL) + { + ub_ctx_delete(m_data->m_ub_context); + } + delete m_data; + } +} +std::vector<std::string> DNSResolver::get_ipv4(const std::string& url, bool& dnssec_available, bool& dnssec_valid) +{ + std::vector<std::string> addresses; + dnssec_available = false; + dnssec_valid = false; + string_ptr urlC(strdup(url.c_str())); + if (!check_address_syntax(urlC)) + { + return addresses; + } + // destructor takes care of cleanup + ub_result_ptr result; + // call DNS resolver, blocking. if return value not zero, something went wrong + if (!ub_resolve(m_data->m_ub_context, urlC, DNS_TYPE_A, DNS_CLASS_IN, &result)) + { + dnssec_available = (result->secure || (!result->secure && result->bogus)); + dnssec_valid = result->secure && !result->bogus; + if (result->havedata) + { + for (size_t i=0; result->data != NULL; i++) + { + addresses.push_back(ipv4_to_string(result->data)); + } + } + } + return addresses; +} +std::vector<std::string> DNSResolver::get_ipv6(const std::string& url, bool& dnssec_available, bool& dnssec_valid) +{ + std::vector<std::string> addresses; + dnssec_available = false; + dnssec_valid = false; + string_ptr urlC(strdup(url.c_str())); + if (!check_address_syntax(urlC)) + { + return addresses; + } + ub_result_ptr result; + // call DNS resolver, blocking. if return value not zero, something went wrong + if (!ub_resolve(m_data->m_ub_context, urlC, DNS_TYPE_AAAA, DNS_CLASS_IN, &result)) + { + dnssec_available = (result->secure || (!result->secure && result->bogus)); + dnssec_valid = result->secure && !result->bogus; + if (result->havedata) + { + for (size_t i=0; result->data != NULL; i++) + { + addresses.push_back(ipv6_to_string(result->data)); + } + } + } + return addresses; +} +std::vector<std::string> DNSResolver::get_txt_record(const std::string& url, bool& dnssec_available, bool& dnssec_valid) +{ + std::vector<std::string> records; + dnssec_available = false; + dnssec_valid = false; + string_ptr urlC(strdup(url.c_str())); + if (!check_address_syntax(urlC)) + { + return records; + } + ub_result_ptr result; + // call DNS resolver, blocking. if return value not zero, something went wrong + if (!ub_resolve(m_data->m_ub_context, urlC, DNS_TYPE_TXT, DNS_CLASS_IN, &result)) + { + dnssec_available = (result->secure || (!result->secure && result->bogus)); + dnssec_valid = result->secure && !result->bogus; + if (result->havedata) + { + for (size_t i=0; result->data != NULL; i++) + { + // plz fix this, but this does NOT work and spills over into parts of memory it shouldn't: records.push_back(result.ptr->data); + char *restxt; + restxt = (char*) calloc(result->len+1, 1); + memcpy(restxt, result->data+1, result->len-1); + records.push_back(restxt); + } + } + } + return records; +} +std::string DNSResolver::get_dns_format_from_oa_address(const std::string& oa_addr) +{ + std::string addr(oa_addr); + auto first_at = addr.find("@"); + if (first_at == std::string::npos) + return addr; + // convert name@domain.tld to name.domain.tld + addr.replace(first_at, 1, "."); + return addr; +} +DNSResolver& DNSResolver::instance() +{ + static DNSResolver* staticInstance = NULL; + if (staticInstance == NULL) + { + staticInstance = new DNSResolver(); + } + return *staticInstance; +} +bool DNSResolver::check_address_syntax(const char *addr) +{ + // if string doesn't contain a dot, we won't consider it a url for now. + if (strchr(addr,'.') == NULL) + { + return false; + } + return true; +} +} // namespace tools diff --git a/src/common/dns_utils.h b/src/common/dns_utils.h new file mode 100644 index 0000000..1e726c8 +++ b/src/common/dns_utils.h @@ -0,0 +1,136 @@ +namespace tools +{ +const static int DNS_CLASS_IN = 1; +const static int DNS_TYPE_A = 1; +const static int DNS_TYPE_TXT = 16; +const static int DNS_TYPE_AAAA = 8; +struct DNSResolverData; +/** + * @brief Provides high-level access to DNS resolution + * + * This class is designed to provide a high-level abstraction to DNS resolution + * functionality, including access to TXT records and such. It will also + * handle DNSSEC validation of the results. + */ +class DNSResolver +{ +public: + /** + * @brief Constructs an instance of DNSResolver + * + * Constructs a class instance and does setup stuff for the backend resolver. + */ + DNSResolver(); + /** + * @brief takes care of freeing C pointers and such + */ + ~DNSResolver(); + /** + * @brief gets ipv4 addresses from DNS query of a URL + * + * returns a vector of all IPv4 "A" records for given URL. + * If no "A" records found, returns an empty vector. + * + * @param url A string containing a URL to query for + * + * @param dnssec_available + * + * @return vector of strings containing ipv4 addresses + */ + std::vector<std::string> get_ipv4(const std::string& url, bool& dnssec_available, bool& dnssec_valid); + /** + * @brief gets ipv6 addresses from DNS query + * + * returns a vector of all IPv6 "A" records for given URL. + * If no "A" records found, returns an empty vector. + * + * @param url A string containing a URL to query for + * + * @return vector of strings containing ipv6 addresses + */ + std::vector<std::string> get_ipv6(const std::string& url, bool& dnssec_available, bool& dnssec_valid); + /** + * @brief gets all TXT records from a DNS query for the supplied URL; + * if no TXT record present returns an empty vector. + * + * @param url A string containing a URL to query for + * + * @return A vector of strings containing a TXT record; or an empty vector + */ + // TODO: modify this to accomodate DNSSEC + std::vector<std::string> get_txt_record(const std::string& url, bool& dnssec_available, bool& dnssec_valid); + /** + * @brief Gets a DNS address from OpenAlias format + * + * If the address looks good, but contains one @ symbol, replace that with a . + * e.g. donate@getmonero.org becomes donate.getmonero.org + * + * @param oa_addr OpenAlias address + * + * @return dns_addr DNS address + */ + std::string get_dns_format_from_oa_address(const std::string& oa_addr); + /** + * @brief Gets the singleton instance of DNSResolver + * + * @return returns a pointer to the singleton + */ + static DNSResolver& instance(); +private: + /** + * @brief Checks a string to see if it looks like a URL + * + * @param addr the string to be checked + * + * @return true if it looks enough like a URL, false if not + */ + bool check_address_syntax(const char *addr); + DNSResolverData *m_data; +}; // class DNSResolver +} // namespace tools diff --git a/src/common/http_connection.h b/src/common/http_connection.h new file mode 100644 index 0000000..c192e79

Quote from: esoum.1003 on July 31, 2015, 01:57:20 PM

Did you stop copy paste your straw?

Thank you for providing us with a companion quote to go along with such gems as "Is it true?".

DID YOU STOP COPY PASTE YOUR STRAW? DID YOU?!

Quote from: esoum.1003 on July 31, 2015, 01:57:20 PM

Please comment the post.

I am littering it with comments as we speak.

Quote from: esoum.1003 on July 31, 2015, 01:57:20 PM

Please compare development of monero with the development of the fathers of the code that your coin uses, ups, sorry, this project was stolen from T_F_T.

steal, verb

1. to take (the property of another or others) without permission or right, especially secretly or by force:

2. to appropriate (ideas, credit, words, etc.) without right or acknowledgment.

This cannot apply to Monero. The license covering the code pre-fork was the MIT license:

https://github.com/bitmonero-project/bitmonero/blob/master/src/cryptonote_config.h#L2

Per the text of the MIT license (http://opensource.org/licenses/mit-license.php):

"Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software"

So we had permission and right. As for acknowledgement, we have included, and continue to include, a CryptoNote copyright notice on all files that originated from before the fork:

https://github.com/monero-project/bitmonero/blob/master/src/cryptonote_config.h#L29

Quote from: esoum.1003 on July 31, 2015, 01:57:20 PM

Yes, thats when you scammers started. You Trolleros Devs couldn't even create a genesis block, so you stole the project from T_F_T.

We forked the repo, and retained the social contract implicit in Monero's launch by not relaunching it, thus ensuring that even thankful_for_today was rewarded for his work (ie. as an early miner). If we had relaunched it then it would have been difficult to do so fairly, someone on either side would feel mistreated.

Quote from: esoum.1003 on July 31, 2015, 01:57:20 PM

Look what another dev think of your work, 6 months ago, and Monero did not change a thing since then.

Looking at it objectively is always better than quoting the opinion of "another dev", especially when that dev is likely involved in the Bytecoin scam.

https://i.imgur.com/UvmK9uu.png

https://i.imgur.com/6wCFUGx.png

You are not just insulting me, you are insulting many others, and your claim is provably untrue. Don't make a statement countering this, provide tangible evidence of your claim.

Quote from: dNote on February 01, 2015, 08:40:29 AM

As for me - monero is a biggest bubble, because there is only speculation about it

Of course there's speculation. But there is also actual use.

Quote from: dNote on February 01, 2015, 08:40:29 AM

blockchain is unusable with its size

How do you suppose you'll have a usable cryptocurrency that is actually used, and yet not have an ever-growing blockchain? What a silly comment, it really shows the desperation of this trolling.

Quote from: dNote on February 01, 2015, 08:40:29 AM

IMO almost everything about monero seems terrible - unusable blockchain size (On average PC, it just can`t be a network node)

Runs just fine on a Raspberry Pi 1 with 256mb RAM, so I guess he must be talking about running it on a 486 DX2 66.

Quote from: dNote on February 01, 2015, 08:40:29 AM

block intervals

Chosen by...thankful_for_today! And here I thought you were hailing him as an incredible human being?

Quote from: dNote on February 01, 2015, 08:40:29 AM

tx generation size

Is ~~duck~~ ~~dark~~ ~~digital~~ dontcareNote really trying to claim that their transactions are significantly smaller?

Quote from: dNote on February 01, 2015, 08:40:29 AM

fee size

0.001 XMR per kilobyte (most transactions are 1kb-2kb), so that's about $0.0000056 per transaction. Gosh. So expensive.

Quote from: dNote on February 01, 2015, 08:40:29 AM

no GUI after almost 1 year since start

Wrong. There are several GUIs: https://getmonero.org/getting-started/choose

Quote from: dNote on February 01, 2015, 08:40:29 AM

totally wrong way of development

As opposed to dNote, which has the "totally right" way of development, no? If that's the case, then why have they attracted NO other contributors?

https://github.com/xdn-project/digitalnote/graphs/contributors vs. https://github.com/monero-project/bitmonero/graphs/contributors

Quote from: dNote on February 01, 2015, 08:40:29 AM

looks like just no understanding of CN technology.

Looks like I've got to rehash some of our research bulletins. Let's do this one:

MRL-0003: Monero is Not That Mysterious

1 Introduction

Recently, there have been some vague fears about the CryptoNote source code and protocol ﬂoating around the internet based on the fact that it is a more complicated protocol than, for instance, Bitcoin. The purpose of this note is to try and clear up some misconceptions, and hopefully remove some of the mystery surrounding Monero Ring Signatures. I will start by comparing the mathematics involved in CryptoNote ring signatures (as described in [CN]) to the mathematics in [FS], on which CryptoNote is based. After this, I will compare the mathematics of the ring signature to what is actually in the CryptoNote codebase.

2 CryptoNote Origins

As noted in ([CN], 4.1) by Saberhagen, group ring signatures have a history starting as early as 1991 [CH], and various ring signature schemes have been studied by a number of researchers throughout the past two decades. As claimed in ([CN] 4.1), the main ring signature used in CryptoNote is based on [FS], with some changes to accomodate blockchain technology.

2.1 Traceable Ring Signatures

In [FS], Fujisaki and Suzuki introduce a scheme for a ring signature designed to “leak secrets anonymously, without the risk of identity escrow.” This lack of a need for identity escrow allows users of the ring signature to hide themselves in a group with an inherently higher level of distrust compared to schemes relying on a group manager.

In ring-signature schemes relying on a group manager, such as the original ring signatures described in [CH], a designated trusted person guards the secrets of the group participants. While anonymous, such schemes rely, of course, on the manager not being compromised. The result of having a group-manager, in terms of currencies, is essentially the same as having a trusted organization or node to mix your coins.

In contrast, the traceable ring signature scheme given in [FS] has no group manager. According to [FS], there are four formal security requirements to their traceable ring signature scheme:

•    Public Traceability -Anyone who creates two signatures for di.erent mes.sages with respect to the same tag can be traced. (In CryptoNote, if the user opts not to use a one-time key for each transaction, then they will be traceable, however if they desire anonymity, then they will use the one-time key. Thus as stated on page 5 of [CN], the traceability property is weakened in CryptoNote.)
•    Tag-Linkability -Every two signatures generated by the same signer with respect to the same tag are linked. (This aspect in CryptoNote refers to each transaction having a key image which prevents double spending.)
•    Anonymity -As long as a signer does not sign on two di.erent messages with respect to the same tag, the identity of the signer is indistinguishable from any of the possible ring members. In addition, any two signatures gen.erated with respect to two distinct tags are always unlinkable. (In terms of CryptoNote, if the signer attempts to use the same key-image more than once, then they can be identiﬁed out of the group. The unlinkability aspect is retained and is a key part of CryptoNote.)
•    Exculpability -An honest ring member cannot be accused of signing twice with respect to the same tag. In other words, it should be infeasible to counterfeit a tag corresponding to another person’s secret key. (In terms of CryptoNote, this says that key images cannot be faked.)

In addition, [FS] provide a ring signature protocol on page 10 of their paper, which is equivalent to the CryptoNote ring signature algorithm, as described on page 9-10 of [CN]. It is worthwhile to note that [FS] is a publicly peer-reviewed publication appearing in Lecture Notes in Computer Science, as opposed to typical crypto.currency protocol descriptions, where it is unclear whether or not they have been reviewed or not.

2.2 Traceability vs CryptoNote

In the original traceable ring signature algorithm described in [FS], it is possible to use the tag corresponding to a signature multiple times. However, multiple uses of the tag allow the user to be traced; in other words, the signer’s index can be determined out of the group of users signing. It is worthwhile to note that, due to the exculpability feature of the protocol ([FS] 5.6, [CN], A2), keys cannot be stolen this way, unless an attacker is able to solve the Elliptic Curve Discrete Logarithm Problem (ECDLP) upon which a large portion of modern cryptography is based ([Si] XI.4).

The process to trace a tag used more than once is described on ([FS], page 10). In the CryptoNote protocol, however, key images (tags) used more than once are rejected by the blockchain as double-spends, and hence traceability is not an aspect of CryptoNote.

2.3 Tag-Linkability vs CryptoNote

In essence, the tag-linkability aspect of the traceable ring signature protocol is what prevents CryptoNote transactions from being double-spends. The relevant protocols are referred to as “Trace” in ([FS], 5) and “LNK” in the CryptoNote paper. Essentially all that is required is to be able to keep track of the key images which have been used before, and to verify that a key image is not used again.

If one key-image is detected on the blockchain before another key-image, then the second key image is detected as a double-spend transaction. As key-images cannot be forged, being exculpable, the double-spender must in fact be the same person, and not another person trying to steal a wallet.

3 One-Time Ring Signatures (mathematics)

The security of the ring signature scheme as described in ([FS] 10, [CN] 10) and implemented in the CryptoNote source relies on the known security properties of Curve25519. Note that this is the same curve used in OpenSSH 6.5, Tor, Apple iOS, and many other[1] security systems.

3.1 Twisted Edwards Curves

The basic security in the CryptoNote Ring Signature algorithm is guaranteed by the ECDLP ([Si], XI.4) on the Twisted Edwards curve ed25519. The security properties of curve ed25519 are described in [Bern], by noted cryptographer Daniel Bernstein, andin([BCPM])byateamfromMicrosoftResearch.Bernsteinnotesabouted25519 the “every known attack is more expensive than performing a brute-force search on a typical 128-bit secret-key cipher.”

The curve ed25519 is a singular curve of genus 1 with a group law, and described

2 .121665 22
by .x+ y2 =1+xy. This curve is considered over the ﬁnite ﬁeld Fq,
121666q =2255 . 19. For those readers unfamiliar with algebraic geometry, an algebraic curve is considered as a one dimensional sort of space, consisting of all points (x, y) satisfying the above equation. All points are also considered modulo q. By virtue of its genus, ed25519 has a “group structure” which, for the purpose of this discussion, means if P =(x1,y1) is a point on the curve, and Q =(x2,y2) is another point on the curve, then these points can be added (or subtracted) and the sum (or di.erence), P + Q (or P . Q) will also be on the curve. The addition is not the naive adding of x1 + x2 and y1 + y2, but instead points are added using the rule

x1y2 + y1x2 y1y2 + x1x2
P + Q = ,
1+ dx1x2y1y2 1 . dx1x2y1y2

.121665

[1]http://ianix.com/pub/curve25519-deployment.html

where d =([BBJLP] 6, [BCPM]). The mathematics of curves of genus one are explained in great detail in [Si] for the interested reader.

Based on the above, we can compute P +P for any such point. In order to shorten notation, we rely on our algebraic intuition and denote 2P = P + P . If n . Z, then nP denotes the repeated sum

P + P + ··· + P
P+_ P
n times

using the above nonlinear addition law. As an example of how this di.ers from ordinary addition, consider the following system of equations:

aP + bQ = X
aP ' + bQ' = Y

where a, b, c, d are integers and P, Q, X are points. If this were a standard system of linear equations then one could use linear algebraic techniques to easily solve for a and b, assuming that P, Q, X, Y, P ', and Q ' are known. However, even if a, b are very small the above system is extremely di.cult to solve using the ed25519 addition law. For example, if a =1 and b =1, we have

xP yQ + yP xQ yP yQ + xP xQ
, =(xX ,yX )
1+ dxP xQyP yQ 1 . dxP xQyP yQ
xP 1 yQ1 + yP 1 xQ1 yP 1 yQ1 + xP 1 xQ1
, =(xY ,yY )
1+ dxP 1 xQ1 yP1 yQ1 1 . dxP 1 xQ1 yP 1 yQ1

So in reality, this is a system of 4 nonlinear equations. To convince yourself that it is in fact di.cult to ﬁgure out a and b, try writing the above systems assuming a =2, b =1. It should become clear that the problem is extremely di.cult when a, b are chosen to be very large. As of yet, there are no known methods available to e.ciently solve this system for large values of a and b.

Consider the following problem. Suppose your friend has a random integer q, and computes qP using the above form of addition. Your friend then tells you the x and y coordinates qP =(x, y), but not what q itself is. Without asking, how do you ﬁnd out what q is? A naive approach might be to start with P and keep adding P + P + P... until you reach qP (which you will know because you will end up at (x, y)). But if q is very large then this naive approach might take billions of years using modern supercomputers. Based on what mathematicians currently know about the problem and the number of possible q, none of the currently known attacking techniques can, as a general rule, do better in any practical sense than brute force.

In CryptoNote, your secret key is essentially just a very, very large number x (for other considerations, see section 4.3.3, we choose x to be a multiple of 8). There is a special point G on the curve ed25519 called “the base point” of the curve which is used as the starting point to get xG. Your public key is just xG, and you are protected by the above problem from someone using known information to determine the private key.

3.2 Relation to Diffie Helman

Included in a ring signature are the following equations involving your secret key x:

P = xG
I = xHp (P )

rs = qs . csx.

Here s is a number giving the index in the group signature to your public key, and Hp (P ) is a hash function which deterministically takes the point P to another point

''
P = x ' G, where x is another very large uniformly chosen number. The value qs is chosen uniformly at random, and cs is computed using another equation involving random values. The particular hash function used in CryptoNote is Keccak1600, used in other applications such as SHA-3; it is currently considered to be secure ([FIPS]). The CryptoNote use of a single hash function is consistent with the standard procedure of consolidating distinct random oracles (in proofs of security in [FS], for example) into a single strong hash function.

The above equations can be written as follows:

P = xG
' ' G '
P = xx
rs = qs . csx

Solving the top two equations is equivalent to the ECDH (as outlined in a previ.ous note ([SN])) and is the same practical di.culty as the ECDLP. Although the equations appear linear, they are in fact highly non-linear, as they use the addition described in 3.1 and above. The third equation (with unknowns qs and x), has the di.culty of ﬁnding a random number (either qs or x) in Fq, a very large ﬁnite ﬁeld; this is not feasible. Note that as the third equation has two unknowns, combining it with the previous two equations does not help; an attacker needs to determine at least one of the random numbers qs or x.

3.3 Time Cost to Guess q or x

Since q and x are assumed to be random very large numbers in Fq , with q = 2255 . 19 (generated as 32-byte integers), this is equivalent to a 128-bit security level ([BCPM]), which is known to take billions of years to compute with current supercomputers.

3.4 Review of Proofs in Appendix

In the CryptoNote appendix, there are four proofs of the four basic properties required for security of the one-time ring-signature scheme:

• Linkability (protection against double-spending)
• Exculpability (protection of your secret key)
• Unforgeability (protection against forged ring signatures)
• Anonymity (ability to hide a transaction within other transactions) These theorems are essentially identical to those in [FS] and show that the ring signature protocol satisﬁes the above traits. The ﬁrst theorem shows that only the secret keys corresponding to the public keys included in a group can produce a signature for that group. This relies on the ECDLP for the solution of two simulta.neous (non-linear) elliptic curve equations, which, as explained in 3.2, is practically unsolvable. The second theorem uses the same reasoning, but shows that in order to create a fake signature that passes veriﬁcation, one would need to be able to solve the ECDLP. The third and fourth theorems are taken directly from [FS].

4 One-Time Ring Signatures (Application)

To understand how CryptoNote is implementing the One-Time Ring signatures, I built a model in Python of Crypto-ops.cpp and Crypto.cpp from the Monero source code using naive Twisted Edwards Curve operations (taken from code by Bernstein), rather than the apparently reasonably optimized operations existing in the CryptoNote code. Functions are explained in the code comments below. Using themodelwillproduceaworkingringsignaturethatdi.ersslightlyfromtheMonero ring signatures only because of hashing and packing di.erences between the used libraries. The full code is hosted at the following address: https://github.com/monero-project/mininero

Note that most of the important helper functions in crypto-ops.cpp in the CryptoNote source are pulled from the reference implementation of Curve25519. This reference implentation was coded by Matthew Dempsky (Mochi Media, now Google)[2].

In addition, after comparing the python code to the paper, and in turn comparing the python code to the actual Monero source, it is fairly easy to see that functions like generate_ring_sig are all doing what they are supposed to based on the pro.tocol describedinthewhitepaper. For example,here is the ring signature generation algorithm used in the CryptoNote source:

Algorithm 1 Ring Signatures

i . 0
while i < numkeys do

if i = s then
k . random Fq element
Li . k · G
Ri . k ·Hp(Pi)

else
k1 . random Fq element
k2 . random Fq element
Li . k1Pi + k2G
Ri . k1I + k2Hp(Pi)
ci . k1
ri . k2

end if
i . i +1
end while
h .Hs(prefix + L.is + R.is))
p
cs . h . i ci
=s
rs . k . xcs
return (I, {ci}, {ri})

Comparing this with [CN] shows that it agrees with the whitepaper. Similarly, here is the algorithm used in the CryptoNote source to verify ring signatures:

Algorithm 2 VER
i =0
while i < numkeys do L.. ciPi + riG
i R.. riHp(Pi)+ ciI
i
i . i +1
end while
h .Hs(prefix + L.is + R.is))
p
h . h .
i
=s ci return (h == 0(mod q)) == 0

4.1 Important Crypto-ops Functions

Descriptions of important functions in Crypto-ops.cpp. Even more references and information is given in the comments in the MiniNero.py code linked above.

[2]http://nacl.cr.yp.to/

4.1.1 ge_frombytes_vartime

Takes as input some data and converts to a point on ed25519. For a reference of (q.5)/837 the equation used, . = uvuv, see ([BBJLP], section 5).

4.1.2 ge_fromfe_frombytesvartime

Similar to the above, but compressed in another form.

4.1.3 ge_double_scalarmult_base_vartime

Takes as inputs two integers a and b and a point A on ed25519 and returns the point aA + bG, where G is the ed25519 base point. Used for the ring signatures when computing, for example, Li with i = s as in ([CN], 4.4)

4.1.4 ge_double_scalarmult_vartime

Takes as inputs two integers a and b and two points A and B on ed25519 and outputs aA + bB. Used, for example, when computing the Ri in the ring signatures with i = s ([CN], 4.4)

4.1.5 ge_scalarmult

Given a point A on ed25519 and an integer a, this computes the point aA. Used for example when computing Li and Ri when i = s.

4.1.6 ge_scalarmult_base

Takes as input an integer a and computes aG, where G is the ed25519 base point.

4.1.7 ge_p1p1_to_p2

There are different representations of curve points for ed25519, this converts between them. See MiniNero for more reference.

4.1.8 ge_p2_dbl

This takes a point in the “p2” representation and doubles it.

4.1.9 ge_p3_to_p2

Takes a point in the “p3” representation on ed25519 and turns it into a point in the “p2” representation.

4.1.10 ge_mul8

This takes a point A on ed25519 and returns 8A.

4.1.11 sc_reduce

Takes a 64-byte integer and outputs the lowest 32 bytes modulo the prime q. This is not a CryptoNote-speciﬁc function, but comes from the standard ed25519 library.

4.1.12 sc_reduce32

Takes a 32-byte integer and outputs the integer modulo q. Same code as above, except skipping the 64.32 byte step.

4.1.13 sc_mulsub

Takes three integers a, b, c in Fq and returns c . ab modulo q.

4.2 Important Hashing Functions

4.2.1 cn_fast_hash

Takes data and returns the Keccak1600 hash of the data.

4.3 Crypto.cpp Functions

4.3.1 random_scalar

Generates a 64-byte integer and then reduces it to a 32 byte integer modulo q for 128-bit security as described in section 3.3.

4.3.2 hash_to_scalar

Inputs data (for example, a point P on ed25519) and outputs Hs (P ), which is the Keccak1600 hash of the data. The function then converts the hashed data to a 32-byte integer modulo q.

4.3.3 generate_keys

Returns a secret key and public key pair, using random_scalar (asdescribed above) to get the secret key. Note that, as described in [Bern], the key set for ed25519 actually is only multiples of 8 in Fq, and hence ge_scalarmult_base includes a ge_mul8 to ensure the secret key is in the key set. This prevents transaction malleability at.tacks as described in ([Bern], c.f. section on “small subgroup attacks”). This is part of the GEN algorithm as described in ([CN], 4.4).

4.3.4 check_key

Inputs a public key and outputs if the point is on the curve.

4.3.5 secret_key_to_public_key

Inputs a secret key, checks it for some uniformity conditions, and outputs the cor.responding public key, which is essentially just 8 times the base point times the point.

4.3.6 hash_to_ec

Inputs a key, hashes it, and then does the equivalent in bitwise operations of mul.tiplying the resulting integer by the base point and then by 8.

4.3.7 generate_key_image

Takes as input a secret key x and public key P , and returns I = xHp (P ), the key image. This is part of the GEN algorithm as described in ([CN], 4.4).

4.3.8 generate_ring_signature

Computes a ring signature, performing SIG as in ([CN], 4.4) given a key image I, a list of n public keys Pi, and a secret index. Essentially there is a loop on i, and if the secret-index is reached, then an if-statement controls the special computation of Li,Ri when i is equal to the secret index. The values ci and ri for the signature are computed throughout the loop and returned along with the image to create the total signature (I,c1, ..., cn,r1, ..., rn) .

4.3.9 check_ring_signature

Runs the VER algorithm in ([CN], 4.4). The veriﬁer uses a given ring signature n
to compute L ' i = riGi, Ri ' = riHp (Pi)+ ciI, and ﬁnally to check ifi=0 ci = Hs (m, L 0' , ..., L ' ,R 0' , ..., R ' ) mod l.nn

4.3.10 generate_key_derivation

Takes a secretkey b,andapublic key P , and outputs 8·bP . (The 8 being for thepur.pose of the secret key set, as described in 4.3.3). This is used in derive_public_key as part of creating one-time addresses.

4.3.11 derivation_to_scalar

Performs Hs (.) as part of generating keys in ([CN], 4.3, 4.4). It hashes an output index together with the point.

4.3.12 derive_public_key

Takes a derivation rA (computed via generate_key_derivation), a point B, and an output index, computes a scalar via derivation_to_scalar, and then computes Hs (rA)+ B.

4.3.13 generate_signature

This takes a preﬁx, a public key, and a secret key, and generates a standard (not ring) transaction signature (similar to a Bitcoin transaction signature).

4.3.14 check_signature

This checks if a standard (not ring) signature is a valid signature.

5 Conclusion

Despite the ring signature functions in the original CryptoNote source being poorly commented, the code can be traced back to established and used sources, and is rel.atively straightfoward. The Python implementation provided with this review gives further indication of the code’s correctness. Furthermore, the elliptic curve mathe.matics underlying the ring signature scheme has been extremely well-studied; the concept of ring signatures is not novel, even if their application to cryptocurrencies is.

References

BBJLP. Bernstein, Daniel J., et al. "Twisted edwards curves." Progress in Cryptology–AFRICACRYPT 2008. Springer Berlin Heidelberg, 2008. 389-405. BCPM. Bos, Joppe W., et al. "Selecting Elliptic Curves for Cryptography: An E.ciency and Security Analysis." IACR Cryptology ePrint Archive 2014 (2014): 130. Bern. Bernstein, Daniel J. "Curve25519: new Di.e-Hellman speed records." Public Key Cryptography-PKC 2006. Springer Berlin Heidelberg, 2006. 207-228. CH. Chaum, David, and Eugene Van Heyst. "Group signatures." Advances in Cryptology—EUROCRYPT’91. Springer Berlin Heidelberg, 1991. CN. van Saberhagen, Nicolas. "CryptoNote v 2.0." (2013). FIPS. SHA, NIST DRAFT. "standard: Permutation-based hash and extendable-output functions." DRAFT FIPS 202 (2014).
Fu.    Fujisaki, Eiichiro. "Sub-linear size traceable ring signatures without random oracles." IEICE TRANSACTIONS on Fundamentals of Electronics, Communications and Computer Sciences 95.1 (2012): 151-166.
FS.    Fujisaki, Eiichiro, and Koutarou Suzuki. "Traceable ring signature." Public Key Cryptography–PKC 2007. Springer Berlin Heidelberg, 2007. 181-200.
IAN. IANIX http://ianix.com/pub/curve25519-deployment.html Si. Silverman, Joseph H. The arithmetic of elliptic curves. Vol. 106. Dordrecht: Springer, 2009.
SN. http://lab.monero.cc/pubs/multiple_equations_attack.pdf

Bitcoin Forum

Alternate cryptocurrencies => Altcoin Discussion => Topic started by: dukey8 on July 30, 2015, 09:15:09 PM