Goomboo's Journal

[Queeq]

WOW that great, thanks you a lot!!

this program can run @ WIN or just @linux ?

& can i humbly ask you draw 5 MIN?

thanks again!

5 min tick for dec.05 - dec.12:


Welcome.

The script can be run on both Win and Linux (or any other OS which has ability to run Python scripts). However it needs some adaptation for that, but I moved my focus to slightly different area. Though anyone can do that as it's open-source and available on Github.

[mmccll]

Very interesting stuff Goomboo. Respect.
I've been running soms backtests using Gekko from Whydifficult (again many respect).

What I've done is creating a script which alters the ema pair in the config file and loops Gekko over this. I'm on my phone now so can't upload the results. Quite similar to queeq, but haven't read his post while making the script...

A very interesting outcome is that the ema pairs which have a short period bigger than the long gives very consistent high profit percentages. Way higher than the other way around. Also looking at the buy sell moments it tends to buy low and sell high pretty good.

data period is last 1.5 weeks and interval is 1h.

Any insights on this?

I will upload files when on laptop.

[mmccll]

See below. Notice especially the area with the short between 41-43 and the long around 25.

http://www.canalphotos.nl/Afbeelding1.jpg

And a sample run of Gekko on the EMA pair short 42, long 25:

Code:

2013-12-14 12:09:10 (INFO):     ADVICE is to BUY @ 728.000 (7.710)      at      2013-12-07 12:59:57
2013-12-14 12:09:10 (INFO):     ADVICE is to SELL @ 962.900 (-0.511)    at      2013-12-09 09:59:59
2013-12-14 12:09:10 (INFO):     ADVICE is to BUY @ 908.610 (0.330)      at      2013-12-12 04:59:59
2013-12-14 12:09:10 (INFO):     ADVICE is to SELL @ 972.010 (-0.255)    at      2013-12-13 11:59:54


2013-12-14 12:09:10 (INFO):             WARNING: BACKTESTING FEATURE NEEDS PROPER TESTING
2013-12-14 12:09:10 (INFO):             WARNING: ACT ON THESE NUMBERS AT YOUR OWN RISK!


2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) start time:                      2013-12-07 11:59:59
2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) end time:                        2013-12-13 22:55:35
2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) timespan:                        6 days

2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) start price:                     1065.1
2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) end price:                       891
2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) Buy and Hold profit:             -16.345883%

2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) amount of trades:                4
2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) original simulated balance:      1000.000000 USD
2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) current simulated balance:       1403.670563 USD
2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) simulated profit:                403.670563 USD (40.367056%)
2013-12-14 12:09:10 (INFO):     (PROFIT REPORT) simulated yearly profit:         23951.120080 USD (2395.112008%)

[Queeq]

Here's my result for almost the same period.



I believe we have different algorithms of backtesting.

In short, mine is the following:
  Build an array of closing prices for, say, 1 hour resolution (configurable).
  Build a number of simple and exponential moving averages from 1 to 50 (configurable) for these closing prices.
  Find all intersections of slow and fast MAs for each unique pair.
  Simulate sell or buy at that points (use closing price) depending on which MA goes below which: if fast goes below slow - sell, and vice versa. Substract a fee from each operation. 0.2% by default (configurable).
  See what amount of initial sum (in percent) is left.
  Show the result for each pair on heatmap.

[mmccll]

That's interesting. I'm using Whydifficults Gekko trading bot which can be found at:

https://bitcointalk.org/index.php?topic=209149.0

(which is by the way pretty amazing stuff!)

Basically it lets you set the EMA pair and some thresholds. Input is a csv file with timestamp, open, high, low and close values. But i assume it does roughly the same as your exponential algorithm.

While making a php script to loop over various settings of Gekko I noticed that there are some very profitable results when you take the short average with a higher setting compared to the long average. That would be in your graphs any values below the diagonal line (which is in your graph blanked out).
Might be interesting to modify your script to see what is shown in this now white area.

Below is the candle-file which I used to make this graph:

http://www.canalphotos.nl/gekko/upload/candles.csv

[Queeq]

That would be in your graphs any values below the diagonal line (which is in your graph blanked out).
Might be interesting to modify your script to see what is shown in this now white area.

That would mean to make my script act in another way round: buy when fast goes below slow and sell when it goes above. That is completely opposite of what Goomboo taught us But for the sake of experiment, here's the result

[Bulbasaur]

Queeq is right, whether an average is fast or slow depends entirely on the number of periods, so to say that the fast one has a higher number than the slow means that it's no longer really the fast one in practice.

I would be hesitant to trade on any such strategy. You would expect it would probably just do as Queeq suggests, buying and selling on the 'wrong' crossovers, but it might also do entirely unexpected things since there's some chance that the code expects the 'fast' input to always actually be faster than the 'slow' one and hasn't been debugged for the reversed case.

On the other hand, your plot does look basically like a mirror image, so where the normal crossover strategy is unprofitable, the corresponding reversed crossover shows a fairly proportional profit. This does seem to make sense, if we assume the losses are due mainly to 'wrong' guesses rather than commission costs. Switching all the trades would transform all the 'wrong' guesses to 'right' guesses.

I would guess that what's happening is, the indicators are lagging enough that the crossover doesn't hit until the trend is over, at which point you would be taking a position just when the trend is reversing. So if you trade against the crossover, you (maybe) catch the new trend rather than the old one. But now you're getting away from the original idea of using the averages as a signal of recent momentum/current trend and relying more on hoping that the trend reversals will happen to align with your choice of lag time.

[xybersurfer]

mmccll, i think you need to check to make sure you didn't accidentally reverse your short and long axii because they seem to be the complete opposite of what one would expect (not just in the normally missing area). mmccll i do enjoy seeing what the heatmap looks like on your more normal axii.

also the csv file from mmccll goes from:
2013-12-06 12:59:28 to 2013-12-13 21:55:35

while the data from Queeq goes from:
2013-12-05 00:00:00 to 2013-12-12 00:00:00

but i doubt this makes that much difference

[mmccll]

I'm checking my scripts very carefully now, just to make sure that there are no mistakes in it. But basically what it does is looping over the gekko config file altering those parameters:

Code:

  // EMA weight (α)
  // the higher the weight, the more smooth (and delayed) the line
  short: 10,
  long: 21,
  // amount of candles to remember and base initial EMAs on

[mmccll]

Checked for 10/21 and 21/10.
Heatmap. Notice the cells with bold lines around them. Showing calculated profit of 11.2% and 8.1%.
http://canalphotos.nl/gekko/upload/Afbeelding3.jpg

And the Gekko output to verify:

Code:

##### SHORT: 10   LONG: 21 #####
2013-12-15 13:52:18 (INFO):     ADVICE is to SELL @ 728.000 (-10.036)   at      2013-12-07 12:59:57
2013-12-15 13:52:18 (INFO):     ADVICE is to BUY @ 820.000 (0.767)      at      2013-12-09 00:59:58
2013-12-15 13:52:18 (INFO):     ADVICE is to SELL @ 938.000 (-0.661)    at      2013-12-11 14:59:59
2013-12-15 13:52:18 (INFO):     ADVICE is to BUY @ 950.000 (0.591)      at      2013-12-13 05:59:46
2013-12-15 13:52:18 (INFO):     ADVICE is to SELL @ 905.000 (-0.287)    at      2013-12-13 20:59:54

2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) start time:                      2013-12-07 11:59:59
2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) end time:                        2013-12-13 22:55:35
2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) timespan:                        6 days

2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) start price:                     1065.1
2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) end price:                       891
2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) Buy and Hold profit:             -16.345883%

2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) amount of trades:                5
2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) original simulated balance:      1000.000000 USD
2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) current simulated balance:       1081.025961 USD
2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) simulated profit:                81.025961 USD (8.102596%)
2013-12-15 13:52:18 (INFO):     (PROFIT REPORT) simulated yearly profit:         4807.540376 USD (480.754038%)

##### SHORT: 21   LONG: 10 #####
2013-12-15 13:54:15 (INFO):     ADVICE is to BUY @ 728.000 (10.036)     at      2013-12-07 12:59:57
2013-12-15 13:54:15 (INFO):     ADVICE is to SELL @ 820.000 (-0.767)    at      2013-12-09 00:59:58
2013-12-15 13:54:15 (INFO):     ADVICE is to BUY @ 938.000 (0.661)      at      2013-12-11 14:59:59
2013-12-15 13:54:15 (INFO):     ADVICE is to SELL @ 950.000 (-0.591)    at      2013-12-13 05:59:46
2013-12-15 13:54:15 (INFO):     ADVICE is to BUY @ 905.000 (0.287)      at      2013-12-13 20:59:54

2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) start time:                      2013-12-07 11:59:59
2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) end time:                        2013-12-13 22:55:35
2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) timespan:                        6 days

2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) start price:                     1065.1
2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) end price:                       891
2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) Buy and Hold profit:             -16.345883%

2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) amount of trades:                5
2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) original simulated balance:      1000.000000 USD
2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) current simulated balance:       1111.949512 USD
2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) simulated profit:                111.949512 USD (11.194951%)
2013-12-15 13:54:15 (INFO):     (PROFIT REPORT) simulated yearly profit:         6642.337697 USD (664.233770%)

And Queeg/Bulbasur you're right! It swaps the BUY/SELL moments as you can see above, effectively trading against the crossovers. Resulting in higher profits in the specific market dynamics of the specified time period. I'll be running some tests on other data sets (varying from/to and interval) to see how this holds against other periods. Might be interesting to see the results...

[lx]

Hi everyone and thanks for this amazing thread.
I just wanted your opinion on the fact that the data we use for backtesting is 'polluted' by events that will most likely not happen again (bitcoin skyrocketing to > $1000 etc).
In the end backtesting is about the relevancy of the data you use to test your strategy, and I just feel like the data is not relevant to determine what's coming.

I've been playing around with Gekko and it seems that EMAs settings have to be adjusted (inverted ?) at some point ; which is not very natural (be gentle, I'm just a geek playing wall street)

[Queeq]

I just wanted your opinion on the fact that the data we use for backtesting is 'polluted' by events that will most likely not happen again (bitcoin skyrocketing to > $1000 etc).
In the end backtesting is about the relevancy of the data you use to test your strategy, and I just feel like the data is not relevant to determine what's coming.

You're free to backtest on any time periods you want. You may exclude periods of fast growth to have a picture of what will be going on when market is steady. However you may potentially lose profits if price goes up/down again rapidly. So I do not only test on the large periods, but also on smaller ones, especially when price goes down to see how much would I lose in such market conditions and to be prepared for that.

[notme]

Daily Goomboos should have sold around $700.  Not bad for a few hours, not that Goomboos care for quick profits.

[solidshotnosh]

Daily Goomboos should have sold around $700.  Not bad for a few hours, not that Goomboos care for quick profits.

The last sell signal I had was @ 857, what interval are you using?

[Queeq]

According to my calculations, Goomboo's strategy is fairly unprofitable in downtrend. That's not surprising at all as signals are generated with lag, thus buying signals are generated with price higher than selling signals. I wonder of how could we minimize lost in such situations. There may be some additional indicators. Could someone point them out so I could add them to backtesting model?

Here are the results from 11th of December till now.

5 min


15 min


30 min


1 hour


2 hours

[helluvaname]

Here are the results from 11th of December till now.

Hi Queeq!
Could you please add some axis on your graphs? It is very hard to understand what MA numbers are on your pictures.

By the way, I have done a little research of my own, it is EMA on 1 hour scale from 08.09.2011 till 08.12.13, btc-e data. I wonder, Queeq, if you could run a similar conditions test so we could compare the results?
The numbers are total balance at the final date, starting with 100$

the picture (I have some "disabled for security errors" so cant post a thumbnail):

http://postimg.org/image/5m32aninp/

And, while getting btc-e data from bitcoincharts I have found several "infinite" prices, which vere replaced vith valid prices taken from previous candles.

[Queeq]

Hi Queeq!
Could you please add some axis on your graphs? It is very hard to understand what MA numbers are on your pictures.

By the way, I have done a little research of my own, it is EMA on 1 hour scale from 08.09.2011 till 08.12.13, btc-e data. I wonder, Queeq, if you could run a similar conditions test so we could compare the results?
The numbers are total balance at the final date, starting with 100$

the picture (I have some "disabled for security errors" so cant post a thumbnail):

http://postimg.org/image/5m32aninp/

And, while getting btc-e data from bitcoincharts I have found several "infinite" prices, which vere replaced vith valid prices taken from previous candles.

If you zoom in you can see MA lengths in braces. I purposely turned off the axis as they don't help much when you look somewhere in the middle of a map.

My numbers are relative percent of the starting sum, so they would be pretty much similar, except that if sum goes under 100 they would be negative.

Bitcoincharts have some gaps in their data, especially in the beginning. They can be easily seen on their graphs. My algorithm just fills these gaps with last known closing price.

Here's the result of my backtesting on the given interval (08.09.2011 - 08.12.2013) with BTC-e data as well. I haven't included longer MAs as they're all red and image becomes insanely big.



So the results are mostly different. What algorithm have you used?

[helluvaname]

If you zoom in you can see MA lengths in braces. I purposely turned off the axis as they don't help much when you look somewhere in the middle of a map.

Oh now I see.

Bitcoincharts have some gaps in their data, especially in the beginning. They can be easily seen on their graphs. My algorithm just fills these gaps with last known closing price.

So do mine.

So the results are mostly different. What algorithm have you used?

Well, I made it the hard way I have loaded all hourly "raw data" from bitcoincharts (there is a button under a candle graph) to excel, then wrote a VBA script imitating buy and sell.
I forgot to tell that I have included a 0,25% fee for each trade.
Now I am moving from excel to MT4 available at btc-e, but it doesn't contain historical data earlier than approx 1st of november 2013 that would be valid for backtesting it with MT4 expert advisor.

My algorythm acts when a new candle is opened, it checks if EMA1-EMA2 was > 0 on the previous candle and EMA1-EMA2 was < 0 on the cande before that or vice versa.

But my excel algorythm and MT4 expert advisor give similar results on timeframe from 1 november 2013 till now.

Right now I am working on feeding MT4 with historical data so I can backtest my expert advisor on all historical data and check if my excel algo is right or wrong.

I have loaded my 1 hour frame from excel to MT4, but I am getting this ugly graphs:

http://s11.postimg.org/ku7mhwuf3/123123_1.jpg

where blue and red lines intersection is a buy/sell point

My guess is that MT4 needs not only 1H scale but all the smaller ones including 1M. Or maybe it needs all ticks, but I can't find aa way to load ticks into it.

I wonder what algorythm do you use to extract candels from that csv data file?

[Queeq]

I think I found an error in my EMA calculation. Different examples use numpy convolution function to do that. However I found that EMA changes more slowly despite that it must be more sensitive. Reversing the weights did the trick. So here's the result using corrected algorithm for data from 1st of the November:





My algorithm tracks each pair of MAs and does buy or sell simulation when they cross. Selection of buying or selling depends on which one goes below (if fast is going below slow - sell, and vice versa). I also subtract 0.2% fee from each operation.
I use an array of closing prices for each time interval (1 hour or other) and another array of corresponding values of MAs for these prices. Prices used for buy/sell simulation is the closing price of such interval.

[helluvaname]

So here's the result using corrected algorithm for data from 1st of the November:

I have run my MT4 script on EMA 21,11 on the same timeframe, here is the report:

https://drive.google.com/file/d/0B3f4gdrrQPzZSXF1NFZhM21CVXM/edit?usp=sharing
https://docs.google.com/document/d/1_8Cm3wC-dQsrecPOqlLmQgLlR49hrhp7DUncCYcF_j4/edit?usp=sharing

It is an .htm file with a gif, can be opened with any browser

starting with $100 on november 1st it appears to end up with $440 at 21:00 December 20

I don't really know how to make those heatcharts with MT4, and I am sick and tired of my excel script, it is so ugly and I just want to forget about it


And, could you please share the way you extract candles from that csv bitcoincharts file, maybe you could post the part of the code responsible for doing that?