Goomboo's Journal

[Queeq]

And, could you please share the way you extract candles from that csv bitcoincharts file, maybe you could post the part of the code responsible for doing that?

I don't really understand what you mean by "extracting candles". I don't use candles, only closing prices.
The algorithm for finding closing price is pretty simple. Firstly, script calculates number of seconds for given interval, 3600 for 1h, for example. Then, when iterating over lines, the following actions are made:
1. Remember current timestamp and price as "last line".
2. Calculate timestamp for start of the next interval.
3. Move on to the next line.
4. If timestamp on this line is greater or equal to calculated "next interval timestamp" - put data from "last line" to database, calculate next interval start and remember current "last line". If current timestamp is smaller - go to step 1 and then to 3.

The code is available at Github. Data manipulation and analysis is made mostly in analysis/analysis.py file.

[Queeq]

I have run my MT4 script on EMA 21,11 on the same timeframe, here is the report:

...

starting with $100 on november 1st it appears to end up with $440 at 21:00 December 20

Now it's very close to my result. Here's mine for 00:00 01.11.2013 - 00:00 21.12.2013:

[mmccll]

For those using php here's a little script I made which generates candle files from bitcoincharts.com (using for example wget http://api.bitcoincharts.com/v1/csv/btceUSD.csv). Alter the $candlesize parameter to set the desired candlesize.
Rename the file you get from bitcoincharts to btc.csv and run the script.

Code:

<?
$handle = fopen('btc.csv',"r");
$candlesize=600;
$i=0;$j=0;
while (($line = fgetcsv($handle,0,",")) != FALSE) {
        $timestamp=$line[0];
        if(!isset($nextstamp)){$nextstamp=$timestamp+$candlesize;}
        $value=round($line[1],4);
        if (($timestamp < $nextstamp)){
                $candle[$i]=$value;
                $i++;$j++;
        }
        else
        {
                if(isset($candle)){
                        $AVG=array_sum($candle)/count($candle);
                        $OPEN=$candle[0];
                        $HIGH=max($candle);
                        $LOW=min($candle);
                        $CLOSE=$candle[$i-1];
                        $TRADES=$i;
                }
                else {
                        $AVG=0;
                }
                print $nextstamp . "," . $OPEN . "," . $HIGH . "," . $LOW . "," . $CLOSE . "," . $TRADES ."\n";
                $i=0;
                $nextstamp=$nextstamp+$candlesize;
                if(isset($candle)){unset($candle);}
        }
}
        fclose($handle);
?>

Output is a csv which can be used for Gekko backtesting.

[helluvaname]

For those using php here's a little script I made which generates candle files from bitcoincharts.com (using for example wget http://api.bitcoincharts.com/v1/csv/btceUSD.csv). Alter the $candlesize parameter to set the desired candlesize.
Rename the file you get from bitcoincharts to btc.csv and run the script.
Output is a csv which can be used for Gekko backtesting.

wow!
Please, wouldn't you be so kind to modify the script so the output would be:
date open (yyyymmdd), time open (mm:hh), open price, high price, low price, close price, volume
I don't know php and hell I am so tired of trying to get that historical data inside MT4.

edit:
well... I am trying to understand your script and as far as I understand it is doing pretty much the same I need...
edit:
no, TRADES is nomber of trades not the voulume. And I wonder what is the format of $nextstamp?

[gandhibt]

Rampions main argument against EMA-systems is fees and slippage, especially if you have rather big holdings and I think these two things are too the toughest ones to beat. One solution could be a compromise. You could adjust your trading wallet size accordingly. If you have big holdings you should maybe hold more and if you have smaller holdings you could trade more. Even if the % win isn't as big with big holdings it's still a lot of added coins vs. buy and hold.

Of course this is a job if you start to trade and it takes time, effort and discipline, you maybe have no time for other jobs and that's of course loss to your holdings and you could end up still losing your trades in a long run. One important thing is to evaluate that is this how you want to use your time? My choice is to be trader nevertheless.

Nice to see that this thread has grown to talk about taking the 21/10 hourly system to another level. I haven't read this in awhile. Keep it up!

[gabbello]

Great tool, Queeq, but is there a way to setup a sell and buy threshold?

E.g. if  threshold_sell  = 0.5 and threshold_buy = 0.06 calculate the difference between EMAs

dif = 100 * (short - long) / ((short + long)/2)

a) buy only if dif < -threshold_sell (-0.5)
b) sell only if dif > threshold_buy  (0.06)


as this is (per my understanding) how the EMA tradebots work.

[Soak]

Hello,

I'm very interested in the heatmap script showed before in the thread. I want to use it for the LTC/BTC pair on BTC-e, but for a little pair like this, I don't know how often to re-evaluate EMA values. Each week? Each month?

Thank you

[Queeq]

Great tool, Queeq, but is there a way to setup a sell and buy threshold?

E.g. if  threshold_sell  = 0.5 and threshold_buy = 0.06 calculate the difference between EMAs

dif = 100 * (short - long) / ((short + long)/2)

a) buy only if dif < -threshold_sell (-0.5)
b) sell only if dif > threshold_buy  (0.06)


as this is (per my understanding) how the EMA tradebots work.

Trade bots may use any strategy. I don't think there are any best practices. I didn't use thresholds either.

Trying to figure out what would be the result of using your formula... Pretty unclear to me. So... It turns out to be some kind of a tool to make decision even more concrete, right? It would put buy or sell action further away from actual EMA cross thus eliminating false positives. But at the same time there would be negative side effect - higher prices to buy and lower prices to sell if trend is advancing. Interesting, needs additional testing.

I have just pushed latest changes to the git repository. Decision function is separated now and it must be easier to add new strategies.

Also I added parabolic SAR implementation and it helped me to exclude some of the false positives giving slightly more profit. It's coded in "Algorithm #2". Buy as usual, sell only when both fast < slow and SAR trending down.


ATTENTION EVERYONE
Bitcoincharts.com have incorrect order of some trades in CSV files!
Thus rendering all of my previous calculations inaccurate. On unix you can correct this using sort:

Code:

sort -t ',' -k 1 -o <output_file> <input_file>

I've included simple check script in my repo, it will show inconsistencies if there are any or otherwise just print number of lines as it iterates over a file.

[Queeq]

Hello,

I'm very interested in the heatmap script showed before in the thread. I want to use it for the LTC/BTC pair on BTC-e, but for a little pair like this, I don't know how often to re-evaluate EMA values. Each week? Each month?

Thank you

I'd choose some kind of universal pair that shows stable profits (or smallest losses) on different time periods in different market conditions, be it rally or aggressive downtrend. That's what Goomboo was talking in the very beginning as far as I understood. So take the script, test it on different time periods and see how it goes

Also if you are going to use get_data.py script to update your CSV files - beware of hardcoded btc_usd pair there!

Sorry for absence of documentation, I'm much more into the analysis now rather than putting everything in good order. Thus the script itself is also pretty messy. However the first priority task is to finish the bot script to unleash it to the wilds

[gabbello]

Trade bots may use any strategy. I don't think there are any best practices. I didn't use thresholds either.

Trying to figure out what would be the result of using your formula... Pretty unclear to me. So... It turns out to be some kind of a tool to make decision even more concrete, right? It would put buy or sell action further away from actual EMA cross thus eliminating false positives. But at the same time there would be negative side effect - higher prices to buy and lower prices to sell if trend is advancing. Interesting, needs additional testing.

Thank you for your feedback, what I wanted to raise is that the buy/sell threshold make very big difference in the final result. For example: http://postimg.org/image/xl9kf5k6h/ in this period of time (with those intervals) there is no ema pair that makes profit, however, for the same period/intervals setting some threshold values will generate profit for various ema pairs. 

[Queeq]

Thank you for your feedback, what I wanted to raise is that the buy/sell threshold make very big difference in the final result. For example: http://postimg.org/image/xl9kf5k6h/ in this period of time (with those intervals) there is no ema pair that makes profit, however, for the same period/intervals setting some threshold values will generate profit for various ema pairs. 

Thanks for clarification. Don't you think that setting thresholds is almost the same that working on larger intervals, with the only difference that buy and sell decisions would be triggered differently? This is pure speculation based on rough estimation as I can't imagine buy/sell graph in my mind for your formula

[gabbello]

I'm not sure that this is the case all the time.

If you want a visual representation of the thresholds they represent the "cloud"/distance between the two EMA lines. So if the two EMA lines cross each other but the difference is not very big the buy/sell trigger will not be fired. So you can have intervals where one one EMA crosses the other and then goes back quite fast (with 0 thresholds this will trigger one buy and one sell, while with some thresholds setup it will not trigger any action (as long as the thresholds are not reached).

Also note that the thresholds do not need to be symmetric (e.g. buy=sell thresholds). So you could sell instantly when the two EMA crosses (sell thresholds=0), but wait for the buy until the difference between the EMAs is large enough.

[Queeq]

All right! Now it's clear. I was also struggling on how to optimize strategy for these cases as they cause pretty much loss during low volatility. I read somewhere about "kind of angle of crossing", but haven't thought up an idea how to do that. Instead I used SAR and according to my test it helped a bit.

How did you choose values of these thresholds? I think it'd be also interesting to iterate over different combinations of those values too, but it would generate enormous amount of data which would take ages to graph.

[gabbello]

I'm not aware of any way to choose these thresholds, but you can start from 0.25 (both buy and sell) as default values. And then iterate up and down in 0.05 steps or smth. similar (until you reach min.0 max??? maybe 0.8-1.0). 

I agree it is a lot of data (if you use 0.0-1.0 interval and 0.05 step you will have 400 combinations for each EMA pair), but that's where computers are good for . If you want to show this visually on your heatmap I would try to calculate best pair of buy/sell thresholds for each EMA pair and display only that one(s) in the relevant block.

But, a first step would be to at least allow users to set these values when running the script (instead of using 0,0 all the time).

[Queeq]

I assume you meant:

a) sell only if dif < -threshold_sell (-0.5)
b) buy only if dif > threshold_buy  (0.06)

Right?

I'm going to hardcode 0.25 for both thresholds now, let's see what would the difference be. I think it would take around half an hour from now to finish the simulations.

[gabbello]

I assume you meant:

a) sell only if dif < -threshold_sell (-0.5)
b) buy only if dif > threshold_buy  (0.06)

Right?
.

yes, sorry.

[Queeq]

I have chosen one month for testing - september 2013 as it is pretty flat.

Traditional strategy:

5 min

15 min

30 min

1 hour

2 hours


Using thresholds of 0.5:
5 min

15 min

30 min

1 hour

2 hours

[Queeq]

Additional text statistics for these heatmaps.
http://www.sendspace.com/filegroup/7dowlksziR9P5QK2y9UX1w

Pushed changes to github so welcome to test different parameters.

[gabbello]

Thank you,

Ok, so having this specific value for thresholds improves the Max. values for all intervals and sometimes improves and sometimes worsen the averages.

Probably that on a period with higher volatility the results (compared to 0 thresholds) would be quite different. And same for different threshold values .

[Queeq]

Welcome.

For me this is a pretty strange picture as relatively high profits are mostly concentrated along pairs of "neighbor" MAs.

Automatic probing of different threshold values will require many changes to the script structure. Maybe I'll take a look at it after finishing the bot script.