Bitcoin Forum
December 11, 2016, 12:12:51 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 ... 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 [55] 56 57 58 59 »
  Print  
Author Topic: Goomboo's Journal  (Read 250999 times)
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
December 21, 2013, 06:43:05 PM
 #1081

And, could you please share the way you extract candles from that csv bitcoincharts file, maybe you could post the part of the code responsible for doing that?

I don't really understand what you mean by "extracting candles". I don't use candles, only closing prices.
The algorithm for finding closing price is pretty simple. Firstly, script calculates number of seconds for given interval, 3600 for 1h, for example. Then, when iterating over lines, the following actions are made:
1. Remember current timestamp and price as "last line".
2. Calculate timestamp for start of the next interval.
3. Move on to the next line.
4. If timestamp on this line is greater or equal to calculated "next interval timestamp" - put data from "last line" to database, calculate next interval start and remember current "last line". If current timestamp is smaller - go to step 1 and then to 3.

The code is available at Github. Data manipulation and analysis is made mostly in analysis/analysis.py file.

Python backtesting and bot script at Github
Development halted due to other priorities.
1481458371
Hero Member
*
Offline Offline

Posts: 1481458371

View Profile Personal Message (Offline)

Ignore
1481458371
Reply with quote  #2

1481458371
Report to moderator
1481458371
Hero Member
*
Offline Offline

Posts: 1481458371

View Profile Personal Message (Offline)

Ignore
1481458371
Reply with quote  #2

1481458371
Report to moderator
Be very wary of relying on JavaScript for security on sites such as blockchain.info and brainwallet.org. The site can change the JavaScript at any time unless you take unusual precautions, and browsers are not generally known for their airtight security.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481458371
Hero Member
*
Offline Offline

Posts: 1481458371

View Profile Personal Message (Offline)

Ignore
1481458371
Reply with quote  #2

1481458371
Report to moderator
1481458371
Hero Member
*
Offline Offline

Posts: 1481458371

View Profile Personal Message (Offline)

Ignore
1481458371
Reply with quote  #2

1481458371
Report to moderator
1481458371
Hero Member
*
Offline Offline

Posts: 1481458371

View Profile Personal Message (Offline)

Ignore
1481458371
Reply with quote  #2

1481458371
Report to moderator
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
December 21, 2013, 07:03:18 PM
 #1082


I have run my MT4 script on EMA 21,11 on the same timeframe, here is the report:

...

starting with $100 on november 1st it appears to end up with $440 at 21:00 December 20

Now it's very close to my result. Here's mine for 00:00 01.11.2013 - 00:00 21.12.2013:


Python backtesting and bot script at Github
Development halted due to other priorities.
mmccll
Newbie
*
Offline Offline

Activity: 7


View Profile
December 22, 2013, 03:48:10 PM
 #1083

For those using php here's a little script I made which generates candle files from bitcoincharts.com (using for example wget http://api.bitcoincharts.com/v1/csv/btceUSD.csv). Alter the $candlesize parameter to set the desired candlesize.
Rename the file you get from bitcoincharts to btc.csv and run the script.

Code:
<?
$handle = fopen('btc.csv',"r");
$candlesize=600;
$i=0;$j=0;
while (($line = fgetcsv($handle,0,",")) != FALSE) {
        $timestamp=$line[0];
        if(!isset($nextstamp)){$nextstamp=$timestamp+$candlesize;}
        $value=round($line[1],4);
        if (($timestamp < $nextstamp)){
                $candle[$i]=$value;
                $i++;$j++;
        }
        else
        {
                if(isset($candle)){
                        $AVG=array_sum($candle)/count($candle);
                        $OPEN=$candle[0];
                        $HIGH=max($candle);
                        $LOW=min($candle);
                        $CLOSE=$candle[$i-1];
                        $TRADES=$i;
                }
                else {
                        $AVG=0;
                }
                print $nextstamp . "," . $OPEN . "," . $HIGH . "," . $LOW . "," . $CLOSE . "," . $TRADES ."\n";
                $i=0;
                $nextstamp=$nextstamp+$candlesize;
                if(isset($candle)){unset($candle);}
        }
}
        fclose($handle);
?>

Output is a csv which can be used for Gekko backtesting.
helluvaname
Newbie
*
Offline Offline

Activity: 28


View Profile
December 22, 2013, 07:08:18 PM
 #1084

For those using php here's a little script I made which generates candle files from bitcoincharts.com (using for example wget http://api.bitcoincharts.com/v1/csv/btceUSD.csv). Alter the $candlesize parameter to set the desired candlesize.
Rename the file you get from bitcoincharts to btc.csv and run the script.
Output is a csv which can be used for Gekko backtesting.

wow!
Please, wouldn't you be so kind to modify the script so the output would be:
date open (yyyymmdd), time open (mm:hh), open price, high price, low price, close price, volume
I don't know php and hell I am so tired of trying to get that historical data inside MT4.

edit:
well... I am trying to understand your script and as far as I understand it is doing pretty much the same I need...
edit:
no, TRADES is nomber of trades not the voulume. And I wonder what is the format of $nextstamp?
gandhibt
Full Member
***
Offline Offline

Activity: 238



View Profile
December 30, 2013, 02:51:13 PM
 #1085

Rampions main argument against EMA-systems is fees and slippage, especially if you have rather big holdings and I think these two things are too the toughest ones to beat. One solution could be a compromise. You could adjust your trading wallet size accordingly. If you have big holdings you should maybe hold more and if you have smaller holdings you could trade more. Even if the % win isn't as big with big holdings it's still a lot of added coins vs. buy and hold.

Of course this is a job if you start to trade and it takes time, effort and discipline, you maybe have no time for other jobs and that's of course loss to your holdings and you could end up still losing your trades in a long run. One important thing is to evaluate that is this how you want to use your time? My choice is to be trader nevertheless.

Nice to see that this thread has grown to talk about taking the 21/10 hourly system to another level. I haven't read this in awhile. Keep it up!

bud trust 1JiRNiERPTVZEhaVcKWbLLHHyCLRaa7nxB
gabbello
Member
**
Offline Offline

Activity: 81


View Profile
January 03, 2014, 06:30:39 PM
 #1086

Great tool, Queeq, but is there a way to setup a sell and buy threshold?

E.g. if  threshold_sell  = 0.5 and threshold_buy = 0.06 calculate the difference between EMAs

dif = 100 * (short - long) / ((short + long)/2)

a) buy only if dif < -threshold_sell (-0.5)
b) sell only if dif > threshold_buy  (0.06)


as this is (per my understanding) how the EMA tradebots work.

Buy and sell bitcoins using credit cards, paypal, skrill, ukash - http://www.virwox.info
Soak
Full Member
***
Offline Offline

Activity: 213



View Profile
January 03, 2014, 07:48:26 PM
 #1087

Hello,

I'm very interested in the heatmap script showed before in the thread. I want to use it for the LTC/BTC pair on BTC-e, but for a little pair like this, I don't know how often to re-evaluate EMA values. Each week? Each month?

Thank you Smiley
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
January 03, 2014, 08:02:09 PM
 #1088

Great tool, Queeq, but is there a way to setup a sell and buy threshold?

E.g. if  threshold_sell  = 0.5 and threshold_buy = 0.06 calculate the difference between EMAs

dif = 100 * (short - long) / ((short + long)/2)

a) buy only if dif < -threshold_sell (-0.5)
b) sell only if dif > threshold_buy  (0.06)


as this is (per my understanding) how the EMA tradebots work.


Trade bots may use any strategy. I don't think there are any best practices. I didn't use thresholds either.

Trying to figure out what would be the result of using your formula... Pretty unclear to me. So... It turns out to be some kind of a tool to make decision even more concrete, right? It would put buy or sell action further away from actual EMA cross thus eliminating false positives. But at the same time there would be negative side effect - higher prices to buy and lower prices to sell if trend is advancing. Interesting, needs additional testing.

I have just pushed latest changes to the git repository. Decision function is separated now and it must be easier to add new strategies.

Also I added parabolic SAR implementation and it helped me to exclude some of the false positives giving slightly more profit. It's coded in "Algorithm #2". Buy as usual, sell only when both fast < slow and SAR trending down.


ATTENTION EVERYONE
Bitcoincharts.com have incorrect order of some trades in CSV files!
Thus rendering all of my previous calculations inaccurate. On unix you can correct this using sort:
Code:
sort -t ',' -k 1 -o <output_file> <input_file>
I've included simple check script in my repo, it will show inconsistencies if there are any or otherwise just print number of lines as it iterates over a file.

Python backtesting and bot script at Github
Development halted due to other priorities.
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
January 03, 2014, 08:16:41 PM
 #1089

Hello,

I'm very interested in the heatmap script showed before in the thread. I want to use it for the LTC/BTC pair on BTC-e, but for a little pair like this, I don't know how often to re-evaluate EMA values. Each week? Each month?

Thank you Smiley

I'd choose some kind of universal pair that shows stable profits (or smallest losses) on different time periods in different market conditions, be it rally or aggressive downtrend. That's what Goomboo was talking in the very beginning as far as I understood. So take the script, test it on different time periods and see how it goes Smiley

Also if you are going to use get_data.py script to update your CSV files - beware of hardcoded btc_usd pair there!

Sorry for absence of documentation, I'm much more into the analysis now rather than putting everything in good order. Thus the script itself is also pretty messy. However the first priority task is to finish the bot script to unleash it to the wilds Smiley

Python backtesting and bot script at Github
Development halted due to other priorities.
gabbello
Member
**
Offline Offline

Activity: 81


View Profile
January 03, 2014, 08:31:51 PM
 #1090



Trade bots may use any strategy. I don't think there are any best practices. I didn't use thresholds either.

Trying to figure out what would be the result of using your formula... Pretty unclear to me. So... It turns out to be some kind of a tool to make decision even more concrete, right? It would put buy or sell action further away from actual EMA cross thus eliminating false positives. But at the same time there would be negative side effect - higher prices to buy and lower prices to sell if trend is advancing. Interesting, needs additional testing.



Thank you for your feedback, what I wanted to raise is that the buy/sell threshold make very big difference in the final result. For example: http://postimg.org/image/xl9kf5k6h/ in this period of time (with those intervals) there is no ema pair that makes profit, however, for the same period/intervals setting some threshold values will generate profit for various ema pairs. 

Buy and sell bitcoins using credit cards, paypal, skrill, ukash - http://www.virwox.info
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
January 03, 2014, 08:38:34 PM
 #1091


Thank you for your feedback, what I wanted to raise is that the buy/sell threshold make very big difference in the final result. For example: http://postimg.org/image/xl9kf5k6h/ in this period of time (with those intervals) there is no ema pair that makes profit, however, for the same period/intervals setting some threshold values will generate profit for various ema pairs. 

Thanks for clarification. Don't you think that setting thresholds is almost the same that working on larger intervals, with the only difference that buy and sell decisions would be triggered differently? This is pure speculation based on rough estimation as I can't imagine buy/sell graph in my mind for your formula Smiley

Python backtesting and bot script at Github
Development halted due to other priorities.
gabbello
Member
**
Offline Offline

Activity: 81


View Profile
January 03, 2014, 08:50:00 PM
 #1092

I'm not sure that this is the case all the time.

If you want a visual representation of the thresholds they represent the "cloud"/distance between the two EMA lines. So if the two EMA lines cross each other but the difference is not very big the buy/sell trigger will not be fired. So you can have intervals where one one EMA crosses the other and then goes back quite fast (with 0 thresholds this will trigger one buy and one sell, while with some thresholds setup it will not trigger any action (as long as the thresholds are not reached).

Also note that the thresholds do not need to be symmetric (e.g. buy=sell thresholds). So you could sell instantly when the two EMA crosses (sell thresholds=0), but wait for the buy until the difference between the EMAs is large enough.

Buy and sell bitcoins using credit cards, paypal, skrill, ukash - http://www.virwox.info
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
January 03, 2014, 09:04:09 PM
 #1093

All right! Now it's clear. I was also struggling on how to optimize strategy for these cases as they cause pretty much loss during low volatility. I read somewhere about "kind of angle of crossing", but haven't thought up an idea how to do that. Instead I used SAR and according to my test it helped a bit.

How did you choose values of these thresholds? I think it'd be also interesting to iterate over different combinations of those values too, but it would generate enormous amount of data which would take ages to graph.

Python backtesting and bot script at Github
Development halted due to other priorities.
gabbello
Member
**
Offline Offline

Activity: 81


View Profile
January 03, 2014, 09:43:10 PM
 #1094

I'm not aware of any way to choose these thresholds, but you can start from 0.25 (both buy and sell) as default values. And then iterate up and down in 0.05 steps or smth. similar (until you reach min.0 max??? maybe 0.8-1.0). 

I agree it is a lot of data (if you use 0.0-1.0 interval and 0.05 step you will have 400 combinations for each EMA pair), but that's where computers are good for Smiley. If you want to show this visually on your heatmap I would try to calculate best pair of buy/sell thresholds for each EMA pair and display only that one(s) in the relevant block.

But, a first step would be to at least allow users to set these values when running the script (instead of using 0,0 all the time).

Buy and sell bitcoins using credit cards, paypal, skrill, ukash - http://www.virwox.info
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
January 03, 2014, 09:53:39 PM
 #1095

I assume you meant:

a) sell only if dif < -threshold_sell (-0.5)
b) buy only if dif > threshold_buy  (0.06)

Right?

I'm going to hardcode 0.25 for both thresholds now, let's see what would the difference be. I think it would take around half an hour from now to finish the simulations.

Python backtesting and bot script at Github
Development halted due to other priorities.
gabbello
Member
**
Offline Offline

Activity: 81


View Profile
January 03, 2014, 10:11:25 PM
 #1096

I assume you meant:

a) sell only if dif < -threshold_sell (-0.5)
b) buy only if dif > threshold_buy  (0.06)

Right?
.

yes, sorry.

Buy and sell bitcoins using credit cards, paypal, skrill, ukash - http://www.virwox.info
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
January 03, 2014, 10:37:39 PM
 #1097

I have chosen one month for testing - september 2013 as it is pretty flat.

Traditional strategy:

5 min

15 min

30 min

1 hour

2 hours


Using thresholds of 0.5:
5 min

15 min

30 min

1 hour

2 hours


Python backtesting and bot script at Github
Development halted due to other priorities.
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
January 03, 2014, 10:50:04 PM
 #1098

Additional text statistics for these heatmaps.
http://www.sendspace.com/filegroup/7dowlksziR9P5QK2y9UX1w

Pushed changes to github so welcome to test different parameters.

Python backtesting and bot script at Github
Development halted due to other priorities.
gabbello
Member
**
Offline Offline

Activity: 81


View Profile
January 03, 2014, 10:57:57 PM
 #1099

Thank you,

Ok, so having this specific value for thresholds improves the Max. values for all intervals and sometimes improves and sometimes worsen the averages.

Probably that on a period with higher volatility the results (compared to 0 thresholds) would be quite different. And same for different threshold values Smiley.


Buy and sell bitcoins using credit cards, paypal, skrill, ukash - http://www.virwox.info
Queeq
Sr. Member
****
Offline Offline

Activity: 426



View Profile
January 03, 2014, 11:07:38 PM
 #1100

Welcome.

For me this is a pretty strange picture as relatively high profits are mostly concentrated along pairs of "neighbor" MAs.

Automatic probing of different threshold values will require many changes to the script structure. Maybe I'll take a look at it after finishing the bot script.

Python backtesting and bot script at Github
Development halted due to other priorities.
Pages: « 1 ... 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 [55] 56 57 58 59 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!