I'm currently using Gekko to help me automate trading and follow the system that Goomboo posted. I don't know much about trading and there's still so much to learn, but I've noticed something during backtesting that I think is related to the latest posts.
When doing tests, does anyone change the number of initial candles to store? I took the H1 data from the last 5 months at BitcoinCharts (3570 candles), changed the format to what the backtester expects and tried this:
[...]
I'm wondering if there's really a need to change that setting or not. Note the huge change from 200 to 300 candles.
I am looking in to the data as to what kind of difference this makes, but do note: The EMA method is supposed to start out with a history, the ideal case would be the complete history but since this is not practicle (and the difference decreases exponentially as you remove old history). If you don't start with a big history you aren't using pure EMA as you are not able to calculate the real EMA (a number which is built up using the complete history). Diabolicus showed the EMA difference in EMA calculation between a big and a small history. If you are looking at the profit as a result of different histories you are just comparing different kind of EMAs (different levels of pure EMA so to say) which each other. It may the case that an EMA without an history would be the most profitable in a backtest, but it pretty hard to turn that into conclusions. Hope that this explanation makes sense. Awesome bot! I love the backtesting feature. I'm getting an error with setting up email though, here's the console: 2013-08-13 23:44:50 (INFO): I'm gonna make you rich, Bud Fox.
2013-08-13 23:44:50 (INFO): Let me show you some Exponential Moving Averages
.
2013-08-13 23:44:50 (INFO): Using normal settings to monitor the live market
2013-08-13 23:44:50 (INFO): NOT trading with real money
2013-08-13 23:44:51 (INFO): Profit reporter active on simulated balance
2013-08-13 23:44:51 (INFO): Calculating EMA on historical data...
2013-08-13 23:44:58 (INFO): ADVICE is to BUY @ 96.709 (0.454)
2013-08-13 23:44:58 (WARN): ERROR SENDING MAIL { [Error: authorization.faile
d]
code: 3,
previous:
{ [Error: bad response on command 'AUTH']
code: 2,
smtp: '334 eyJzdGF0dXMiOiI0MDAiLCJzY2hlbWVzIjoiQmVhcmVyIiwic2NvcGUiOiJodHRw
czovL21haWwuZ29vZ2xlLmNvbS8ifQ==\n' },
smtp: undefined }
Hi, i got exactly the same error after updating to the latest version. Darn APIs that keep changing, working on it! |
|
|
|
nm... looks like even at 100 it was not enough to calculate 'accurate' EMAs, running
[..]
Anyone know off the top of their head how many back candles are really needed to prime the EMA for 'good' values?
Glad you got it sorted, if you got any feedback on ways to make the backtesting easier / better I am all ears  Here's a little comparison:
3rd column = EMA(50), started January 1st
4th column = EMA(50), started 100 candles later (April 10th)
5th column = deviation 2nd <-> 1st EMA in %
It takes 48 days (May 28th) to bring the deviation down to 10%, and 110 days (July 29th) to get to less than 1%.
This is very variable information, thanks! Do note that due to the nature of EMA this number should be lower when using lower EMA values (like both 21 and 10). But I can't really tell if 1% is the number we should aim for. |
|
|
|
any chance that Bitstamp is working again? e02bddc..78589c9 master -> origin/master
Already up-to-date. /methods/realtime-candle-fetcher.js:98
throw 'Failed to load historical trades from ' + this.watcher.name;
^
Failed to load historical trades from Bitstamp
I am working on my own candle platform that will be available through an API so this problem will be fixed, no matter how often the API changes. You can currently use these solutions. |
|
|
|
Do not do that only for me but if other are interested too, it would be useful, the more data we have, the more reliable a strategy can be, especially if you test on the long-term! In 2011 we had the biggest bitcoin bubble until now followed by a long deflation, it can be interesting to backtest on it, the years 2012 and 2013 are very different I agree that the more data there is the better strategies you can develop. But keep in mind that markets change, I don't think the way the market behaved in 2011 is comparable to it's current behaviour. |
|
|
|
I tried to lower values but this is what I got now:' 'Calculating EMA on historical data...'' and just reports that without actually going to the next step.Could you suggest the values for candles and interval to get it started? Anything above 4 min in interval gives me first error, while setting it to 4 min just hangs...Ty for your answer !
Ah, that probably means it is working and checking every hour but has nothing to report so far. I agree that the messaging needs to be improved to indicate that it is checking. If you want to be sure you can turn on debug at the bottom of the config, this way Gekko will spit out a lot more information (and errors). |
|
|
|
Because Goomboo's method only works in certain scenario's.
If the market varies in a certain way, you'll lose a whole lot. If the market varies in a certain way, you'll make about 20% of your full investment.
The failure is (IMO) that the bot uses only one method and one strategy to win. "1 size fits all".
Using TA in a trade bot means that you are building a beta model (in this case a pretty simple one using only a single simple indicator). If you create a more complicated model taking more into account you are not better off per definition. Goomboo's point is that you have complex models and simple ones, and he found that this simple one is making him money. It also seems to lack awareness of what is a good decision vs a bad decision at any given moment. Which means it is not taking into account many different pieces of data. I am not even sure if the B-Bot is taking into account the fee structure at the exchanges. So far, from looking at it, it is only looking at EMA values and that seems to be about the majority of the decision. (?)
Otherwise, there is (or should be) no way to lose actual money on fees.
Well it would be like looking into the future to know whether a trading decision is good or bad. Here you can see that trading is more like smart gambling (there is always risk, etc.) instead of a solid stream of profit. The bot uses EMA to detect trends, you shouldn't take fees into account because (with EMA) you can only detect which direction the market will probably move to, but never how long that move will last (and thus what the profit per trade is and whether you should change position based on that profit and the costs = fees). It is my hope that in a future revision, the B-Bot needs to evolve not only to use "A different strategy" but to use "more than one" depending on prior conditions.
Agreed (except for the depending on prior conditions part), but I think the target audience for this bot is people who are just getting into trading to see if they can make a little profit, etc. Using only EMA is not only simple in the math sense, it's also pretty easy to comprehend. If you are serious about getting into (algo) trading you probably would not use a browser plugin (security issues, stability issues, networking / UI / processing / data storage limitations and the fact that auto browser updates can break it anytime of the day). I think the max you can squeeze out of the market is 20% to 40% of whatever you put in (per month).
The problem is the increased risk and losing just as much as you are gaining. GoomBoo's method seems to be about making sure bets at the cost of significant opportunities. So if there were 20 opportunities to increase your wealth then you only took 2 of them because the EMA lines were blatantly profitable.
True, (only using) EMA is extremely bad when you want to manage risk. And 20 - 40% per month is extreme from a profit perspective. One of the smartest groups in the market are doing that on a yearly basis (so 12 times less profit per month). And they are not using EMA as their model, they got an army of quants who all using their Phd's in battle. |
|
|
|
Hi, thank you for your wonderful project! I have a small question. How often should Gekko to send email ? I have received only one letter after the start. Gekko sends an email when a change in the market is detected. With the default setup it uses EMA to detect such a change on an hourly basis. When there is a lot of volatility Gekko will probably report a couple of times a day (max) but more realistically you should expect it once a day. If something is wrong Gekko currently only logs it (when you enable debug = true, as you did), if you are using forever use `forever list` to find the logfile, you can read it by typing `tail [name of logfile]` (or use `cat` if you want to read the whole thing). I'm trying to start it and here's what I get: "Failed to load historical trades from Bitstamp". Any suggestions?
I am currently working on a solution for this as this is a big problem, what you can try for now: lessen the `candles` in the settings, lessen the `interval` in the settings. Note that setting `candles` to a lower number will decrease the trend detection performance in the start (until it build up the history it can't fetch right now). If you are only using paper trading this means it's better to ignore the first couple of days. If you lessen the `interval` Gekko will look at smaller timeframes (so it will detect trends on a smaller timescale so to say). This problem is not present at Mt. Gox right now btw. |
|
|
|
Is that NVD3? Because I am building something similar myself |
|
|
|
Here is the original blogpost by the guy. The biggest thing I got out of the article was this: Meiklejohn and fellow researcher Damon McCoy, an assistant professor of computer science at George Mason University, have been mapping out a network of bitcoin wallets that are used exclusively by the curators of the Silk Road. If you wish to transact with merchants on the Silk Road, you need to fund your account with bitcoins. The act of adding credits appears to be handled by a small number of bitcoin purses.
“All Silk Road purchases are handled internally by Silk Road, which means money trades hands from the Silk Road account of the buyer to the Silk Road account of the seller,” explained Meiklejohn, author of the paper, A Fistful of Bitcoins: Characterizing Payments Among Men with No Names, to be released in October 2013 at the ACM Internet Measurement Conference in Barcelona, Spain. I knew about Silk Road mixing all coins internally, but I didn't knew research was being done on an academic level in trying to track this kind of flow of money. The same kind of research could be used to track all other kinds of monetary flows. For example where a company offering services for bitcoin will spent those coins on again, what employee gets how much coins. Or how a kid getting pocket money decides to spent it, etc. Not pretty sure if the current the blockchain (and in our current economical culture where almost all transactions are on the blockchain, written in stone so to say) is the kind of system I really want to have.
|
|
|
|
Does this still use BTC-e historical data from bitcoincharts?
My concern is Bitcoincharts downtime/recently they missed a bunch of data from BTC-e
At the moment it does, however I am in the process of storing all trade data on my server, precalculating most standard candles and exposing them through an API. Though this still is a central point of failure. Sneak peak: > db.stats()
{
"db" : "coindata",
"collections" : 144,
"objects" : 151908,
"avgObjSize" : 85.47955341390842,
"dataSize" : 12985028,
"storageSize" : 32378880,
"numExtents" : 329,
"indexes" : 285,
"indexSize" : 12476576,
"fileSize" : 201326592,
"nsSizeMB" : 16,
"dataFileVersion" : {
"major" : 4,
"minor" : 5
},
"ok" : 1
}
(currently watching different 144 markets, got 150k+ trades stored). I suspect it may take a while before this is stable enough to be included in Gekko. |
|
|
|
Hi again,
Firstly, thanks for your help on this!
Realise that Gekko is dependent on external sources - don't get me wrong, I wasn't complaining about Gekko at all, just thought I'd help with a bugreport... Fully understand that it might be external sources. But since it works with USD, it is a bit strange, I thought...
As I said, it's working absolutely fine with USD... Only when I change the currency to EUR, then I get this issue.
Tried it with JPY and SEK as well (just changing the conf) to try out other pairs - doesn't seem to be working either, only USD so far for me...
The issue already appears at the start of gekko. Interval is at 30mins at the moment. Running well with USD...
Let me know if I can help somehow to enable/simplify/troubleshoot a fix. I'm on windows, if it helps. Can't find the logs anywhere though...
Hosting standard candle intervals would be interesting. But realise that this might cost you time/money/effort, so thanks for considering it!
Hi, Sorry if I came across as trying to push the issues to external dependencies, I meant that in most markets besides USD there is little volatility. Currently when Gekko tries to calculate a candle which doesn't have any trade, you'll get NaN for the rest of the time (so also for all other candles that come after it). I'll fix this bug but I don't know how to handle situations where candles don't have any trades at all. Right now you could set the interval to something higher than 30 minutes to lessen the chance that a candle will have 0 trades, but I'll patch it ASAP so that it will still work. |
|
|
|
This is very interesting indeed... Will the server be open source too so we can run our own instances? I think this is especially important for people who want to use non-standard candles. Also will it be able to be used for the backtesting too?
Thanks. I am planning on hosting all standard candle intervals (15min, 1h, 4h, 1d, etc.) for all those markets and providing them for free (as long as I can afford it). I could open source it all but storing all the data is not something as small as running a Gekko instance, there is a lot of trade data flowing from all exchanges. Take a look at the mtgox sqlite database for example, it is over 4GB already. I am now thinking of maybe turning this into a service where all standard candles for all those markets can be retrieved by everyone (free). And maybe I can see what other stuff can be calculated easily. I am thinking about things like spread graphs between different BTC/LTC (or other pure crypto) markets for arbitrage or maybe realtime datastreams to popular charting software like sierrachart. Right now I am just exploring all the different things I can do with this stuff. -- This will be an optional solution for Gekko, in no way will this be forced upon Gekko's users because nobody can easily check whether the candle data is not messed with. |
|
|
|
Hi, Using MtGox, I get NaN errors when using EUR... USD seems to be working fine. Enabled verbose to check what's going on. With USD I get: 2013-07-30 09:59:29 (DEBUG): calced EMA properties for new candle:
2013-07-30 09:59:29 (DEBUG): short ema 101.048
2013-07-30 09:59:29 (DEBUG): long ema 101.037
2013-07-30 09:59:29 (DEBUG): diff ema 0.010
2013-07-30 09:59:29 (DEBUG): fetching new trades for new bucket at Mt. Gox
2013-07-30 09:59:31 (DEBUG): calculated candle: 0
2013-07-30 09:59:31 (DEBUG): calced EMA properties for new candle:
2013-07-30 09:59:31 (DEBUG): short ema 101.038
2013-07-30 09:59:31 (DEBUG): long ema 101.033
2013-07-30 09:59:31 (DEBUG): diff ema 0.005
2013-07-30 09:59:31 (DEBUG): we are currently not in an up or down trend @ 100.981 (0.005) When changing to EUR, I get something like: 2013-07-30 10:04:03 (DEBUG): calculated candle: 1
2013-07-30 10:04:03 (DEBUG): calced EMA properties for new candle:
2013-07-30 10:04:03 (DEBUG): short ema NaN
2013-07-30 10:04:03 (DEBUG): long ema NaN
2013-07-30 10:04:03 (DEBUG): diff ema NaN
2013-07-30 10:04:03 (DEBUG): fetching new trades for new bucket at Mt. Gox
2013-07-30 10:04:04 (DEBUG): calculated candle: 0
2013-07-30 10:04:04 (DEBUG): calced EMA properties for new candle:
2013-07-30 10:04:04 (DEBUG): short ema NaN
2013-07-30 10:04:04 (DEBUG): long ema NaN
2013-07-30 10:04:04 (DEBUG): diff ema NaN
2013-07-30 10:04:04 (DEBUG): we are currently not in an up or down trend @ 75.500 (NaN) This is the config I changed: config.normal = {
enabled: true,
exchange: 'MtGox', // 'MtGox', 'BTCe' or 'Bitstamp'
currency: 'EUR',
asset: 'BTC',
tradingEnabled: false,Simply changed the currency in there - and then the error appears. Anybody had this issue? What is the interval you have set? The volatility is very low at the moment, any time Gekko encouters an interval in which there were no trades this happens. So al small interval will trigger this sooner. See below: realtime-candle-fetcher.js:98
throw 'Failed to load historical trades from ' + this.watcher.name;
^
Failed to load historical trades from bitcoincharts
I have been getting this all weekend for Bitstamp too. Gekko is currently not dependent on external factors (besides the exchange APIs and bitcoincharts for BTC-e), this is great and I want to keep it like this for as long as possible. But with the recent problems I am facing I am going to calculate candles on my server expose those to Gekko/the world (it stays optional).This solves a lot of small issues: - No 'failed to load data from exchange X'
- No sudden API changes exposed to Gekko
- Able to keep track of a lot of exchanges, even those without a historical data API. This finally means altcoins as well!
- Way more efficient: This way the candles are only calculated once and everyone uses that directly
Problems: - If for whatever reason I can't fetch the candles no one using Gekko is able to (paper) trade
- Central point of failure
- I have to pay for the server / bandwidth, if I can't it will be shutdown
- It's going to be hard to offer different non-standard intervals
I set up a script that is watching a bunch of exchanges with different currency pairs at the moment.Those can all be exposed to Gekko (assuming the exchanges have trading APIs) I am currently watching: > show collections
bit2c_BTC/NIS
bitstamp_BTC/USD
btcchina_BTC/CNY
btce_BTC/EUR
btce_BTC/RUR
btce_BTC/USD
btce_EUR/USD
btce_FTC/BTC
btce_LTC/BTC
btce_LTC/RUR
btce_NMC/BTC
btce_NVC/BTC
btce_PPC/BTC
btce_TRC/BTC
btce_USD/RUR
bter_BQC/BTC
bter_BQC/CNY
bter_BQC/LTC
bter_BTB/BTC
bter_BTC/CNY
bter_CNC/BTC
bter_CNC/CNY
bter_CNC/LTC
bter_FRC/BTC
bter_FRC/CNY
bter_FRC/LTC
bter_FTC/BTC
bter_FTC/CNY
bter_FTC/LTC
bter_LTC/BTC
bter_LTC/CNY
bter_MYMINER/BTC
bter_NMC/BTC
bter_NME/LTC
bter_PPC/BTC
bter_PPC/CNY
bter_PPC/LTC
bter_TRC/BTC
bter_TRC/CNY
bter_TRC/LTC
bter_WDC/BTC
bter_WDC/CNY
bter_WDC/LTC
bter_YAC/BTC
bter_YAC/CNY
cryptsy_ALF/BTC
cryptsy_AMC/BTC
cryptsy_ANC/BTC
cryptsy_ARG/BTC
cryptsy_BQC/BTC
cryptsy_BTB/BTC
cryptsy_BTE/BTC
cryptsy_BTG/BTC
cryptsy_CAP/BTC
cryptsy_CGB/BTC
cryptsy_CNC/BTC
cryptsy_CRC/BTC
cryptsy_CSC/BTC
cryptsy_DBL/LTC
cryptsy_DGC/BTC
cryptsy_DMD/BTC
cryptsy_DVC/LTC
cryptsy_ELC/BTC
cryptsy_EMD/BTC
cryptsy_EZC/LTC
cryptsy_FLO/LTC
cryptsy_FRC/BTC
cryptsy_FRK/BTC
cryptsy_FST/BTC
cryptsy_FTC/BTC
cryptsy_GLD/LTC
cryptsy_HYC/BTC
cryptsy_IFC/LTC
cryptsy_IXC/BTC
cryptsy_JKC/LTC
cryptsy_KGC/BTC
cryptsy_LKY/BTC
cryptsy_LTC/BTC
cryptsy_MEC/BTC
cryptsy_MEM/LTC
cryptsy_MNC/BTC
cryptsy_MST/LTC
cryptsy_NAN/BTC
cryptsy_NBL/BTC
cryptsy_NMC/BTC
cryptsy_NRB/BTC
cryptsy_NVC/BTC
cryptsy_PPC/BTC
cryptsy_PXC/BTC
cryptsy_QRK/BTC
cryptsy_RYC/LTC
cryptsy_SBC/BTC
cryptsy_TRC/BTC
cryptsy_WDC/BTC
cryptsy_XNC/LTC
cryptsy_XPM/BTC
cryptsy_YAC/BTC
fxbtc_BTC/CNY
fxbtc_LTC/BTC
fxbtc_LTC/CNY
mtgox_BTC/AUD
mtgox_BTC/BTC
mtgox_BTC/CAD
mtgox_BTC/CHF
mtgox_BTC/CNY
mtgox_BTC/CZK
mtgox_BTC/DKK
mtgox_BTC/EUR
mtgox_BTC/GBP
mtgox_BTC/HKD
mtgox_BTC/JPY
mtgox_BTC/NOK
mtgox_BTC/NZD
mtgox_BTC/PLN
mtgox_BTC/RUB
mtgox_BTC/SEK
mtgox_BTC/SGD
mtgox_BTC/THB
mtgox_BTC/USD
mtgox_USD/BTC
vircurex_BTC/DGC
vircurex_BTC/DVC
vircurex_BTC/FRC
vircurex_BTC/FTC
vircurex_BTC/IXC
vircurex_BTC/LTC
vircurex_BTC/NMC
vircurex_BTC/NVC
vircurex_BTC/PPC
vircurex_BTC/RUR
vircurex_BTC/TRC
vircurex_BTC/USD
vircurex_BTC/XPM
virwox_BTC/SLL
It's been running for a day or something and I currently have 120k trades stored (this does not include historical data). |
|
|
|
Would this position be available to college students? (IE I can do heavy, daily work during summers but then during the school year practicaly available only on weekends)
Wondering something similar: are you looking for interns? (I'm currently not based in the US) |
|
|
|
People also seem to be aware of the need of decriminalizing controlled substances, stopping waging wars overseas, opposing the military industrial complex and are generally aware of how rotten to the core the system is on here.
The perceived difference comes from the fact that you share most beliefs with the rest of the people on here. So instead of reading about someone blaming those damn jews, you are reading about how we (the bitcoiners) will kick the Bilderberg group of the new world order throne. Which may not be regarded as propaganda but instead as conspiracy by most intellectual societies I know (most academics, etc). |
|
|
|
As most candlestick generators assume open price to be equal the close price from the previous bar, it turns out the algorithms at Cryptotrader handle market data correctly.
In traditional trading (markets that open in the morning and close in the afternoon) it happens quite often that the close of the previous day is lower / higher than the open of the new day.  Markets like BTC/USD don't close at all but I have seen candles where the open and the close were not the same at all. |
|
|
|
I think there has been misunderstanding about my explanation, as i tried to explain how internal calculation of a simulated purchase is done, but not how trends are detected.
Oops my bad, I read it to quick probably  |
|
|
|
Once the bar is closed, an algorithm handles it immediately. if you will have bars in 1min or 30s? it will not be as immediate as it is now. you cannot buy for average price of current bar because that period is just closed, you buy on the next bar avg price, your trade is being aggregated to this next bar. That's why you are counting the price as next bar. You can perform simulation based on real executed price of orders ~ estimated by both methods with lag/with no lag. When you have 30 minutes bar the engine will trigger when a bar closes, within a very short amount the engine will determine whether it needs to change the portfolio (and if to buy or sell to reflect this). This will all probably happen within 1 second after the bar closed, with the low liquidity in the BTC market I really don't think the effects are big. And if they are I don't see how the average price of the next bar is any better price indicator as a lot can happen after in the 29 minutes and 59 seconds after the decision (maybe the open of the next candle would be a little better but I really don't see it right now). About the lag: I don't know if this data is available at all, but it is not going to help you much: when there is a 10 second lag it means the trading engine is lagging 10 seconds behind. All orders from between 10 seconds ago and now are queued and they are probably not available, so they are unknown at the time you make your decision (not 100% sure, maybe there are in the order book already). If I got your question correctly: Trade price = Close price for the current bar + Trading fee Remember, this is only a simulation so the formula does not consider slippage and other real trading costs. I don't agree on your price calculation: While the trading fee obviously needs to be payed while ordering, the alpha (EMA/MACD indicator) is only responsible for detecting the trend, you are now manipulating the data so that trends happening do not get properly detected IMO (difference is pretty small, but it's there).
This is a hard problem because the current alpha (a simple trend indicator) can't tell you anything about how long a trend will probably last (statistically) or how much the price will change. If you would have a model predicting this you can better determine if you should reposition your portfolio while taking fees into account. My bot only uses the fees to simulate the profit, it does not use them to determine what position to take.EDIT: It appears I failed to read properly. |
|
|
|
This is a big step towards implementing a system that supports over a 100 different indicators ( ta-lib) because it needs to get candles fed. May I ask how far are you with the ta-lib implementation? Also, could you make it somewhat similar to the code that cryptotrader.org accepts? It would be so much easier if one could just simply copy&paste coffescript snippets back and forth. I was pretty far. But because of the API changes in both bitcoincharts (BTC-e data) and bitstamp I probably have to change core. I want to have the core stabilised before I'm adding all the advanced stuff. I was planning to do something similar for a while, I am not a fan of coffeescript myself. I think plain javascript is a lot easier for most people, as it's very similar to a lot of big languages like Java and C. But adding coffeescript ofcourse is a piece of cake since Gekko is a js project, so this will make it in. I don't think I can make it exactly the same (with all function calls, etc). But I'll see what I can do |
|
|
|
It's not looking very good for the algos I think this is a good demonstration to show how hard trading profitably is as the market continues to evolve. Of course it can always be fun to experiment with small amount, but investing can be like gambling when you are not into it. |
|
|
|
|
Show Posts
