Best Historical Data Source for Backtesting

[forexample]

Sorry to rehash an old topic but every search I do just results in pages of EAs.

 

I am currently testing a few EAs from this site. I use Alpari's MT4 and so had downloaded Metaquotes data at end of Feb. At the end of March I redownloaded so I could test to end of March and checked data by retesting same as before (to end of Feb) and I got very different results - some trades completely missed, some different trades etc. I deleted historical data, redownloaded and got diff results again! It seems that the data is different every time and that there are lots of holes. This is not for a scalper either so not so fiddly. It is giving me a headache.

 

I know Birt's thread for testing at 99% but for first run of tests I'm happy with 90%, but I would like to have reliable data. Does anyone have a good source? I hope in the beginning for EURUSD, 2001-present.

 

As an aside, how to keep a set spread? I remember a while back someone's advice was to open a demo, ge tthe feed, then delete the demo and dont re-login but if I do this backtester just doesn't work.

 

Mad kudos for anything that works and is accurate!

 

Thanks in advance,

 

FE

[konorti]

I have been using these:

http://www.forextester.com/data/datasources.html

http://global.fxdd.com/ru/mt1m-data.html

They also say that data before 2006 is not really reliable.

[forexample]

The reason I am asking was based on a bigger issue that I have just realised. I have a live account with Alpari UK and I have been using the same platform for backtesting and for live trading. I recently re-downloaded the historical data from the history center (EU) and my charts ended up with a 13 day gap in March. Support told me my platform was "corrupt" and to try a fresh install. I did this, and the chart was fine, UNTIL i downloaded history again and the same gap appeared (candle chart attached). You can see inside the history center how much data is missing (picture attached). While I accept that historical data is not trustworthy, I am shocked that as soon as data is downloaded charts use this feed.

 

http://i43.tinypic.com/5ds712.gif

 

This is completely different from what support informed me, and also disturbing in the face that they disclaim how the data is not true Alpari data and can't be used in disputes. I trade daily technicals and gaps like this throw off the whole charts - I first noticed because my TLs looked so different from one day to the next (after the gap appeared).

 

I am now using two platforms, one for backtesting and one for live trading.

 

http://i43.tinypic.com/200biwo.jpg

Edited April 29, 2010 by forexample

[pro]

Alpari Uk/mt4 uses every dirty trick to fool the clients. Once i detected a 8 months data gap on USD/JPY....

[thesecret]

My friend once found GMT shift +3h in the middle of the year for couple of months in some data.

 

No its not DST.

[fx4a]

Use "Page Up" to scroll back as far as you can with TF you are testing include 1 min. This way you get Broker data, not metaquotes, which can have missing data.

[forexample]

Use "Page Up" to scroll back as far as you can with TF you are testing include 1 min. This way you get Broker data, not metaquotes, which can have missing data.

 

Last night after having issues with downloaded data not compiling M5, M30 etc I had this idea and spent an hour or two scrolling back to the start of 2006 on M1, retested and its working fine now from what I can see. Good advice though, thanks given!

 

I have backtested for the last two years and thought I was only getting "90%" accuracy, what a joke that was!

Edited April 29, 2010 by forexample

[forexample]

I live in Japan, where companies and/or individuals take full responsibility for any issues. Leaders and Executives step down (they even commit suicide at times!) if their company of any of their workers commit atrocities rather than blame it on someone else. I thought I was past all these broker issues a long time ago and was happy in my blissful ignorance (another brick in the wall). Anyway, if anyone cares to see how Alpari UK treats a customer with a live account, and I am not betting either way that they will blame me/my platform or take responsibility (well I have my opinion based on past issues but I will spare judgement), here is the mail I sent support today:

 

Firstly, I am not accusing you personally of any intent to mislead the customer. I assume that the issue at hand stems from poor teaching at a higher level and I ask you to question your advisors in more detail on this.

 

I did as you said and reinstalled a fresh platform. The charts were fine. After re-downloading history data from Metaquotes the charts went back to having a large gap as can be seen here:

 

http://i43.tinypic.com/5ds712.gif

 

If I then deleted all history and refreshed the chart it went back to normal. It is not my data or platform etc that is corrupt, it is the Metaquotes data as can be seen here (I did this multiple times to be sure; and this is just one example of large holes):

 

http://i43.tinypic.com/200biwo.jpg

 

So firstly, for anyone experiencing the same issue, please just advise them to delete all history and refresh chats, a fresh install is not necessary.

 

Also, for your records, if you wish to advise customers how to get good quality data for backtesting, they simply have to open their live feed, start with a Monthly chart and work down to M1 chart, allow 4,000,000 bars in history in Options (if they wish to test back to 2001), and scroll back in time to however far they need data. This takes a long time but the data is of good quality. It took me a while playing with the platform to realise this, but it is not so difficult to work out and it is something support should be aware of.

 

While I understand that downloaded history is not accurate, your statement that charts come from direct feed is incorrect and misleading. As soon as history is downloaded from Metaquotes, charts run off this feed, and I have proven this multiple times. Whoever told you otherwise is either ill-advised or directly misleading customers. I of course take the position that they are merely ill-advised and am not accusing anyone of anything, however I do ask you to do more research to help customers make advised decisions. As a long-term technical trader with a large account at Oanda I have been testing Alpari and a number of other MT4 brokers with multiple small accounts with the intent of funding a pro account for EA trading as I am also a software developer, but wish to find a broker that treats the little guy as well as I am treated at Oanda. Things like this make me question whether Alpari will be my broker of choice. A lot of people use MT4 to trade EAs as well as live accounts and once they download data from Metaquotes their live data is corrupt and you should make customers aware of this.

 

I hope you take this mail not as an offensive attack but as an exposition for facts, and I hope my comments help you better support customers with the same issue in the future, as I know you value your clients’ opinions and strive to always offer better services.

 

Best Regards,

[forexample]

could have been worse, more a turnaround and doublestep than a blame game:

 

Thank you for your email and your comments, which were in no way taken as personal accusations.

 

I would like to address some of the points that you raised.

 

Firstly, we are aware of the Tools => Options => Charts max bars in history and we advise clients to use this when they report that there is no or insufficient data on their results.

 

With regards to the charts coming from direct feed, this is in no way incorrect or misleading information; the charts do run on our live feed, but when historical data is downloaded, it overrides our live feed up to the latest date on the download. After than point the data on the charts will come from the live feed again, unless another download is done.

 

Perhaps I did not make myself clear enough, making you think that the data on the charts comes from the direct feed even when downloading historical data, so apologies if my explanation was in any way confusing.

 

Finally, we do warn the customers that they are downloading data from MetaQuotes, and that the downloaded history data can differ from the data stored on the trade server; in no way we try to mislead or provide inaccurate information:

 

Once again, thank you for your comments/feedback.

 

Please do not hesitate to contact us should you require further assistance.

 

Kind Regards

 

[Msdogan]

I have same problems with data. When i delete history i can get good chart with no GAP but when i do backtest i get bad quality. But when i download data from metaquetos then look my chart i see big GAPs but %90 quality. But at the end it makes different results. How can i fix this ?

[hyperdimension]

I have similarly been through a lot of trouble, headache and wasted hours/days with historical data. My advice is don't bother with the data from History Center.

 

Ideally it is best to log tick data from your live account that you intend to trade through and use that as your historical data for backtesting. However, this is impractical because as soon as something goes wrong such as your terminal crashes, you have introduced a hole and therefore corrupted your data. I have thought of using multiple terminals in different locations/VPSs to log data from a single broker's server, however I need a good way to combine the data and discard duplicate ticks.

 

The next best thing is to use Dukascopy's data. Just read birt's tick data web page. It's worth it to save a lot of headache dealing with crap data from other sources.

[fx1st]

maybe this could be interesting for your consideration :

hxxp://www.pronetfx.com/datacentre.asp

 

sorry for bad english..

[forexample]

I found a good resource for backtesting on someones blog - is it ok to post the link here? It is a good help. I am using fxdd's data from konorti's post on the first page.

 

I am using Alpari's MT4, I tried just backscrolling M1 data but could only get to Feb of this year frustratingly. I wish I had Alpari's M1 data as they used to upload on their site but they are holding out on us.

 

Interestingly, since FXDD doesn't go lower than 1.0 lots on the data I have, I have to test with 1.0 lots or bigger or the EAs won't work. I guess a lot of info is held in this historical data.

[Ajata]

I found a good resource for backtesting on someones blog - is it ok to post the link here? It is a good help. I am using fxdd's data from konorti's post on the first page.

 

I am using Alpari's MT4, I tried just backscrolling M1 data but could only get to Feb of this year frustratingly. I wish I had Alpari's M1 data as they used to upload on their site but they are holding out on us.

 

Interestingly, since FXDD doesn't go lower than 1.0 lots on the data I have, I have to test with 1.0 lots or bigger or the EAs won't work. I guess a lot of info is held in this historical data.

 

Please post the link if you think this is good blog.

[swordfish]

Dukascopy data

 

Can someone figure out how to do this and let us know? I did one try and got data corrupted during download.

http://eareview.net/tick-data