Running identical backtests getting varied results | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Identical High or Low Values Indicator 3 replies

MTF indicators don't work during backtests? 6 replies

Interesting Picture, 2 identical currency pairs! 3 replies

Question regarding automatic backtests 0 replies

Validity of Backtests in MT3? 1 reply

Attachments: Running identical backtests getting varied results
Exit Attachments

Running identical backtests getting varied results

  • #1
  • First Post: Feb 20, 2019 2:40pm Feb 20, 2019 2:40pm
I have been backtesting an EA with the exact same settings, the exact same dates, and receiving different net profit results. Admittedly, the profit variance isn't huge but it is an unsettling variable. Has anybody experienced this?
  • #2
  • Feb 20, 2019 10:01pm Feb 20, 2019 10:01pm
Quoting STD-Deviant
I have been backtesting an EA with the exact same settings, the exact same dates, and receiving different net profit results. Admittedly, the profit variance isn't huge but it is an unsettling variable. Has anybody experienced this?
If you are using the current spread setting that can make a difference but if you are using a fixed spread???
Trade what you see not what you hope
  • #3
  • Feb 20, 2019 10:11pm Feb 20, 2019 10:11pm
Quoting STD-Deviant
I have been backtesting an EA with the exact same settings, the exact same dates, and receiving different net profit results. Admittedly, the profit variance isn't huge but it is an unsettling variable. Has anybody experienced this?
You're setting the Spread to "Current". That means the spread used is the moment spread when you click Start button. Spread changes almost every tick as seen in ALT+T. It is different case if you type yes type the Spread into the textfield itself. Says 30. That means you're using fixed spread. It won't change during the backtest.
  • #4
  • Feb 20, 2019 10:42pm Feb 20, 2019 10:42pm
Oh how I wish I was. Unfortunately the spread value is inputted and fixed.

The copy of MT4 that I am backtesting on is logged into the broker. Is there any way the historical data is changing slightly? Man what a dirty trick that would be.

I am working on setting up an offline copy using Dukascopy data.
  • #5
  • Feb 20, 2019 11:00pm Feb 20, 2019 11:00pm
Quoting STD-Deviant
Oh how I wish I was. Unfortunately the spread value is inputted and fixed. The copy of MT4 that I am backtesting on is logged into the broker. Is there any way the historical data is changing slightly? Man what a dirty trick that would be. I am working on setting up an offline copy using Dukascopy data.
Maybe post some detail screenshot will help us to examine it.
Go with the flow at Shi*ting Market
  • #6
  • Feb 20, 2019 11:02pm Feb 20, 2019 11:02pm
By any chance are you running one from a Demo account and the other from a Real account? The reason I ask is because, depending on the broker, the data stream is often different -- not by a lot, but enough to cause small differences in the results.

Also -- after running a back test take a closer look at the Results tab to locate which trades actually differ -- this will help you figure it out. If you are using the same dates, symbol, period and fixed spread then the results should match. The data doesn't change.

Another thing just occurred to me -- is the "To:" date today? If it is and the Period is a lower time frame then the second back test may actually include some additional bars.

Another thing -- for the 'Use date' specify something like 2018.10.01 thru 2018.12.21 -- i.e. do NOT include today's date.

Another thing -- do you get different results for all periods -- i.e. M1 through D1?

Back testing is sort of an art -- it can provide clues and a sanity check that your EA doesn't crash MT4 -- but don't take the results too seriously.
  • #7
  • Feb 20, 2019 11:23pm Feb 20, 2019 11:23pm
Attached Image


Attached Image



There we go. Two tests run with identical parameters and conditions but differing profits. On the $559 trade the lot size even changed from .67 to .68

I'm about to put my head through the screen. :*(
  • #8
  • Feb 20, 2019 11:27pm Feb 20, 2019 11:27pm
Quoting robots4me
By any chance are you running one from a Demo account and the other from a Real account? The reason I ask is because, depending on the broker, the data stream is often different -- not by a lot, but enough to cause small differences in the results. Also -- after running a back test take a closer look at the Results tab to locate which trades actually differ -- this will help you figure it out. If you are using the same dates, symbol, period and fixed spread then the results should match. The data doesn't change. Another thing just occurred to me...
Both are run from the same live account, nothing different.

The data tested is a couple months in a previous year. Nothing recent.

The test is only being run on one time frame as that is what the EA was programmed for.
  • #9
  • Feb 21, 2019 1:44am Feb 21, 2019 1:44am
What are the column headings? This looks much different than the 'Results' tab of the MT4 Strategy Testers that I'm familiar with. Mine look like the image below:
Attached Image (click to enlarge)
Click to Enlarge Name: StrategyTester-Results.png Size: 49 KB
  • #10
  • Feb 21, 2019 1:53am Feb 21, 2019 1:53am
You mentioned lot sizes -- are these being calculated on the fly? Obviously they're not fixed. In the mql4 language you are often supposed to "Normalize" floating point values to maintain accuracy. Also, it makes a difference if the broker is using 4- or 5-digit floating point values. Perhaps the small errors are caused by floating point values not being normalized. Here's a couple of links that might be relevant:

https://docs.mql4.com/convert/normalizedouble
https://docs.mql4.com/check/digits
  • #11
  • Feb 23, 2019 11:30pm Feb 23, 2019 11:30pm
It appears the problem I was experiencing was due to a constant recalculation of the exchange rate.

Note to self: Don't run backtests connected to the server.

Credit to Sangmane for pointing out the errors of my ways.

Thanks to all of you for the input.

-SD
  • #12
  • Feb 25, 2019 1:22pm Feb 25, 2019 1:22pm
I don't want to start another thread and clog this beautiful place up so...

https://www.upwork.com/messages/att/1099932088323076096

Does anybody know why my EA would only be buying in mini-lot denominations while running in strategy tester? No micro-lots at all.


One other question if you don't mind..

https://www.upwork.com/messages/att/1099931799332806656

The EA is set up to risk 1% in each trade. Any idea why it would be showing losses lower than 1% of the previous balance?


https://www.upwork.com/messages/att/1099932940165246976

Same thing for profits. The calculated risk/reward (based on pips) showed this as a greater than 2:1 reward/risk trade but it closes out lower than 2x 1% of the account. Ideas?

The pair I'm testing has USD as the base currency = USD/xxx

Thanks for any tips.

-SD
  • #13
  • Last Post: Feb 25, 2019 2:47pm Feb 25, 2019 2:47pm
Quoting STD-Deviant
I don't want to start another thread and clog this beautiful place up so... {image} Does anybody know why my EA would only be buying in mini-lot denominations while running in strategy tester? No micro-lots at all. One other question if you don't mind.. {image} The EA is set up to risk 1% in each trade. Any idea why it would be showing losses lower than 1% of the previous balance? {image} Same thing for profits. The calculated risk/reward (based on pips) showed this as a greater than 2:1 reward/risk trade but it closes out lower than 2x 1% of the...
I figured it out. *sigh*

I went to another account that was connected to a live feed and the micro-lots as well as the proper lot size risk (1%) started occurring. I guess if you keep the platform disconnected from your broker you will have the benefit of not seeing your net profit bounce around with the latest exchange calculation but you'll have to give up on your EA calculating the correct risk/profit target down to the micro-lot. Perhaps it balances out in the end?

The key for me is to be able to see the change in profit from test to test. As long as I'm getting consistent changes as I modify parameters I should be happy. Not perfect, but it will do.
0 traders viewing now