Why "Walk Forward Analysis" is still unreliable and useless!

Could the WFE be used to determine the best OOS window size? I notice for a given system that the results vary greatly based on the OOS window size used, and that, depending on the system, a very large OOS (say seven years) is not necessarily better than a smaller one (say three or even two years). I am trying to figure out a method by which one can assess the best OOS window size to use for a system, rather than just assuming larger is better, which I don't think is true after one hits enough trades for statistical robustness. One thought is to run an analysis over the same span of data, but with a series of sequentially different OOS window sizes and then then assess WFE from those results. Of course, that suffers from the same WFA problems that you are trying to address. But there should be a method to determine the proper OOS window size (and forward window size as well) for a system. I just don't think that "longer is simply better" for some systems, particularly those that trade lower TF than the daily. As an example, the Euro today is not trading anything like it was three years ago on the 15M charts. If one is using a 5 year OOS window for a 15M breakout system, you are watching expectancy tank right now even though your past WFA may have looked great over many different settings. Now if one uses a 2 year OOS window, the same system has adjusted for market conditions. Thoughts?

Sorry for the late answer!

Well, what you say is true, absolutely
To find the best insample-timespan (for out of sample: the shorter, the better should be true), DATFRA has a different optimisation approach.

It first evaluates all parametersets, and then calculates the in-sample & out-of-sample pairs afterwards - based on the single trades taken.
That way, you can test multiple in-sample & out-of-sample timespan combinations, without re-doing backtests and not suffering from lost data, like WFA

But for WFA, all you could do is: Test multiple In-Sample timespans and record stuff like WFE, variance of WFE, profit, drawdown etc.
And then use gnuplot or something to plot different in-sample timespans against 2 of the other values, like the heatmaps i showed.
That way you should be able to estimate an optimum if you are lucky. but it is all very uncertain, as i stated in my article

-Darwin

I am just doing some tests with DATFRA, and I noticed something that is very well visualising what I want to state in this thread.

Currently, DATFRA's System Builder module does a Walk Forward Analysis on new trading systems. But well, I think I have to get rid of this approach completly.
Here you can see the SAME walk forward analysis (EURUSD H4, system uses 2 moving averages for entry and 2 for exit, optimisation was done using MT4's genetic algorithms), repeated 5 times:

http://s7.directupload.net/images/140326/lahlxh4n.png
http://s1.directupload.net/images/140326/xy474ibb.png
http://s1.directupload.net/images/140326/9vzngyg5.png
http://s14.directupload.net/images/140326/a4o8v4vr.png
http://s1.directupload.net/images/140326/mwom6fzw.png

You can see that the outcomes fluctuate so heavily, that WFA is even useless for a first pre-evaluation of a system, I guess.


-Darwin

PS: The system was actually a proftiable one (tough, very far from perfect!).
Here is a Step-By-Step Analysis of it using DATFRA: http://darwins-fx-tools.com/?example
Btw, you can download the free version of DATFRA and do the analysis steps in the video on your own, without any registration or stuff like that.

Sorry might be oot, but i dont know where to share, since this is the forum of most Robbust Trader build the system

its about optimizing, ok here it is

in 2014 i do WFA,
tons of learning also in asirikuy about how robust is WFA method, trade on open bar only, etc2
but its an effort too much, i always WFA my EA for weeks or month. And an Open Bar EA surely had long stagnant Drawdown period.
As at the end , im coming to my own conclusion, continous WFA is also curve fitting it self. In a different way.

I even see another silly method 'WALKBACKWARD analysist' LMAO Hahahahah.
Hey, i mean it in esential way,
If we curve fit the last period, and we test the ea in the oldest period as out of sampling, it also an OUT OF SAMPLING METHOD
Even lanchester the developer of Multiple ea do that. Optimized for last 6 month, but make sure the results in the early year also profitable.
In early years period means = OUT OF SAMPLE.

No mater what we trade in the past or future, esentially it still OOS for testing robustness
In WFA The Best Out of sample we pick, is the best that we choose to live trade.
So whats different than curve fitting??

The results in Live trade, the market always change beyond our expectation.
Even with continous development.
The last market behaviour is the least we could adapt with our optimization.

Even i know a friend works in a real Financial Institution, a 20 years trading experience in a hedge fund.
From what i see, the way he do to adapt market change is no different than curve fitting.
Im not saying WFA is a wrong method, i also still had WFA tools, but rarely using it now.

So, as a simple statistic approach, why dont we build a most robust system that all in sample data we could had?
Even it tooks hundreds hour for a single curency. (With TickData also now!)

The more sample we optimized is the more statisticly robust proven. No matter what the market behavior change, historicly some of it repeated itself.
Market Trade only know 4 behaviour : Trend, Reversal, Breakout, and Ranging. Only the volume volatility that changing.
Then i optimized all the historical data as robust at we can with as much data as i had.
And build porftolio with those 4 character inside to minimize stagnant period. And took every chance in every condition change.
Then picking up the best KPI from it optimization results.

Such as Profit factor, CAGR, etc. But Not get too shiny on the score itself, since those Profit Factor, CAGR, Payoff, etc always had flaws statisticly
So i pick with the most trades as posible first, that mean the more data, more sample and most market condition as we could had.
Run montecarlo, stagnancy period, drawdown % and period analysis with the risk that we could stomach.
And dont change the rules and system before that max risk happened. Wich mean market still in our portfolio prediction.
(So actually i do WFA also now, but my out of sample is a live trade. LOL. Kidding)

However this is my personal conclusion. Dont pick it if you dont walk the path yourself.
Im not bashing out Asirikuy with WFA approach, but i felt like im wasting times backthen sticking to others personal rules to build a systems.
But however, he build a great platform and different insight.

No matter what system we build, tick scalping, swing, to long trend trading, or even a Grid.
Whatever approach we choose, Even now im doing curve fitting to 'latest market condition' approach. From the base parameters that already robust entire period as much i can have. And back using TDS Variable spreads ON.

As It IS the 'TRUE real market condition'. Not open bar.
At the end of the day, we trade to made money.
Not to be a smart academicly.

So for me the hints is as i mention above 'Run the system with the max risk we could stomach with'.
No matter we use WFA or Curve fitting the whole period method.
Managing Risk first, then profits follow.

Cheers,


My two cents
Once again : this is my personal conclusion. Dont pick it if you dont walk the path yourself.

BEST WAY TO TREAT EA Portofolio : Risk Weighting, 12 Curency porto. Keep optimizing fo curvefitting all time??

Im a MAM and PAMM manager trading in multi broker house

We hoping to built system as robust as posible
so we can leave it alone 24/7/365 days

but what i found is
https://www.mql5.com/en/forum/370289

Yes, there no such a thing as WFA robustnest guarantee,
If WFA posible as out of sample,
So Does WBA , Walk Back Ward Analysis, its still out of sample

Do it with your TICK DATA broker that differ EACH!
Live Spread, Slippage Simulation,
YEAHH! Thats when OUR Wife started to leave us by ourself never going out in weekend!
LMAO

This is where our duty END statistically whatever tools and coding we do!



But at least with Portfolio Analyzer you should had it, we can find projection in monte carlo, if our system is already breach the max drawdown and stagnant area





NEXT !!!!!!!!!!!!

I have simple Solution wich i practice and work in live trade!
We dont need recode EA changing strategy multiple times!


Just Curve Fit The RISK Weighting for Each Curency!
And Use Monte Carlo back for predictig Worst Case Scenario



The logic is KISS Keep it simple and stupid :
Whatever we did to Our Strategy is traning the system to be adapt with each Curency
Each curency depent on Fundamental decicission
Thats where Pattern began and change
Drawdown and stagnatn period happened during those change

But not every country curency or CFD had the same moment of fundamental change,
Even in Covid 2020 era
If we had 12 curency running! 4 in Drawdown moment that those country still doing some policy to do. lets say 8 running Good !

We lowering the risk of the 4, until those 4 went back on track
We rising the risk of the rest of 8, until they start going wrong by our statistic measure

But Use Portofolio Analyzer as your bench mark on the New Curve fitted Risk,
So we know, if WCS breach, we do Re-Curvitting more!


This is never ending Cuve fitting i know, Exhausting right?
But, believe in me, i know you guys never leave your PC, desktop etc, even you have already had 24:! Profit factor system

So,
By doing this way, I climbing up the roller coaster faster than the MonteCarlo WCS Worscasescenario


Thanks God!

My Private MAM keep going strong in this Covid and Coin transistion Era
And Now im living bigger from FX rather than my Automotive Business

I eat MAM nd PAMM Fee bigger than my own profit!
Client Profit, me profit

Every one happy






Just My 2 Cent!












NB : NEver publish your strategy description detail in any forum, even to calm down or supporting client there
Brokers do spy on us, and silly things like 15-25 pips slippage always happened after i reveal some algo change in any forum and social media

O Simpati, [8/19/2022 2:12 AM]
ROBUST HOW WE KEEP ENJOYING PROFITS REGULARLY
O Simpati, [8/19/2022 2:12 AM]
We Trade and scalp and swing few comodities, small position, Risk Reduced
This strategy proven
We've withdrawl 15-20% profits 3 times in this WAR ERA.
regular system keep running with 0.7 times risk, the 0.3 is covered by comodities, yes its Oild and Gold, Few Strong Fundamental
IMPORTANT
To make every COMPOUNDED profits SAFE and periodictly : DONT THINK COMPOUND CAPITAL
WITHDRAWL every sudden profits made in the range above 5% Gain
We cant hindered Drawdown even Musk and Buffet Too now
But ORDequity V2 Modified Strategy proven no need much time to regain New High,
after 5-7% Gain new high, withdawl it, or Keep it in Wallet
Compounded Drawdown reduced!
Like Now and also 3 times at few month ago.
Then, after we kept the protits in Wallet or Withdrawl
We Just Waiting few Weeks for our Basic Capital Equity to re-gain new 5% High, Then Withdrawl it again
Repeat.
We talking about compounded DRAWDOWN, talk on risk first, then safety follow, and profits achieved.

Keep Safe and Stay Alive!

MAM has gone 100% Profit
Tickmill Small Signal Main Account had Gone 4000% Profit
Alpari on the way To NEW HIGH with 343% Profits Since Inception
myfxbook.com/members/ORDExhaust
We Keep GUNNIN !

Hello I think WFA can be reliable I hope your up for a discussion

Hello I think WFA can be reliable I hope your up for a discussion