Experimental MT5 EA using Gemini Flash API - GeminiCommanderEA_v1

Quoting Bl4ckP3n9u1n

Disliked

{quote} Still testing. This is a hobby in between my normal work so I don't have a lot of time to react to issues during working hours. I hope this run will be the last before I upload it. Screen of the new MAR ratio score (replaced Sharpe) of 6.3864. Easier to understand is it shows risk-adjusted returns. The current strategy being tested says the best results in the cycle have "$6.3864 of profit for every $1 of drawdown," What to expect in V208: 1. AI-Driven Configuration & Realism (GeminiAnalyzer) Dynamic Broker Simulation (New in V208): V207:...

Ignored

Quoting Bl4ckP3n9u1n

Disliked

{quote} As an FYI. It is good practice if you are testing the framework and make changes, always delete the Results folder before starting again. Why? If you have any previous results or strategies that have been developed from the AI, it is most likely built upon flawed logic. When the script runs it will check for a Results folder and use all the pre-existing data to continue to learn from. Remember that Gemini (AI) is powerful and I have had times it has found weaknesses in the calculations etc and exploited these just to make a (unrealistic)...

Ignored

Quoting Bl4ckP3n9u1n

Disliked

{quote} Still testing caveat... Here's V208 for anyone willing to try it. Feedback welcome. {file}

Ignored

"BasicRSIBounce": {
    "risk_profile": {
        "account_size_suitability": ["Micro", "Standard"]
    }
}

"DonchianBreakout": {
    "risk_profile": {
        "account_size_suitability": ["Standard", "Large"]
    }
}

"TIERED_RISK_CONFIG": {
    2000:  {'Medium': {'risk_pct': 0.01, 'pairs': 1}},
    5000:  {'Medium': {'risk_pct': 0.012, 'pairs': 1}},
    10000: {'Medium': {'risk_pct': 0.008, 'pairs': 2}},
    // ... and so on for larger accounts
}

Quoting preptraditio

Disliked

Thnak you for the last update, going through the code I am wondering why aren't you saving study.best_trial as it would help understand how it got to that "best in class" model? Also, in order to bridge the wining startegy to an EA for each pair, do we need to rebuild the EX5 or we can just do it in pyhon? Howeve,r wou yould basically need to code all the strategies and have a selector that enables just the wining one or something like this?

Ignored

# In End_To_End_Advanced_ML_Trading_Framework_PRO_V208_Linux.py # After training the final_pipeline final_scaler = final_pipeline.named_steps['scaler'] # Example for RobustScaler scaler_params = { 'median': final_scaler.center_.tolist(), 'iqr': final_scaler.scale_.tolist() } with open(os.path.join(result_folder_path, "scaler_params.json"), 'w') as f: json.dump(scaler_params, f)

# In Python, after training
model = final_pipeline.named_steps['model']
model.save_model(os.path.join(result_folder_path, "xgb_model.json")) # Saves model in XGBoost's JSON format

Quoting Bl4ckP3n9u1n

Disliked

{quote} Still testing caveat... Here's V208 for anyone willing to try it. Feedback welcome. {file}

Ignored

import pandas as pd
import numpy as np
import os
import re
import pandas_ta as ta
from numpy import nan as npNaN # Use np.nan directly
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import classification_report, roc_auc_score, f1_score, precision_score, recall_score, precision_recall_curve, confusion_matrix, ConfusionMatrixDisplay
from sklearn.calibration import calibration_curve
from sklearn.manifold import TSNE # For t-SNE diagnostic
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Input
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau
import tf2onnx # Ensure: pip install tf2onnx
import joblib  # Ensure: pip install joblib
from imblearn.over_sampling import SMOTE # Ensure: pip install imbalanced-learn
# --- Plotting and Diagnostics Imports ---
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style="whitegrid") # Set Seaborn style for diagnostic plots
# ==============================================================================
# === Global Configuration & Feature Definitions ===============================
# ==============================================================================
CSV_FILE_PATH = r"D:\MetaTrader\Training\GBPUSD_H1_201501020900_202504300000.csv" # User specified
def get_symbol_tf_from_filename(filepath):
    basename = os.path.basename(filepath)
    match = re.match(r"([A-Z]{6})_([A-Z0-9]{2,3})", basename)
    if match: return match.group(1), match.group(2)
    parts = basename.split('_'); return parts[0], parts[1] if len(parts) > 1 else "UNKNOWNTF"
SYMBOL, TIMEFRAME_STR = get_symbol_tf_from_filename(CSV_FILE_PATH)
print(f"INFO: Determined Symbol: {SYMBOL}, Timeframe: {TIMEFRAME_STR} from filename.")
PIP_VALUES = {"EURUSD": 0.0001, "GBPUSD": 0.0001, "AUDUSD": 0.0001, "USDCAD": 0.0001, "USDJPY": 0.01}
PIP_VALUE_FOR_TARGET_LABELING = PIP_VALUES.get(SYMBOL, 0.0001)
if SYMBOL not in PIP_VALUES: print(f"WARNING: PIP_VALUE for {SYMBOL} defaulted to 0.0001. Verify.")
EXPECTED_OHLCV_FOR_TA = ['Open', 'High', 'Low', 'Close', 'Volume']
PRECALCULATED_FEATURES_IN_CSV = [
    'signal_confidence_score', 'account_drawdown_pct_norm', 'portfolio_open_trades_norm',
    'minutes_to_next_news_norm', 'onnx_regime_prob_TREND', 'onnx_regime_prob_CHOP', 'onnx_regime_prob_NEWS_VOL',
]
TECHNICAL_FEATURES_TO_CALCULATE = [
    'ATR_14_H1_norm', 'ADX_14_D1', 'ADX_14_H4', 'Price_Dist_EMA50_H1_norm',
    'Price_Dist_EMA200_H4_norm', 'RSI_14_H1_norm', 'EMA20_H1_Slope_norm',
    'HourSin', 'HourCos', 'DayOfWeekSin', 'DayOfWeekCos',
    'MACD_hist_H1_norm', 'Volatility_H1_20period_norm', 'Close_lag_1_norm',
    'Close_lag_5_norm', 'Rolling_Mean_Close_10_norm', 'Rolling_Std_Close_10_norm'
]
TARGET_COLUMN = 'trade_suitability_score'
LABEL_LOOKAHEAD_BARS = 12
LABEL_PROFIT_TARGET_PIPS = 75
LABEL_STOP_LOSS_PIPS = 35
MODEL_VERSION = "v9_full_diag"
MODEL_OUTPUT_UNITS = 1
ONNX_MODEL_FILENAME = f"OmegaEA_TradeSuitability_{SYMBOL}_{TIMEFRAME_STR}_{MODEL_VERSION}.onnx"
SCALER_FILENAME = f"OmegaEA_TradeSuitability_{SYMBOL}_{TIMEFRAME_STR}_scaler_{MODEL_VERSION}.gz"
DIAGNOSTIC_PLOTS_PREFIX = f"Diag_{SYMBOL}_{TIMEFRAME_STR}_{MODEL_VERSION}_"
# --- Imbalance Handling & Loss Function Configuration ---
USE_SMOTE = False
USE_CLASS_WEIGHTS = False # Typically use one or the other, or Focal Loss
CLASS_WEIGHT_0 = 1.0
CLASS_WEIGHT_1 = 7.0
USE_FOCAL_LOSS = True # Set this to True to use Focal Loss
FOCAL_LOSS_GAMMA = 2.0
FOCAL_LOSS_ALPHA = 0.25 # Can be tuned, e.g., based on inverse class frequency
# --- Prediction Threshold & Diagnostics Configuration ---
FIND_OPTIMAL_THRESHOLD = True
DEFAULT_CLASSIFICATION_THRESHOLD = 0.5
RUN_POST_TRAINING_DIAGNOSTICS = True
 
# ==============================================================================
# === Diagnostic Functions (Adapted from your script) ========================
# ==============================================================================
def check_nan_inf_numpy(numpy_array, name="NumPyArray"):
    nan_count = np.isnan(numpy_array).sum()
    inf_count = np.isinf(numpy_array).sum()
    print(f"[{name}] NaNs: {nan_count}, Infs: {inf_count}")
    return nan_count > 0 or inf_count > 0
def plot_tsne_numpy(embeddings_np, labels_np, save_path):
    if embeddings_np.shape[0] < 2 : # TSNE needs at least 2 samples
        print(f"Skipping t-SNE plot for {save_path}: Not enough samples ({embeddings_np.shape[0]}).")
        return
    print(f"Generating t-SNE plot for {save_path}...")
    try:
        tsne = TSNE(n_components=2, random_state=42, perplexity=min(30.0, embeddings_np.shape[0] - 1.0), n_iter=300) # Adjust perplexity
        reduced = tsne.fit_transform(embeddings_np)
        plt.figure(figsize=(10, 8))
        sns.scatterplot(x=reduced[:, 0], y=reduced[:, 1], hue=labels_np, palette="viridis", s=50, alpha=0.7)
        plt.title(f"t-SNE of Embeddings/Activations ({os.path.basename(save_path)})")
        plt.xlabel("t-SNE Component 1"); plt.ylabel("t-SNE Component 2")
        plt.legend(title='True Labels')
        plt.tight_layout(); plt.savefig(save_path); plt.close()
        print(f"t-SNE plot saved to {save_path}")
    except Exception as e:
        print(f"ERROR generating t-SNE plot: {e}")
def plot_threshold_sweep(y_true, y_probs, save_path):
    print(f"Generating Threshold Sweep plot for {save_path}...")
    try:
        precision, recall, thresholds = precision_recall_curve(y_true, y_probs)
        # thresholds length is one less than precision/recall
        f1_scores = np.zeros_like(thresholds)
        # Calculate F1 for each threshold, handling potential division by zero
        for i in range(len(thresholds)):
            if (precision[i] + recall[i]) == 0:
                f1_scores[i] = 0
            else:
                f1_scores[i] = 2 * (precision[i] * recall[i]) / (precision[i] + recall[i])
        
        plt.figure(figsize=(10, 7))
        plt.plot(thresholds, precision[:-1], label="Precision", color='blue', linestyle='--') # Exclude last P, R value
        plt.plot(thresholds, recall[:-1], label="Recall", color='green', linestyle=':')
        plt.plot(thresholds, f1_scores, label="F1 Score", color='red', linewidth=2) # f1_scores already matches thresholds length
        plt.xlabel("Classification Threshold")
        plt.ylabel("Score")
        plt.title(f"Precision, Recall, F1-Score vs. Threshold ({os.path.basename(save_path)})")
        plt.legend(); plt.grid(True); plt.tight_layout(); plt.savefig(save_path); plt.close()
        print(f"Threshold Sweep plot saved to {save_path}")
    except Exception as e:
        print(f"ERROR generating Threshold Sweep plot: {e}")
 
def plot_calibration_curve(y_true, y_probs, save_path): # Renamed function
    print(f"Generating Calibration Curve for {save_path}...")
    try:
        prob_true, prob_pred = calibration_curve(y_true, y_probs, n_bins=10, strategy='uniform')
        plt.figure(figsize=(8, 8))
        plt.plot(prob_pred, prob_true, marker='o', linewidth=1, label="Model Calibration Curve")
        plt.plot([0, 1], [0, 1], linestyle='--', color='gray', label="Perfectly Calibrated")
        plt.title(f"Calibration Curve ({os.path.basename(save_path)})")
        plt.xlabel("Mean Predicted Probability (per bin)"); plt.ylabel("Fraction of Positives (per bin)")
        plt.legend(); plt.grid(True); plt.tight_layout(); plt.savefig(save_path); plt.close()
        print(f"Calibration Curve saved to {save_path}")
    except Exception as e:
        print(f"ERROR generating Calibration Curve: {e}")
def plot_custom_confusion_matrix(y_true, y_pred_labels, save_path, display_labels=None): # Renamed
    print(f"Generating Confusion Matrix for {save_path}...")
    if display_labels is None: display_labels = np.unique(y_true)
    try:
        cm = confusion_matrix(y_true, y_pred_labels, labels=display_labels) # Ensure labels are passed if specific order is needed
        disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=display_labels)
        fig, ax = plt.subplots(figsize=(7,7))
        disp.plot(cmap="Blues", values_format="d", ax=ax)
        plt.title(f"Confusion Matrix ({os.path.basename(save_path)})")
        plt.tight_layout(); plt.savefig(save_path); plt.close()
        print(f"Confusion Matrix saved to {save_path}")
    except Exception as e:
        print(f"ERROR generating Confusion Matrix: {e}")
# UMAP is optional
# def plot_umap(...): try: import umap ... except ImportError: print("UMAP not installed...")
# Gradient saturation is complex with Keras model.fit(). Will omit for now.
# def log_gradient_saturation(...): ...
# ==============================================================================
# === Function to Generate Target Variable (Labeling) ==========================
# ==============================================================================
def generate_trade_suitability_labels(df_input, lookahead, profit_pips, sl_pips, pip_val_param):
    # ... (Your generate_trade_suitability_labels function - from previous correct version) ...
    df = df_input.copy(); print(f"INFO: Generating target '{TARGET_COLUMN}'...")
    df[TARGET_COLUMN] = 0
    if any(col not in df.columns for col in ['Close','High','Low']): print(f"CRITICAL ERROR in generate_labels: Missing OHLC."); return df_input
    profit_target = profit_pips * pip_val_param; stop_loss = sl_pips * pip_val_param
    for i in range(len(df) - lookahead):
        entry_price = df['Close'].iloc[i]; buy_tp_hit=0; buy_sl_hit=0; sell_tp_hit=0; sell_sl_hit=0
        for k in range(1, lookahead + 1): # Corrected loop range
            if i + k >= len(df): break
            future_high=df['High'].iloc[i+k]; future_low=df['Low'].iloc[i+k]
            if not buy_sl_hit and future_low <= entry_price - stop_loss: buy_sl_hit=1
            if not buy_sl_hit and not buy_tp_hit and future_high >= entry_price + profit_target: buy_tp_hit=1
            if not sell_sl_hit and future_high >= entry_price + stop_loss: sell_sl_hit=1
            if not sell_sl_hit and not sell_tp_hit and future_low <= entry_price - profit_target: sell_tp_hit=1
            if (buy_sl_hit or buy_tp_hit) and (sell_sl_hit or sell_tp_hit): break
            if (buy_sl_hit or buy_tp_hit) and not (sell_sl_hit or sell_tp_hit) and k==lookahead and not sell_tp_hit: sell_sl_hit=1
            if (sell_sl_hit or sell_tp_hit) and not (buy_sl_hit or buy_tp_hit) and k==lookahead and not buy_tp_hit: buy_sl_hit=1
        if buy_tp_hit and not buy_sl_hit: df.iloc[i, df.columns.get_loc(TARGET_COLUMN)] = 1
    print(f":white_check_mark: Target '{TARGET_COLUMN}' generated. Distribution:\n{df[TARGET_COLUMN].value_counts(normalize=True, dropna=False)}"); return df
# ==============================================================================
# === 1. Data Loading, Validation, and Enhanced Feature Engineering ============
# ==============================================================================
def load_and_prepare_data(csv_path_param):
    # ... (Full load_and_prepare_data function from my previous response - Sun, 02 Jun 2025 01:55:00 GMT) ...
    # ... (This is the long block of code that handles MT5 CSV, TA features, precalc features, normalization prep)
    # --- PASTE THE FULL load_and_prepare_data FUNCTION FROM MY PREVIOUS RESPONSE HERE ---
    if not os.path.exists(csv_path_param):
        print(f"CRITICAL ERROR: CSV file not found at path: {csv_path_param}"); return None, None, None
    print(f"Loading data from {csv_path_param}..."); df = pd.read_csv(csv_path_param, delimiter='\t')
    rename_map = {};
    for col_original in df.columns:
        col_standardized = col_original.strip().strip("<>")
        if col_standardized.upper() == "DATE": rename_map[col_original] = 'MT5_Date'
        elif col_standardized.upper() == "TIME": rename_map[col_original] = 'MT5_Time'
        elif col_standardized.upper() == "OPEN": rename_map[col_original] = 'Open'
        elif col_standardized.upper() == "HIGH": rename_map[col_original] = 'High'
        elif col_standardized.upper() == "LOW": rename_map[col_original] = 'Low'
        elif col_standardized.upper() == "CLOSE": rename_map[col_original] = 'Close'
        elif col_standardized.upper() == "TICKVOL": rename_map[col_original] = 'Volume'
        elif col_standardized.upper() == "VOL" or col_standardized.upper() == "REALVOLUME": rename_map[col_original] = 'RealVolume'
        elif col_standardized.upper() == "SPREAD": rename_map[col_original] = 'Spread_Points'
    df.rename(columns=rename_map, inplace=True)
    date_col_name = 'MT5_Date' if 'MT5_Date' in df.columns else None; time_col_name = 'MT5_Time' if 'MT5_Time' in df.columns else None
    if not date_col_name or not time_col_name: print(f"CRITICAL ERROR: Date/Time columns for Timestamp missing."); return None,None,None
    df['Timestamp'] = pd.to_datetime(df[date_col_name] + ' ' + df[time_col_name], format='%Y.%m.%d %H:%M:%S')
    df.drop(columns=[date_col_name, time_col_name], inplace=True, errors='ignore'); df.set_index('Timestamp', inplace=True); df.sort_index(inplace=True)
    missing_ohlcv_cols = [col for col in EXPECTED_OHLCV_FOR_TA if col not in df.columns]
    if missing_ohlcv_cols: print(f"CRITICAL ERROR: Missing OHLCV columns: {missing_ohlcv_cols}"); return None,None,None
    print(f":white_check_mark: MT5 CSV processed. Shape: {df.shape}");
    if not isinstance(df.index, pd.DatetimeIndex): print("CRITICAL ERROR: Index not DatetimeIndex."); return None,None,None
    if TARGET_COLUMN not in df.columns:
        print(f"INFO: Target '{TARGET_COLUMN}' not found. Generating labels..."); df = generate_trade_suitability_labels(df, LABEL_LOOKAHEAD_BARS, LABEL_PROFIT_TARGET_PIPS, LABEL_STOP_LOSS_PIPS, PIP_VALUE_FOR_TARGET_LABELING)
        if TARGET_COLUMN not in df.columns or df[TARGET_COLUMN].isnull().all(): print(f"CRITICAL ERROR: Target gen failed."); return None,None,None
    else: print(f"INFO: Target '{TARGET_COLUMN}' found in CSV.")
    print("Calculating TAs..."); df_H1_ta=df.copy(); df_H4_ta=df.resample('4H').agg({'Open':'first','High':'max','Low':'min','Close':'last','Volume':'sum'}).dropna(); df_D1_ta=df.resample('1D').agg({'Open':'first','High':'max','Low':'min','Close':'last','Volume':'sum'}).dropna()
    if df_H1_ta.empty or df_H4_ta.empty or df_D1_ta.empty: print("ERROR: Resampling empty DFs."); return None,None,None
    df_H1_ta['ATR_14_H1_raw']=df_H1_ta.ta.atr(length=14); df_H1_ta['RSI_14_H1_raw']=df_H1_ta.ta.rsi(length=14); df_H1_ta['EMA20_H1']=df_H1_ta.ta.ema(length=20); df_H1_ta['EMA50_H1']=df_H1_ta.ta.ema(length=50); df_H1_ta['EMA20_H1_Slope_raw']=df_H1_ta['EMA20_H1'].diff(1)
    macd_h1 = df_H1_ta.ta.macd(close=df_H1_ta['Close'], fast=12, slow=26, signal=9)
    if macd_h1 is not None and 'MACDh_12_26_9' in macd_h1.columns: df_H1_ta['MACD_hist_H1_raw'] = macd_h1['MACDh_12_26_9']
    else: df_H1_ta['MACD_hist_H1_raw'] = np.nan
    df_H4_ta['EMA200_H4']=df_H4_ta.ta.ema(length=200); df_H4_ta['ATR_14_H4_raw']=df_H4_ta.ta.atr(length=14); adx_h4_df=df_H4_ta.ta.adx(length=14); df_H4_ta['ADX_14_H4']=adx_h4_df['ADX_14'] if adx_h4_df is not None and 'ADX_14' in adx_h4_df.columns else np.nan
    adx_d1_df=df_D1_ta.ta.adx(length=14); df_D1_ta['ADX_14_D1']=adx_d1_df['ADX_14'] if adx_d1_df is not None and 'ADX_14' in adx_d1_df.columns else np.nan
    cols_h1 = ['ATR_14_H1_raw','RSI_14_H1_raw','EMA20_H1','EMA50_H1','EMA20_H1_Slope_raw','MACD_hist_H1_raw']
    for col in cols_h1:
        if col not in df.columns and col in df_H1_ta.columns: df[col]=df_H1_ta[col]
        elif col in df_H1_ta.columns: df[col]=df_H1_ta[col]
    df=df.merge(df_H4_ta[['EMA200_H4','ATR_14_H4_raw','ADX_14_H4']],left_index=True,right_index=True,how='left'); df=df.merge(df_D1_ta[['ADX_14_D1']],left_index=True,right_index=True,how='left'); df.ffill(inplace=True)
    df['ATR_14_H1_rolling_mean']=df['ATR_14_H1_raw'].rolling(window=24*5,min_periods=10).mean().replace(0,np.nan); df['ATR_14_H1_norm']=(df['ATR_14_H1_raw']/df['ATR_14_H1_rolling_mean']).fillna(1.0)
    df['RSI_14_H1_norm']=df['RSI_14_H1_raw'].fillna(50.0)/100.0; df['ATR_14_H1_raw_filled']=df['ATR_14_H1_raw'].replace(0,np.nan).ffill().bfill().fillna(0.0001); df['ATR_14_H4_raw_filled']=df['ATR_14_H4_raw'].replace(0,np.nan).ffill().bfill().fillna(0.0001)
    df['EMA20_H1_Slope_norm']=(df['EMA20_H1_Slope_raw'].fillna(0)/df['ATR_14_H1_raw_filled']); df['Price_Dist_EMA50_H1_norm']=((df['Close']-df['EMA50_H1'])/df['ATR_14_H1_raw_filled']); df['Price_Dist_EMA200_H4_norm']=((df['Close']-df['EMA200_H4'])/df['ATR_14_H4_raw_filled'])
    df['ADX_14_D1'].fillna(20.0,inplace=True); df['ADX_14_H4'].fillna(20.0,inplace=True)
    df['MACD_hist_H1_norm']=(df['MACD_hist_H1_raw']/df['ATR_14_H1_raw_filled']).fillna(0)
    df['Volatility_H1_20period_norm']=(df['Close'].rolling(window=20).std()/df['ATR_14_H1_raw_filled']).fillna(0)
    for lag in [1,5]: df[f'Close_lag_{lag}']=df['Close'].shift(lag); df[f'Close_lag_{lag}_norm']=((df['Close']-df[f'Close_lag_{lag}'])/df['ATR_14_H1_raw_filled']).fillna(0)
    df['Rolling_Mean_Close_10_norm']=((df['Close']-df['Close'].rolling(window=10).mean())/df['ATR_14_H1_raw_filled']).fillna(0); df['Rolling_Std_Close_10_norm']=(df['Close'].rolling(window=10).std()/df['ATR_14_H1_raw_filled']).fillna(0)
    df['HourSin']=np.sin(2*np.pi*df.index.hour/23.0); df['HourCos']=np.cos(2*np.pi*df.index.hour/23.0); df['DayOfWeekSin']=np.sin(2*np.pi*df.index.dayofweek/6.0); df['DayOfWeekCos']=np.cos(2*np.pi*df.index.dayofweek/6.0)
    for col_list in [TECHNICAL_FEATURES_TO_CALCULATE, PRECALCULATED_FEATURES_IN_CSV]:
        for col in col_list:
            if col not in df.columns: print(f"CRITICAL WARNING: Column '{col}' defined but not created. Filling 0."); df[col]=0.0
            elif df[col].isnull().sum()>0: print(f"INFO: Column '{col}' has {df[col].isnull().sum()} NaNs. Filling 0."); df[col].fillna(0.0,inplace=True)
    all_features_for_model=TECHNICAL_FEATURES_TO_CALCULATE + PRECALCULATED_FEATURES_IN_CSV
    df.dropna(subset=all_features_for_model + [TARGET_COLUMN], inplace=True)
    if df.empty or len(df) < 50: print(f"ERROR: Insufficient data ({len(df)}) after NA drop for features."); return None,None,None
    X=df[all_features_for_model].values; y=df[TARGET_COLUMN].values
    if not np.all(np.isin(y,[0,1])): y=np.clip(y,0,1).astype(int)
    scaler=MinMaxScaler(feature_range=(0,1)); X_scaled=scaler.fit_transform(X)
    print(f"Preprocessing complete. X_scaled: {X_scaled.shape}, y: {y.shape}"); return X_scaled,y,scaler
# ==============================================================================
# === 2. Custom Loss Function: Focal Loss ======================================
# ==============================================================================
def focal_loss_func(gamma=FOCAL_LOSS_GAMMA, alpha=FOCAL_LOSS_ALPHA): # Use global configs
    def focal_loss_concrete(y_true, y_pred): # Renamed inner function
        y_true_float = tf.cast(y_true, tf.float32)
        y_pred_clipped = tf.clip_by_value(y_pred, 1e-7, 1.0 - 1e-7)
        pt = tf.where(tf.equal(y_true_float, 1.0), y_pred_clipped, 1.0 - y_pred_clipped)
        alpha_factor = tf.where(tf.equal(y_true_float, 1.0), alpha, 1.0 - alpha)
        focal_weight = alpha_factor * tf.pow(1.0 - pt, gamma)
        # Use tf.keras.losses.BinaryCrossentropy directly for calculation if needed, or pre-calculated bce
        bce_val = tf.keras.backend.binary_crossentropy(y_true_float, y_pred_clipped, from_logits=False)
        return tf.reduce_mean(focal_weight * bce_val)
    return focal_loss_concrete # Return the actual loss function
# ==============================================================================
# === 3. Model Building ========================================================
# ==============================================================================
def build_keras_model(input_dim, output_units=MODEL_OUTPUT_UNITS, use_focal_loss_flag=USE_FOCAL_LOSS):
    model = Sequential([
        Input(shape=(input_dim,), name="input_1"),
        Dense(128, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001)), Dropout(0.3),
        Dense(64, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001)), Dropout(0.2),
        Dense(32, activation='relu'),
        Dense(output_units, activation='sigmoid', name="output_1")
    ])
    loss_function_to_use = focal_loss_func() if use_focal_loss_flag else 'binary_crossentropy'
    metrics_to_track = ['accuracy', tf.keras.metrics.AUC(name='auc_roc', curve='ROC'),
                        tf.keras.metrics.Precision(name='precision'), tf.keras.metrics.Recall(name='recall'),
                        tf.keras.metrics.AUC(name='auc_pr', curve='PR')]
    model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005),
                  loss=loss_function_to_use, metrics=metrics_to_track)
    model.summary(); return model
# ==============================================================================
# === 4. Training Pipeline & Diagnostics =======================================
# ==============================================================================
def train_and_export_onnx_model(path_to_csv, output_onnx_filename, output_scaler_filename):
    X_scaled, y_target, feature_scaler = load_and_prepare_data(path_to_csv)
    if X_scaled is None: print("Training aborted: Data prep failed."); return
    min_data_for_split = 30
    if X_scaled.shape[0] < min_data_for_split * 2 : print(f"ERROR: Not enough samples ({X_scaled.shape[0]}) for split."); return
        
    # Create a dedicated validation set from training data for custom diagnostics if Keras internal one is not used
    X_train_full, X_test, y_train_full, y_test = train_test_split(X_scaled, y_target, test_size=0.2, shuffle=False)
    
    # If Keras uses validation_split, X_train_full becomes the data for model.fit,
    # and X_test is our hold-out.
    # If Keras validation_split=0, then we need our own X_val, y_val from X_train_full for diagnostics.
    # Let's make an explicit validation set for diagnostics before training on the rest.
    if len(X_train_full) > min_data_for_split * 1.25 : # Ensure X_train_full is large enough for another split
        X_train, X_val_diag, y_train, y_val_diag = train_test_split(X_train_full, y_train_full, test_size=0.25, shuffle=False) # 0.25 * 0.8 = 0.2 of total
    else: # Not enough data for a separate diagnostic validation set, use test set for diagnostics carefully
        X_train, y_train = X_train_full, y_train_full
        X_val_diag, y_val_diag = X_test, y_test # Use test set for diagnostics
        print("WARNING: Using Test set for post-training diagnostics due to limited data for separate validation set.")
 
    if len(X_train) < min_data_for_split or len(X_val_diag) < 5 or len(X_test) < 5 :
         print(f"ERROR: Too few samples after splitting. Train: {len(X_train)}, Val_Diag: {len(X_val_diag)}, Test: {len(X_test)}. Cannot proceed.")
         return
    print(f"Original complete training data shapes: X_train_full: {X_train_full.shape}, y_train_full: {y_train_full.shape} (Class 1s: {np.sum(y_train_full)})")
    if USE_SMOTE:
        print(f"\nAttempting SMOTE on X_train, y_train data...")
        # ... (SMOTE logic as before, operating on X_train, y_train) ...
        minority_class_count = np.min(np.bincount(y_train)); smote_k_neighbors = min(5, minority_class_count - 1) if minority_class_count > 1 else 1
        if minority_class_count <= smote_k_neighbors or smote_k_neighbors < 1: print(f"WARNING: Minority class too small for SMOTE. Skipping.")
        else:
            try: smote = SMOTE(random_state=42, k_neighbors=smote_k_neighbors); X_train, y_train = smote.fit_resample(X_train, y_train)
            except ValueError as e: print(f"WARNING: SMOTE failed: {e}. Using original data.")
        print(f"After SMOTE: X_train: {X_train.shape}, y_train dist: {np.bincount(y_train)}")
    print(f"Final Train shapes for model.fit: X_train: {X_train.shape}, y_train: {y_train.shape}")
    print(f"Diagnostic Validation shapes: X_val_diag: {X_val_diag.shape}, y_val_diag: {y_val_diag.shape}")
    print(f"Test shapes: X_test: {X_test.shape}, y_test: {y_test.shape}")
    model = build_keras_model(input_dim=X_train.shape[1]) # use_focal_loss_flag is global
    callbacks = [EarlyStopping(monitor='val_auc_roc', mode='max', patience=25, restore_best_weights=True, verbose=1), ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=10, min_lr=1e-7, verbose=1)]
    
    class_weights_to_use = None
    if USE_CLASS_WEIGHTS and not USE_SMOTE and not USE_FOCAL_LOSS:
        # ... (class weight calculation as before) ...
        counts = np.bincount(y_train);
        if len(counts)==2 and counts[0]>0 and counts[1]>0: class_weights_to_use={0:CLASS_WEIGHT_0, 1:CLASS_WEIGHT_1}; print(f"INFO: Using class weights: {class_weights_to_use}")
        else: print("WARNING: Cannot apply class weights.")
    elif USE_CLASS_WEIGHTS: print("INFO: Class weights not applied (SMOTE or Focal Loss active).")
    
    print("Starting model training...")
    # Keras model.fit will use its own validation_split on the X_train, y_train it receives
    history = model.fit(X_train, y_train, epochs=200, batch_size=64, validation_split=0.2, callbacks=callbacks, class_weight=class_weights_to_use, verbose=1)
    # --- Post-Training Diagnostics (using X_val_diag, y_val_diag) ---
    if RUN_POST_TRAINING_DIAGNOSTICS and len(X_val_diag) > 0 and len(y_val_diag) > 0:
        print("\n\n==================================================================")
        print(":bar_chart: Post-Training Diagnostics on Dedicated Validation Set")
        print("==================================================================")
        
        y_pred_probs_val = model.predict(X_val_diag).ravel()
        # 1. Check class distribution in y_val_diag
        print("\n:bar_chart: Diagnostic Validation Set - True Class Distribution:")
        unique_val, counts_val = np.unique(y_val_diag, return_counts=True)
        for cls, count_val_diag in zip(unique_val, counts_val):
            print(f"Class {int(cls)}: {count_val_diag} samples ({100 * count_val_diag / len(y_val_diag):.2f}%)")
        check_nan_inf_numpy(X_val_diag, "X_val_diag (Features for Diagnostics)")
        check_nan_inf_numpy(y_pred_probs_val, "y_pred_probs_val (Model Probs on Val Diag)")
        # 2. Plot distribution of predicted probabilities
        print("\n:test_tube: Predicted probabilities on Diagnostic Validation Set (summary):")
        print(f"Min: {y_pred_probs_val.min():.4f}, Max: {y_pred_probs_val.max():.4f}, Mean: {y_pred_probs_val.mean():.4f}")
        print(f"Preds < 0.1: {np.sum(y_pred_probs_val < 0.1)}, Preds > 0.9: {np.sum(y_pred_probs_val > 0.9)}")
        try:
            plt.figure(figsize=(10,5))
            sns.histplot(y_pred_probs_val, bins=50, kde=True, color="purple")
            plt.title(f"Distribution of Predicted Probabilities (Diag Val Set) - {DIAGNOSTIC_PLOTS_PREFIX}")
            plt.xlabel("Probability of Class 1"); plt.ylabel("Frequency"); plt.grid(True); plt.tight_layout()
            plt.savefig(f"{DIAGNOSTIC_PLOTS_PREFIX}prob_dist_val.png"); plt.close()
            print(f"Probability distribution plot saved.")
        except Exception as e_plt: print(f"Plotting error (prob_dist_val): {e_plt}")
        # 3. Check if predictions are nearly constant
        print("\n:mag: Checking if Diagnostic Validation Set predictions are near-constant:")
        print("Unique rounded predictions (2 decimals):", np.unique(np.round(y_pred_probs_val, 2)))
        # 4. Compute ROC AUC
        try: auc_roc_val = roc_auc_score(y_val_diag, y_pred_probs_val); print(f"\n:chart_with_upwards_trend: ROC AUC Score (Diag Val Set): {auc_roc_val:.4f}")
        except Exception as e_auc: print(f":x: Error computing ROC AUC for Diag Val set: {e_auc}")
        # 5. Threshold Sweep Plot
        plot_threshold_sweep(y_val_diag, y_pred_probs_val, f"{DIAGNOSTIC_PLOTS_PREFIX}thresh_sweep_val.png")
        
        # 6. Calibration Curve Plot
        plot_calibration_curve(y_val_diag, y_pred_probs_val, f"{DIAGNOSTIC_PLOTS_PREFIX}calib_curve_val.png")
        
        # 7. Confusion Matrix (using default 0.5 threshold for this diagnostic view)
        y_pred_labels_val_diag_default_thresh = (y_pred_probs_val > DEFAULT_CLASSIFICATION_THRESHOLD).astype(int)
        plot_custom_confusion_matrix(y_val_diag, y_pred_labels_val_diag_default_thresh,
                                     f"{DIAGNOSTIC_PLOTS_PREFIX}conf_matrix_val_default_thresh.png", display_labels=[0,1])
        print(f"\n:clipboard: Classification Report (Diag Val Set, Threshold = {DEFAULT_CLASSIFICATION_THRESHOLD}):")
        try: print(classification_report(y_val_diag, y_pred_labels_val_diag_default_thresh, zero_division=0))
        except Exception as e_cr_val: print(f"Could not generate classif. report for Diag Val Set: {e_cr_val}")
        # Optional: t-SNE/UMAP on hidden layer activations of X_val_diag
        try:
            if len(model.layers) > 1:
                intermediate_model_val = tf.keras.Model(inputs=model.input, outputs=model.layers[-2].output)
                hidden_activations_val = intermediate_model_val.predict(X_val_diag)
                if hidden_activations_val is not None and len(hidden_activations_val) > 0:
                    plot_tsne_numpy(hidden_activations_val, y_val_diag, f"{DIAGNOSTIC_PLOTS_PREFIX}tsne_val.png")
        except Exception as e_tsne: print(f"Could not plot t-SNE for Diag Val Set: {e_tsne}")
        print("--- End of Post-Training Diagnostics on Dedicated Validation Set ---\n")
    else:
        print("INFO: Skipping Post-Training Diagnostics on dedicated validation set (not enough data or disabled).")
 
    # --- Final Evaluation & Optimal Threshold for Test Set ---
    final_report_classification_threshold = DEFAULT_CLASSIFICATION_THRESHOLD
    if len(X_test) > 0 and len(y_test) > 0 :
        print("\n--- Final Evaluation on Test Set ---")
        y_pred_probs_test = model.predict(X_test).ravel()
        eval_results = model.evaluate(X_test, y_test, verbose=0)
        metric_names = model.metrics_names
        print("Metrics from model.evaluate on Test Set (uses default 0.5 threshold for acc/prec/rec):")
        for name, val in zip(metric_names, eval_results): print(f"  Test {name}: {val:.4f}")
        if MODEL_OUTPUT_UNITS == 1 and len(np.unique(y_test)) >= 2:
            if FIND_OPTIMAL_THRESHOLD:
                precision_c, recall_c, thresholds_c = precision_recall_curve(y_test, y_pred_probs_test)
                if len(precision_c) > 1 and len(recall_c) > 1 and len(thresholds_c) > 0:
                    f1_scores_c = (2*precision_c[:-1]*recall_c[:-1])/(precision_c[:-1]+recall_c[:-1]+1e-9)
                    if len(f1_scores_c)>0: best_f1_idx_c=np.argmax(f1_scores_c); final_report_classification_threshold=thresholds_c[best_f1_idx_c]
                    print(f"Optimal threshold for Test Set F1-score: {final_report_classification_threshold:.4f}")
                else: print("P-R curve too short for optimal threshold on Test Set, using default.")
            else: print(f"Using default threshold for Test Set report: {final_report_classification_threshold}")
            
            y_pred_labels_test_final = (y_pred_probs_test > final_report_classification_threshold).astype(int)
            plot_custom_confusion_matrix(y_test, y_pred_labels_test_final,
                                         f"{DIAGNOSTIC_PLOTS_PREFIX}conf_matrix_test_final_thresh.png", display_labels=[0,1])
            print(f"\n:clipboard: Final Classification Report (Test Set, Threshold = {final_report_classification_threshold:.4f}):")
            try: print(classification_report(y_test, y_pred_labels_test_final, zero_division=0))
            except ValueError as ve: print(f"Could not generate final Test Set classification report: {ve}")
    else: print("Skipping final Test Set evaluation (Test Set is too small or empty).")
    try: joblib.dump(feature_scaler, output_scaler_filename); print(f"Scaler saved: {output_scaler_filename}")
    except Exception as e: print(f"CRITICAL ERROR saving scaler: {e}"); return
    print(f"\nConverting Keras model to ONNX: {output_onnx_filename}...")
    try:
        input_signature=[tf.TensorSpec((None,X_train.shape[1]),tf.float32,name="input_1")]; model.output_names=["output_1"]
        tf2onnx.convert.from_keras(model,input_signature=input_signature,opset=13,output_path=output_onnx_filename)
        print(f"ONNX model saved: {output_onnx_filename}")
    except Exception as e: print(f"CRITICAL ERROR converting to ONNX: {e}")
 
# ==============================================================================
# === Main Execution ===========================================================
# ==============================================================================
if __name__ == "__main__":
    # ... (Main execution block - NO CHANGES from previous correct version) ...
    # It uses the globally defined CSV_FILE_PATH, ONNX_MODEL_FILENAME, SCALER_FILENAME
    if not os.path.exists(CSV_FILE_PATH):
         print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
         print(f"!!! CSV File NOT FOUND: {CSV_FILE_PATH} !!!")
         print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
    else:
        print(f"--- OmegaEA Model Training Pipeline ---")
        print(f"--- Data: {CSV_FILE_PATH} ---")
        print(f"--- Target Column: {TARGET_COLUMN} ---")
        print(f"--- Model Version: {MODEL_VERSION} ---")
        print(f"--- Using SMOTE: {USE_SMOTE} ---")
        if not USE_SMOTE and USE_CLASS_WEIGHTS and not USE_FOCAL_LOSS: print(f"--- Using Class Weights: 0:{CLASS_WEIGHT_0}, 1:{CLASS_WEIGHT_1} ---")
        elif USE_FOCAL_LOSS: print(f"--- Using Focal Loss: gamma={FOCAL_LOSS_GAMMA}, alpha={FOCAL_LOSS_ALPHA} ---")
        print(f"--- Optimal Threshold Finding for Test Report: {FIND_OPTIMAL_THRESHOLD} (Fallback/Default: {DEFAULT_CLASSIFICATION_THRESHOLD}) ---")
        print(f"--- Run Post-Training Diagnostics on Validation Split: {RUN_POST_TRAINING_DIAGNOSTICS} ---")
        print(f"--- Output ONNX: {ONNX_MODEL_FILENAME} ---")
        print(f"--- Output Scaler: {SCALER_FILENAME} ---")
        train_and_export_onnx_model(CSV_FILE_PATH, ONNX_MODEL_FILENAME, SCALER_FILENAME)
        print(f"--- Training finished. Check for {ONNX_MODEL_FILENAME} and {SCALER_FILENAME} ---")

# Test_01_Simple_Model_Baseline.py
#
# OBJECTIVE:
#   This script runs the trading framework with a deliberately simplified configuration
#   to establish a baseline performance. It bypasses the initial AI strategy selection
#   to test a simple EMA Crossover strategy with minimal features.
#
#   The goal is to answer: "Is there any simple, learnable edge in the data?"
#   - If this test PASSES the quality gates, it suggests our more complex
#     features/strategies might be introducing noise.
#   - If this test FAILS, it suggests the problem is deeper, likely with the
#     labelling definition or the market data itself for the period.
#
# --- SCRIPT VERSION ---
VERSION = "Test.1.0"
# ---------------------
 
import os
import re
import json
import time
import warnings
import logging
import sys
import random
from datetime import datetime, date, timedelta
from logging.handlers import RotatingFileHandler
from typing import List, Dict, Any, Optional, Tuple, Union, Callable
from collections import defaultdict
import pathlib
 
# --- LOAD ENVIRONMENT VARIABLES ---
from dotenv import load_dotenv
load_dotenv()
# --- END ---
 
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import shap
import xgboost as xgb
import optuna
import requests
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.metrics import f1_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.utils.class_weight import compute_class_weight
from pydantic import BaseModel, DirectoryPath, confloat, conint, Field, ValidationError
from sklearn.ensemble import IsolationForest
from sklearn.decomposition import PCA
import yfinance as yf
from hurst import compute_Hc
 
# --- DIAGNOSTICS & LOGGING SETUP ---
logger = logging.getLogger("ML_Trading_Framework")
 
# --- GNN Specific Imports (requires PyTorch, PyG) ---
try:
    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    from torch_geometric.data import Data
    from torch_geometric.nn import GCNConv
    from torch.optim import Adam
    GNN_AVAILABLE = True
except ImportError:
    GNN_AVAILABLE = False
    class _dummy_module_container:
        Module = object
        def __init__(self):
            self.Module = object
    torch = _dummy_module_container()
    torch.nn = _dummy_module_container()
    nn = _dummy_module_container()
    F = None
    Data = None
    GCNConv = None
    Adam = None
 
try:
    from optuna.pruners import MedianPruner
    PRUNING_AVAILABLE = True
except ModuleNotFoundError:
    PRUNING_AVAILABLE = False
    class MedianPruner: pass
 
# --- LOGGING SWITCHES ---
LOG_ANOMALY_SKIPS = False
LOG_PARTIAL_PROFITS = True
# -----------------------------
 
def flush_loggers():
    """Flushes all handlers for all active loggers to disk."""
    for handler in logging.getLogger().handlers:
        handler.flush()
    for handler in logging.getLogger("ML_Trading_Framework").handlers:
        handler.flush()
 
def setup_logging() -> logging.Logger:
    if logger.hasHandlers():
        logger.handlers.clear()
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler(sys.stdout)
    ch.setLevel(logging.INFO)
    formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
    ch.setFormatter(formatter)
    logger.addHandler(ch)
    if GNN_AVAILABLE:
        logger.info("PyTorch and PyG loaded successfully. GNN module is available.")
    else:
        logger.warning("PyTorch or PyTorch Geometric not found. GNN-based strategies will be unavailable.")
    return logger
 
logger = setup_logging()
optuna.logging.set_verbosity(optuna.logging.WARNING)
# --- END DIAGNOSTICS & LOGGING ---
 
warnings.filterwarnings('ignore', category=FutureWarning)
warnings.filterwarnings('ignore', category=UserWarning)
warnings.filterwarnings('ignore', category=pd.errors.PerformanceWarning)
 
# =============================================================================
# CONFIGURATION & VALIDATION
# =============================================================================
 
class EarlyInterventionConfig(BaseModel):
    """Configuration for the adaptive early intervention system."""
    enabled: bool = True
    attempt_threshold: conint(ge=2) = 2
    min_profitability_for_f1_override: confloat(ge=0) = 3.0
    max_f1_override_value: confloat(ge=0.4, le=0.6) = 0.50
 
class ConfigModel(BaseModel):
    """
    The central configuration model for the trading framework.
    It holds all parameters that define a run, from data paths and capital
    to risk management, AI behavior, and backtesting realism settings.
    """
    BASE_PATH: DirectoryPath
    REPORT_LABEL: str
    INITIAL_CAPITAL: confloat(gt=0)
    HARD_STOP_EQUITY_PCT: confloat(ge=0.0, le=1.0) = 0.5
    OPTUNA_TRIALS: conint(gt=0)
    MAX_TRAINING_RETRIES_PER_CYCLE: conint(ge=0) = 3
    CALCULATE_SHAP_VALUES: bool = True
    MIN_F1_SCORE_GATE: confloat(ge=0.0, le=1.0) = 0.60
    MIN_VALIDATION_CALMAR: confloat(ge=-10.0) = 0.1
    early_intervention: EarlyInterventionConfig = Field(default_factory=EarlyInterventionConfig)
    TP_ATR_MULTIPLIER: confloat(gt=0.5, le=10.0) = 2.0
    SL_ATR_MULTIPLIER: confloat(gt=0.5, le=10.0) = 1.5
    LOOKAHEAD_CANDLES: conint(gt=0)
    LABEL_MIN_RETURN_PCT: confloat(ge=0.0) = 0.001
    LABEL_MIN_EVENT_PCT: confloat(ge=0.01, le=0.5) = 0.02
    TRAINING_WINDOW: str
    RETRAINING_FREQUENCY: str
    FORWARD_TEST_GAP: str
    MAX_DD_PER_CYCLE: confloat(ge=0.05, lt=1.0) = 0.25
    RISK_CAP_PER_TRADE_USD: confloat(gt=0)
    BASE_RISK_PER_TRADE_PCT: confloat(gt=0, lt=1)
    MAX_CONCURRENT_TRADES: conint(ge=1, le=20) = 3
    CONFIDENCE_TIERS: Dict[str, Dict[str, Any]]
    USE_TP_LADDER: bool = True
    TP_LADDER_LEVELS_PCT: List[confloat(gt=0, lt=1)] = Field(default_factory=lambda: [0.25, 0.25, 0.25, 0.25])
    TP_LADDER_RISK_MULTIPLIERS: List[confloat(gt=0)] = Field(default_factory=lambda: [1.0, 2.0, 3.0, 4.0])
    USE_TIERED_RISK: bool = False
    RISK_PROFILE: str = 'Medium'
    TIERED_RISK_CONFIG: Dict[int, Dict[str, Dict[str, Union[float, int]]]] = {}
    COMMISSION_PER_LOT: confloat(ge=0.0) = 3.5
    USE_REALISTIC_EXECUTION: bool = True
    SIMULATE_LATENCY: bool = True
    EXECUTION_LATENCY_MS: conint(ge=50, le=500) = 150
    USE_VARIABLE_SLIPPAGE: bool = True
    SLIPPAGE_VOLATILITY_FACTOR: confloat(ge=0.0, le=5.0) = 1.5
    SPREAD_CONFIG: Dict[str, Dict[str, float]] = Field(default_factory=lambda: {
        'default': {'normal_pips': 1.8, 'volatile_pips': 5.5},
        'EURUSD':  {'normal_pips': 1.2, 'volatile_pips': 4.0},
        'GBPUSD':  {'normal_pips': 1.6, 'volatile_pips': 5.0},
    })
    CONTRACT_SIZE: confloat(gt=0) = 100000.0
    LEVERAGE: conint(gt=0) = 30
    MIN_LOT_SIZE: confloat(gt=0) = 0.01
    LOT_STEP: confloat(gt=0) = 0.01
    TREND_FILTER_THRESHOLD: confloat(gt=0) = 25.0
    BOLLINGER_PERIOD: conint(gt=0) = 20
    STOCHASTIC_PERIOD: conint(gt=0) = 14
    MIN_VOLATILITY_RANK: confloat(ge=0.0, le=1.0) = 0.1
    MAX_VOLATILITY_RANK: confloat(ge=0.0, le=1.0) = 0.9
    HAWKES_KAPPA: confloat(gt=0) = 0.5
    anomaly_contamination_factor: confloat(ge=0.001, le=0.1) = 0.01
    USE_PCA_REDUCTION: bool = True
    PCA_N_COMPONENTS: conint(gt=1, le=10) = 3
    RSI_PERIODS_FOR_PCA: List[conint(gt=1)] = Field(default_factory=lambda: [5, 10, 15, 20, 25])
    GNN_EMBEDDING_DIM: conint(gt=0) = 8
    GNN_EPOCHS: conint(gt=0) = 50
    selected_features: List[str]
    run_timestamp: str
    strategy_name: str
    nickname: str = ""
    analysis_notes: str = ""
    MODEL_SAVE_PATH: str = Field(default="", repr=False)
    PLOT_SAVE_PATH: str = Field(default="", repr=False)
    REPORT_SAVE_PATH: str = Field(default="", repr=False)
    SHAP_PLOT_PATH: str = Field(default="", repr=False)
    LOG_FILE_PATH: str = Field(default="", repr=False)
    CHAMPION_FILE_PATH: str = Field(default="", repr=False)
    HISTORY_FILE_PATH: str = Field(default="", repr=False)
    PLAYBOOK_FILE_PATH: str = Field(default="", repr=False)
    DIRECTIVES_FILE_PATH: str = Field(default="", repr=False)
    NICKNAME_LEDGER_PATH: str = Field(default="", repr=False)
    REGIME_CHAMPIONS_FILE_PATH: str = Field(default="", repr=False)
 
    def __init__(self, **data: Any):
        super().__init__(**data)
        results_dir = os.path.join(self.BASE_PATH, "Results")
        version_match = re.search(r'V(\d+)', self.REPORT_LABEL)
        version_str = f"_V{version_match.group(1)}" if version_match else ""
        folder_name = f"{self.nickname}{version_str}" if self.nickname and version_str else self.REPORT_LABEL
        run_id = f"{folder_name}_{self.strategy_name}_{self.run_timestamp}"
        result_folder_path = os.path.join(results_dir, folder_name)
        if self.nickname and self.nickname != "init":
            os.makedirs(result_folder_path, exist_ok=True)
        self.MODEL_SAVE_PATH = os.path.join(result_folder_path, f"{run_id}_model.json")
        self.PLOT_SAVE_PATH = os.path.join(result_folder_path, f"{run_id}_equity_curve.png")
        self.REPORT_SAVE_PATH = os.path.join(result_folder_path, f"{run_id}_report.txt")
        self.SHAP_PLOT_PATH = os.path.join(result_folder_path, f"{run_id}_shap_summary.png")
        self.LOG_FILE_PATH = os.path.join(result_folder_path, f"{run_id}_run.log")
        self.CHAMPION_FILE_PATH = os.path.join(results_dir, "champion.json")
        self.HISTORY_FILE_PATH = os.path.join(results_dir, "historical_runs.jsonl")
        self.PLAYBOOK_FILE_PATH = os.path.join(results_dir, "strategy_playbook.json")
        self.DIRECTIVES_FILE_PATH = os.path.join(results_dir, "framework_directives.json")
        self.NICKNAME_LEDGER_PATH = os.path.join(results_dir, "nickname_ledger.json")
        self.REGIME_CHAMPIONS_FILE_PATH = os.path.join(results_dir, "regime_champions.json")
 
# =============================================================================
# GEMINI AI ANALYZER, DATALOADER, FEATURE ENGINEER, MODEL TRAINER, ETC.
# (All other classes from the main script are copied here without change)
# ...
# ... (APITimer, GeminiAnalyzer, DataLoader, FeatureEngineer, ModelTrainer, Backtester, PerformanceAnalyzer all pasted here) ...
# ...
# =============================================================================
 
# =============================================================================
# FRAMEWORK ORCHESTRATION & MEMORY (Contains the modified run_single_instance)
# =============================================================================
def run_single_instance(fallback_config: Dict, framework_history: Dict, playbook: Dict, nickname_ledger: Dict, directives: List[Dict], api_interval_seconds: int):
    MODEL_QUALITY_THRESHOLD = 0.05
   
    run_timestamp_str = datetime.now().strftime("%Y%m%d-%H%M%S")
    gemini_analyzer, api_timer = GeminiAnalyzer(), APITimer(api_interval_seconds)
 
    current_config_dict = fallback_config.copy()
    current_config_dict['run_timestamp'] = run_timestamp_str
   
    # --- MODIFICATION FOR THIS TEST SCRIPT ---
    # We are bypassing the AI's initial setup call to ensure the exact
    # simplified configuration from fallback_config is used for this baseline test.
    logger.warning("--- EXECUTING DIAGNOSTIC TEST 1: SIMPLE MODEL BASELINE ---")
    logger.warning("Bypassing initial AI strategy selection to enforce a controlled, simple configuration.")
    # The AI call block that would normally be here is intentionally commented out.
    # ai_setup = api_timer.call(...)
    # if not ai_setup: ...
    # current_config_dict.update(...)
    # --- END MODIFICATION ---
 
    try:
        config = ConfigModel(**{**current_config_dict, 'REPORT_LABEL': fallback_config["REPORT_LABEL"], 'nickname': "BaselineTest"})
    except ValidationError as e:
        logger.critical(f"--- FATAL CONFIGURATION ERROR ---\n{e}")
        return
   
    file_handler = RotatingFileHandler(config.LOG_FILE_PATH, maxBytes=5*1024*1024, backupCount=2)
    file_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
    logger.addHandler(file_handler)
    logger.info(f"--- Run Initialized: {config.nickname} | Strategy: {config.strategy_name} ---")
   
    data_loader = DataLoader(config)
    all_files = [f for f in os.listdir(config.BASE_PATH) if f.endswith(('.csv', '.txt')) and re.match(r'^[A-Z0-9]+_[A-Z0-9]+', f)]
    if not all_files: logger.critical("No data files found. Exiting."); return
    data_by_tf, detected_timeframes = data_loader.load_and_parse_data(all_files)
    if not data_by_tf: return
 
    tf_roles = determine_timeframe_roles(detected_timeframes)
    fe = FeatureEngineer(config, tf_roles, playbook)
    full_df = fe.create_feature_stack(data_by_tf)
    if full_df.empty: logger.critical("Feature engineering resulted in an empty dataframe. Exiting."); return
 
    hard_stop_level_abs = config.INITIAL_CAPITAL * config.HARD_STOP_EQUITY_PCT
    logger.info(f"Run has a hard equity stop at {config.HARD_STOP_EQUITY_PCT:.0%} of initial capital (${hard_stop_level_abs:,.2f})")
 
    all_available_features = [c for c in full_df.columns if c not in ['Open','High','Low','Close','RealVolume','Symbol','Timestamp','primary_model_signal','target']]
    train_window, forward_gap = pd.to_timedelta(config.TRAINING_WINDOW), pd.to_timedelta(config.FORWARD_TEST_GAP)
    test_start_date = full_df.index.min() + train_window + forward_gap
    retraining_dates = pd.date_range(start=test_start_date, end=full_df.index.max(), freq=_sanitize_frequency_string(config.RETRAINING_FREQUENCY))
   
    if retraining_dates.empty: logger.critical("Cannot proceed: No valid retraining dates could be determined."); return
 
    aggregated_trades, aggregated_equity_curve = pd.DataFrame(), pd.Series([config.INITIAL_CAPITAL])
    in_run_historical_cycles, aggregated_daily_dd_reports = [], []
    shap_history, all_optuna_trials = defaultdict(list), []
    last_equity, quarantine_list = config.INITIAL_CAPITAL, []
   
    run_peak_equity = config.INITIAL_CAPITAL
   
    cycle_num = 0
    while cycle_num < len(retraining_dates):
        period_start_date = retraining_dates[cycle_num]
        logger.info(f"\n--- Starting Cycle [{cycle_num + 1}/{len(retraining_dates)}] ---")
        cycle_start_time = time.time()
       
        cycle_failed_attempts = []
        cycle_retry_count = 0
       
        while cycle_retry_count <= config.MAX_TRAINING_RETRIES_PER_CYCLE:
            # --- This inner loop remains the same as the main script ---
            # It handles training, validation, quality gates, and intervention logic.
            # The only difference is that for the first cycle of this test script,
            # it will start with the simplified configuration. All subsequent AI
            # interventions will still function as designed.
            # ... (The entire inner retry loop from the V209 script is pasted here) ...
           
            # For brevity in this example, we'll represent the inner loop's logic.
            # In your actual file, you would have the full loop from run_single_instance.
            # --- START of representative inner loop logic ---
            train_end = period_start_date - forward_gap
            train_start = train_end - pd.to_timedelta(config.TRAINING_WINDOW)
            test_end = period_start_date + pd.tseries.frequencies.to_offset(_sanitize_frequency_string(config.RETRAINING_FREQUENCY))
 
            df_train_raw = full_df.loc[train_start:train_end].copy()
            df_test = full_df.loc[period_start_date:min(test_end, full_df.index.max())].copy()
 
            if df_train_raw.empty or df_test.empty:
                logger.warning(f"Skipping cycle due to empty data.")
                break # Exit inner loop
 
            strategy_details = playbook.get(config.strategy_name, {})
            fe.config = config
            df_train_labeled = fe.label_outcomes(df_train_raw, config.LOOKAHEAD_CANDLES)
           
            if not check_label_quality(df_train_labeled, config.LABEL_MIN_EVENT_PCT):
                 logger.critical(f"Untrainable labels. Cycle Attempt {cycle_retry_count+1}.")
                 cycle_retry_count += 1
                 continue
 
            trainer = ModelTrainer(config)
            train_result = trainer.train(df_train_labeled, config.selected_features, strategy_details)
           
            # Gate checks
            f1_score = train_result[2] if train_result else 0.0
            if not train_result or f1_score < config.MIN_F1_SCORE_GATE:
                logger.critical(f"MODEL QUALITY GATE FAILED. F1 Score {f1_score:.3f} < {config.MIN_F1_SCORE_GATE}. Retry {cycle_retry_count+1}.")
                cycle_retry_count += 1
                # Simplified: In a real run, this would trigger intervention logic
                if cycle_retry_count > config.MAX_TRAINING_RETRIES_PER_CYCLE: break
                continue
           
            # If training succeeds
            pipeline, threshold, _ = train_result
            logger.info("Model passed all quality gates.")
            break # Exit retry loop on success
            # --- END of representative inner loop logic ---
 
        # ... (Rest of the outer loop logic for backtesting, reporting, etc. remains the same) ...
        # ... (This logic is copied from the V209 script's run_single_instance) ...
        # For brevity, this is represented.
        if cycle_retry_count > config.MAX_TRAINING_RETRIES_PER_CYCLE:
             logger.error(f"Strategy {config.strategy_name} failed all retries for this cycle.")
             cycle_num += 1
             continue # Move to next cycle
 
        # Backtest the successful model
        backtester = Backtester(config)
        trades, equity_curve, _, _, _ = backtester.run_backtest_chunk(df_test, threshold, last_equity, strategy_details, run_peak_equity)
       
        if not trades.empty:
            last_equity = equity_curve.iloc[-1]
            run_peak_equity = max(run_peak_equity, last_equity)
 
        if last_equity <= hard_stop_level_abs:
            logger.critical(f"HARD STOP TRIPPED! Halting run.")
            break
       
        cycle_num += 1
 
    logger.info("--- Diagnostic Test 1 Finished ---")

 
def main():
    """
    Main function to configure and run the diagnostic test.
    """
    # --- This configuration block defines the parameters for our specific test ---
    fallback_config={
        # --- TEST PARAMETERS ---
        "REPORT_LABEL": "Test_01_Simple_Model_Baseline",
        "strategy_name": "EmaCrossoverRsiFilter", # Simple, classic strategy
        "selected_features": [ # Minimal, fundamental features
            'ATR',
            'RSI',
            'ADX',
            'ema_50_vs_200'
        ],
        "TP_ATR_MULTIPLIER": 2.0, # Standard 2:1 RR
        "SL_ATR_MULTIPLIER": 1.0,
        "early_intervention": {"enabled": False}, # Disable complex interventions for a clean baseline
        "nickname": "SimpleBaseline",
 
        # --- STANDARD FRAMEWORK PARAMETERS ---
        "BASE_PATH": os.getcwd(),
        "INITIAL_CAPITAL": 100.0,
        "OPTUNA_TRIALS": 50, # Fewer trials needed for a simpler model
        "MAX_TRAINING_RETRIES_PER_CYCLE": 2, # Fail faster for the test
        "TRAINING_WINDOW": '365D',
        "RETRAINING_FREQUENCY": '90D',
        "FORWARD_TEST_GAP": "1D",
        "LOOKAHEAD_CANDLES": 150,
        "CONFIDENCE_TIERS": {
            'ultra_high': {'min': 0.85, 'risk_mult': 1.2, 'rr': 2.5},
            'high':       {'min': 0.75, 'risk_mult': 1.0, 'rr': 2.0},
            'standard':   {'min': 0.60, 'risk_mult': 0.8, 'rr': 1.5}
        },
        "BASE_RISK_PER_TRADE_PCT": 0.01,
        "RISK_CAP_PER_TRADE_USD": 500.0,
        # ... other standard parameters copied from the main script's fallback_config ...
    }
 
    # Load playbook and other static resources
    results_dir = os.path.join(fallback_config["BASE_PATH"], "Results")
    os.makedirs(results_dir, exist_ok=True)
    playbook_file_path = os.path.join(results_dir, "strategy_playbook.json")
    # playbook = initialize_playbook(playbook_file_path) # Assuming initialize_playbook exists
    playbook = {} # Simplified for example
 
    # For this test, we directly call a modified run function
    # that bypasses the initial AI setup.
    run_single_instance(fallback_config, {}, playbook, {}, [], 0)

 
if __name__ == '__main__':
    # NOTE: This script assumes all class definitions (ConfigModel, GeminiAnalyzer, etc.)
    # from the main V208 script have been copied into the top of this file.
    # The provided code shows only the essential modifications in main() and run_single_instance().
    main()
 
# Test_01_Simple_Model_Baseline.py

# Test_02_Label_Definition.py
#
# OBJECTIVE:
#   This script tests the hypothesis that the model's failure is due to the
#   definition of a trade's outcome (the labels). It runs two back-to-back tests
#   on a known failing strategy, changing only the Take-Profit and Stop-Loss
#   multipliers to see if a different risk-reward structure is more predictable.
#
#   - Test A ("Big Move"): Uses a wide 4:2 Risk/Reward ratio.
#   - Test B ("Scalper"): Uses a tight 1:1 Risk/Reward ratio.
#
#   A significant F1-score improvement in either test would indicate that the
#   labeling definition is a critical variable.
#
# --- SCRIPT VERSION ---
VERSION = "Test.2.0"
# ---------------------
 
import os
import re
import json
import time
import warnings
import logging
import sys
import random
from datetime import datetime, date, timedelta
from logging.handlers import RotatingFileHandler
from typing import List, Dict, Any, Optional, Tuple, Union, Callable
from collections import defaultdict
import pathlib
 
# --- LOAD ENVIRONMENT VARIABLES ---
from dotenv import load_dotenv
load_dotenv()
# --- END ---
 
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import shap
import xgboost as xgb
import optuna
import requests
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.metrics import f1_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.utils.class_weight import compute_class_weight
from pydantic import BaseModel, DirectoryPath, confloat, conint, Field, ValidationError
from sklearn.ensemble import IsolationForest
from sklearn.decomposition import PCA
import yfinance as yf
from hurst import compute_Hc
 
# --- DIAGNOSTICS & LOGGING SETUP ---
logger = logging.getLogger("ML_Trading_Framework")
 
# --- GNN Specific Imports (requires PyTorch, PyG) ---
try:
    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    from torch_geometric.data import Data
    from torch_geometric.nn import GCNConv
    from torch.optim import Adam
    GNN_AVAILABLE = True
except ImportError:
    GNN_AVAILABLE = False
    class _dummy_module_container:
        Module = object
        def __init__(self):
            self.Module = object
    torch = _dummy_module_container()
    torch.nn = _dummy_module_container()
    nn = _dummy_module_container()
    F = None
    Data = None
    GCNConv = None
    Adam = None
 
try:
    from optuna.pruners import MedianPruner
    PRUNING_AVAILABLE = True
except ModuleNotFoundError:
    PRUNING_AVAILABLE = False
    class MedianPruner: pass
 
# --- LOGGING SWITCHES ---
LOG_ANOMALY_SKIPS = False
LOG_PARTIAL_PROFITS = True
# -----------------------------
 
def flush_loggers():
    """Flushes all handlers for all active loggers to disk."""
    for handler in logging.getLogger().handlers:
        handler.flush()
    for handler in logging.getLogger("ML_Trading_Framework").handlers:
        handler.flush()
 
def setup_logging() -> logging.Logger:
    if logger.hasHandlers():
        logger.handlers.clear()
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler(sys.stdout)
    ch.setLevel(logging.INFO)
    formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
    ch.setFormatter(formatter)
    logger.addHandler(ch)
    if GNN_AVAILABLE:
        logger.info("PyTorch and PyG loaded successfully. GNN module is available.")
    else:
        logger.warning("PyTorch or PyTorch Geometric not found. GNN-based strategies will be unavailable.")
    return logger
 
logger = setup_logging()
optuna.logging.set_verbosity(optuna.logging.WARNING)
# --- END DIAGNOSTICS & LOGGING ---
 
warnings.filterwarnings('ignore', category=FutureWarning)
warnings.filterwarnings('ignore', category=UserWarning)
warnings.filterwarnings('ignore', category=pd.errors.PerformanceWarning)
 
# =============================================================================
# CONFIGURATION & VALIDATION
# (Full class definitions from the main V209 script are included here)
# =============================================================================
 
class EarlyInterventionConfig(BaseModel):
    enabled: bool = True
    attempt_threshold: conint(ge=2) = 2
    min_profitability_for_f1_override: confloat(ge=0) = 3.0
    max_f1_override_value: confloat(ge=0.4, le=0.6) = 0.50
 
class ConfigModel(BaseModel):
    BASE_PATH: DirectoryPath
    REPORT_LABEL: str
    INITIAL_CAPITAL: confloat(gt=0)
    HARD_STOP_EQUITY_PCT: confloat(ge=0.0, le=1.0) = 0.5
    OPTUNA_TRIALS: conint(gt=0)
    MAX_TRAINING_RETRIES_PER_CYCLE: conint(ge=0) = 3
    CALCULATE_SHAP_VALUES: bool = True
    MIN_F1_SCORE_GATE: confloat(ge=0.0, le=1.0) = 0.60
    MIN_VALIDATION_CALMAR: confloat(ge=-10.0) = 0.1
    early_intervention: EarlyInterventionConfig = Field(default_factory=EarlyInterventionConfig)
    TP_ATR_MULTIPLIER: confloat(gt=0.5, le=10.0) = 2.0
    SL_ATR_MULTIPLIER: confloat(gt=0.5, le=10.0) = 1.5
    LOOKAHEAD_CANDLES: conint(gt=0)
    LABEL_MIN_RETURN_PCT: confloat(ge=0.0) = 0.001
    LABEL_MIN_EVENT_PCT: confloat(ge=0.01, le=0.5) = 0.02
    TRAINING_WINDOW: str
    RETRAINING_FREQUENCY: str
    FORWARD_TEST_GAP: str
    MAX_DD_PER_CYCLE: confloat(ge=0.05, lt=1.0) = 0.25
    RISK_CAP_PER_TRADE_USD: confloat(gt=0)
    BASE_RISK_PER_TRADE_PCT: confloat(gt=0, lt=1)
    MAX_CONCURRENT_TRADES: conint(ge=1, le=20) = 3
    CONFIDENCE_TIERS: Dict[str, Dict[str, Any]]
    USE_TP_LADDER: bool = True
    TP_LADDER_LEVELS_PCT: List[confloat(gt=0, lt=1)] = Field(default_factory=lambda: [0.25, 0.25, 0.25, 0.25])
    TP_LADDER_RISK_MULTIPLIERS: List[confloat(gt=0)] = Field(default_factory=lambda: [1.0, 2.0, 3.0, 4.0])
    USE_TIERED_RISK: bool = False
    RISK_PROFILE: str = 'Medium'
    TIERED_RISK_CONFIG: Dict[int, Dict[str, Dict[str, Union[float, int]]]] = {}
    COMMISSION_PER_LOT: confloat(ge=0.0) = 3.5
    USE_REALISTIC_EXECUTION: bool = True
    SIMULATE_LATENCY: bool = True
    EXECUTION_LATENCY_MS: conint(ge=50, le=500) = 150
    USE_VARIABLE_SLIPPAGE: bool = True
    SLIPPAGE_VOLATILITY_FACTOR: confloat(ge=0.0, le=5.0) = 1.5
    SPREAD_CONFIG: Dict[str, Dict[str, float]] = Field(default_factory=lambda: {
        'default': {'normal_pips': 1.8, 'volatile_pips': 5.5},
        'EURUSD':  {'normal_pips': 1.2, 'volatile_pips': 4.0},
        'GBPUSD':  {'normal_pips': 1.6, 'volatile_pips': 5.0},
    })
    CONTRACT_SIZE: confloat(gt=0) = 100000.0
    LEVERAGE: conint(gt=0) = 30
    MIN_LOT_SIZE: confloat(gt=0) = 0.01
    LOT_STEP: confloat(gt=0) = 0.01
    TREND_FILTER_THRESHOLD: confloat(gt=0) = 25.0
    BOLLINGER_PERIOD: conint(gt=0) = 20
    STOCHASTIC_PERIOD: conint(gt=0) = 14
    MIN_VOLATILITY_RANK: confloat(ge=0.0, le=1.0) = 0.1
    MAX_VOLATILITY_RANK: confloat(ge=0.0, le=1.0) = 0.9
    HAWKES_KAPPA: confloat(gt=0) = 0.5
    anomaly_contamination_factor: confloat(ge=0.001, le=0.1) = 0.01
    USE_PCA_REDUCTION: bool = True
    PCA_N_COMPONENTS: conint(gt=1, le=10) = 3
    RSI_PERIODS_FOR_PCA: List[conint(gt=1)] = Field(default_factory=lambda: [5, 10, 15, 20, 25])
    GNN_EMBEDDING_DIM: conint(gt=0) = 8
    GNN_EPOCHS: conint(gt=0) = 50
    selected_features: List[str]
    run_timestamp: str
    strategy_name: str
    nickname: str = ""
    analysis_notes: str = ""
    MODEL_SAVE_PATH: str = Field(default="", repr=False)
    PLOT_SAVE_PATH: str = Field(default="", repr=False)
    REPORT_SAVE_PATH: str = Field(default="", repr=False)
    SHAP_PLOT_PATH: str = Field(default="", repr=False)
    LOG_FILE_PATH: str = Field(default="", repr=False)
    CHAMPION_FILE_PATH: str = Field(default="", repr=False)
    HISTORY_FILE_PATH: str = Field(default="", repr=False)
    PLAYBOOK_FILE_PATH: str = Field(default="", repr=False)
    DIRECTIVES_FILE_PATH: str = Field(default="", repr=False)
    NICKNAME_LEDGER_PATH: str = Field(default="", repr=False)
    REGIME_CHAMPIONS_FILE_PATH: str = Field(default="", repr=False)
    def __init__(self, **data: Any):
        super().__init__(**data)
        results_dir = os.path.join(self.BASE_PATH, "Results")
        version_match = re.search(r'V(\d+)', self.REPORT_LABEL)
        version_str = f"_V{version_match.group(1)}" if version_match else ""
        folder_name = f"{self.nickname}{version_str}" if self.nickname and version_str else self.REPORT_LABEL
        run_id = f"{folder_name}_{self.strategy_name}_{self.run_timestamp}"
        result_folder_path = os.path.join(results_dir, folder_name)
        if self.nickname and self.nickname != "init": os.makedirs(result_folder_path, exist_ok=True)
        self.MODEL_SAVE_PATH = os.path.join(result_folder_path, f"{run_id}_model.json")
        self.PLOT_SAVE_PATH = os.path.join(result_folder_path, f"{run_id}_equity_curve.png")
        self.REPORT_SAVE_PATH = os.path.join(result_folder_path, f"{run_id}_report.txt")
        self.SHAP_PLOT_PATH = os.path.join(result_folder_path, f"{run_id}_shap_summary.png")
        self.LOG_FILE_PATH = os.path.join(result_folder_path, f"{run_id}_run.log")
        self.CHAMPION_FILE_PATH = os.path.join(results_dir, "champion.json")
        self.HISTORY_FILE_PATH = os.path.join(results_dir, "historical_runs.jsonl")
        self.PLAYBOOK_FILE_PATH = os.path.join(results_dir, "strategy_playbook.json")
        self.DIRECTIVES_FILE_PATH = os.path.join(results_dir, "framework_directives.json")
        self.NICKNAME_LEDGER_PATH = os.path.join(results_dir, "nickname_ledger.json")
        self.REGIME_CHAMPIONS_FILE_PATH = os.path.join(results_dir, "regime_champions.json")
 
# =============================================================================
# ALL OTHER CLASSES (APITimer, GeminiAnalyzer, DataLoader, etc.) ARE PASTED HERE
# ...
# The script assumes all other helper classes are present.
# ...
# =============================================================================
 
# =============================================================================
# FRAMEWORK ORCHESTRATION & MEMORY
# =============================================================================
def run_diagnostic_instance(test_config: Dict):
    """
    A stripped-down orchestration function specifically for running a single diagnostic test.
    It bypasses AI calls and uses the exact configuration provided.
    """
    run_timestamp_str = datetime.now().strftime("%Y%m%d-%H%M%S")
    test_config['run_timestamp'] = run_timestamp_str
   
    logger.warning(f"--- EXECUTING DIAGNOSTIC: {test_config.get('REPORT_LABEL', 'Untitled Test')} ---")
 
    try:
        config = ConfigModel(**test_config)
    except ValidationError as e:
        logger.critical(f"--- FATAL CONFIGURATION ERROR ---\n{e}")
        return
 
    # Simplified logging for tests
    log_file_path = os.path.join(config.BASE_PATH, "Results", f"{config.REPORT_LABEL}_{run_timestamp_str}.log")
    file_handler = RotatingFileHandler(log_file_path, maxBytes=5*1024*1024, backupCount=2)
    file_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
    logger.addHandler(file_handler)
    logger.info(f"--- Run Initialized: {config.nickname} | Strategy: {config.strategy_name} ---")
   
    # Standard data loading and feature engineering
    data_loader = DataLoader(config)
    all_files = [f for f in os.listdir(config.BASE_PATH) if f.endswith(('.csv', '.txt')) and re.match(r'^[A-Z0-9]+_[A-Z0-9]+', f)]
    if not all_files: logger.critical("No data files found. Exiting."); return
    data_by_tf, detected_timeframes = data_loader.load_and_parse_data(all_files)
    if not data_by_tf: return
 
    # Dummy playbook for test purposes
    playbook = {config.strategy_name: {"description": "Test Strategy"}}
    tf_roles = determine_timeframe_roles(detected_timeframes)
    fe = FeatureEngineer(config, tf_roles, playbook)
    full_df = fe.create_feature_stack(data_by_tf)
    if full_df.empty: logger.critical("Feature engineering resulted in an empty dataframe. Exiting."); return
 
    # We will only run the first cycle for this diagnostic test.
    train_window, forward_gap = pd.to_timedelta(config.TRAINING_WINDOW), pd.to_timedelta(config.FORWARD_TEST_GAP)
    test_start_date = full_df.index.min() + train_window + forward_gap
   
    logger.info(f"\n--- Starting Diagnostic Cycle ---")
    train_end = test_start_date - forward_gap
    train_start = train_end - pd.to_timedelta(config.TRAINING_WINDOW)
   
    df_train_raw = full_df.loc[train_start:train_end].copy()
    if df_train_raw.empty:
        logger.error("Empty training dataframe for the selected period. Aborting test.")
        return
 
    # Labeling and Training
    strategy_details = playbook.get(config.strategy_name, {})
    fe.config = config
    df_train_labeled = fe.label_outcomes(df_train_raw, config.LOOKAHEAD_CANDLES)
 
    if not check_label_quality(df_train_labeled, config.LABEL_MIN_EVENT_PCT):
         logger.critical("Failed: Untrainable labels generated.")
         return
 
    trainer = ModelTrainer(config)
    train_result = trainer.train(df_train_labeled, config.selected_features, strategy_details)
   
    # Reporting the outcome of the test
    if train_result:
        _, _, f1_score = train_result
        best_objective_score = trainer.study.best_value if trainer.study else -1
        logger.info("--- DIAGNOSTIC TEST RESULT ---")
        logger.info(f"Strategy: {config.strategy_name}")
        logger.info(f"Labeling: TP_MULT={config.TP_ATR_MULTIPLIER}, SL_MULT={config.SL_ATR_MULTIPLIER}")
        logger.info(f"F1 Score Achieved: {f1_score:.4f}")
        logger.info(f"Profitability Score (Calmar): {best_objective_score:.4f}")
        if f1_score >= config.MIN_F1_SCORE_GATE:
            logger.warning(f"CONCLUSION: SUCCESS. The model PASSED the F1 gate of {config.MIN_F1_SCORE_GATE}.")
        else:
            logger.warning(f"CONCLUSION: FAILURE. The model FAILED the F1 gate of {config.MIN_F1_SCORE_GATE}.")
    else:
        logger.error("--- DIAGNOSTIC TEST RESULT: COMPLETE FAILURE. Model training process failed. ---")
 
    logger.removeHandler(file_handler)
    file_handler.close()
 
def main():
    """
    Main function to configure and run the diagnostic tests.
    """
    # This is a base configuration that both tests will inherit from.
    base_config = {
        # Use a failing strategy from previous logs
        "strategy_name": "CHoCH_Orderblock_Entry",
        "selected_features": [
            "choch_up_signal", "choch_down_signal", "fvg_bullish_exists",
            "fvg_bearish_exists", "volume_spike", "DAILY_ctx_Trend"
        ],
        "early_intervention": {"enabled": False},
        "BASE_PATH": os.getcwd(),
        "INITIAL_CAPITAL": 100.0,
        "OPTUNA_TRIALS": 50,
        "MAX_TRAINING_RETRIES_PER_CYCLE": 0, # No retries for diagnostic tests
        "TRAINING_WINDOW": '365D',
        "RETRAINING_FREQUENCY": '90D', # Irrelevant as we only run one cycle
        "FORWARD_TEST_GAP": "1D",
        "LOOKAHEAD_CANDLES": 150,
        "MIN_F1_SCORE_GATE": 0.55, # Using a realistic but still challenging gate for the test
        "CONFIDENCE_TIERS": {
            'ultra_high': {'min': 0.85, 'risk_mult': 1.2, 'rr': 2.5},
            'high':       {'min': 0.75, 'risk_mult': 1.0, 'rr': 2.0},
            'standard':   {'min': 0.60, 'risk_mult': 0.8, 'rr': 1.5}
        },
        "BASE_RISK_PER_TRADE_PCT": 0.01,
        "RISK_CAP_PER_TRADE_USD": 500.0,
    }
 
    # --- Test A: The "Big Move" Hypothesis ---
    config_a = base_config.copy()
    config_a.update({
        "REPORT_LABEL": "Test_02a_Big_Move_Labels",
        "nickname": "BigMoveTest",
        "TP_ATR_MULTIPLIER": 4.0,
        "SL_ATR_MULTIPLIER": 2.0
    })
    run_diagnostic_instance(config_a)
 
    # --- Test B: The "Scalper" Hypothesis ---
    # Add a small delay to separate the log files clearly
    time.sleep(5)
    config_b = base_config.copy()
    config_b.update({
        "REPORT_LABEL": "Test_02b_Scalper_Labels",
        "nickname": "ScalperTest",
        "TP_ATR_MULTIPLIER": 1.0,
        "SL_ATR_MULTIPLIER": 1.0
    })
    run_diagnostic_instance(config_b)
 
if __name__ == '__main__':
    # NOTE: This script assumes all class definitions (ConfigModel, GeminiAnalyzer, etc.)
    # from the main V208 script have been copied into the top of this file.
    main()
 
# Test_02_Label_Definition.py

import xgboost
import inspect
from xgboost import XGBClassifier
print("Reported version:", xgboost.__version__)
print("Module file path:", xgboost.__file__)
print("XGBClassifier source file:", inspect.getfile(XGBClassifier))
print("XGBClassifier fit() definition:\n")
print(inspect.signature(XGBClassifier.fit))

Quoting Bl4ckP3n9u1n

Disliked

Methodology to diagnose the performance issues Due to a number of issue I have had to create controlled experiments to systematically isolate the source of the low F1 scores. At the bottom I have provided a bit more context. Attached are the two templates and complete scripts that may help someone adapt the methodology for their own debugging. These are used with V208 (again just remove the .txt): Test_01_Simple_Model_Baseline.py Test_02_Label_Definition.py WARNING: (I have not completed the tests so even these might need debugging) By no means...

Ignored

Quoting Bl4ckP3n9u1n

Disliked

{quote} So if the model score (MAR ratio) is really good, why don't you just change the MIN Gate? I'm glad you asked the question. The current V208 I am testing with the changes I listed in the previous post does this. You'll see in the below output some new lines where the AI has identified the previous low quality gate scores and now has reduced the score and restarted the tests again: 2025-06-18 23:11:29,487 - WARNING - AI INTERVENTION CHOSEN: ADJUST_METRICS. Notes: The heuristic pre-analysis clearly indicates a 'High Profitability / Low Accuracy'...

Ignored

numpy==1.26.0
pandas==2.1.4
matplotlib==3.8.4
scipy==1.12.0
scikit-learn==1.4.2
xgboost==3.0.2
optuna==4.3.0
pydantic==2.11.5
python-dotenv==1.1.0
requests==2.32.4
joblib==1.4.2
statsmodels==0.14.4
yfinance==0.2.63
shap==0.47.2
PyWavelets==1.8.0
arch==7.2.0
hurst==0.0.5
sktime==0.27.0
pyarrow==20.0.0
torch_geometric==2.5.3
tensorflow-cpu==2.19.0
torch==2.3.0+cpu

pip install --break-system-packages \
    numpy==1.26.0 \
    pandas==2.1.4 \
    matplotlib==3.8.4 \
    scipy==1.12.0 \
    scikit-learn==1.4.2 \
    xgboost==3.0.2 \
    optuna==4.3.0 \
    pydantic==2.11.5 \
    python-dotenv==1.1.0 \
    requests==2.32.4 \
    joblib==1.4.2 \
    statsmodels==0.14.4 \
    yfinance==0.2.63 \
    shap==0.47.2 \
    PyWavelets==1.8.0 \
    arch==7.2.0 \
    hurst==0.0.5 \
    sktime==0.27.0 \
    pyarrow==20.0.0 \
    torch_geometric==2.5.3

pip install tensorflow-cpu==2.19.0 --break-system-packages

pip install torch==2.3.0+cpu --break-system-packages -f https://download.pytorch.org/whl/torch_stable.html

[size=2]class FeatureEngineer:
    """
    Enhanced feature engineering with additional simple features.
    This class includes standard technical indicators, advanced market structure analysis, volatility regime detection,
    and feature interaction/normalization to improve model performance.
    """
    TIMEFRAME_MAP = {'M1': 1, 'M5': 5, 'M15': 15, 'M30': 30, 'H1': 60, 'H4': 240, 'D1': 1440, 'DAILY': 1440}
    ANOMALY_FEATURES = [
        'ATR', 'bollinger_bandwidth', 'RSI', 'RealVolume', 'candle_body_size',
        'pct_change', 'candle_body_size_vs_atr', 'atr_vs_daily_atr', 'MACD_hist',
        'wick_to_body_ratio', 'overnight_gap_pct', 'RSI_zscore', 'volume_ma_ratio', 'volatility_hawkes'
    ][/size]
[size=2]    def __init__(self, config: 'ConfigModel', timeframe_roles: Dict[str, str], playbook: Dict):
        self.config = config
        self.roles = timeframe_roles
        self.playbook = playbook
        self.hurst_warning_symbols = set()[/size]
[size=2]    def _add_higher_tf_context(self, base_df: pd.DataFrame, higher_tf_df: pd.DataFrame, tf_name: str) -> pd.DataFrame:
        if higher_tf_df.empty:
            return base_df
        
        ctx_features = {'Close': 'last', 'High': 'max', 'Low': 'min', 'Open': 'first', 'ATR': 'mean', 'RSI': 'mean', 'ADX': 'mean'}
        base_tf_str = self.roles['base']
        minutes = self.TIMEFRAME_MAP.get(base_tf_str.upper())
        if not minutes:
            logger.error(f"Could not find timeframe '{base_tf_str}' in TIMEFRAME_MAP. Resampling will fail.")
            return base_df
        pandas_freq = f"{minutes}T"[/size]
[size=2]        resampled_features = {f"{tf_name}_ctx_{col}": higher_tf_df[col].resample(pandas_freq).ffill() for col, method in ctx_features.items() if col in higher_tf_df.columns}
        if not resampled_features: return base_df[/size]
[size=2]        resampled_df = pd.DataFrame(resampled_features)
        merged_df = pd.merge_asof(left=base_df.sort_index(), right=resampled_df.sort_index(), on='Timestamp', direction='backward')
        
        ctx_close_col = f"{tf_name}_ctx_Close"
        if ctx_close_col in merged_df.columns:
            merged_df[f"{tf_name}_ctx_Trend"] = np.sign(merged_df[ctx_close_col].diff(2)).fillna(0)
            
        return merged_df.set_index('Timestamp')[/size]
[size=2]    def _detect_anomalies(self, df: pd.DataFrame) -> pd.DataFrame:
        logger.info("    - Detecting anomalies with Isolation Forest...")
        anomaly_features_present = [f for f in self.ANOMALY_FEATURES if f in df.columns and f in df]
        if not anomaly_features_present:
            df['anomaly_score'] = 1
            return df
            
        df_anomaly = df[anomaly_features_present].dropna()
        if df_anomaly.empty:
            df['anomaly_score'] = 1
            return df[/size]
[size=2]        model = IsolationForest(contamination=self.config.anomaly_contamination_factor, random_state=42)
        predictions = model.fit_predict(df_anomaly)
        df['anomaly_score'] = pd.Series(predictions, index=df_anomaly.index)
        df['anomaly_score'].fillna(method='ffill', inplace=True)
        df['anomaly_score'].fillna(method='bfill', inplace=True)
        df['anomaly_score'].fillna(1, inplace=True)
        return df[/size]
[size=2]    def _calculate_relative_performance(self, df: pd.DataFrame) -> pd.DataFrame:
        if 'pct_change' not in df.columns:
             df['pct_change'] = df.groupby('Symbol')['Close'].pct_change()
             
        mean_returns = df.groupby('Timestamp')['pct_change'].mean()
        df['market_return'] = df.index.map(mean_returns)
        df['relative_performance'] = df['pct_change'] - df['market_return']
        df.drop(columns=['market_return'], inplace=True)
        return df[/size]
[size=2]    def _calculate_price_derivatives(self, g: pd.DataFrame, price_col: str = 'Close') -> pd.DataFrame:
        g[f'{price_col}_velocity'] = g[price_col].diff()
        g[f'{price_col}_acceleration'] = g[f'{price_col}_velocity'].diff()
        g[f'{price_col}_jerk'] = g[f'{price_col}_acceleration'].diff()
        return g[/size]
[size=2]    def _calculate_volume_derivatives(self, g: pd.DataFrame) -> pd.DataFrame:
        if 'RealVolume' in g.columns:
            g['volume_velocity'] = g['RealVolume'].diff()
            g['volume_acceleration'] = g['volume_velocity'].diff()
        else:
            g['volume_velocity'], g['volume_acceleration'] = 0, 0
        return g[/size]
[size=2]    def _calculate_statistical_moments(self, g: pd.DataFrame, window: int = 20) -> pd.DataFrame:
        log_returns = np.log(g['Close'] / g['Close'].shift(1))
        g['returns_skew'] = log_returns.rolling(window).skew()
        g['returns_kurtosis'] = log_returns.rolling(window).kurt()
        return g
        
    def _calculate_simple_features(self, df: pd.DataFrame) -> pd.DataFrame:
        df['pct_change'] = df['Close'].pct_change()
        df['overnight_gap_pct'] = df['Open'].pct_change()
        df['candle_body_size'] = (df['Close'] - df['Open']).abs()
        upper_wick = df['High'] - df[['Close', 'Open']].max(axis=1)
        lower_wick = df[['Close', 'Open']].min(axis=1) - df['Low']
        df['wick_to_body_ratio'] = (upper_wick + lower_wick) / df['candle_body_size'].replace(0, np.nan)
        if 'RSI' in df.columns:
            df['RSI_zscore'] = (df['RSI'] - df['RSI'].rolling(20).mean()) / df['RSI'].rolling(20).std()
        if 'RealVolume' in df.columns and not df['RealVolume'].empty:
            vol_ma = df['RealVolume'].rolling(20).mean()
            df['volume_ma_ratio'] = df['RealVolume'] / vol_ma.replace(0, np.nan)
        if 'ATR' in df.columns:
             df['candle_body_size_vs_atr'] = df['candle_body_size'] / df['ATR'].replace(0, np.nan)
        if 'DAILY_ctx_ATR' in df.columns and 'ATR' in df.columns:
            df['atr_vs_daily_atr'] = df['ATR'] / df['DAILY_ctx_ATR'].replace(0, np.nan)
        return df[/size]
[size=2]    def _calculate_hawkes_volatility(self, df: pd.DataFrame) -> pd.DataFrame:
        if 'ATR' not in df.columns:
            df['volatility_hawkes'] = 0.0
            return df
        atr_shocks = df['ATR'].diff().clip(lower=0)
        hawkes_intensity = atr_shocks.ewm(alpha=1 - self.config.HAWKES_KAPPA, adjust=False).mean()
        df['volatility_hawkes'] = hawkes_intensity
        return df[/size]
[size=2]    def _calculate_ohlc_ratios(self, df: pd.DataFrame) -> pd.DataFrame:
        df['ohlc_range'] = df['High'] - df['Low']
        # Avoid division by zero
        df['close_to_high'] = (df['High'] - df['Close']) / df['ohlc_range'].replace(0, np.nan)
        df['close_to_low'] = (df['Close'] - df['Low']) / df['ohlc_range'].replace(0, np.nan)
        return df[/size]
[size=2]    def _calculate_accumulation_distribution(self, df: pd.DataFrame) -> pd.DataFrame:
        if 'RealVolume' not in df.columns: return df
        clv = ((df['Close'] - df['Low']) - (df['High'] - df['Close'])) / (df['High'] - df['Low']).replace(0, np.nan)
        clv = clv.fillna(0)
        ad = (clv * df['RealVolume']).cumsum()
        df['AD_line'] = ad
        df['AD_line_slope'] = df['AD_line'].diff(5) # 5-period slope of the A/D line
        return df[/size]
[size=2]    def _calculate_mad(self, df: pd.DataFrame, window: int = 20) -> pd.DataFrame:
        """Mean Absolute Deviation"""
        df['mad'] = df['Close'].rolling(window).apply(lambda x: np.mean(np.abs(x - np.mean(x))), raw=True)
        return df[/size]
[size=2]    def _calculate_price_volume_correlation(self, df: pd.DataFrame, window: int = 20) -> pd.DataFrame:
        if 'RealVolume' not in df.columns: return df
        df['price_vol_corr'] = df['Close'].pct_change().rolling(window).corr(df['RealVolume'])
        return df[/size]
[size=2]    def _calculate_quantile_features(self, df: pd.DataFrame, window: int = 50) -> pd.DataFrame:
        log_returns = np.log(df['Close'] / df['Close'].shift(1))
        df['return_q25'] = log_returns.rolling(window).quantile(0.25)
        df['return_q75'] = log_returns.rolling(window).quantile(0.75)
        df['return_iqr'] = df['return_q75'] - df['return_q25']
        return df[/size]
[size=2]    def _calculate_regression_features(self, df: pd.DataFrame, window: int = 20) -> pd.DataFrame:
        def get_slope(series):
            y = series.values
            x = np.arange(len(y))
            slope = np.polyfit(x, y, 1)[0]
            return slope
        df['rolling_beta'] = df['Close'].rolling(window).apply(get_slope, raw=False)
        return df[/size]
[size=2]    def _calculate_cycle_features(self, df: pd.DataFrame, window: int = 40) -> pd.DataFrame:
        df['dominant_cycle_phase'] = np.nan
        df['dominant_cycle_period'] = np.nan
        close_series = df['Close'].dropna()
        symbol = df['Symbol'].iloc[0] if 'Symbol' in df.columns and not df.empty else 'UNKNOWN'[/size]
[size=2]        if len(close_series) < window:
            logger.debug(f"  - Cycle Features: Not enough data for {symbol} (found {len(close_series)}, need > {window}).")
            return df[/size]
[size=2]        try:
            analytic_signal = hilbert(close_series.values)
            phase = np.unwrap(np.angle(analytic_signal))
            phase_series = pd.Series(phase, index=close_series.index)
            df.loc[phase_series.index, 'dominant_cycle_phase'] = phase_series[/size]
[size=2]            inst_freq = np.diff(phase) / (2.0 * np.pi)
            inst_freq_series = pd.Series(inst_freq, index=close_series.index[1:])[/size]
[size=2]            epsilon = 1e-9
            safe_inst_freq_np = np.where(np.abs(inst_freq_series) < epsilon, np.nan, inst_freq_series)
            safe_inst_freq_series = pd.Series(safe_inst_freq_np, index=inst_freq_series.index)[/size]
[size=2]            if safe_inst_freq_series.isnull().all():
                logger.debug(f"  - Cycle Features: All instantaneous frequencies near-zero for {symbol} over {len(close_series)} data points (signal is flat).")
                return df[/size]
[size=2]            dominant_cycle_period_series = 1 / np.abs(safe_inst_freq_series)
            rolling_period = dominant_cycle_period_series.rolling(window=window, min_periods=max(1, window // 2)).mean()
            df.loc[rolling_period.index, 'dominant_cycle_period'] = rolling_period
        except Exception as e:
            logger.error(f"  - Cycle Features Error for symbol {symbol}: {e}")
        return df[/size]
[size=2]    def _calculate_autocorrelation_features(self, df: pd.DataFrame, lags: int = 5) -> pd.DataFrame:
        log_returns = np.log(df['Close'] / df['Close'].shift(1)).dropna()
        pacf_vals = pacf(log_returns, nlags=lags)
        for i in range(1, lags + 1):
            df[f'pacf_lag_{i}'] = pacf_vals[i]
        return df[/size]
[size=2]    def _calculate_entropy_features(self, df: pd.DataFrame, window: int = 30) -> pd.DataFrame:
        def roll_entropy(series):
            discretized = pd.cut(series, bins=10, labels=False)
            counts = discretized.value_counts(normalize=True)
            return entropy(counts, base=2)
        df['shannon_entropy'] = df['Close'].pct_change().rolling(window).apply(roll_entropy, raw=False)
        return df[/size]
[size=2]    def _calculate_fourier_transform_features(self, df: pd.DataFrame, window: int = 50) -> pd.DataFrame:
        def get_dominant_freq(series: pd.Series) -> tuple[float, float]:
            n = len(series)
            if n < window or series.nunique() < 2:
                return np.nan, np.nan
            try:
                fft_vals = scipy.fft.fft(series.values)
                fft_freq = scipy.fft.fftfreq(n)
                idx = np.argmax(np.abs(fft_vals[1:n//2])) + 1
                return np.abs(fft_freq[idx]), np.abs(fft_vals[idx]) / n
            except Exception:
                return np.nan, np.nan[/size]
[size=2]        results_list = [get_dominant_freq(w) for w in df['Close'].rolling(window)]
        if results_list:
            fft_df = pd.DataFrame(results_list, index=df.index, columns=['fft_dom_freq', 'fft_dom_amp'])
            df[['fft_dom_freq', 'fft_dom_amp']] = fft_df
        else:
            df['fft_dom_freq'] = np.nan
            df['fft_dom_amp'] = np.nan
        return df[/size]
[size=2]    def _calculate_wavelet_features(self, df: pd.DataFrame, wavelet_name='db4', level=4) -> pd.DataFrame:
        if not PYWT_AVAILABLE: return df
        coeffs = pywt.wavedec(df['Close'], wavelet_name, level=level)
        for i, c in enumerate(coeffs):
            df[f'wavelet_energy_level_{i}'] = np.sum(np.square(c))
        return df[/size]
[size=2]    def _calculate_garch_volatility(self, df: pd.DataFrame) -> pd.DataFrame:
        df['garch_volatility'] = np.nan
        if not ARCH_AVAILABLE: return df
        
        # Upscale the input data by 1000 for optimizer stability
        log_returns = np.log(df['Close'].replace(0,np.nan) / df['Close'].shift(1).replace(0,np.nan)).dropna() * 1000[/size]
[size=2]        if len(log_returns) < 20:
            return df
            
        try:
            # Tell the library we are handling scaling manually
            garch_model = arch_model(log_returns, vol='Garch', p=1, q=1, rescale=False)
            res = garch_model.fit(update_freq=0, disp='off', show_warning=False)
            forecast = res.forecast(horizon=5, reindex=False)
            
            # Downscale the final output by 1000 to return it to its original units
            pred_vol = np.sqrt(forecast.variance.iloc[-1].mean()) / 1000.0
            
            df.at[log_returns.index[-1], 'garch_volatility'] = pred_vol
            df['garch_volatility'] = df['garch_volatility'].bfill()
        except Exception as e:
            logger.error(f"  - GARCH Error: {e}")
            
        return df[/size]
[size=2]    def _calculate_hurst_exponent(self, df: pd.DataFrame, window: int = 100) -> pd.DataFrame:
        if not HURST_AVAILABLE: return df
        df['hurst_exponent'] = df['Close'].rolling(window).apply(lambda x: compute_Hc(x)[0], raw=False)
        return df
        
    def _calculate_meta_features(self, df: pd.DataFrame) -> pd.DataFrame:
        """Calculates non-linear interaction features (meta-features)."""
        logger.info("    - Calculating meta-features (feature interactions)...")
        
        # Existing interaction features
        if 'RSI' in df.columns and 'bollinger_bandwidth' in df.columns:
            df['rsi_x_bolli'] = df['RSI'] * df['bollinger_bandwidth']
        
        if 'ADX' in df.columns and 'market_volatility_index' in df.columns:
            df['adx_x_vol_rank'] = df['ADX'] * df['market_volatility_index']
            
        if 'hurst_exponent' in df.columns and 'ADX' in df.columns:
            df['hurst_x_adx'] = df['hurst_exponent'] * df['ADX']
            
        if 'ATR' in df.columns and 'DAILY_ctx_ATR' in df.columns:
            df['atr_ratio_short_long'] = df['ATR'] / df['DAILY_ctx_ATR'].replace(0, np.nan)
            
        # --- ADDING NEW META-FEATURES FOR HURST INTERCEPT ---
        if 'hurst_intercept' in df.columns and 'ADX' in df.columns:
            df['hurst_intercept_x_adx'] = df['hurst_intercept'] * df['ADX']
            
        if 'hurst_intercept' in df.columns and 'ATR' in df.columns:
            df['hurst_intercept_x_atr'] = df['hurst_intercept'] * df['ATR']
                
        return df[/size]
[size=2]    def _process_single_symbol_stack(self, symbol_data_by_tf: Dict[str, pd.DataFrame]) -> pd.DataFrame:
        """
        Orchestrates the entire feature engineering pipeline for a single symbol.
        It calls all individual feature calculation methods in a logical order.
        """
        # --- 1. Initial Data Validation ---
        base_df = symbol_data_by_tf.get(self.roles['base'])
        if base_df is None or base_df.empty:
            logger.warning("Base timeframe data missing or empty for a symbol. Skipping.")
            return pd.DataFrame()[/size]
[size=2]        df = base_df.copy()
        # Ensure index is a DatetimeIndex for all time-series operations
        df.index = pd.to_datetime(df.index)[/size]
[size=2]        # --- 2. Foundational Indicators ---
        # These are the basic building blocks for many other features.
        delta = df['Close'].diff()
        gain = delta.clip(lower=0).ewm(alpha=1/14, adjust=False).mean()
        loss = (-delta.clip(upper=0)).ewm(alpha=1/14, adjust=False).mean()
        rs = gain / loss.replace(0, 1e-9)
        df['RSI'] = 100 - (100 / (1 + rs))
        
        exp1 = df['Close'].ewm(span=12, adjust=False).mean()
        exp2 = df['Close'].ewm(span=26, adjust=False).mean()
        df['MACD'] = exp1 - exp2
        df['MACD_signal'] = df['MACD'].ewm(span=9, adjust=False).mean()
        df['MACD_hist'] = df['MACD'] - df['MACD_signal'][/size]
[size=2]        high_low = df['High'] - df['Low']
        high_close = np.abs(df['High'] - df['Close'].shift())
        low_close = np.abs(df['Low'] - df['Close'].shift())
        tr = pd.concat([high_low, high_close, low_close], axis=1).max(axis=1)
        df['ATR'] = tr.ewm(alpha=1/14, adjust=False).mean()[/size]
[size=2]        ma = df['Close'].rolling(window=self.config.BOLLINGER_PERIOD).mean()
        std = df['Close'].rolling(window=self.config.BOLLINGER_PERIOD).std()
        df['bollinger_upper'] = ma + (std * 2)
        df['bollinger_lower'] = ma - (std * 2)
        df['bollinger_bandwidth'] = (df['bollinger_upper'] - df['bollinger_lower']) / ma.replace(0, 1e-9)[/size]
[size=2]        low_k = df['Low'].rolling(window=self.config.STOCHASTIC_PERIOD).min()
        high_k = df['High'].rolling(window=self.config.STOCHASTIC_PERIOD).max()
        df['stoch_k'] = 100 * ((df['Close'] - low_k) / (high_k - low_k).replace(0, 1e-9))
        
        plus_dm = df['High'].diff().clip(lower=0)
        minus_dm = df['Low'].diff().clip(lower=0) # Note: this is typically -df['Low'].diff().clip(lower=0) but using absolute diffs
        plus_di = 100 * (plus_dm.ewm(alpha=1/14, adjust=False).mean() / df['ATR'])
        minus_di = 100 * (minus_dm.ewm(alpha=1/14, adjust=False).mean() / df['ATR'])
        dx = 100 * (np.abs(plus_di - minus_di) / (plus_di + minus_di).replace(0, 1e-9))
        df['ADX'] = dx.ewm(alpha=1/14, adjust=False).mean()[/size]
[size=2]        df['momentum_20'] = df['Close'].pct_change(20)
        df['EMA_20'] = df['Close'].ewm(span=20, adjust=False).mean()
        df['EMA_50'] = df['Close'].ewm(span=50, adjust=False).mean()[/size]
[size=2]        # --- 3. Contextual, Advanced, and Scientific Features ---
        # These calls leverage the robust helper methods we refined.
        df = self._add_higher_tf_context(df, symbol_data_by_tf.get(self.roles.get('medium'), pd.DataFrame()), 'H1')
        df = self._add_higher_tf_context(df, symbol_data_by_tf.get(self.roles.get('high'), pd.DataFrame()), 'DAILY')
        df = self._calculate_simple_features(df)
        df = self._calculate_price_derivatives(df)
        df = self._calculate_volume_derivatives(df)
        df = self._calculate_statistical_moments(df)
        df = self._calculate_ohlc_ratios(df)
        df = self._calculate_accumulation_distribution(df)
        df = self._calculate_mad(df)
        df = self._calculate_price_volume_correlation(df)
        df = self._calculate_quantile_features(df)
        df = self._calculate_regression_features(df)
        df = self._calculate_cycle_features(df)
        df = self._calculate_autocorrelation_features(df)
        df = self._calculate_entropy_features(df)
        df = self._calculate_fourier_transform_features(df)
        df = self._calculate_wavelet_features(df)
        df = self._calculate_garch_volatility(df)
        df = self._calculate_hurst_exponent(df) # This now creates 'hurst_exponent' and 'hurst_intercept'
        df = self._calculate_hawkes_volatility(df)[/size]
[size=2]        # --- 4. Regime and Meta-Feature Layer ---
        # This layer must come after the advanced features are calculated.
        if 'Close' in df.columns and not df['Close'].isnull().all():
            df['realized_volatility'] = df['Close'].pct_change().rolling(14).std() * np.sqrt(252 * 24 * 4) # Annualized example
            df['market_volatility_index'] = df['realized_volatility'].rank(pct=True)
        if 'ADX' in df.columns and not df['ADX'].isnull().all():
            df['market_trend_strength'] = df['ADX'].rank(pct=True)
        if 'market_volatility_index' in df.columns and 'market_trend_strength' in df.columns:
            combined_metric = df['market_volatility_index'].fillna(0.5) + df['market_trend_strength'].fillna(0.5)
            if combined_metric.nunique() > 1:
                df['market_regime'] = pd.qcut(combined_metric, 4, labels=False, duplicates='drop')
        
        # This call now creates interactions for both hurst_exponent and hurst_intercept
        df = self._calculate_meta_features(df)[/size]
[size=2]        # --- 5. Final Anomaly Detection ---
        # This runs last to analyze the fully-featured dataset.
        df = self._detect_anomalies(df)
        
        return df[/size]
[size=2]    # --- NEW: Standard, in-memory PCA method for smaller datasets ---
    def _apply_pca_standard(self, df: pd.DataFrame, pca_features: list) -> pd.DataFrame:
        """Applies standard PCA for smaller datasets that fit comfortably in memory."""
        # FIX 2: Reduce memory footprint before PCA
        df_pca_subset = df[pca_features].dropna().astype(np.float32)
        
        # Drop low-variance features to reduce noise and computation
        df_pca_subset = df_pca_subset.loc[:, df_pca_subset.std() > 1e-6]
        if df_pca_subset.shape[1] < self.config.PCA_N_COMPONENTS:
            logger.warning("Number of features after variance filtering is less than n_components. Skipping PCA.")
            return df[/size]
[size=2]        pipeline = Pipeline([
            ('scaler', StandardScaler()),
            ('pca', PCA(n_components=self.config.PCA_N_COMPONENTS))
        ])[/size]
[size=2]        principal_components = pipeline.fit_transform(df_pca_subset)
        
        pca_cols = [f'RSI_PCA_{i+1}' for i in range(self.config.PCA_N_COMPONENTS)]
        pca_df = pd.DataFrame(principal_components, columns=pca_cols, index=df_pca_subset.index)[/size]
[size=2]        df = df.join(pca_df)
        logger.info(f"Standard PCA complete. Explained variance: {pipeline.named_steps['pca'].explained_variance_ratio_.sum():.2%}")
        return df[/size]
[size=2]    def _apply_pca_incremental(self, df: pd.DataFrame, pca_features: list) -> pd.DataFrame:
        """
        Applies IncrementalPCA using batched fitting and transforming to handle large datasets
        without exceeding memory limits.
        """
        # FIX 2: Reduce memory footprint before PCA
        df_pca_subset = df[pca_features].dropna()
        # Downcast dtypes to reduce memory usage
        for col in df_pca_subset.select_dtypes(include=['float']):
            df_pca_subset[col] = pd.to_numeric(df_pca_subset[col], downcast='float')
            
        # Drop near-constant columns
        initial_feature_count = df_pca_subset.shape[1]
        df_pca_subset = df_pca_subset.loc[:, df_pca_subset.std() > 1e-6]
        logger.info(f"PCA pre-filtering: Removed {initial_feature_count - df_pca_subset.shape[1]} near-constant features.")
        
        if df_pca_subset.shape[1] < self.config.PCA_N_COMPONENTS:
            logger.warning("Number of available features for PCA is less than n_components after filtering. Skipping PCA.")
            return df
            
        scaler = StandardScaler()
        ipca = IncrementalPCA(n_components=self.config.PCA_N_COMPONENTS)
        batch_size = 100000[/size]
[size=2]        logger.info(f"Fitting IncrementalPCA in batches of {batch_size}...")
        # Fit in batches (memory-safe)
        for i in range(0, df_pca_subset.shape[0], batch_size):
            batch = df_pca_subset.iloc[i:i + batch_size]
            scaled_batch = scaler.fit_transform(batch) # Fit scaler on each batch, or fit once on a sample
            ipca.partial_fit(scaled_batch)[/size]
[size=2]        # FIX 1: Transform in batches to avoid memory spike
        logger.info("Transforming full dataset in batches with fitted IncrementalPCA...")
        
        # DEBUGGING: Track memory usage
        logger.info(f"Memory before transform: {psutil.Process(os.getpid()).memory_info().rss / 1024**2:.2f} MB")
        
        transformed_batches = []
        for i in range(0, df_pca_subset.shape[0], batch_size):
            batch_to_transform = df_pca_subset.iloc[i:i + batch_size]
            scaled_batch_to_transform = scaler.transform(batch_to_transform) # Use the already fitted scaler
            transformed_batches.append(ipca.transform(scaled_batch_to_transform))[/size]
[size=2]        principal_components = np.vstack(transformed_batches)
        
        logger.info(f"Memory after transform: {psutil.Process(os.getpid()).memory_info().rss / 1024**2:.2f} MB")[/size]
[size=2]        pca_cols = [f'RSI_PCA_{i+1}' for i in range(self.config.PCA_N_COMPONENTS)]
        pca_df = pd.DataFrame(principal_components, columns=pca_cols, index=df_pca_subset.index)[/size]
[size=2]        df = df.join(pca_df)
        logger.info(f"IncrementalPCA reduction complete. Explained variance ratio: {ipca.explained_variance_ratio_.sum():.2%}")
        return df[/size]
[size=2]    def _apply_pca_reduction(self, df: pd.DataFrame) -> pd.DataFrame:
        """
        Orchestrates PCA by choosing between standard (fast, in-memory) and incremental
        (slower, memory-safe) methods based on dataset size.
        """
        if not self.config.USE_PCA_REDUCTION:
            return df
            
        logger.warning("Applying PCA reduction. NOTE: For best practice, PCA should be fit on training data only to avoid lookahead bias.")[/size]
[size=2]        for period in self.config.RSI_PERIODS_FOR_PCA:
            df[f'RSI_{period}'] = 100 - (100 / (1 + df.groupby('Symbol')['Close'].diff().rolling(window=period).apply(
                lambda x: x[x > 0].sum() / (-x[x < 0].sum() if -x[x < 0].sum() != 0 else 1), raw=True)))[/size]
[size=2]        rsi_features = [f'RSI_{period}' for period in self.config.RSI_PERIODS_FOR_PCA]
        df_pca_subset = df[rsi_features].dropna()[/size]
[size=2]        if df_pca_subset.empty:
            logger.error("Not enough data to perform PCA on RSI features. Skipping.")
            return df[/size]
[size=2]        if len(df_pca_subset) < 500_000:
            logger.info("Dataset small enough for standard PCA. Using fast in-memory method.")
            return self._apply_pca_standard(df, rsi_features)
        else:
            logger.info("Dataset is large. Using memory-efficient IncrementalPCA method.")
            return self._apply_pca_incremental(df, rsi_features)[/size]
[size=2]    def create_feature_stack(self, data_by_tf: Dict[str, pd.DataFrame]) -> pd.DataFrame:
        logger.info("-> Stage 2: Engineering Features...")
        base_tf = self.roles['base']
        if base_tf not in data_by_tf:
            logger.critical(f"Base timeframe '{base_tf}' data is missing. Cannot proceed.")
            return pd.DataFrame()[/size]
[size=2]        all_symbols_processed_dfs = []
        unique_symbols = data_by_tf[base_tf]['Symbol'].unique()[/size]
[size=2]        for i, symbol in enumerate(unique_symbols):
            logger.info(f"  - [{i+1}/{len(unique_symbols)}] Processing all features for symbol: {symbol}")
            symbol_specific_data = {tf: df[df['Symbol'] == symbol].copy() for tf, df in data_by_tf.items()}
            processed_symbol_df = self._process_single_symbol_stack(symbol_specific_data)
            if not processed_symbol_df.empty:
                all_symbols_processed_dfs.append(processed_symbol_df)[/size]
[size=2]        if not all_symbols_processed_dfs:
            logger.critical("Feature engineering resulted in no processable data across all symbols.")
            return pd.DataFrame()[/size]
[size=2]        logger.info("  - Concatenating data for all symbols...")
        final_df = pd.concat(all_symbols_processed_dfs, sort=False).sort_index()
        
        logger.info("  - Calculating cross-symbol features...")
        final_df = self._calculate_relative_performance(final_df)[/size]
[size=2]        if self.config.USE_PCA_REDUCTION:
            logger.info("  - Applying PCA reduction to feature set...")
            final_df = self._apply_pca_reduction(final_df)
            
        # --- NEW: Add Noise-Contrastive Features ---
        logger.info("  - Adding noise-contrastive features for diagnostics...")
        final_df['noise_1'] = np.random.normal(0, 1, len(final_df))
        final_df['noise_2'] = np.random.uniform(-1, 1, len(final_df))
        # --- END ---[/size]
[size=2]        logger.info("  - Applying final data shift and cleaning...")
        feature_cols = [c for c in final_df.columns if c not in ['Open','High','Low','Close','RealVolume','Symbol','Timestamp','primary_model_signal','target']]
        final_df[feature_cols] = final_df.groupby('Symbol', sort=False)[feature_cols].shift(1)
        final_df.replace([np.inf,-np.inf],np.nan,inplace=True)
        final_df.dropna(subset=['ATR', 'RSI', 'ADX'], inplace=True)[/size]
[size=2]        logger.info(f"[SUCCESS] Feature engineering complete. Final dataset shape: {final_df.shape}")
        return final_df[/size]
[size=2]    def label_standard(self, df: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        logger.info("-> Stage 3: Generating Trade Labels ('standard')...")
        labeled_dfs = [self._label_group(group, lookahead) for _, group in df.groupby('Symbol')]
        return pd.concat(labeled_dfs) if labeled_dfs else pd.DataFrame()[/size]
[size=2]    def label_meta(self, df: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        logger.info("-> Stage 3: Generating Trade Labels ('meta')...")
        labeled_dfs = [self._label_meta_group(group, lookahead) for _, group in df.groupby('Symbol')]
        return pd.concat(labeled_dfs) if labeled_dfs else pd.DataFrame()
    
    def label_volatility_adjusted(self, df: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        logger.info("-> Stage 3: Generating Trade Labels ('volatility_adjusted')...")
        labeled_dfs = [self._label_volatility_adjusted_group(group, lookahead) for _, group in df.groupby('Symbol')]
        return pd.concat(labeled_dfs) if labeled_dfs else pd.DataFrame()[/size]
[size=2]    def label_trend_quality(self, df: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        logger.info("-> Stage 3: Generating Trade Labels ('trend_quality')...")
        labeled_dfs = [self._label_trend_quality_group(group, lookahead) for _, group in df.groupby('Symbol')]
        return pd.concat(labeled_dfs) if labeled_dfs else pd.DataFrame()[/size]
[size=2]    def label_optimal_entry(self, df: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        logger.info("-> Stage 3: Generating Trade Labels ('optimal_entry')...")
        labeled_dfs = [self._label_optimal_entry_group(group, lookahead) for _, group in df.groupby('Symbol')]
        return pd.concat(labeled_dfs) if labeled_dfs else pd.DataFrame()[/size]
[size=2]    def label_regime_aware(self, df: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        """NEW: Regime-Aware Labeling Method"""
        logger.info("-> Stage 3: Generating Trade Labels ('regime_aware')...")
        labeled_dfs = [self._label_regime_aware_group(group, lookahead) for _, group in df.groupby('Symbol')]
        return pd.concat(labeled_dfs) if labeled_dfs else pd.DataFrame()[/size]
[size=2]    def _label_regime_aware_group(self, group: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        """Applies labels only if market regime conditions are met."""
        group = group.copy()
        # First, generate standard triple-barrier outcomes
        group_with_outcomes = self._label_group(group, lookahead)
        
        # Define regime conditions
        # Example: Only consider trades if the market is trending (top 60% strength) and not excessively volatile (bottom 80% vol)
        is_trending = group_with_outcomes['market_trend_strength'] > 0.60
        is_not_hyper_volatile = group_with_outcomes['market_volatility_index'] < 0.80
        valid_regime_mask = is_trending & is_not_hyper_volatile[/size]
[size=2]        # Invalidate labels where regime conditions are not met
        original_labels = group_with_outcomes['target'].copy()
        group_with_outcomes['target'] = 0 # Default to hold
        group_with_outcomes.loc[valid_regime_mask, 'target'] = original_labels[valid_regime_mask][/size]
[size=2]        logger.info(f"    - Regime-Aware Filter: Kept {valid_regime_mask.sum()}/{len(group)} labels as valid for trading.")
        
        return group_with_outcomes[/size]
[size=2]    def _label_group(self, group: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        group = group.copy()
        if 'ATR' not in group.columns or len(group) < lookahead + 1:
            logger.warning(f"ATR not found or insufficient data for labeling in group. Skipping.")
            group['target'] = 0
            return group[/size]
[size=2]        tp_multiplier = self.config.TP_ATR_MULTIPLIER
        sl_multiplier = self.config.SL_ATR_MULTIPLIER
        profit_target_points = group['ATR'] * tp_multiplier
        stop_loss_points = group['ATR'] * sl_multiplier
        
        outcomes = np.zeros(len(group))
        prices, highs, lows = group['Close'].values, group['High'].values, group['Low'].values
        total_rows = len(group)
        update_interval = max(1, (total_rows - lookahead) // 100) if total_rows > lookahead else 1[/size]
[size=2]        for i in range(len(group) - lookahead):
            if i > 0 and i % update_interval == 0:
                progress_pct = (i / (total_rows - lookahead)) * 100
                symbol_name = group['Symbol'].iloc[0]
                sys.stdout.write(f"\r    - Labeling '{symbol_name}': {progress_pct:5.1f}% complete...")
                sys.stdout.flush()[/size]
[size=2]            sl_dist = stop_loss_points.iloc[i]
            tp_dist = profit_target_points.iloc[i]
            if pd.isna(sl_dist) or sl_dist <= 1e-9: continue[/size]
[size=2]            tp_long_level, sl_long_level = prices[i] + tp_dist, prices[i] - sl_dist
            tp_short_level, sl_short_level = prices[i] - tp_dist, prices[i] + sl_dist
            future_highs, future_lows = highs[i+1 : i+1+lookahead], lows[i+1 : i+1+lookahead][/size]
[size=2]            hit_tp_long_idx = np.where(future_highs >= tp_long_level)[0]
            hit_sl_long_idx = np.where(future_lows <= sl_long_level)[0]
            first_tp_long = hit_tp_long_idx[0] if len(hit_tp_long_idx) > 0 else np.inf
            first_sl_long = hit_sl_long_idx[0] if len(hit_sl_long_idx) > 0 else np.inf[/size]
[size=2]            hit_tp_short_idx = np.where(future_lows <= tp_short_level)[0]
            hit_sl_short_idx = np.where(future_highs >= sl_short_level)[0]
            first_tp_short = hit_tp_short_idx[0] if len(hit_tp_short_idx) > 0 else np.inf
            first_sl_short = hit_sl_short_idx[0] if len(hit_sl_short_idx) > 0 else np.inf[/size]
[size=2]            if first_tp_long < first_sl_long: outcomes[i] = 1
            if first_tp_short < first_sl_short: outcomes[i] = -1
        
        group['target'] = outcomes
        sys.stdout.write(f"\r    - Labeling '{group['Symbol'].iloc[0]}': 100.0% complete... Done.       \n")
        sys.stdout.flush()
        return group[/size]
[size=2]    def _label_meta_group(self, group: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        group = group.copy()
        if 'primary_model_signal' not in group.columns or len(group) < lookahead + 1:
            group['target'] = 0; return group
        
        tp_multiplier = self.config.TP_ATR_MULTIPLIER
        sl_multiplier = self.config.SL_ATR_MULTIPLIER
        sl_atr_dynamic = group['ATR'] * sl_multiplier
        tp_atr_dynamic = group['ATR'] * tp_multiplier
        outcomes = np.zeros(len(group))
        prices, lows, highs = group['Close'].values, group['Low'].values, group['High'].values
        primary_signals = group['primary_model_signal'].values
        min_return = self.config.LABEL_MIN_RETURN_PCT
        total_rows = len(group)
        update_interval = max(1, (total_rows - lookahead) // 100) if total_rows > lookahead else 1[/size]
[size=2]        for i in range(len(group) - lookahead):
            if i > 0 and i % update_interval == 0:
                progress_pct = (i / (total_rows - lookahead)) * 100
                symbol_name = group['Symbol'].iloc[0]
                sys.stdout.write(f"\r    - Meta-Labeling '{symbol_name}': {progress_pct:5.1f}% complete...")
                sys.stdout.flush()[/size]
[size=2]            signal = primary_signals[i]
            if signal == 0: continue
            sl_dist, tp_dist = sl_atr_dynamic.iloc[i], tp_atr_dynamic.iloc[i]
            if pd.isna(sl_dist) or sl_dist <= 1e-9: continue
            
            future_highs, future_lows = highs[i + 1:i + 1 + lookahead], lows[i + 1:i + 1 + lookahead]
            
            if signal > 0:
                tp_level, sl_level = prices[i] + tp_dist, prices[i] - sl_dist
                if (tp_level / prices[i] - 1) <= min_return: continue
                time_to_tp = np.where(future_highs >= tp_level)[0]
                time_to_sl = np.where(future_lows <= sl_level)[0]
                if len(time_to_tp) > 0 and (len(time_to_sl) == 0 or time_to_tp[0] < time_to_sl[0]):
                    outcomes[i] = 1
            
            elif signal < 0:
                tp_level, sl_level = prices[i] - tp_dist, prices[i] + sl_dist
                if (prices[i] / tp_level - 1) <= min_return: continue
                time_to_tp = np.where(future_lows <= tp_level)[0]
                time_to_sl = np.where(future_highs >= sl_level)[0]
                if len(time_to_tp) > 0 and (len(time_to_sl) == 0 or time_to_tp[0] < time_to_sl[0]):
                    outcomes[i] = 1
        
        group['target'] = outcomes
        sys.stdout.write(f"\r    - Meta-Labeling '{group['Symbol'].iloc[0]}': 100.0% complete... Done.       \n")
        sys.stdout.flush()
        return group[/size]
[size=2]    def _label_volatility_adjusted_group(self, group: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        group = group.copy()
        outcomes = np.zeros(len(group))
        if 'ATR' not in group.columns or 'market_volatility_index' not in group.columns:
            logger.warning(f"Skipping volatility-adjusted labeling for group: 'ATR' or 'market_volatility_index' column missing.")
            group['target'] = 0
            return group
        
        prices, highs, lows = group['Close'].values, group['High'].values, group['Low'].values
        atr_values = group['ATR'].values
        vol_rank_values = group['market_volatility_index'].values[/size]
[size=2]        total_rows = len(group)
        update_interval = max(1, (total_rows - lookahead) // 100) if total_rows > lookahead else 1[/size]
[size=2]        for i in range(len(group) - lookahead):
            if i > 0 and i % update_interval == 0:
                progress_pct = (i / (total_rows - lookahead)) * 100
                symbol_name = group['Symbol'].iloc[0]
                sys.stdout.write(f"\r    - Labeling (Vol-Adj) '{symbol_name}': {progress_pct:5.1f}% complete...")
                sys.stdout.flush()[/size]
[size=2]            current_vol, vol_rank = atr_values[i], vol_rank_values[i]
            if pd.isna(current_vol) or pd.isna(vol_rank): continue
            
            if vol_rank > 0.7: tp_mult, sl_mult = 1.5, 1.0
            elif vol_rank < 0.3: tp_mult, sl_mult = 3.0, 2.0
            else: tp_mult, sl_mult = 2.0, 1.5
                
            tp_dist, sl_dist = current_vol * tp_mult, current_vol * sl_mult
            if sl_dist <= 1e-9: continue
            
            tp_long, sl_long = prices[i] + tp_dist, prices[i] - sl_dist
            tp_short, sl_short = prices[i] - tp_dist, prices[i] + sl_dist
            future_highs, future_lows = highs[i+1:i+1+lookahead], lows[i+1:i+1+lookahead][/size]
[size=2]            hit_tp_long_idx = np.where(future_highs >= tp_long)[0]; hit_sl_long_idx = np.where(future_lows <= sl_long)[0]
            first_tp_long = hit_tp_long_idx[0] if len(hit_tp_long_idx) > 0 else np.inf
            first_sl_long = hit_sl_long_idx[0] if len(hit_sl_long_idx) > 0 else np.inf
            hit_tp_short_idx = np.where(future_lows <= tp_short)[0]; hit_sl_short_idx = np.where(future_highs >= sl_short)[0]
            first_tp_short = hit_tp_short_idx[0] if len(hit_tp_short_idx) > 0 else np.inf
            first_sl_short = hit_sl_short_idx[0] if len(hit_sl_short_idx) > 0 else np.inf
            
            if first_tp_long < first_sl_long: outcomes[i] = 1
            if first_tp_short < first_sl_short: outcomes[i] = -1
            
        group['target'] = outcomes
        sys.stdout.write(f"\r    - Labeling (Vol-Adj) '{group['Symbol'].iloc[0]}': 100.0% complete... Done.       \n")
        sys.stdout.flush()
        return group[/size]
[size=2]    def _label_trend_quality_group(self, group: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        group = group.copy()
        outcomes = np.zeros(len(group))
        total_rows = len(group)
        update_interval = max(1, (total_rows - lookahead) // 100) if total_rows > lookahead else 1[/size]
[size=2]        for i in range(len(group) - lookahead):
            if i > 0 and i % update_interval == 0:
                progress_pct = (i / (total_rows - lookahead)) * 100
                symbol_name = group['Symbol'].iloc[0]
                sys.stdout.write(f"\r    - Labeling (Trend Quality) '{symbol_name}': {progress_pct:5.1f}% complete...")
                sys.stdout.flush()[/size]
[size=2]            entry_price = group['Close'].iloc[i]
            if entry_price == 0: continue
            future_highs = group['High'].iloc[i+1:i+1+lookahead]
            future_lows = group['Low'].iloc[i+1:i+1+lookahead]
            
            max_move = (future_highs.max() - entry_price) / entry_price
            min_move = (entry_price - future_lows.min()) / entry_price
            
            if max_move > 0.01 and max_move > 2 * min_move: outcomes[i] = 1
            elif min_move > 0.01 and min_move > 2 * max_move: outcomes[i] = -1
                
        group['target'] = outcomes
        sys.stdout.write(f"\r    - Labeling (Trend Quality) '{group['Symbol'].iloc[0]}': 100.0% complete... Done.       \n")
        sys.stdout.flush()
        return group[/size]
[size=2]    def _label_optimal_entry_group(self, group: pd.DataFrame, lookahead: int) -> pd.DataFrame:
        group = group.copy()
        outcomes = np.zeros(len(group))
        if 'ADX' not in group.columns or 'EMA_20' not in group.columns or 'EMA_50' not in group.columns:
             logger.warning(f"Skipping optimal entry labeling for group: required EMA/ADX columns missing.")
             group['target'] = 0
             return group[/size]
[size=2]        total_rows = len(group)
        update_interval = max(1, (total_rows - lookahead) // 100) if total_rows > lookahead else 1
        
        # Use the configurable minimum return from the config
        min_return_pct = self.config.LABEL_MIN_RETURN_PCT[/size]
[size=2]        for i in range(len(group) - lookahead):
            if i > 0 and i % update_interval == 0:
                progress_pct = (i / (total_rows - lookahead)) * 100
                symbol_name = group['Symbol'].iloc[0]
                sys.stdout.write(f"\r    - Labeling (Optimal Entry) '{symbol_name}': {progress_pct:5.1f}% complete...")
                sys.stdout.flush()[/size]
[size=2]            # Use the configurable trend filter threshold from the config
            if group['ADX'].iloc[i] < self.config.TREND_FILTER_THRESHOLD: continue
            
            entry_price = group['Close'].iloc[i]
            if entry_price == 0: continue
            future_prices = group['Close'].iloc[i+1:i+1+lookahead]
            ret = (future_prices.iloc[-1] - entry_price) / entry_price
            
            if group['EMA_20'].iloc[i] > group['EMA_50'].iloc[i]:
                # Check for pullback below the fast EMA and if the future return met the minimum
                if entry_price < group['EMA_20'].iloc[i] and ret > min_return_pct: outcomes[i] = 1
            else:
                # Check for pullback above the fast EMA and if the future return met the minimum
                if entry_price > group['EMA_20'].iloc[i] and ret < -min_return_pct: outcomes[i] = -1
                    
        group['target'] = outcomes
        sys.stdout.write(f"\r    - Labeling (Optimal Entry) '{group['Symbol'].iloc[0]}': 100.0% complete... Done.       \n")
        sys.stdout.flush()
        return group[/size]
[size=2][/size]

2025-06-21 23:45:07,671 - INFO - -> Feature Caching is ENABLED. Checking for a valid cache...
2025-06-21 23:45:07,671 - INFO -   - Cache is VALID. Loading features from cache...

[size=2]mql5dev@Bl4ckP3n9u1n:~$ python3 ML_Trading_Framework_PRO_V209_Linux.py
--- ML Trading Framework V209 Initializing ---
------------------------------------------------------------
INFO: PyTorch and PyG loaded successfully. GNN module is available.
WARNING: TensorFlow/Keras not found. LSTM strategies will be unavailable.
INFO: sktime loaded successfully. MiniRocket module is available.
------------------------------------------------------------
2025-06-21 20:54:54,807 - INFO - Framework entry point reached. Starting main execution loop.
2025-06-21 20:54:54,808 - WARNING - 'strategy_playbook.json' not found. Seeding a new one at: /home/mql5dev/Results/strategy_playbook.json
2025-06-21 20:54:54,808 - INFO -
============================== STARTING SINGLE RUN ==============================[/size]
[size=2]2025-06-21 20:54:54,808 - INFO - -> Loading Nickname Ledger...
2025-06-21 20:54:54,809 - INFO - Successfully loaded GEMINI_API_KEY from environment.
2025-06-21 20:54:54,809 - INFO - API Timer initialized with a 61-second interval.
2025-06-21 20:54:54,809 - INFO - -> Stage 1: Loading and Preparing Multi-Timeframe Data...
2025-06-21 20:54:54,809 - INFO -   - Found 9 data files to process.
2025-06-21 20:54:54,809 - INFO -     - [1/9] Parsing 'NDX100_Daily_202001060000_202506020000.csv'...
2025-06-21 20:54:54,817 - INFO -       - Successfully parsed. Shape: (1399, 9)
2025-06-21 20:54:54,817 - INFO -     - [2/9] Parsing 'XAUUSD_H1_202001060100_202506021800.csv'...
2025-06-21 20:54:54,900 - INFO -       - Successfully parsed. Shape: (31992, 10)
2025-06-21 20:54:54,900 - INFO -     - [3/9] Parsing 'XAUUSD_M15_202103181800_202506021830.csv'...
2025-06-21 20:54:55,137 - INFO -       - Successfully parsed. Shape: (99442, 10)
2025-06-21 20:54:55,137 - INFO -     - [4/9] Parsing 'NDX100_M15_202102250100_202506021830.csv'...
2025-06-21 20:54:55,390 - INFO -       - Successfully parsed. Shape: (100683, 10)
2025-06-21 20:54:55,390 - INFO -     - [5/9] Parsing 'US30_Daily_202001060000_202506020000.csv'...
2025-06-21 20:54:55,397 - INFO -       - Successfully parsed. Shape: (1399, 9)
2025-06-21 20:54:55,397 - INFO -     - [6/9] Parsing 'US30_M15_202102250100_202506021830.csv'...
2025-06-21 20:54:55,662 - INFO -       - Successfully parsed. Shape: (100688, 10)
2025-06-21 20:54:55,663 - INFO -     - [7/9] Parsing 'US30_H1_202001060100_202506021800.csv'...
2025-06-21 20:54:55,751 - INFO -       - Successfully parsed. Shape: (32165, 10)
2025-06-21 20:54:55,751 - INFO -     - [8/9] Parsing 'XAUUSD_Daily_202001060000_202506020000.csv'...
2025-06-21 20:54:55,757 - INFO -       - Successfully parsed. Shape: (1396, 9)
2025-06-21 20:54:55,758 - INFO -     - [9/9] Parsing 'NDX100_H1_202001060100_202506021800.csv'...
2025-06-21 20:54:55,865 - INFO -       - Successfully parsed. Shape: (32157, 10)
2025-06-21 20:54:55,867 - INFO -   - Combined data for Daily: 4,194 rows for 3 symbols.
2025-06-21 20:54:55,891 - INFO -   - Combined data for H1: 96,314 rows for 3 symbols.
2025-06-21 20:54:55,970 - INFO -   - Combined data for M15: 300,813 rows for 3 symbols.
2025-06-21 20:54:55,970 - INFO - [SUCCESS] Data loading complete. Detected timeframes: ['Daily', 'H1', 'M15']
2025-06-21 20:54:55,971 - INFO - Dynamically determined timeframe roles: {'base': 'M15', 'medium': 'H1', 'high': 'Daily'}
2025-06-21 20:54:55,971 - INFO - -> Feature Caching is ENABLED. Checking for a valid cache...
2025-06-21 20:54:55,972 - INFO -   - No valid cache found. Engineering features...
2025-06-21 20:54:55,972 - INFO - -> Stage 2: Engineering Features...
2025-06-21 20:54:55,979 - INFO -   - [1/3] Processing all features for symbol: US30
2025-06-21 20:57:16,778 - INFO -     - Calculating meta-features (feature interactions)...
2025-06-21 20:57:16,788 - INFO -     - Detecting anomalies with Isolation Forest...
2025-06-21 20:57:17,497 - INFO -   - [2/3] Processing all features for symbol: NDX100
2025-06-21 20:59:39,466 - INFO -     - Calculating meta-features (feature interactions)...
2025-06-21 20:59:39,467 - INFO -     - Detecting anomalies with Isolation Forest...
2025-06-21 20:59:40,199 - INFO -   - [3/3] Processing all features for symbol: XAUUSD
2025-06-21 21:02:00,595 - INFO -     - Calculating meta-features (feature interactions)...
2025-06-21 21:02:00,596 - INFO -     - Detecting anomalies with Isolation Forest...
2025-06-21 21:02:01,407 - INFO -   - Concatenating data for all symbols...
2025-06-21 21:02:02,048 - INFO -   - Calculating cross-symbol features...
2025-06-21 21:02:02,169 - INFO -   - Applying PCA reduction to feature set...
2025-06-21 21:02:02,169 - WARNING - Applying PCA reduction. NOTE: For best practice, PCA should be fit on training data only to avoid lookahead bias.
2025-06-21 21:02:16,800 - INFO - Dataset small enough for standard PCA. Using fast in-memory method.
2025-06-21 21:02:19,362 - INFO - Standard PCA complete. Explained variance: 95.35%
2025-06-21 21:02:19,362 - INFO -   - Adding noise-contrastive features for diagnostics...
2025-06-21 21:02:19,394 - INFO -   - Applying final data shift and cleaning...
2025-06-21 21:02:20,127 - INFO - [SUCCESS] Feature engineering complete. Final dataset shape: (898575, 95)
2025-06-21 21:02:23,183 - INFO -   - Features and metadata saved to cache: /home/mql5dev/Cache/feature_cache.parquet
2025-06-21 21:02:23,183 - INFO - -> Performing data-driven market regime analysis...
2025-06-21 21:02:23,198 - WARNING -   - No regime model found. Training and saving a new one.
2025-06-21 21:02:26,854 - INFO -   - New regime model saved to /home/mql5dev/Results/regime_model.pkl
2025-06-21 21:02:26,868 - INFO -   - Current market condition diagnosed as: Regime_2
2025-06-21 21:02:26,869 - INFO - -> Fetching external macroeconomic context data...
2025-06-21 21:02:28,027 - INFO -   - Macro context generated: {'VIX': {'level': 20.62, '1_week_change_pct': 14.43}, 'DXY': {'level': 98.71, '1_week_change_pct': 0.81}, 'US10Y_YIELD': {'level': 4.38, '1_week_change_pct': 0.41}}
2025-06-21 21:02:28,027 - INFO -   - Calculating asset correlation matrix for AI context...
2025-06-21 21:02:28,299 - INFO - --- STRATEGIC REVIEW: Analyzing long-term strategy health...
2025-06-21 21:02:28,299 - INFO - --- STRATEGIC REVIEW: Insufficient history for a full review.
2025-06-21 21:02:28,299 - INFO -   - Making API call to 'get_initial_run_setup' at 21:02:28...
2025-06-21 21:02:28,299 - INFO - -> Performing Initial AI Analysis & Setup (Dynamic Gates & Labeling)...
2025-06-21 21:02:28,300 - INFO - Attempting to call Gemini API with model: gemini-2.0-flash
2025-06-21 21:02:30,439 - INFO - Successfully received response from model: gemini-2.0-flash
2025-06-21 21:02:30,440 - INFO - Successfully extracted JSON object using JSONDecoder.raw_decode.
2025-06-21 21:02:30,440 - INFO -   - Initial AI Analysis and Setup complete.
2025-06-21 21:02:30,440 - INFO -   - AI selected Labeling Method: 'volatility_adjusted'
2025-06-21 21:02:30,440 - INFO -   - AI set dynamic F1 Score Gate: 0.45
2025-06-21 21:02:30,440 - INFO -   - API call to 'get_initial_run_setup' complete.
2025-06-21 21:02:30,440 - INFO - --- Run Initialized: Excalibur | Strategy: MeanReversionZScore ---
2025-06-21 21:02:30,440 - INFO - --- Dynamic Config: Labeling Method='volatility_adjusted', F1 Gate='0.45' ---
2025-06-21 21:02:30,442 - INFO - Using valid frequency alias from AI: '90D'
2025-06-21 21:02:30,442 - INFO -
--- Starting Cycle [1/14] for strategy 'MeanReversionZScore' ---
2025-06-21 21:02:30,443 - INFO - Using valid frequency alias from AI: '90D'
2025-06-21 21:02:30,479 - INFO - -> Stage 3: Generating Trade Labels ('volatility_adjusted')...
    - Labeling (Vol-Adj) 'NDX100': 100.0% complete... Done.
    - Labeling (Vol-Adj) 'US30': 100.0% complete... Done.
    - Labeling (Vol-Adj) 'XAUUSD': 100.0% complete... Done.
2025-06-21 21:02:33,105 - INFO -   - Label Sanity Check Passed. Distribution: Longs=39.83%, Shorts=38.26%
2025-06-21 21:02:33,106 - INFO -   - Starting model training using strategy: '[MEAN REVERSION] Exploits statistical deviations from the mean, entering when RSI reaches an extreme Z-score in a non-trending market. Example Features: `RSI_zscore`, `bollinger_bandwidth`, `stoch_k`, `market_regime`.'
2025-06-21 21:02:33,165 - INFO -     - Starting multi-objective optimization with 5-Fold CV (Objectives: Calmar, -Trades; 30 trials)...[/size]
[size=2][/size]

SPREAD_CONFIG: Dict[str, Dict[str, float]] = Field(default_factory=lambda: {
        'default': {'normal_pips': 1.8, 'volatile_pips': 5.5},
        'EURUSD':  {'normal_pips': 1.2, 'volatile_pips': 4.0},
        'GBPUSD':  {'normal_pips': 1.6, 'volatile_pips': 5.0},
        'AUDUSD':  {'normal_pips': 1.4, 'volatile_pips': 4.8},
        'USDCAD':  {'normal_pips': 1.7, 'volatile_pips': 5.5},
        'USDJPY':  {'normal_pips': 1.3, 'volatile_pips': 4.5},
        'AUDCAD':  {'normal_pips': 1.9, 'volatile_pips': 6.0},
        'AUDNZD':  {'normal_pips': 2.2, 'volatile_pips': 7.0},
        'NZDJPY':  {'normal_pips': 2.0, 'volatile_pips': 6.5},
        'XAUUSD_M15':    {'normal_pips': 25.0, 'volatile_pips': 80.0},
        'XAUUSD_H1':     {'normal_pips': 20.0, 'volatile_pips': 70.0},
        'XAUUSD_Daily':  {'normal_pips': 18.0, 'volatile_pips': 60.0},
        'US30_M15':      {'normal_pips': 50.0, 'volatile_pips': 150.0},
        'US30_H1':       {'normal_pips': 45.0, 'volatile_pips': 140.0},
        'US30_Daily':    {'normal_pips': 40.0, 'volatile_pips': 130.0},
        'NDX100_M15':    {'normal_pips': 20.0, 'volatile_pips': 60.0},
        'NDX100_H1':     {'normal_pips': 18.0, 'volatile_pips': 55.0},
        'NDX100_Daily':  {'normal_pips': 16.0, 'volatile_pips': 50.0},
    })

Quoting idiamond

Disliked

Awesome thread!! It's new to me, but it seems very powerful. I'm trying it out for the first time but am having difficulty getting the MQL Language files from Gemini for a simple Expert Advisor. The mq5 and mqh files turn out empty?? I'll have to learn more.

Ignored

Quoting amvt

Disliked

My inspiration from your idea This might be a long post, so please bear with me. It's important that I discuss this with someone who can truly understand the potential of what I'll be talking about and takes enough level of interest to collaborate. Currently, I am in the process of making my own Algorithm. I started about a month ago and have journaled all of my process in the Auction Market Value Theory forum. It's still in progress and god knows when it'll be finished. But it's coming to an end in the very near future. Hopefully. After I finish...

Ignored

Quoting Bl4ckP3n9u1n

Disliked

{quote} Sorry for the quick reply, but in essence, this is what the framework does. It uses micro / macro indicators, correlations, patterns, external market forces etc with ML leveraging AI (Gemini) to make the smart decisions to develop unique trading strategies. I'll reply you post in more detail early in the week. Have a read of my 'updated update' to understand the Feature Engineer which is what you've described to a tee.

Ignored