Disliked{quote} I haven't worked on 'STREAMLINED' for a while now but this was the last 'version' I have. There's chances of bugs still and I think there's a memory leak. I posted about that issue last week. {file}Ignored
This release is just to provide some insight into the possibilities letting AI find patterns in the data.
Attached:
- AxiomEdge_PRO_V211_2_STREAMLINED.py.txt
By simplifying the feature engineering, I removed a layer of pre-defined complexity and shifted the "intelligence" to the AI's dynamic pattern discovery.
Complex Features Left Out
The feature engineering process was streamlined by removing functions that calculate highly specialized, derivative, or complex indicators. These often represent fixed, pre-conceived ideas about market behaviour. The core idea is to provide the AI with fundamental building blocks rather than complex, opinionated signals.
The main categories of removed features include:
- Advanced Statistical & Econometric Models: Features derived from models like GARCH (for volatility forecasting), PACF (autocorrelation), Hawkes Processes (self-excitation), and information theory metrics like Shannon Entropy were excluded.
- Frequency & Cyclical Analysis: Complex signal processing techniques like Fourier, Wavelet, and Hilbert Transforms, which attempt to find dominant market cycles, were removed.
- Market Microstructure Features: Proxies for order flow, market depth, and liquidity (e.g., volume_imbalance, estimated_spread) were left out. These are often noisy and difficult to model without true Level 2 data.
- Pre-defined Technical Patterns: Specific, hardcoded patterns like trend pullbacks and indicator divergences were removed, as the AI is now tasked with discovering these types of relationships on its own.
- Advanced Volatility Metrics: Academic volatility estimators like Parkinson and Yang-Zhang were replaced by simpler, more robust metrics like ATR and realized volatility.
- Signal Enhancement: Post-processing steps like applying a Kalman Filter to smooth indicators were removed to provide the AI with more raw, unfiltered inputs.
Benefits of AI-Discovered Features
Introducing a larger set of AI-discovered features built from a "bare essential" base has several key advantages:
Adaptability to Market Regimes: This is the most significant benefit. The AI generates features specifically tailored to the current diagnosed market regime (e.g., "Trending," "Ranging"). Hardcoded features are static and may fail when market conditions change, whereas the AI's features are dynamic and context-aware.
Discovery of Novel Patterns: The AI can identify complex, non-linear relationships between fundamental features (e.g., how session times interact with volatility and statistical skew) that a human might never think to code. It moves beyond standard technical analysis to find a unique, data-driven "edge."
Reduced Human Bias: The extensive list of removed features represents a developer's specific hypotheses about what drives markets. By reducing this list and empowering the AI, we lessen the influence of preconceived biases and allow the model to learn directly from the data's underlying structure.
Hypothesis Generation: The AI is prompted to provide a clear, testable hypothesis for every feature it creates. This transforms it from a "black box" into a powerful research assistant, helping you understand why a particular signal might be predictive and providing new avenues for strategy development.
The Updated Function 'def discover_behavioral_patterns'
Modify the 'Discovery number' in the OBJECTIVE to increase or decease the AI Discovered Features. Currently set between 15-25
def discover_behavioral_patterns(self, base_features: List[str], diagnosed_regime: str) -> Dict[str, Dict[str, str]]:
"""Engages the AI to discover novel non-technical or behavioral patterns."""
logger.info("-> Engaging AI to discover novel behavioral/statistical patterns...")
if not self.api_key_valid:
logger.warning(" - API key invalid or missing.")
return {}
# --- Step 1: Tag features into conceptual categories ---
feature_categories = {
"Time-Based": ['hour', 'day_', 'session', 'month', 'week_'],
"Statistical & Fractal": ['skew', 'kurtosis', 'entropy', 'hurst', 'pacf', 'garch', 'zscore'],
"Volatility": ['volatility', 'ATR', 'bollinger_bandwidth', 'bb_width', 'vix'],
"Structural & Event-Based": ['gap', 'displacement', 'breakout', 'structural_break', 'orb_'],
"Sentiment & Emotion": ['sentiment', 'fear_greed'],
"Behavioral Insights": ['drawdown_percent', 'anchor_price', 'insidebar'],
"Fundamental Price/Volume": ['Close', 'Open', 'High', 'Low', 'RealVolume', 'pct_change', 'prev_session_high', 'orb_high']
}
categorized_features = {
cat_name: [f for f in base_features if any(key in f for key in keywords)]
for cat_name, keywords in feature_categories.items()
}
# Logging categorized features
for category, feats in categorized_features.items():
logger.info(f" - {category}: {len(feats)} features")
# --- Step 2: Build the prompt to the AI ---
category_strings = "\n".join(
[f"{i+1}. {cat}: {json.dumps(feats)}"
for i, (cat, feats) in enumerate(categorized_features.items())]
)
prompt = f"""
You are a Quantitative Behavioral Analyst specializing in financial markets.
---
**OBJECTIVE**:
Discover 15-25 novel, non-obvious meta-features by combining base market features to capture **behavioral**, **time-sensitive**, or **statistical** phenomena. Avoid standard TA signals.
---
**CURRENT MARKET REGIME**:
- `{diagnosed_regime}` — Tailor hypotheses accordingly.
---
**FEATURE CONSTRUCTION RULES**:
- Combine features from **at least two different categories**.
- Include **a clear, testable hypothesis** for each feature.
- Use **safe lambda code**:
- Start with `lambda row:`
- Use `row.get('feature', default_value)`
- Use `max(..., 1e-6)` to avoid zero division.
- Do NOT use typical TA crossovers (e.g., MACD, RSI).
- Use creative combinations that reflect **market psychology, volatility structure, sentiment reaction**, etc.
---
**AVAILABLE FEATURE CATEGORIES**:{category_strings}
---
**RESPONSE FORMAT** (Return exactly this JSON format with 15–25 entries):
```json
{{
"feature_name_example": {{
"lambda": "lambda row: ...",
"description": "HYPOTHESIS: ..."
}}
}}
"""
# --- Step 3: Send prompt to model and parse response ---
response_text = self._call_gemini(prompt)
suggestions = self._extract_json_from_response(response_text)
# --- Step 4: Validate and return ---
if isinstance(suggestions, dict) and all(isinstance(v, dict) and 'lambda' in v and 'description' in v for v in suggestions.values()):
logger.info(f" - AI successfully proposed {len(suggestions)} new behavioral patterns.")
return suggestions
logger.error(" - AI failed to return a valid dictionary of behavioral patterns.")
return {} EXMAPLE AI DISCOVERY OUTPUT
2025-07-22 19:01:01,120 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:717 - - Making API call to 'discover_behavioral_patterns' at 19:01:01...
2025-07-22 19:01:01,120 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2028 - -> Engaging AI to discover novel behavioral/statistical patterns...
2025-07-22 19:01:01,120 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2051 - - Time-Based: 9 features
2025-07-22 19:01:01,120 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2051 - - Statistical & Fractal: 5 features
2025-07-22 19:01:01,120 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2051 - - Volatility: 6 features
2025-07-22 19:01:01,120 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2051 - - Structural & Event-Based: 10 features
2025-07-22 19:01:01,120 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2051 - - Sentiment & Emotion: 5 features
2025-07-22 19:01:01,120 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2051 - - Behavioral Insights: 1 features
2025-07-22 19:01:01,121 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2051 - - Fundamental Price/Volume: 3 features
2025-07-22 19:01:01,121 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:851 - Attempting to call Gemini API with model: gemini-2.5-flash-lite-preview-06-17
2025-07-22 19:01:09,960 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:871 - Successfully received and extracted text response from model: gemini-2.5-flash-lite-preview-06-17
2025-07-22 19:01:09,960 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:904 - Successfully extracted JSON object from within backticks.
2025-07-22 19:01:09,960 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2106 - - AI successfully proposed 21 new behavioral patterns.
2025-07-22 19:01:09,960 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:719 - - API call to 'discover_behavioral_patterns' complete.
2025-07-22 19:01:09,960 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2760 - -> Injecting 21 AI-discovered alpha features...
2025-07-22 19:01:10,387 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'vol_skew_interaction' (Desc: HYPOTHESIS: In Regime_2, periods of high realized volatility combined with negative skew (indicating heavier left tails and potential for sharp downside moves) might signal increased fear and panic, leading to further downward pressure or a shift in trend.)
2025-07-22 19:02:13,404 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'vix_anchor_deviation' (Desc: HYPOTHESIS: In Regime_2, when the VIX (a measure of expected volatility) deviates significantly from a perceived anchor price (representing stability or a reference point), it may indicate a disconnect between expected fear and current market pricing, potentially preceding a correction or volatility event.)
2025-07-22 19:03:14,654 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'sentiment_overnight_gap' (Desc: HYPOTHESIS: In Regime_2, a positive sentiment score coupled with a significant overnight price gap (up or down) could indicate that existing sentiment is either amplified or challenged by overnight news/events, potentially leading to follow-through or reversal.)
2025-07-22 19:04:15,939 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'hurst_vix_ratio' (Desc: HYPOTHESIS: In Regime_2, a declining Hurst exponent (suggesting increased randomness) coupled with a rising VIX might indicate a market losing its trending behavior and becoming more erratic, a precursor to increased volatility and potential range-bound trading or sharp reversals.)
2025-07-22 19:05:18,523 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'orb_breakout_sentiment' (Desc: HYPOTHESIS: In Regime_2, ORB breakouts that occur with a positive sentiment score may be more likely to sustain, while breakouts with negative sentiment could be false signals or indicative of underlying weakness.)
2025-07-22 19:06:20,127 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'atr_day_of_week_interaction' (Desc: HYPOTHESIS: In Regime_2, the impact of daily volatility (ATR) might be amplified or dampened on certain days of the week, perhaps due to differing trader participation or event cycles, leading to predictable patterns in breakout strength or consolidation.)
2025-07-22 19:07:23,894 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'kurtosis_anchor_deviation' (Desc: HYPOTHESIS: In Regime_2, a high kurtosis (fat tails) combined with a low anchor price relative to the current market might suggest speculative excesses or significant outlier events are being priced in, potentially leading to mean reversion or volatility spikes.)
2025-07-22 19:08:27,040 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'london_ny_session_overlap_vol' (Desc: HYPOTHESIS: In Regime_2, the overlap of London and New York trading sessions typically sees increased volume and volatility. This feature aims to capture that heightened volatility specifically during the overlap, potentially signaling opportunities for faster price movements.)
2025-07-22 19:09:30,513 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'prev_session_vol_ratio' (Desc: HYPOTHESIS: In Regime_2, a significant increase in current realized volatility compared to the previous session's volatility could indicate a shift in market regime or the onset of a trending move, especially if driven by external factors.)
2025-07-22 19:10:41,054 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'fear_greed_vix_diff' (Desc: HYPOTHESIS: In Regime_2, a large divergence between the fear and greed index (sentiment) and the average VIX (expected volatility) could signal market irrationality. For instance, high fear and low VIX might suggest oversold conditions that could reverse, or vice-versa.)
2025-07-22 19:11:44,557 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'bollinger_breakout_sentiment_interaction' (Desc: HYPOTHESIS: In Regime_2, wide Bollinger Bandwidths coupled with positive sentiment might indicate strong trending potential, while wide bandwidths with negative sentiment could signal increased risk of a sharp reversal or consolidation.)
2025-07-22 19:12:49,092 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'RSI_zscore_hour_interaction' (Desc: HYPOTHESIS: In Regime_2, the typical intraday patterns of RSI divergence or convergence might be amplified or occur at different times of the day. This feature explores if extreme RSI Z-scores at certain hours predict reversals or continuations differently.)
2025-07-22 19:13:54,167 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'overnight_gap_hurst' (Desc: HYPOTHESIS: In Regime_2, large overnight gaps might behave differently depending on the market's underlying trend-following or mean-reverting tendencies (Hurst exponent). A large gap with a high Hurst might indicate trend continuation, while with a low Hurst, it could signal a reversal.)
2025-07-22 19:14:56,894 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'tweet_sentiment_ATR_ratio' (Desc: HYPOTHESIS: In Regime_2, the impact of social media sentiment on price action might be more pronounced when volatility is lower. This feature captures the ratio of sentiment to volatility, hypothesizing that high sentiment relative to low ATR could signal a sentiment-driven move.)
2025-07-22 19:16:03,286 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'breakout_H1_vix_diff' (Desc: HYPOTHESIS: In Regime_2, an upward H1 breakout occurring when the VIX is relatively low might be a stronger signal of bullish momentum than when the VIX is already elevated, suggesting less immediate fear or risk aversion.)
2025-07-22 19:17:08,843 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'sentiment_breakdown_H4_interaction' (Desc: HYPOTHESIS: In Regime_2, a significant downward H4 breakout accompanied by negative sentiment might indicate a strong bearish continuation, whereas a downward breakout with positive sentiment could signal a potential bear trap.)
2025-07-22 19:18:12,692 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'prev_session_high_pct_change' (Desc: HYPOTHESIS: In Regime_2, the difference between the current price change and the previous session's high might indicate the extent of momentum beyond previous resistance levels. A large positive difference suggests strong upward continuation.)
2025-07-22 19:19:16,527 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'vix_5d_avg_kurtosis' (Desc: HYPOTHESIS: In Regime_2, elevated kurtosis (fat tails) combined with a higher average VIX might indicate a market where extreme moves are more probable and potentially correlated with broader market fear, signaling increased risk.)
2025-07-22 19:20:19,665 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'day_of_week_sentiment_interaction' (Desc: HYPOTHESIS: In Regime_2, specific days of the week might exhibit different reactions to prevailing sentiment. For instance, positive sentiment on a historically weak day could be a contrarian indicator.)
2025-07-22 19:21:24,235 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'hurst_intercept_volatility_interaction' (Desc: HYPOTHESIS: In Regime_2, the interplay between the Hurst intercept (indicating persistent behavior) and realized volatility could reveal the market's sensitivity to underlying trends. A high intercept with high volatility might suggest trend acceleration.)
2025-07-22 19:22:28,407 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:2783 - - Applying pattern: 'orb_low_sentiment_ratio' (Desc: HYPOTHESIS: In Regime_2, a low ORB opening range low relative to positive sentiment might suggest a weak market that is being supported by optimistic sentiment, potentially a reversal signal. Conversely, a low ORB low with negative sentiment could confirm downside.)
2025-07-22 19:23:32,900 - INFO - AxiomEdge_PRO_V211_2_STREAMLINED.py:7698 - Successfully injected 21 new behavioral features into the main dataframe.