Uncovering Hidden Causal Relationships in Financial Markets: A Guide to Ensuring Robust AI Investment Strategies with DoWhy

Predictive models in financial markets often rely on correlations, leading to misleading results. With DoWhy, we can go beyond simple predictions to identify the true causal relationships hidden behind the data. Based on this understanding, we can build robust and explainable AI investment strategies that remain stable even in volatile markets. This is a key to innovatively enhancing the reliability and efficiency of AI-driven financial systems.

1. The Pitfall of Correlation, and the Necessity of Causation

The introduction of advanced AI models in financial markets is no longer a new phenomenon. Machine learning (ML) and deep learning (DL) models sometimes show astonishing predictive performance in various areas such as time series forecasting, portfolio optimization, and high-frequency trading. However, whether the "predictions" these models show actually lead to "explanation" or "understanding" is a separate issue. We often misunderstand that "sentiment analysis causes stock price increases" when a positive sentiment score in news articles shows a strong correlation with stock price increases.

However, financial markets are systems where countless complex factors are intertwined, and there is no guarantee that two variables showing a strong correlation today will do so tomorrow. Even if there is a statistically significant correlation, it is highly likely to be caused by a third hidden variable (confounder) or simply by chance. Such correlation-based strategies easily collapse when market conditions change or new information is introduced, which can lead to massive losses.

To ensure the robustness of AI investment strategies, we must be able to answer the causal question: "What causes what?" It is crucial to understand what impact interest rate hikes actually have on the stock prices of specific industry groups, or whether an increase in a company's ESG score has a direct causal relationship with long-term investment returns. This causal understanding is essential for us to answer "What-if" questions, respond more flexibly to market changes, and clearly identify the causes of strategy failures.

2. Deep Dive: DoWhy and the Power of Causal Inference

DoWhy, a Python library developed by Microsoft, simplifies the complex process of Causal Inference into a systematic and intuitive 4-step framework. This helps even non-domain experts explore and verify inherent causal relationships in data. The core of DoWhy lies in the following 4-step process:

1. Model: Define hypothetical causal relationships between variables based on domain knowledge using a Causal Graph. Specify which variables are the cause (Treatment), which are the outcome, and which are confounders.
2. Identify: Based on the defined causal graph and assumptions, identify statistically estimable causal effects (Estimand). This step involves finding a mathematical expression to calculate the causal effect.
3. Estimate: Estimate the identified causal effect using actual data and various statistical/machine learning techniques (e.g., Propensity Score Matching, Instrumental Variables, G-estimation, etc.).
4. Refute: Verify how robust the estimated causal effect is to changes in assumptions or noise in the data. This is a very important step to increase the reliability of causal inference results.

DoWhy integrates these four steps, supporting not only the estimation of causal effects but also the verification of their robustness. This plays a decisive role in avoiding misleading conclusions and deriving more reliable insights in noisy and complex systems like financial markets.

3. Step-by-Step Guide / Implementation: Causal Relationship Analysis using DoWhy

In this section, we will implement with DoWhy the process of analyzing the causal effect of changes in a specific economic indicator (e.g., manufacturing PMI) on the stock price of a particular industry sector (e.g., tech stocks) using hypothetical financial data. We aim to estimate the pure causal effect, beyond just the correlation between PMI and tech stock prices, by controlling for other macroeconomic variables (e.g., interest rates, consumer sentiment).

Step 1: Environment Setup and Data Preparation

Install DoWhy and necessary libraries, and generate hypothetical financial data for analysis. In a real environment, actual market data and economic indicators should be collected and used.


# 필요한 라이브러리 설치
# pip install dowhy econml pandas networkx matplotlib

import pandas as pd
import numpy as np
import dowhy
from dowhy import CausalModel
import networkx as nx
import matplotlib.pyplot as plt

# 1. 가상의 금융 데이터 생성 (시뮬레이션)
np.random.seed(42)
n_samples = 1000

# 교란 변수(Confounder): 금리 (interest_rate), 소비자 심리 지수 (consumer_sentiment)
# 제조업 PMI (treatment), 기술주 주가 (outcome)
interest_rate = np.random.normal(loc=0.05, scale=0.01, size=n_samples)
consumer_sentiment = np.random.normal(loc=100, scale=10, size=n_samples)

# 제조업 PMI (Treatment): 금리, 소비자 심리에 영향을 받음 (가정)
manufacturing_pmi = (
    120
    - 500 * interest_rate
    + 0.5 * consumer_sentiment
    + np.random.normal(loc=0, scale=5, size=n_samples)
)

# 기술주 주가 (Outcome): PMI, 금리, 소비자 심리에 영향을 받음 (가정)
tech_stock_price = (
    500
    + 2 * manufacturing_pmi  # PMI가 주가에 긍정적인 인과 효과 (가정)
    - 2000 * interest_rate   # 금리가 주가에 부정적인 인과 효과 (교란)
    + 1.5 * consumer_sentiment # 소비자 심리가 주가에 긍정적인 인과 효과 (교란)
    + np.random.normal(loc=0, scale=20, size=n_samples)
)

data = pd.DataFrame({
    'interest_rate': interest_rate,
    'consumer_sentiment': consumer_sentiment,
    'manufacturing_pmi': manufacturing_pmi,
    'tech_stock_price': tech_stock_price
})

print("데이터 미리보기:")
print(data.head())

Step 2: Causal Model Definition

Define the causal graph between variables based on the data. We want to know if manufacturing PMI affects tech stock prices, and we assume that interest rates and consumer sentiment are confounders affecting both. DoWhy allows defining causal graphs using GML (Graphical Model Language) or dot language.


# 인과 모델 정의
# treatment: manufacturing_pmi (원인)
# outcome: tech_stock_price (결과)
# common_causes: interest_rate, consumer_sentiment (교란 변수)

# GML 언어를 사용하여 인과 그래프 정의
# graph = """
# digraph {
#     interest_rate -> manufacturing_pmi;
#     consumer_sentiment -> manufacturing_pmi;
#     interest_rate -> tech_stock_price;
#     consumer_sentiment -> tech_stock_price;
#     manufacturing_pmi -> tech_stock_price;
# }
# """
# DoWhy는 더 간편하게 인과 관계를 설정할 수 있습니다.

model = CausalModel(
    data=data,
    treatment='manufacturing_pmi',
    outcome='tech_stock_price',
    common_causes=[
        'interest_rate',
        'consumer_sentiment'
    ]
    # 기타 변수들을 정의하고 싶다면 additional_variables=[...] 사용
)

# 인과 그래프 시각화 (optional, requires graphviz)
# model.view_model()
# plt.show() # 이미지를 직접 확인하려면 주석 해제
print("\n인과 모델이 성공적으로 정의되었습니다.")

Step 3: Identification Strategy for Causal Effect

Identify methods to statistically estimate the causal effect from the defined causal model. DoWhy automatically applies methods like the Back-door Criterion based on the given causal graph and assumptions to find identifiable causal effects (Estimand).


# 인과 효과 식별
identified_estimand = model.identify_effect(
    estimand_type="ate", # 평균 처리 효과 (Average Treatment Effect)
    # method_name="backdoor", # DoWhy가 자동으로 적절한 방법을 찾음
    # proceed_when_unidentifiable=True # 식별 불가능하더라도 진행
)

print("\n식별된 인과 효과:")
print(identified_estimand)

Step 4: Estimating the Causal Effect

Calculate the identified causal effect using actual data and various estimation techniques. DoWhy supports several estimators such as Linear Regression, Inverse Probability Weighting (IPW), and Propensity Score Matching (PSM). Here, we will use linear regression to estimate the effect after controlling for confounders.


# 인과 효과 추정 (선형 회귀 사용)
# method_name: 'backdoor.linear_regression'은 백도어 경로를 닫기 위해 선형 회귀를 사용한다는 의미
causal_estimate = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.linear_regression",
    control_value=data['manufacturing_pmi'].median(), # 비교 기준점
    treatment_value=data['manufacturing_pmi'].median() + data['manufacturing_pmi'].std(), # 처리 효과 평가 지점
    target_units="ate" # 평균 처리 효과
)

print("\n추정된 인과 효과:")
print(f"제조업 PMI의 1 표준편차 증가가 기술주 주가에 미치는 평균 인과 효과: {causal_estimate.value:.2f}")

# 다른 추정기 예시: Propensity Score Matching
# causal_estimate_psm = model.estimate_effect(
#     identified_estimand,
#     method_name="backdoor.propensity_score_matching",
#     target_units="ate"
# )
# print(f"PSM으로 추정된 인과 효과: {causal_estimate_psm.value:.2f}")

Step 5: Refuting the Causal Estimate

This step verifies how sensitive the estimated causal effect is to our assumptions or data characteristics. DoWhy provides various refutation methods to increase the reliability of the results. This is especially important in highly uncertain environments like financial markets.

Random Common Cause: Checks how much the result changes when a randomly generated confounder is added to existing confounders.
Random Replace Treatment: Checks if the causal effect disappears when the treatment variable is replaced with a random variable. If it does not disappear, there might be an issue with the original estimate.
Data Subset Refuter: Checks how consistent the results are when estimated using a subset of the data.


# 인과 추정 결과 강건성 검증

# 1. 무작위 교란 변수 추가 (Adding a random common cause)
refute_random_common_cause = model.refute_estimate(
    identified_estimand, causal_estimate,
    method_name="random_common_cause"
)
print("\n[Refutation 1] 무작위 교란 변수 추가:")
print(refute_random_common_cause)

# 2. 원인 변수를 무작위 변수로 대체 (Replacing treatment with a random variable)
refute_random_treatment = model.refute_estimate(
    identified_estimand, causal_estimate,
    method_name="random_treatment_on_outcome"
)
print("\n[Refutation 2] 원인 변수를 무작위 변수로 대체:")
print(refute_random_treatment)

# 3. 데이터 서브셋 사용 (Data subset refuter)
refute_subset = model.refute_estimate(
    identified_estimand, causal_estimate,
    method_name="data_subset_refuter",
    subset_fraction=0.8 # 데이터의 80%만 사용
)
print("\n[Refutation 3] 데이터 서브셋 사용:")
print(refute_subset)

# refutation 결과 해석:
# - new_effect가 original_effect와 크게 다르지 않다면 강건하다고 볼 수 없음.
# - Random common cause: 새로운 교란 변수가 추가되었을 때 결과가 안정적이면 좋음.
# - Random treatment: 처리 효과가 0에 가까워져야 함 (처리 효과가 우연이 아님을 의미).
# - Data subset refuter: 서브셋에서도 원래 결과와 유사해야 함.

4. Real-world Use Case / Example: Causal Relationship between News Sentiment Analysis and Stock Price Fluctuations

In financial markets, News Sentiment Analysis is utilized as a crucial element of investment strategies. Many quantitative models generate trading signals under the hypothesis that if news sentiment for a specific company or industry is positive, its stock price will rise, and if negative, it will fall. However, such strategies often become powerless in the face of market Regime Shifts or unforeseen macroeconomic shocks. One case I experienced was when, despite consistently high positive news sentiment indicators for tech stocks, they plummeted significantly during a period of rapid interest rate hikes by the Federal Reserve. Here, news sentiment continued to have a positive correlation with stock prices, but its causal influence was overshadowed or weakened by the larger confounding variable of interest rate hikes.

By leveraging DoWhy, we can build more robust strategies in such situations. For example:

Treatment: Change in news sentiment score for a specific company/industry
Outcome: Stock return of that company/industry
Confounders: Overall market return, sector return, interest rate changes, exchange rate changes, VIX index, changes in the company's financial indicators, etc.

Through DoWhy's causal graph, we can specify not only the direct path between news sentiment and stock prices but also the indirect effects of overall market factors on this relationship. For instance, positive news may boost stock prices, but at the same time, concerns about interest rate hikes can worsen overall market investor sentiment, offsetting that effect. DoWhy enables us to isolate and estimate the pure causal effect of news sentiment scores in such complex situations. If, through the Refutation stage, the causal effect estimate changes significantly when a specific macroeconomic variable is added, we know that we must include that variable in the model to control for confounding effects.

My personal insight is that we need to change the question from simply "What indicator predicts stock prices?" to "What indicator causes stock prices to move under what conditions?" DoWhy provides a systematic framework to answer this question, which is particularly useful for preparing for situations where past correlations break down in black swan events or rapidly changing market environments. In other words, DoWhy contributes to simultaneously enhancing the Explainability and Robustness of investment strategies by answering the 'why' question, beyond blindly trusting predictive accuracy.

5. Pros & Cons / Critical Analysis

Pros:
- Robust Strategy Building: By basing strategies on causal relationships rather than correlations, it reduces the vulnerability of predictive models and allows for the establishment of strategies resilient to market volatility.
- Improved Explainability: Provides clear causal explanations for "why a certain outcome occurred," increasing the reliability of investment decisions. This also offers significant advantages for regulatory compliance (e.g., AI ethics, explainable AI).
- "What-if" Scenario Analysis: Facilitates hypothetical scenario analysis of how the outcome changes when a specific variable (Treatment) is altered.
- Systematic Framework: Structures the complex process of causal inference into four stages: modeling, identification, estimation, and refutation, making it more accessible.
- Open Source & Active Community: Developed by Microsoft as an open-source library, it benefits from continuous development and community support.
Cons:
- Domain Knowledge Requirement: Accurate definition of the causal graph requires deep domain knowledge of the field (financial markets) and the ability to formulate hypotheses. An incorrect graph leads to incorrect inferences.
- Data Quality and Availability: High-quality data that can accurately measure and include confounders is crucial. In financial markets, it can be difficult to obtain all relevant variables.
- Complexity of High-Dimensional Data: As the number of variables becomes very large, the definition of the causal graph and the estimation process can become complex.
- Time Complexity: The computational cost and time of causal inference can be burdensome for direct application in real-time decision-making systems like high-frequency trading. (Primarily used in strategy development and validation stages).
- Still in Research Phase: Some causal

Uncovering Hidden Causality in Financial Markets: A Guide to Building Robust AI Investment Strategies with DoWhy