Dynamic Portfolio Optimization Based on Reinforcement Learning (RL): Building Autonomous Investment Strategies that Adapt to Financial Market Changes

Static portfolio strategies are revealing their limitations in unpredictable financial markets. Reinforcement Learning (RL) presents a new paradigm for autonomously optimizing portfolios by actively responding to real-time market changes. This article deeply explores practical methodologies for utilizing RL to maximize investment returns and manage risk in volatile markets.

1. The Challenge / Context

Financial markets are inherently non-linear, non-stationary, and complex systems that are constantly changing. Traditional portfolio theory, such as Markowitz's mean-variance optimization, focuses on deriving a fixed asset allocation based on market data at a specific point in time. However, such static approaches struggle to effectively respond to rapidly changing market conditions, such as interest rate hikes, geopolitical risks, and technological innovations. Investors are keenly aware of the need for dynamic strategies that can detect subtle market changes and adjust portfolios quickly and objectively in response.

The problem is that such dynamic adjustments are very difficult with human intervention alone. Humans are susceptible to emotions and have limitations in analyzing vast amounts of data in real-time and making optimal decisions. It is at this point that reinforcement learning emerges as a powerful solution. RL is a machine learning paradigm where an agent learns an optimal behavior policy through trial and error by interacting with an environment, making it one of the most suitable tools for the complexity and dynamism of financial markets.

2. Deep Dive: Dynamic Portfolio Optimization Based on Reinforcement Learning

Dynamic portfolio optimization based on reinforcement learning goes beyond simply predicting asset prices; it is a process of learning a 'policy' on what investment actions (buy, sell, hold) to take in a specific market state to achieve optimal long-term rewards. The key components are as follows:

Agent: Acts as the portfolio manager. It observes market data and makes investment decisions according to the learned policy.
Environment: The financial market itself. All market information, including stock prices, trading volume, technical indicators, and macroeconomic data, forms part of the environment.
State: The current situation of the environment observed by the agent. This can include OHLCV (Open, High, Low, Close, Volume) data for a certain past period, technical indicators (RSI, MACD), current portfolio composition, and cash holdings.
Action: The decisions an agent can take in the market. This includes buy/sell/hold decisions for each asset, or adjusting the weight of each asset within the portfolio (rebalancing). It can be defined as a continuous action space (e.g., asset weights between 0 and 1) or a discrete action space (e.g., buy, sell, hold).
Reward: Feedback given as a result of the agent's actions. Changes in portfolio returns, Sharpe ratio, reduction in Max Drawdown, and net profit considering transaction costs can be used as reward functions. The reward function is crucial as it defines the 'goal' for the agent to learn.
Policy: The agent's strategy on what action to take in a specific state. RL aims to optimize this policy to maximize long-term cumulative rewards.

Various RL algorithms such as Deep Q-Network (DQN), Proximal Policy Optimization (PPO), and Advantage Actor-Critic (A2C) can be applied, and recently, Transformer-based Sequence Models are also being utilized for state representation. These algorithms contribute to learning complex market patterns and non-linear relationships to derive optimal investment policies that are difficult for humans to predict.

3. Step-by-Step Guide / Implementation

The process of building a reinforcement learning-based dynamic portfolio optimization system is as follows. We will explain it focusing on examples using Python and the stable-baselines3, Gym libraries.

Step 1: Data Collection and Preprocessing

The core of portfolio optimization is high-quality data. Historical stock prices (OHLCV), trading volume, technical indicators (moving averages, RSI, MACD, etc.), and even news data or corporate financial data can be utilized. After collecting the data, it must be preprocessed into a suitable format for model training.


import pandas as pd
import numpy as np
import yfinance as yf # 주식 데이터 수집 라이브러리 예시

# 데이터 수집 (예시: Apple, Microsoft, Google 주식)
tickers = ['AAPL', 'MSFT', 'GOOG']
start_date = '2010-01-01'
end_date = '2023-01-01'

data = yf.download(tickers, start=start_date, end=end_date)['Adj Close']
# 결측치 처리 및 수익률 계산 (NaN이 있으면 RL 환경 구성에 문제 발생)
data = data.dropna()
returns = data.pct_change().dropna()

# 기술적 지표 추가 (예시: 단순 이동 평균)
for ticker in tickers:
    data[f'{ticker}_SMA20'] = data[ticker].rolling(window=20).mean()
    data[f'{ticker}_RSI'] = ... # RSI 계산 로직 추가
    # ... 기타 지표 추가

# 학습에 사용할 최종 데이터셋 준비
# RL 상태(State)는 주로 가격, 지표, 포트폴리오 상태 등을 포함
# 여기서는 간단히 수정 종가와 기술적 지표를 활용한다고 가정
print("Data head:")
print(data.head())

Step 2: Defining the Reinforcement Learning Environment (Custom Gym Environment)

The OpenAI Gym interface provides a standardized environment for training RL agents. We build a custom portfolio optimization environment by inheriting from gym.Env. This environment is a core part that defines the agent's 'state', 'action', and 'reward'.

Observation Space: Market and portfolio information observed by the agent. This can include the value of currently held assets, cash balance, and market indicators. It is generally represented by continuous values.
Action Space: The actions an agent can take. It is defined as continuous values that directly adjust the weight of each asset in the portfolio (e.g., expressing buy/sell intensity with values between [-1, 1]) or discrete values (e.g., 'buy', 'sell', 'hold' decisions for each asset).
Reward Function: Defines the agent's goal. Typically, it uses the portfolio's daily return, Sharpe ratio, or return including negative transaction costs.


import gym
from gym import spaces

class StockPortfolioEnv(gym.Env):
    def __init__(self, df, initial_amount=100000, transaction_cost_rate=0.001):
        super(StockPortfolioEnv, self).__init__()
        self.df = df # 전처리된 데이터프레임
        self.stock_dim = len(df.columns) # 자산 개수
        self.initial_amount = initial_amount
        self.transaction_cost_rate = transaction_cost_rate
        self.current_step = 0

        # 행동 공간 정의: 각 자산에 대한 투자 비중 (0~1 사이)
        # 마지막은 현금 비중. 총합이 1이 되도록 함.
        self.action_space = spaces.Box(low=0, high=1, shape=(self.stock_dim + 1,), dtype=np.float32)

        # 상태 공간 정의: [포트폴리오 가치, 보유 현금, 각 자산 가격, 각 자산 보유 수량] + 시장 지표
        # 간단하게 현재 자산 가격과 보유 자산 수량, 현금 비중으로 구성
        # 실제 구현에서는 df의 모든 정보와 포트폴리오 가치/현금 등을 포함
        self.observation_space = spaces.Box(
            low=0, high=np.inf, shape=(self.stock_dim * 2 + 2,), dtype=np.float32
        )

        self.reset()

    def reset(self):
        self.current_step = 0
        self.portfolio_value = self.initial_amount
        self.cash_in_hand = self.initial_amount
        self.holdings = np.zeros(self.stock_dim) # 각 자산 보유 수량

        # 초기 상태 반환
        return self._get_observation()

    def _get_observation(self):
        # 현재 시장 가격
        current_prices = self.df.iloc[self.current_step].values[:self.stock_dim]
        # 현재 보유 자산 가치
        current_holdings_value = self.holdings * current_prices
        
        # 상태 = [현재 포트폴리오 가치, 현재 현금, 각 자산 가격, 각 자산 보유 수량]
        # 실제 구현에서는 기술적 지표 등을 추가
        return np.concatenate([
            [self.portfolio_value],
            [self.cash_in_hand],
            current_prices,
            self.holdings
        ])

    def step(self, actions):
        # 액션 정규화: 모든 비중의 합이 1이 되도록 (softmax 등)
        normalized_actions = actions / np.sum(actions)

        # 다음 스텝으로 이동
        self.current_step += 1
        if self.current_step >= len(self.df) - 1:
            done = True
        else:
            done = False

        # 다음 시점의 시장 가격
        next_prices = self.df.iloc[self.current_step].values[:self.stock_dim]
        
        # 이전 포트폴리오 가치 계산
        prev_portfolio_value = self.cash_in_hand + np.sum(self.holdings * self.df.iloc[self.current_step - 1].values[:self.stock_dim])

        # 행동 (리밸런싱) 수행
        # 총 포트폴리오 가치에 따라 각 자산 비중 재조정
        target_values = normalized_actions[:-1] * self.portfolio_value # 현금 제외한 자산 비중
        target_cash = normalized_actions[-1] * self.portfolio_value # 현금 비중

        # 각 자산에 대해 매수/매도 결정
        current_holdings_value = self.holdings * self.df.iloc[self.current_step - 1].values[:self.stock_dim]
        
        transaction_cost = 0
        for i in range(self.stock_dim):
            diff = target_values[i] - current_holdings_value[i]
            if diff > 0: # 매수
                num_shares_to_buy = diff / self.df.iloc[self.current_step - 1].values[i]
                cost = num_shares_to_buy * self.df.iloc[self.current_step - 1].values[i]
                transaction_cost += cost * self.transaction_cost_rate
                self.cash_in_hand -= cost
                self.holdings[i] += num_shares_to_buy
            elif diff < 0: # 매도
                num_shares_to_sell = abs(diff) / self.df.iloc[self.current_step - 1].values[i]
                revenue = num_shares_to_sell * self.df.iloc[self.current_step - 1].values[i]
                transaction_cost += revenue * self.transaction_cost_rate
                self.cash_in_hand += revenue
                self.holdings[i] -= num_shares_to_sell
        
        # 현금 비중 조정
        cash_diff = target_cash - self.cash_in_hand
        if cash_diff > 0: # 현금 추가 매수 개념
            # 이 로직은 실제 현금을 주식으로 바꾸는 것과 반대이므로,
            # 실제로는 현금 비중이 낮아지면 매수, 높아지면 매도 형태로 구현해야 함.
            # 여기서는 단순히 현금 목표 비중과의 차이를 반영한다고 가정
            pass 

        # 현재 포트폴리오 가치 업데이트
        self.portfolio_value = self.cash_in_hand + np.sum(self.holdings * next_prices)

        # 보상 계산: (현재 포트폴리오 가치 / 이전 포트폴리오 가치) - 1 - 거래 비용
        reward = (self.portfolio_value / prev_portfolio_value) - 1 - (transaction_cost / prev_portfolio_value)
        
        # 새로운 상태 반환
        observation = self._get_observation()
        info = {} # 추가 정보

        return observation, reward, done, info

    def render(self, mode='human'):
        # 포트폴리오 가치 등을 출력
        print(f"Step: {self.current_step}, Portfolio Value: {self.portfolio_value:.2f}")

    def close(self):
        pass

# 환경 테스트
# env = StockPortfolioEnv(data)
# obs = env.reset()
# print(f"Initial Observation: {obs}")
# action = env.action_space.sample() # 무작위 행동
# obs, reward, done, info = env.step(action)
# print(f"First Step - Reward: {reward}, Done: {done}, New Obs: {obs}")

Step 3: RL Agent Selection and Training

Once the environment is ready, you can select and train an RL agent using libraries like stable-baselines3. PPO (Proximal Policy Optimization) is widely used as a stable and effective algorithm.


from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv

# 데이터 준비 (예시를 위해 Simple Env 사용)
# 실제로는 Step 1에서 전처리한 data DataFrame을 사용
# 여기서는 data DataFrame을 'Adj Close'와 몇 가지 기술적 지표로 구성했다고 가정
# StockPortfolioEnv는 실제 가격 데이터와 기술적 지표를 상태 공간에 포함하도록 확장되어야 함.
# 이 예시에서는 ticker들의 Adj Close만 사용했다고 가정
df_for_env = data[tickers] # Adj Close price만 사용

env = DummyVecEnv([lambda: StockPortfolioEnv(df_for_env, initial_amount=100000)])

# PPO 모델 초기화
# policy="MlpPolicy"는 다층 퍼셉트론 정책 네트워크를 사용한다는 의미
# learning_rate, n_steps, batch_size, gamma 등 하이퍼파라미터 튜닝이 중요
model = PPO("MlpPolicy", env, verbose=1, 
            learning_rate=0.0001, 
            n_steps=2048, 
            batch_size=64, 
            gamma=0.99,
            tensorboard_log="./ppo_portfolio_tensorboard/")

# 모델 학습
print("Starting training...")
model.learn(total_timesteps=100000, reset_num_timesteps=False)
print("Training finished.")

# 학습된 모델 저장
model.save("ppo_portfolio_optimizer")

Step 4: Backtesting and Evaluation

The trained policy should be backtested against a separate validation or test dataset similar to real market data. This is essential for evaluating the model's generalization performance and determining whether overfitting has occurred.


# 학습된 모델 로드
# model = PPO.load("ppo_portfolio_optimizer")

# 테스트 데이터셋 준비 (학습 데이터와 겹치지 않아야 함)
test_start_date = '2023-01-01'
test_end_date = '2024-01-01'
test_data = yf.download(tickers, start=test_start_date, end=test_end_date)['Adj Close']
test_data = test_data.dropna()
df_test_for_env = test_data[tickers]

test_env = StockPortfolioEnv(df_test_for_env, initial_amount=100000)

obs = test_env.reset()
portfolio_values = [test_env.portfolio_value]
rewards = []

done = False
while not done:
    action, _states = model.predict(obs, deterministic=True) # deterministic=True로 고정된 행동 선택
    obs, reward, done, info = test_env.step(action)

Dynamic Portfolio Optimization with Reinforcement Learning: Building Adaptive Investment Strategies for Volatile Markets

Dynamic Portfolio Optimization Based on Reinforcement Learning (RL): Building Autonomous Investment Strategies that Adapt to Financial Market Changes

1. The Challenge / Context

2. Deep Dive: Dynamic Portfolio Optimization Based on Reinforcement Learning

3. Step-by-Step Guide / Implementation

Step 1: Data Collection and Preprocessing

Step 2: Defining the Reinforcement Learning Environment (Custom Gym Environment)

Step 3: RL Agent Selection and Training

Step 4: Backtesting and Evaluation

`Heeviz Engineering Team`

`Related Posts`

Building an Automated Macroeconomic Indicator Analysis Pipeline with Prefect and LLM: Uncovering Investment Strategy Insights

Serverless LLM Agent Orchestration: Building Real-time AI Automation Pipelines with AWS Lambda

Multi-Agent Orchestration for Enterprise Workflows: A Guide to Building Autonomous AI Assistant Systems