Building a Hybrid Investment Decision-Making System Combining Human Expert Intuition and AI Analytical Power: LLM-based Proactive Questioning and Bias Mitigation Strategies

The gap between human deep intuition and AI's vast data analysis capabilities in the investment decision-making process has long been a challenge. This article proposes a method for building a hybrid system that utilizes LLMs to bridge this gap, proactively mitigate potential expert biases, and innovatively enhance the quality of investment decisions. Through specific technology stacks and implementation strategies that can be applied immediately, it will provide practical insights to elevate your investment process to the next level.

1. The Challenge / Context

In today's complex financial markets, investment decision-making demands extreme precision and speed. Human experts often demonstrate excellent insights through 'intuition' that is difficult to quantify, such as subtle market trends, geopolitical factors, and psychological elements. However, this intuition can easily be swayed by cognitive biases, emotional fluctuations, and information overload, risking irrational decisions. On the other hand, artificial intelligence, especially quantitative analysis models, excels at processing vast amounts of data at superhuman speeds and identifying patterns, but has limitations in understanding the context of complex unstructured data or making 'common sense' inferences about new market variables. In essence, we face a dilemma where humans are biased and slow but wise, while AI is fast and objective but blind.

This gap can no longer be solved by passive data provision alone. Simple AI reporting only assists human expert judgment and does not fundamentally revolutionize the quality of decision-making. Here, the emergence of large language models (LLMs) like ChatGPT has provided us with a new solution. LLMs have the potential to go beyond simple information retrieval, mimicking and enhancing human thought processes through natural language understanding, reasoning, and even 'proactive questioning'. It is now time to build an interactive system where LLMs not only present analysis results but also validate expert intuition and proactively mitigate potential biases.

2. Deep Dive: LLM-based Proactive Questioning and Bias Mitigation Framework

The core of a hybrid investment decision-making system lies in a framework where LLMs actively engage in dialogue with human experts, going beyond simple Q&A, to enhance the quality of decisions. This is designed for the LLM to act like a highly trained 'coach', facilitating the expert's thought process and correcting potential errors.

2.1. Expanding LLM's Role: From Simple Analysis to Proactive Reasoning

Traditional AI systems primarily focused on analyzing quantitative data and discovering patterns. However, LLMs expand their role in the following ways:

  • Context Understanding and Knowledge Synthesis: It understands and synthesizes vast amounts of unstructured text data, such as news articles, analyst reports, and corporate disclosures, in addition to structured financial data, to derive deep insights.
  • Proactive Question Generation: It poses critical questions about an expert's initial judgment or hypothesis, helping to uncover hidden assumptions or overlooked factors.
  • Chain-of-Thought Provision: It clearly explains the process by which the LLM arrived at a specific conclusion, providing transparency that allows experts to trust and, if necessary, correct the AI's judgment.
  • Bias Recognition and Mitigation: It learns patterns of cognitive biases (confirmation bias, anchoring effect, availability bias, etc.) and warns experts or suggests alternative perspectives at points where these biases might appear.

2.2. How LLMs Work for Bias Mitigation

LLM-based bias mitigation is not simply saying, "You are biased." It is achieved through sophisticated prompt engineering and expert feedback loops:

  • Hypothesis Testing Request: After receiving an expert's initial investment hypothesis, the LLM asks questions that encourage the expert to look for counter-evidence or weaknesses. 
    "현재 [특정 기업]에 대한 투자 의견을 [긍정적/부정적]으로 보고 계신데, 이와 상반되는 시장 데이터나 분석 보고서는 없는지 다시 한번 검토해 주시겠습니까? 특히 [특정 지표] 측면에서 리스크 요인을 찾아주세요."
  • Alternative Scenario Exploration: To prevent getting stuck in a single powerful narrative, it asks questions that assume various market scenarios and predict investment performance for each scenario. 
    "현재 시장에 가장 영향을 미 미칠 수 있는 '블랙스완' 이벤트는 무엇이라고 생각하십니까? 만약 [특정 이벤트]가 발생한다면, 이 투자 포트폴리오에 어떤 영향을 미칠 것이며, 이에 대한 헤징 전략은 무엇이 있을까요?"
  • Inducing Exclusion of Emotional Factors: It encourages caution against emotional judgments based on rapid market changes or personal experiences. 
    "최근 [관련 업종]의 급락으로 인해 투자 심리가 위축된 상황이지만, [해당 기업]의 펀더멘탈은 여전히 견고한 편입니다. 현재의 시장 분위기와 별개로, 오직 기업의 내재 가치와 장기적 성장 가능성에만 집중하여 다시 한번 평가해 주시겠습니까?"

3. Step-by-Step Guide / Implementation

Now, let's look at the specific steps to build an LLM-based hybrid investment decision-making system. This process includes everything from the data pipeline to building the interaction interface with the LLM.

Step 1: Data Integration and Knowledge Graph Construction

For the LLM to perform accurate and in-depth analysis, extensive and refined data is required. Data from various sources is integrated and stored in the form of a Knowledge Graph or Vector Database for efficient access by the LLM.

  • Data Sources: Real-time stock price data, corporate financial statements (disclosures), news articles (finance, general industry), analyst reports, social media trends, macroeconomic indicators, etc.
  • Technology Stack: Apache Kafka (real-time streaming), Apache Spark (data processing), PostgreSQL/MongoDB (structured/unstructured storage), Pinecone/Weaviate/ChromaDB (vector database).
  • Implementation Example (Knowledge Storage using LangChain and Vector DB):
# Python 예시: 데이터 로드 및 벡터 스토어 인덱싱
from langchain.document_loaders import PyPDFLoader, WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma

# 1. 문서 로드 (예: PDF 애널리스트 리포트, 웹 기반 뉴스 기사)
# loader = PyPDFLoader("path/to/analyst_report.pdf")
loader = WebBaseLoader("https://news.naver.com/main/read.naver?mode=LSD&mid=shm&sid1=101&oid=000&aid=0000000000") # 예시 URL
documents = loader.load()

# 2. 문서 분할 (LLM 컨텍스트 윈도우에 맞게)
text_splitter = RecursiveCharacterTextTextSplitter(chunk_size=1000, chunk_overlap=200)
texts = text_splitter.split_documents(documents)

# 3. 임베딩 및 벡터 스토어 저장
embeddings = OpenAIEmbeddings() # 또는 다른 임베딩 모델
vectorstore = Chroma.from_documents(texts, embeddings, persist_directory="./chroma_db")
vectorstore.persist()

print("데이터가 벡터 스토어에 성공적으로 저장되었습니다.")

Step 2: LLM-based Investment Analysis and Initial Proposal Generation

Based on integrated data, the LLM generates initial investment ideas and identifies potential risks and opportunities. This step primarily uses the RAG (Retrieval Augmented Generation) pattern to reduce LLM hallucination and elicit answers based on the latest information.

  • Technology Stack: OpenAI API (GPT-4), Anthropic Claude, or self-hosted models (e.g., Llama 2). LangChain/LlamaIndex (RAG framework).
  • Implementation Example (Initial Analysis Prompt):
# Python 예시: LLM을 활용한 초기 투자 분석 (LangChain)
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA

# 이전에 저장된 벡터 스토어 로드
vectorstore = Chroma(persist_directory="./chroma_db", embedding_function=OpenAIEmbeddings())
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

llm = ChatOpenAI(model_name="gpt-4", temperature=0.7)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True
)

query = """
다음은 최근 [특정 기업]에 대한 애널리스트 보고서와 뉴스 기사입니다. 이 기업에 대한 투자 매력도를 종합적으로 평가하고, 주요 강점과 약점, 그리고 잠재적 리스크 요인을 상세히 분석하여 300자 내외로 요약해 주세요.
시장 동향: [LLM이 직접 데이터에서 추출하거나, 추가 정보를 제공]
경쟁사 분석: [LLM이 직접 데이터에서 추출하거나, 추가 정보를 제공]
"""

response = qa_chain({"query": query})
print("
LLM 초기 분석 결과:
")
print(f"
{response['result']}
")
print("
참고 문서:
")
for doc in response['source_documents']:
    print(f"
  • {doc.metadata.get('source', 'Unknown source')} - {doc.page_content[:100]}...
    • ")

    Step 3: Proactive Questioning and Bias Mitigation Module Implementation

    This is the core step. The LLM detects potential biases in the expert's initial judgment or its own analysis and generates proactive questions to mitigate them. This can be triggered by specific events (e.g., strong expressions of confidence from the expert, use of specific keywords).

    • Technology Stack: LLM API, Python (prompt management and logic).
    • Implementation Example (BiasDetector Class and Question Generation):
    # Python 예시: 편향 감지 및 능동적 질문 생성
from langchain.prompts import PromptTemplate
from langchain.schema import HumanMessage, SystemMessage

class BiasMitigationLLM:
    def __init__(self, llm):
        self.llm = llm
        # 편향 감지 및 질문 생성을 위한 프롬프트 템플릿
        self.bias_question_template = PromptTemplate(
            template="""
            당신은 인간 전문가의 투자 의사결정 과정을 보조하고 편향을 완화하는 AI 코치입니다.
            인간 전문가가 제시한 다음 투자 의견과 당신이 분석한 내용을 바탕으로,
            인간 전문가가 인지적 편향(예: 확증 편향, 앵커링 효과, 가용성 편향)에 빠지지 않도록
            능동적이고 비판적인 질문 1~2가지를 생성해 주세요.
            질문은 전문가의 사고를 확장하고, 다른 관점을 고려하도록 유도해야 합니다.
            결론을 강요하거나 훈계하는 어조가 아닌, 탐색적인 질문이어야 합니다.

            ---
            인간 전문가의 초기 투자 의견:
            {expert_opinion}

            LLM의 초기 분석 요약:
            {llm_analysis}
            ---

            생성할 질문:
            """,
            input_variables=["expert_opinion", "llm_analysis"]
        )

    def generate_bias_mitigation_questions(self, expert_opinion: str, llm_analysis: str) -> str:
        prompt = self.bias_question_template.format(
            expert_opinion=expert_opinion,
            llm_analysis=llm_analysis
        )
        response = self.llm.invoke([SystemMessage(content="You are a helpful AI assistant."), HumanMessage(content=prompt)])
        return response.content

# 사용 예시:
expert_initial_opinion = "최근 A기업의 주가가 급락했지만, 과거에도 이런 패턴 후 강하게 반등했으므로, 지금이 매수 적기라고 확신합니다."
llm_analysis_summary = "A기업의 재무 상태는 견고하나, 최근 급락은 시장 전체의 불확실성과 동종업계 경쟁 심화가 복합적으로 작용한 것으로 분석됩니다. 과거 패턴과 현재 시장 환경에는 차이점이 존재합니다."

bias_llm = BiasMitigationLLM(ChatOpenAI(model_name="gpt-4", temperature=0.5))
questions = bias_llm.generate_bias_mitigation_questions(expert_initial_opinion, llm_analysis_summary)

print("
    LLM이 생성한 편향 완화 질문:
    ")
print(f"
    {questions}
    ")

    In the code above, the LLM will recognize the expert's conviction of 'strong rebound after past patterns' as a potential signal of 'anchoring bias' and generate questions that encourage considering differences from the current market environment.

    Step 4: Human Expert Feedback Loop and Decision Refinement

    Human experts review and respond to the questions generated by the LLM. These responses are then fed back to the LLM to refine the analysis or generate additional questions. This iterative interaction improves the quality of the final decision.

    • Technology Stack: Streamlit/Dash/React (frontend), FastAPI/Flask (backend API), LLM API.
    • Workflow:
      1. Expert submits initial investment idea to the system.
      2. LLM analyzes data, generates initial investment proposal, and identifies potential bias points.
      3. LLM presents proactive questions to the expert (via web interface).
      4. Expert responds to questions and provides additional information.
      5. LLM refines analysis based on expert's response, presents new perspectives, or asks further questions.
      6. This process repeats until a final investment decision is reached.

    4. Real-world Use Case / Example

    Let's assume 'Mr. Kim Investor', a private investor or small hedge fund manager, is trying to make an investment decision on S-Tech stock. Mr. Kim Investor is familiar with S-Tech's past stock price movements and is encouraged by S-Tech's recent new technology announcement. He is strongly convinced that S-Tech will rebound soon.

    Traditional Approach: Mr. Kim Investor is likely to make a buy decision focusing only on his past experience and positive news. Negative news or market analysis might be unconsciously overlooked.

    Hybrid System Application:

    1. Initial Analysis: Mr. Kim Investor inputs "Review S-Tech stock purchase" into the system. The LLM comprehensively analyzes S-Tech's financial statements, recent news, competitor trends, and overall macroeconomic indicators to generate an initial report. This report includes not only the potential of the new technology but also two-sided information such as lower-than-expected last quarter's performance, high debt ratio, and global economic slowdown concerns.
    2. Bias Detection and Questioning: The LLM detects potential signs of 'confirmation bias' and 'anchoring bias' in Mr. Kim Investor's initial decision-making context. That is, it identifies Mr. Kim Investor's tendency to focus only on positive information and be bound by past successful experiences. The LLM presents the following questions on Mr. Kim Investor's dashboard: 
      "김 투자님, S-Tech의 신기술 발표는 고무적이지만, 지난 분기 실적은 시장 예상치를 하회했습니다. 이 기업의 높은 부채 비율이 현재의 금리 인상 환경에서 추가적인 리스크로 작용할 가능성은 없다고 보시는지 궁금합니다. 또한, S-Tech 주가의 '과거 반등 패턴'이 현재 글로벌 공급망 문제나 원자재 가격 상승과 같은 거시경제적 변수 하에서도 유효할 것이라는 가정에 대해 재고해 보시는 것은 어떠신가요?"
    3. Expert Feedback and Refinement: Mr. Kim Investor reflects on his thought process through the LLM's questions. He realizes he overlooked the debt ratio and macroeconomic variables and requests additional data from the LLM on how these factors might affect S-Tech's future performance. The LLM provides relevant data and summarizes the company's debt management strategies and key differences between past rebound patterns and the current market environment.
    4. Final Decision: After this iterative interaction, Mr. Kim Investor reconsiders his initial 'strong buy' opinion on S-Tech and decides to invest only a small amount to mitigate risk, or to observe the market until the situation becomes clearer. In this process, his decision-making moves away from biases caused by emotion or limited information, leading to a much more rational and data-driven outcome.

    Personal Insight: What I've learned over the past 10 years of building various decision support systems is that simply 'providing information' is not enough to change deep-seated human thinking habits. True innovation is possible when AI goes beyond being a passive information provider to become an active partner that 'guides', 'challenges', and even 'triggers' human thought processes. LLM-based 'proactive questioning' is at the pinnacle of this AI-human collaboration, and it is the most powerful point I discovered after countless trials and errors. It's not just about throwing data at experts, but about instructing how data should be reflected in their thinking, and furthermore, expanding their thinking. This approach is the realization of 'augmented intelligence' beyond a mere 'copilot'.

    5. Pros & Cons / Critical Analysis

    • Pros:
      • Improved Decision Quality: Combines human intuition and AI analytical power to reduce bias and make more rational decisions.
      • Reduced Cognitive Load: LLMs handle vast information processing and initial analysis, allowing experts to focus on critical judgments.
      • Transparency and Explainability: The LLM's chain of thought and questions make the decision-making process more transparent and easier to understand the results.
      • Consistency Maintenance: Can reduce inconsistent decision-making due to emotional factors or temporary market fluctuations.
      • Rapid Learning and Adaptation: LLMs can quickly learn from new data and market changes and question experts based on the latest information.
    • Cons:
      • Setup and Operating Costs: Significant costs can arise from high-performance LLM API usage fees, data integration and vector DB construction, and self-model training.
      • Hallucination Risk: LLMs may generate factually incorrect information, which can be fatal for investment decisions. (Needs mitigation through RAG and expert review)
      • Data Security and Privacy: If sensitive investment data is transmitted to external LLM services, security and regulatory compliance issues may arise.
      • Over-reliance: Experts may become overly reliant on LLM results, leading to a decline in critical thinking skills. It is important to recognize that AI provides a 'framework for thought' rather than 'the answer'.
      • Bias Learning Potential: If the LLM's training data itself contains biases, there is a risk that the LLM may generate or reinforce new biases.

    6. FAQ

    • Q: What is the minimum technology stack required to build this system?
      A: Python (main development language), LangChain or LlamaIndex (LLM orchestration framework), OpenAI API (GPT-4) or Anthropic Claude API (large language model), ChromaDB or Pinecone (vector database), Streamlit or Dash (for building simple frontends and dashboards) are key.
    • Q: How can LLM bias be effectively mitigated?
      A: First, use prompt engineering to clearly instruct the LLM on its role in bias detection and presenting diverse perspectives. Second, use the RAG (Retrieval Augmented Generation) pattern to ensure the LLM always answers based on reliable, up-to-date data. Third, continuously improve the LLM's bias mitigation questioning ability through ongoing feedback from human experts. Finally, efforts to reduce bias in the training data should be pursued concurrently.
    • Q: Is it safe to transmit sensitive investment information to an LLM?
      A: When using cloud-based LLM APIs, you must carefully review the service provider's data processing and security policies and sign enterprise-level agreements for sensitive information protection. The safest method is to build a self-hosted LLM (e.g., Llama 2, Mistral) in an on-premise environment and complete all data processing within your internal network. Strong encryption should always be applied during API communication.

    7. Conclusion

    A hybrid investment decision-making system combining human expert intuition and LLM analytical power is a powerful solution for navigating the complexities of financial markets. The LLM's role in going beyond a simple information provider to enhance experts' thought processes through proactive questioning and bias mitigation mechanisms has the potential to change the paradigm of investment decision-making. This system holds universal value and can be applied not only in the investment domain but also in all fields where human judgment is critical, such as law and medicine.

    Build a prototype of this powerful hybrid system right now to revolutionize your investment strategy. The first step is to link an LLM API with a vector database and implement a simple proactive questioning prompt. Refer to the code snippets and architecture presented in this article to experience small successes and gradually expand the system. Feel free to join the discussion if you have any questions!