Automating E-commerce Price Monitoring and Competitor Analysis with n8n and Python

The core of competitive e-commerce operations lies in real-time price monitoring and competitor analysis. By combining n8n's powerful workflow automation capabilities with Python's flexible web scraping abilities, you can automatically collect and analyze price information 24/7 to formulate immediate response strategies. This solution saves you time, accelerates decision-making, and ultimately contributes to increased sales.

1. The Challenge / Context

The e-commerce market is constantly changing, and price fluctuations occur in the blink of an eye. Manually monitoring competitor prices is time-consuming, inefficient, and prone to errors. Furthermore, analyzing and responding to collected data in real-time presents difficulties. Failure to address these issues can lead to delayed responses to market changes and a decrease in sales. Small businesses and individual sellers, in particular, face greater challenges due to a lack of time and resources to invest in competitor analysis. Therefore, an automated price monitoring and competitor analysis system is an essential component for modern e-commerce operations.

2. Deep Dive: n8n

n8n is a low-code workflow automation platform. It connects various applications and services to automate data flows and efficiently handle repetitive tasks. A core feature of n8n is its node-based visual interface. Each node performs a specific task, and by connecting nodes, complex workflows can be easily built. Specifically, you can use the HTTP Request node to fetch data from websites, the Function node to process and transform data, and the Email node to receive notifications. n8n supports both cloud and self-hosted versions, with the self-hosted version offering greater advantages in terms of data security and privacy.

3. Step-by-Step Guide / Implementation

Below is a step-by-step guide to automating e-commerce price monitoring and competitor analysis by integrating n8n and Python.

Step 1: Write a Python Script (Web Scraping)

Write a Python script to extract price information from competitor websites. You can use the `requests` and `beautifulsoup4` libraries to parse HTML and extract the desired data.


import requests
from bs4 import BeautifulSoup

def get_price(url, selector):
    try:
        response = requests.get(url)
        response.raise_for_status()  # HTTP 에러 확인
        soup = BeautifulSoup(response.content, 'html.parser')
        price_element = soup.select_one(selector)
        if price_element:
            price = price_element.text.strip()
            return price
        else:
            return None
    except requests.exceptions.RequestException as e:
        print(f"Error fetching URL: {e}")
        return None
    except Exception as e:
        print(f"Error parsing HTML: {e}")
        return None

if __name__ == '__main__':
    url = "https://www.example.com/product/123"  # 대상 웹사이트 URL
    selector = ".price"  # 가격 정보 CSS 선택자
    price = get_price(url, selector)
    if price:
        print(f"가격: {price}")
    else:
        print("가격을 찾을 수 없습니다.")

    # JSON 형식으로 결과를 반환하기 위한 예시
    import json
    data = {"price": price}
    print(json.dumps(data))

In the code above, enter the URL of the competitor's website from which to extract price information into the `url` variable, and enter the CSS selector of the HTML element containing the price information into the `selector` variable. You need to analyze the website's HTML structure to find the appropriate selector. Since each website has a different HTML structure, you must use a selector specific to each website. You can easily find the desired HTML elements using Chrome Developer Tools (F12).

Step 2: Build an n8n Workflow

Build an n8n workflow to automate the process of executing the Python script, processing the results, and sending notifications.

  • (1) Add a Webhook node: Use a Webhook node to trigger the workflow. For example, you can set the workflow to run at a specific time each day.
  • (2) Add an Execute Command node: Use an Execute Command node to execute the Python script. Enter `python /path/to/your/script.py` in the "Command" field. Ensure that the `PATH` environment variable is correctly set so that n8n can find the Python executable.
  • (3) Add a Function node: Use a Function node to parse the output of the Execute Command node and extract the desired data (price). Since the output of the Execute Command node is a string, it needs to be converted to JSON format.

// Function 노드 JavaScript 코드 예시
const output = $input.first().json.stdout;
try {
  const data = JSON.parse(output);
  return [{json: data}];
} catch (e) {
  return [{json: {error: "JSON 파싱 오류", rawOutput: output}}];
}
  • (4) HTTP Request node (Optional): If necessary, you can use an HTTP Request node to store price information in a database or integrate with other services via an API. For example, you can record price information in a spreadsheet or send price change notifications to a Slack channel.
  • (5) Add an Email node: Use an Email node to send notifications if a price change is detected. The Function node compares the current price with the previous price to determine if there's a change, and then passes the notification content to the Email node.

When executing a Python script using the Execute Command node in n8n, the script execution might fail due to permission issues. In such cases, you need to grant Python script execution permissions to the user running the n8n process.


# 파일 권한 변경 예시 (리눅스)
chmod +x /path/to/your/script.py
chown n8n_user:n8n_group /path/to/your/script.py

Step 3: Data Storage and Visualization (Optional)

Collected price data can be stored in a database (e.g., PostgreSQL, MySQL) and analyzed for trends using visualization tools (e.g., Grafana, Tableau). You can use n8n's HTTP Request node to send data to a database API or connect directly to the database.


// PostgreSQL 데이터베이스에 데이터 저장 예시 (n8n HTTP Request 노드 설정)
// Method: POST
// URL: http://your_database_api_endpoint/prices
// Headers: Content-Type: application/json
// Body:
{
  "product_id": "123",
  "price": "{{$json.price}}",
  "timestamp": "{{$now}}"
}

4. Real-world Use Case / Example

While operating an online shopping mall, I used to spend over 2 hours daily manually monitoring product prices. After automating price monitoring and competitor analysis using n8n and Python, I saved more than 2 hours every day and increased sales by 15% by responding immediately to price changes. In particular, setting up instant notifications whenever a competitor's specific product price dropped was a great help in maintaining price competitiveness and attracting customers. As soon as I received an alert, I was able to adjust my store's prices to stay competitive.

5. Pros & Cons / Critical Analysis

  • Pros:
    • Time Saving: Significantly reduces the time spent manually monitoring prices.
    • Improved Accuracy: Automated systems reduce human error and provide accurate data.
    • Real-time Response: Allows for immediate response to price changes, maintaining competitiveness.
    • Data-driven Decision Making: Enables the formulation of effective pricing strategies by analyzing collected data.
    • Scalability: Easily expandable to monitor various websites and products.
    • Cost-effectiveness: Minimal additional costs beyond initial setup.
  • Cons:
    • Initial Setup Complexity: Requires knowledge of n8n and Python, and initial setup may take time.
    • Maintenance for Website Changes: If a competitor's website structure changes, the Python script needs to be modified.
    • Potential for Website Blocking: Excessive requests may lead to your IP being blocked by competitor websites. Adjust the interval between requests to reduce the chance of blocking. It's also important to check the robots.txt file to confirm if crawling is prohibited.
    • Data Quality Issues: Analysis results may vary depending on the quality of data provided by the website. A data validation step should be added to improve data quality.

6. FAQ

  • Q: How do I install n8n?
    A: n8n can be installed via npm or using Docker. Refer to the official n8n documentation for installation guides.
  • Q: What libraries are needed to run the Python script?
    A: Generally, the `requests` and `beautifulsoup4` libraries are required. Other libraries can be added as needed.
  • Q: How can I prevent my IP from being blocked by a website?
    A: You can adjust the interval between requests, change the User-Agent, and use proxy servers, among other methods. Also, check the website's robots.txt file to confirm if crawling is prohibited.
  • Q: How do I store price data in a database?
    A: You can use n8n's HTTP Request node to send data to a database API or connect directly to the database. The method of connecting to a database varies depending on the database type.

7. Conclusion

Automating e-commerce price monitoring and competitor analysis with n8n and Python is an essential tool for competitive e-commerce operations. This solution allows you to save time and costs, and drive sales growth through data-driven decision-making. Apply the code now to build an automated price monitoring system and gain a competitive advantage. Refer to the official n8n documentation to further expand your workflows and customize them to suit your business.