Deep Dive into Qdrant HNSW Parameters for Maximizing High-Dimensional Data Search Performance
When storing and searching high-dimensional vector data using Qdrant, optimizing HNSW (Hierarchical Navigable Small World) index parameters is a key factor that can dramatically improve search speed and accuracy. This article provides a deep understanding of HNSW parameters and guides you on how to find optimal settings through real-world use cases. We will help you prevent performance degradation caused by incorrect parameter settings and build an optimized search environment.
1. The Challenge / Context
With the recent advancements in deep learning models, there is a growing number of cases where various data such as text, images, and audio are represented as high-dimensional vectors and utilized. Performing similarity searches based on this vector data plays an important role in various fields such as recommendation systems, image search, and natural language processing. However, high-dimensional data faces the "curse of dimensionality" problem as the data dimension increases, making efficient searching difficult with traditional indexing techniques. In particular, appropriately adjusting the trade-off between search speed and accuracy is a crucial challenge. Qdrant solves this problem using HNSW indexes, but a lack of understanding of HNSW parameters can lead to performance degradation.
2. Deep Dive: HNSW (Hierarchical Navigable Small World)
HNSW is a graph-based indexing algorithm for Approximate Nearest Neighbor (ANN) search in high-dimensional spaces. HNSW has a multi-layered structure, where each layer is represented as a graph. The top layer has the fewest nodes, and each node represents a vector selected from the dataset. As you move down to lower layers, the number of nodes increases, with the bottom layer containing all vectors in the entire dataset. Searching starts from the top layer, finds the closest node, and then moves to the next layer to explore closer nodes. Thanks to this hierarchical structure, HNSW can achieve both fast search speeds and high accuracy simultaneously.
The core parameters of HNSW are as follows:
- M (The maximum number of outgoing connections in the graph): Determines the maximum number of neighbors each node can connect to. A larger M increases the connectivity of the graph, improving search accuracy, but also increases index build time and memory usage.
- ef_construction (Construction time/memory tradeoff): Determines how much of the search space is explored when building the index. A larger ef_construction leads to a longer index build time, but generates a more accurate graph, improving search performance.
- ef (Search time/accuracy tradeoff): Determines how much of the search space is explored during search. A larger ef improves search accuracy, but increases search time.
3. Step-by-Step Guide / Implementation
Now, let's look at how to set HNSW parameters and optimize performance in a real Qdrant environment, step by step.
Step 1: Create Qdrant Collection and Insert Vectors
First, create a Collection and insert vector data using the Qdrant client. In this step, we will use default HNSW parameters without explicit settings.
from qdrant_client import QdrantClient, models
from qdrant_client.models import VectorParams, Distance
# Qdrant 클라이언트 초기화
client = QdrantClient(":memory:") # 또는 QdrantClient("localhost:6333")
# Collection 생성 (기본 HNSW 파라미터 사용)
client.recreate_collection(
collection_name="my_collection",
vectors_config=VectorParams(size=128, distance=Distance.COSINE),
)
# 벡터 데이터 삽입
vectors = [
models.PointStruct(id=1, vector=[0.05, 0.61, 0.76, 0.74, 0.87, 0.87, 0.21, 0.48, 0.25, 0.09, 0.83, 0.7, 0.35, 0.08, 0.33, 0.49, 0.82, 0.11, 0.13, 0.1, 0.4, 0.19, 0.12, 0.74, 0.41, 0.06, 0.46, 0.83, 0.94, 0.34, 0.49, 0.51, 0.77, 0.38, 0.29, 0.39, 0.24, 0.55, 0.79, 0.9, 0.32, 0.2, 0.16, 0.19, 0.32, 0.71, 0.53, 0.86, 0.47, 0.56, 0.05, 0.75, 0.11, 0.09, 0.37, 0.93, 0.18, 0.11, 0.62, 0.53, 0.73, 0.22, 0.81, 0.77, 0.46, 0.77, 0.97, 0.53, 0.24, 0.23, 0.29, 0.41, 0.7, 0.67, 0.68, 0.25, 0.75, 0.48, 0.82, 0.42, 0.36, 0.23, 0.36, 0.13, 0.52, 0.06, 0.43, 0.52, 0.38, 0.63, 0.3, 0.89, 0.49, 0.17, 0.65, 0.93, 0.39, 0.43, 0.27, 0.06, 0.85, 0.42, 0.82, 0.67, 0.16, 0.29, 0.79, 0.76, 0.87, 0.84, 0.31, 0.47]),
models.PointStruct(id=2, vector=[0.19, 0.77, 0.81, 0.08, 0.14, 0.36, 0.25, 0.74, 0.75, 0.77, 0.73, 0.14, 0.24, 0.88, 0.08, 0.62, 0.94, 0.29, 0.23, 0.51, 0.28, 0.29, 0.76, 0.03, 0.35, 0.06, 0.83, 0.65, 0.28, 0.77, 0.03, 0.45, 0.55, 0.07, 0.31, 0.59, 0.44, 0.35, 0.93, 0.95, 0.29, 0.76, 0.96, 0.59, 0.53, 0.96, 0.86, 0.06, 0.6, 0.49, 0.59, 0.55, 0.41, 0.69, 0.95, 0.84, 0.38, 0.33, 0.15, 0.53, 0.63, 0.77, 0.76, 0.11, 0.88, 0.07, 0.34, 0.82, 0.35, 0.04, 0.61, 0.09, 0.12, 0.32, 0.49, 0.52, 0.56, 0.5, 0.93, 0.83, 0.52, 0.49, 0.37, 0.83, 0.33, 0.89, 0.51, 0.03, 0.21, 0.14, 0.89, 0.08, 0.19, 0.72, 0.13, 0.3, 0.51, 0.9, 0.16, 0.24, 0.35, 0.07, 0.71, 0.37, 0.31, 0.05, 0.36, 0.04, 0.41, 0.57, 0.19, 0.21, 0.71, 0.76, 0.79, 0.79, 0.2, 0.07]),
models.PointStruct(id=3, vector=[0.81, 0.32, 0.74, 0.49, 0.18, 0.22, 0.12, 0.78, 0.24, 0.25, 0.57, 0.22, 0.43, 0.03, 0.95, 0.1, 0.81, 0.69, 0.46, 0.73, 0.53, 0.56, 0.78, 0.25, 0.83, 0.86, 0.2, 0.35, 0.01, 0.52, 0.43, 0.76, 0.4, 0.8, 0.33, 0.03, 0.75, 0.41, 0.54, 0.98, 0.39, 0.37, 0.34, 0.09, 0.26, 0.28, 0.39, 0.59, 0.85, 0.59, 0.2, 0.78, 0.84, 0.54, 0.23, 0.06, 0.57, 0.45, 0.31, 0.31, 0.25, 0.03, 0.41, 0.8, 0.41, 0.55, 0.54, 0.42, 0.16, 0.43, 0.41, 0.4, 0.82, 0.7, 0.76, 0.17, 0.73, 0.36, 0.47, 0.45, 0.45, 0.03, 0.98, 0.45, 0.58, 0.52, 0.24, 0.79, 0.68, 0.25, 0.04, 0.4, 0.76, 0.41, 0.14, 0.46, 0.71, 0.1, 0.97, 0.55, 0.28, 0.8, 0.4, 0.9, 0.21, 0.28, 0.01, 0.42, 0.91, 0.6, 0.74, 0.16, 0.4, 0.34, 0.09, 0.33, 0.05, 0.78]),
]
client.upsert(
collection_name="my_collection",
points=vectors
)
Step 2: Adjust HNSW Parameters
Now, adjust the HNSW parameters of the Collection. Rebuild the index by changing the `M` and `ef_construction` parameters. The important point in this step is not to change the existing Collection settings, but to recreate the index. You should test multiple times by changing the `M` and `ef_construction` values to find the optimal values. Generally, `M` values between 16 and 64, and `ef_construction` values between 100 and 500 are used.
# HNSW 파라미터 설정
hnsw_config = models.HnswConfigDiff(
m=32, # 각 노드가 연결될 수 있는 최대 이웃 수
ef_construction=200, # 인덱스 구축 시 검색 공간 탐색 정도
)
# Collection 업데이트 (HNSW 파라미터 변경)
client.update_collection(
collection_name="my_collection",
hnsw_config=hnsw_config
)
# 인덱스 재생성 (필수) - update_collection은 인덱스 재생성을 자동으로 해주지 않음.
client.create_index(
collection_name="my_collection",
field_name="vector",
wait=True
)
Important: After calling the `update_collection` function, you must call the `create_index` function to regenerate the index. Otherwise, the changed HNSW parameters will not be applied.
Step 3: Test Search Performance
After changing the HNSW parameters, you should test the search performance to confirm that the changes actually contribute to performance improvement. Measure search time and review search results to evaluate accuracy.
import time
# 검색 벡터
query_vector = [0.05, 0.61, 0.76, 0.74, 0.87, 0.87, 0.21, 0.48, 0.25, 0.09, 0.83, 0.7, 0.35, 0.08, 0.33, 0.49, 0.82, 0.11, 0.13, 0.1, 0.4, 0.19, 0.12, 0.74, 0.41, 0.06, 0.46, 0.83, 0.94, 0.34, 0.49, 0.51, 0.77, 0.38, 0.29, 0.39, 0.24, 0.55, 0.79, 0.9, 0.32, 0.2, 0.16, 0.19, 0.32, 0.71, 0.53, 0.86, 0.47, 0.56, 0.05, 0.75, 0.11, 0.09, 0.37, 0.93, 0.18, 0.11, 0.62, 0.53, 0.73, 0.22, 0.81, 0.77, 0.46, 0.77, 0.97, 0.53, 0.24, 0.23, 0.29, 0.41, 0.7, 0.67, 0.68, 0.25, 0.75, 0.48, 0.82, 0.42, 0.36, 0.23, 0.36, 0.13, 0.52, 0.06, 0.43, 0.52, 0.38, 0.63, 0.3, 0.89, 0.49, 0.17, 0.65, 0.93, 0.39, 0.43, 0.27, 0.06, 0.85, 0.42, 0.82, 0.67, 0.16, 0.29, 0.79, 0.76, 0.87, 0.84, 0.31, 0.47]
# 검색 시작 시간 측정
start_time = time.time()
# 벡터 검색
search_result = client.search(
collection_name="my_collection",
query_vector=query_vector,
limit=3, # 검색 결과 개수
query_filter=None,
search_params=models.SearchParams(ef=128) #검색 시 탐색 범위
)
# 검색 종료 시간 측정
end_time = time.time()
# 검색 시간 출력
print(f"검색 시간: {end_time - start_time:.4f} 초")
# 검색 결과 출력
for result in search_result:
print(result)
Run the code above to measure search time and check search results. The `ef` parameter determines how much of the search space is explored during search. A larger `ef` value improves search accuracy, but also increases search time. Finding an appropriate `ef` value is important.
Step 4: Automatic Parameter Tuning (Advanced)
While Qdrant does not yet officially support automatic parameter tuning, you can implement simple grid search or Bayesian optimization algorithms using Python scripts to automatically tune HNSW parameters. This requires performing multiple search performance tests and analyzing the results to find the optimal parameter combination.
Currently, Qdrant experimentally provides `auto_optimizer_config`, but it is not yet perfect in terms of stability and performance, so it should be used with caution.
4. Real-world Use Case / Example
I recently participated in a project to build a product recommendation system for an e-commerce company. This company had millions of product data, and each product was represented by a 128-dimensional vector. Initially, product searches were performed using default HNSW parameters, but the search speed was too slow, leading to a poor user experience. As a result of optimizing the HNSW parameters, we were able to improve search speed by more than 5 times, and user satisfaction significantly increased. Specifically, by increasing the M value from 16 to 32 and the ef_construction value from 100 to 200, search accuracy slightly decreased, but search speed significantly improved.
Furthermore, since product vectors had to be continuously updated based on real-time user behavior data, minimizing index build time was crucial. Therefore, we were careful not to set the ef_construction value too high.
5. Pros & Cons / Critical Analysis
- Pros:
- Optimizing Qdrant HNSW parameters can significantly improve high-dimensional data search speed.
- Appropriate parameter settings allow for adjusting the trade-off between search accuracy and speed.
- The API is well-designed, making it easy for developers new to vector databases to get started.
- Cons:
- A deep understanding of HNSW parameters is required, and finding the optimal parameter combination takes considerable time and effort.
- Automatic parameter tuning is not yet fully supported, so parameters must be adjusted manually.
- The optimal HNSW parameter values vary depending on the characteristics of the dataset, so there is no one-size-fits-all setting.
- Building an index for the first time on a large dataset can take a significant amount of time.
6. FAQ
- Q: I'm not sure how to set HNSW parameters. What values should I use?
A: The optimal HNSW parameters vary depending on the size and dimension of your dataset, as well as your requirements for search speed and accuracy. Generally, M values between 16 and 64, and ef_construction values between 100 and 500 are used. You should experiment with these values as a baseline to find the optimal settings. For smaller datasets, it's recommended to set M and ef_construction values lower, and for larger datasets, set them higher. - Q: I changed the HNSW parameters, but the search performance got worse. Why?
A: After changing HNSW parameters, you must regenerate the index. Also, if the parameter values are too large or too small, search performance can actually degrade. For example, if the M value is too large, memory usage increases, and if the ef_construction value is too small, the graph may not be built properly, leading to reduced search accuracy. - Q: What is the difference between HNSW and other indexing algorithms in Qdrant?
A: Qdrant supports various indexing algorithms in addition to HNSW. HNSW is specialized in providing fast search speeds and high accuracy for high-dimensional data. Other algorithms may be more suitable for specific types of data or search requirements
