What is PQ?

PQ, or Product Quantization, is a lossy data compression technique, mainly used for efficient similarity search in high-dimensional spaces. It works by dividing the high-dimensional vector space into smaller subspaces and then quantizing each subspace using k-means clustering. The centroids of these clusters become the representative codebook for that subspace. A vector is then represented by the indices of the nearest centroids in each subspace. This allows for significant reduction in storage space and faster distance calculations, as distances are computed based on these quantized representations rather than the original vectors. Commonly used in image retrieval, recommendation systems, and large-scale data analysis where speed and memory efficiency are crucial.