Dynamic Entropy-Encoded Arrays in O(1) Time with Nearly Optimal Space

Abstract

We show how to implement a dynamic array $A[1, n]$ with symbols from a fixed alphabet $Σ$, while supporting $O(1)$-time queries and updates, and using a total space of $$ \log \binom{|Σ|}{m} + \left(1 + O\left(\frac{\log \log n}{\log n}\right)\right) \cdot \left(\sum_{σ\in Σ} f_σ\log (n / f_σ)\right) + n / \text{polylog } n $$ bits, where $f_σ$ denotes the frequency of each symbol $σ\in Σ$ and $m$ denotes the number of distinct symbols with non-zero frequencies. This resolves a long-standing open question as to whether one can achieve space bounds close to that of arithmetic coding, while supporting $O(1)$-time operations, whenever the entropy is at least $n/\text{polylog } n$. We also prove a nearly matching space lower bound: up to a factor of $O(\log \log n)$, the entropy-dependent multiplicative overhead of our construction is optimal among $O(1)$-time solutions when $|Σ|=O(\sqrt n)$ and the entropy $\sum_{σ\in Σ} f_σ\log (n / f_σ)$ lies between $n/\log^{O(1)}n$ and $(1/100)n\log n$. Finally, we present several applications of our results, resolving two open problems having to do with space-efficient dictionaries and filters.