Towards Enhanced Quantum Resistance for RSA via Constrained R\'enyi Entropy Optimization: A Theoretical Framework for Backward-Compatible Cryptography

Abstract

The advent of quantum computing poses a critical threat to RSA cryptography, as Shor's algorithm can factor integers in polynomial time. While post-quantum cryptography standards offer long-term solutions, their deployment faces significant compatibility and infrastructure challenges. This paper proposes the Constrained R\'enyi Entropy Optimization (CREO) framework, a mathematical approach to potentially enhance the quantum resistance of RSA while maintaining full backward compatibility. By constraining the proximity of RSA primes ($|p-q| < \gamma \sqrt{pq}$), CREO reduces the distinguishability of quantum states in Shor's algorithm, as quantified by R\'enyi entropy. Our analysis demonstrates that for a $k$-bit modulus with $\gamma = k^{-1/2+\epsilon}$, the number of quantum measurements required for reliable period extraction scales as $\Omega(k^{2+\epsilon})$, compared to $\mathcal{O}(k^3)$ for standard RSA under idealized assumptions. This represents a systematic increase in quantum resource requirements. The framework is supported by constructive existence proofs for such primes using prime gap theorems and establishes conceptual security connections to lattice-based problems. CREO provides a new research direction for exploring backward-compatible cryptographic enhancements during the extended transition to post-quantum standards, offering a mathematically grounded pathway to harden widely deployed RSA infrastructure without requiring immediate protocol or infrastructure replacement.