Ternary cyclotomic polynomials having a large coefficient

Abstract

Let $\Phi_n(x)$ denote the $n$th cyclotomic polynomial. In 1968 Sister Marion Beiter conjectured that $a_n(k)$, the coefficient of $x^k$ in $\Phi_n(x)$, satisfies $|a_n(k)|\le (p+1)/2$ in case $n=pqr$ with $p<q<r$ primes (in this case $\Phi_n(x)$ is said to be ternary). Since then several results towards establishing her conjecture have been proved (for example $|a_n(k)|\le 3p/4$). Here we show that, nevertheless, Beiter's conjecture is false for every $p\ge 11$. We also prove that given any $\epsilon>0$ there exist infinitely many triples $(p_j,q_j,r_j)$ with $p_1<p_2<... $ consecutive primes such that $|a_{p_jq_jr_j}(n_j)|>(2/3-\epsilon)p_j$ for $j\ge 1$.