On some determinants involving the tangent function

Abstract

Let $p$ be an odd prime and let $a,b\in\mathbb Z$ with $p\nmid ab$. In this paper we mainly evaluate $$T_p^{(\delta)}(a,b,x):=\det\left[x+\tan\pi\frac{aj^2+bk^2}p\right]_{\delta\le j,k\le (p-1)/2}\ \ (\delta=0,1).$$ For example, in the case $p\equiv3\pmod4$ we show that $T_p^{(1)}(a,b,0)=0$ and $$T_p^{(0)}(a,b,x)=\begin{cases} 2^{(p-1)/2}p^{(p+1)/4}&\text{if}\ (\frac{ab}p)=1, \\p^{(p+1)/4}&\text{if}\ (\frac{ab}p)=-1,\end{cases}$$ where $(\frac{\cdot}p)$ is the Legendre symbol. When $(\frac{-ab}p)=-1$, we also evaluate the determinant $\det[x+\cot\pi\frac{aj^2+bk^2}p]_{1\le j,k\le(p-1)/2}.$ In addition, we pose several conjectures one of which states that for any prime $p\equiv3\pmod4$ there is an integer $x_p\equiv1\pmod p$ such that $$\det\left[\sec2\pi\frac{(j-k)^2}p\right]_{0\le j,k\le p-1}=-p^{(p+3)/2}x_p^2.$$