General monotone formula for homogeneous $k$-Hessian equation in the exterior domain and its applications

Abstract

In this paper, we deal with an overdetermined problem for the $k$-Hessian equation ($1\leq k<\frac n2$) in the exterior domain and prove the corresponding ball characterizations. Since that Weinberger type approach seems to fail to solve the problem, we give a new perspective to solve exterior overdetermined problem by combining two integral identities and geometric inequalities inspired by Brandolini-Nitsch-Salani's results \cite{BNS}. Meanwhile, we establish general monotone formulas to derive geometric inequalities related to $k$-admissible solution $u$ in $\mathbb R^n\setminus\Omega$, where $\Omega$ is smooth, $k$-convex and star-shaped domain, which constructed by Ma-Zhang\cite{MZ} and Xiao\cite{xiao}.