Solutions of Fully Nonlinear Nonlocal Systems

Abstract

In this paper we consider the system involving fully nonlinear nonlocal operators: $$ \left\{ \begin{array}{ll} F_{\alpha}(u(x)) = C_{n,\alpha} PV \int_{{R}^n} \frac{G(u(x)-u(y))}{|x-y|^{n+\alpha}} dy=f(v(x)), F_{\beta}(v(x)) = C_{n,\beta} PV \int_{{R}^n} \frac{G(v(x)-v(y))}{|x-y|^{n+\beta}} dy=g(u(x)). \end{array} \right. $$ A \textit{narrow region principle} and a \textit{decay at infinity} for the system for carrying on the method of moving planes are established. Then we prove the radial symmetry and monotonicity for positive solutions to the nonlinear system in the whole space. Non-existence of positive solutions to the nonlinear system on a half space is proved.