Degenerate principal series representations and their holomorphic extensions

Original · EN

Let X=H/L be an irreducible real bounded symmetric domain realized as a real form in an Hermitian symmetric domain D=G/K. The intersection S of the Shilov boundary of D with X defines a distinguished subset of the topological boundary of X and is invariant under H and can also be realized as S=H/P for certain parabolic subgroup P of H. We study the spherical representations Indₚʰ() of H induced from P. We find formulas for the spherical functions in terms of the Macdonald ₂F₁ hypergeometric function. This generalizes the earlier result of Faraut-Koranyi for Hermitian symmetric spaces D. We consider a class of H-invariant integral intertwining operators from the representations Indₚʰ() on L²(S) to the holomorphic representations of G on D restricted to H. We construct a new class of complementary series for the groups H=SO(n, m), SU(n, m) (with n-m >2) and Sp(n, m) (with n-m>1). We realize them as a discrete component in the branching rule of the analytic continuation of the holomorphic discrete series of G=SU(n, m), SU(n, m)× SU(n, m) and SU(2n, 2m) respectively.