THE SELBERG TRACE FORMULA FOR NON-UNITARY REPRESENTATIONS OF THE LATTICE

WERNER MÜLLER

Abstract. Let X = Γ\G/K be a compact locally symmetric space. In this paper we establish a version of the Selberg trace formula for non-unitary representations of the lattice Γ. On the spectral side appears the spectrum of the “flat Laplacian” Δ^{#}, acting in the space of sections of the associated flat bundle. In general, this is a non-self-adjoint operator.

1. Introduction

Let G be a connected real semisimple Lie group with finite center and of non-compact type. Let K be a maximal compact subgroup of G. Then S = G/K is a Riemannian symmetric space of nonpositive curvature. We fix an invariant metric on S which we normalize using the Killing form. Let Γ ⊂ G be a discrete subgroup such that Γ\G is compact. For simplicity we assume that Γ is torsion free. Then Γ acts properly discontinuously on S and X = Γ\S is a compact locally symmetric manifold.

Let χ: Γ → GL(V_{χ}) be a finite-dimensional unitary representation. Denote by E_{χ} → Γ\S the associated flat vector bundle. It is equipped with a canonical Hermitian fiber metric h^{χ} and a compatible flat connection ∇^{χ}. Let d_{χ}: C^{∞}(X, E_{χ}) → Λ^{1}(X, E_{χ}) be the associated exterior derivative and let δ_{χ} be the formal adjoint of d_{χ} with respect to the inner products in C^{∞}(X, E_{χ}) and C^{∞}(X, T^{∗}(X) ⊗ E_{χ}), respectively, induced by the invariant metric on S and the fiber metric h^{χ} in E_{χ}. Let Δ_{χ} = δ_{χ}d_{χ} be the associated Laplace operator. Then Δ_{χ} is a second order elliptic, formally self-adjoint, nonnegative differential operator. The Selberg trace formula computes the distributional trace of cos t\sqrt{Δ_{χ}} in terms of a sum of distributions on G, which are associated to the conjugacy classes of Γ.

The trace formula has many applications. Of particular interest for the present paper are applications to Ruelle and Selberg zeta functions. Especially the analytic continuation and the functional equation of twisted Ruelle and Selberg zeta functions rely on the twisted Selberg trace formula [BO], [Se2]. Also spectral invariants of locally symmetric spaces such as analytic torsion and eta invariants can be studied with the help of the trace formula (see [Fr], [Mil], [MS1], [MS2]). So far, these applications are restricted to unitary representations of Γ and it this very desirable to extend the scope of the trace formula so that all finite-dimensional representations are covered. This is the main goal of this paper.

Date: June 11, 2009.

1991 Mathematics Subject Classification. Primary: 11F72.

Key words and phrases. Selberg trace formula, spectral theory.

To begin with we recall the trace formula for a unitary representation χ (see [Se1], [Se2]). Let spec(Δ_{χ}) be the spectrum of Δ_{χ}. It consists of a sequence 0 ≤ λ_{1} < λ_{2} < ⋯ of eigenvalues of finite multiplicities. Denote by m(λ_{k}) the multiplicity of λ_{k}. Let φ ∈ S(ℝ) be even and assume that the Fourier transform φ̂ of φ belongs to C_{c}^{∞}(ℝ). Then φ((Δ_{χ})^{1/2}) is a trace class operator and its trace is given by

(1.1)  Tr φ((Δ_{χ})^{1/2}) = ∑_{λ∈spec(Δ_{χ})} m(λ)φ(λ^{1/2}).

Let Δ̃ be the Laplacian of S, and let h_{φ} be the kernel of the invariant integral operator φ(Δ̃^{1/2}). It belongs to the space C_{c}^{∞}(G//K) of K-bi-invariant compactly supported smooth functions on G. Given γ ∈ Γ let {γ}_{Γ} denote its Γ-conjugacy class. Furthermore, let G_{γ} and Γ_{γ} denote the centralizer of γ in G and Γ, respectively. Then the first version of the trace formula is the following identity.

∑_{λ∈spec(Δ_{χ})} m(λ)φ(λ^{1/2}) = vol(Γ\S) dim V_{χ}h_{φ}(e)

+ ∑_{{γ}_{Γ}≠e} tr χ(γ) vol(Γ_{γ}\G_{γ}) ∫_{G_{γ}\G} h_{φ}(g^{−1}γg) dġ.

(1.2)

To make this formula more explicite, one can use the Plancherel formula to express h_{φ} in terms of φ. Furthermore, the orbital integrals

I(g; φ) = ∫_{G_{γ}\G} h_{φ}(g^{−1}γg) dġ

are invariant distributions and therefore, one can use Harish-Chandra’s Fourier inversion formula to compute them (see [DKV, §4]). In the higher rank case this is rather complicated and no closed formula is available. In the rank one case, however, the situation is much better. There is a simple formula expressing the orbital integrals in terms of characters which leads to an explicit form of the trace formula [Wa, Theorem 6.7].

To extend the Selberg trace formula to all finite-dimensional representations of Γ, we first note that the sum on the right hand side of (1.2) is finite and therefore, it is well defined for all finite-dimensional representations χ. The question is what is the appropriate operator which replaces the Laplacian on the left hand side. In general there is no Hermitian metric on E_{χ} which is compatible with the flat connection ∇^{χ}. A special case has been studied by Fay [Fa]. He considered the analytic torsion T_{M}(χ) of a Riemann surface M = Γ\ℍ of genus g > 1 and a unitary character χ ∈ Hom(Γ, S^{1}) and established the analytic continuation of T_{M}(χ) to all characters χ ∈ Hom(Γ, ℂ^{∗}). To this end he introduced a non-self-adjoint Laplacian. We use a similar approach in the general case. The operator that replaces Δ_{χ} is the “flat Laplacian” Δ_{χ}^{#} which is defined as follows. Let ∗: Λ^{p}(T^{∗}X) → Λ^{n−p}(T^{∗}X) be the Hodge star operator associated to the Riemannian metric of X. Extend ∗ to an operator ∗_{χ} in Λ^{p}(T^{∗}X) ⊗ E_{χ} by ∗_{χ} = ∗ ⊗ Id_{E_{χ}}. Define δ_{χ}^{#} := (−1)^{n+1} ∗_{χ} d_{χ} ∗_{χ} . Then the flat Laplacian Δ^{#} is defined as

Δ_{χ}^{#} = δ_{χ}^{#}d_{χ}.

If χ is unitary, Δ_{χ}^{#} equals Δ_{χ}. For an arbitrary χ we pick any Hermitian fiber metric in E_{χ} and use it together with the Riemannian metric on X to introduce an inner product in C^{∞}(X, E_{χ}). In general, Δ_{χ}^{#} is a not self-adjoint w.r.t. this inner product. However, if we define the corresponding Laplace operator Δ_{χ} as above by δ_{χ}d_{χ}, where the formal adjoint δ_{χ} is taken w.r.t. to the inner product, then Δ_{χ}^{#} has the same principal symbol as Δ_{χ}. This implies that the operator Δ_{χ}^{#} has nice spectral properties. Its spectrum is discrete and contained in a positive cone C ⊂ ℂ with ℝ^{+} ⊂ C (see [Sh]). It follows that an Agmon angle θ exists for Δ_{χ}^{#} and we can define φ ((Δ_{χ}^{#})_{θ}^{1/2}) by the usual functional

calculus [Sh]. This is a trace class operator. Since φ is assumed to be even, φ ((Δ_{χ}^{#})_{θ}^{1/2}) is independent of θ and we can delete θ from the notation. Lidkii’s theorem [GK, Theorem 8.4] generalizes (1.1). As mentioned above, the spectrum spec(Δ_{χ}^{#}) of Δ_{χ}^{#} is discrete and consists of eigenvalues only. For λ ∈ spec(Δ_{χ}^{#}) let m(λ) denote the algebraic multiplicity of λ, i.e., m(λ) is the dimension of the root space which consists of all f ∈ C^{∞}(X, E_{χ}) such that there is N ∈ ℕ with (Δ_{χ}^{#} − λ I)^{N}f = 0. Then by Lidskii’s theorem we have

(1.3)  Tr φ((Δ_{χ}^{#})^{1/2}) = ∑_{λ∈spec(Δ_{χ}^{#})} m(λ)φ(λ^{1/2}).

The first version of our trace formula generalizes (1.2) with Tr φ ((Δ_{χ}^{#})^{1/2}) on the left hand side.

Actually, we prove a more general result. Let τ be an irreducible representation of K and E_{τ} → Γ\S the associated locally homogeneous vector bundle, equipped with its canonical invariant connection ∇^{τ}. Let ∇ = ∇^{τ,χ} be the product connection in E_{τ} ⊗ E_{χ}, and let Δ_{τ,χ}^{#} = − Tr(∇^{2}) be the corresponding connection Laplacian. Then for φ as above φ ((Δ_{τ,χ}^{#})^{1/2}) is a trace class operator and we establish a trace formula for this operator which is similar to the scalar case.

If G has split rank one, we get an explicite version of the trace formula. To describe it we need to introduce some notation. Let G = KAN be an Iwasawa decomposition of G. Then dim A = 1. Let 𝔞 be the Lie algebra of A. The restriction of the Killing form to 𝔞^{∗} defines an inner product on 𝔞^{∗}. Let |ρ| denote the norm of the half-sum ρ of positive roots of (G, A). Let γ ∈ Γ \ {e}. Then there is a unique closed geodesic τ_{γ} that corresponds to the Γ-conjugacy class {γ}_{Γ} of γ. Denote by l(γ) the length of τ_{γ}. Furthermore, let γ_{0} ∈ Γ be the unique primitive element such that γ = γ_{0}^{k} for some k ∈ ℕ. Finally let D(γ) be the discriminant of γ (see (6.2) for its definition). Let β(λ)dλ be the Plancherel measure for spherical functions on G [He]. We can now state our main result in the scalar case which is the following theorem.

Theorem 1.1. For all even φ ∈ S(ℝ) with φ̂ ∈ C_{c}^{∞}(ℝ) we have

∑_{λ∈spec(Δ_{χ}^{#})} m(λ)φ ((λ − |ρ|^{2})^{1/2}) = dim(V_{χ})\frac{vol(Γ\S)}{2} ∫_{ℝ} φ(λ)β(λ) dλ

+ ∑_{\{γ\}_{Γ}≠e} tr χ(γ)\frac{l(γ_{0})}{D(γ)}φ̂(l(γ)).

(1.4)

Note that for every c > 0 there are only finitely many conjugacy classes {γ}_{Γ} with l(γ) ≤ c. Therefore the sum on the right hand side is finite.

To describe our method we restrict attention to the scalar case, i.e, we consider the operator Δ_{χ}^{#}. Our method is based on the approach of Bunke and Olbrich [BO] to the Selberg trace formula in the unitary case. We consider the wave equation

(1.5)  (\frac{∂^{2}}{∂t^{2}} + Δ_{χ}^{#}) u(t) = 0, u(0) = f, u_{t}(0) = 0,

for any initial conditions f ∈ C^{∞}(X, E_{χ}). Since the principal symbols of Δ_{χ^{#}} is given by σ(x, ξ) =∥ ξ ∥^{2} Id_{E_{x}}, the operator L = \frac{∂^{2}}{∂t^{2}} + Δ_{χ}^{#} is strictly hyperbolic in the sense of [Ta1, Chapt. IV, §3]. Therefore (6.2) has a unique solution u(t; f). Let φ ∈ S(ℝ) be even such that φ̂ ∈ C_{c}^{∞}(ℝ). Then it follows that

(1.6)  φ ((Δ_{χ}^{#})^{1/2}) f = \frac{1}{\sqrt{2π}} ∫_{ℝ} φ̂(t)u(t; f) dt.

Let ũ(t; f) and f̃ denote the lift of u(t, f) and f, respectively, to the universal covering S of X. Then the corresponding wave equation on S with initial conditions u(0) = f̃, u_{t}(0) = 0 is also strictly hyperbolic and by finite propagation speed it follows that it has a unique solution u(t; f̃). Thus we obtain ũ(t, x̃; f) = u(t, x̃; f̃). Since the lift of E_{χ} to S is trivial, the lifted operator Δ̃_{χ}^{#} takes the form Δ̃_{χ}^{#} = Δ̃ ⊗ Id_{V_{χ}}, where Δ̃ is the Laplace operator

on S. Let h_{φ} ∈ C_{c}^{∞}(G//K) be the kernel of the G-invariant integral operator φ \binom{}{Δ̃^{1/2}}.

Then it follows that the kernel K_{φ}(x, y) of φ ((Δ_{χ}^{#})^{1/2}) is given by

(1.7)  K_{φ}(x, y) = ∑_{γ∈Γ} h_{φ}(g_{1}^{−1}γg_{2})χ(γ),

where x = Γg_{1}K and y = Γg_{2}K. One can now proceed in the same way as in the case of a unitary representation χ and derive the twisted Selberg trace formula.

Besides unitary representations of Γ, there is a second class of representations of Γ for which the usual trace formula can be applied. These are representations which are the restriction to Γ of a finite-dimensional representation η: G → GL(E). Let E_{η} → X be the flat vector bundle associated to η|_{Γ}. Then E_{η} is canonically isomorphic to the locally homogeneous vector bundle E_{τ} associated to the principal K-bundle Γ\G → X via the representation τ = η|_{K}. The bundle carries a canonical Hermitian fiber metric and the Laplacian in C^{∞}(X, E_{η}) with respect to this metric is closely related to the Casimir operator acting in C^{∞}(X, E_{τ}). This brings us back to the usual framework of the Selberg trace formula for locally homogeneous vector bundles. Details will be discussed in section 7.

The paper is organized as follows. In section 2 we collect a number of facts about spectral theory of elliptic operators with leading symbol of Laplace type and we develop some functional calculus for such operators. The kernels of the associated integral operators are studied in section 3. Especially, we prove (1.6) and (1.7). In section 4 we apply these results to the case of twisted Bochner-Laplace operators. In section 5 we turn to the locally symmetric case and we prove the first version of the trace formula which is Theorem 5.1. In section 6 we specialize to the case where G has split rank one and we prove Theorem 1.1. In the final section 7 we are concerned with representations of Γ which are the restriction of a representation of G.

2. Functional calculus

In this section we develop the necessary facts of the functional calculus we are going to use in this paper.

Let X be a closed Riemannian manifold of dimension n and E → X a Hermitian vector bundle over X. We denote by C^{∞}(X, E) the space of smooth sections of E, and by L^{2}(X, E) the space of L^{2}-sections of E w.r.t. the metrics on X and E. Let

P: C^{∞}(X, E) → C^{∞}(X, E)

be an elliptic differential operator of order 2 with leading symbol

(2.1)  σ(P)(x, ξ) =∥ ξ ∥_{x}^{2} · Id_{E_{x}} .

For I ⊂ [0, 2π] let

Λ_{I} = {re^{iθ}: 0 ≤ r < ∞, θ ∈ I}.

be the solid angle attached to I. The following lemma describes the structure of the spectrum of P.

Lemma 2.1. For every 0 < ε < π/2 there exists R > 0 such that the spectrum of P is contained in the set B_{R}(0) ∪ Λ_{[−ε,ε]}. Moreover the spectrum of P is discrete.

Proof. The first statement follows from [Sh, Theorem 9.3]. The discreteness of the spectrum follows from [Sh, Theorem 8.4]. □

It follows from Lemma 2.1 that there exists an Agmon angle θ for P and we can define the square root P_{θ}^{1/2} as in [Sh]. For the convenience of the reader we include some details. Denote by spec(P) the spectrum of P. Let ε > 0. For simplicity we assume that 0 ∉ σ(P). By Lemma 2.1 there exist 0 < θ < 2π and ε > 0 such that

spec(P) ∩ Λ_{[θ−ε,θ+ε]} = ∅.

θ is called an Agmon angle for P. Since spec(P) is discrete and 0 ∉ σ(P), there exists also r_{0} > 0 such that

spec(P) ∩ {z ∈ ℂ: |z| < 2r_{0}} = ∅.

Define the contour Γ = Γ_{θ,r_{0}} ⊂ ℂ as the union of three curves Γ = Γ_{1} ∪ Γ_{2} ∪ Γ_{3}, where

\begin{aligned}Γ_{1} = \{re^{iθ}: r_{0} & ≤ r < ∞\}, Γ_{2} = \{r_{0}e^{iα}: θ ≤ α ≤ θ + 2π\}, \\ & Γ_{3} = \{re^{i(θ+2π)} : r_{0} ≤ r < ∞\}. \end{aligned}

Put

P_{θ}^{−1/2} = \frac{i}{2π} ∫_{Γ_{θ,r_{0}}} λ^{−1/2}(P − λ)^{−1} dλ.

By [Sh, Corollary 9.2, Chapt. II, §9] we have ∥ (P −λ)^{−1} ∥≤ C|λ|^{−1} for λ ∈ Γ_{θ,r_{0}}. Therefore the integral is absolutely convergent. Put

P_{θ}^{1/2} = P · P_{θ}^{−1/2}.

Then P_{θ}^{1/2} satisfies (P_{θ}^{1/2})^{2} = P. If θ is fixed, we simply denote this operator by P^{1/2}. We recall [See], [Sh, Theorem 11.2] that P^{1/2} is a classical pseudo-differential operator with principal symbol

σ(P^{1/2})(x, ξ) =∥ ξ ∥_{x} · Id_{E_{x}} .

In any coordinate chart, the complete symbol q(x, ξ) of P^{1/2} has an asymptotic expansion

q(x, ξ) ∼ ∑_{j=0}^{∞} q_{1−j}(x, ξ),

where q_{1−j}(x, ξ) is homogeneous in ξ of order 1 − j. The same holds for Δ_{E}^{1/2}. Since the principal symbols coincide, we get

(2.2)  P^{1/2} = Δ_{E}^{1/2} + B,

where B is a pseudo-differential operator of order zero. Especially, B is a bounded operator in L^{2}(X, E).

Let R_{λ}(P^{1/2}) = (P^{1/2} − λ I)^{−1} and R_{λ}(Δ_{E}^{1/2}) = (Δ_{E}^{1/2} − λ I)^{−1} be the resolvents of P^{1/2} and Δ_{E}^{1/2}, respectively. For λ ∉ spec(Δ_{E}^{1/2}) we have the following equality

(2.3)  P^{1/2} − λ I = (I +BR_{λ}(Δ_{E}^{1/2}))(Δ_{E}^{1/2} − λ I).

Since Δ_{E}^{1/2} is self-adjoint, the resolvent of Δ_{E}^{1/2} satisfies

(2.4)  ∥ R_{λ}(Δ_{E}^{1/2}) ∥≤ | Im(λ)|^{−1}

[Ka, Chapt. V, §3.5]. Let b = 2 ∥ B ∥. It follows from (2.4) that for | Im(λ)| ≥ b we have ∥ BR_{λ}(Δ_{E}^{1/2}) ∥≤ 1/2. Thus in this range of λ the operator I +BR_{λ}(Δ_{E}^{1/2}) is invertible and

R_{λ}(P^{1/2}) = R_{λ}(Δ_{E}^{1/2})(I +BR_{λ}(Δ_{E}^{1/2}))^{−1}.

Combined with (2.4) we get

(2.5)  ∥ R_{λ}(P^{1/2}) ∥≤ 2| Im(λ)|^{−1}, | Im(λ)| ≥ b.

We can now summarize the spectral properties of P^{1/2}.

Lemma 2.2. The resolvent of P^{1/2} is compact. The spectrum of P^{1/2} is discrete. There exist b > 0 and c ∈ ℝ such that the spectrum of P^{1/2} is contained in the domain

(2.6)  Ω_{b,c} = {λ ∈ ℂ: Re(λ) > c, | Im(λ)| < b}.

Proof. Since P^{1/2} is an elliptic pseudo-differential operator of order 1 on a closed manifold, its resolvent is compact and hence, its spectrum is discrete. The remaining statements are a consequence of (2.5). □

Though P is not self-adjoint in general, it still has nice spectral properties [Sh, Chapt. I, §8]. Given λ_{0} ∈ spec(P), let Γ_{λ_{0}} be a small circle around λ_{0} which contains no other points of spec(P). Put

Π_{λ_{0}} = \frac{i}{2π} ∫_{Γ_{λ_{0}}} R_{λ}(P) dλ.

Then Π_{λ_{0}} is the projection onto the root subspace V_{λ_{0}}. This is a finite-dimensional subspace of C^{∞}(X, E) which is invariant under P and there exists N ∈ ℕ such that (P −λ_{0} I)^{N}V_{λ_{0}} = 0. Furthermore, there is a closed complementary subspace V′_{λ_{0}} to V_{λ_{0}} in L^{2}(X, E) which is invariant under the closure P̄ of P in L^{2} and the restriction of (P̄ − λ_{0} I) to V′_{λ_{0}} has a bounded inverse. The algebraic multiplicity m(λ_{0}) of λ_{0} is defined as

m(λ_{0}) = dim V_{λ_{0}}.

If λ_{1}, λ_{2} ∈ spec(P) with λ_{1} ≠ λ_{2}, then the projections Π_{λ_{1}} and Π_{λ_{2}} are disjoint, i.e.,

Π_{λ_{1}}Π_{λ_{2}} = Π_{λ_{2}}Π_{λ_{1}} = 0.

Since the principal symbols of P and Δ_{E} are the same, we have P = Δ + D, where D is a first order differential operator. Let R_{λ}(Δ) be the resolvent of Δ. Then DR_{λ}(Δ) is a compact operator. This means that D is compact relative to Δ and it follows from [Mk, I,§4, Theorem 4.3] that the root vectors are complete. Thus L^{2}(X, E) is the closure of the algebraic direct sum of finite-dimensional P -invariant subspaces V_{k}

(2.7)  L^{2}(X, E) = \overline{⨁_{k≥1} V_{k}}

such that the restriction of P to V_{k} has a unique eigenvalue λ_{k} and |λ_{k}| → ∞. In general, the sum (2.7) is not a sum of mutually orthogonal subspaces.

It follows from (2.7) that P_{θ}^{1/2} has a similar spectral decomposition with eigenvalues λ_{θ}^{1/2}, λ ∈ spec(P), and m(λ_{θ}^{1/2}) = m(λ).

Given r > 0, let

N(r, P) := ∑_{λ∈spec(P), |λ|≤r} m(λ).

be the counting function of the eigenvalues of P, where eigenvalues are counted with their algebraic multiplicity.

Lemma 2.3. Let n = dim X. We have

N(r, P) = \frac{rk(E) vol(X)}{(4π)^{n/2}Γ(n/2 + 1)}r^{n/2} + o(r^{n/2}), r → ∞.

Proof. It is well know that Tr(e^{−tΔ_{E}}) has an asymptotic expansion of the form

Tr(e^{−tΔ_{E}}) ∼ t^{−n/2} ∑_{k≥0} a_{k}t^{k}, t → +0

(see [Gi, Lemma 1.8.3]), and by [Gi, Lemma 4.1.4] the leading coefficient a_{0} is given by a_{0} = (4π)^{−n/2} rk(E) vol(X). Let N(r, Δ_{E}) be the counting function of the eigenvalues of Δ_{E}. Using the Tauberian theorem (see [Sh, Chapt. II, §14]), we get

(2.8)  N(r, Δ_{E}) = \frac{rk(E) vol(X)}{(4π)^{n/2}Γ(n/2 + 1)}r^{n/2} + o(r^{n/2}), r → ∞.

The lemma follows from [Mk, I, §8, Corollary 8.5]. □

Let h ∈ C_{c}^{∞}(ℝ) be even and set

φ(λ) = \frac{1}{\sqrt{2π}} ∫_{ℝ} h(r) cos(tλ) dr, λ ∈ ℂ.

Then φ(λ) is rapidly decreasing in each strip | Im(λ)| < δ, δ > 0. For b > 0 and d ∈ ℝ let Γ = Γ_{b,d} ⊂ ℂ be the contour which is union of the two half-lines L_{±b,d} = {z ∈ ℂ: Im(z) = ±b, Re(z) ≥ d} and the semi-circle S = {d + be^{iθ}: π/2 ≤ θ ≤ 3π/2}. By Lemma 2.2 there exist b > 0, d ∈ ℝ such that spec(P^{1/2}) is contained in the interior of Γ_{b,d}. Put

(2.9)  φ(P^{1/2}) := \frac{i}{2π} ∫_{Γ} φ(λ)(P^{1/2} − λ I)^{−1} dλ.

It follows from (2.5) that the integral is absolutely convergent.

Lemma 2.4. φ(P^{1/2}) is an integral operator with a smooth kernel.

Proof. First note that

∫_{Γ} φ(λ)(P − λ^{2})(P^{1/2} − λ)^{−1} dλ = ∫_{Γ} φ(λ)(P^{1/2} + λ) dλ = 0.

This implies that for k, l ∈ ℕ we have

P^{k}φ(P^{1/2})P^{l} = \frac{i}{2π} ∫_{Γ} λ^{2(k+l)}φ(λ)(P^{1/2} − λ)^{−1} dλ.

The function λ ↦ λ^{2(k+l)}φ(λ) is rapidly decreasing on | Im(λ)| = ±b. Hence P^{k}φ(P^{1/2})P^{l} is a bounded operator in L^{2}. Since P is elliptic, it follows that for all s, r ∈ ℝ, φ(P^{1/2}) extends to a bounded operator from H^{s}(X, E) to H^{r}(X, E), which shows that φ(P^{1/2}) is a smoothing operator and hence, it is an integral operator with a smooth kernel. □

In order to continue, we need to establish an auxiliary result about smoothing operators. Let

A: L^{2}(X, E) → L^{2}(X, E)

be an integral operator with a smooth kernel H ∈ C^{∞}(X × X, E ⊠ E^{∗}).

Proposition 2.5. A is a trace class operator and

(2.10)  Tr(A) = ∫_{X} tr H(x, x) dμ(x).

Proof. We generalize the proof of Theorem 1 in [La, Chapt VII, §1]. Let ∇^{E} be a Hermitian connection in E and let Δ_{E} = (∇^{E})^{∗}∇^{E} be the associated Bochner-Laplace operator. Then Δ_{E} is a second order elliptic operator which is essentially self-adjoint and non-negative. Its spectrum is discrete. Let {ϕ_{j}}_{j∈ℕ} be an orthonormal basis of L^{2}(X, E) consisting of eigensections of Δ_{E} with eigenvalues 0 ≤ λ_{1} ≤ λ_{2} ≤ ⋯ → ∞. We can expand H in the orthonormal basis as

(2.11)  H(x, y) = ∑_{i,j=1}^{∞} a_{i,j}ϕ_{i}(x) ⊗ ϕ_{j}^{∗}(y),

where

(2.12)  a_{i,j} = ⟨Aϕ_{i}, ϕ_{j}⟩.

Since H is smooth, the coefficients a_{i,j} are rapidly decreasing. Indeed, for every N we have

(1 + λ_{i} + λ_{j})^{N}a_{i,j} = ⟨(I +Δ_{E} ⊗ I + I ⊗Δ_{E})^{N}H, ϕ_{i} ⊗ ϕ_{j}^{∗}⟩.

Hence for every N ∈ ℕ there exists C_{N} > 0 such that

|a_{i,j}| ≤ C_{N}(1 + λ_{i} + λ_{j})^{−N}, i, j ∈ ℕ.

This implies that the series (2.11) converges in the C^{∞} topology. Let P_{i,j} be the integral operator with kernel ϕ_{i} ⊗ ϕ_{j}^{∗}. Thus

P_{i,j}(ϕ_{k}) = \begin{cases}0, & k ≠ j; \\ ϕ_{i}, & k = j. \end{cases}

Let P_{j} be the orthogonal projection of L^{2}(X, E) onto the 1-dimensional subspace ℂϕ_{j}. Put

B = ∑_{i,j} a_{i,j}(1 + λ_{j})^{n}P_{i,j}, C = ∑_{j}(1 + λ_{j})^{−n}P_{j}.

Then A = BC and it follows from (2.8) that B and C are Hilbert-Schmidt operators. Thus A is a trace class operator. Furthermore, by (2.11) and (2.12) we get

∫_{X} tr H(x, x) dx = ∑_{i,j=1}^{∞} a_{i,j} ∫_{X}⟨ϕ_{i}(x), ϕ_{j}(x)⟩ dx = ∑_{i=1}^{∞} a_{i,i} = Tr(A).

□

Now we apply this result to φ(P^{1/2}). Let K_{φ}(x, y) be the kernel of φ(P^{1/2}). Then by Proposition 2.5, φ(P^{1/2}) is a trace class operator and we have

(2.13)  Tr φ(P^{1/2}) = ∫_{X} tr K_{φ}(x, x) dμ(x).

By Lidskii’s theorem [GK, Theorem 8.4] the trace is equal to the sum of the eigenvalues of φ(P^{1/2}), counted with their algebraic multiplicities. The eigenvalues of φ(P^{1/2}) and their algebraic multiplicities can be determined as follows. Given N ∈ ℕ, let Π_{N} denote the projection onto the direct sum of the root subspaces V_{k}, k ≤ N, of P. As explained above, we have

P Π_{N} = ∑_{k=1}^{N}(λ_{k}Π_{k} + D_{k}),

where Π_{k} is the projection onto V_{k} and D_{k} is a nilpotent operator in V_{k}. Then it follows from [Ka, I, (5.50)] that

φ(P^{1/2})Π_{N} = ∑_{k=1}^{N}(φ(λ_{k}^{1/2})Π_{k} + D′_{k}),

where D′_{k} is again a nilpotent operator in V_{k}. Thus φ(P^{1/2}) leaves the decomposition (2.7) invariant and the restriction of φ(P^{1/2}) to V_{k} has a unique eigenvalue φ(λ_{k}^{1/2}). Of course, some of the eigenvalues φ(λ_{k}^{1/2}) may coincide in which case the root space is the sum of the corresponding root spaces V_{k}. Now, applying Lidskii’s theorem [GK, Theorem 8.4] and (2.13), we get the following proposition.

Proposition 2.6. Let φ ∈ S(ℝ) be even and suppose that φ̂ ∈ C_{c}^{∞}(ℝ). Then we have

(2.14)  ∑_{λ∈spec(P)} m(λ)φ(λ^{1/2}) = ∫_{X} tr K_{φ}(x, x) dx.

By Lemma 2.3, the series on the left hand side is absolutely convergent.

Remark. Recall that P^{1/2} = P_{θ}^{1/2} depends on the choice of an Agmon angle θ, and so do the eigenvalues λ_{k}^{1/2} = (λ_{k})_{θ}^{1/2}. Let 0 < θ < θ′ < 2π be two Agmon angles. Then it follows from Lemma 2.1 that there are only finitely many eigenvalues λ_{1}, ..., λ_{m} of P which are contained in Λ_{[θ,θ′]}. Therefore for λ ∈ spec(P) we have

(λ)_{θ′}^{1/2} = \begin{cases}(λ)_{θ}^{1/2}, & if λ ∉ \{λ_{1}, ..., λ_{m}\}; \\ −(λ)_{θ}^{1/2}, & if λ ∈ \{λ_{1}, ..., λ_{m}\}. \end{cases}

Since φ is even φ ((λ)_{θ}^{1/2}) is independent of θ. This justifies the notation on the left hand side of (2.14). □

3. The kernel and the wave equation

In this section we give a description of the kernel K_{φ} of the smoothing operator φ(P^{1/2}) in terms of the solution of the wave equation. Consider the wave equation

(3.1)  \frac{∂^{2}u}{∂t^{2}} + P u = 0, u(0, x) = f(x), u_{t}(0, x) = 0.

Proposition 3.1. For each f ∈ C^{∞}(X, E) there is a unique solution u(t; f) ∈ C^{∞}(ℝ × X, E) of the wave equation (3.1) with initial condition f. Moreover for every T > 0 and s ∈ ℝ there exists C > 0 such that for every f ∈ C^{∞}(X, E)

(3.2)  ∥ u(t, f) ∥_{s}≤ C ∥ u(0, f) ∥_{s}, |t| ≤ T,

where ∥ · ∥_{s} denotes the s-Sobolev norm.

Proof. We proceed in the same way as in [Ta1, Chapt. IV, §§1,2] and replace (3.1) by a first oder system. Let Δ_{E} be the Bochner-Laplace operator associated to the connection ∇^{E} in E. Put Λ = (Δ_{E} + Id)^{1/2} and

(3.3)  L := (\begin{matrix}0 & Λ \\ −P Λ^{−1} & 0 \end{matrix}) : \begin{matrix}C^{∞}(X, E) \\ ⊕ \\ C^{∞}(X, E) \end{matrix} ⟶ \begin{matrix}C^{∞}(X, E) \\ ⊕ \\ C^{∞}(X, E) \end{matrix}.

Then L is a pseudo-differential operator of order 1. Let u be a solution of (3.1). Put

u_{1} = Λu, u_{2} = \frac{∂}{∂t}u.

Then (u_{1}, u_{2}) satisfies

(3.4)  \frac{∂}{∂t} (\begin{matrix}u_{1} \\ u_{2} \end{matrix}) = L (\begin{matrix}u_{1} \\ u_{2} \end{matrix}) , u_{1}(0) = Λf, u_{2}(0) = 0.

On the other hand, let (u_{1}, u_{2}) be a solution of the initial value problem (3.4). Put u = Λ^{−1}u_{1}. Then u is a solution of (3.1). Thus it suffices to consider (3.4). By (2.1) it follows that P = Δ_{E} + D where D is a differential operator of order ≤ 1. Therefore we get

P Λ^{−1} = (Δ_{E} + Id)Λ^{−1} + (D − Id)Λ^{−1} = Λ + B_{1},

where B_{1} is a pseudo-dfifferential operator of order 0. Therefore

L + L^{∗} = (\begin{matrix}0 & Λ \\ −Λ − B_{1} & 0 \end{matrix}) + (\begin{matrix}0 & −Λ − B_{1}^{∗} \\ Λ & 0 \end{matrix}) = − (\begin{matrix}0 & B_{1}^{∗} \\ B_{1} & 0 \end{matrix}) ,

is a pseudo-differential operator of order zero. Hence (3.4) is a symmetric hyperbolic system in the sense of [Ta1, Chapt. IV, §2]. So we can proceed as in the proof of Theorem 2.3 in [Ta1, Chapt. IV, §2] to establish existence and uniqueness of solutions of (3.1). The estimation (3.2) follows from the proof using Cronwall’s inequality. □

Proposition 3.2. Let φ ∈ S(ℝ) be even such that φ̂ ∈ C_{c}^{∞}(ℝ). Then for every f ∈ C^{∞}(X, E) we have

φ(P^{1/2})f = \frac{1}{\sqrt{2π}} ∫_{ℝ} φ̂(t)u(t; f) dt.

Proof. Let Γ ⊂ ℂ be as in (2.9). Let c ≥ 0 be such that the spectrum of P + c is contained in Re(z) > 0. For σ > 0 define the operator cos(tP^{1/2})e^{−σ(P +c)} by the functional integral

cos(tP^{1/2})e^{−σ(P +c)} = \frac{i}{2π} ∫_{Γ} cos(tλ)e^{−σ(λ^{2}+c)}(P^{1/2} − λ)^{−1} dλ.

By (2.5) the integral is absolutely convergent. For f ∈ C^{∞}(X, E) and σ > 0 put

(3.5)  u(t; σ, f) := cos(tP^{1/2})e^{−σ(P +c)}f.

Then u(t; σ, f) satisfies

(\frac{∂^{2}}{∂t^{2}} + P ) u(t; σ, f) = \frac{i}{2π} ∫_{Γ} cos(tλ)e^{−σ(λ^{2}+c)}(P − λ^{2})(P^{1/2} − λ)^{−1} dλ

= \frac{i}{2π} ∫_{Γ} cos(tλ)e^{−σ(λ^{2}+c)}(P^{1/2} + λ) dλ = 0.

and u(0; σ, f) = e^{−σ(P +c)}f. Thus u(t; σ, f) is the unique solution of (3.1) with initial condition e^{−σ(P +c)}f. Then u(t; f) − u(t; σ, f) is the solution of (3.1) with initial condition f − e^{−(P +c)}f. Hence by (3.2) we get for all s ∈ ℝ

(3.6)  ∥ u(t; f) − u(t; σ, f) ∥_{H^{s}}≤ C ∥ f − e^{−σ(P +c)}f ∥_{H^{s}}, |t| ≤ T.

Now note that for every f ∈ C^{∞}(X, E) we have

lim_{σ→0} ∥ e^{−σ(P +c)}f − f ∥= 0.

This follows from the parametrix construction. Hence we get

∥ f − e^{−σ(P +c)}f ∥_{H^{s}} = ∥ (P + c)^{s/2}f − e^{−σ(P +c)}(P + c)^{s/2}f ∥_{L^{2}}→ 0

as σ → 0. Combined with (3.6) we get

(3.7)  lim_{σ→0} ∥ u(t; f) − u(t; σ, f) ∥_{H^{s}}= 0.

Furthermore we have

\frac{1}{\sqrt{2π}} ∫_{ℝ} φ̂(t)u(t; σ, f) dt = \frac{1}{\sqrt{2π}} ∫_{ℝ} φ̂(t)\frac{i}{2π} ∫_{Γ} cos(tλ)e^{−σ(λ^{2}+c)}(P^{1/2} − λ)^{−1}f dλ dt

= \frac{i}{2π} ∫_{Γ} (\frac{1}{\sqrt{2π}} ∫_{ℝ} φ̂(t) cos(tλ) dt) e^{−σ(λ^{2}+c)}(P^{1/2} − λ)^{−1}f dλ

= \frac{i}{2π} ∫_{Γ} φ(λ)e^{−σ(λ^{2}+c)}(P^{1/2} − λ)^{−1}f dλ.

For σ → 0, the right hand side converges to φ(P^{1/2})f. By (3.7) the left hand side converges to (2π)^{−1/2} ∫_{ℝ} φ̂(t)u(t; f) dt. □

Let p: X̃ → X be the universal covering of X, Ẽ = p^{∗}E, and P̃: C^{∞}(X̃, Ẽ) → C^{∞}(X̃, Ẽ) the lift of P to X̃. Let ũ(t, x̃, f) and f̃be the pull back to X̃ of u(t, x; f) and f, respectively. Then ũ(t, f) satisfies

(3.8)  (\frac{∂^{2}}{∂t^{2}} + P̃) ũ(t; f) = 0, ũ(0; f) = f̃, ũ_{t}(0, f) = 0.

By (2.1) we have P̃ = Δ̃_{E} + D̃, where D̃ is a differential operator of order ≤ 1. Then it follows from energy estimates as in [Ta2, Chapt. 2, §8] that solutions of (\frac{∂^{2}}{∂t^{2}} +P̃)u = 0 have finite propagation speed. This implies that for every ψ ∈ C^{∞}(X̃, Ẽ) the wave equation

(\frac{∂^{2}}{∂t^{2}} + P̃) u(t; ψ) = 0, u(0; ψ) = ψ, u_{t}(0; ψ) = 0,

has a unique solution. Hence we get

(3.9)  ũ(t; f) = u(t, f̃).

Let d(x, y) denote the geodesic distance of x, y ∈ X̃. For δ > 0 let

U_{δ} = {(x, y) ∈ X̃ × X̃: d(x, y) < δ}.

Proposition 3.3. There exist δ > 0 and H_{φ} ∈ C^{∞}(X̃ ×X̃, Hom(Ẽ, Ẽ)) with supp H_{φ} ⊂ U_{δ} such that for all ψ ∈ C^{∞}(X̃, Ẽ) we have

\frac{1}{\sqrt{2π}} ∫_{ℝ} φ̂(t)u(t, x̃; ψ) dt = ∫X̃ H_{φ}(x̃, ỹ)(ψ(ỹ)) dỹ.

Proof. Suppose that supp φ̂ ⊂ [−T, T]. Let V ⊂ X̃ be an open relatively compact subset. For r > 0 let

V_{r} = {y ∈ V : d(y, V) < r}.

Let χ ∈ C_{c}^{∞}(V_{2T}) such that χ = 1 on V_{T}. By finite propagation speed, we have

u(t, x̃; ψ) = u(t, x̃; χψ), x̃ ∈ V, |t| < T,

for all ψ ∈ C^{∞}(X̃, Ẽ). Thus we are reduced to the case of a compact manifold and the proof follows from Lemma 2.4 and Proposition 3.2. □

Using Proposition 3.3 together with (3.9) and Proposition 3.2, we obtain

(3.10)  φ(P^{1/2})f(x̃) = ∫X̃ H_{φ}(x̃, ỹ)(f̃(ỹ)) dỹ

for all f ∈ C^{∞}(X, E). Let F ⊂ X̃ be a fundamental domain for Γ. Given γ ∈ Γ, let R_{γ}: Ẽ → Ẽ be the induced bundle map. Thus for each ỹ ∈ X̃, we have a linear isomorphism R_{γ}: Ẽỹ → Ẽ_{γỹ}. Note that f̃ satisfies

f̃(γỹ) = R_{γ}(f̃(ỹ)), γ ∈ Γ.

Then we get

∫X̃ H_{φ}(x̃, ỹ)(f̃(ỹ)) dỹ = ∑_{γ∈Γ} ∫_{γF} H_{φ}(x̃, ỹ)(f̃(ỹ)) dỹ

= ∑_{γ∈Γ} ∫_{F} H_{φ}(x̃, γỹ)(f̃(γỹ)) dỹ

= ∫_{F} (∑_{γ∈Γ} H_{φ}(x̃, γỹ) ∘ R_{γ}) (f̃(ỹ)) dỹ.

Combining this expression with (3.10), it follows that the kernel K_{φ} of φ(P^{1/2}) is given by

(3.11)  K_{φ}(x, y) = ∑_{γ∈Γ} H_{φ}(x̃, γỹ) ∘ R_{γ},

where x̃ and ỹ are any lifts of x and y to the fundamental domain F. So by Proposition 2.6 we get

Proposition 3.4. Let φ ∈ S(ℝ) be even and suppose that φ̂ ∈ C_{c}^{∞}(ℝ). Then we have

∑_{λ∈spec(P)} m(λ)φ(λ^{1/2}) = ∑_{γ∈Γ} ∫_{F} tr(H_{φ}(x̃, γx̃) ∘ R_{γ}) dx̃.

Note that the sum on the right is finite.

4. The twisted Bochner-Laplace operator

Let E → X be a complex vector bundle with covariant derivative ∇. Define the invariant second covariant derivative ∇^{2} by

∇_{U,V}^{2} = ∇_{U}∇_{V} − ∇_{∇_{U} V} ,

where U, V are any two vector fields on X. Then the connection Laplacian Δ^{#} is defined by

Δ^{#} = − Tr(∇^{2}).

Let (e_{1}, ..., e_{n}) be a local frame field. Then

Δ^{#} = − ∑_{j} ∇_{e_{j},e_{j}}^{2}.

This formula implies that the principal symbol of Δ^{#} is given by

σ(Δ^{#})(x, ξ) =∥ ξ ∥_{x}^{2} Id_{E_{x}} .

Thus the results of the previous section can be applied to Δ^{#}.

Assume that E is equipped with a Hermitian fiber metric and ∇ is compatible with the metric. Then it follows that

(4.1)  ∇^{∗}∇ = − Tr ∇^{2},

[LM, p.154], i.e., the connection Laplacian equals the Bochner-Laplace operator Δ_{E} = ∇^{∗}∇.

Now let ρ: π_{1}(X) → GL(V) be a finite-dimensional complex representation of π_{1}(X). Let F → X be the associated flat vector bundle with connection ∇^{F}. Let E be a Hermitian vector bundle with Hermitian connection ∇^{E}. We equip E⊗F with the product connection ∇^{E⊗F}, which is defined by

∇_{Y}^{E⊗F} = ∇_{Y}^{E} ⊗ 1 + 1 ⊗ ∇_{Y}^{F} ,

for Y ∈ C^{∞}(X, T X). Let Δ_{E,ρ}^{#} be the connection Laplacian associated to ∇^{E⊗F}. Locally it can be described as follows. Let U ⊂ X be an open subset such that F |_{U} is trivial. Then (E ⊗ F)|_{U} is isomorphic to the direct sum of m = rank(F) copies of E|_{U}:

(E ⊗ F)|_{U} ≅ ⊕_{i=1}^{m}E|_{U}.

Let e_{1}, ..., e_{m} be a basis of flat sections of F |_{U}. Then each φ ∈ C^{∞}(U, (E ⊗ F)|_{U}) can be written as

φ = ∑_{j=1}^{m} φ_{j} ⊗ e_{j},

where φ_{i} ∈ C^{∞}(U, E|_{U}), i = 1, ..., m. Then

∇_{Y}^{E⊗F} (φ) = ∑_{j}(∇_{Y}^{E}φ_{j}) ⊗ e_{j}.

Let Δ_{E} = (∇^{E})^{∗}∇^{E} be the Bochner-Laplace operator associated to ∇^{E}. Using (4.1), we get

(4.2)  Δ_{E,ρ}^{#}φ = ∑_{j}(Δ_{E}φ_{j}) ⊗ e_{j}.

Let Ẽ and F̃ be the pullback to X̃ of E and F, respectively. Then F̃ ≅ X̃ × V and

C^{∞}(X̃, Ẽ ⊗ F̃) ≅ C^{∞}(X̃, Ẽ) ⊗ V.

It follows from (4.2) that with respect to this isomorphism, the lift Δ̃_{E,ρ}^{#} of Δ_{E,ρ}^{#} to X̃ takes the form

Δ̃_{E,ρ}^{#} = Δ̃_{E} ⊗ Id,

where Δ̃_{E} is the lift of Δ_{E} to X̃. Let ψ ∈ C_{c}^{∞}(X̃, Ẽ) ⊗ V. Then the unique solution of the wave equation

(\frac{∂^{2}}{∂t^{2}} + Δ̃_{E,ρ}^{#}) u(t; ψ) = 0, u(0; ψ) = ψ, u_{t}(0, ψ) = 0,

is given by

u(t; ψ) = (cos (t(Δ̃_{E})^{1/2}) ⊗ Id) ψ.

Let φ be as above and let k_{φ}(x̃, ỹ) be the kernel of

φ ((Δ̃_{E})^{1/2}) = \frac{1}{\sqrt{2π}} ∫_{ℝ} φ̂(t) cos (t(Δ̃_{E})^{1/2}) dt.

Then the kernel H_{φ} of Proposition 3.3 is given by H_{φ}(x̃, ỹ) = k_{φ}(x̃, ỹ) ⊗ Id. Let R_{γ}: Ẽỹ → Ẽ_{γỹ} be the canonical isomorphism. Then it follows from (3.11) that the kernel of the operator φ((Δ_{E,ρ}^{#})^{1/2}) is given by

(4.3)  K_{φ}(x, y) = ∑_{γ∈Γ} k_{φ}(x̃, γỹ) ∘ (R_{γ} ⊗ ρ(γ)).

Combined with (3.4) we get

Proposition 4.1. Let F_{ρ} be a flat vector bundle over X, associated to a finite-dimensional complex representation ρ: π_{1}(X) → GL(V). Let Δ_{E,ρ}^{#} be the twisted connection Laplacian acting in C^{∞}(X, E ⊗ F_{ρ}). Let φ ∈ S(ℝ) be even with φ̂ ∈ C_{c}^{∞}(ℝ) and denote by k_{φ}(x̃, ỹ) the kernel of φ ((Δ̃_{E})^{1/2}). Then we have

(4.4)  ∑_{λ∈spec(Δ_{E,ρ}^{#})} m(λ)φ(λ^{1/2}) = ∑_{γ∈Γ} tr ρ(γ) ∫_{F} tr (k_{φ}(x̃, γx̃) ∘ R_{γ}) dx̃.

5. Locally symmetric spaces

In this section we specialize to the case where X is a locally symmetric manifold. Let G be a connected semisimple real Lie group of non-compact type with finite center. Let K ⊂ G be a maximal compact subgroup of G. Denote by 𝔤 and 𝔨 the Lie algebras of G and K, respectively. Let

(5.1)  𝔤 = 𝔭 ⊕ 𝔨

be the Cartan decomposition. Put S = G/K. This is a Riemannian symmetric space of non-positive curvature. The invariant metric is obtained by translation of the restriction of the Killing form to 𝔭 ≅ T_{e}(G/K). Let Γ ⊂ G be a discrete, torsion free, cocompact subgroup. Then Γ acts freely on S by isometries and X = Γ\S is a compact locally symmetric manifold.

Let τ: K → GL(V_{τ}) be a finite-dimensional unitary representation of K, and let

Ẽ_{τ} = (G × V_{τ})/K → G/K

be the associated homogeneous vector bundle, where K acts on the right as usual by

(g, v)k = (gk, τ(k^{−1})v), g ∈ G, k ∈ K, v ∈ V_{τ}.

Let

(5.2)  C^{∞}(G; τ) := {f: G → V_{τ} | f ∈ C^{∞}, f(gk) = τ(k^{−1})f(g), g ∈ G, k ∈ K} .

Similarly, by C_{c}^{∞}(G; τ) we denote the subspace of C^{∞}(G; τ) of compactly supported functions and by L^{2}(G; τ) the completion of C_{c}^{∞}(G; τ) with respect to the inner product

⟨f_{1}, f_{2}⟩ = ∫_{G/K}⟨f_{1}(g), f_{2}(g)⟩ dġ.

There is a canonical isomorphism

(5.3)  C^{∞}(S, Ẽ_{τ}) ≅ C^{∞}(G; τ).

[Mi, p.4]. Similarly, there are isomorphisms C_{c}^{∞}(S, Ẽ_{τ}) ≅ C_{c}^{∞}(G; τ) and L^{2}(S, Ẽ_{τ}) ≅ L^{2}(G; τ).

Let ∇^{τ} be the canonical G-invariant connection on Ẽ_{τ}. It is defined by

∇_{g_{∗}Y}^{τ} f(gK) = \frac{d}{dt}|_{t=0} (g exp(tY))^{−1} f(g exp(tY)K),

where f ∈ C^{∞}(G; τ) and Y ∈ 𝔭. Let Δ̃_{τ} be the associated Bochner-Laplace operator. Then Δ̃_{τ} is G-invariant, i.e., Δ̃_{τ} commutes with the right action of G on C^{∞}(S, Ẽ_{τ}). Let Ω ∈ Z(𝔤_{ℂ}) and Ω_{K} ∈ Z(𝔨_{ℂ}) be the Casimir elements of G and K, respectively. Assume that τ is irreducible. Then with respect to (5.3), we have

(5.4)  Δ̃_{τ} = −R(Ω) + λ_{τ} Id,

where λ_{τ} = τ(Ω_{K}) is the Casimir eigenvalue of τ [Mi, Proposition 1.1]. We note that λ_{τ} ≥ 0.

Let φ ∈ S(ℝ) be even with φ̂ ∈ C_{c}^{∞}(ℝ). Then φ(Δ̃_{τ}^{1/2}) is a G-invariant integral operator. Therefore its kernel k_{φ} satisfies

k_{φ}(gx̃, gỹ) = k_{φ}(x̃, ỹ), g ∈ G.

With respect to the isomorphism (5.3) it can be identified with a compactly supported C^{∞}-function

h_{φ}: G → End(V_{τ}),

which satisfies

h_{φ}(k_{1}gk_{2}) = τ(k_{1}) ∘ h_{φ}(g) ∘ τ(k_{2}), k_{1}, k_{2} ∈ K.

Then φ(Δ̃_{τ}^{1/2}) acts by convolution

(5.5)  (φ(Δ̃_{τ}^{1/2})f) (g_{1}) = ∫_{G} h_{φ}(g_{1}^{−1}g_{2})(f(g_{2})) dg_{2}.

Let

E_{τ} = Γ\Ẽ_{τ}

be the locally homogeneous vector bundle over Γ\S induced by Ẽ_{τ}. Let χ: Γ → GL(V_{χ}) be a finite-dimensional complex representation and let F_{χ} be the associated flat vector bundle over Γ\S. Let Δ_{τ,χ}^{#} be the twisted connection Laplacian acting in C^{∞}(Γ\S, E_{τ} ⊗F_{χ}). Then it follows from (4.3) that the kernel K_{φ} of φ(Δ̃_{τ}^{1/2}) is given by

K_{φ}(g_{1}K, g_{2}K) = ∑_{γ∈Γ} h_{φ}(g_{1}^{−1}γg_{2}) ⊗ χ(γ).

By Proposition 4.1 we get

(5.6)  Tr φ ((Δ_{τ,χ}^{#})^{1/2}) = ∑_{γ∈Γ} tr χ(γ) ∫_{Γ\G} tr h_{φ}(g^{−1}γg) dġ.

We now proceed in the usual way, grouping terms together into conjugacy classes. Given γ ∈ Γ, denote by {γ}_{Γ}, Γ_{γ}, and G_{γ} the Γ-conjugacy class of γ, the centralizer of γ in Γ, and the centralizer of γ in G, respectively. With the conjugacy class {e}_{Γ} separated from the others as usual, we get a first version of the trace formula.

Proposition 5.1. For all even φ ∈ S(ℝ) with φ̂ ∈ C_{c}^{∞}(ℝ) we have

Tr φ ((Δ_{τ,χ}^{#})^{1/2}) = dim(V_{χ}) vol(Γ\S) tr h_{φ}(e)

+ ∑_{{γ}_{Γ}≠e} tr χ(γ) vol(Γ_{γ}\G_{γ}) ∫_{G_{γ}\G} tr h_{φ}(g^{−1}γg) dġ. (5.7)

In order to make this formula more explicit, one needs to express the kernel h_{φ} in terms of φ, and to evaluate the orbital integrals on the right hand side. The kernel h_{φ} can be determined using Harish-Chandra’s Plancherel formula. The orbital integrals can be computed using the Fourier inversion formula. However, both formulae are pretty complicated in the higher rank case. A sufficiently explicit formula can be obtained in the rank one case which we discuss in the next section.

6. The rank one case

Let G and K be as above. We introduce some notation following [Wa]. Let G = KAN be an Iwasawa decomposition of G (see [He]). Then A is a maximal vector subgroup of G and N is a maximal unipotent subgroup of G. In this section we assume that G has split rank one, i.e., dim A = 1. Let M be the centralizer of A in K. We set P = MAN. Then P is a parabolic subgroup of G. Since G has split rank 1, every proper parabolic subgroup of G is conjugate to P.

Denote by Ĝ and M̂ the set of equivalence classes of irreducible unitary representations of G and M, respectively. For π ∈ Ĝ we denote by H_{π} the Hilbert space in which π operates.

Let 𝔞 and 𝔫 be the Lie algebras of A and N, respectively. Choose H ∈ 𝔞 such that ad(H)|_{𝔫} has eigenvalues 1 and possibly 2. Then 𝔞 = ℝH. For t ∈ R we set a_{t} = exp(tH) and log a_{t} = t. Let A^{+} = {a_{t}: t > 0}.

Let ρ be the half-sum of positive roots of (𝔤, 𝔞). Its norm |ρ| with respect to the normalized Killing form is given as follows. Let p and q be the dimensions of the eigenspaces of ad(H)|_{𝔫} with eigenvalues 1 and 2, respectively. Then p > 0 and 0 ≤ q < p. Then

(6.1)  |ρ| = \frac{1}{2}(p + 2q).

For σ ∈ M̂ and λ ∈ ℝ let π_{σ,λ} be the unitarily induced representation from P to G which is defined as in [Wa, p. 177]. Let Θ_{σ,λ} denote the character of π_{σ,λ}.

If γ ∈ Γ, γ ≠ e, then there exists g ∈ G such that gγg^{−1} ∈ MA^{+}. Thus there are m_{γ} ∈ M and a_{γ} ∈ A^{+} such that gγg^{−1} = m_{γ}a_{γ}. By [Wa, Lemma 6.6], a_{γ} depends only on γ and m_{γ} is determined by γ up to conjugacy in M. Let

l(γ) = log a_{γ}.

Then l(γ) is the length of the unique closed geodesic of Γ\S determined by {γ}_{Γ}. Furthermore, by the above remark

(6.2)  D(γ) := e^{−l(γ)|ρ|}| det (Ad(m_{γ}a_{γ})|_{𝔫} − Id) |

is well defined. Let

u(γ) = vol(G_{m_{γ}a_{γ}}/A).

Let h ∈ C_{c}^{∞}(G) be K-finite. Then by [Wa, pp. 177-178] (correcting a misprint) we have

(6.3)  ∫_{G_{γ}\G} h(gγg^{−1}) dġ = \frac{1}{2π} \frac{1}{u(γ)D(γ)} ∑_{σ∈M̂} \overline{tr σ(γ)} ∫_{ℝ} Θ_{σ,λ}(h) · e^{−il(γ)λ} dλ.

Since h is K-finite, Θ_{σ,λ}(h) ≠ 0 only for finitely many σ. Thus the sum over σ ∈ M̂ is finite. The volume factors in (5.7) are computed as follows. Since G has rank one, Γ_{γ} is infinite cyclic [DKV, Proposition 5.16]. Thus there is γ_{0} ∈ Γ_{γ} such that γ_{0} generates Γ_{γ} and γ = γ_{0}^{n(γ)} for some integer n(γ) ≥ 1. Then

(6.4)  \frac{vol(Γ_{γ}\G_{γ})}{u(γ)} = l(γ_{0}).

Inserting (6.3) and (6.4) into (5.7), we get the following form of the trace formula in the rank one case.

Proposition 6.1. Let φ ∈ S(ℝ) be even with φ̂ ∈ C_{c}^{∞}(ℝ). Then

Tr φ ((Δ_{τ,χ}^{#})^{1/2}) = dim(V_{χ}) vol(Γ\S) tr h_{φ}(e)

+ ∑_{\{γ\}_{Γ}≠e} tr χ(γ)\frac{1}{2π} \frac{l(γ_{0})}{D(γ)} ∑_{σ∈M̂} \overline{tr σ(γ)} ∫_{ℝ} Θ_{σ,λ}(h_{φ}) · e^{−il(γ)λ} dλ. (6.5)

The right hand side is still not in an explicite form. First of all we can use the Plancherel formula [Kn] to express tr h_{φ}(e) in terms of characters. In this way we are reduced to the computation of the characters Θ_{π}, π ∈ Ĝ, evaluated on tr h_{φ}. This is our next goal.

For simplicity we assume that K is multiplicity free in G, i.e., for each τ ∈ K̂ and π ∈ Ĝ, we have [π|_{K} : τ] ≤ 1. By [Ko] this condition is satisfied for G = SO_{0}(n, 1) and G = SU(n, 1). Let

Ĝ(τ) = {π ∈ Ĝ: [π|_{K} : τ] = 1} .

Then for each π ∈ Ĝ(τ) we can identify the τ-isotypical subspace H_{π}(τ) of τ in H_{π} with V_{τ}. Let P_{τ} be the orthogonal projection of H_{π} onto H_{π}(τ). Define the τ-spherical function Φ_{τ}^{π} on G by

Φ_{τ}^{π}(g) := P_{τ}π(g)P_{τ}, g ∈ G.

Then Φ_{τ}^{π} is a C^{∞}-map

Φ_{τ}^{π}: G → End(V_{τ})

which satisfies

(6.6)  \begin{aligned}Φ_{τ}^{π}(g)^{∗} & = Φ_{τ}^{π}(g^{−1}) \\ Φ_{τ}^{π}(k_{1}gk_{2}) & = τ(k_{1})Φ_{τ}^{π}(g)τ(k_{2}), g ∈ G, k_{1}, k_{2} ∈ K. \end{aligned}

Let v ∈ V_{τ} and set

f_{τ,v}^{π}(g) = Φ_{τ}^{π}(g^{−1})(v).

(6.7)

Then f_{τ,v}^{π} ∈ C^{∞}(G; τ) and it follows from (5.4) that

(6.8)  Δ̃_{τ}f_{τ,v}^{π} = (−π(Ω) + λ_{τ})f_{τ,v}^{π}.

Let u(t, x; f_{τ,v}^{π}) be the unique solution of

(\frac{∂^{2}}{∂t^{2}} + Δ̃_{τ}) u(t) = 0, u(0) = f_{τ,v}^{π}, u_{t}(0) = 0.

Lemma 6.2. For t ∈ ℝ, τ ∈ K̂ and π ∈ Ĝ(τ), we have −π(Ω) + λ_{τ} ≥ 0 and

u(t, x; f_{τ,v}^{π}) = cos (t\sqrt{−π(Ω) + λ_{τ}}) f_{τ,v}^{π}(x).

Proof. Let ⟨·, ·⟩ be the Killing form on 𝔤. Its restriction to 𝔭 (resp. 𝔨) is positive (resp. negative) definite. Let X_{1}, ..., X_{d} ∈ 𝔭 and Y_{1}, ..., Y_{m} ∈ 𝔨 be bases of 𝔭 and 𝔨, respectively, such that ⟨X_{i}, X_{j}⟩ = δ_{ij}, ⟨Y_{i}, Y_{j}⟩ = −δ_{ij}. Then Ω = ∑_{i} X_{i}^{2} − ∑_{j} Y_{j}^{2} and Ω_{K} = − ∑_{j} Y_{j}^{2}. Let v ∈ H_{π}(τ), ∥ v ∥= 1. Then we get

−π(Ω) + λ_{τ} = −⟨π(Ω)v, v⟩ + λ_{τ} = ∑_{i} ∥ π(X_{i})v ∥^{2}≥ 0,

which proves the first statement. For the second statement, we note that by definition, we have

(6.9)  \frac{∂^{2}}{∂t^{2}} u(t, x; f_{τ,v}^{π}) = −Δ̃_{τ}u(t, x; f_{τ,v}^{π}).

Fix x_{0} ∈ S. Let χ ∈ C_{c}^{∞}(S) be such that

χ(y) = \begin{cases}1, & y ∈ B_{2t}(x); \\ 0, & y ∈ S \ B_{3t}(x). \end{cases}

Then by finite propagation speed we have

u(t, x; f_{τ,v}^{π}) = u(t, x; χf_{τ,v}^{π}), x ∈ B_{t}(x_{0}).

Since χf_{τ,v}^{π} ∈ C_{c}^{∞}(S, Ẽ_{τ}), we have

u(t, x; f_{τ,v}^{π}) = (cos (t(Δ̃_{τ})^{1/2}) (χf_{τ,v}^{π})) (x), x ∈ B_{t}(x_{0}).

Using that Δ̃_{τ} commutes with cos (t(Δ̃_{τ})^{1/2}), and finite propagation speed, we get

Δ̃_{τ}u(t, x; f_{τ,v}^{π}) = u(t, x; Δ̃_{τ}f_{τ,v}^{π}).

By (6.8) it follows that

u(t, x; Δ̃_{τ}f_{τ,v}^{π}) = −(−π(Ω) + λ_{τ})u(t, x; f_{τ,v}^{π}).

Combined with (6.9) it follows that for every x ∈ S, u(t, x; f_{τ,v}^{π}) satisfies the following differential equation in t

(\frac{d^{2}}{dt^{2}} − π(Ω) + λ_{τ}) u(t, x; f_{τ,v}^{π}) = 0, u(0, x; f_{τ,v}^{π}) = f_{τ,v}^{π}(x), u_{t}(0, x; ϕ_{λ}) = 0.

This implies the claimed equality. □

Let φ ∈ S(ℝ) be even with φ̂ ∈ C_{c}^{∞}(ℝ). Since the kernel of the integral operator φ((Δ̃_{τ})^{1/2}) is given by h_{φ} ∈ C_{c}^{∞}(G), φ((Δ̃_{τ})^{1/2})(f_{τ,v}^{π}) is well defined and it follows from Lemma 6.2 that

\begin{aligned}φ((Δ̃_{τ})^{1/2})(f_{τ,v}^{π}) = \frac{1}{\sqrt{2π}} ∫_{ℝ} φ̂(t)u(t; f_{τ,v}^{π}) dt & = \frac{1}{\sqrt{2π}} ∫_{ℝ} φ̂(t) cos(t\sqrt{−π(Ω) + λ_{τ}})f_{τ,v}^{π} dt \\ & = φ (\sqrt{−π(Ω) + λ_{τ}}) f_{τ,v}^{π}. \end{aligned}

If we rewrite this equality in terms of the kernel h_{φ} and use the definition of f_{τ,v}^{π}, we get

(6.10)  ∫_{G} h_{φ}(g^{−1}g_{1})Φ_{τ}^{π}(g_{1}^{−1})v dg_{1} = φ (\sqrt{−π(Ω) + λ_{τ}}) Φ_{τ}^{π}(g^{−1})v.

Let d_{τ} := dim V_{τ}. Putting g = 1 and taking the trace of both sides, we get

∫_{G} Tr[h_{φ}(g)Φ_{τ}^{π}(g^{−1})] dg = d_{τ}φ (\sqrt{−π(Ω) + λ_{τ}}) .

We continue by rewriting the left hand side. To this end put

ϕ_{τ}^{π}(g) := tr Φ_{τ}^{π}(g), g ∈ G.

Note that ϕ_{τ}^{π} satisfies ϕ_{τ}^{π}(g) = ϕ_{τ}^{π}(g^{−1}). Using the Schur orthogonality relations (see [Kn, Chapt. I, §5]), we get

\begin{aligned}& Φ_{τ}^{π}(g) = d_{τ} ∫_{K} Tr[τ(k^{−1})Φ_{τ}^{π}(g)]τ(k) dk = d_{τ} ∫_{K} ϕ_{τ}^{π}(k^{−1}g)τ(k) dk. \\ (6.11) \end{aligned}

Using (6.11), we get

∫_{G} Tr[h_{φ}(g)Φ_{τ}^{π}(g^{−1})] dg = d_{τ} ∫_{G} ∫_{K} ϕ_{τ}^{π}(k^{−1}g^{−1}) Tr[h_{φ}(g)τ(k)] dkdg
= d_{τ} ∫_{K} ∫_{G} ϕ_{τ}^{π}(gk) tr h_{φ}(gk) dgdk
= d_{τ} ∫_{G} tr h_{φ}(g)ϕ_{τ}^{π}(g) dg.

(6.12)

Together with (6.10) we obtain

(6.13)  ∫_{G} tr h_{φ}(g)ϕ_{τ}^{π}(g) dg = φ (\sqrt{−π(Ω) + λ_{τ}}) .

Now let τ′ ∈ K̂ be any other representation which occurs in π|_{K}. Repeating the argument used in (6.12), we get

∫_{G} tr h_{φ}(g)ϕ_{τ′}^{π}(g) dg = ∫_{G} Tr [(∫_{K} ϕ_{τ′}^{π}(k^{−1}g^{−1})τ(k) dk) h_{φ}(g)] dg.

Again by the Schur orthogonality relations, we have

∫_{K} ϕ_{τ′}^{π}(k^{−1}g^{−1})τ(k) dk = 0,

if τ′ ≅ τ̸. Hence we get

(6.14)  ∫_{G} tr h_{φ}(g)ϕ_{τ′}^{π}(g) dg = 0, τ′ ∈ K̂, τ′ ≅ τ̸.

Choose an orthonormal basis of H_{π} which is adapted to the decomposition of π|_{K} into irreducible representations of K. Then it follows from (6.14) that

Θ_{π}(tr h_{φ}) = Tr [∫_{G} tr h_{φ}(g)π(g) dg] = ∑_{τ′} ∫_{G} tr h_{φ}(g)ϕ_{τ′}^{π}(g) dg

= ∫_{G} tr h_{φ}(g)ϕ_{τ}^{π}(g) dg.

(6.15)

Combined with (6.13) we obtain the following lemma.

Proposition 6.3. Let φ ∈ S(ℝ) be even with φ̂ ∈ C_{c}^{∞}(ℝ). Let h_{φ} be the kernel of φ((Δ̃_{τ})^{1/2}). Then for all π ∈ Ĝ(τ) we have

Θ_{π}(tr h_{φ}) = φ (\sqrt{−π(Ω) + λ_{τ}}) .

Since G has split rank one, the tempered dual of G (which is the support of the Plancherel measure) is the union of the unitarily induced representations π_{σ,λ}, σ ∈ M̂, λ ∈ ℝ, and the discrete series, where the latter exists only if rank G = rank K. First consider the induced representation π_{σ,λ}. Let T ⊂ M be a maximal torus and 𝔱 the Lie algebra of T. Let Λ_{σ} ∈ i𝔱 be the infinitesimal character of σ ∈ M̂ and ρ_{M} the half-sum of positive roots of (M, T). Then by [Kn, Proposition 8.22]

(6.16)  π_{σ,λ}(Ω) = −λ^{2} − |ρ|^{2} + |Λ_{σ} + ρ_{M}|^{2} − |ρ_{M}|^{2},

where |ρ| is given by (6.1). Let τ ∈ K̂. By Frobenius reciprocity [Kn, p.208] we have

(6.17)  [π_{σ,λ}|_{K} : τ] = [τ|_{M} : σ], σ ∈ M̂.

Since we are assuming that K is multiplicity free in G, it follows that [τ|_{K} : σ] ≤ 1. Let

M̂(τ) = {σ ∈ M̂: [τ|_{M} : σ] = 1}.

Then by (6.17) it follows that π_{σ,λ} ∈ Ĝ(τ) if and only if σ ∈ M̂(τ), and by Proposition 6.3 we get

(6.18) Θ_{σ,λ}(tr h_{φ}) = φ (\sqrt{λ^{2} + |ρ|^{2} + |ρ_{M}|^{2} − |Λ_{σ} + ρ_{M}|^{2} + λ_{τ}}) , σ ∈ M̂(τ), λ ∈ ℝ.

Now suppose that rank G = rank K. Then G has a non-empty discrete series. Let H ⊂ G be a compact Cartan subgroup with Lie algebra 𝔥. Let L ⊂ i𝔥 be the lattice of all μ ∈ i𝔥 such that ξ_{μ}(exp Y) = e^{μ(Y)}, Y ∈ 𝔥_{ℂ} exists. Let L′ ⊂ L be the subset of regular elements. According to Harish-Chandra the discrete series of G is parametrized by L′, i.e., for each μ ∈ L′ there is a discrete series representation π_{μ}. Moreover π_{μ} ≅ π_{μ′} iff there exists w ∈ W such that μ = wμ′, and each discrete series representation is of the form π_{μ} for some μ ∈ L′. Then by [Ar, (6.8)] we have

(6.19)  π_{μ}(Ω) = |μ + ρ|^{2} − |ρ|^{2}, μ ∈ L′.

So Proposition 6.3 gives in this case

(6.20)  Θ_{π_{μ}}(tr h_{φ}) = φ (\sqrt{|μ + ρ|^{2} − |ρ|^{2} + λ_{τ}}) , μ ∈ L′, π_{μ} ∈ Ĝ(τ).

Using the Plancherel formula, (6.18) and (6.20), we get an explicit form of the trace formula (6.5).

Now we consider the case τ = 1, where 1 denotes the trivial representation. Then the h_{φ} belongs to the space C_{c}^{∞}(G//K) of K-bi-invariant, smooth, compactly supported functions on G. Let c(λ) be Harish-Cahndra’s c-function. Then the Plancherel measure for the spherical Fourier transform is given by |c(λ)|^{−2}dλ, and the Plancherel formula for spherical functions (see [He]) gives

(6.21)  h_{φ}(e) = \frac{1}{2} ∫_{ℝ} φ (\sqrt{λ^{2} + |ρ|^{2}}) |c(λ)|^{−2} dλ.

Furthermore, note that by Frobenius reciprocity M̂(1) consists only of the trivial representation 1 of M, and by (6.18) we have

(6.22)  Θ_{1,λ}(tr h_{φ}) = φ (\sqrt{λ^{2} + |ρ|^{2}}) , λ ∈ ℝ.

Inserting (6.10) and (6.22) into (6.5) we get the final trace formula for the Laplacian on functions. If we replace Δ_{χ}^{#} by Δ_{χ}^{#} − |ρ|^{2}, then Theorem 1.1 follows.

7. Restrictions of representations of G

In this section we consider representations of Γ which are the restriction of a finitedimensional complex representation η: G → GL(E) of G. For such representations there exists another approach to the Selberg trace formula.

Denote the flat bundle associated to η|_{Γ} by E_{η}. There is a different description of E_{η} as follows. Let E_{τ} = Γ\Ẽ_{τ} be the locally homogeneous vector bundle associated to the restriction τ of η to K. Then there is a canonical isomorphism

(7.1)  E_{η} ≅ E_{τ}

[MM, Proposition 3.1]. Note that the space of C^{∞}-sections of E_{τ} can be identified with the space (C^{∞}(Γ\G) ⊗ E)^{K} of K-invariant vectors in (C^{∞}(Γ\G) ⊗ E), where K acts by k ↦ R(k) ⊗ η(k), k ∈ K. Thus there is a canonical isomorphism

(7.2)  ϕ: C^{∞}(X, E_{η}) ≅ (C^{∞}(Γ\G) ⊗ E)^{K} .

Let 𝔤 = 𝔨⊕𝔭 be the Cartan decomposition. By [MM, Lemma 3.1] there exists a Hermitian inner product ⟨·, ·⟩_{E} in E which satisfies the following properties.

\begin{matrix}⟨η(Y)u, v⟩_{E} = −⟨u, η(Y)v⟩_{E}, for Y ∈ 𝔨, u, v ∈ E; \\ ⟨η(Y)u, v⟩_{E} = ⟨u, η(Y)v⟩_{E}, for Y ∈ 𝔭, u, v ∈ E. \end{matrix}

In particular, ⟨·, ·⟩_{E} is K-invariant. Therefore, it defines a G-invariant Hermitian fiber metric in Ẽ_{τ} which descends to a fiber metric in E_{τ}. By (7.1) it corresponds to a fiber metric in E_{η}. Let Δ_{η} = (∇^{η})^{∗}∇^{η} be the associated Laplacian in C^{∞}(X, E_{η}). It is a formally self-adjoint operator. Its spectral decomposition can be determined as follows. By Kuga’s lemma [MM, (6.9)] we have

(7.3)  Δ_{η} = −R(Ω) + η(Ω) Id .

Assume that η is absolutely irreducible. Then there is a scalar λ_{η} ≥ 0 such that

η(Ω) = λ_{η} Id .

Let R_{Γ} be the right regular representation of G in L^{2}(Γ\G). Let

(7.4)  L^{2}(Γ\G) = ˆ⨁_{π,∈Ĝ}m_{Γ}(π)H_{π}

be the decomposition of R_{Γ} into irreducible subrepresentations, where H_{π} denotes the Hilbert space of the representation π. Denote by (H_{π} ⊗ E)^{K} the space of K invariant vectors of H_{π} ⊗ E, where the action of K is given by k ↦ π(k) ⊗ η(k). By (7.1) and (7.4) we get

(7.5)  L^{2}(X, E_{η}) ≅ (L^{2}(Γ\G) ⊗ E)^{K} ≅ ˆ⨁_{π∈Ĝ}m_{Γ}(π)(H_{π} ⊗ E)^{K}.

For π ∈ Ĝ let

λ_{π} = π(Ω)

be the Casimir eigenvalue of π. Then R(Ω) acts in (H_{π} ⊗ E)^{K} by λ_{π}. By (7.3) it follows that w.r.t. the isomorphism (7.5), Δ_{η} acts in (H_{π} ⊗ E)^{K} as (−λ_{π} + λ_{η}) Id. Thus (7.5) is the eigenspace decomposition of Δ_{η}.

Let φ ∈ S(ℝ) be even. By Lemma 2.4 φ ((Δ_{η})^{1/2}) is a smoothing operator. So it is a trace class operator. It acts in (H_{π} ⊗E)^{K} by φ ((−λ_{π} + λ_{η})^{1/2}). Then it follows from (7.5) that

(7.6)  Tr φ ((Δ_{η})^{1/2}) = ∑_{π∈Ĝ} m_{Γ}(π) dim(H_{π} ⊗ E)^{K}φ ((−λ_{π} + λ_{η})^{1/2}) .

To derive the trace formula, we can proceed as in section 5. The lift Δ̃_{η} of Δ_{η} to S is a G-invariant elliptic differential operator which is symmetric and non-negative. Let h_{η,φ}: Γ\G → End(E) be the kernel of φ((Δ̃_{η})^{1/2}). Applying Proposition 5.1 with χ = 1 and (7.6), we get

∑_{π∈Ĝ} m_{Γ}(π) dim(H_{π} ⊗ E)^{K}φ ((−λ_{π} + λ_{η})^{1/2}) = vol(Γ\S) tr h_{η,φ}(e)

+ ∑_{{γ}_{Γ}≠e} vol(Γ_{γ}\G_{γ}) ∫_{G_{γ}\G} tr h_{η,φ}(g^{−1}γg) dġ.

(7.7)

Remark. Let χ = η|_{Γ}. Then we also have the trace formula of Proposition 5.1 with τ = 1. The two formulas are, of course, different, since the operators are different. In the present case, the advantage is that we can work with self-adjoint operators. On the other hand, the formula (5.7) is more suitable for applications to Ruelle- and Selberg zeta functions. □

If the split rank of G is 1, we can use (6.3) to express the orbital integrals in terms of characters. This gives

Proposition 7.1. Assume that the split rank of G is 1. Let η: G → GL(E) be an absolutely irreducible finite-dimensional complex representation of G. Let φ ∈ S(ℝ) be even with φ̂ ∈ C_{c}^{∞}(ℝ). Then with the same notation as above we have

∑_{π∈Ĝ} m_{Γ}(π) dim(H_{π} ⊗ E)^{K}φ ((−λ_{π} + λ_{η})^{1/2}) = vol(Γ\S) tr h_{η,φ}(e)

+ ∑_{\{γ\}_{Γ}≠e} \frac{1}{2π} \frac{l(γ_{0})}{D(γ)} ∑_{σ∈M̂} \overline{tr σ(γ)} ∫_{ℝ} Θ_{σ,λ}(h_{η,φ}) · e^{−il(γ)λ} dλ.

(7.8)

The characters Θ_{σ,λ}(h_{η,φ}) can be computed by the method explained in section 6.

So there are two classes of finite-dimensional representations of Γ for which we can work with self-adjoint operators and apply the usual Selberg trace formula. These are unitary representations and restrictions of rational representations of G. In general, not every representation of Γ belongs to one of these classes. However, if rank(G) ≥ 2, the superrigidity theorem of Margulis [Ma, Chapt. VII, §5] implies that a general representation of Γ is not to far from a representation which is either unitary or the restriction of a rational representation. See [BW, p. 245] for more details.

References

[Ar] Arthur, J., Harmonic analysis of tempered distributions on semisimple Lie groups of real rank one,

[BW] Borel, A., Wallach, N., Continuous cohomology, discrete subgroups, and representations of reductive groups. Second edition. Amer. Math. Soc., Providence, RI, 2000.

[BO] Bunke, U., Olbrich, M. Selberg zeta and theta functions. Akademie Verlag, Berlin, 1995.

[DKV] Duistermaat, J. J., Kolk, J. A. C.; Varadarajan, V. S. Spectra of compact locally symmetric manifolds of negative curvature. Invent. Math. 52 (1979), no. 1, 27–93.

[Fa] Fay, J., Analytic torsion and Prym differentials. In: Riemann surfaces and related topics, pp. 107–122, Ann. of Math. Stud., 97, Princeton Univ. Press, Princeton, N.J., 1981.

[Fr] Fried, D., Analytic torsion and closed geodesics on hyperbolic manifolds. Invent. Math. 84 (1986), 523–540.

[Gi] Gilkey, P.B., Invariance theory, the heat equation, and the Atiyah-Singer index theorem. Second edition. Studies in Advanced Mathematics. CRC Press, Boca Raton, FL, 1995.

[GK] Gohberg, Krein, M., Introduction to spectral theory

[He] Helgason,S., Groups and geometric analysis, Mathematical Surveys and Monographs, vol. 83, American Mathematical Society, Providence, RI, 2000.

[Ho] Hörmander, L., The spectral function of an elliptic operator. Acta Math. 121 (1968), 193–218.

[Ka] Kato, T., Perturbation theory for linear operators, Springer-Verlag, Berlin, 1966.

[Kn] Knapp, A.W., Representation theory of semisimple groups, Princeton University Press, Princeton, New Jersey 1986.

[Ko] Koornwinder, Tom H.,A note on the multiplicity free reduction of certain orthogonal and unitary groups. Nederl. Akad. Wetensch. Indag. Math. 44 (1982).

[La] Lang, S., SL_{2}(ℝ), Addison-Wesley Publishing Co., Reading, Mass.-London-Amsterdam, 1975.

[LM] Lawson, Michelson, Spin geometry,

[Ma] Margulis, G.A., Discrete subgroups of semisimple Lie groups, Springer-Verlag, Berlin, 1991.

[Mk] Markus, A.S., Introduction to the spectral theory of polynomial operator pencils, Translation of Math. Monogr. Vol. 71, Providence, RI, 1988.

[MM] Matsushima, Y., Murakami, S., On vector bundle valued harmonic forms and automorphic forms on symmetric riemannian manifolds. Ann. of Math. (2) 78 1963 365–416.

[Mi] Miatello, R., The Minakshisundaram-Pleijel coefficients for the vector-valued heat kernel on compact locally symmetric spaces of negative curvature. Trans. Amer. Math. Soc. 260 (1980), no. 1, 1–33.

[Mil] Millson, J., Closed geodesics and the η-invariant. Ann. of Math. (2) 108 (1978), 1–39.

[MS1] Moscovici, H., Stanton, R., Eta invariants of Dirac operators on locally symmetric manifolds, Invent. Math. 95 (1989), 629–666.

[MS2] Moscovici, H., Stanton, R., R-torsion and zeta functions for locally symmetric manifolds. Invent. Math. 105 1991), 185–216.

[See] Seeley, R. T., Complex powers of an elliptic operator. In: Proc. Sympos. Pure Math., Vol. 10, pp. 288–307, Amer. Math. Soc., Providence, R.I., 1967.

[Se1] Selberg, A., Harmonic analysis and discontinuous groups in weakly symmetric Riemannian spaces with applications to Dirichlet series. J. Indian Math. Soc. (N.S.) 20 (1956), 47–87.

[Se2] Selberg, A., Harmonic analysis, in ”Collected Papers”, Vol. I, Springer-Verlag, Berlin-Heidelberg-New York (1989), 626–674.

[Sh] Shubin, M. A., Pseudodifferential operators and spectral theory. Springer-Verlag, Berlin, 1987.

[Ta1] Taylor, M., Pseudo-differential operators, Princeton University Press, Princeton, NJ, 1981.

[Ta2] Taylor, M., Partial differential equations. I. Basic theory. Applied Mathematical Sciences, 115. Springer-Verlag, New York, 1996.

[Wa] Wallach, N., On the Selberg trace formula in the case of compact quotient, Bull. Amer. Math. Soc. 82 (1976), 171 –195.

Universität Bonn, Mathematisches Institut, Beringstrasse 1, D – 53115 Bonn, Germany

  E-mail address: mueller@math.uni-bonn.de