The Annals of Statistics
2017, Vol. 45, No. 1, 415–460
DOI: 10.1214/16-AOS1463
Institute of Mathematical Statistics, 2017
EXTREME EIGENVALUES OF LARGE-DIMENSIONAL SPIKED
FISHER MATRICES WITH APPLICATION
BY QINWEN WANG AND JIANFENG YAO
University of Hong Kong
Consider two p-variate populations, not necessarily Gaussian, with co-
variance matrices 1 and 2, respectively. Let S1 and S2 be the correspond-
ing sample covariance matrices with degrees of freedom m and n. When the
difference between 1 and 2 is of small rank compared to p,m and n,
the Fisher matrix S := S 12 S1 is called a spiked Fisher matrix. When p,m
and n grow to infinity proportionally, we establish a phase transition for the
extreme eigenvalues of the Fisher matrix: a displacement formula showing
that when the eigenvalues of (spikes) are above (or under) a critical value,
the associated extreme eigenvalues of S will converge to some point outside
the support of the global limit (LSD) of other eigenvalues (become outliers);
otherwise, they will converge to the edge points of the LSD. Furthermore, we
derive central limit theorems for those outlier eigenvalues of S. The limiting
distributions are found to be Gaussian if and only if the corresponding pop-
ulation spike eigenvalues in are simple. Two applications are introduced.
The first application uses the largest eigenvalue of the Fisher matrix to test
the equality between two high-dimensional covariance matrices, and explicit
power function is found under the spiked alternative. The second application
is in the field of signal detection, where an estimator for the number of signals
is proposed while the covariance structure of the noise is arbitrary.
1. Introduction. Consider two p-variate populations with covariance matri-
ces 1 and 2, and let S1 and S2 be the sample covariance matrices from samples
of the two populations with degrees of freedom m and n, respectively. When the
difference between 1 and 2 is of finite rank, the Fisher matrix S := S 12 S1
is called a spiked Fisher matrix. In this paper, we derive three results related to
the extreme eigenvalues of the spiked Fisher matrix for general populations in
the large-dimensional regime, that is, the dimension (p) grows to infinity together
with the two sample sizes (m and n). Our first result is a phase transition phe-
nomenon for the extreme eigenvalues of S: a displacement formula showing that
when the eigenvalues of (spikes) are above (or under) a critical value, the asso-
ciated extreme eigenvalues of S will converge to some point outside the support
of the global limit (LSD) of other eigenvalues (become outliers), and the loca-
tion of this limit only depends on the corresponding population spike of and
Received April 2015; revised March 2016.
MSC2010 subject classifications. Primary 62H12; secondary 60F05.
Key words and phrases. Large-dimensional Fisher matrices, spiked Fisher matrix, spiked popu-
lation model, extreme eigenvalue, phase transition, central limit theorem, signal detection, high-
dimensional data analysis.
415
416 Q. WANG AND J. YAO
two dimension-to-sample-size ratios; otherwise, they will converge to the edge
points of the LSD. The second result is on the second-order behavior of those out-
lier eigenvalues of S. We show that after proper normalization, a packet of those
outlier eigenvalues (corresponding to the same spike in ) converge to the eigen-
values’ distribution of some structured Gaussian random matrix. In particular, the
limiting distribution of the outlier eigenvalue of S (after normalization) is Gaus-
sian if and only if the corresponding spike in is simple. Finally, as an extension,
we consider the joint distribution of all those outlier eigenvalues (correspond to
different spikes in ) as a whole, and it is shown that those outlier eigenvalues
(after normalization) converge to the eigenvalues’ distribution of some block ran-
dom matrix, whose structure can be fully identified. Also as a special case, if all
the spikes in are simple, then the joint distribution of the outlier eigenvalues of
S is multivariate Gaussian.
There exists a vast literate on the spectral analysis of multivariate Fisher matri-
ces under the assumption that both populations are Gaussian and share the same
covariance matrix, that is, 1 =2. The joint distribution of the eigenvalues of the
corresponding Fisher matrix S was first simultaneously and independently pub-
lished in 1939 by R. A. Fisher, S. N. Roy, P. L. Hsu and M. A. Girshick. Later in
1980, Wachter (1980) finds a deterministic limit, the celebrated Wacheter distribu-
tion, for the empirical measure of these eigenvalues when the dimension p grows
to infinity proportionally with the degrees of freedom m and n (large-dimensional
regime). Wachter’s result has been later extended to non-Gaussian populations us-
ing the tools from the random matrix theory and two early examples of such ex-
tensions are Silverstein (1985) and Bai, Yin and Krishnaiah (1987).
In this paper, we are also interested in the large-dimensional regime, while al-
lowing 1 and 2 to be separated by a (finite) rank-M matrix . Besides, the
two populations can have arbitrary distributions other than Gaussian. From the
perturbation theory, when M is a fixed integer while p, m and n grow to infinity
proportionally, the empirical measure of the p eigenvalues of S will be affected
by a difference of order M/p (→ 0), so that its limit remains the Wachter dis-
tribution. Therefore, our main concern is the local asymptotic behaviors of the
M extreme eigenvalues of S (other than the global limit). In a recent preprint
Dharmawansa, Johnstone and Onatski (2014), by assuming both population are
Gaussian and M = 1, these authors show that, when the norm of the rank-1 dif-
ference (spike) exceeds a phase transition threshold, the asymptotic behavior of
the log-ratio of the joint density of these characteristic roots under a local deviation
from the spike depends only on the largest eigenvalue lp,1 and the statistical exper-
iment of observing all the eigenvalues is locally asymptotically normal (LAN). As
a by-product of their analysis, the authors also establish joint asymptotic normal-
ity of a few of the largest eigenvalues when the corresponding spikes in (with
M > 1) exceed the phas
e transition threshold. The analysis given in this refer-
ence highly relies on the Gaussian assumption so that the joint density function of
the eigenvalues has indeed an explicit form, and the main results are obtained via
LARGE AND SPIKED FISHER MATRICES WITH APPLICATION 417
an accurate asymptotic approximation of the log-ratio of these density functions.
Therefore, one of the main objectives of our work is to develop a general theory
without such Gaussian assumption. It is thus apparent that the joint density of the
eigenvalues of the Fisher matrix S has then no more an analytic formula and new
techniques are needed to solve the questions.
Our approach relies on the tools borrowed from the theory of random matrices.
A methodology particularly successful both in theory and applications within this
approach relies on the spiked population model coined in Johnstone (2001). This
model assumes the population covariance matrix has the structure p = Ip +
where the rank of is M (M is a fixed integer). Again for small rank M , the
empirical eigenvalue distribution of the corresponding sample covariance matrix
remains the standard Marcˇenko–Pastur law. What makes a difference is the local
asymptotic behaviors of the extreme sample eigenvalues. For example, the fluctu-
ation of largest eigenvalues of a sample covariance matrix from a complex spiked
Gaussian population is studied in Baik, Ben Arous and Péché (2005), where the
authors uncover a phase transition phenomenon: the weak limit and the scaling of
these extreme eigenvalues are different depending on whether the eigenvalues of
(spikes) are above, equal or below a critical value, situations refereed as super-
critical, critical and sub-critical, respectively. In Baik and Silverstein (2006), the
authors consider the spiked population model with general populations (not nec-
essarily Gaussian). For the almost sure limits of the extreme sample eigenvalues,
they find that if a population spike (in ) is large or small enough, the correspond-
ing spiked sample eigenvalues will converge to a limit outside the support of the
limiting spectrum (become outliers). In Paul (2007), a CLT is established for these
outliers, that is, the super-critical case, under the Gaussian assumption and assum-
ing that population spikes are simple (multiplicity 1). The CLT for super-critical
outliers with general populations and arbitrary multiplicity numbers is developed
in Bai and Yao (2008). Joint distributions for the outlier sample eigenvalues and
eigenvectors can be found in Shi (2013) and Wang, Su and Yao (2014). A recent
related application to high-dimensional regression can be found in Kargin (2015).
Within the theory of random matrices, the techniques we use in this paper for
spiked models are closely connected to other random matrix ensembles through the
concept of small-rank perturbations. Theories on perturbed Wigner matrices can be
found in Péché (2006), Féral and Péché (2007), Capitaine, Donati-Martin and Féral
(2009), Pizzo, Renfrew and Soshnikov (2013) and Renfrew and Soshnikov (2013).
In a more general setting of finite-rank perturbation including both the additive and
the multiplicative one, referees include Benaych-Georges and Nadakuditi (2011),
Benaych-Georges, Guionnet and Maida (2011) and Capitaine (2013).
Apart from the theoretical results, we also propose two applications both in
high-dimensional hypothesis testing and signal detection, respectively. The first
application uses the largest eigenvalue of the Fisher matrix to test the following
hypotheses:
(1.1) H0 : 1 =2 vs. H1 : 1 =2 +,
418 Q. WANG AND J. YAO
where is a nonnegative definite matrix of rank M . Under this spiked alterna-
tive H1, explicit formula for the power function is derived. Our second application
is to propose an estimator for the number of signals based on noisy observations.
Other than the existing approaches [see, e.g., Kritchman and Nadler (2008), Nadler
(2010), Passemier and Yao (2012, 2014)], our method allows the covariance struc-
ture of the noise to be arbitrary.
The rest of the paper is organized as follows. First, in Section 2, the exact setting
of the spiked Fisher matrix S = S 12 S1 is introduced. Then in Section 3, we estab-
lish the phase transition phenomenon for the extreme eigenvalues of S: a displace-
ment formula is found as well as the transition boundary is explicitly obtained.
Next, CLTs for those outlier eigenvalues fluctuating around their limit (i.e., in the
super-critical case) are established first in Section 4 for one group of sample eigen-
values corresponding to a same population spike, and then in Section 6 for all the
groups jointly. Section 5 contains numerical illustrations that demonstrate the finite
sample performance of our results. In Section 7, we develop in detail two appli-
cations in high-dimensional statistics. Proofs of the main theorems (Theorems 3.1
and 4.1) are included in Section 8 while some technical lemmas are grouped in the
Appendix.
2. Spiked Fisher matrix and preliminary results. In what follows, we will
assume that 2 = Ip . This assumption does not lose any generality since the eigen-
values of the Fisher matrix S = S 12 S1 are invariant under the transformation
S1 → 1/22 S1 1/22 , S2 → 1/22 S2 1/22 .(2.1)
Also we will write p for 1 to signify the dependence on the dimension p. Let
Z = (z1, . . . , zn)= (zij )1≤i≤p,1≤j≤n(2.2)
and
W = (w1, . . . ,wm)= (wkl)1≤k≤p,1≤l≤m(2.3)
be two independent arrays, with respective size p × n and p ×m, of independent
real-valued random variables with mean 0 and variance 1. Now suppose we have
two samples {zi}1≤i≤n and {xi =1/2p wi}1≤i≤m, where {zi} and {wi} are given by
(2.2) and (2.3), and p is a rank M (M is a fixed integer) perturbation of Ip , that
is,
(2.4) p =
(
M 0
0 Ip M
)
.
Here, M is a M ×M covariance matrix, containing k nonzero and nonunit eigen-
values (ai), with multiplicity numbers (ni) (n1 + · · · + nk = M). That is, M
has the eigen-decomposition U diag(a1, . . . , a1︸ ︷︷ ︸
n1
, . . . , ak, . . . , ak︸ ︷︷ ︸
nk
)U , where U is a
M ×M orthogonal matrix.
LARGE AND SPIKED FISHER MATRICES WITH APPLICATION 419
The sample covariance matrices of the two observations {xi} and {zi} are
(2.5) S1 = 1
m
m∑
l=1
xlx