*Published with minor revisions as "Refining Ratings" in the August 1998 issue of
*Risk. * All figures appear at the end of the article.*

A Nonparametric Test for Credit Rating
Refinements

by

Ross M. Miller

Miller Risk Advisors

2255 Algonquin Road

Niskayuna, NY 12309 USA

millerrm@alumni.caltech.edu

May 1998

While the intensive quantification of market risk has led to measurements accurate to
the basis point (and beyond), difficulties in quantifying credit risk have resulted in the
practice of measuring this risk with far less precision. Indeed, financial institutions
that develop their own internal measures of credit risk usually employ a "1" to
"9" scale of creditworthiness for their exposures, taking their lead from the
rating agencies and their AAA to D (and similar) systems. Even with the further refinement
of "notches," designated with a "+" or "-" the vast universe
of credit risk is reduced to at most thirty buckets. In reality, the broad range of
pricing for corporate debt obligations in the marketplace indicates that there are far
more than thirty categories of credit risk. In the high-yield debt market the illusion of
"stability" provided by a broad categorization scheme can be easily outweighed
by its imprecision.

The difference in the precision with which market risk and credit risk have been
measured is due in large part to the rich body of quantitative theory that has been
developed for analyzing market risk. In particular, there are many cases where the
arbitrage-based techniques of option valuation theory can be applied to market risk.
Despite the numerous shortcomings of this theory (elaborated upon in this and other
journals), its practical application leads to a fairly precise quantification of risk in a
wide variety of settings.

Fortunately, those same option-theoretic roots that have been so useful in quantifying
market risk can be applied to the quantification of credit risk in corporate settings.
Indeed, the modern theory’s developers—Fischer Black, Myron Scholes, and Robert
Merton—noted in their earliest work on options that the equity in any firm could
itself be valued as an option. The value of the equityholder’s implicit option is
directly related to the likelihood of financial distress, at which point they will choose
to limit any further liability from their stake in the firm by "putting" it at a
price of zero to the debtholders. Generally, such financial distress will also result in a
shortfall to the debtholders and the firm will default on its debt. Hence, the credit risk
of the firm, i.e., its propensity to default, is directly and quantitatively linked to the
value of its equity. In those cases where the value of the equity is directly observable,
such as when it takes the form of an actively-traded security, one should be able, in
principle, to "reverse engineer" the probability distribution of default from
the market value of the equity given the volatility and financial structure of the
firm’s assets. Without delving into the messy details here, the option-valuation
approach to credit risk is far more difficult to make practical than its market risk
analogue.

This paper develops and applies the statistical machinery that is necessary to
determine whether a particular quantitative method of measuring credit risk, using option
valuation or other methods, truly represents a refinement over broader categorization
methods. Here the term "refinement" is used in a technical sense of partitioning
the creditworthiness of firms in a way that yields more information that the original
partition, in this case the letter-based ratings. This is a particularly challenging
statistical problem because defaults themselves are infrequent occurrences and the error
structure of their estimates cannot be assumed to be normal or any other well-behaved
distribution. The solution to this problem is to get the most out of the data that exists
and to apply nonparametric tests, which do not rely on distributional assumptions, to
these data. Although this paper applies the methodology to a specific refinement of
letter-based ratings, it can be applied quite broadly.

**Alternative Credit Ratings**

The methodology for testing whether a quantitative credit rating system is a
statistically significant refinement of a broader rating system that is described here was
developed as part of an independent study commissioned by the Capital Markets Assurance
Corporation (CapMAC), a financial guarantee company based in New York that subsequently
merged with MBIA Inc., another financial guarantee company. Specifically, CapMAC was
interested in the degree to which the quantitative credit ratings provided by KMV
Corporation’s Credit Monitor system, the first option-based credit rating system,
were a refinement of Standard & Poor’s (S&P) notch-level ratings for U.S.
companies.

For this study S&P’s senior unsecured debt ratings as they appear in Compustat
were used. The S&P company ratings that were used were (in descending order of credit
quality): AAA, AA+, AA, AA-, A+, A, A-, BBB+, BBB, BBB-, BB+, BB, BB-, B+, B, B-. Ratings
below B- (CCC+, CCC, etc.) were excluded from the study because of the sparseness of these
data. S&P ratings are designed to provide intermediate to long-run indications of
creditworthiness.

In contrast, KMV’s ratings take the form of expected default frequencies (EDFs),
which range from 0.02% (2 basis points) to 20% and are reported with basis point
precision, making for 1,999 different possible ratings. Because the likelihood of default
varies over the business cycle, there is no fixed mapping between KMV and S&P ratings.
KMV provides EDFs for horizons ranging from one to five years (see Kealhofer, Kwok, and
Weng, 1998 for more information about Credit Monitor and the relationship between EDFs and
agency ratings). For this study, the one-year EDF, for which KMV provides the most
information, is used. Although the EDF provides a cardinal measure of credit risk, the
nonparametric methods used in this study will only examine its ordinal properties relative
to S&P ratings.

The test of KMV’s predictive power relative to an S&P rating can be expressed
as the following null hypothesis: given an S&P rating for a company, the KMV rating
(EDF) provides no additional information as to the likelihood the firm will default over a
specified period of time. Figure 1 gives a grid of the EDFs for all companies rated by KMV
Credit Monitor at the end of 1989 that had an S&P company rating of B- in ascending
order of EDF (descending creditworthiness). The null hypothesis states that the pattern of
defaults should be random within that cohort; however, it is clear from Figure 1 that over
the next three years (1990-1992) defaults came disproportionately from the companies with
the highest EDFs. Of course, this is a single cohort at a single point in time. In the
following section we will demonstrate how to aggregate all cohorts at all points in time
into a single nonparametric statistical test of the null hypothesis.

**Aggregating Defaults**

The study covered U.S. companies on an end-of-month basis from June 1986 to November
1996. Only U.S. companies with both S&P ratings and KMV Credit Monitor EDFs were
used—a total of over 1,000 companies at some point during the 10-year sample period.
Defaults were taken from KMV’s default database, includes both the defaults reported
by the rating agencies for rated issues as well as defaults for companies having no rated
debt. Although KMV performed a major revision of its model in 1994, only the EDFs
contemporaneously reported by KMV were used in order to minimize the extent to which we
were testing the model on data to which it had been backtested or backfitted.

The trick to aggregating the KMV data over both time and S&P ratings categories is
to convert each EDF into an ordinal ranking within its month and (notch) rating cohort. A
natural ordinal measure is a percentile rank—with 100 being the best (lowest EDF) and
0 the worst (highest EDF). A means for computing percentile that both takes account of
ties in ranking, which are quite rare for companies with higher EDFs, and is consistent
across cohorts of different sizes is given by:

where *Total* is the size of the cohort, *Worse* is the number of the firms
in the cohort with worse credit ratings (EDFs), and *Better* is the number of firms
with better ratings.

For example, in a cohort with 20 companies (the smallest allowed in the study), the
company with the lowest EDF (assuming it is the only one) has *Better* = 0, *Worse*
= 19, so by substitution above its *Percentile* = 97.5 (the midpoint of the top 5%).
If two companies are tied for 3^{rd} and 4^{th} best EDFs, they would have
*Better* = 2, *Worse* = 16, giving *Percentile* = 85.

The reason for a lower limit on the size of a cohort is to make the distribution of
percentiles as continuous as possible, facilitating meaningful comparisons between
cohorts. Also note that by converting an EDF to a percentile we are completely eliminating
any bias that might result from EDFs as a whole being more or less optimistic about the
overall rate of defaults than S&P ratings.

The null hypothesis that EDFs provide no information beyond that contained in S&P
ratings can be made operational using EDF percentiles. Given the way that the percentiles
were constructed, the null hypothesis implies that the percentiles of the population of
defaulting firms will have a flat (or uniform) distribution. In particular, looking at
defaulting firms any fixed number of months prior to the default, the distribution of
percentiles should be uniform.

The histograms in Figure 2 show that at 6 months prior to default this distribution is
obviously far from uniform. As time to default increases, the distribution becomes more
uniform, which is what one would expect for any system with predictive power, i.e., as the
time to the event increases, the predictive power decreases. Nonetheless, just from
looking at the graph of the distribution, it is apparent that there is still some
predictive power 36 months prior to default.

**Applying Kolmogorov-Smirnov**

The Kolmogorov-Smirnov test is used to quantify the degree to which the observed
distribution differs from uniformity as a statistical significance level. This test is a
basic and well-understood tool of nonparametric statistics (see Siegel and Castellan, 1988
for details). In the one-sample version of this test we can determine the significance
with which a sample of data differs from a given baseline frequency distribution—in
this case the uniform distribution.

The Kolmogorov-Smirnov test works on the distributional characteristics of the maximum
distance between the cumulative density functions of the sample and the baseline. This
maximum distance is known as the D-statistic. By aggregating over all months and ratings
we get at least fifty data points for which both KMV and S&P ratings are known 6, 12,
18, 24, 36 and 48 months prior to default, more than enough to perform a meaningful
statistical test.

The results of this test are given in the table below. KMV Credit Monitor has very
strong predictive value for as many as 18 months prior to default. The predictive power
remains statistically significant out to 36 months. At 48 months the significance level
finally drops below 95%.

An alternative nonparametric approach to showing predictive power of EDFs relative to
S&P ratings is to apply a binomial test to a division of the sample into two parts,
with the 50^{th} percentile as a natural cutoff point. The results of this test,
which are not given here, as well as for other natural percentile cutoffs (90^{th},
75^{th}, etc.) confirm the results of the Kolmogorov-Smirnov test. The advantage
of the Kolmogorov-Smirnov test is that it does not require the arbitrary choice of a
percentile cutoff.

**Discussion**

This paper has demonstrated a statistical methodology for inferring that a quantitative
credit rating system is a refinement of a traditional rating system, i.e., it provides
additional information. This demonstration was made with the use of a single, innocuous
statistical assumption—that the number of firms in a cohort is large enough to make
the distribution of percentiles nearly continuous. Generating results beyond those given
here almost always comes at the cost of additional assumptions that could affect the
validity of the analysis. For example, it is natural to try to turn the analysis around
and ask whether for a given EDF a letter rating provides additional information. To do
this analysis the technique developed above cannot be applied without making further
assumptions. The problem here is that while each letter rating-based cohort has associated
with it a near-continuum of EDFs (cf. Figure 1), virtually all cohorts based on a single
EDF will contain few, if any, companies because the number of companies is small relative
to the number of possible EDFs. The remedy of grouping EDFs together to get around this
problem not only will require the use of arbitrary cutoffs, but also will throw away
information that may then be incorrectly attributed to the letter rating.

Additional assumptions are also required to gauge the relative contribution of both the
letter-based and quantitative rating systems simultaneously using regression analysis. In
a regression the two rating systems are used to generate two or more dependent variables.
The independent variable is then either the default rate (for grouped data) or a dummy
variable representing default (for individual data). Whichever way the analysis is set up,
explicit assumptions as to the error distribution of the independent variables must be
made in order to perform the regression. Given that neither rating is generated by a
"natural" process that can be expected to have a predictable error term, e.g.,
normally distributed, the results are likely to be quite sensitive to the distributional
assumptions that underlie the regression.

On a more positive note, the analysis performed in this paper can be readily extended
to compare two different quantitative credit rating systems to a common letter-based
baseline. Such a comparison can be directly implemented using the two-sample version of
the Kolmogorov-Smirnov test, a standard textbook extension of the single-sample version
that we used.

**References**

**Kealhofer S, S Kwok and W Weng, 1998**

*Uses and abuses of bond default rates*

CreditMetrics Monitor, First Quarter, pages 37-55

**Siegel, S and N. J. Castellan Jr., 1988**

*Nonparametric statistics for the behavioral sciences, second edition*

McGraw-Hill

Figure 1. Defaults for S&P B- Companies at the end of 1989 ordered by EDF

Figure 2. Distribution of KMV Percentiles of Defaulting Firms 6 to 48
Months Prior to Default

Copyright © 1998 by Miller Risk Advisors. All rights reserved.