FSP Index Methodology

Data collected to create the 2006-07 FSP Index database include book publications, journal publications, citations of journal articles, awards and honors, and federal grant dollars awarded. Faculty names are the unit of record in the database and the productivity measures are tied to specific individuals prior to the aggregation of data into disciplinary groups. This provides the ability to monitor the productivity of individual programs or departments while taking into account the fact that faculty members may move to other institutions, thus offering an advantage over ranking schemes that are based on data at the level of the academic program as a whole (e.g., “Department of Physics”). The results for FSP Index 2006-07 are the aggregation of unit record data for individuals currently listed as members of the Ph.D. program faculty in a particular discipline at a particular university. Interdisciplinary programs are properly credited with the work done by their entire faculty. Similarly, graduate programs that draw on the intellectual resources of faculty who are not employed by the university (e.g., by national laboratories, museums etc.) are also accorded the appropriate publications and grants so that inter-institutional programs are not disadvantaged. Faculty may be members of multiple programs. FSP Index 2006-07 counts faculty fully in each program in which they are listed, on the basis that the faculty member will be fully available intellectually to any student who works with him/her. To avoid the distorting effects that could occur by counting one person’s accomplishments more than once, our faculty lists are de-duplicated so that a person and their accomplishments are counted only once at each level of analysis. For example, a person who is a member of the graduate faculties in Biochemistry, Pharmacology, and Chemistry is counted as a full FTE in each of these three programs, and their citations, grants, honorific awards, and publications are fully attributed to all three programs. At the next level of aggregation, the person would be counted as one FTE in Physical Sciences and one FTE in Biomedical Sciences, again with their full productivity in each of the two areas.

The FSP database is the only one of its kind that provides data on faculty scholarly work at the discipline level, on an annual basis, and in a comparative and objective context.

Taxonomy

The taxonomy for the FSP index was developed to cover as many fields of study as possible while still providing meaningful results: the taxonomy has four levels. The names of all Ph.D. programs in the United States were collected using available databases and informed internet-based searches. Programs that exist at ten or more US universities were considered appropriate groups for study and this resulted in the recognition of 172 categories under which a program can be reported. From level one, disciplines were aggregated to a level two category, e.g., individual biomedical science fields, such as Biochemistry and Developmental Biology, are aggregated as Biomedical Sciences. This approach allows us to match the structure of the analysis to the university being analyzed. Thus, a university with a combined Biomedical Sciences Ph.D. program (first reported at level two in AA’s taxonomy) can be compared with universities that have individual programs in the various fields of biomedical sciences (reported at level one), with these being aggregated up to create a “virtual” Biomedical Sciences program for those schools with level one disciplines. Similarly, programs in disciplines that exist at less than ten universities are aggregated to the second level, such as Area, Ethnic, Cultural and Gender Studies, to allow inclusion in the study and comparison with a meaningful group. The third level of the taxonomy is a group of 11 broad fields (Physical Sciences and Mathematics, Humanities and Fine Arts, Education etc.). The fourth level at which comparisons are made is the whole institution.

For institutional level comparisons it does not make sense to compare a university with one Ph.D. discipline to very large, comprehensive universities, so we developed the following classification system for FSP 2006-07. We followed the approach established by Webster and Skinner (May/June 1996, Change, Rating PhD Programs: What the NRC Report Says…and Doesn’t Say, pp. 22-44, David S. Webster and Tad Skinner), which defined comprehensive institutions as those offering a substantial number of Ph.D. programs (included in the 1995 NRC Assessment). Webster and Skinner set this number as 15 or more Ph.D. programs and we have used that threshold also for Large Research Universities. The remaining universities in our study fall into two major groups: those that specialize by discipline (here defined as activity falling into a limited number of related fields among the eleven discipline areas), and those that cover a broad range of fields but with a limited (<15) number of programs. The first of these groups is termed Specialized Research Universities and the fields of specialization are defined to include a minimum of 20 universities in each group. This leads to the recognition of Specialized Research Universities in: STEM Fields, Biomedical & Health Sciences, Business, Education, & Social Sciences, and Theology. The remaining group of universities offers Ph.D. programs in a broad range of disciplines and have one to 14 programs in total. This group is classified as Smaller, Research Universities.

Ph.D. Programs and faculty names collection

Faculty names and their program affiliations were collected from the academic bulletins and websites of every university included in the 2006-07 Index. The research staff of Academic Analytics manually collected these data to ensure the highest level of accuracy. In addition to manual collection, lists of faculty were sent to the graduate deans and directors of IR at each university included in FSP Index 2006-07 with a request to review and update the lists. One hundred ninety two schools responded to this request either by updating their list or verifying that it was correct as sent for FSP Index 2006-07. A total of 179,250 unique faculty members in 8,182 programs is included in the FSP Index 2006-07 study.

Publication data

Journal publication and citation data were provided by Scopus™ for the years 2004, 2005 and 2006 for publications, and 2003, 2004, 2005, and 2006 for citations, and cover over 15,000 journals (list available at: info.scopus.com/detail/what). Every author of a paper is credited for having authored the paper in our authorship count. Papers with multiple authors were credited to every author at each relevant level of comparison. For example, if a paper has three authors, two in one program and one in a second at the same University then the publication would be treated as a single paper for the purpose of counting the average number of (unique) publications per faculty member in both programs. All of the authors would contribute to both numerator and denominator in calculating the percentage of faculty who have authored a paper. At the second and third levels of aggregation, the journal publications and citations are also de-duplicated so as not to distort either the measure of the percentage of faculty who have published or the average number of papers published at any level in the comparisons. Double counting will occur across institutions in which faculty members at each institution who have co-authored a publication will be credited for that publication (and the resultant citations). A similar process is used to apportion grants.

Book Publications

Book publication data were supplied by Baker & Taylor, who maintain a comprehensive catalogue of millions book publications. Books written in the five year period between 2002 and 2006 were matched to faculty members in our database. Criteria used to match faculty members were limited to only parsed name elements (first name, middle initial, last name) with varying degrees of match sensitivity (e.g., omitted middle names and initials do not result in missing books matches; short-forms of first names also do not prohibit a positive match, etc.); this liberal matching criteria is necessary because book databases often do not have geographic or affiliation information for authors. Our matching approach was supplemented by examining each matched book’s fields of classification (BISAC and LOC), title, and online resources such as faculty CV’s, determining the likelihood that this book is correctly matched to the faculty member of the same name. This labor intensive process results in what is likely the first and only comprehensive and accurate database of books attributable to American scholars in Ph.D. Programs - in total, nearly 60,000 books were positively matched to scholars.

Grant data collection

Research grants awarded between 2004 and 2006 were collected from publicly available databases of the federal agencies listed below. Only new and competitive grants are counted (e.g., non-competitive renewals of grants are not counted). Individual grants are awarded to researchers only once - in the year of the original award. The “dollars per year” associated with each grant is calculated based on the total value of the grant divided by the duration of the grant. For example, a $300,000 grant over three years awarded in 2004 is counted as one grant in 2004 worth $100,000 per year. Research agencies included in the FSP study are: National Institutes of Health (NIH), National Science Foundation (NSF), National Oceanic and Atmospheric Administration (NOAA), U.S. Department of Agriculture (USDA), U.S. Department of Education (DoEd), U.S. Department of Energy (DOE; three divisions of DOE are included - Division of Materials Science and Engineering, Office of Biological and Environmental Research, Office of Nuclear Physics).

Awards and honors data collection

Data were collected from the websites of award and honor granting organizations including the name of the recipient, their university affiliation at the time of the award, and their field of academic work. Recipients were then located to record their current institutional affiliation, and the award was then credited to programs in which the person is a current member. It is generally agreed that awards and honors are not all of equal stature, so we have accorded these with different weights by recording them for different periods of time. For an award that is given annually to large numbers of people data are included for awards given in the last five years. For more competitive and prestigious awards the award may be carried for ten years or twenty years. A Nobel Prize is carried for fifty years. The exact weight given to each award will be a matter for fruitful discussion within the academy but this approach offers a simple way to accord different weights to a variety of important and prestigious honors and awards.

Threshold for Inclusion

For a program to be included in FSP Index 2006-07, it must have either ten or more faculty members, or, if it has fewer than ten faculty members, it must have one-half the median number of faculty members for a program in that discipline. If one-half the median is not an integer, the required number is rounded up. As such, given a median number of faculty in a discipline of 17, a program must have nine (rounded up from 8.5) faculty members to be included in FSP Index 2006-07.