Administrative record-keeping operations can and do constitute rich sources of statistical-reporting and research data useful for many purposes. For example, the Federal government uses Internal Revenue Service records as a source of data for the quinquennial Census of Business and Manufacturers; hospital records are used to develop research data banks on particular diseases or disabilities; school and college records are used to study the relationship between academic performance and subsequent career achievement. Unfortunately, however, the mere existence of an administrative data base can create a strong temptation to use it for statistical reporting and research without sufficient attention to the appropriateness of doing so.
Three conditions that encourage sound use of data systems for statistical reporting and research are often absent from the environment in which administrative systems are designed and operated They are:
- knowledge of the social processes by which data come to be collected;
- management of data collection and analysis by individuals with strong statistical and research competence; and
- independent expert scrutiny of analytic methods and results.
Knowledge of Data Collection Processes. Detailed understanding of how and why data come to be collected is often difficult, if not impossible, to achieve. For example, not everyone who is eligible for public assistance applies for it, and the amount and kind of information collected from each applicant may vary in subtle ways.5 Hence, if data from administrative systems are used for statistical reporting and research, the results must take account of systematic bias resulting from incompleteness in the data base. Measuring such bias can be expensive and time-consuming, and corrections for it can be even harder to make. Highly trained people are needed to conduct careful studies of the processes by which data in a system are being generated. Because of their expense and difficulty, however, and also because they can bring to light inadequacies in the overall performance of an organization, such studies tend not to be done.
Statistical and Research Competence. Because most administrative systems are committed to day-to-day record-keeping operations, they are seldom managed or staffed by persons with strong statistical and research competence. It is true that the statistical offices of a few large government agencies-notably the Social Security Administration and the Internal Revenue Service-have substantially influenced the statistical uses made of their principal data sources, which are mainly administrative records. Similar examples can be found at other levels of government and among private organizations, but there are also numerous instances in which such statistical and research competence is brought to bear only through informal or sporadic consulting arrangements, if at all.
Independent Scrutiny. Because administrative data systems are not created expressly for statistical reporting and research, they also tend to lack the strong ties to external groups of data users, and to the formal systems of professional peer review that characterize general purpose statistical-reporting and research operations. This isolation from independent expert scrutiny, coupled with the management orientation of administrative data systems, weakens the incentive to maintain high standards in the secondary statisticalreporting and research uses that are made of them.
Neglect of these three conditions is particularly dangerous in a governmental setting. In business, the quality of statistical reporting and research may be measured by the usefulness of such work to the planning and marketing functions that maintain a firm's competitive position. In government, however, feedback from the marketplace is attenuated. Save for the occasional newsworthy statistical report, the ancillary uses of administrative data systems may be ignored by outside professionals and invisible to the general public and its elected representatives.
In the Federal Government, formal arrangements for implementing the Federal Reports Act are supposed to serve as a check on the uses made of administrative record-keeping systems for statistical reporting and research. However, at other levels of government, the low visibility of such uses, coupled with the uneven impact of public information laws, can create an open invitation to misguided use of statistical reports and research findings based on administrative data.
We learned, for example, that one agency of a State government recently attempted to compare earnings declarations made by some public assistance beneficiaries to county welfare offices, with earnings of those same beneficiaries reported by their employers to a second State agency. This complex comparison of data derived from two quite different administrative record-keeping systems was undertaken mainly to verify the beneficiaries' eligibility for public assistance payments on a case-by-case basis, but it also resulted in a statistical report "showing" that a substantial percentage of the State's public assistance beneficiaries were engaged in "apparent fraud." The design of the comparison, and thus the resulting data, supported no such conclusion. Few people are aware of its technical failings, however, and it seems unlikely that many more will discover them, since appropriately documented data from the study have not been made available outside the sponsoring State agencies.