HHS Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated to the Public. V. Agency Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Disseminated to the Public

10/01/2002

The quality assurance process begins at the inception of the information development process. AHRQ reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including creation, collection, maintenance, and dissemination. AHRQ will demonstrate in its Paperwork Reduction Act (PRA) clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and AHRQ information quality guidelines.

Work performed by AHRQ is conducted in compliance with generally accepted professional standards appropriate to the field(s) of study being undertaken, e.g., clinical, statistics, economics, or qualitative health services research. Moreover, work produced by the Agency undergoes many levels of review before being finalized and disseminated to ensure quality, i.e., utility, objectivity (in substance and presentation), and integrity. Reviews are both hierarchical, i.e., performed by higher level supervisors and managers, and by peer experts both inside and outside of the Agency. Generally, it is the responsibility of AHRQ's Center Directors to ensure the utility, objectivity, and integrity of work produced within their research Centers.

The discussion that follows outlines the general quality standards and quality assurance procedures that AHRQ has in place. More detailed information is available on AHRQ's Web site and that information will be expanded to reflect new products (e.g. new databases) or revised over time to reflect advances in generally accepted professional standards.

Statistical information and database development

Quality standards: Statistical activities are carried out in accordance with generally accepted professional standards and modern statistical theory and practice, including scientific sampling, statistical inference and analytical techniques and practices. All statistical programs employ or have access to experts in statistics and research design.

Quality assurance procedures: Each of the Agency's data products undergoes a series of standardized quality assurance procedures to assure the validity, reliability, and consistency of the data. These procedures are posted on the AHRQ Web site. For example, the Medical Expenditure Panel Survey (MEPS), which involves data collection, uses quality assurance procedures such as the validation of interviewers' work, performance of quality control checks on the source variables across all analytic groups, and benchmarking the data (e.g. comparisons with prior years and comparable data sources such as the Current Population Survey of the U.S. Census Bureau and the National Health Interview Survey). The Healthcare Cost and Utilization Project (HCUP), which aggregates and standardizes existing data from states, has quality control procedures in place to verify that the databases contain information that is valid, internally consistent, and consistent with established norms. Procedures include automated edit checks, careful review of all summary statistics, and benchmarking data with other sources (e.g. comparison with the National Hospital Discharge Survey).

Before disseminating information in any format, data are reviewed internally to ensure that they are scientifically sound and meet or exceed standards for data quality, statistical integrity, and confidentiality and privacy protections. Similarly, description and documentation of databases, tools, products, and on-line query systems are carefully reviewed, edited, and re-reviewed before release to the public.

Results of research projects

Quality standards: AHRQ staff adhere to the highest professional standards in their appropriate/respective fields, e.g., economics, social science and statistics. They seek to create transparency in their activities to enable others to fully understand their research processes, methodologies, and assumptions so that the work can be evaluated and replicated. Staff must ensure that strong measures are in place to guarantee the integrity of research information and to protect it from unauthorized access or revision.

Quality assurance procedures: All intramural research projects  — whether they involve qualitative or quantitative analysis  — and all projects on which AHRQ staff collaborate with authors outside the Agency are subject to AHRQ's intramural research planning process. This is intended to ensure conformance with strategic planning, accountability, peer review, scientific objectivity, confidentiality, and privacy standards. Research reports that are the products of intramural research undergo many levels of review prior to release, including hierarchical and peer reviews (sometimes including external reviews by other agencies and outside experts). Manuscripts submitted to scientific journals are subject to additional peer-review by the journal, prior to acceptance and publication.

In keeping with widely-accepted scientific research practices, research reports published by AHRQ describe the methods, data sources, analytical techniques, assumptions, and limitations of the research so the study can be substantially assessed and replicated.

Program Evaluations

Quality standards: Program evaluations are developed and assessed in accordance with the highest professional standards of evaluation practice.

Quality assurance procedures: Evaluation proposals are reviewed by Agency management for suitability, utility, and methodology, in accordance with sound evaluation design and standards of evaluation practice. Many evaluation projects have technical advisory committees that oversee the design and conduct of the evaluation in accordance with standard evaluation theory and practice; they often provide an expert review of the final report. Results of evaluation activities are released to the public only after Agency management has completed a review of the quality, accuracy and completeness of the report.

Software tools

Quality standards: Software tools are developed by teams of subject area experts in compliance with generally acceptable professional standards appropriate to their respective fields. Many levels of review and testing are built into the development process to ensure that the final product is useful, objective, and of high quality.

Quality assurance procedures: Reviews of tools in development are stringent. To ensure that they are needed and useful, the Agency generally relies upon peer experts and user groups both inside and outside of the Department. The Agency conducts extensive testing prior to release. Feedback is obtained from users of the tools or software to identify enhancements that will make the tools or software more fully meet user needs. As an example, the NGC is currently conducting its third annual on-line user survey to assess experiences with NGC and gather their recommendations for how to improve the system.

Consumer and professional health information

Quality standards: Consumer and professional information developed by AHRQ is conducted in compliance with generally accepted professional standards appropriate to the field of study being undertaken, e.g., clinical, economics, statistics, and social science research, as well as sound health care communication principles.

Quality assurance procedures: AHRQ works closely with stakeholder groups to determine the need for consumer and professional publications. Publications are based on information emanating from research that has been reviewed for scientific and medical accuracy and completeness by experts within and outside of the Agency before it is disseminated. Consumer publications are tested with targeted audiences to ensure relevance, clarity, and comprehension. In addition, AHRQ's publications containing health care recommendations indicate the level of substantiating evidence.