HHS Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated to the Public

A. Summary of HHS Agency Guidelines

The U.S. Department of Health and Human Services (HHS) developed these guidelines to implement Office of Management and Budget (OMB) January 2002 requirement that all federal agencies issue guidelines for ensuring the quality of the information that they disseminate to the public. The Department of Health and Human Services is the United States government's principal agency for protecting the health of all Americans and providing essential human services, especially to those who are least able to help themselves. The Department includes more than 300 programs, covering a wide spectrum of activities.

In the course of carrying out program missions, agencies and staff offices within HHS disseminate a wide variety of information to the public, ranging from research and statistical reports to authoritative health and medical information. Many of these information dissemination activities and products rank among the highest quality scientific, statistical and programmatic information among federal agencies, and in many cases set the national and international standard for quality.

HHS is committed to disseminating information that meets the standards of quality set forth in OMB and in the guidelines discussed in this document. It is HHS's goal to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination. The guidelines that follow describe the quality assurance policies and practices that support information dissemination activities in HHS.

Following an overview of the OMB Guidelines and HHS guidelines, the quality assurance policies of each of the major operating agencies and staff offices of HHS are described. Each set of agency guidelines includes a description of a) the mission of the agency, b) the scope and applicability of the guidelines within the agency, c) the types of information that the agency disseminates, d) the dissemination methods employed by the agency, e) the policies, standards and practices that the agency employs to ensure the quality of the information it disseminates, and f) an administrative mechanism and contact points for each agency so that individuals may seek correction of any information that is believed not to meet the OMB, HHS, or agency-specific guidelines along with an administrative appeals process.

B. OMB Guidelines

On September 28, 2001, and as amended on February 22, 2002, OMB issued final Guidelines to implement section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554). The statute directs OMB to "issue government wide guidelines that provide policy and procedural guidance to federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by federal agencies." By October 1, 2002, agencies must issue their own implementing guidelines. The guidelines only apply to information that is disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.

In general, the OMB Guidelines require agencies to adopt a basic standard of quality as a performance goal and take appropriate steps to incorporate information quality criteria into agency information dissemination practices. Quality is to be ensured and established at levels appropriate to the nature and timeliness of the information to be disseminated, and specific standards may be adopted that are appropriate to the various categories of information that is disseminated. Agencies are to develop a process for reviewing the quality of information before it is disseminated. Further, information quality is to be treated as an integral step in every aspect of the information development process.

In issuing the Guidelines, OMB outlined several guiding principles. First, OMB designed the Guidelines to apply to a wide variety of government dissemination activities that may range in importance and scope. OMB also designed the Guidelines to be generic enough to fit all media, whether printed or electronic. OMB specifically sought to avoid the problems inherent in developing detailed, prescriptive, "one size fits all" guidelines that would artificially require all types of dissemination activities to be treated in the same manner. Second, OMB designed the Guidelines so that agencies will meet basic information quality standards. The Guidelines recognize that some government information may need to meet higher or more specific standards than others, depending on their purpose and scope. The more important the information, the higher the quality standards to which it might be held, for example, "influential scientific, financial or statistical information" described below. At the same time, OMB recognizes that information quality comes at a cost. Accordingly, agencies are encouraged to weigh the costs and benefits of higher information quality in the development of information, and the level of quality to which it will be held.

Third, OMB designed the Guidelines so that agencies can apply them in a common sense, workable manner. OMB expects agencies to use existing processes rather than create new and potentially duplicative or contradictory processes. Finally, OMB recognizes that the Guidelines cannot be implemented in the same way by all agencies. While the implementation may differ, the essence of the Guidelines will apply. The agencies must make their methods transparent by providing documentation, ensure quality by reviewing the underlying methods used, by consulting as needed with both experts and users, and by keeping users notified about corrections and revisions. These underlying principles apply equally well across the diversity of HHS agencies and information dissemination activities, and they have been adopted in the approach to the HHS Guidelines described below.

C. HHS Responsibilities

In accordance with the OMB Guidelines, agencies subject to the Paperwork Reduction Act (44USC 3502(1)) are required to:

• Issue their own information quality guidelines ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by the agency not later than one year after the issuance of the OMB Guidelines;
• Establish administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated by the agency that does not comply with the guidelines. Agencies also are to specify appropriate time periods for agency decisions on whether and how to correct the information, and are to notify the affected persons of the action taken. If the person who requested the correction does not agree with the agency's decision (including the corrective action, if any), that person may file for reconsideration within the agency. The agency is to establish an administrative appeal process to review the initial decision, and specify appropriate time limits in which to resolve such requests for reconsideration.
• Report periodically to the OMB Director on the number and nature of complaints received by the agency regarding the accuracy of information disseminated by the agency and how such complaints were resolved by the agency.

The HHS guidelines described in this plan incorporate the underlying principles that OMB used in designing their government-wide guidelines. First, the HHS guidelines apply to a wide range of government information dissemination activities across HHS and are generic enough to fit all types of media, including print, electronic, and other forms within HHS. Second, the HHS guidelines are intended to assure that all the information that is disseminated meets a basic level of quality and that more important information meets a more rigorous quality standard. Third, the HHS guidelines explicitly recognize the very different types of information that various HHS agencies disseminate depending on their missions, including the need for flexibility in implementation and avoidance of a "one size fits all" approach. Fourth, the statement of HHS information quality policies and procedures are issued in the form of guidelines and not a regulation.

HHS itself encompasses a broad and diverse range of health and human services programs which, while unified in their pursuit of broad goals, are themselves very diverse, encompassing the nation's largest health insurance plan, the nation's preeminent biomedical research agency, as well as most of the nation's federal capacity for public health protection and preparedness and income assistance to needy families. Accordingly, the HHS approach to implementation of the OMB Guidelines is designed to allow HHS agencies and offices to use existing agency quality assurance mechanisms, and apply the guidelines in a flexible manner that recognizes the mission of the agency, the wide range of data that is disseminated and the frequent reliance on third party sources.

D. Framework for HHS Guidelines

1. Purpose

These Guidelines describe the policies and procedures that HHS agencies employ to ensure the quality of the information they disseminate and the administrative complaint mechanisms that HHS agencies make available to the public. The Guidelines provide policy and procedural guidance to HHS staff and are intended to inform the public about agency quality assurance policies and procedures. HHS views the guidelines as an evolving document and process. HHS will continually review the performance of the guidelines in the context of agency statutes and missions and will make revisions to the guidelines as necessary.

2. Definitions

1. "Quality" is an encompassing term comprising utility, objectivity, and Integrity. Therefore, the Guidelines sometimes refer to these four statutory terms, collectively, as "quality."
2. "Utility" refers to the usefulness of the information to its intended users, including the public. In assessing the usefulness of information that the agency disseminates to the public, the agency needs to consider the uses of the information not only from the perspective of the agency but also from the perspective of the public. As a result, when transparency of information is relevant for assessing the information's usefulness from the public's perspective, the agency must take care to ensure that transparency has been addressed in its review of the information.
3. "Objectivity" involves two distinct elements, presentation and substance. "Objectivity" includes whether disseminated information is being presented in an accurate, clear, complete, and unbiased manner. This involves whether the information is presented within a proper context. Sometimes, in disseminating certain types of information to the public, other information must also be disseminated in order to ensure an accurate, clear, complete, and unbiased presentation. Also, the agency needs to identify the sources of the disseminated information (to the extent possible, consistent with confidentiality protections) and, in a scientific, financial, or statistical context, the supporting data and models, so that the public can assess for itself whether there may be some reason to question the objectivity of the sources. Where appropriate, data should have full, accurate, transparent documentation, and error sources affecting data quality should be identified and disclosed to users. In addition, "objectivity" involves a focus on ensuring accurate, reliable, and unbiased information. In a scientific, financial or statistical context, the original and supporting data shall be generated, and the analytic results shall be developed, using sound statistical and research methods.
   1. If data and analytic results have been subjected to formal, independent, external peer review, the information may generally be presumed to be of acceptable objectivity. However, this presumption is rebuttable based on a persuasive showing by the petitioner in a particular instance. If agency-sponsored peer review is employed to help satisfy the objectivity standard, the review process employed shall meet the general criteria for competent and credible peer review recommended by OMB-OIRA to the President's Management Council (9/20/01) (www.whitehouse.gov/omb/inforeg/oira_review-process.html), namely, that (a) peer reviewers be selected primarily on the basis of necessary technical expertise, (b) peer reviewers be expected to disclose to agencies prior technical/policy positions they may have taken on the issues at hand, (c) peer reviewers be expected to disclose agencies their sources of personal and institutional funding (private or public sector), and (d) peer reviews be conducted in an open and rigorous manner.
   2. If an agency is responsible for disseminating influential scientific, financial, or statistical information, agency guidelines shall include a high degree of transparency about the data and methods to facilitate the reproducibility of such information by qualified third parties. Original and supporting data must be subject to commonly accepted scientific, financial, and statistical standards related thereto. However, agency guidelines shall not require that all disseminated data be subjected to a reproducibility requirement. Agencies may identify, in consultation with the relevant scientific and technical communities, those particular types of data that can practicably be subjected to a reproducibility requirement, given ethical, feasibility, or confidentiality constraints. It is understood that reproducibility of data is an indication of transparency about research design and methods and thus a replication exercise (i.e., a new experiment, test, or sample) shall not be required prior to each dissemination. With regard to analytic results, agency guidelines shall generally require sufficient transparency about data and methods that an independent reanalysis could be undertaken by a qualified member of the public. These transparency standards apply to agency analysis of data from a single study as well as to analyses that combine information from multiple studies. Making the data and methods publicly available will assist in determining whether analytic results are reproducible. However, the objectivity standard does not override other compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections. In situations where public access to data and methods will not occur due to other compelling interests, agencies shall apply especially rigorous robustness checks to analytic results and document what checks were undertaken. However, agency guidelines, in all cases, shall require a disclosure of the specific data sources that have been used and the specific quantitative methods and assumptions that have been employed. Each agency is authorized to define the type of robustness checks, and level of detail for documentation thereof, in ways appropriate for it given the nature and multiplicity of issues for which the agency is responsible. With regard to analysis of risks to human health, safety and the environment maintained or disseminated by the agencies, agencies shall either adopt or adapt the quality principles applied by Congress to risk information used and disseminated pursuant to the Safe Drinking Water Act Amendments of 1996 (42 U.S.C. 300g-1(b)(3)(A) & (B)). Agencies responsible for dissemination of vital health and medical information shall interpret the reproducibility and peer-review standards in a manner appropriate to assuring the timely flow of vital information from agencies to medical providers, patients, health agencies, and the public. Information quality standards may be waived temporarily by agencies under urgent situations (e.g., imminent threats to public health or homeland security) in accordance with the latitude specified in agency-specific guidelines.
4. "Integrity" refers to the security of information — protection of the information from unauthorized access or revision, to ensure that the information is not compromised through corruption or falsification.
5. "Information" means any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes information that an agency disseminates from a web page, but does not include the provision of hyperlinks to information that others disseminate. This definition does not include opinions, where the agency's presentation makes it clear that what is being offered is someone's opinion rather than fact or the agency's views.
6. "Government information" means information created, collected, processed, disseminated, or disposed of by or for the Federal Government.
7. "Information dissemination product" means any book, paper, map, machine-readable material, audiovisual production, or other documentary material, regardless of physical form or characteristic, an agency disseminates to the public. This definition includes any electronic document, CD-ROM, or web page.
8. "Dissemination" means agency initiated or sponsored distribution of information to the public (see 5 C.F.R. 1320.3(d) (definition of "Conduct or Sponsor"). Dissemination does not include distribution intended to be limited to government employees or agency contractors or grantees; intra- or inter-agency use or sharing of government information; and responses to request for agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act or other similar law. This definition also does not include distribution limited to correspondence with individuals or persons, archival records, public filings, subpoenas or adjudicative processes.
9. "Influential," when used in the phrase "influential scientific, financial, or statistical information," means that the agency can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important public policies or important private sector decisions. Each agency is authorized to define "influential" in ways appropriate for it given the nature and multiplicity of issues for which the agency is responsible.
10. "Reproducibility" means that the information is capable of being substantially reproduced, subject to an acceptable degree of imprecision. For information judged to have more (less) important impacts, the degree of imprecision that is tolerated is reduced (increased). If agencies apply the reproducibility test to specific types of original or supporting data, the associated guidelines shall provide relevant definitions of reproducibility (e.g., standards for replication of laboratory data). With respect to analytic results, "capable of being substantially reproduced" means that independent analysis of the original or supporting data using identical methods would generate similar analytic results, subject to an acceptable degree of imprecision or error.

3. Scope and Applicability of the Guidelines

The HHS guidelines described in this implementation plan apply to substantive information dissemination activities that are initiated or sponsored by HHS agencies. The pre-dissemination review described in the guidelines only applies to information that is disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information. The guidelines do not apply to the large proportion of extramural scientific research activity supported by HHS whose dissemination is the sole responsibility of the academic researcher rather than HHS. The guidelines do apply to the dissemination of information by federal intramural researchers if the dissemination is agency-initiated or sponsored. Otherwise, appropriate disclaimers are to be included in the report or speech to distinguish the research from agency views and positions. The guidelines do not apply to distribution of information limited to correspondence with individuals or persons, press releases that support the announcement or give public notice of information that the agency disseminates elsewhere, archival material, public filings, subpoenas or adjudicative processes. Nor do they apply to opinions, when the agency's presentation makes clear that what is disseminated is someone's opinion rather than fact or agency views.

4. Overview of HHS Information Dissemination and Quality Assurance

a. The HHS Mission

The Department of Health and Human Services is the United States government's principal agency for protecting the health of all Americans and providing essential human services, especially to those who are least able to help themselves. The Department includes more than 300 programs, covering a wide spectrum of activities. Program responsibilities include:

• Ensuring a safe and healthy America
• Conducting and supporting medical and social science research
• Preventing outbreaks of infectious diseases
• Assuring the safety of foods and drugs
• Administering the Medicare and Medicaid programs
• Providing financial assistance for low income families
• Improving maternal and child health
• Administering the Head Start program
• Preventing child abuse and domestic violence
• Providing assistance to States for substance abuse treatment and prevention
• Providing services for older Americans
• Assuring comprehensive health services for Native Americans

b. Categories of Information Disseminated

The development and dissemination of timely and high quality data and information is a critical component of the missions of many HHS programs, as well as those of HHS partners in the health and human services communities. As a result, HHS plays a major role in information dissemination, as a producer and user of high quality data and information, as a collaborator with partners in the health and human services communities, and as a national leader in health and human services information policy. In carrying out their diverse statutes and missions, HHS agencies develop and support the dissemination of the following types of substantive information:

• Scientific research studies (the results of biomedical, behavioral, services research and social science research)
• Statistical and analytic studies (the results of surveys, statistical systems as well as analytical and modeling studies and public use data files
• Programmatic, administrative and regulatory information, including program evaluations
• Authoritative health, medical and human services information aimed at consumers and health and human services professionals
• Public health surveillance, epidemiology and risk assessment studies and information

c. Types of Dissemination Mechanisms

HHS agencies disseminate information through a wide range of methods and print and electronic media. These include publication in peer reviewed literature, published reports, periodicals, newsletters, brochures, clearinghouses, websites, CD-ROM and other electronic media.

d. Overview of Quality Assurance Policies and Practices in HHS

Depending upon their specific statutes and missions and the nature of the information they disseminate to the public, HHS agencies currently use a variety of quality assurance methods and procedures. These methods and procedures are designed to maximize the quality of HHS information, including the objectivity, utility, and integrity.

• Objectivity involves a focus on ensuring that information is accurate, reliable and unbiased and that information products are presented in an accurate, clear, complete and unbiased manner. Objectivity is achieved by using reliable data sources and sound analytical techniques, and carefully reviewing information products prepared by qualified people using proven methods.
• Utility involves the usefulness of the information to its intended users. Utility is achieved by staying informed of information needs and developing new data, and information products where appropriate. Based on internal analyses of information requirements, convening and attending conferences, working with advisory committees and stakeholders, sponsoring outreach activities, and where appropriate, testing publications with targeted audiences to ensure relevance, clarity, and comprehensiveness, HHS agencies keeps abreast of information needs.
• Integrity refers to the security of information from unauthorized access or revision to ensure that the information is not compromised through corruption or falsification. HHS agencies have in place rigorous controls to ensure the integrity of its administrative information. Three distinct objectives are pursued in protecting the integrity of information: ensuring that there is no deliberate or accidental improper disclosure of sensitive automated information; protecting against deliberate or accidental corruption of automated information; and protecting against deliberate or accidental actions that cause automated information resources to be unavailable to users when needed. Information is accorded protection against disclosure, alteration, loss, or destruction based on the degree of sensitivity. In addition, HHS agencies use appropriate safeguards to protect data from improper disclosure by backing up critical data periodically, and, if a security incident occurs, by following proper incident response procedures. Managers are responsible for ensuring that employees, both Government and contractors, observe all security requirements, and that employees receive appropriate security training. HHS also is instituting an enterprise-wide structured management control review process that applies throughout the system life cycle. As part of this process, risk analyses are conducted to establish a balance between an acceptable level of risk and the costs associated with safeguards.

 

In addition, HHS is subject to a number of statutory requirements that protect the sensitive information it gathers and maintains on individuals. Among these are:

 

• Health Insurance Portability and Accountability Act of 1996
• Privacy Act of 1974
• Computer Security Act of 1987
• Office of Management and Budget (OMB) Circulars A-123, A-127, and A-130
• Government Information Security Reform Act
• Federal Managers' Financial Integrity Act (FMFIA) of 1982

e. HHS Information Quality Goals

The development of data and information within HHS is generally undertaken within the context of two overarching goals:

• attention to information quality as a total and continuing process, and
• commitment to making data and information supported with public funds available to the public, consistent with confidentiality concerns and resource availability. Further, when HHS agencies prepare a Paperwork Reduction Act (PRA) clearance submission, they strive to engage in a data development effort that will result in information that will be collected maintained, and used in a way that is consistent with OMB, HHS, and agency-specific information quality guidelines. As a general policy, HHS views data and information quality as a continuing process that begins at the inception of the information development process with project conceptualization and carries through all phases of data planning, design and execution, including information dissemination activities. Further, HHS agencies that support or sponsor research and statistical activities are encouraged to not only describe the methods and data sources in a clear and transparent manner in the reports they release, but also to make the data used for the report available to the public through public use data files, restricted access research files, research data centers, data archives and other mechanisms consistent with confidentiality, legal and proprietary restrictions. In addition, HHS agencies review the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treat information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination. The quality assurance processes that are used to ensure the quality of specific categories of information are described below.

• Research and scientific studies disseminated by HHS are subject to an external, objective peer review process at both the inception stage and the pre-dissemination stage as a part of the publication process in peer reviewed journals. In addition, the quality of all of the intramural research programs of HHS agencies is continually reviewed and monitored by advisory committees and boards of scientific advisors. In accordance with widely accepted scientific research practice in the U.S., research reports disseminated by HHS agencies describe the methods, data sources, analytical techniques, measures, assumptions and limitations of the research, so that the study could be substantially reproduced. If original data are employed, it is the policy of HHS to make every effort to make the data available to the public in de-identified form consistent with confidentiality requirements, proprietary restrictions and resource availability.
• Statistical activities of HHS agencies are based on reliable data sources and are carried out in accordance with modern statistical theory and practice, including scientific sampling, statistical inference and analytical techniques and practices. All statistical programs employ or have access to experts in statistics and research design. HHS houses the National Center for Health Statistics, the federal government's designated general purpose statistical agency for health statistics, as well as programmatic and special purpose statistical activities. All proposals for original data collection activities in HHS undergo a rigorous and exacting review process in connection with the Paperwork Reduction Act, which also provides opportunity for public comment in the design of the information collection. Frequent meetings with user groups are common, and individual surveys and statistical systems often employ project specific technical advisory groups. Substantive reports from HHS statistical activities undergo a quality review process within their organizations before they are released, including expert review by supervisors, internal peer review by qualified scientists and statisticians, and in some cases external peer review as well as expert review by other offices prior to dissemination.
• Programmatic and administrative information — A significant amount of substantive information is disseminated by HHS agencies in connection with and as a byproduct of the administration of programs. Often the programmatic and administrative information disseminated is obtained from third parties, such as States, grantees, health plan contractors or intermediaries, or community-based organizations. In their stewardship function, agencies often collect, compile, standardize, analyze and disseminate such programmatic information. While the reliance on third parties places limits on the federal quality assurance authorities, a variety of techniques are employed to promote the accuracy, completeness and timeliness of the information. These include use of generally accepted accounting and financial management procedures and principles, internal controls, legal certifications and assurances on the part of the organizations supplying the information, audited financial reports and statements, as well as sample audits and site visits, and checks for completeness and consistency with trends and external controls. Programmatic reports are typically subject to supervisory review before release.
• Program evaluation studies are often undertaken by HHS agencies to assess program functioning and identify opportunities for improvement. Agencies employ quality assurance procedures in the choice and development of evaluation projects. Proposals for evaluation activities in HHS agencies are usually reviewed by agency management for suitability, utility and methodology in accordance with sound evaluation design and standards of evaluation practice. Many evaluation projects have specific technical advisory committees that oversee the design and conduct of the evaluation in accordance with standard evaluation theory and practice, and they often provide an expert review of the final report. Results of evaluation activities are released to the public only after agency management has completed a review of the quality, accuracy and completeness of the report.
• Regulatory information — A variety of information is used in support of regulatory development and decision-making. Regulatory activity undertaken by HHS agencies closely tracks statutory authorities and program responsibilities. Scientific, financial, and statistical information used in support of regulatory decision-making is subject to a quality review process within the agency involving appropriate experts depending upon the nature of the information. In addition, all significant proposed regulations are reviewed by OMB prior to issuance for public comment, and all proposed regulatory actions provide for extensive public comment.
• Authoritative health, scientific and consumer information — Several HHS agencies develop and disseminate authoritative health and human services information intended for consumers and the professional community. In some instances, the agency simply provides a link to information developed by other authoritative organizations. In other instances, the agency develops its own consumer and professional practice information. In the latter case, the information is reviewed for scientific and medical accuracy and completeness by experts within the agency before it is disseminated. In a number of instances the information also is reviewed by scientific and medical advisory bodies before dissemination as well, depending on the nature of the information.
• Public health surveillance and epidemiological information — Several HHS agencies compile, analyze and disseminate information from public health surveillance systems and epidemiological activities. In many surveillance systems, the primary information is developed by State and local government agencies, clinical laboratories and other health care entities and reported to CDC for national aggregation and analysis. Data quality is assured through use of reliable data sources, appropriate statistical techniques, agreement on national reporting standards, quality control procedures, standard case definitions and reports, adherence to professional practices and standards for public health reporting in the U.S., and frequent consultation with the user community. Before such information is disseminated, it is reviewed for medical, scientific and public health accuracy, soundness and utility by agency experts. Comment and feedback is encouraged on such information, and HHS agencies work closely with the relevant professional and public health organizations.

f. Influential Scientific, Financial, and Statistical Information

From time to time, HHS agencies disseminate influential scientific, financial and statistical information as defined in the OMB Guidelines. In such instances, care is taken to ensure that the information is substantially reproducible and replicable. This goal is accomplished by using reliable data sources and sound analytical techniques, and by employing a high degree of transparency about the data, sources, methods, measures, assumptions and limitations used to develop the information in order to facilitate reproducibility by qualified third parties. In the case of original or supporting data, most major epidemiological and statistical activities sponsored by HHS agencies have well developed public use data dissemination programs that make much of the data available to the public in standardized, de-identified micro-data files. Because of confidentiality, ethical and feasibility constraints and legal obligations to third parties supplying the information, there may be instances where original or supporting data may not be available to the public, but HHS agencies typically will work with qualified third parties to facilitate understanding, and transparency in data sources and methods will be emphasized in the report or in reference documents. In the case of analytical studies, HHS agencies will make provisions for sufficient transparency about data and methods so that an independent reanalysis could be undertaken by a qualified member of the public.

g. Health, Safety, and Environmental Information

Several HHS agencies have science-based missions and use such information in decision making. These agencies adapt the quality standards discussed in the Safe Drinking Water Act Amendments of 1996 (SDWA) (42 U.S.C. 300g-1(b)(3)(A) and (B)). The adaptation involves the commitment of these agencies to use: a.) the best available science and supporting studies conducted in accordance with sound and objective scientific practices, including peer reviewed studies when available and b.) data collected by accepted methods (if reliability of the method and the nature of the decision justify use of the data.) Additional detail relating to the standards employed by individual agencies is described in Part II, the individual agency section of these guidelines.

h. Urgent Public Health and Safety Information

Several HHS agencies are responsible for dissemination of authoritative health, medical and safety information on a real time basis in order to protect the health of the public against urgent and emerging threats. Accordingly, nothing in these guidelines relating to reproducibility or peer review shall be construed to limit or delay the timely flow of vital information from agencies to medical providers, patients, health agencies, and the public. HHS reserves the right to waive information quality standards temporarily for agencies addressing urgent situations (e.g., imminent threats to public health or homeland security) in accordance with the latitude described in both the OMB and agency specific guidelines.

E. Overview of HHS Agency Complaint Procedures

The OMB Guidelines require all agencies to establish administrative mechanisms allowing affected persons to seek and obtain correction of information disseminated by the agency that does not comply with OMB, HHS or agency-specific guidelines. Agencies also are to specify appropriate time periods for agency decisions on whether and how to correct the information, and are to notify the affected persons of the action taken. If the person who requested the correction does not agree with the agency's decision (including the corrective action, if any), that person may file for reconsideration within the agency. The agency is to establish an administrative appeal process to review the initial decision, and specify appropriate time limits in which to resolve such requests for reconsideration. The agency need not respond substantively to information not covered by the guidelines or to frivolous or repetitive requests for correction.

Requests for correction of information quality pursuant to Section 515 are to be directed to the respective agency. A common format for complaints has been developed across HHS. The approach is described below. To accompany the actual implementation of the Guidelines in October 2002, HHS has created a department-wide website describing in user friendly terms the procedures and contact persons for submitting requests for corrections. The web site can be accessed at www.hhs.gov/infoquality.

• Responsibility of the Complainant
  In general, to seek an information quality request for correction of information disseminated by any HHS agency, individuals should follow the procedures described below. Requests for correction that are specific and provide evidence to support the need for correction will enable the agency to provide a satisfactory response. Complainants should be aware that they bear the "burden of proof" with respect to the necessity for correction as well as with respect to the type of correction they seek. A complaint or request for correction of information must be in written hard copy or electronic form, be sent to the agency designated address by mail or electronic-mail (e-mail); and state that an information quality request for correction is being submitted. In terms of content, the complaint letter must contain:
  • a detailed description of the specific material that needs to be corrected including where the material is located, i.e. the publication title, date, and publication number, if any, or the website and web page address (url), or the speech title, presenter, date and place of delivery;
  • the specific reasons for believing the information does not comply with OMB, HHS, or agency-specific guidelines and is in error and supporting documentation, if any;
  • the specific recommendations for correcting the information;
  • a description of how the person submitting the complaint is affected by the information error; and
  • the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.
• Responsibility of the Agency
  Based on a review of the information provided, the agency will determine whether a correction is warranted and if, so what action to take. The agency will respond to the requestor by letter or e-mail. The agency's response will explain the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information, the magnitude of the correction and the resource requirements for the correction. The response will describe how the complainant may request reconsideration. The agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.
• Appeals for Reconsideration
  If the individual submitting the complaint does not agree with the agency's decision (including the corrective action), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal shall state the reasons why the agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the agency response to it, clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal to the specific agency appeals address. The agency official who handles the original complaint will not have responsibility for resolving the appeal. The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.
• Rulemakings and Other Public Comment Procedures
  Existing public comment procedures for rule-makings and other formal agency actions already provide well established procedural safeguards that allow affected persons to raise information quality issues on a timely basis. Accordingly, agencies will use these existing procedures to respond to information quality complaints that arise in this process. In cases where the agency disseminates a study, analysis, or other information prior to the final agency action or information product, requests for correction will be considered prior to the final agency action or information product in those cases where in the agency's judgment issuing an earlier response would not unduly delay issuance of the agency action or information product and the complainant has shown a reasonable likelihood of suffering actual harm from the agency's dissemination if the agency does not resolve the complaint prior to the final agency action or information product.

A. Administration for Children and Families

I. Agency Mission

The Administration for Children and Families (ACF) is responsible for administering numerous federal programs: Temporary Assistance to Needy Families (TANF), child support, child care, Head Start, child welfare and other programs relating to children and families; services for those with developmental disabilities and mental retardation, refugee services, and Native American/Tribal programs. Actual services are provided by state, county, city, and tribal governments, and public and private local agencies. ACF assists these organizations through funding, policy direction, and information services.

II. Scope and Applicability of Guidelines for Agency

ACF is committed to ensuring that disseminated information meets the standards of quality set forth in the OMB, HHS and ACF guidelines. It is ACF's policy to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

The pre-dissemination review described in the guidelines applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information. This section identifies the types of information covered by the Guidelines, and also lists the types of information that are exempt. ACF information subject to the Information Quality Guidelines includes:

• Statistical information;
• Studies and summaries prepared for public dissemination to inform the public about the impact of programs administered by the agency; and
• Studies and summaries prepared for use in formulating broad program policy.

The following types of information are not subject to the Information Quality Guidelines:

• Information relating to ACF programs, archival information, clearinghouses, and Internet distribution of studies, reports, documents, summaries, and articles not authored by the agency and not representing the agency's views, including materials authored and distributed by ACF grantees;
• Information limited in dissemination to government employees (intra- or interagency) or agency contractors or grantees;
• Information pertaining to basic agency operations or management information produced primarily for internal use;
• Procedural or policy manuals;
• Responses to requests for agency records under the Freedom of Information Act (FOIA), the Privacy Act, the Federal Advisory Committee Act, or other similar laws;
• Information relating solely to correspondence with individuals or persons; and
• Press releases that support the announcement or give public notice of information that ACF has disseminated elsewhere.

III. Types of Information Disseminated by the Agency to the Public

Given the wide variety of programs administered by the Administration for Children and Families, and the numerous types of information generated by these programs, it is not possible to offer a comprehensive list of all dissemination activities conducted by the agency. Consequently, the following examples have been drawn selectively from a cross-section of ACF programs. However, it should be noted that not all of the information described below is subject to the OMB Guidelines.

1. Program Information — descriptions of programs such as Head Start, child care, child welfare and youth services, child support enforcement, community services, Temporary Assistance for Needy Families (TANF), and low-income family assistance programs; budget and funding information; data on numbers and locales of grantees; types of services supported through agency programs; and other descriptive information. For example, the Children's Bureau makes available a variety of materials relating to basic program information. Fact sheets describing the purpose and funding level of each of the Children's Bureau's programs are available on the ACF web site. The Children's Bureau web site also has information of a general nature relating to child welfare, for instance, information on how to report child abuse and neglect or how to become a foster or adoptive parent. Examples of information offered by a typical web site (Administration on Developmental Disabilities) to the public include: a home page; staff directory; calendar of major program events; a fact sheet; links to state-based grantees; updates on various major initiatives; links to national disability organizations and related disability sites; connectivity to the Developmental Disabilities Assistance and Bill of Rights Act of 2000, current regulations, program highlights and outcomes; and a section devoted to frequently asked questions, where the public can search for answers to their questions or email a question to the program staff. The Family and Youth Services Bureau's (FYSB) National Clearinghouse on Families and Youth (NCFY) disseminates:
   • The Exchange, a periodical focusing on issues of interest to youth service professionals;
   • FYSB Update, a periodical sharing information about the Bureau's research and demonstration projects;
   • Technical assistance publications, providing guidance about how to address specific policy or operational issues;
   • Publications for young people, parents and community members (developed by FYSB and NCFY); and
   • Publications from all sources on youth issues (including those not developed by FYSB).
2. Statistical information — information on numbers of sub-populations served through ACF programs supported wholly or in part by Federal dollars (e.g., characteristics and financial circumstances of families served by TANF; information on program recipients' participation in work activities; information on the child support caseload; number and characteristics of children who are victims of child abuse and neglect or who are adopted from public child welfare systems; number of children served in Head Start; percentage of Head Start teachers with a college degree; numbers and characteristics of children served by Runaway and Homeless Youth Programs; number of children in each State receiving a child care subsidy under the Child Care and Development Fund.). Each year, the Children's Bureau publishes a report called "Child Maltreatment," presenting the latest statistics and trends in child abuse and neglect. In addition, the Children's Bureau publishes an annual report on "Outcomes in Child Welfare" with information from each State.
3. Research and evaluation reports focused on social science research and evaluation pertinent to ACF programs and policy. The web site of each program within ACF makes the most recent reports available to members of the research community and the general public. Examples of research and evaluation reports disseminated include the "1998 National Estimates of the Number of Boarder Babies, Abandoned Infants and Discarded Infants" and the final report of the "Third National Incidence Study on Child Abuse and Neglect."
4. Reports to the Congress — Several programs submit annual reports to Congress in compliance with legislative provisions authorizing their implementation. For example, the Children's Bureau's report, "Blending Perspectives and Building Common Ground: A Report to Congress on Substance Abuse and Child Protection" complies with such a legislative provision. In August 2000, TANF submitted the Third Annual Report to Congress to comply with section 411(b) of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) of 1996. The Family and Youth Services Bureau submits a biennial report to Congress that describes Runaway and Homeless Youth (RHY) operations and other activities. After release to Congress, this report is made available in the public domain. Annual RHY caseload data from the Runaway and Homeless Youth Management Information System (RHYMIS) are included in the Report to Congress and provided on request to grantees, researchers, advocacy organizations, and the general public.

IV. Types of Dissemination Methods

Information is disseminated through printed, electronic, and presentation materials. Dissemination vehicles include brochures and targeted mailings to State officials, grantees, or others responsible for administering programs; posting information on the ACF web site; making information available through clearinghouses and resource centers; presenting and distributing information at appropriate conferences; and electronically collecting and distributing aggregated and disaggregated program-related data. Research funded by several agency programs is also disseminated through peer-reviewed journals.

Research and evaluation reports are disseminated in a number of ways. Reports with particular policy relevance and findings are distributed directly by mail to State program directors or other targeted audiences. Information on newly released reports is also shared with organizations that warehouse and broadcast information through their own networks (e.g., the Welfare Information Network and the Research Forum on Children, Families and the New Federalism). ACF's web site provides information on current research and evaluation projects, posts reports (or links to reports), and disseminates information through a listserv. In addition, ACF sponsors research conferences such as an annual Welfare Reform Research and Evaluation Conference that attracts State program staff, Federal regional staff researchers in the field and other organizations interested in ACF programs.

V. Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Disseminated to the Public

At ACF, the quality assurance process begins at the inception of the information development process. Further, ACF reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination.

The review procedures may vary among programs depending upon the requirements of the program and the type of information to be disseminated:

1. Statistical Information: Statistical information pertaining to several programs disseminated by ACF is gathered from State agencies or local grantees. To assure the quality, integrity and consistency of this information, ACF staff review and analyze the data, subject the data to electronic validation procedures such as logic and edit checks, and contact the State or grantee to seek clarification in case discrepancies arise. States are provided opportunities to review and revise their information prior to publication in a federally published document or their placement on the web site. For example, TANF data collection is subject to edit and consistency checks and additional statistical analyses through a series of frequency distributions and cross tabulations to ensure the quality of data. The TANF program shares compiled information (e.g., work participation rates and High Performance Bonus data) with the States before publication to ensure accuracy.
2. Surveys: Surveys sponsored by ACF are conducted using methodologies that are consistent with generally accepted professional standards for all aspects of survey development, including sample frame development, statistical design of the survey sample, questionnaire design and testing, data collection, sampling and coverage errors, nonresponse analysis, imputation of missing data, weights and variance estimates. ACF surveys follow guidelines and policies set forth in the Paperwork Reduction Act (PRA) and other regulations related to the conduct of government surveys. ACF is committed to demonstrating in its PRA clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and ACF information quality guidelines. For example, ACF currently is funding a major multi-year longitudinal survey — the National Survey of Child and Adolescent Well-Being (NSCAW) — designed and implemented by a reputable national research organization. This project consults with a technical advisory group composed of state officials, members of the academic community, child welfare professionals and practitioners, and federal experts in the implementation of this study. Major research and evaluation studies sponsored by the agency usually rely upon technical advisory groups or federal experts or both for project guidance.
3. Analytic Reports and Policy Studies: Reports submitted by grantees and contractors for projects funded by ACF are subject to internal reviews by several agency officials for quality, objectivity, and accuracy. Federal experts from within and outside of the Department often monitor interim and final reports of projects with crosscutting implications. Information contained in analytical reports and policy studies is based on estimates derived from reliable administrative data and external data sources. All data sources are identified. All analytic reports and policy studies are reviewed by technically qualified staff to ensure valid, complete, unbiased, objective and relevant analysis. Analytic reports and policy studies considered to be more technically complex are also reviewed by subject matter experts outside of the originating organizational component to provide additional perspective and expertise.
4. Policy documents are cleared through ACF and Departmental clearance procedures, including review by the Office of the General Counsel (to assure proper legal interpretation), as well as review by numerous other knowledgeable officials. In some cases, a technical advisory group convened for a specific project may review and comment on a report prior to its release. For example, the Office of Family Assistance (OFA) has developed intra-Departmental procedures for assuring the quality of disseminated information. The Office of the General Counsel, key OFA staff, and other offices review significant policy documents, where appropriate, and the documents are cleared by the Office Director before issuance. All ACF regulations and reports to Congress are subject to review and clearance under both ACF and Office of the Secretary's clearance processes in addition to other reviews.

Policy for Correcting Errors:

If an error is detected in the agency's reports and publications before mailing, it is corrected. If these materials have already been mailed, ACF issues a special notification to the recipients or includes an errata sheet with the subsequent publications. Errors in materials in the agency's web site are corrected online.

VI. Agency Administrative Complaint Procedures

ACF has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and ACF guidelines.

External complaints about information disseminated can be made in the form of written correspondence. The Chief Information Officer (CIO), is the ACF official designated to receive and resolve complaints regarding information that does not comply with either the OMB guidelines or the agency's guidelines. The CIO's address is: Chief Information Officer, Deputy Assistant Secretary for Administration , Administration for Children and Families, Aerospace Building, 6th Floor, 370 L'Enfant Promenade SW, Washington, D.C. 20447. Feedback and complaints may be sent electronically to InfoQuality@acf.hhs.gov.

1. Responsibility of the Complainant
   To seek a correction of information disseminated by the agency, individuals should follow the procedures described below:
   1. A complaint or request for review and correction of information shall be in written hard copy or electronic form;
   2. It shall be sent to the agency by mail or electronic-mail (e-mail);
   3. It shall state that an information quality request for correction is being submitted.
   
   The complaint shall contain
   4. A detailed description of the specific material that needs to be corrected including where the material is located, i.e., the publication title, date, and publication number, if any, or the website and web page address (URL), or the speech title, presenter, date and place of delivery;
   5. The specific reasons for believing the information does not comply with OMB, HHS or ACF guidelines and is in error and supporting documentation, if any;
   6. The specific recommendations for correcting the information;
   7. A description of how the person submitting the complaint is affected by the information error; and
   8. The name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.
   
   Complainants should be aware that they bear the "burden of proof" with respect to the necessity for correction as well as with respect to the type of correction they seek.
2. Responsibility of the Agency
   Based on a review of the information provided, the agency will determine whether a correction is warranted and, if so, what action to take. The agency will respond to the requestor by letter or e-mail. The agency's response will explain the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. The agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.
3. Appeals
   If the individual submitting the complaint does not agree with the agency's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal shall state the reasons why the agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the agency response to it, clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal to the specific agency appeals address. The agency official who resolved the original complaint will not have responsibility for the appeal. The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

VII. Influential Scientific, Financial, and Statistical Information

The Guidelines apply to certain statistical information disseminated by ACF in view of the potential substantial impact on important public policies. Since statistical information for major ACF programs such as foster care, child abuse and neglect, TANF, etc., are provided by States, ACF will work closely with the States to assure the completeness, reliability, and integrity of the data and the transparency of methodology and analytical techniques.

B. Administration for Community Living

I. Agency Mission

The mission of the Administration on Aging (AoA) is to ensure that older Americans have the opportunity to age with dignity, have choices in managing their own lives, and remain active and productive members of their families and communities. We fund states, tribal agencies, city and county governments, and non-profit organizations at the community level. Core programs include home and community-based services such as transportation, chore and personal care; home and community-based nutrition services; preventive health services; support to caregivers; services that protect the rights of older persons; services for Native Americans and persons with Alzheimer's Disease. AoA assists these organizations by providing policy direction, technical assistance and information. Results and accomplishments of AoA projects and activities are made available to the aging community and to the public at large.

II. Scope and Applicability of Guidelines

A. Covered Information 

AoA is committed to ensuring that disseminated information meets the standards of quality set forth in the OMB, HHS and AoA guidelines. It is AoA's goal to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

The pre-dissemination review described in these guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information. The guidelines apply to the dissemination of substantive information that is endorsed or sponsored by the Agency. This includes:

1. Programmatic Analyses and Evaluations
2. Reports, Monographs, and Technical Assistance Materials, including brochures, documents, newsletters, electronic documents, and audiovisual productions, as well as reports and proceedings from AoA sponsored conferences and meetings
3. Consumer Information
4. Oral information, including speeches, interviews, expert opinions, only if representing AoA's views, official positions, or policies as well as written op-ed and similar editorial materials
    

B. Information Not Covered

1. CCCS bibliographic information or other archival records
2. Documents not authored by the agency and not representing the agency's views, including information authored and distributed by AOA grantees. Information that is intended to be limited in dissemination to Government employees or agency contractors or grantees
3. Information pertaining to basic agency operations, e.g., information about agency authorities, activities, programs, along with contact information for the public, organizational charts, AoA Status Reports, solicitations, program announcements (PAs), receipt and review materials (e.g., summary statements developed in connection with grant reviews)
4. Information intended solely for intra- or interagency use or sharing of Government information
5. Responses to requests for agency records under the Freedom of Information Act (FOIA), the Privacy Act, the Federal Advisory Committee Act, or other similar laws
6. Information relating solely to correspondence with individuals or persons
7. Press releases that support the announcement or give public notice of information that AoA has disseminated elsewhere.
8. Information intended to be limited to public filings, subpoenas, or adjudicative processes
9. Opinions where the agency's presentation makes it clear that what is being offered is personal opinion rather than fact or the agency's views
10. Information that is intended to be limited in dissemination to Government employees or agency contractors or grantees

III. Types of Information Disseminated by AoA to the Public

Each year, AoA produces a number of publications of various types, and about 500 static Web pages. All publications that carry the AoA imprimatur, i.e. are considered official AoA publications or releases and must follow AoA procedures for appropriate for that type of material (see Section V). The types of information disseminated by AoA to the public include the following:

1. Programmatic Analyses and Evaluations

   This category includes program reviews, evaluations and recurring reports. Analyses of project results may be developed and used for many purposes, including meeting annual reporting requirements, such as the Government Performance and Results Act (GPRA). Highlights of evaluations and analyses may be on the AoA Web site, and used in testimonies and speeches by AoA officials.

2. Reports, Monographs, and Technical Assistance Materials

   These are generally authored or co-authored by AoA staff analysts as part of their official duties or leading professionals in the field under contract. Such reports typically either provide technical assistance information to agencies and service providers working in aging, provide statistical compilations (typically of Census demographical data which is needed by aging agencies and they plan services) or the reports may address a program or policy issue of concern.

3. Consumer Information

   AoA provides a number of resources for the general consumer to learn about a variety of aging related concerns such as health information, housing programs available to older people and how to access the service system. A considerable amount of this information is distributed through AoA Center for Communications and Consumer Services (CCCS). Other sources of consumer information include distribution by other AoA staff and the AoA web site.

4. Oral information, including speeches, interviews

   AoA executives and experts make frequent speeches on a variety of issues relating to aging and are interviewed by the media (only if representing AoA's views, official positions, or policies) as well as written op-ed and similar editorial materials.

IV. Types of Dissemination Methods

AoA information is disseminated in many mediums, with the following being most common:

1. Print -- publications, books, newsletters, brochures, booklets, pamphlets, and reports.
2. Oral -- formal speeches, oral presentations, interviews, or commentaries for publication or broadcast; letters-to-the-editor or correspondence likely to result in similar publications.
3. Electronic -- The AoA Web site is a Central Information Resource in Aging. It is also a vehicle for enhancing the way AoA carries out its business operations. To accomplish this dual mission, AoA must put the needs of its users first. It serves as a dynamic source of credible, reliable and timely information and technical assistance designed to support our network, aging service providers, older people and their families in the daily conduct of their lives and work as well as helping the agency carry out its various business operations.

V. Quality Assurance Policies, Standards, and Processes for Ensuring the Quality of Information Disseminated to the Public

A. Overview

All AoA documents must be prepared in accordance with professional and ethical standards, as well as generally accepted standards of good taste. They must be appropriate for dissemination by this agency, and must undergo appropriate review and approval prior to release. At AoA, the quality assurance process begins at the inception of the information development process. AoA efforts to ensure and maximize information quality begin at the preparation stage, and continue through the review and approval stages. Existing AoA policies developed in concert with Federal computer security laws provide appropriate security safeguards to ensure integrity of AoA documents, i.e., ensure that the information is protected from unauthorized access, revision, corruption, or falsification.

AoA has quality control measures embedded in its grant application review process to ensure that the information ultimately disseminated is of high quality. AoA grant applications undergo extensive professional review.

B. AoA Information Review and Approval Policies and Procedures

Publications are expected to meet high standards of quality, make a substantial contribution to the field, and contain sufficient information for the informed audience to assess its validity. The review, approval, and dissemination of substantive information by AoA require adherence to appropriate clearance procedures for this type of material and are consistent with HHS guidelines. The originating office is responsible for obtaining the necessary clearances for reproduction and distribution of printed materials and should ensure that written material distributed is appropriate and consistent with HHS policy. A document that has obtained publication clearance for paper printing is often posted on the AoA web site for greater accessibility. AoA Web documents with no print counterpart require content clearance by the appropriate AoA office or contact person to ensure that the information observes all applicable requirements governing information for release to the public.

As noted above, AoA issues four types of information:

1. Programmatic Analyses and Evaluations
2. Reports, Monographs, and Technical Assistance Materials
3. Consumer Information
4. Oral Information

AoA reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination. The information review and approval process is essentially the same for each of these types of information. Such information must meet the following guidelines which AoA employs to assure the quality of its information products, including their utility, objectivity, and validity:

Utility involves the usefulness of the information to its intended users. Utility is achieved by staying informed of information needs and developing and information products which are appropriate to these needs. This is achieved by: internal analyses of information requirements, continuous informal discussions and consultation with members of the aging services network, project grantees and other experts in the field, consultation with consumer groups, convening and attending conferences, working with professional groups, and sponsoring outreach activities. Where appropriate, contact information is available on each publication to allow feedback and questions by users.

Objectivity involves a focus on ensuring that information is accurate, reliable and unbiased and that information products are presented in an accurate, clear, complete and unbiased manner. Objectivity is achieved by using reliable data sources and sound analytical techniques, and preparing information products that use proven methods by qualified people that are carefully reviewed.

Integrity -- Information contained in analytical reports and technical assistance materials is based on valid and reliable data sources which are identified. All reports and materials are reviewed by technically qualified staff to ensure that analysis is valid, complete, unbiased, objective and relevant. Materials that are considered to be more technically complex are also reviewed by subject matter experts and other AoA program experts outside of the originating component to provide additional perspective and expertise.

C. Administrative Requirements

AoA printed publications must be cleared through the Assistant Secretary for Aging and the Office of the Assistant Secretary for Public Affairs (OASPA), HHS. Similarly, all AoA Audio/Visual projects and exhibits must be cleared through OASPA. Further, AoA will demonstrate in its PRA clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and AoA information quality guidelines.

In general, any writing by an AoA employee on a work-related subject, whether intended for electronic or print publication, or for oral delivery, must be prepared according to accepted AoA standards of quality, reviewed for substantive content, and administratively approved. The purpose of the AoA clearance process is to improve the quality of information, and to ensure the accuracy, objectivity, utility, and validity of information. Directors of AoA component units (or their delegates) are responsible for establishing and maintaining controls to ensure competent and timely clearance of professional writing and presentations by developing procedures appropriate to the type of information.

Oral information, including speeches, interviews, expert opinions, only if representing AoA views, official positions, or policies, including any statements, comments, or discussion of Federal policies or practices that are relevant to the employee's position or duties, draw conclusions, advocate or oppose professional practices or positions on subjects related to AoA duties, or might otherwise be construed as reflecting an official position by AoA, HHS, or the Federal Government, are covered by the OMB Guidelines, and must be approved in the Office of the Assistant Secretary for Aging or the Director of CCCS, AoA.

No review or approval is required for nonofficial and private writing, speaking, and publishing by an employee unless his/her AoA employment is likely to be regarded as influencing the content.

D. Special Considerations for Agency Dissemination

Disclaimers -- Normally, the need for a disclaimer is eliminated through the clearance process. However, a disclaimer may still be needed even after official clearance to clarify that the presentation should not be construed as necessarily representing AoA views, and/or to distinguish the status of information (e.g., preliminary, based on partial data set). The Department's regulations (Standards of Conduct) to which the AoA subscribes, require that disclaimers be used in all unofficial writing and editing related to the employee's official duties and/or affiliation with programs of the Federal Government in which the employee's identification with AoA is to be shown, can be inferred, or is well-known.

E. AoA Center for Communications and Consumer Services (CCCS)

The Center for Communications and Consumer Services (CCCS), a unit of AoA, is a central source for a wide variety of consumer as well as program- and policy-related materials and demographic and other statistical data on the health, economic, and social status of older Americans. CCCS services include the following: inquiry response, repository of AoA-funded research products, a bibliographic database of AoA-funded r&d products and other aging-related products, a reading room and reference collection, and statistical information.

At AoA, there are essentially three types of materials being disseminated through information services to the public: (1) Materials produced by AoA staff or contractors that undergo usual AoA review and approval processes; (2) materials produced by AoA grantees that are subject to policies and procedures in the AoA Grants Administration Manual; and (3) other materials not produced by AoA but available from a variety of professional sources (e.g., brochures handbooks, statistical tables, articles) whether in print or in electronic format, with appropriate disclaimers attached. Non-Federal materials typically undergo careful professional review before they are disseminated by AoA information staff. Such materials are accompanied by appropriate disclaimers.

F. Web Site Information

AoA has designated content area experts to develop or recommend material for inclusion on the AoA web site. These content area specialists periodically review material on the Web page to determine whether it is accurate and up to date. Information, particularly time-sensitive information, should be posted as soon as possible. Web page creators are expected to promptly update or remove out-of-date information.

Unless noted otherwise, information posted on pages within the "AoA.Gov" domain is considered to be "in the public domain." As such, others are free to establish links to AoA online resources. In establishing such links, AoA requests that others avoid creating the impression that AoA is endorsing or promoting any particular product or service. In the same vein, any outside link to an external resource from an AoA Web site is examined on a case-by-case basis. In general, the content manager for each page determines when links to outside entities are justified.

AoA web pages containing links to external web pages not located on AoA servers should include a link to a statement that releases AoA from responsibility for the material included in the external Web page. Again, it is important to avoid giving a user the impression that AoA is endorsing information or a service or product described in an external site.

VI. Agency Administrative Complaint Procedures

Responsible Official

AoA has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and AoA guidelines.

The AoA official with overall responsibility for receiving and resolving complaints regarding information that does not comply with agency's guidelines for data quality is the Director of the Center for Communications and Consumer Services (CCCS). The Director of CCCS will have overall responsibility for implementing AoA Information Quality Guidelines, and will work collaboratively with other units in AoA.

1. Responsibility of the Complainant

   To seek a correction of information disseminated by the agency, individuals should follow the procedures described below.

   1. A complaint or request for review and correction of information shall be in written hard copy or electronic form;
   2. it shall be sent to the agency by mail or electronic-mail (e-mail); and
   3. it shall state that an information quality  request for correction is being submitted.
       

   The complaint shall contain

   4. a detailed description of the specific material that needs to be corrected including where the material is located, i.e., the publication title, date, and publication number, if any, or the web site and web page address (url), or the speech title, presenter, date and place of delivery; and
   5. the specific reasons for believing the information does not comply with OMB, HHS or AoA guidelines and is in error and supporting documentation, if any ;
   6. the specific recommendations for correcting the information;
   7. a description of how the person submitting the complaint is affected by the information error; and
   8. the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.

   
   Complainants should be aware that they bear the "burden of proof" with respect to the necessity for correction as well as with respect to the type of correction they seek. All complaints shall be sent to:
    

   Director, Center for Communications and Consumer Services
   Administration on Aging
   330 Independence Avenue, SW
   Washington, DC 20201

   
   Alternatively, complaints submitted via electronic mail should be sent to: dataquality@AoA.gov.

2. Responsibility of the Agency

   Based on a review of the information provided, the agency will determine whether a correction is warranted and if, so what action to take. The agency will respond to the requestor by letter or e-mail. The agency's response will explain the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. The agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

3. Appeals

   If the individual submitting the complaint does not agree with the agency's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal shall state the reasons why the agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the agency response to it, clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal to the specific agency appeals address. The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date. The agency official who resolved the original complaint will not have responsibility for the appeal. Appeals will be decided by the Director, Center for Wellness and Aging and should be addressed to:

   
   Deputy Assistant Secretary for Policy and Programs
   Administration on Aging
   330 Independence Avenue, SW
   Washington, DC 20201

VII. Influential Scientific, Financial, and Statistical Information

The OMB Information Quality Guidelines require that "influential" scientific, financial, or statistical information in official Government documents must be based on studies that can be substantially reproduced if the original or supporting data were to be independently reanalyzed using the same methods. "Influential" when used in the phrase "influential scientific, financial, or statistical information" means that the AoA can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important public policies or important private sector decisions, or will have important consequences for specific health practices, technologies, substances, products, or firms.

AoA is committed to applying rigorous professional standards to ensure the accuracy and reliability of research results. For professional and technical documents, AoA regularly seeks input from qualified analysts and experts from both within and outside AoA to insure that these influential materials are appropriate, accurate, complete, and of high quality (including objective, useful, and valid) prior to dissemination.

AoA continues to encourage the sharing of data and methods where practicable. Since the influence and implications of AoA-disseminated information cannot always be fully anticipated, all AoA reports are expected to state clearly how analytic results are generated -- the specific data used, various assumptions, specific analytic methods, statistical procedures, sources of error -- making the analysis sufficiently transparent so as to be capable of being reproduced. AoA advocates the archiving of data where feasible to facilitate the reproducibility of influential information.

VIII. References

HHS Standards of Conduct Regulations (45 CFR 73.735-705) (updated October 1, 2000).

The HHS Printing Handbook (September 1998).

AoA Administrative Procedures Manual (2001).

AOA Grants Administration Manual.

Office of Management and Budget (OMB) "Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies" -- Final Guidelines (January 3, 2002).

OMB, Circular No. A-130, Revised (Transmittal Memorandum No. 4). Management of Federal Information Resources (November 30, 2000).

Last revised: December 13, 2006

C. Agency for Healthcare Research and Quality

I. Agency Mission

The mission of the Agency for Healthcare Research and Quality (AHRQ) is to develop scientific evidence that enables health care decision makers — patients and clinicians, those who manage and purchase health care services, and policy makers  — to make more informed health care choices. AHRQ accomplishes its mission by conducting, supporting, and disseminating scientific research designed to improve the outcomes, quality, and safety of health care, reduce its cost, broaden access to effective services, and improve the efficiency and effectiveness of the ways health care services are organized, delivered, and financed.

The Agency was created December 1989 as the Agency for Health Care Policy and Research (AHCPR) and reauthorized on December 6, 1999, as the Agency for Healthcare Research and Quality. AHRQ is an agency of the Public Health Service (PHS) within the U.S. Department of Health and Human Services (HHS).

At AHRQ, the Director will have overall responsibility for implementing the AHRQ Information Quality Guidelines.

II. Scope and Applicability of Guidelines for Agency

AHRQ will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and AHRQ guidelines. It is AHRQ's goal to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. AHRQ strives to provide information that is accurate, reliable, clear, complete, unbiased, and useful. It is committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

The pre-dissemination review described in the guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.  The guidelines apply to information in all media (e.g., print, electronic) and apply primarily to the dissemination of substantive information (e.g., reports, studies, and summaries relating to the findings and related products of AHRQ research) rather than on information pertaining to basic Agency operations not intended for public dissemination.

Many AHRQ activities result in publicly disseminated information that is subject to the OMB Guidelines including:
 

• Statistical information and databases
• Results of research projects
• Program evaluations
• Software tools
• Consumer and professional health information

The following types of information produced by the Agency are not covered: scientific information produced by extramural researchers who conduct and disseminate their own work in the same manner as their academic colleagues and whose views do not represent the views of the Agency; information in which the results are identified as not being Agency views or positions; archival information; information intended soley for intra-or-interagency use or sharing of government information; press releases that support the announcement or give public notice of information that AHRQ has disseminated elsewhere; correspondence with individuals; responses to requests for Agency records under the Freedom of Information Act (FOIA); public filings; subpoenas; and the findings and determinations that AHRQ makes in the course of adjudications  involving specific parties.

III. Types of Information Disseminated by the Agency to the Public

Data products  — Includes print and electronic materials that describe or present aggregate statistical information such as data from the Medical Expenditure Panel Survey (MEPS), a survey of households regarding how they use and pay for health care. MEPS Public Use Data Files (on the Web site and CD-ROM) allow access to aggregate data. Another example is the Healthcare Cost and Utilization Project (HCUP), a family of health care databases and related tools with data provided by state data organizations, state hospital associations, and other sources. Web-based interactive analytic tools (i.e., MEPSnet, HCUPnet) are also available.

Results of research projects  — Research reports and other information products such as fact sheets, research syntheses, and AHRQ's monthly newsletter Research Activities are produced in both print and electronic formats and describe the findings and methods of AHRQ-funded research projects.

Program evaluation reports  — Reports summarizing the results of program evaluations are included in the departmental catalog of program evaluations maintained by the Office of the Assistant Secretary for Planning and Evaluation. Many of the Agency's program evaluations are driven by the goals established under the Government Performance and Results Act (GPRA).

Software tools  — Tools developed to assist health care policy makers, health system leaders, purchasers, and others in carrying out their tasks and making health care choices. An example is the National Guideline Clearinghouse (NGC), which provides standardized information on evidence-based clinical practice guidelines.

Consumer and professional health information  — AHRQ produces a wide range of print and electronic consumer and professional information. Examples include consumer publications on: various health conditions/diseases, on how to select a health plan, personal health guides, and how to improve the quality of health care you receive as a patient. Examples of publications for professionals include The Physicians Handbook of Preventive Services and evidence reports on a wide array of clinical topics that can be used by public and private entities to develop and implement their own practice guidelines, performance measures, review criteria, and other clinical quality improvement tools.

IV. Types of Dissemination Methods

AHRQ principally uses print and electronic methods of dissemination.

Print methods include:
 

• Reports, fact sheets, and research syntheses communicating the results of surveys and other research activities
• Publication of Research Activities, AHRQ's monthly newsletter AHRQ

Electronic methods include:
 

• The AHRQ Web site as well as AHRQ program specific sites
• AHRQ's Electronic Newsletter

Information Clearinghouse: AHRQ also has an Information Clearinghouse used to disseminate print and other media.

Secondary methods of dissemination include, but are not limited to: workshops, audiovisual techniques, oral presentations, and teleconferences. These dissemination mechanisms largely draw on information developed and reviewed first through one of the primary print or electronic modes of information dissemination.

V. Agency Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Disseminated to the Public

The quality assurance process begins at the inception of the information development process. AHRQ reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including creation, collection, maintenance, and dissemination. AHRQ will demonstrate in its Paperwork Reduction Act (PRA) clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and AHRQ information quality guidelines.

Work performed by AHRQ is conducted in compliance with generally accepted professional standards appropriate to the field(s) of study being undertaken, e.g., clinical, statistics, economics, or qualitative health services research. Moreover, work produced by the Agency undergoes many levels of review before being finalized and disseminated to ensure quality, i.e., utility, objectivity (in substance and presentation), and integrity. Reviews are both hierarchical, i.e., performed by higher level supervisors and managers, and by peer experts both inside and outside of the Agency. Generally, it is the responsibility of AHRQ's Center Directors to ensure the utility, objectivity, and integrity of work produced within their research Centers.

The discussion that follows outlines the general quality standards and quality assurance procedures that AHRQ has in place. More detailed information is available on AHRQ's Web site and that information will be expanded to reflect new products (e.g. new databases) or revised over time to reflect advances in generally accepted professional standards.

Statistical information and database development

Quality standards: Statistical activities are carried out in accordance with generally accepted professional standards and modern statistical theory and practice, including scientific sampling, statistical inference and analytical techniques and practices. All statistical programs employ or have access to experts in statistics and research design.

Quality assurance procedures: Each of the Agency's data products undergoes a series of standardized quality assurance procedures to assure the validity, reliability, and consistency of the data. These procedures are posted on the AHRQ Web site. For example, the Medical Expenditure Panel Survey (MEPS), which involves data collection, uses quality assurance procedures such as the validation of interviewers' work, performance of quality control checks on the source variables across all analytic groups, and benchmarking the data (e.g. comparisons with prior years and comparable data sources such as the Current Population Survey of the U.S. Census Bureau and the National Health Interview Survey). The Healthcare Cost and Utilization Project (HCUP), which aggregates and standardizes existing data from states, has quality control procedures in place to verify that the databases contain information that is valid, internally consistent, and consistent with established norms. Procedures include automated edit checks, careful review of all summary statistics, and benchmarking data with other sources (e.g. comparison with the National Hospital Discharge Survey).

Before disseminating information in any format, data are reviewed internally to ensure that they are scientifically sound and meet or exceed standards for data quality, statistical integrity, and confidentiality and privacy protections. Similarly, description and documentation of databases, tools, products, and on-line query systems are carefully reviewed, edited, and re-reviewed before release to the public.

Results of research projects

Quality standards: AHRQ staff adhere to the highest professional standards in their appropriate/respective fields, e.g., economics, social science and statistics. They seek to create transparency in their activities to enable others to fully understand their research processes, methodologies, and assumptions so that the work can be evaluated and replicated. Staff must ensure that strong measures are in place to guarantee the integrity of research information and to protect it from unauthorized access or revision.

Quality assurance procedures: All intramural research projects  — whether they involve qualitative or quantitative analysis  — and all projects on which AHRQ staff collaborate with authors outside the Agency are subject to AHRQ's intramural research planning process. This is intended to ensure conformance with strategic planning, accountability, peer review, scientific objectivity, confidentiality, and privacy standards. Research reports that are the products of intramural research undergo many levels of review prior to release, including hierarchical and peer reviews (sometimes including external reviews by other agencies and outside experts). Manuscripts submitted to scientific journals are subject to additional peer-review by the journal, prior to acceptance and publication.

In keeping with widely-accepted scientific research practices, research reports published by AHRQ describe the methods, data sources, analytical techniques, assumptions, and limitations of the research so the study can be substantially assessed and replicated.

Program Evaluations

Quality standards: Program evaluations are developed and assessed in accordance with the highest professional standards of evaluation practice.

Quality assurance procedures: Evaluation proposals are reviewed by Agency management for suitability, utility, and methodology, in accordance with sound evaluation design and standards of evaluation practice. Many evaluation projects have technical advisory committees that oversee the design and conduct of the evaluation in accordance with standard evaluation theory and practice; they often provide an expert review of the final report. Results of evaluation activities are released to the public only after Agency management has completed a review of the quality, accuracy and completeness of the report.

Software tools

Quality standards: Software tools are developed by teams of subject area experts in compliance with generally acceptable professional standards appropriate to their respective fields. Many levels of review and testing are built into the development process to ensure that the final product is useful, objective, and of high quality.

Quality assurance procedures: Reviews of tools in development are stringent. To ensure that they are needed and useful, the Agency generally relies upon peer experts and user groups both inside and outside of the Department. The Agency conducts extensive testing prior to release. Feedback is obtained from users of the tools or software to identify enhancements that will make the tools or software more fully meet user needs. As an example, the NGC is currently conducting its third annual on-line user survey to assess experiences with NGC and gather their recommendations for how to improve the system.

Consumer and professional health information

Quality standards: Consumer and professional information developed by AHRQ is conducted in compliance with generally accepted professional standards appropriate to the field of study being undertaken, e.g., clinical, economics, statistics, and social science research, as well as sound health care communication principles.

Quality assurance procedures: AHRQ works closely with stakeholder groups to determine the need for consumer and professional publications. Publications are based on information emanating from research that has been reviewed for scientific and medical accuracy and completeness by experts within and outside of the Agency before it is disseminated. Consumer publications are tested with targeted audiences to ensure relevance, clarity, and comprehension. In addition, AHRQ's publications containing health care recommendations indicate the level of substantiating evidence.

VI. Agency Administrative Complaint Procedures

AHRQ has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and AHRQ guidelines. Persons seeking correction of AHRQ-produced information should follow the procedures below.

1. Responsibility of the Complainant

   To seek a correction of information disseminated by the Agency, complainants should follow the procedures described below. Complaints or requests for review and correction of information shall be in written hard copy or electronic form. They shall be sent to the Agency by regular or electronic mail. Letters (written hard copy) should be sent to: Office of the Director
   540 Gaither Road
   Rockville, MD 20850 Electronic mail should be sent to info@ahrq.gov. Complaints should state "Information Quality Concern" in the subject line. The body of the complaint shall state that a request for correction of information is being submitted. Complaints shall contain a detailed description of the specific material that needs to be corrected including a detailed citation, e.g., the publication title, date, and publication number, if any, or the Web site and web page address (url). Complaints should describe the specific reasons for believing the information does not comply with OMB, HHS, or AHRQ guidelines and is in error and supporting documentation, if any. In addition, they should include specific recommendations for correcting the information; a description of how the person submitting the complaint is affected by the information error; and the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual or organization making the complaint. Requesters should be aware that they bear the "burden of proof" with respect to the necessity for correction as well as with respect to the type of correction they seek. In determining whether and how to address a request, the Agency will consider such factors as the Agency's budget, resources, and priorities, as well as the complexity of the correction task itself. Also, it needs to be understood that it may not be in the public interest for the Agency to devote significant resources to correcting information where the expenditure of such resources is not, in the Agency's view, cost effective in light of the significance of the asserted error, the benefits that are likely to be derived from such a correction, the costs of the correction, and the Agency's more pressing priorities and obligations.

2. Responsibility of the Agency

   Based on a review of the information provided, the Agency will determine whether a correction is warranted and if, so what action to take. The Agency will respond to the requestor by letter or e-mail. The Agency's response will explain the findings of the review and the actions that the Agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. The Agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the Agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

3. Appeals

   If the individual submitting the complaint does not agree with the Agency's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the Agency's decision. The appeal shall state the reasons why the Agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the Agency response to it, clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal to the addresses for hard copy or electronic submissions identified above. The Agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the Agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date. The Agency official who resolved the original complaint will not have responsibility for the appeal.

VII. Influential Scientific, Financial and Statistical Information

From time to time, AHRQ disseminates influential information, including statistical information, as defined in the OMB Guidelines. By "influential," OMB is referring to information "that the agency can reasonably determine...will have or does have a clear and substantial impact on important public policies or important private sector decisions." In accordance with standard Agency quality procedures, care is taken to ensure that information disseminated is substantially reproducible and replicable. This goal is accomplished by using reliable data sources and sound analytical techniques, and by employing a high degree of transparency about the data, methods, measures, assumptions and limitations used to develop the information in order to facilitate reproducibility by qualified third parties.

Because of ethical and feasibility constraints as well as confidentiality and legal obligations to third parties supplying information, there may be instances where not all of the original or supporting data may be available to the public. In such instances, AHRQ will promote understanding and transparency by clearly identifying, to the extent practicable, the resulting research methodologies and assumptions in reference documents.

D. Centers for Disease Control and Prevention and Agency for Toxic Substances and Disease Registry

I. Agency Mission

The Centers for Disease Control and Prevention (CDC) and the Agency for Toxic Substances and Disease Registry (ATSDR) are two of the operating components of the HHS. CDC has remained at the forefront of public health efforts to prevent and control infectious and chronic diseases, injuries, workplace hazards, disabilities, and environmental and occupational health threats for more than 50 years. CDC is the lead federal agency for protecting the health and safety of people — at home and abroad, providing credible information to enhance health decisions, and promoting health through strong partnerships.

CDC seeks to accomplish its mission by working with partners throughout the nation and world to monitor health, detect and investigate health problems, conduct research to enhance prevention, develop and advocate sound public health policies, implement prevention strategies and programs, promote healthy behaviors, foster safe and healthful environments, and provide leadership and training.

CDC has developed and sustained many vital partnerships with public and private entities that improve service to the American people. In FY 2000, the workforce of CDC comprised approximately 8,500 FTE in 170 disciplines with a public health focus. Although CDC's national headquarters is in Atlanta, Georgia, more than 2,000 CDC employees work at other locations nationwide including virtually all States. Approximately 160 are assigned overseas in 45 countries. In addition, CDC is comprised of 12 Centers, Institutes, and Offices (CIOs). These organizational components, listed below, respond individually in their areas of expertise and pool their resources and expertise on cross-cutting issues and specific health threats.

• National Center on Birth Defects and Developmental Disabilities
• National Center for Chronic Disease Prevention and Health Promotion
• National Center for Environmental Health
• National Center for Health Statistics
• National Center for HIV, STD, and TB Prevention
• National Center for Infectious Diseases
• National Center for Injury Prevention and Control
• National Immunization Program
• National Institute for Occupational Safety and Health
• Epidemiology Program Office
• Public Health Practice Program Office
• Office of the Director

ATSDR was established in 1980 by the Comprehensive Environmental Response, Compensation, and Liability Act, also known as Superfund. ATSDR works to prevent exposures to hazardous wastes and to environmental releases of hazardous substances. Working with States and other Federal agencies, ATSDR seeks to prevent exposure and adverse health effects associated with exposure to hazardous substances from waste sites. The agency conducts public health assessments, health studies, surveillance activities and health education training in communities around waste sites or exposed to environmental releases. ATSDR also develops toxicological profiles of hazardous chemicals found at these sites. The agency has 10 regional offices and an office in Washington, DC, and a staff of about 400 persons.

Although CDC and ATSDR are separate agencies, both strive to protect and improve the health of the American public. The Director of CDC also serves as the Administrator of ATSDR.

Unless otherwise specified, all subsequent references to CDC also include ATSDR and all practices and procedures described in this document apply to both agencies.

II. Scope and Applicability of Guidelines for CDC

CDC will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and CDC guidelines. It is CDC's policy to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination. CDC guidelines do not apply to the National Center for Health Statistics (NCHS). While NCHS is a component of CDC, NCHS is the nation's principal health statistics agency and as such has separate guidelines.

The pre-dissemination review described in the guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.

The guidelines apply to information in all media — print, electronic, audiovisual, and oral. They apply to substantive information, such as studies and reports, rather than to information pertaining to basic agency operations. Information that is disseminated at the request of CDC or with specific CDC approval through a contract, a grant, or a cooperative agreement is subject to these guidelines.

Examples are provided below of the types of information that the CDC considers within and outside the scope of the guidelines.

1. Covered Information
   • Scientific research papers, books, journal articles, reports, and similar materials, unless they have disclaimers to distinguish the research from CDC views and positions;
   • Other official reports, brochures, documents, newsletters, and audiovisual products;
   • Oral information, including speeches, interviews, expert opinions only if representing CDC's views, official positions, or policies;
   • Statistical information - statistical analyses, aggregated information by programs.
2. Information Not Covered
   • Documents not authored by CDC (either directly or by contract) and not representing official views, including research and science supported by CDC funding;
   • Opinions where the presentation makes it clear that what is being offered is personal opinion rather than fact or CDC's views;
   • Archival information disseminated by CDC (for example, Internet distribution of published articles);
   • Information dissemination limited to government employees or agency contractors or grantees;
   • Information intended solely for intra- or inter-agency use or sharing of government information, such as evaluation of a specific public health program to assess the success in achieving its objectives, technical assistance reports, training materials, manuals;
   • Information intended to be limited to public filings, subpoenas, or adjudicative processes;
   • Press releases that support the announcement or give public notice of information that CDC has disseminated elsewhere.

III. Types of Information Disseminated by NCHS to the Public

Annually, CDC produces hundreds of publications of various types and provides over 100,000 pages of Web content for access by the public. All publications that carry the CDC logo are considered official publications or releases, and must follow CDC policy and procedures for preparation, review, approval, and distribution (www.cdc.gov/od/foia/policies/clearance.htm).

Examples of the types of information disseminated by CDC to the public are listed below. Some types fit into more than one category and are mentioned in each.

1. Scientific research studies.

   CDC encourages professional dissemination of scientific research by employees and those funded by CDC to conduct research. These research studies may be published by CDC, such as the Morbidity and Mortality Weekly Report (MMWR) or non-CDC publications including journals, books, chapters, editorials, reviews, proceedings or abstracts. These are usually authored by or co-authored by CDC staff scientists as part of their official duties or may be authored by CDC partners, CDC advisory committees, or working groups convened by CDC.

2. Statistical products

   CDC releases data sets and disseminates statistical reports produced by its data collection programs. These include vital statistics, population-based health surveys, and surveys of health care providers.

3. Programmatic and administrative information.

   CDC disseminates community health assessments and information in connection with and as a byproduct of the administration of programs, such as Program-in-Brief documents, At-A-Glance documents, and program brochures.

4. Authoritative health, medical and human services information aimed at consumers and health and human services professionals.

   CDC publishes the MMWR which includes Recommendations and Reports. CDC generates Health Alerts, Public Health Advisories, and guidelines for dealing with specific public health threats. CDC also provides the website Travelers' Health, which publishes guidelines for international travelers including the "Yellow Book" and official expert opinions. CDC produces and broadcasts science educational materials and training modules, including Public Health Grand Rounds Satellite broadcasts, Web-assisted Audio Conferences for State and Local Health Policymakers, and the Health Training Network Satellite Broadcast.

5. Public heath surveillance, and epidemiology information.

   CDC publishes the MMWR Summary of Notifiable Diseases and CDC Surveillance Summaries, and other surveillance summaries on a variety of infectious diseases such as HIV/AIDS and tuberculosis, as well as other non-infectious conditions such as Birth Defects Surveillance, National Oral Health Surveillance, Pediatric Nutrition Surveillance, Pregnancy Nutrition Surveillance, Hazardous Substance Release/Health Effects Database, Flu Bulletin, Influenza Season Reports and Occupational Morbidity and Mortality Surveillance, Adult Blood Lead Epidemiology and Surveillance, Coal Workers X-ray Surveillance Program, National Surveillance System of Pneumoconiosis Mortality, National Traumatic Occupational Fatalities Surveillance System. In addition CDC publishes outbreak investigations or other items reported in the MMWR that are not authoritative or urgent. ATSDR disseminates information products including Public Health Assessments, Public Health Consultations, Fact Sheets, health study reports, Toxicological Profiles, Case Studies in Environmental Medicine, and Hazardous Substances and Public Health (newsletter).

IV. Types of Dissemination Methods

CDC disseminates information through a wide range of methods, often using more than one medium for the same information.

1. Print,
   including publications in peer-reviewed literature, published reports, periodicals, brochures, books, and correspondence;
2. Electronic,
   such as the CDC Website, CD -ROM, Listserv, e-mail, automated voice and fax systems, hotlines and clearinghouses;
3. Audiovisual,
   broadcast scripts, audio or videotapes, and videocasting. CDC's Public Health Training Network makes satellite broadcasts and Webcasts available nationally.
4. Oral,
   formal speeches, oral presentations, and interviews, or commentaries for publication or broadcast.

V. Agency Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Dissemination to the Public.

1. Overview

   CDC's policies and procedures are designed to ensure and maximize the quality of its information products with regard to their utility, objectivity, and integrity. The agency's quality assurance process begins at the inception of the information development process. CDC has guidelines to address the general principles concerning the responsibilities of the CDC staff in the collection and recording of data, publication practices, authorship determination, peer review, confidentiality of information, collaborations, and human subjects research. Authorship issues and review and clearance procedures are set forth in the "Authorship of CDC Publications and the Clearance Procedures for Scientific and Technical Documents" (www.cdc.gov/od/foia/policies/clearance.htm). CDC reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination. Further, CDC is committed to demonstrating in its Paperwork Reduction Act clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and CDC information quality guidelines. The individual CIO Associate Directors for Science (ADS) or designee are responsible for assuring the quality of information disseminated by CDC and that the quality assurance methods and procedures described in Overview of Quality Assurance Policies and Practices in HHS are met. To meet the standards for external merit review of research and scientific studies and intramural research programs, CDC policy is to peer review extramural research and intramural research studies and programs. The CIO ADS or designee are responsible for clearance of documents originating in that CIO before dissemination and for ensuring that the necessary clearances are obtained and that written material distributed is appropriate and consistent with HHS policy. While each CIO can determine preparation, review and approval procedures, all must meet standards provided by the ADS, CDC and those provided in the HHS Part I Overview D.4.d.

   • Utility CDC addresses utility, a measure of the usefulness of information products to its intended users, by staying informed of user needs through information product research and user needs assessment, user feedback, consultation with advisory committees, and conference participation.
   • Objectivity CDC provides assurance that information is accurate, reliable, and unbiased. Objectivity is achieved through existing review and clearance procedures and, in many cases, the peer review of disseminated information.
   • Integrity CDC assures the integrity of its data and information products through the enforcement of rigorous controls that protect against unauthorized access, revision, or corruption. Some of the controls used at CDC include access control, user authentication, encryption, access monitoring, provision of unalterable electronic content, and audit trails.

2. CDC Information Review and Approval Policies and Procedures by Type of Information

   1. Health and Public Health Information
      1. Scientific research studies
         CDC encourages professional dissemination of scientific research and other information by its employees. Publications or presentations by CDC employees are expected to meet high standards of quality, make a substantial contribution to the field, and contain sufficient information for the informed audience to assess its validity. Publication of scientific information by individual employees must undergo a formal review and clearance process by the CIO ADS or designee before dissemination. This review includes the evaluation of data collection measures for completeness, accuracy and timeliness, data management and analysis, clarity and accuracy of presentation, and validity of interpretation of findings. Oral presentations undergo appropriate supervisory review. Laboratory data are reviewed to assure that good laboratory data practice was followed for sampling, methodology, instrumentation and analysis. Intramural research programs will be subject to review and monitoring by external, objective peer review through an advisory committee or board of scientific counselors. Scientific research studies submitted to journals are subject to peer review of methods and findings by the journal prior to publication. ATSDR has a mandated policy for external peer review of all intramural and extramural research study protocols and findings prior to public dissemination.
      2. Authoritative health, medical and human services information aimed at consumers and health and human services professionals
         CDC disseminates authoritative health and medical information routinely as part of its mission. As an example, articles or reports for publication in the MMWR are subject to routine CDC review and approval procedures in the originating CIO. Because information disseminated in the MMWR often has impact on the practice of public health, the CDC ADS must also review and approve it. Health Alerts related to bioterrorism that are disseminated by CDC are also reviewed and approved at the CDC ADS level prior to release.
      3. Public heath surveillance and epidemiology information
         CDC often obtains surveillance information from third parties, such as States, grantees, or community-based organizations. Reliance on third parties places limits on CDC's quality assurance, although the accuracy, completeness and timeliness of the information are subject to sample audits, site visits, and an evaluation for completeness and consistency with trends and external controls. The MMWR Summary of Notifiable Diseases, for example, depends on data reported from States. CDC conducts audits and checks for consistency for trends before reporting these data. ATSDR produces Toxicological Profiles for hazardous substances found at National Priorities List sites as well as other documents that undergo public comment periods before being finalized and distributed. The Toxicological Profiles and other ATSDR documents are first produced as drafts and are then subject to public comments following announcement in the Federal Register and using other means. Only after considering the comments, the profiles and documents are finalized and then distributed to the public. ATSDR has a government to government policy on Tribal Nations that specifies how the agency works with and respects Tribal rights, sovereignty, and culture. Data or information collected from American Indian/Alaska Native communities requires approval from the Tribal government and direct involvement in the research or study from concept to completion. The Tribe reserves the right to review and critique the design and findings. Issues of release and ownership of data, information or other products must be agreed to by the Tribal government. Close collaboration and involvement of the Tribe is essential to ensure quality, utility, objectivity and integrity of information prior to being disseminated.
   2. Statistical products CDC routinely employs a number of widely accepted methods and procedures for ensuring quality, including independent assessments of statistical methodologies, peer reviews, and observance of professional standards. To insure the utility of CDC statistical and analytic information products, CDC conducts independent research and consults experts in areas such as data collection, data analysis and a variety of substantive topics and areas. Additionally, CDC maintains ongoing contact with users, and participates in conferences, and workshops in order to objectively assess and identify the current and future data needs of CDC's constituents. Further, CDC employs a wide variety of dissemination mechanisms to make its statistical and analytic information products widely available and broadly accessible. To assure that statistical and analytic information products are accurate, reliable, and unbiased, CDC obtains these data through generally accepted statistical theory and practice. Dissemination of data also follows generally recognized guidelines in terms of defining acceptable standards regarding minimum response rates, maximum standard errors, cell size suppression, quality of coding and other processing operations. CDC also maintains staff expertise in areas such as concept development, survey planning and design, data collection, data processing and editing, data analysis, evaluation procedures, and methods of data dissemination. All CDC statistical and analytic information products undergo a formal clearance process before dissemination. Publications and reports, whether in electronic or paper form, are reviewed by a CIO ADS or designee. These reviews cover the clarity of descriptive text, the appropriateness of the methodology, the soundness of the analysis, the adherence to confidentiality and disclosure avoidance restrictions, the readability of tabular and graphic presentations of data. Finally, all products undergo editorial review, (e.g., formatting, proofreading, spell checks, proper punctuation). Oral presentations undergo appropriate supervisory review. The CIO ADS or designee may also review for programmatic and policy implications on behalf of and in consultation with other division or senior staff. In addition, all public-use tapes are reviewed by the CIO ADS or designee for accuracy and appropriate confidentiality protections. CDC statistical and analytic information products are derived using generally acceptable statistical practices and methodologies which are clearly documented and available to the public. These procedures enable responsible statisticians and analysts outside of CDC to replicate CDC's statistical methods and obtain results consistent with those obtained by CDC.

VI. Agency Administrative Complaint Procedures

CDC has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and CDC guidelines.

CDC will establish a Website to advise information consumers of the agency's information quality guidelines, the process to submit a complaint, information needed by the complainant, and a description of the complaint adjudication process. CDC will centralize the initial receipt, logging, and tracking of all complaints received under this provision in the Management Analysis and Services Office (MASO), Office of Program Services. Complaints will be forwarded to the office that has subject matter responsibility for the information product in question.

1. Responsibility of the Complainant To seek a correction of information disseminated by the agency, individuals must follow the procedures described below:
   1. complaints or requests for review and correction of information must be in written (hard copy or electronic) form;
   2. requests shall be sent to CDC by mail at CDC/ATSDR, Attn: MASO, MS-E11, 1600 Clifton Road, N.E.; Atlanta, GA 30333 or by e-mail at: InfoQuality@cdc.gov; and
   3. requests shall state that an information quality request for correction is being submitted.
   
   The complaint must contain:
   4. a detailed description of the specific information that needs to be corrected including where the information is located, i.e. the publication title, date, and publication number, if any, or the Website and Web page address (url), or the speech title, presenter, date and place of delivery;
   5. the specific reasons for believing the information does not comply with OMB, HHS or CDC guidelines and is in error and supporting documentation, if any;
   6. the specific recommendations for correcting the information;
   7. a description of how the person submitting the complaint is affected by the information error; and
   8. the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.
   
   Complainants should be aware that they bear the 'burden of proof' with respect to the necessity for correction as well as with respect to the type of correction they seek.
    
2. CDC/ATSDR Responsibility CDC will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the requestor will be informed that more time is required, notified of the reason why, and provided an estimated decision date. Based on a review of the information provided, the agency will determine whether a correction is warranted and, if so, what action to take. CDC will respond to the requestor by letter or e-mail, explaining the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration of the CDC decision.
    
3. Appeals If the individual submitting the complaint does not agree with CDC's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal must state the reasons why the agency response is insufficient or inadequate. Complainants must attach a copy of their original request and the agency's response to it. Clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal by mail to CDC/ATSDR, Attn: MASO, MS-E11; 1600 Clifton Road, N.E., Atlanta, GA 30333 or by e-mail to InfoQuality@cdc.gov. The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date. The agency official who resolved the original complaint will not have responsibility for the appeal. MASO will direct all appeals to an appropriate CDC official in the Office of the Director based on the nature of the information product and complaint.

VII. Influential Scientific, Financial and Statistical Information

CDC considers the information disseminated in the MMWR Recommendations and Reports, the Hazardous Substance Release/Health Effects Database, Toxicological Profiles, ATSDR Public Health Assessments, and Federal Register publications related to science as influential scientific information.

Risk Assessment

Some of the influential information that we disseminate is based on an analysis of the risks to the public of certain actions or exposures to hazardous substances. For purposes of this guidance, we are defining risk as the likelihood that injury or damage is or can be caused by a substance, technology, or activity. We use risk analysis (the integration of risk assessment with risk management and risk communication) as a tool to enhance the scientific basis for all of our regulatory decisions.

The OMB Guidelines provide special considerations that must be taken into account in certain risk assessments, those that provide the basis for the dissemination of influential information. The guidelines state that "With regard to analysis of risks to human health, safety, and the environment maintained or disseminated by the agencies, agencies shall either adopt or adapt the quality principles applied by Congress to risk information used and disseminated pursuant to the Safe Drinking Water Act Amendments of 1996 (SDWA) (42 U.S.C. 300g-1(b)(3)(A) and (B))."

The SDWA risk assessment principles are as follows:

1. To the degree that the agency action is based on science, the agency shall use
   1. the best available, peer-reviewed science and supporting studies conducted in accordance with sound and objective scientific practices
   2. data collected by accepted methods (if reliability of the method and the nature of the decision justify use of the data)
2. In the dissemination of public information about risks, the agency shall ensure that the presentation of information about risk effects is comprehensive, informative, and understandable.
3. In a document made available to the public in support of a regulation, the agency shall specify, to the extent practicable
   1. Each population addressed by any estimate of applicable risk effects
   2. The expected risk or central estimate of risk for the specific populations affected
   3. Each appropriate upper-bound or lower-bound estimate of risk
   4. Each significant uncertainty identified in the process of the assessment of risk effects and the studies that would assist in resolving the uncertainty and
   5. Peer-reviewed studies known to the agency that support, are directly relevant to, or fail to support any estimate of risk effects and the methodology used to reconcile the inconsistencies in the scientific data

Many of our actions are based on scientific experts' judgments using available data, are essentially qualitative and do not lend themselves to the types of quantitative risk assessments contemplated by the SDWA principles. As a result, we have adapted the general principles for risk assessments from the SDWA to fit these situations.

1. The agency will use
   1. the best available science and supporting studies conducted in accordance with sound and objective scientific practices, including peer-reviewed science and supporting studies when available
   2. data collected by accepted methods (if reliability of the method and the nature of the decision justify use of the data)
2. In the dissemination of public information about risks, the agency will ensure that the presentation of information about risk effects is comprehensive, informative, and understandable.

In situations where a quantitative risk assessment is conducted, we generally follow basic risk assessment principles in the NAS paradigm of 1983. Our needs for quantitative risk assessments range over a wide variety of hazards including physical hazards encountered during exposure to toxic substances and antimicrobial resistance to antibiotic therapy. Thus, we also ascribe to the statement from NAS when it revisited the risk assessment process in 1994 (Science and Judgment in Risk Assessment, NAS 1994): "Risk assessment is not a single process, but a systematic approach to organizing and analyzing scientific knowledge and information." In each of the areas we regulate, we apply risk assessment practices to the specific task that are widely accepted among relevant domestic and international public health agencies.

For quantitative risk assessments in support of the dissemination of influential information, CDC intends to apply the following principles:

1. The agency will use
   1. the best available science and supporting studies conducted in accordance with sound and objective scientific practices, including peer-reviewed science and supporting studies when available.
   2. data collected by accepted methods (if reliability of the method and the nature of the decision justifies use of the data).
2. In the dissemination of public information about health risks, the agency shall ensure that the presentation of information is comprehensive, informative, and understandable, within the context of its intended purpose.
3. In a document made available to the public, the agency shall specify, to the extent practicable-
   1. Each population addressed by any estimate of applicable effects;
   2. The expected or central estimate of risk for the specific populations affected;
   3. Each appropriate upper-bound and/or lower-bound risk estimates;
   4. Data gaps and other significant uncertainties identified in the process of the risk assessment and the studies that would assist in reducing the uncertaintaies; and
   5. Additional studies not used in the risk assessment that support or fail to support the findings of the assessment and the rationale of why they were not used.

VIII. Special Considerations for Agency Dissemination

Special consideration also applies to information products that are urgent in nature and because of the potential risk to human health and safety, certain information products may be disseminated in an expedited manner without having fully complied with all normal quality guidelines; however, basic quality principles and processes will still apply and be followed.

National Center for Health Statistics

I. Agency Mission

The National Center for Health Statistics (NCHS), a component of the Centers for Disease Control and Prevention (CDC), is one of the federal statistical agencies belonging to the Interagency Council on Statistical Policy (ICSP). The ICSP, which is led by the Office of Management and Budget (OMB), is comprised of the heads of the nation's ten principal statistical agencies plus the heads of the statistical units of four non-statistical agencies. The ICSP coordinates statistical work across organizations, enabling the exchange of information about organization programs and activities, and provides advice and counsel to OMB on statistical activities. The statistical activities of these agencies are predominantly the collection, compilation, processing or analysis of information for statistical purposes. Within this framework, NCHS functions as the federal agency responsible for the collection and dissemination of the nation's vital and health statistics. Its mission is to provide statistical information that will guide actions and policies to improve the health of the American people.

To carry out its mission, NCHS conducts a wide range of annual, periodic, and longitudinal sample surveys and administers the national vital statistics systems. As the nation's principal health statistics agency, NCHS leads the way with accurate, relevant, and timely data. To assure the accuracy, relevance, and timeliness of its statistical products, NCHS assumes responsibility for determining sources of data, measurement methods, methods of data collection and processing while minimizing respondent burden; employing appropriate methods of analysis, and ensuring the public availability of the data and documentation of the methods used to obtain the data. Within the constraints of resource availability, NCHS continually works to improve its data systems to provide information necessary for the formulation of sound public policy. As appropriate, NCHS seeks advice on its statistical program as a whole, including the setting of statistical priorities and on the statistical methodologies it uses. NCHS strives to meet the needs for access to its data while maintaining appropriate safeguards for the confidentiality of individual responses.

II. Scope and Applicability of Guidelines

NCHS will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and NCHS guidelines. It is NCHS's policy to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

The pre-dissemination review described in the guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.

The guidelines cover all NCHS statistical and analytic information products originated by the agency, except as specified below:

• Documents not authored by NCHS (either directly or by contract) and not representing official views, including research supported by NCHS funding;
• Archival information disseminated by NCHS;
• Information dissemination limited to government employees or agency contractors or grantees;
• Information intended solely for intra- or inter-agency use or sharing of government information, such as evaluation of a specific methodological approach to assess success in achieving its objectives;
• Press releases that support the announcement or give public notice of information that NCHS has disseminated elsewhere.
• Information for public filings, subpoenas, or adjudicative processes;
• Opinions where the presentation makes it clear that what is being offered is personal opinion rather than fact or the agency's views.

III. Types of Information Disseminated by NCHS to the Public

NCHS disseminates a variety of statistical and analytic information products. These include: statistical reports on topics such as births, deaths, life tables, health status, health services utilization and the provision of health care; statistical tabulations; publications in its "Healthy People 2010" series; "Health, US," the Secretary's annual report to the Congress; and public and restricted use data files. NCHS also disseminates the results of epidemiologic, demographic and methodological research.

IV. Types of Dissemination Methods

NCHS statistical and analytic information products are distributed in one or a combination of the following media:

• Print — publications, reports, books, brochures, and pamphlets.
• Electronic — the NCHS website, CD ROMS, listservs, e-mail, automated voice and fax systems, hotlines and clearinghouses.
• Oral — speeches, presentations, commentaries for publication or broadcast, etc.
• Audio-Visual — broadcast scripts, audio or videotapes, and videocasting.

V. Agency Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Dissemination to the Public

NCHS is recognized as one of the major statistical agencies in the federal government. In developing its statistical and analytic information products, NCHS recognizes that the quality assurance process begins at the inception of the information development process routinely employs a number of widely accepted methods and procedures for ensuring quality, including independent assessments of statistical methodologies and observance of professional standards. In order to ensure the utility of its statistical and analytic information products, NCHS conducts independent research and consults with experts in areas such as data collection, data analysis and a variety of substantive topics and issues. Additionally, NCHS maintains ongoing contact with users, and participates in conferences, workshops, etc., in order to objectively assess and identify the current and future data needs of NCHS constituents. NCHS reviews the quality (including the objectivity, utility, and integrity) of information before it's disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination. Further, NCHS employs a wide variety of dissemination mechanisms to make its statistical and analytic information products widely available and broadly accessible.

In order to assure the objectivity of its statistical and analytic information products, i.e., that they are accurate, reliable, and unbiased, NCHS obtains these data through accepted statistical theory and practice. Dissemination of data also follows generally recognized guidelines in terms of defining acceptable standards regarding minimum response rates, maximum standard errors, cell size suppression, quality of coding and other processing operations. NCHS also maintains staff expertise in areas such as concept development, survey planning and design (including questionnaire development and testing), data collection, data processing and editing, data analysis, evaluation procedures, and methods of dissemination. Further, NCHS strives to demonstrate in its Paper Reduction Act (PRA) clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and NCHS information quality guidelines.

NCHS assures the security of its statistical and analytic information products through the enforcement of rigorous controls that protect against unauthorized access to the data, revision or corruption of the data, or unauthorized use of the data. Some of the major controls used at NCHS include access control, user authentication, encryption, access monitoring, provision of unalterable electronic content, and audit trails.

All NCHS statistical and analytic information products, undergo a formal clearance process before dissemination. Publications and reports, whether in electronic or paper form, are reviewed by a designated official within the author's office or division and by the Associate Director for Science (ADS). These reviews cover the clarity of descriptive text, the appropriateness of the methodology, the soundness of the analysis, the adherence to confidentiality and disclosure avoidance restrictions, the readability of tabular and graphic presentations of data, etc. Finally, all products undergo editorial review (e.g., formatting, proofreading, spell checks, proper punctuation, etc.,) In addition, all public use tapes are reviewed for accuracy and appropriate confidentiality protections. Oral presentations are subject to appropriate supervisory review.

NCHS statistical and analytic information products are derived using generally acceptable statistical practices and methodologies which are well documented and available to the public. These procedures enable responsible statisticians and analysts outside of NCHS to replicate the NCHS statistical methods and obtain results consistent with those obtained by NCHS.

VI. Agency Administrative Complaint Procedures

1. Responsibility of the Complainant
   NCHS has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and NCHS guidelines. To seek a correction of information disseminated by NCHS, individuals should follow the procedures described below.
   1. A complaint or request for review and correction of information shall be in written hard copy or electronic form;
   2. it shall be sent to NCHS by mail or electronic-mail (e-mail); and
   3. it shall state that an information quality request for correction is being submitted.
   
   The complaint shall contain
   4. a detailed description of the specific material that needs to be corrected, including where the material is located, i.e., the publication title, date, and publication number, if any, or the website and web page address (url), or the speech title, presenter, date and place of delivery;
   5. the specific reason(s) for believing the information does not comply with OMB, HHS or NCHS guidelines and is in error and supporting documentation, if any;
   6. the specific recommendations for correcting the information;
   7. a description of how the complainant is affected by the information error; and
   8. the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the complainant.
   Complainants should be aware that they bear the "burden of proof" with respect to the necessity for correction as well as with respect to the type of correction they seek. The complaint or request for review and correction of information shall be addressed to: Dr. Jennifer Madans
   Associate Director for Science, NCHS. Dr. Madans may be contacted by e-mail at JMadans@cdc.gov. Her mailing address is: National Center for Health Statistics
   6525 Belcrest Road, Room 1140
   Hyattsville, MD 20782
    
2. Responsibility of NCHS Based on a review of the information provided, NCHS will determine whether a correction is warranted and, if so, what action to take. NCHS will respond to the requester by letter or e-mail. The NCHS response will explain the findings of the review and the actions that it will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. NCHS will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, NCHS will inform the complainant that more time is required and indicate the reason(s) why. NCHS will also provide an estimated decision date.
    
3. Appeals If the complainant does not agree with the NCHS decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the NCHS decision. The appeal shall state the reason(s) why the NCHS response is insufficient or inadequate. Complainants shall attach a copy of their original request and the NCHS response to it, clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal to the NCHS appeals address, which will be included in the NCHS response. The NCHS official who reviewed the original complaint will not have responsibility for responding to the appeal. The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

VII. Influential Scientific, Financial and Statistical Information

NCHS statistical and analytic data information products are used extensively to evaluate and profile the health of the American people. Often, findings from NCHS surveys serve as benchmarks for investigating the need to modify or implement policies that will affect initiatives addressing specific health concerns of the United States population or specific health concerns of certain population sub-groups. In recognition of the potential impact some of its statistical and analytic data products may have on health policy and initiatives, NCHS requires that these products meet the highest possible quality standards. In order to accomplish this goal, quality standards are embedded throughout all survey processes and procedures, beginning with the conceptualization and design of a survey, through to data dissemination. The appropriateness of many of these standards is measured by independent assessments (e.g., peer review) conducted by qualified persons. Further, NCHS strives to assure that its processes and procedures are appropriate, well documented, and capable of being replicated by qualified persons outside of NCHS in order to derive results consistent with those obtained by NCHS. Whenever confidentiality restrictions preclude the general availability of NCHS data sets, methodologies, etc., NCHS makes every effort to provide qualified researchers limited access to confidential files through its Research Data Center, which was established specifically for that purpose.

E. Centers for Medicare & Medicaid Services

I. Agency Mission

The Centers for Medicare & Medicaid Services (CMS) administers the Medicare program, providing health care security and choice for aged and disabled people in this country. Jointly with the State governments, CMS administers the Medicaid program and the State Children's Health Insurance Program (SCHIP). CMS is the largest purchaser of health care in the United States — our programs account for more than one third of the dollars spent on health care in the U.S. economy. In 2001, CMS programs provided health care coverage for 70 million people — nearly one out of every four Americans.

In administering the Medicare, Medicaid and SCHIP, CMS activities are performed with one thing in mind: a commitment to the people who rely upon our programs for access to and quality of the health care they need. CMS' mission, though complex in execution, is simply stated: We assure health care security for beneficiaries. And our vision is no less ambitious: In the stewardship of our programs, we lead the Nation's health care system toward improved health for all.

CMS could not accomplish its mission by working alone. We work in partnership with many other organizations and individuals: other Federal and State agencies, and Territorial and Tribal governments; private-sector contractors; health plans, practitioners, and health care facilities; professional organizations, academia, and advocacy groups; and beneficiaries and their families.

II. Scope and Applicability of Guidelines

CMS will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and CMS guidelines. It is CMS's goal to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

The pre-dissemination review described in the guidelines only applies to substantive information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.
 

1. Information Covered Information that IS subject to the Information Quality Guidelines includes:
   • statistics and information designed for public dissemination to help CMS program beneficiaries make informed choices
   • statistical or actuarial information
   • studies and summaries prepared for public dissemination to inform the public about the impact of CMS programs
   • studies and summaries prepared for use in formulating broad program policy.
   
   To correct information that is subject to the Information Quality Guidelines, please follow the procedure for correcting CMS information outlined in Section VI of this notice.
    
2. Information Not Covered Information that is NOT subject to the Information Quality Guidelines includes:
   • personal information maintained for purposes of enrollment or adjudicative decisions
   • data used to verify and pay individual claims
   • information pertaining to basic agency operations, such as the Chief Financial Officer's Report
   • procedural and policy manuals
   • management information that is produced primarily for internal use
   • information which is disseminated only to CMS employees or agency contractors
   • opinions where the agency's presentation makes it clear that what is being offered is personal opinion rather than fact or the agency views
   • press releases that support the announcement or give public notice of information that CMS has disseminated elsewhere.

III. Types of Information Disseminated

Generally speaking, information released by CMS falls into one of two broad categories: Medicare consumer information, and program information and evaluation. Typically, CMS does not provide detailed consumer information about Medicaid or SCHIP because these functions are carried out by individual State governments.
 

1. Medicare consumer information This information is designed to help people enrolled in Medicare make informed decisions about the types of insurance and service providers they use. For example,
   • Medicare & You is an annual handbook mailed to Medicare enrollees. The book provides basic benefit information, and also provides information about managed care plans and other types of health insurance available to enrollees.
   • Nursing Home Compare is a web site containing information about long-term care facilities. In combination with other factors, this information can be used by people who are contemplating nursing home care for themselves or others, to decide which facility is best for them.
2. Program information and evaluation This type of information is designed to help improve the performance of CMS programs. Examples include:
   • Program information provides summary statistics describing use of CMS program services, the number of people using those services, expenditure for those services, etc. For example, CMS publishes an annual Medicare and Medicaid Statistical Supplement to the Health Care Financing Review.
   • Statistical data sets are claims-level, person-level, or provider-level data used for program evaluation, health services research, or policy development. For example, CMS prepares Standard Analytic Files (SAFs) extracting Medicare hospital inpatient claims that can be used by outside researchers who abide by a data use agreement. The agency also releases information from the Medicare Current Beneficiary Survey (MCBS), a survey of a random sample of Medicare enrollees that collects social, demographic, and economic information about those enrollees.
   • Research and evaluation reports focus on issues germane to CMS programs and policy. Typically, each innovation in CMS programs must be evaluated; often, such innovations are preceded by research performed by CMS staff or by outside researchers under contract to the agency. Many of these evaluations are released through Reports to the Congress.
   • Technical reports summarize specific aspects of CMS and its program activities. For example, each year CMS actuaries produce the data underlying the report of Medicare's trustees on the financial health of the program.
   • Payment updates establish the amount Medicare will pay for particular services or for capitated care of beneficiaries. For example, each year the agency publishes a fee schedule update that determines payments for physician services.

IV. Types of Dissemination Methods

CMS uses four modes to disseminate information.
 

1. Internet The agency maintains two web sites. One of them — www.medicare.gov — is devoted to consumer information. It is intended for use by program beneficiaries, their families and care-givers, and their advocates. The second site — www.cms.hhs.gov — is designed to serve researchers, policymakers, and health care providers. The sites are cross-linked for ease of access.
    
2. Printed matter CMS produces a wide variety of pamphlets, handbooks, and directories related to our programs. For example, CMS publishes the Health Care Financing Review, containing scholarly articles relevant to CMS programs as well as data highlights from those programs. In addition to the Medicare & You handbook described above, smaller leaflets are prepared to describe program eligibility, enrollment, and benefits.
    
3. Storage Media Data from CMS operations are available in a variety of storage media upon request for use at non-CMS computing facilities. Aside from download files available through the Internet, this form of information dissemination is ad hoc in nature.
    
4. Telephone CMS maintains a national toll-free telephone number, 1-800-MEDICARE, to process questions and provide information about the Medicare program.
    
5. Multi-media CMS is engaged in an ongoing multimedia effort to provide information about the Medicare program to its enrollees. The National Medicare Education Program (NMEP) is unique among the agency's information products in that it is intentionally designed to incorporate Internet, printed, and telephone information seamlessly.

V. Agency Quality Assurance Policies, Standards and Processes

At CMS, the quality assurance process begins at the inception of the information development process. Information released by CMS is developed from reliable data sources using accepted methods for data collection and analysis, and is based on thoroughly reviewed analyses and models. Further, CMS reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination.  The guidelines below describe procedures that CMS employs to assure the quality of its information products, including their utility, objectivity, integrity, transparency, and reproducibility.

1. Utility Utility involves the usefulness of the information to its intended users. Utility is achieved by staying informed of information needs and developing new data, models, and information products where appropriate. Based on internal analyses of information requirements, convening and attending conferences, working with advisory committees, and sponsoring outreach activities, CMS keeps abreast of information needs with respect to the analysis of CMS programs. To the extent possible, we base our analysis on CMS administrative data and surveys by other federal agencies or established survey organizations. When major needs for data related to CMS populations are identified, CMS conducts special-purpose surveys to address these needs. CMS's ongoing publication series and other information products are reviewed to ensure that they remain relevant and address current information needs. Based on internal product reviews, consultation with users, and in response to changing needs and emphases, content of ongoing information products is changed, new products are introduced and others discontinued. CMS prepares special reports and topical studies that address emerging information needs stemming from proposed changes in the law and related policy debates. Where appropriate, contact information is available on each publication (and in some cases on each table of a publication) to allow feedback and questions by users. New and revised information products are tested with focus groups of intended recipients. In many cases, the structure of the content itself is a collaborative process involving providers, consumers, academicians, and policy analysts.
    
2. Objectivity Objectivity involves a focus on ensuring that information is accurate, reliable and unbiased and that information products are presented in an accurate, clear, complete and unbiased manner. Objectivity is achieved by using reliable data sources and sound analytical techniques, and carefully reviewing information products prepared by qualified people using proven methods. Using reliable data sources. Much of the information disseminated by CMS is based on administrative data files. These files contain information used to manage CMS programs, including data to determine benefit entitlement, and to compute and pay benefits. CMS staff conduct ongoing reviews of claims information in CMS data systems to ensure its accuracy. CMS administrative data are also covered under CMS's Financial Management Systems and conform to the high standards of financial accountability demanded by these Systems. These financial management systems are mandated by the Office of Management and Budget and are designed to provide complete, reliable, consistent, timely and useful management information to enable agencies to carry out their fiduciary responsibilities. CMS-sponsored surveys are conducted using methodologies that are consistent with generally accepted professional standards for all aspects of survey development, including sample frame development, statistical design of the survey sample, questionnaire design and testing, data collection, sampling and coverage errors, nonresponse analysis, imputation of missing data, weights and variance estimates. CMS surveys follow guidelines and policies set forth in the Paperwork Reduction Act (PRA) and other regulations related to the conduct of government surveys. Through the PRA process, CMS ensures that information will be collected, maintained, and used in a way that is consistent with OMB, HHS and CMS information quality guidelines. CMS also prepares information products using data produced or maintained by other federal agencies and established survey organizations. Where possible, external data used by CMS are produced using generally accepted methodologies. Where samples from administrative data files are employed for analysis, sound statistical methods are employed to develop samples. Staff involved in producing statistical publications are knowledgeable about the content, structure and limitations of the administrative data files and maintain working relations with staff who create, update and maintain these files. Where administrative files are linked to surveys for analysis, sound procedures for extracting and linking data from external sources are employed, based on a thorough understanding of the relevant components of the data sources. Computer programs for sampling, extracting and linking external data files are developed and tested by technically qualified staff. Samples from administrative files are evaluated to ensure that samples are representative of the underlying administrative data files. Estimates prepared from external data sources are reviewed to ensure that the data extraction and linkage processes were implemented correctly. Using sound analytic techniques. Analytical reports are prepared using a variety of analytical techniques. These techniques range from simple tabulations and descriptive summary statistics to multivariate statistical methods and econometric models. Analytical techniques are reviewed for their appropriateness to the data and the analysis being conducted and are clearly identified in reports. Actuarial estimates are subject to the standards of that profession and are certified by the Chief Actuary. Preparing statistical data products. Estimates in statistical data products are prepared from representative random samples of CMS administrative data files and from reliable external data sources. Procedures for sampling from administrative files and linking external data files to administrative data files are prepared using accepted statistical methods. Output is reviewed by knowledgeable staff within the originating component and across components as appropriate. Estimates are compared to prior year estimates and estimates from other sources to ensure consistency, reasonableness and reliability. All data sources used in producing statistical data products are identified, either for the publication as a whole or for individual tables. Documentation includes specification of variables used, definitions of variables when appropriate, sampling errors and disclosure avoidance rules or techniques. Preparing analytical reports and policy studies. Information contained in analytical reports and policy studies is based on estimates derived from reliable administrative data files and external data sources. Analysts apply sound statistical and analytical techniques and are knowledgeable about the data sources and models being used. All data sources are identified. When analyses are based on simulation model projections, the assumptions used to produce the projections are also identified as well as the rationale for the assumptions used and the impact of using alternative assumptions. All analytic reports and policy studies are reviewed by technically qualified staff to ensure that analysis is valid, complete, unbiased, objective and relevant. Analytic reports and policy studies that are considered to be more technically complex are also reviewed by subject matter experts outside of the originating component to provide additional perspective and expertise. Establishing editorial review for accuracy and clarity of information in publications. All information products are edited and proofread before release to ensure clarity and coherence of the final report. Text is edited to ensure that the report is easy to read and grammatically correct, that thoughts and arguments flow logically, and that information is worded concisely and lucidly. Tables and charts are edited to ensure that they clearly and accurately illustrate and support points made in the text, and include concise but descriptive titles. Tables and charts clearly indicate the unit of measure and the universe being examined and all internal labels (column heads, row stubs, and panel headings) accurately describe the information they contain. All changes made to a manuscript during the editing process are checked by a proofreader and reviewed and approved by the author.
    
3. Integrity Integrity refers to the security of information from unauthorized access or revision to ensure that the information is not compromised through corruption or falsification. CMS has in place rigorous controls to ensure the integrity of its administrative information. CMS is highly protective of the confidentiality of information it holds through its policies and practices. CMS administers social insurance programs that — directly or indirectly — touch the lives of almost every American. CMS has in place programs and policies for securing CMS resources as required by the Government Information Security Reform Act (P.L.106-398, Title X, Subtitle G). These security procedures address all major components of information security. CMS is subject to a number of statutory requirements that protect the sensitive information it gathers and maintains on individuals. Among these are:
   • Health Insurance Portability and Accountability Act of 1996
   • Privacy Act of 1974
   • Computer Security Act of 1987
   • Office of Management and Budget (OMB) Circulars A-123, A-127, and A-130
   • Government Information Security Reform Act
   • Federal Managers' Financial Integrity Act (FMFIA) of 1982
4. Transparency and Reproducibility If an agency is responsible for disseminating "influential" information, guidelines for dissemination should include a high degree of transparency about data and methods to facilitate its reproducibility by qualified third parties. Information is considered influential if it will have a substantial impact on important public policies or important private sector decisions. CMS's guidelines call for identification and documentation of data sets used in producing estimates and projections, and for clear descriptions of the methods used. Many estimates and projections included in CMS information products are not directly reproducible by the public because the underlying data sets used to produce them are confidential. However, some statistical publications that are based on publicly available data and whose programs are made available on request are fully reproducible by the public. Further, some CMS data is available to people outside the agency through adherence to a data use agreement, provided that the use of the data is consistent with the appropriate systems of records notice. Where estimates and projections may not be easily reproduced by third parties due to the complexity and detail of the methods and data, greater emphasis is placed on periodic review by outside panels of technical experts. CMS also achieves transparency through wide dissemination of its information. Many reports and other data products are available both as printed and electronic documents. New documents posted on our Web sites are section 508 compliant, making information available to an audience that includes persons who have a visual impairment and read online using assistive technology. (Older documents are being converted to meet section 508 standards.)

VI. Agency Administrative Complaint Procedures

CMS has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and CMS guidelines.

Existing public comment procedures for rule-makings and other formal agency actions already provide well established procedural safeguards that allow affected persons to raise information quality issues on a timely basis. Accordingly, CMS will use these existing procedures to respond to information quality complaints that arise in this process.

In cases where agency disseminates a study, analysis, or other information prior to the final agency action or information product, requests for correction will be considered prior to the final agency action or information product in those cases where in the agency's judgment issuing an earlier response would not unduly delay issuance of the agency action or information product and the complainant has shown a reasonable likelihood of suffering actual harm from the agency's dissemination if the agency does not resolve the complaint prior to the final agency action or information product.
 

1. Responsibility of the Complainant To seek a correction of information disseminated by the agency, individuals shall follow the procedures described below.
   1. A complaint or request for review and correction of information shall be in written hard copy or electronic form;
   2. it shall be sent to the agency by mail or electronic-mail (e-mail) at the address below; and
   3. it shall state that an information quality request for correction is being submitted.
   
   The complaint shall contain
   4. a detailed description of the specific material that needs to be corrected including where the material is located, i.e. the publication title, date, and publication number, if any, or the web site and web page address (url), or the speech title, presenter, date and place of delivery;
   5. the specific reasons for believing the information does not comply with OMB, HHS, or CMS guidelines and is in error and supporting documentation, if any;
   6. the specific recommendations for correcting the information;
   7. a description of how the person submitting the complaint is affected by the information error; and
   8. the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.
   
   Complainants should be aware that they bear the "burden of proof" with respect to the necessity for correction as well as with respect to the type of correction they seek.
    
2. Responsibility of the Agency Based on a review of the information provided, the agency will determine whether a correction is warranted and if, so what action to take. The agency will respond to the requestor by letter or e-mail. The agency's response will explain the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. The agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.
    
3. Appeals If the individual submitting the complaint does not agree with the agency's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal shall state the reasons why the agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the agency response to it, clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal to the specific agency appeals address. The agency official who resolved the original complaint will not have responsibility for the appeal. The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.
    
4. Contact Complaints filed under this guideline shall be mailed to:
   • Information Quality
     Centers for Medicare & Medicaid Services
     7500 Security Boulevard
     Baltimore, MD 21244-1850
   
   Alternatively, the complaint can be emailed to infoquality@cms.hhs.gov Appeals filed under this guideline should be mailed or emailed to the same address, and should contain the word "appeal" on the envelope or in the subject line.

VII. Influential Scientific, Financial, and Statistical Information

The OMB Information Quality Guidelines require that "influential" scientific, financial, or statistical information in official Government documents must be based on data or studies that could be substantially reproduced if the original or supporting data were to be independently reanalyzed using the same methods. "Influential" means that CMS can reasonably determine that dissemination of the information will have a substantial impact on important public policies or important private sector decisions or will have important consequences for specific health practices, technologies, substances, produces, or firms. Examples of the types of information disseminated by CMS that have the potential to be influential include the annual reports of the Medicare Board of Trustees and annual publication of provider payment rates.

CMS is committed to applying rigorous scientific standards to ensure the accuracy and reliability of program evaluation results. The scientific/research, financial, and statistical community recognizes peer review as the primary means of quality control. CMS routinely seeks input from qualified peer reviewers, inside and outside the Federal government prior to dissemination of this type of information.

F. Food and Drug Administration

This guidance provides the information quality guidelines for the Food and Drug Administration (FDA) requested under the Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies (OMB Guideline). ¹ The OMB Guidelines implement section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Pub. L. 106-554; H.R. 5658). The OMB Guidelines direct agencies to issue their own information quality guidelines ensuring and maximizing the quality, objectivity, utility, and integrity of information, including statistical information, disseminated by the agency. This guidance is part of the U.S. Department of Health and Human Services (HHS) implementation plan developed to comply with the OMB Guidelines. The principles described in the HHS plan and OMB guidance are incorporated by reference into this guidance.

This guidance describes the nature of the information disseminated by the FDA and explains FDA's standards, policies, and procedures for ensuring the quality of the information it disseminates. The guidance also explains the administrative mechanisms that are in place to enable persons to seek and obtain correction of information maintained and disseminated by the FDA that they believe does not comply with the OMB and HHS Guidelines and this guidance.

I. Agency Mission

The Agency's mission, as defined in Section 406 of the Food and Drug Administration (FDA) Modernization Act of 1997, is as follows:

The Administration shall --

1. promote the public health by promptly and efficiently reviewing clinical research and taking appropriate action on the marketing of regulated products in a timely manner;
    
2. with respect to such products, protect the public health by ensuring that --
   • 1. foods are safe, wholesome, sanitary, and properly labeled;
     2. human and veterinary drugs are safe and effective;
     3. there is reasonable assurance of the safety and effectiveness of devices intended for human use;
     4. cosmetics are safe and properly labeled;
     5. public health and safety are protected from electronic product radiation; and
     6. when the Family Smoking Prevention and Tobacco Control Act became law on June 22, 2009, it gave FDA the authority to regulate tobacco products.
3. participate through appropriate processes with representatives of other countries to reduce the burden of regulation, harmonize regulatory requirements, and achieve appropriate reciprocal arrangements; and
    
4. as determined to be appropriate by the Secretary, carry out paragraphs (1) through (3) in consultation with experts in science, medicine, and public health, and in cooperation with consumers, users, manufacturers, importers, packers, distributors, and retailers of regulated products.

II. Scope and Applicability of Guidelines

FDA is a scientific regulatory agency that regulates one trillion dollars worth of consumer goods each year. This amounts to more than 20 percent of all consumer spending. FDA regulates most food that we eat, all prescription drug and over-the-counter drug products that we take, and all medical devices that we use. For foods, we ensure that products are safe, wholesome, sanitary, and properly labeled. For drugs, we ensure the products are both safe and effective for use; and for medical devices, there is a reasonable assurance of their safety and effectiveness. We also are responsible for ensuring that electronic and radiation-emitting equipment, such as x-ray machines, microwave ovens, and metal detectors, are safe for use. We certify and inspect annually all mammography facilities. We regulate animal feed and most animal drugs. We ensure that cosmetics are labeled honestly and cause no harm. Our regulatory activities include inspection and surveillance of marketed products, standard setting, bioresearch monitoring, and human subject protection. We also conduct research in support of our regulatory programs.

FDA, which employs about 10,000 people, is organized primarily by centers:

• Center for Biologics Evaluation and Research (CBER)
• Center for Devices and Radiological Health (CDRH)
• Center for Drug Evaluation and Research (CDER)
• Center for Food Safety and Applied Nutrition (CFSAN)
• Center for Veterinary Medicine (CVM)
• National Center for Toxicological Research (NCTR)
• Office of the Commissioner (OC)
• Office of Regulatory Affairs (ORA)
• Center for Tobacco Products (CTP)

Our Office of the Ombudsman, which is responsible for monitoring complaints regarding information dissemination and our response to those complaints, resides in the Office of the Commissioner.

As described in detail in the paragraphs that follow, we disseminate many different types of information to a wide variety of audiences, including the regulated industry, health care professionals and organizations, consumers, patients, other governmental agencies, and international organizations and agencies. Information dissemination is an important part of our mission to promote and protect the public health. FDA recognizes that public access to high quality information is critical to achieving this mission and public input, in turn, improves the quality of the information we disseminate.

Because of the nature of this information, our goal has been and remains to ensure that all the information we disseminate meets the high standards of quality (including objectivity, utility, and integrity) described in the OMB and HHS Guidelines. As discussed in detail in Section V, we have established several policies, standards, and processes to ensure the quality of the information we make available to the public. FDA also will ensure that information collected under the Paperwork Reduction Act of 1995, 44 U.S.C. chapter 35, will be collected, maintained, and used in a manner consistent with the OMB and HHS Guidelines and this guidance document. With the following specific exceptions, the OMB Guidelines apply to most categories of FDA-disseminated information (see discussion in following sections):

1. Documents relating to internal FDA procedures
2. Internal government correspondence
3. Correspondence with individuals that is not normally made public
4. Press releases (unless they contain new substantive information not covered by previous information dissemination)
5. Archival records
6. Distributions intended to be limited to subpoenas or adjudicative documents including findings and determinations made in the course of an administrative proceeding pursuant to 21 CFR Parts 10, 12, 13, 14, 15, 16, 17 and 19.
7. Scientific publications that only contain the views of the authors and are not used to support an official agency position
8. Responses to requests for information under the Freedom of Information Act (FOIA), the Privacy Act, the Federal Advisory Committee Act, or other similar laws.

In the pages that follow, we describe the types of information we disseminate (Section III), the methods we use to disseminate this information (Section IV), and the quality assurance policies, standards, and processes that have been put in place to ensure the quality of the information we distribute (Section V). Some of the information described below may include information that falls under one of the types of information specifically excluded above. To the extent that information in one of the categories listed below includes information listed in one of the exceptions, the OMB Guidelines do not apply. In Section VI, we discuss the procedures to be used by persons who want to request that information we have disseminated be corrected. Section VII discusses the types of information we have identified to be influential according to the OMB Guidelines and the principles we apply to information that will be disseminated regarding risks to human health, safety, and the environment. The principles that are used have been adapted from the quality principles applied by Congress to risk information pursuant to the Safe Drinking Water Act Amendments of 1996 (42 U.S.C. 300g-1(b)(3)(A) and (B)). Section VIII describes some special circumstances that may apply to certain information dissemination activities.

The pre-dissemination review described in the HHS guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.

III. Types of Information Disseminated

We make a large number of documents and information available to a variety of audiences. The major types, with examples of each, are provided here.

1. Public Communications About Risk

   As part of our mission to protect the public health and safety, we provide the public with a wide variety of information on risk, including information on food safety and the risks involved with using medical products. Some examples of our communications are listed here.

   • Consumer advice and fact sheets (for example, News Release No. 0020.01, January 2001 provides advice to consumers to reduce the risk of illness from foodborne Listeria monocytogenes)
   • "Dear Health Care Professional" letters (for example, Agency August 2001 letter to health care professionals warning about rhabdomyolsis, a serious, potentially fatal adverse effect of all statin drugs)
   • Public health and safety alerts (for example, "FDA Announces New Alcohol Warnings for Pain Relievers and Fever Reducers," October 1998; "Seniors and Food Safety -- Preventing Foodborne Illness," May 1999; "Learning About Lasik Eye Surgery," on the Internet; "FDA Patient Safety News," a video targeting hospitals; "Tips and Warnings for Consumers About Buying Medicines and Medical Products Online," on the Internet; "An Important Message for Pregnant Women and Women of Childbearing Age About the Risk of Mercury in Fish," March 2001; "FDA Public Health Notification: Reducing Radiation Risk from Computed Tomography for Pediatric and Small Adult Patients," November 2001)
   • FDA Talk Papers (for example, FDA Announces Availability of Vibrioparahaemolyticus Risk Assessment, January 18, 2001)
   • Various subject matter brochures intended for consumers. Some are produced as low-literacy brochures aimed at consumers with no more than a fifth-grade reading level. Some are also produced in other languages, particularly in Spanish. Some are mailed directly to specific audiences; some are disseminated at large professional meetings (for example, "Reprocessing of Single-Use Medical Devices by Hospitals" (November 2000), "Buying Contact Lenses on the Internet, by Phone, or by Mail" (August 2001)).
   • Posters are distributed to health clinics and schools with consumer information on FDA-related health issues.

2. Rulemaking Documents

   Like many Federal agencies, we engage in rulemaking. This process includes publishing a proposed rule and explanatory material in the Federal Register, obtaining public comment, and publishing a final rule and response to the comments. Some examples include:

   • Additional Criteria for Classifying Over the Counter Drugs as Generally Recognized as Safe and Effective and Not Misbranded, a final rule that published on January 23, 2002
   • Medical Devices; Device Tracking, a final rule that published on February 8, 2002
   • State Certification of Mammography Facilities, a final rule that published on February 6, 2002
   • Implantation or Injectable Dosage Form New Animal Drugs: Trenbolone and Estradiol; a final rule that published on February 7, 2002
   • Foreign Establishment Registration and Listing, a final rule that published on November 27, 2001
   • Substances Affirmed as Generally Recognized as Safe: Menhaden Oil; a proposed rule that published on February 26, 2002
   • Hazard Analysis and Critical Control Point (HACCP): Procedures for the Safe and Sanitary Processing and Importing of Juice; a final rule that published on January 19, 2001

   As part of the rulemaking process, we may also publish advanced notice of proposed rulemaking documents (ANPRs) and direct final rules.

3. Product Approvals

   1. Medical Products

      When we evaluate applications for approval to market medical products, we produce reviews of the data collected, analyzed, and submitted by applicants. On approval of drugs², we compile our reviews into an approval package that provides the basis for the Agency clearance of a decision to approve a product. To provide as much information as possible to health care practitioners and consumers so they can make informed decisions about treatment, we make the medical product approval packages, including generic drug approvals, available on the Internet. An approval package can range from 100 to more than 1,000 pages, redacted to remove confidential and trade secret information. Contents usually include individual discipline reviews; correspondence between the company and FDA; administrative documents; and labeling. For example, the package for Clarinex [desloratadine], an antihistamine to treat seasonal allergic rhinitis, was approved on December 21, 2001. We posted the 500-page package on the Internet on February 12, 2002.

      We also post on the Internet, for animal drug products, an FOI summary of the approval and for device premarket approvals, a detailed summary of safety and effectiveness, the approval order, and the draft labeling.

   2. Food and Color Additives

      When we evaluate applications for premarket approval of food additives and color additives, we produce reviews of the data and analyses submitted by the applicant. For direct food additives and color additives, we publish a final rule in the Federal Register explaining the basis for Agency approval of the product and issuance of a regulation in the U.S. Code of Federal Regulations. For other food additives, we announce their approval via Internet listings that are updated at least monthly. In all cases, the published listings include specifications and use limitations necessary to ensure the safe use of the product. In addition, the documents and information that form the basis for Agency approvals are available under the Freedom of Information Act.

4. Guidance and Regulatory Assistance

   We develop guidance and other policy documents usually with input from the public, to assist industry, consumers, hospitals, reviewers, and other health care related organizations and individuals interested in our statutes and regulations. In addition, we provide procedural guidance to our field offices. In compliance with our policy involving good guidance practices, under 21 CFR 10.115, we publish notices announcing the availability of guidances in the Federal Register, and make the guidances available in the public docket and on the Internet. Some examples of guidances are provided here:

   • Guidance to Hospitals, Nursing Homes, and Other Health Care Organizations -- FDA Public Advisory on the risks of death and injury related to medical gas mix-ups (April 2001)
   • Small Business Compliance Guides -- guidances to help small businesses; required by the Regulatory Flexibility Act (5 U.S.C. 602) for all rulemakings that will have a significant impact on small entities (for example, Sterility Requirement for Aqueous-Based Drug Products for Oral Inhalation -- Small Entity Compliance Guide, November 2001)
   • Formal Dispute Resolution: Appeals Above the Division Level (March 2001)
   • Food Security Guidance (January 2002)
   • Food Code; 2001 Revision (December 2001)
   • Guidance for Industry: Studies to Evaluate the Safety of Residues of Veterinary Drugs in Human Food: Genotoxicity Testing, VICH GL23
   • Guidance on Enforcement Priorities for Single-Use Devices Reprocessed by Third Parties and Hospitals (August 2000)
   • The Policy Guidance Help System (January 2000; revised November 2001) -- a computerized system containing all Mammography Quality Standards Act regulations and final guidance (Internet and as a stand-alone downloadable program)
   • CDRH develops "Plain Talk" guidance on how to comply with our regulations and provides a unique interactive Website called "Device Advice" to answer specific device related questions.
   • Compliance Policy Guides are issued to the field offices to ensure that our regulations and policies are followed consistently. These Compliance Policy Guides are made public and treated as guidances.
   • Regulatory Procedures Manual provides FDA procedural guidance and instruction for use by FDA and the public. The manual is publicly available and is treated as guidance.

5. Reports

   We also develop reports on a variety of topics. Some examples include:

   • Managing the Risks From Medical Product Use -- Report to the FDA Commissioner from the Task Force on Risk Management (May 1999)
   • Prescription Drug Marketing Act of 1987 -- Report to Congress (June 2001)
   • FDA Fiscal Year 2002 Congressional Budget Request (annual report)
   • Import Detentions Reports (IDRs) -- IDRs provide information on the products detained by the Agency (that is, products for which our District Offices have issued a "Notice of Detention and Hearing"). The IDR is generated from data collected by FDA's Operational and Administrative System for Import Support (OASIS) and is updated monthly and is posted on the FDA website for a period of time (on the average of 12 months). Archived data are not available on the Internet but is available under the Freedom of Information Act.
   • FDA Enforcement Report -- Published weekly, this online publication contains information on recalls and other actions taken in connection with Agency regulatory activities.
   • Mammography Facility Adverse Event Report -- an annual report of adverse actions taken against mammography facilities issued to help health professionals and consumers in evaluating the performance of their mammography facilities

6. Citizen Petitions and Responses

   When citizens petition the FDA to address an issue, we write a response to the petitioner explaining our position. Although these responses are letters addressed to individuals or organizations, the petitions and our responses are made available through the public docket and often on the Internet. In recent years, we have responded to an average of 250 citizen petitions per year. For example, on February 15, 2002, we issued a response to a petitioner asking us to refrain from approving a generic version of an antibiotic (Ceftin) if the generic drug product's active ingredient were wholly or partially in crystalline form.

7. Press Items and Publications

   The Agency releases much information through the press and related media. Some examples include:

   • News releases (for example, HHS and USDA Release Listeria monocytogenes Risk Assessment and Listeria monocytogenes Action Plan, Release No. 0020.01, January 18, 2001; Cape May Foods Recalls Chopped Clams Because of Possible Health Risk, January 15, 2002)
   • FDA Consumer Magazine (bimonthly magazine targeting consumers) and reprints of selected feature articles from FDA Consumer.
   • FDA Veterinarian (bimonthly newsletter targeting veterinarians and the food animal industry)
   • Frequently Asked Questions: What Can I Do to Protect Myself from Food Poisoning? (Internet only)

IV. Types of Dissemination Methods

Transparency is one of the Agency's key goals. It is critical that our audience understand what we do, how we do what we do, and why we do something. Because our audience is so diverse, we use a variety of media to disseminate public health and safety information. Some examples are provided here:

• Oral Presentations in public forums sponsored by FDA or outside parties, such as professional societies or trade associations (for example, the FDA Science Forum)
• Internet (medical product approval packages; device summaries of safety and effectiveness; safety alerts; guidance documents; special issue papers, such as those on "Online Medicine Sales" and the "Agency's Bioterrorism Activities")
• Federal Register (proposed and final rules; notices announcing the availability of guidances; meeting notices; other notices)
• The public docket (citizen petitions and responses; transcripts of certain meetings; information about advisory committee meetings; public comments on guidances, regulations, and any other documents that publish in the Federal Register)
• Videos (on the Internet and for dissemination to outside organizations and use in meetings and conferences. For example, "Science and our Food Supply: Investigating Food Safety from Farm to Table")
• CD ROM (for example, Listeria monocytogenes draft risk assessment documents and models)
• FDA's E-mail lists (see topics at www.fda.gov/emaillist.html)

V. Agency Quality Assurance Policies, Standards, and Processes

As described in the HHS Implementation Plan, as one of the HHS operating divisions, we have established a number of quality assurance policies, standards, and processes for ensuring the quality of the information we disseminate to the public. Our documents undergo a rigorous review and clearance evaluation according to pre-established procedures, documented in our regulations and guidances.

Generally, Agency documents are cleared as follows:

• Document is developed by an individual or team
• Document is circulated to working group members, and often an editor, and comments are incorporated
• Document is circulated and cleared by center managers
• We publish many documents (for example, guidances, proposed rules) for comment by members of the public, and some documents are reviewed by outside advisory committees comprising experts in the subject matter of the document
• If required by regulation or policy, documents are circulated to and cleared by the Office of the Chief Counsel, Office of Policy, Planning, and Legislation, the Department (HHS), and the Office of Management and Budget (OMB).

In addition to these clearance procedures, we use a number of mechanisms to ensure the quality of the information we disseminate. FDA reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance, and dissemination. Quality, as defined in the OMB Guidelines, encompasses (1) utility, the usefulness of the information to its intended users, including the public; (2) objectivity, whether information is being presented in an accurate, clear, complete, and unbiased manner; and (3) integrity, the information is protected from unauthorized access or revision.

1. Utility

   We only disseminate information that we believe will be useful to the public or a segment of the public. In fact, often we disseminate information because members of the public or the regulated industry have requested it. We develop many guidances as a result of public questions about a specific topic. We also have processes (21 CFR 10.30) by which members of the public can petition us to take certain actions, such as initiating rulemaking or taking specific administrative or enforcement actions. Requests for dissemination of information also can be submitted through petitions.

   We developed our good guidance practice (GGP) policy as a result of public request. Congress later enacted the policy into law, and we codified our GGP policy in our regulations at 21 CFR 10.115. The GGPs describe our procedures for developing, issuing, and using guidance documents and include detailed procedures on how members of the public can suggest areas for guidance development, submit drafts of proposed guidance documents, and request the revision or withdrawal of an existing guidance document. We also maintain a guidance Agenda, which is a list of guidances we are planning to develop in the coming year. We post the list on the Internet and publish it annually in the Federal Register. We publish the Agenda to keep the public up-to-date on guidance development plans and solicit input from the public on what guidances are needed.

   In addition, we are subject to the Freedom of Information Act and the Electronic Freedom of Information Act Amendments (5 U.S.C. 552), which provide for the dissemination of information to members of the public and posting on the Internet certain information that is, or is likely to be, responsive to multiple information requests.

   In accordance with the Regulatory Flexibility Act (5 U.S.C. 602), the General Services Administration publishes a semiannual regulatory agenda describing the regulatory actions being developed. The Secretary welcomes comments on this agenda and suggestions for improvements and initiatives.

2. Objectivity

   As already mentioned, we have many different systems in place to ensure that the information we disseminate is presented in an accurate, clear, and unbiased manner. We have a strong commitment to writing all our new documents in plain English. We have provided plain English training to many of the employees who write our documents. We also continue to solicit feedback from stakeholders on our efforts to present written information clearly.

   We also take steps to ensure that our regulatory decisions are based on objective information. If we rely on information submitted to us by third parties in support of an application for product approval or in a rulemaking proceeding, we make sure that this information meets the appropriate standards for quality and objectivity. We have a number of regulations and guidances that set standards for the generation of information in support of regulatory decisions. For example, the supporting data may be generated in new research using good laboratory practices (GLPs) ³, in clinical studies subject to Good Clinical Practices (GCPs) ⁴, in reviews of existing information obtained primarily from peer-reviewed scientific literature, or obtained from surveys based on widely accepted scientific survey techniques. Interpretations of quantitative results of Agency studies are commonly subjected to statistical analyses. Even though we may be unable to fully disclose the information on which our actions are based because it has been submitted by a third party with a proprietary interest in maintaining its confidentiality, we make special efforts to ensure that the information submitted meets these standards for objectivity and is of high quality.

   The methods by which we ensure the objectivity of the information for some of our major regulatory activities are described here.

   1. Product Review Activities

      One key FDA responsibility is the evaluation of data submitted to the Agency in medical and veterinary medical product applications and in food and color additive petitions or notifications. In general, firms that want to market certain products (for example, drugs and medical devices) submit applications to FDA. These applications contain data or information on which the firm relies to claim that its product is safe and effective for its intended uses. We base our decisions about safety and effectiveness primarily on our analyses of the integrity of the submitted data. When we approve a product and post a drug review package on the Internet, we are ensuring that our analyses of the data submitted to us are available for public scrutiny.

      We develop regulations and guidance documents to help ensure that the data submitted to us result from the best available studies, that the studies are conducted in accordance with sound and objective scientific practices, and that the data are collected using scientifically accepted methods. For example, FDA regulations specify the format and content of the clinical studies that are submitted in support of an application to market a new drug product. They specify how the data are to be collected and the types of analyses that are to be performed. In the case of biological products, we have developed guidance on the format and content of reports on clinical studies that are submitted to the Agency. Other FDA guidances provide detailed descriptions of appropriate methodologies, analyses, and procedures.

      Since the early 1990s, we have been involved in an intensive international effort to harmonize technical requirements for the conduct of studies in support of marketing applications and the content and format of applications with the goal of allowing the submission of a common application for marketing around the world. The International Conference on Harmonisation for Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) brings together scientific experts from different countries to develop a consensus on the appropriate requirements. We also are engaged in international activities in the device, food, and animal drug areas. For example, International Cooperation on Harmonisation of Technical Requirements for Registration of Veterinary Products (VICH) is the veterinary counterpart to ICH. The Global Harmonization Task Force (GHTF) is working to harmonize device regulations and guidance. The Codex Alimentarius committees are working to harmonize international food regulations. Many agreements reached are then embodied in regulations issued through notice-and-comment rulemaking and in guidance documents that describe in more detail appropriate ways to comply with the regulations. As a result of these efforts, most of our actions on product approval applications are consistent with international standards for data collection and quality of analysis.

      We also are in the process of developing good review practices (GRPs) for drug reviews with the goal of making our drug product review process consistent across all divisions in the Center for Drug Evaluation and Research. A major emphasis of the GRP project is to ensure that the reviews we make available to the public are consistently formatted and clearly written so interested individuals can access important health and safety information.

      We frequently consult with scientific experts on product approval applications and broader issues. We have 31 standing Advisory Committees ⁵, whom we routinely consult on whether the data in particular applications are sufficient to support an approval decision. As noted above, we incorporate our approval decisions into drug approval packages and device summaries of safety and effectiveness that contain our analyses of the submitted data. These packages and summaries do not include confidential commercial, trade secret, and other information exempt from disclosure when we place them on the Internet.

   2. Food Safety Activities

      One of FDA's major areas of responsibility is ensuring the safety of the food we eat. Food safety activities include research, risk assessment, inspections, surveillance, compliance, education, and system coordination activities. We must make sure that the information we provide on food safety is presented in an accurate, clear, complete and unbiased manner. This means that the data on which we base our decisions must be collected in an objective manner using sound scientific principles for data collection.

      We collect information to support our food safety activities through many sources, including research, risk assessment, inspection and surveillance, peer-reviewed literature, and advisory committee opinions. We conduct in-house research on a variety of food safety topics and also fund a substantial amount of extramural research every year. The Center for Food Safety and Applied Nutrition (CFSAN) participates in collaborative research on processing and packaging through the National Center for Food Safety and Technology, which is a consortium of government, industry, and academia, and we coordinate food safety research activities through a cooperative program with the University of Maryland. Topics of both internal and external research interest are determined by CFSAN's 3-year research plan. This plan is developed in conjunction with other federal agencies to prioritize our research to inform our most critical food safety efforts and to avoid duplication of effort.

      We also gather information for certain food safety activities through risk assessment. Risk assessments are a very useful tool for evaluating the benefits of pursuing various rulemaking strategies. To date, we have conducted or been involved in four risk assessments related to food safety. We conducted a joint risk assessment with USDA on Salmonella Enteritidis in shell eggs that was published in 1998. More recently, we published risk assessments on the Public Health Impact of Vibrio parahaemolyticus in Raw Molluscan Shellfish and the Relative Risk to Public Health from Foodborne Listeria monocytogenes Among Selected Ready-to-Eat Foods. We also intend to use the risk assessment model developed for USDA by Harvard University on bovine spongiform encephalopathy to determine the risk reduction outcomes of various rulemaking efforts we are considering.

      We gather data for FDA food safety activities through use of surveys designed for specific purposes and advisory committee opinions. The surveys include the Health and Diet Survey and the Total Diet Study, both of which are used in our development of safety and exposure assessments for various compounds. When information is not available through research or literature, we have several advisory committees that are able to render expert opinions on particular matters. These committees include the National Advisory Committee on Microbiological Criteria for Foods, which considers a variety of food safety issues for FDA and USDA, and FDA's Transmissible Spongiform Encephalopathy Advisory Committee (TSEAC), which specifically considers issues related to TSE diseases. Finally, our food safety activities are informed through the participation of FDA scientists in a variety of professional organizations such as Codex Alimentarius, International Commission on Microbiological Specifications for Foods, Institute of Food Technologists, American Society for Microbiology, the International Association for Food Protection, Society for Toxicology, American Chemical Society, the National Academy of Sciences, International Life Sciences Institute, and editorial review boards of several publications including Journal of Food Protection and Journal of Food Science.

   3. Adverse Events Analysis for Medical Products

      Once products are marketed, we continue to monitor their safety after approval and disseminate information about their risks to health care providers, patients, and consumers. We undertake a number of data collection activities to ensure the objectivity of the information we disseminate on medical products.

      1. Human Drug and Biological Products

         The Adverse Event Reporting System (AERS) is an Oracle-based computerized information system designed to support the Agency's postmarketing safety surveillance program for all approved drug and therapeutic biologic products. The structure of the database complies with the international safety reporting guidance (E2B Guidance on Data Elements for Transmission of Individual Case Safety Reports, January 1998 ⁶), including content and format for electronic submission of the reports from the manufacturers. The ultimate goal of AERS is to help reduce the risks associated with medical product use by providing the best available tools for storing and analyzing safety reports. Information from this system is used to support decisions to disseminate information of product safety. By systematizing the submission of data to the Agency, we have greatly improved the quality (and objectivity) of related decisions and information dissemination efforts.

         A separate system is used to monitor the safety of vaccines after approval. The Vaccine Adverse Events Reporting System (VAERS) is a cooperative program for vaccine safety of the FDA and the Centers for Disease Control and Prevention (CDC). VAERS is a postmarketing safety surveillance program, collecting information about adverse events that occur after the administration of U.S. licensed vaccines. Other systems are in place to monitor the quality of manufacturing of drugs and biological products and blood related products.

      2. Medical Devices

         We use a variety of tools to identify problems and safety issues related to medical devices that are approved and being used by health care practitioners and consumers. Tools include both voluntary and mandatory reporting of adverse events; monitoring of product performance through other data sources, such as registries and various research efforts; and the use of both mandatory and voluntary postmarket studies aimed at examining specific safety issues.

         Adverse events related to either product problems or issues associated with the use of the device are reported by both manufacturers and device users. Although both the manufacturing and user communities have mandatory reporting requirements for device-related problems, this surveillance system is virtually a passive system that depends on the reporter to recognize an event and follow through in reporting. This passive surveillance system is augmented by a more active reporting network composed of hospitals and other health care facilities, where reporting is encouraged and supported through educational activities and feedback. Adverse event reports are immediately triaged to quickly identify problems that require urgent attention; all reports are then reviewed by clinical analysts and others with appropriate expertise to decide if further follow-up is needed.

      3. Animal Drugs

         An Adverse Drug Event (ADE) report for veterinary medicinal products consists of either an undesired side effect or the lack of a desired effect associated with drugs administered to animals. Reports may also involve product defects and potential harm posed to persons administering or using animal drugs. For example, in the year 2000, FDA's Center forVeterinary Medicine reviewed 14,497 ADE reports consisting of: 13,757 undesired side effects and lack of desired effect; and 740 product defects.

         Adverse event reports related to animal drugs are maintained in a separate database, the Center for Veterinary Medicine's adverse drug event reporting system (ADERS). This database is used to identify adverse effects not detected during pre-market testing of FDA-approved animal drugs and to monitor the performance of drugs not approved for use in animals. The ADERS depends upon the detection of an adverse clinical event by veterinarians and animal owners, the attribution of the clinical event to the use of a particular drug ("suspect" drug), and the reporting of the ADE either to the manufacturer of the suspected drug or directly to FDA. Data from these ADE reports are reviewed, coded and entered into the computerized ADERS.

         The ADE for veterinary drugs generates current information on the safety and efficacy of veterinary drugs. These data expand the knowledge base used in animal drug approvals and ultimately contribute to reducing the risks associated with veterinary medical products. Summary information from this system is available to support decisions about disseminating information on product safety.

      4. Foods, Including Dietary Supplements, and Cosmetics

         We use several reporting systems to identify problems associated with foods, including dietary supplements, and cosmetics.

         • The Adverse Reaction Monitoring System (ARMS) collects spontaneous reports from consumers and health professionals regarding alleged adverse effects from food products.
         • The Special Nutritional Adverse Event Monitoring System (SN/AEMS) collects spontaneous reports from consumers and health professionals regarding adverse effects from special nutritional.
         • The Cosmetic Adverse Reaction Monitoring System (CARMS) collects spontaneous reports from consumers and health professionals regarding alleged adverse effects from cosmetic products.
         • CFSAN receives adverse event reports linked to the products it regulates through FDA's MedWatch program.

   4. Research to Support the Regulatory Functions of FDA

      FDA conducts peer-reviewed research that supports the regulatory functions of the Agency. To accomplish this mission, the FDA seeks feedback from its stakeholders and partners, including its center scientists, other government agencies, industry, and academia. In addition, the FDA Science Board (SB) provides guidance on specific complex and technical issues as well as emerging issues within the scientific community, in industry, and academia. The SB conducts ongoing reviews of the research programs themselves and how these programs are keeping pace with technical and scientific evolutions in the fields of regulatory science. Their review includes advice on developing an appropriate research agenda and on upgrading FDA's scientific and research facilities to keep pace with these changes. The SB is composed of non-governmental scientists from industry, academia, and consumer organizations. This board is further supplemented with subject matter experts and scientists representing all FDA centers. Research programs can be via collaborations through partnerships with other scientific organizations. Scientific and monetary collaborations include interagency agreements with other government agencies, Cooperative Research and Development Agreements and technology transfer with industry, and grants or informal agreements with academic institutions.

      Each FDA Center and Office also uses several strategies to ensure the quality (including objectivity) of its extramural research and the accuracy of data collected in association with these studies. For example, study protocols may be developed collaboratively by principal investigators and program officials. A project advisory group (PAG) consisting of experts who oversee extramural research projects in a particular field may be appointed.

      To ensure the data are accurate and timely, FDA monitors research progress at the project level on a recurring basis. FDA quality assurance staff assess experiments that fall within the scope of FDA's Good Laboratory practice guidelines for compliance. Additionally, agency scientists publish research manuscripts, book chapters, and abstracts in recognized, peer-reviewed scientific journals. FDA's research findings are also presented at national and international scientific meetings, many of which are sponsored or co-sponsored by FDA. Many FDA scientists also serve on international scientific advisory boards.

3. Integrity

   We strive to maintain the integrity of the information we collect and use, and protect it against disclosure, alteration, loss, or destruction. We require all of our operating divisions to adhere to a series of Agency guidelines to ensure our data integrity operations. Guidelines include:

   • FDA Staff Manual Guide 3250.17 Data Security\Data Integrity
   • Computer Security Act of 1987
   • Computer Fraud and Abuse Act of 1986
   • Clinger-Cohen Act of 1996
   • Federal Managers Financial Integrity Act
   • Government Information Security Reform Act (GISRA)
   • Office of Management and Budget (OMB) Circular A-130, Appendix III, Management of Federal Information Resources, Revised 2/96
   • HHS Automated Information Systems Security Program Handbook (Release 2.0), 5/94
   • NIST Special Publication 800-14, Generally Accepted Principles and Practices for Securing Information Technology Systems (September 1996)

   All data submitted for inclusion in our systems must be accompanied by information about origin, sensitivity, reliability, and the date of most recent revision. We have systems in place to ensure that data modifications are accomplished in a managed and controlled manner and that all information is protected from unauthorized access, revision, corruption, or falsification.

   Transactions affecting sensitive or valuable information can only be processed if the originating individual or system has been validated as authorized to submit such transactions. Additionally, transactions must be initiated only through source documents or computerized messages in which the originating individual or system is clearly identified. All transactions intended for input into a multi-user production computer system must first be subjected to reasonableness checks, edit checks, and/or validation checks.

   Transactions that fail such checks must either be:

   • Rejected with a notification of the rejection sent to the submitter
   • Corrected and resubmitted or
   • Suspended pending further investigation

   Management has established and maintains preventive and detective security measures that ensure that our information is protected from undetected alteration. All rejected input transactions are placed in a suspense file and listed in exception reports until they are successfully resubmitted for processing. Resubmission and corrections are subject to the same validation procedures that original input transactions receive. Management reviews the reasonableness and accuracy of all changes to internal records. If a client or customer brings record errors to our attention, an investigation of the errors is initiated promptly.

   The Office of Public Affairs provides guidance to the Agency on best communications principles and practices to help assure that public messages are accurate, understandable and consistent with current Agency policies. The quality, objectivity, utility and integrity of information posted on the FDA Internet is assured by the internal clearance process used to clear printed documents.

VI. Agency Administrative Complaint Procedures

1. Format for Submitting Complaints

   As described below, we intend to use existing complaint mechanisms to address complaints from the public concerning our information dissemination activities. You may use any of the mechanisms outlined below to request correction of information disseminated by FDA that you believe does not meet the applicable OMB or HHS Information Quality Guidelines, or this Guidance.

   If you request a correction of any information disseminated by us, we would appreciate it if you clearly designate the request as a request for correction of information under this guidance and the OMB and HHS Guidelines. You should use the following format for your request:

   1. Your name, mailing address, fax number or e-mail address, telephone number, and organizational affiliation, if any, of the requestor,
   2. A detailed description of the specific material that needs to be corrected including where the material is located, i.e., the publication title, date, and publication number, if any, or the website and web page address (Uniform Resource Locator or url),
   3. The specific reasons for believing the information does not meet applicable OMB, HHS, or FDA guidelines and is in error, and supporting documentation,
   4. The specific recommendations for correcting the information,
   5. A description of how the person submitting the complaint is affected by the information error.

2. Procedures for Submitting Complaints

   We have clear procedures in place to address complaints from the public. Requests should be submitted to the Agency in accordance with the procedures described below for dispute resolution (i.e., beginning with the employee or division that disseminated the information, or by contacting the center, the Agency, or an ombudsman). To help us track and monitor complaints, regardless of which procedures you use, we ask that you also send a copy of your request for correction to:

   Office of the Ombudsman
   Food and Drug Administration
   10903 New Hampshire Avenue
   WO Building 32, Room 4260
   Silver Spring, MD 20993

   It will also be helpful to those wishing to submit a request for correction electronically
   if the Office e-mail address is included: OMBUDS@OC.FDA.HHS.gov

   FDA regulations at 21 CFR 10.75 provide a mechanism for any interested person (a person who submits a petition, comment, or objection, or otherwise asks to participate in an informal or formal administrative proceeding or court action) to obtain formal review of any Agency decision or action by raising the matter with the supervisor of the employee who made the decision. These procedures can be used to submit an initial complaint about an FDA information dissemination. If the issue is not resolved at the primary supervisory level, the interested person may request that the matter be reviewed at the next higher supervisory level. This process may continue throughout the Agency's chain of command, through the centers to the Commissioner of the FDA.

   Regulations for dispute resolution during the application review process (21 CFR 312.48; 314.103; and 814.42 (d), 814.46(c), 814.112(b), and 808.25 (e)) specify procedures similar to those outlined above. CDRH also established a Medical Devices Dispute Resolution Panel to hear scientific disputes. Regulations for CDER and CBER also provide that a sponsor may request that we seek the advice of outside experts. In addition, we may refer major issues to an appropriate advisory committee for its recommendations (§§ 312.48(c)(3) and 314.103(c)(3)).

   Several guidances explaining the dispute resolution process also are available:

   • Formal Dispute Resolution: Appeals Above the Division Level (for drug and biological products, February 2000),
   • Resolving Scientific Disputes Concerning the Regulation of Medical Devices, a Guide to Use of the Medical Devices Dispute Resolution Panel (July 2001)
   • Medical Device Appeals and Complaints: Guidance on Dispute Resolution (February 1998)
   • A Suggested Approach to Resolving Least Burdensome Issues (September 2000)

   Finally, 21 CFR 5.200 provides for the establishment of an Agency ombudsman. We have established an Ombudsman Office within the Office of the Commissioner, and each center has identified or is identifying an ombudsman. Information about when and how to contact an Agency or center ombudsman can be found on our Internet site ⁷. We encourage interested parties who may be reluctant to contact a program person in a specific program, division, office or center to feel free to contact a center or Agency ombudsman.

   Existing public comment procedures for rule-makings and other formal agency actions already provide well established procedural safeguards that allow affected persons to raise information quality issues on a timely basis. Accordingly, FDA will use these existing procedures to respond to information quality complaints that arise in this process.

   In cases where the agency disseminates a study, analysis, or other information prior to the final agency action or information product, requests for correction will be considered prior to the final agency action or information product in those cases where in the agency's judgment issuing an earlier response would not unduly delay issuance of the agency action or information product and the complainant has shown a reasonable likelihood of suffering actual harm from the agency's dissemination if the agency does not resolve the complaint prior to the final agency action or information product.

   You should be aware that you bear the "burden of proof" with respect to your request for correction.

3. FDA's Response to Complaints

   Based on a review of the information you provide in your complaint, we will determine whether a correction is warranted and, if so, what action, if any, we will take. We will respond to you in a manner appropriate to the nature and extent of your complaint (for example, by letter, e-mail, fax, press release, mass mailing) and will indicate whether, and if so how, we intend to correct the information. We will respond in accordance with the time frame specified in the complaint mechanism you use. Where a procedure does not specify a time frame for a response, we will respond within 60 days, in accordance with the OMB and HHS Guidelines. We will respond by either issuing a decision or by informing you that more time is required, explaining why, and providing you with an estimated decision date. In deciding how to respond, we will consider all of our legal obligations and policies.

4. Requests for Reconsideration

   If you do not agree with FDA's decision about your complaint (including any corrective action), you may send a request for reconsideration within 30 days of receipt of our decision. You may use any of the Procedures for Submitting Complaints described above. A request for reconsideration should state the reasons why you believe the response is inadequate, should be designated as an "Information Quality Appeal," and should include a copy of your original request and the agency's decision. The agency will respond to all requests for appeals within the time frame specified in the procedure you use. Where a procedure does not specify a time frame for a response to your appeal, we will respond in a timely manner, in accordance with the OMB and HHS Guidelines.

VII. Influential Scientific, Financial, and Statistical Information

As illustrated by the number and types of information we disseminate and the variety of methods we use to disseminate them, it is clear that we strive for a high degree of transparency with regard to all of our information dissemination activities. The OMB Guidelines, however, apply special quality standards to the dissemination of information that is considered influential. Such information must meet high standards of transparency of the data and methods used to facilitate the reproducibility of such information by third parties. As defined below, influential scientific, financial and statistical information that FDA disseminates will meet the high standards in the OMB Guidelines for such information.

1. Definition of the Term Influential

   The term influential information, when used in the OMB Guideline in the phrase "influential scientific, financial, or statistical information," applies when the agency can "reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important public policies or important private sector decisions" (67 FR 8452; February 22, 2002). However, because each agency is different, and there are vast differences in the types of information they disseminate, each agency has been asked to elaborate on the definition of influential in the context of their missions and duties, "with due consideration of the nature of the information they disseminate." As stated in the OMB Guideline (V.9), "[e]ach agency is authorized to define 'influential' in ways appropriate for it given the nature and multiplicity of issues for which the agency is responsible."

   For purposes of this guidance, influential information is defined as disseminated information that results from or is used in support of agency actions that are expected to have an annual effect on the economy of $100 million or more or will adversely affect in a material way the economy, a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or State, local or tribal governments or communities. It should be noted that the definition applies to "information" itself, not to decisions that the information may support. Even if a decision or action by FDA is itself very important, a particular piece of information supporting it may or may not be "influential."

   Two examples of what FDA considers to be influential information follow.

   • Quality Mammography Standards

     On October 28, 1997, we issued a final rule (62 FR 55852) amending our regulations governing mammography to provide increased assurance of adequate and consistent evaluation of mammography facilities on a nationwide level and compliance of the facilities with quality standards. Costs of the regulation include replacing below standard mammography units and film processors, providing written results of tests to patients, providing telephone results of tests to referring physicians, and conducting required weekly image quality tests. Because this rulemaking was expected to have an annual effect on the economy of more than $100 million, the information FDA disseminated in support of this rulemaking would be considered influential information.

   • Hazard Analysis and Critical Control Point (HACCP); Procedures for the Safe and Sanitary Processing and Importing of Juice

     On January 19, 2001, we adopted a final rule (66 FR 6138) to ensure the safe and sanitary processing of fruit and vegetable juices. The regulations mandate the application of HACCP principles to the processing of these foods. HACCP is a preventive system of hazards control. FDA adopted this rule in response to a number of food hazards associated with juice products and because preventive control measures are the most effective and efficient way to ensure that these products are safe. The final regulation involves costs to the manufacturers and processors of juice products for implementing procedures consistent with the regulation that exceeded $100 million. Therefore, the information disseminated in connection with this rulemaking would be considered influential information.

2. Transparency

   If information that meets the criteria for influential information is disseminated, the OMB Guidelines provide that it must meet certain higher standards of transparency and methods to facilitate the reproducibility of information by qualified third parties. When we disseminate information, but particularly in those cases involving influential information, we strive to ensure that the information is accurate and unbiased, as well as substantially reproducible and replicable. This goal is accomplished by using reliable data sources and sound analytical techniques, and by employing a high degree of transparency about the data, methods, measures, assumptions and limitations used to develop the information to facilitate reproducibility by third parties. Our goal is to provide a clear explanation of the assumptions and data upon which we base our conclusions, the criteria used to determine the suitability of the data for use, the methods used in our analysis, and the conclusions we have drawn.

   Because of legal obligations to maintain the confidentiality of data supplied by third parties, there may be instances when original or supporting data may not be available to the public. In such cases, we will disclose the specific data sources used and the specific quantitative methods and assumptions employed. We will also conduct especially rigorous robustness checks of any models used in the analysis and the analytic results so that there will be a high degree of confidence in the results.

   Biases, if any, will be revealed. All assumptions used in the analysis, the scientific rationale, and data used to estimate the impact of the various factors influencing the analysis should be clearly stated. This ensures that biases will be eliminated or minimized and that any introduced biases will be clearly identified.

   Clarity includes ensuring the information disseminated is clear and understandable. When detailed technical information is needed to provide sufficient information so that a qualified third party could reproduce the analysis, the resulting document may be lengthy and difficult the public to understand. One approach that can provide additional transparency in such cases is to develop an interpretative summary document as a companion to the technical analysis. The summary document can provide a non-technical explanation of the data, process, results, and conclusions in a manner that the public can understand. As discussed under "Objectivity," we have a strong commitment to writing all our new documents in plain English. As we revise and update existing documents, we will ensure that they are written in plain English. Our goal is to make our written communications more understandable.

   A participatory process should be used. The process for generating information defined as influential should be transparent. One approach is to invite public comment on the information to be disseminated and encourage stakeholders to submit scientific data and information that can be used in preparing the information. As appropriate, we will solicit advice and opinions of advisory committees as well as peer review from experts within and outside of the agency. To the extent practicable under confidentiality laws, we will strive to make supporting data and analyses available to the public for technical review and comment. This can be accomplished by posting the information on our web pages and providing printed copies as requested.

3. Risk Assessment

   Some of the influential information that we disseminate is based on an analysis of the risks to the public of certain actions or exposures to hazardous substances. For purposes of this guidance, we are defining risk as the likelihood that injury or damage is or can be caused by a substance, technology, or activity. We use risk analysis (the integration of risk assessment with risk management and risk communication) as a tool to enhance the scientific basis for all of our regulatory decisions.

   The OMB Guidelines provide that in addition to the ordinary standards for utility, objectivity, and integrity that apply to dissemination of information, special considerations must be taken into account in certain risk assessments, i.e., those that provide the basis for the dissemination of influential information. The Guidelines state that "With regard to analysis of risks to human health, safety, and the environment maintained or disseminated by the agencies, agencies shall either adopt or adapt the quality principles applied by Congress to risk information used and disseminated pursuant to the Safe Drinking Water Act Amendments of 1996 (SDWA) (42 U.S.C. 300g-1(b)(3)(A) and (B)).

   The SDWA risk assessment principles are as follows:

   1. To the degree that the agency action is based on science, the agency shall use

      1. the best available, peer-reviewed science and supporting studies conducted in accordance with sound and objective scientific practices
      2. data collected by accepted methods (if reliability of the method and the nature of the decision justify use of the data)

   2. In the dissemination of public information about risks, the agency shall ensure that the presentation of information about risk effects is comprehensive, informative, and understandable.

   3. In a document made available to the public in support of a regulation, the agency shall specify, to the extent practicable

      1. Each population addressed by any estimate of applicable risk effects
      2. The expected risk or central estimate of risk for the specific populations affected
      3. Each appropriate upper-bound or lower-bound estimate of risk
      4. Each significant uncertainty identified in the process of the assessment of risk effects and the studies that would assist in resolving the uncertainty and
      5. Peer-reviewed studies known to the agency that support, are directly relevant to, or fail to support any estimate of risk effects and the methodology used to reconcile the inconsistencies in the scientific data

   Many of our actions are based on scientific experts' judgments using available data, are essentially qualitative, and are generally carried out for non-cancer-causing hazards. Such assessments provide useful answers in most instances that are sufficient for regulatory purposes, and much more elaborate, quantitative estimates extrapolating beyond the data are unnecessary. For example, we may issue regulations on submission requirements for product approval applications, electronic submission of product labeling, or periodic reporting by manufacturers of adverse events from drugs; devices; and biologics, including blood, vaccines, and tissues. Regulations like these do not always lend themselves to the types of quantitative risk assessments contemplated by the Safe Drinking Water Act principles.

   Other actions are based on research and supporting data that are generated outside FDA. For example, most product approval actions are based on scientific studies conducted by sponsors seeking marketing approval in accordance with our regulations and guidance documents. Our regulations and guidance documents describe sound scientific practices for conducting human and animal studies of medical products and analyzing the resulting data. Most information in these studies is considered confidential commercial information and is closely held by the sponsors. As a result, formal peer-review of the data is rare. However, for certain drug approval applications, the safety and/or effectiveness information is presented to scientific advisory committees for recommendations. Evaluations of food safety and nutritional data are also presented to scientific advisory committees.

   As a result, we have adapted the general principles for risk assessments from the SDWA to fit these situations. The principles we intend to apply to risk assessments involving the dissemination of influential information affecting product approval actions or regulations that do not lend themselves to quantitative risk assessment are as follows:

   1. The Agency will use

      1. the best available science and supporting studies conducted in accordance with sound and objective scientific practices, including peer reviewed science and supporting studies when available
      2. data collected by accepted methods (if reliability of the method and the nature of the decision justify use of the data)

   2. In the dissemination of public information about risks, the Agency will ensure that the presentation of information about risk effects is comprehensive, informative, and understandable.

   In situations requiring a quantitative risk assessment, we generally follow basic risk assessment principles in the NAS paradigm of 1983. Our needs for quantitative risk assessments range over a wide variety of hazards including physical hazards encountered during use of a medical device, food chemical residues, and antimicrobial resistance genes in bacteria. Thus, we also ascribe to the statement from NAS when it revisited the risk assessment process in 1994 (Science and Judgment in Risk Assessment, NAS 1994): "Risk assessment is not a single process, but a systematic approach to organizing and analyzing scientific knowledge and information. "In each of the areas we regulate, we apply risk assessment practices to the specific task that are widely accepted among relevant domestic and international public health agencies.

   For quantitative risk assessments in support of the dissemination of influential information, FDA intends to apply the following principles:

   1. The agency will use-

      1. the best available science and supporting studies conducted in accordance with sound and objective scientific practices, including peer reviewed science and supporting studies when available;
      2. data collected by accepted methods (if reliability of the method and the nature of the decision justifies use of the data)

   2. In the dissemination of public information about health risks, the agency shall ensure that the presentation of information is comprehensive, informative, and understandable, within the context of its intended purpose.

   3. In a risk assessment document made available to the public, the agency shall specify, to the extent practicable-

      1. Each population addressed by any estimate of applicable effects;
      2. The expected or central estimate of risk for the specific populations affected;
      3. Each appropriate upper-bound and/or lower-bound risk estimate and the methodology used to reconcile the inconsistencies in the scientific data;
      4. Data gaps and other significant uncertainties identified in the process of the risk assessment and the studies that would assist in characterizing the uncertainties; and
      5. Additional studies not used to produce the risk estimate that support or fail to support the findings of the assessment, and the rationale of why they were not used.

VIII. Special Considerations for Agency Dissemination

Under certain circumstances, we may need to disseminate information without fully applying the principles for ensuring the quality, objectivity, utility and integrity of the information outlined above. Even in these cases, however, FDA intends to use its internal review process to evaluate the data received and the information it plans to disseminate to ensure to the degree practicable the accuracy, objectivity, and transparency of the relevant information, and will entertain requests for correction after the exigent circumstances have passed. The specific situations where this may occur are as follows:

• Public Health Emergencies: In the case of a public health emergency, there may not be time for the Agency to submit relevant information to all levels of review or review by an Advisory Committee prior to dissemination of the information.
• Statutory or Other Legal Requirement: If a statutory requirement, Executive Order, or court order requires immediate implementation of a policy, we may have insufficient time to apply the OMB requirements prior to disseminating information relevant to that policy.
• Other Circumstances: There may be unforeseen circumstances in carrying out our mission that could prevent the Agency from applying all of the OMB guidelines when disseminating information to the public.

As mentioned above, in all such special circumstances, the Agency will be particularly careful to use its internal review processes to the extent practicable when considering the dissemination of relevant information to the public.

IX. References

Clinger-Cohen Act of 1996.
Code of Federal Regulations (CFR), Title 21, Parts 5, 10, 50, 56, 58, 312, and 314.
Computer Fraud and Abuse Act of 1986.
Computer Security Act of 1987.
Consolidated Appropriations Act, 2001 (Public Law 106-554), Appendix C -- H.R. 5658.
HHS Automated Information Systems Security Program Handbook (Release 2.0), May 1994.
FDA guidance, Formal Dispute Resolution: Appeals Above the Division Level, February 2000.
FDA guidance, Medical Device Appeals and Complaints: Guidance on Dispute Resolution, February 1998.
FDA guidance, Resolving Scientific Disputes Concerning the Regulation of Medical Devices, a Guide to Use of the Medical Devices Dispute Resolution Panel, July 2001.
FDA Internet world Wide Web Site Guidelines, rev. March 23, 2001.
FDA Staff Manual Guide 3250.17,Data Security\Data Integrity.
Federal Food, Drug, and Cosmetic Act (21 U.S.C. 301 et seq.).
Federal Managers Financial Integrity Act.
Food and Drug Administration Modernization Act of 1995 (Public Law 105-115).
Freedom of Information Act and Electronic Freedom of Information Act Amendments (5 U.S.C. 552).
Government Information Security Reform Act (GISRA).
ICH guidance, E2B Guidance on Data Elements for Transmission of Individual Case Safety Reports, January 1998.
NIST Special Publication 800-14, Generally Accepted Principles and Practices for Securing Information Technology Systems, September 1996.
Office of Management and Budget (OMB) Circular A-130, Appendix III, Management of Federal Information Resources, rev. February 1996.
Public Health Service Act (42 U.S.C. 201 et seq.).
Regulatory Flexibility Act (5 U.S.C. 601-612).
Safe Drinking Water Act Amendments (Public Law 104-182).
Treasury and General Government Appropriations Act, 2001.

Endnotes

¹ These guidelines published in the Federal Register on January 3, 2002 (67 FR 369) and were republished with corrections on February 22, 2002 (57 FR 8452).

² All product reviews undergo extensive review through a hierarchical process (see Section V).

³ Good laboratory practices (GLPs) for nonclinical laboratory studies are discussed in 21 CFR 58.

⁴ Regulations at 21 CFR 312, guidances developed as part of the Agency's international harmonization efforts (for example, E6 and E8), and guidances developed by FDA that address clinical development of drugs to treat specific indications provide requirements and recommendations on good clinical practice (GCP). In addition 21 CFR parts 50 and 56 address issues related to informed consent and investigational review boards (IRBs), respectively.

⁵ FDA also administers an HHS Advisory Committee that has 18 panels.

⁶ This guidance is being updated and should be available soon.

⁷ Information on the FDA Ombudsman Program can be found at www.fda.gov/oc/ombudsman/homepage.htm.

G. Health Resources and Services Administration

I. Agency Mission

The Health Resources and Services Administration (HRSA) is the lead Federal agency in promoting access to health care services that improve the Nation's health. With a statutory emphasis on special needs, underserved, and vulnerable populations, HRSA mobilizes its bureaus, programs, staff, and partners to assure access to quality health care. The four main strategies to achieve this goal are to: eliminate barriers to care; eliminate health disparities; assure quality of care; and, improve public health and health care systems. To fulfill HRSA's mission, its programmatic portfolio includes a range of programs or initiatives designed to increase access to care, improve quality, and safeguard the health and well-being of the Nation's most vulnerable populations.

Collectively, HRSA programs work to improve access to care for the more than 38.7 million Americans who are uninsured and the 40 million who live in medically undeserved areas. HRSA supports over 700 community health centers; funds services for people living with HIV/AIDS through the Ryan White CARE (Comprehensive AIDS Resources Emergency) Act; assists States and health care organizations in improving services to mothers and children; oversees the National system that allocates organs, tissue, and bone marrow for transplantation; and works with academic health centers and other training programs to enhance the diversity and distribution of the Nation's health care workforce.

The primary operating units in HRSA each contribute to the overall mission and major goals and objectives:

• The Bureau of Health Professions: provides national leadership to assure a health professions workforce that meets the health care needs of the public.
• The HIV/AIDS Bureau: provides leadership in the delivery of high quality HIV primary care and supporting services for uninsured and underinsured individuals and families affected by HIV/AIDS.
• The Maternal and Child Health Bureau: works on behalf of America's mothers, children, and families in ways that will assure continued improvement in their health, safety, and well-being.
• The Office of Rural Health Policy: serves as the leading Federal proponent for better rural health care services.
• The Office of Special Programs: ensures access and capacity to scarce resources including assurance of access to organ and bone marrow transplantation programs.
• The Bureau of Primary Health Care: increases access to comprehensive primary and preventive health care and works to improve the health status of underserved and vulnerable families and individuals.

More information about HRSA, its programs and activities is available on the HRSA web site www.hrsa.gov.

II. Scope and Applicability of Guidelines for Agency/Office

HRSA will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and HRSA guidelines.  It is HRSA's policy to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

The guidelines apply to substantive information that is disseminated by Federal agencies subject to the Paper Reduction Act (PRA), excluding information that is someone's opinion rather than fact or the agency's views. More specifically, it applies to agency initiated or sponsored information. The HRSA information to which these guidelines would apply are aggregate statistical types of information that are abstracted from grantee applications, policy and program information (i.e., data and state profile information, community information, official reports), as well as other electronic documents, newsletters, and brochures that are provided to the public. For example, this includes the HIV/AIDS State Profiles, the Title V Information System web site, the current findings from the National Sample Survey of Registered Nurses, the Area Resource File, and the HRSA State Profiles.

The pre-dissemination review described in the guidelines only applies to information first disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.

The guidelines would not include data provided directly to HRSA by grantees, such as individual grantee application forms, reporting requirements, performance plans, etc., but would apply to information that is disseminated from such collections of information. Examples of other types of information collections to which these guidelines would not apply is the information that is collected for the National Practitioner Data Bank (NPDB), since this information is not disseminated to the public but is restricted in access to those entities and persons who, under specified conditions may request information from the NPDB. Other information that is not covered includes papers, books, journal articles, and other documents that have the disclaimer that the information contained therein does not represent the agency's views. Information that is not covered also includes press releases, correspondence with individual persons, archival records, or opinions.  In addition, the guidelines do not apply to press releases that support the announcement or give public notice of information that HRSA has disseminated elsewhere.

III. Types of Information Disseminated by the Agency to the Public

HRSA disseminates a range of information to the public via the HRSA web site, the HRSA Information Center, and the Office of Communications. The HRSA Information Center is a gateway to information about HRSA programs. All information supplied through this center has been cleared through HRSA and ASPA. Through the Information Center, health care professionals, policymakers, researchers, and members of the public can obtain material on HRSA-supported health programs. The HRSA website also provides information, data, and reports on community health resources, health professions training, HIV/AIDS, managed care, maternal and child health, minority health, oral health, primary health care, rural health, and women's health. Within the HRSA web site there is a wealth of public information. Users of this site may review information on-line or download it for future use.

1. Examples of Program, Policy, and Administrative data:

   1. The HRSA State Profiles features resources granted by the Agency within each state targeted to serve low-income, uninsured, and medically undeserved populations. To provide an understanding of each state's unique environment, the Profiles also include demographic data and health status indicators. Each State Profile features:

      • An overview of funds granted within a state
      • Selected demographic data
      • Health status and health care access indicators
      • Health care provider resources
      • Highlights of HRSA resources targeted in the areas of primary health care, health care provider resources, maternal and child health, public health, rural health, and HIV/AIDS.

      The State Profiles are six-page reports highlighting the health status of each State's population and HRSA's investment of health resources within each State and the District of Columbia for the years 1997 through 1999. Two categories of data are provided: program data describing resources provided to States by HRSA, and demographic and health status indicator data. For each data element, there are data definitions, source information, types of calculations and derivations, value ranges, and a description of the HRSA Program from which the data were collected. Users can create customized reports by selecting specific data elements, and can generate data to be imported into a spreadsheet or database.

   2. The HRSA HIV/AIDS Bureau State Profiles describe spending and service information of the Ryan White CARE Act programs. Information is provided on location of grantee, clients served, grantee accomplishments, and the characteristics of the HIV epidemic in the state (e.g., co-morbidities and other funding sources such as Medicaid, the largest payer of HIV/AIDS services in the nation), reporting requirements and progress reports for the Title I, II, III, and IV grantees; CDC Surveillance Reports; and other sources including Census and GAO.

   3. The Title V Information System (IS) provides information on the status of maternal and child health in the United States. Data are compiled from annual Title V Block Grant applications and reports submitted by all U.S. States, Territories, and Jurisdictions. Information is provided on key measures of maternal and child health, budgets and expenditures, program data, performance measures, and summary data. Users may download prepared tables, graphs, brochures, fact sheets, and reports on maternal and child health, as well as the Electronic Reporting Package (ERP) that States use to report their data.

   4. The HRSA Preview: the HRSA Preview provides the general public with a single source of program and application information related to the Agency's competitive grant offerings. It contains a description of competitive and other grant programs scheduled for awards in Fiscal Year 2002, and includes instructions on how to contact the Agency for information and receive application kits for all programs. The following specific information is included in the HRSA Preview: (1) program title; (2) legislative authority; (3) purpose; (4) eligibility; (5) funding priorities and/or preferences; (6) estimated dollar amount of competition; (7) estimated number of awards; (8) estimated project period; (9) Catalog of Federal Domestic Assistance (CFDA) identification number; (10) application availability date; (11) letter of intent deadline (if any); (12) application deadline; (13) projected award date; (14) programmatic content, with telephone and e-mail addresses. Certain other information, including how to obtain and use the HRSA Preview and grant terminology, can also be found in the HRSA Preview.

   5. HRSA's Bureau of Health Professions Area Resource File (ARF): the ARF is a county level data set with information on health professions, health facilities, utilization, expenditures, population characteristics, and geographic environment. The data set includes county level estimates for the entire U.S. of the number of physicians, including specialties and limited demographic information, dentists, dental hygienists, optometrists, pharmacists, podiatrists, nurses, physician assistants, and other health professions. Information is available on hospitals, nursing homes, HMOs, inpatient days and other utilization measures, hospital and Medicare expenditures, and various selected variables containing information on the resident population and environmental characteristics. The ARF utilizes secondary data obtained from Census, the American Medical Association, National Center for Health Statistics, and others and compiles a wide array of data sources. As such, the limitations of these data sources all apply, as do any cross source comparability issues.

2. Example of statistical data:

   The National Sample Survey of Registered Nurses (NSSRN): The Seventh NSSRN was conducted in 2000 and is the nation's most extensive and comprehensive source of statistics on all those with current licenses to practice in the United States. It provides information on the number of registered nurses, their educational background and specialty areas; their employment settings, position levels, and salaries; their geographic distribution; their personal characteristics including gender, racial/ethnic background, age, family status, and satisfaction with their job. The substantial database resulting from the 2000 study may be used for many different types of analyses concerning a variety of subjects. The report presents an overview of the personal, professional, and employment characteristics of the almost 2.7 million registered nurses in the country as of March 2000. A summary of the findings from the study and some comparisons to the findings of prior studies in this series, are presented in the report. Appendix A contains a series of tables summarizing the data. A review of the survey methodology and the statistical techniques used in sample selection, response weighting, and identification of sampling errors are found in Appendix B. The survey instrument is included in Appendix C of the report.

IV. Types of Dissemination Methods

Information is disseminated by HRSA by a variety of mechanisms: the HRSA Office of Communications (OC) serves as the Agency point of contact for clearing and producing HRSA information products. Information is available from the HRSA Information Center in the form of publications, brochures, fact sheets, and other resources on health care services. Through the Information Center, health care professionals, policymakers, researchers, and members of the public can obtain material on HRSA-supported health programs. Information on HRSA publications is provided in the Information Center catalog, and may also be ordered directly from the HRSA web site.

For each of the earlier examples given above of information disseminated by the Agency to the public, the following dissemination methods are used:

1. The HRSA State Profiles are available through the HRSA web site on stateprofiles.hrsa.gov and the information can be viewed and downloaded using Adobe Acrobat and provides a formatted six page profile for each state.

2. The HIV/AIDS Bureau State Profiles can be accessed at hab.hrsa.gov. The State Profiles can be viewed or downloaded in Adobe Acrobat. The site has a phone number and email contact information for further requests and feedback.

3. The Title V Information System is available on the HRSA Bureau of Maternal and Child Health web site at www.mchdata.net. Program materials can be downloaded to obtain prepared tables and graphs, brochures, fact sheets, annual reports, and more. Hard copy versions are available from the Information Center. The web site also includes specific contact information for questions regarding the content of the site and questions, problems or comments regarding the functionality of the site.

4. The HRSA Preview was published in the Federal Register on 8/9/2001 and is also available for downloading on the web at www.hrsa.gov/grants.htm. Additional information is available at:

   HRSA Grants Application Center
   Attention: Grants Management Officer
   901 Russell Avenue, Suite 450
   Gaithersburg, MD 20879

   or by calling the HRSA Grants Application Center at 1-877-477-2123.

5. The Area Resource File is available in electronic format only. Contact information is provided on the HRSA web site and includes an address, phone number, fax number, email, and web site specific to ARF.

6. Findings from the 2000 National Sample Survey are provided on the HRSA web site at bhpr.hrsa.gov/healthworkforce/rnsurvey and can be downloaded in Adobe Acrobat. Contact information for the Nursing Data and Analysis Staff in the Bureau of Health Professions, HRSA, is also provided. Preliminary findings were published in a report and are available from the HRSA Information Center.

V. Agency Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Disseminated to the Public

HRSA reviews the quality (including the objectivity, utility, and integrity) of information before its is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination.

1. Program information:

   Program information provided by grantees in the form of application data, reporting requirements, performance reports and progress reports, receives a series of reviews prior to any dissemination to the public. Grantees conduct internal reviews in order to provide data that are accurate, consistent, and complete. The HRSA Office or Bureau conducts a standard review to ensure data quality, completeness, and reliability.

2. Reports, publications, and other products:

   HRSA publications, audiovisual products, and exhibits are required to be consistent with government-wide and HHS public affairs policies. HRSA has published guidelines which closely follow the regulations described in the HHS Public Affairs Management Manual and Government Printing and Binding Regulations to assist program staff in obtaining appropriate review and clearance of information products. To produce a publication, a complete HHS-615 (Publication Planning and Clearance Request) form must be submitted to, and approved by, ASPA.

   Publications to be produced at HRSA's request under contract are subject to HHS clearance and the Joint Committee on Printing (JCP) printing procedures. An HHS-524 clearance is required for publications to be developed as part of a larger public affairs services contract. The HHS-524 must be approved before a Request for Proposals (RFP) can be issued.

3. Statistical data disseminated to the public:

   At HRSA, the quality assurance process begins at the inception of the information development process.

   For information disseminated from sample surveys and other research related or evaluative activities, the widely accepted standards of technical and scientific review are utilized to ensure data quality. These standards include, where appropriate, peer review, internal expert review, Institutional Review Board (IRB) review and OMB review.

   For each of the examples provided earlier, the following quality assurance procedures are employed.

   1. The HRSA State Profiles are reviewed by program staff from each Bureau providing input into the profile prior to dissemination. This is an interval review by each Office and Bureau from which data are obtained. The originating office is responsible for ensuring that all necessary internal review, approval, and clearance is obtained prior to dissemination. Where indicators are taken from other sources, that source is clearly indicated and the limitations of the data are described. For the demographic and other data derived from other sources, HRSA provides an explicit list of the source information, the year the data were obtained, citations, and related URLs. For example, for health indicators such as percent of people insured in the State who had private insurance, the source listed is the U.S. Census Bureau, Health Insurance Statistical Tables, and the citation is Table HI-4, Health Insurance Coverage Status and Type of Coverage by State-All Persons, 1998. Each data element contains such a source listing along with definitions.
       
   2. The HIV/AIDS Bureau uses CAREWare, a software package for use by the HIV/AIDS Bureau providers to track clients and services. The software and manual are available online, and the system captures all the data items required for the reports needed. CAREWare contains consistency and edit checks on inputted data for quality assurance. This system was designed specifically to collect the exact data elements and generate complete reports. The HIV/AIDS program staff review all data for verification prior to dissemination. The CARE Act State Profiles data provides, for each element, a source list and year of data, as well as notes relating to limitations, rounding, and restrictions, where appropriate. For example, for the Title II Profile Report, Clients Served by gender, age, race/ethnicity, exposure category, the source is listed as the Annual Administrative Report to HRSA, with a note that Accounts are rounded to the nearest 10. Data are not reported from cells of fewer than six clients.
       
   3. The Maternal and Child Health Bureau Title V annual report uses an electronic reporting package that was developed in collaboration with the states in order to improve accuracy and completeness of the report. This reporting package provides for automatic calculations of ratios, rates, and percentages, and carries data over from year to year, and assures that data used in multiple tables is entered only once. This system has in place a mechanism that warns states when conflicting or unacceptable figures appear as a quality check on the final data. Knowledgeable MCHB program staff monitor and review all information submitted in the grant annual report for completeness, accuracy, and reliability.
       
   4. The HRSA Preview undergoes appropriate internal Division, Bureau, and Office review and approval prior to dissemination. HRSA adheres to all accepted standards applying to the publication of information in the Federal Register. The HRSA Preview undergoes extensive Office/Bureau, and Agency wide review to ensure accuracy, reliability, and comprehensiveness prior to dissemination to the pubic.
       
   5. The Area Resource File provides county level data from a variety of data sources. As a result the data have a number of limitations and users of the file are provided with a disclaimer as to the applicability and detail of the data. The ARF utilizes secondary data only, and the years and detail for which all county level data are available are considerably limited. Each data source conducts its own review prior to the release of information. Knowledgeable staff within the originating office and across offices conduct standard reviews for completeness and reliability of the data.
       
   6. The National Sample Survey of Registered Nurses received extensive review during each phase of operation: survey design, sample selection, fielding of the survey, data editing, and analysis. Clearance was required of all proposed sampling methods, survey procedures, the data collection instrument, and analytic plans by HRSA, HHS, and OMB. This clearance requires a series of expert review. A scientific review, an internal review, and IRB review were conducted to ensure that survey procedures, data collection, editing, and analyses would adhere to the highest standards.

   As in all research activities involving human subjects, it is HRSA's policy that adequate protection of participants be ensured in accordance with the provisions of 45 CFR Part 46. Project plans and procedures must be submitted to the appropriate IRB for approval, and the program must execute all necessary Assurances of Compliance with OPRR. The NSSRN obtained IRB review and approval for all survey methods and procedures.

   For the National Sample Survey of Registered Nurses, widely accepted standards as established by the statistical community are utilized to assure data survey quality. For example, the median design effect for the survey was 1.66, an accepted level by the statistical community for a large national survey. The information disseminated in the report provides extensive detail on the analytic issues (sampling and nonsampling errors, standard error calculation, variance estimation, etc.). As with any sample survey, the results are subject to sampling error. The report provides detail on measures of variability, standard errors, information on the computation of the sampling variance and the design effect.

   Further, HRSA strives to demonstrate in its Paper Reduction Act (PRA) clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and HRSA information quality guidelines.

VI. Agency Administrative Complaint Procedures

1. Responsibility of the Complainant

   To seek a correction of information disseminated by the agency, individuals should follow the procedures described below.

   1. A complaint or request for review and correction of information shall be in written hard copy or electronic form;
   2. it shall be sent to the agency by mail or electronic-mail(e-mail); and
   3. it shall state that an information quality request for correction is being submitted.

   The complaint shall contain

   4. a detailed description of the specific material that needs to be corrected including where the material is located, i.e. the publication title, date, and publication number, if any, or the website and web page address (url), or the speech title, presenter, date and place of delivery; and
   5. the specific reasons for believing the information does not comply with OMB, HHS or HRSA guidelines and is in error and supporting documentation, if any;
   6. the specific recommendations for correcting the information;
   7. a description of how the person submitting the complaint is affected by the information error; and
   8. the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.

   Complainants should be aware that they bear the ‘burden of proof’ with respect to the necessity for correction as well as with respect to the type of correction they seek.

   Complaints by mail should be directed to:

   HRSA Reports Clearance Officer
   HRSA/OPE, Room 14-45
   5600 Fishers Lane
   Rockville, MD. 20857

   E-Mail complaints should be sent to www.infoquality@hrsa.gov

2. Responsibility of the Agency

   Based on a review of the information provided, the agency will determine whether a correction is warranted and if, so what action to take. The agency will respond to the requestor by letter or e-mail. The agency's response will explain the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. The agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

3. Appeals

   If the individual submitting the complaint does not agree with the agency's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal shall state the reasons why the agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the agency response to it, clearly mark the appeal with the words, Information Quality Appeal, and send the appeal to the specific agency appeals address:

   Division of Information and Analysis
   Office of Planning and Evaluation
   HRSA, Room 14-45
   5600 Fishers Lane
   Rockville, MD. 20857

   The agency official who resolved the original complaint will not have responsibility for the appeal.  The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

4. Comments and Requests for Information:

   HRSA's Office of Communications (OC) has maintained a toll-free phone number to answer questions and address comments on subject specific information. This office has served as the point of entry for queries regarding HRSA information, requests for publications, comments and corrections to information disseminated to the public. To date this Office has not received any complaints about information disseminated by HRSA. Calls have been made, however, by States and grantees to correct and/or update information that is maintained on the HRSA web site. These infrequent calls are received by the OC and directed toward the appropriate Bureau for review, verification, and revision. The States/grantees verify the correction/modification of their data to Bureau staff and after confirmation with program staff the revisions are implemented. In addition, the HRSA web site contains a detailed list of HRSA phone numbers by Office and Bureau for comment or for additional information. Comments and questions can be mailed to ask@hrsa.gov or comments@hrsa.gov, or by calling 1-800-275-4772.

VII. Influential Scientific, Financial and Statistical Information

From time to time HRSA disseminates information that would generally be regarded as influential. In those circumstances, we use the highest standards of reproducibility and transparency for such information to be disseminated to the public.

VIII. References

(Currently HRSA is in the process of updating these references)

HRSA Circular, No. 95-01, Identification of HRSA in Communications Materials Logo Guidelines. HRSA, OC, 5600 Fishers Lane, Rockville, Md. 20857

HRSA Policy Circular, No. 96.05, Protection of Participants in HRSA Research Programs. HRSA, 5600 Fishers Lane, Rockville, MD. 20857

Planning, Clearing, and Producing a HRSA Information Product. 1999, HRSA, Office of Communications, 5600 Fishers Lane, Rockville, MD. 20857

Publications Catalog, 2001. HRSA, Information Center, 5600 Fishers Lane, Rockville, MD. 20857

Skinner, CJ. Aggregated analysis: standard errors and significance tests. In: Skinner, CJ, Holt, D. Smith TMF, eds. Analysis of complex surveys. New York: John Wiley and Sons, Inc. 1989.

The Registered Nurse Population, Preliminary Findings, 2001. The National Sample Survey of Registered Nurses, 2002. HRSA, Division of Nursing, 5600 Fishers Lane, Rockville, MD. 20857

IX. Other Agency Specific Policies and Procedures

HRSA, through its Bureaus and Offices, administers a variety of service delivery and demonstration programs. HRSA Bureaus and Offices also administer some specific research programs and a variety of epidemiological and service utilization studies, as well as evaluations which might be considered research. Participants must be protected from potential risks that may be associated with any such research projects. HRSA Policy Circular No. 96.05 establishes policies and procedures to protect human subjects in research programs conducted or supported by HRSA.

This circular gives guidance for compliance with HHS regulations dealing with protection of human research subjects. It states HSRA policy for various levels of protection applicable to HRSA programs, and explains how to determine the protection level appropriate to each program. It provides guidance on exemption for certain research activities or public benefit or service programs, requiring evaluation of such claims by program staff and approval by a HRSA Human Subjects Protection Committee.

If the public checks the HRSA resources for public information but does not find adequate information, they may file a Freedom of Information Act (FOIA) request. The HRSA FOIA Home Page, newsroom.hrsa.gov/efoia.htm has the location and contact information for making such requests. The site contains a guide for requesting information and records from HRSA as well as a copy of the Freedom of Information Act, As Amended. In addition, this site provides the HRSA Annual FOIA Report for Fiscal Years 1998 through 2001.

H. Indian Health Service

I. Agency Mission

The Indian Health Service (IHS) is responsible for providing federal health services to American Indians and Alaska Natives. The IHS provides a comprehensive health services delivery system for American Indians and Alaska Natives with opportunity for maximum tribal involvement in developing and managing programs to meet their health needs. The goal of the IHS is to raise the health status of American Indian and Alaska Native people to the highest possible level. IHS services are provided directly and also through tribally contracted and operated health programs.

II. Scope and Applicability of Guidelines for IHS

IHS will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and IHS guidelines. It is IHS’s goal to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

The guidelines apply only to the dissemination of substantive information that the IHS initiates or sponsors. The IHS information to which these guidelines would apply are programmatic statistics and IHS program and policy information.

The pre-dissemination review described in the guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.

1. Information that is subject to the Information Quality Guidelines includes:

   • Statistical clinical information
   • Actuarial information
   • Epidemiological information
   • Studies and summaries prepared for public dissemination to inform the public about the impact of IHS programs
   • Studies and summaries prepared for use in formulating broad program policy

2. Information that is not subject to the Information Quality Guidelines includes:

   • Personal information received as a result of agency adjudicative decisions
   • Information pertaining to basic agency operations
   • Program publications, such as pamphlets and notices, that describe Area and regional programs
   • Clinical program information
   • Information on how to apply for third party program benefits
   • Procedural and policy manuals
   • Management information that is produced primarily for internal use
   • Press releases that support the announcement or give public notice of information that IHS has disseminated elsewhere.

III. Types of Information Disseminated by the IHS to the Public

IHS disseminates statistical reports and IHS policy and programmatic information to the public via the IHS web site and through IHS publications. The IHS website also contains links to other useful sites relating to health information and IHS related activities. The Indian Health Service (IHS) provides information to the public about the current IHS programs, clinical epidemiological data on American Indian and Alaska Natives(AI/AN), the projected scope and impact of IHS programs of AI/AN people in the future, and the effect of proposed changes to the IHS and tribal health care systems. Information products describe the impact of IHS programs on American Indian and Alaska Natives communities and upon its beneficiary populations. IHS provides detailed demographic and economic information on health benefits for AI/AN people. The IHS also provides information on research among AI/AN communities. IHS information products are used by government planners and policymakers as well as many actuaries, economists and other social scientists, the media and the public to analyze IHS programs and their impact on the health of the Indian Nation.

IV. Types of Dissemination Methods

IHS disseminates information in print and electronic form.

V. Agency Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Disseminated to the Public

The quality assurance process begins at the inception of the information development process.  Further, IHS reviews the quality (including the objectivity, utility, and integrity) of information before its is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination.  In addition, IHS demonstrates in its Paperwork Reduction Act clearance submission that information will be collected, maintained, and used in a way that is consistent with 0MB, HHS and IHS information quality guidelines.

1. Administrative and program data:

   Information is provided by tribal governments in the form of program data, reporting requirements, performance reports and progress reports. This information is often compiled and aggregated by IHS and is subject to a series of internal quality reviews prior to any dissemination to the public. Tribal governments may conduct internal reviews in order to provide data that are accurate, consistent, and complete.

2. Reports, publications, and other products:

   IHS publications are required to be consistent with government-wide and HHS public affairs policies. IHS has guidelines which closely follow the regulations described in the HHS Public Affairs Management Manual and Government Printing and Binding Regulations to assist program staff in obtaining appropriate review and clearance of information products.

3. Statistical data disseminated to the public:

   For IHS demographic and statistical information, IHS uses widely accepted standards for descriptive and analytical statistics to ensure data quality. The IHS statistical program conducts a standard review to ensure data quality, completeness, and reliability. The review includes the reliability of data sources and soundness of statistical methods and presentation.

   Information released by IHS is developed from reliable data sources utilizing accepted methods for data collection. The information is based on thoroughly reviewed analyses and models. The guidelines below describe procedures that IHS employs to assure the quality of its information products, including their utility, objectivity, integrity, transparency, and reproducibility.

4. Utility:

   Utility involves the usefulness of the information to its intended users. Utility is achieved by staying informed of information needs and developing new data, models, and information products where appropriate. Based on internal analyses of information requirements, convening and attending conferences, working with advisory committees, and sponsoring outreach activities, IHS keeps abreast of information needs with respect to the analysis of IHS programs. To the extent possible, we base our analysis on IHS administrative data and surveys by other federal agencies or established survey organizations. When major needs for data related to IHS populations are identified, IHS conducts special-purpose surveys to address these needs or assigns the task to its sister agencies in the Department of Human Services to assist in obtaining the needed data.

   IHS's ongoing publication series and other information products are reviewed to ensure that they remain relevant and address current information needs. Based on internal product reviews, consultation with users, and in response to changing needs and emphases, content of ongoing information products is changed, new products are introduced and others discontinued. IHS prepares special Clinical Trends reports and topical studies that address emerging information needs stemming from data coming from the agency clinical services. It also addresses proposed changes in the law and related policy debates. IHS also identifies requirements for simulation models to support the preparation of analytic reports and policy studies and modifies its current models or develops new models accordingly. Where appropriate, contact information is available on each publication (and in some cases on each table of a publication) to allow feedback and questions by users. In addition, IHS's Office of Program Support has recently begun a periodic customer satisfaction survey to obtain input and guidance by users.

5. Objectivity:

   Objectivity involves a focus on ensuring that information is accurate, reliable and unbiased and that information products are presented in an accurate, clear, complete and unbiased manner. Objectivity is achieved by using reliable data sources and sound analytical techniques, and preparing information products that use proven methods by qualified people that are carefully reviewed.

   Much of the information disseminated by IHS is based on patient data files. These files contain information used to manage IHS programs, including data to determine benefits, and plan for improving the health of the American Indian and Alaska Native people IHS's Office of Public Health conducts ongoing reviews of data and information in IHS data systems to ensure its accuracy. In support of these activities, IHS employs an outside contractor to review IHS's quality control methodology and processes to confirm the validity of its data management processes. IHS administrative data are also covered under IHS Financial Management System and conform to the high standards of financial accountability demanded by these Systems. These financial management systems are mandated by the Office of Management and Budget and are designed to provide complete, reliable, consistent, timely and useful management information to enable agencies to carry out their fiduciary responsibilities.

   IHS-sponsored surveys are conducted using methodologies that are consistent with generally accepted professional standards for all aspects of survey development, including sample frame development, statistical design of the survey sample, questionnaire design and testing, data collection, sampling and coverage errors, non-response analysis, imputation of missing data, weights and variance estimates. IHS surveys follow guidelines and policies set forth in the Paperwork Reduction Act and other regulations related to the conduct of government surveys. IHS also prepares information products using data produced or maintained by other federal agencies and established survey organizations. All such external survey and administrative data used by IHS are produced using generally accepted methodologies.

   Where samples from administrative data files are employed for analysis, sound statistical methods are employed to develop samples. Staff producing statistical publications are knowledgeable about the content, structure and limitations of the administrative data files and maintain working relations with staff who create, update and maintain these files.

6. Use sound analytic techniques:

   Analytical reports are prepared using a variety of analytical techniques from simple tabulations and descriptive summary statistics to multivariate statistical methods and econometric models. Analytical techniques are reviewed for their appropriateness to the data and the analysis being conducted and are clearly identified in reports.

   IHS utilizes several simulation models to make estimates of the effects of demographic and economic trends and legislative and policy options on Indian Health programs and beneficiary populations now and in the future. Some simulation models have been or are being developed within IHS. Others are being developed under contracts. All contracts to develop simulation models provide for detailed documentation that describes the goals and objectives of the model, the data sources being used and the methodologies and assumptions employed. Contract reports are being made available on the Internet. Documentation is available for some simulation models that have been developed within IHS and will be prepared for others.

   All simulation models are extensively tested and reviewed within IHS to verify that the computer programs that were developed to implement the model conform to the stated objectives. Where appropriate, historical simulations are developed to evaluate the success of a model in producing reasonable projections.  Simulation models are based on IHS's and its contractor's best judgments of current and future behavioral relationships and methods of projecting key program outcomes. These models are periodically updated to reflect input from internal and external reviews and research findings on behavioral relationships. These updates are also documented.

7. Preparation of statistical data products:

   Estimates in statistical data products are prepared from representative samples of IHS data files and from reliable external data sources. Procedures for collection of data from administrative files and linking external data files to administrative data files are prepared using accepted statistical methods. Output is reviewed by knowledgeable staff within the originating component and across components as appropriate. Estimates are compared to prior year estimates and estimates from other sources to ensure consistency, reasonableness and reliability. All data sources used in producing statistical data products are identified, either for the publication as a whole or for individual tables. Documentation includes specification of variables used, definitions of variables when appropriate, sampling errors and disclosure avoidance rules or techniques.

8. Preparation of analytical reports and policy studies:

   Information contained in analytical reports and policy studies is based on estimates derived from reliable administrative data files and external data sources. Analysts apply sound statistical and analytical techniques and are knowledgeable about the data sources and models being used. All data sources are identified. When analyses are based on simulation model projections, the assumptions used to produce the projections are also identified as well as the rationale for the assumptions used and the impact of using alternative assumptions. All analytic reports and policy studies are reviewed by technically qualified staff to ensure that analysis is valid, complete, unbiased, objective and relevant. Analytic reports and policy studies that are considered to be more technically complex are also reviewed by subject matter experts outside of the originating component to provide additional perspective and expertise.

9. Editorial review for accuracy and clarity of information in publications:

   All information products are edited and proofread before release to ensure clarity and coherence of the final report. Text is edited to ensure that the report is easy to read and grammatically correct thoughts and arguments flow logically, and information, is worded concisely and lucidly. Tables and charts are edited to ensure that they clearly and accurately illustrate and support points made in the text, and include concise but descriptive, titles. Tables and charts clearly indicate the unit of measure and the universe being examined and all internal labels (column heads, row stubs, and panel headings) should accurately describe the information they contain. All changes made to a manuscript during the editing process are checked by a proofreader and reviewed and approved by the author.

   If an error is detected before an initial mailing, IHS includes an errata notice with the mailing. If the mailing has been sent out, IHS issues an errata sheet with all subsequent publications, and as appropriate, sends the errata sheet to all those who received the initial notice. Errata notices are put on the first page of the Web version to inform both new and repeat site visitors about the mistake, and the corrected version of the document is posted on the Web.

10. Integrity:

    Integrity refers to the security of information from unauthorized access or revision to ensure that the information is not compromised through corruption or falsification. To ensure the integrity of its administrative information, IHS has in place rigorous controls that have been identified as representing sound security practices. IHS is highly protective of the confidentiality of information it holds through its policies and practices. IHS has in place programs and policies for securing IHS resources as required by the Government Information Security Reform Act (P.L. 106-398, Title X, Subtitle G). These security procedures address all major components of information security and apply to all IHS operating components.

    IHS is subject to statutory requirements to protect the sensitive information it gathers and maintains on individuals. These requirements are contained in the following documents:

    • Privacy Act of 1974
    • Computer Security Act of 1987
    • Office of Management and Budget (OMB) Circulars A-123 A-127 and A-130
    • Government Information Security Reform Act
    • Federal Managers' Financial Integrity Act (FMFIA) of 1982
    • Section 6103 of the Internal Revenue Code
    • Section 1106 of the Social Security Act
    • IRS Tax Information Security Guidelines for Federal, State and Local Agencies

VI. Agency Administrative Complaint Procedures

1. Responsibility of the Complainant

   To seek a correction of information disseminated by the IHS, individuals shall follow the procedures described below.

   1. A complaint or request for review and correction of information shall be in written hard copy or electronic form;
   2. it shall be sent to the agency by mail or electronic-mail (e-mail); and
   3. it shall state that an information quality request for correction is being submitted

   The complaint should contain

   4. a description of the specific material that needs to be corrected including where the material is located, i.e. the publication title, date, and publication number, if any, or the website and web page address (url), or the speech title, presenter, date and place of delivery; and
   5. the specific reasons for believing the information does not comply with OMB, HHS, or IHS guidelines and is in error and supporting documentation, if any;
   6. the specific recommendations for correcting the information;
   7. a description of how the person submitting the complaint is affected by the information error; and
   8. the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.

   Complainants should be aware that they bear the "burden of proof" with respect to the necessity for correction as well as with respect to the type of correction they seek.

   Complaints by mail should be directed to:

   Indian Health Service
   Data Quality Management
   801 Thompson Avenue
   Rockville, MD 20852

   Alternatively, complaints may be e-mailed to Tony.Knedrick@mail.ihs.gov

2. Responsibility of the Agency

   Based on a review of the information provided, IHS will determine whether a correction is warranted and if, so what action to take. The agency will respond to the requestor by letter or e-mail. The IHS response will explain the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. The agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

3. Appeals

   If the individual submitting the complaint does not agree with the IHS's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal should state the reasons why the agency response is insufficient or inadequate. Complainants should attach a copy of their original request and the agency response to it, clearly mark the appeal with the words, "Information Quality Appeal," and send the appeal to the specific agency appeals address.

   The agency program official who resolved the original complaint will not have responsibility for the appeal. The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

VII. Influential Scientific, Financial and Statistical Information

If an agency is responsible for disseminating "influential" information, guidelines for dissemination should include a high degree of transparency about data and methods to facilitate its reproducibility by qualified third parties. Information is considered influential if it will have a substantial impact on important public policies or important private sector decisions. Since much of IHS's actuarial, statistical and analytical information products potentially have an impact on important public policies, IHS's information that is subject to section 515 should be highly transparent and capable of being reproduced by qualified persons.

IHS's guidelines call for identification and documentation of data sets used in producing estimates and projections and clear descriptions of methods used to produce estimates and to develop model projections to make its results as transparent as possible. Many estimates and projections included in IHS information products are not directly reproducible by the public because the underlying data sets used to produce them are confidential. However, some statistical publications that are based on publicly available data and whose programs are made available on request are fully reproducible by the public. And IHS is in the process of developing public-use versions of several data files, which will increase the reproducibility of estimates and projections to the extent possible while still protecting confidentiality. Some estimates and projections may not be easily reproduced by third parties due to the complexity and detail of the methods and data. In these cases greater emphasis is placed on periodic review by outside panels of technical experts.

IHS also achieves transparency through wide dissemination of its information. Most reports and other data products are available both as printed and electronic documents. They are announced on the IHS web site and most electronic versions can be accessed and downloaded directly from the IHS web site. All documents posted on our Web site since year 2000 are section 508 compliant making information available to an audience that includes persons who have a visual impairment and read online using assistive technology.

I. National Institutes of Health

These guidelines were developed to implement the Office of Management and Budget (OMB) Guidelines and the Department of Health and Human Services (HHS) guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies, hereafter referred to as the OMB Information Quality Guidelines and the HHS Part I: Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated to the Public. This report provides the information quality guidelines for the National Institutes of Health (NIH), and explains how these guidelines will ensure and maximize the quality, objectivity, utility, and integrity of information, including statistical information, disseminated by NIH to the public. This report also details our administrative mechanisms for allowing affected persons to seek and obtain appropriate correction of information maintained and disseminated by the NIH.

The OMB Information Quality Guidelines can be found in the Federal Register, September 28, 2001 The guidelines apply primarily to the dissemination of substantive information (e.g., scientific reports, articles, studies, summaries, speeches, official expert opinions, brochures, statistical information, or compendiums) rather than information pertaining to basic agency operations. Such information can be in any media -- printed, electronic, audiovisual, and the like -- and must be authored or issued by the agency or its contractors, and represent our view.

NIH will ensure that disseminated information meets the standards set forth in the OMB, HHS, and NIH guidelines. It is NIH's policy to ensure and maximize the quality, objectivity, utility, and integrity of the information it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. The quality assurance process begins at the inception of the information development process. NIH is committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination. Each federal agency is already required to demonstrate the "practical utility" of a proposed collection of information in its Paperwork Reduction Act (PRA) submission, i.e., for draft information collections designed to gather information that the agency plans to disseminate. NIH will demonstrate in its PRA clearance packages that each such draft information collection will result in information that will be collected, maintained, and used in a way consistent with OMB, HHS, and NIH information quality guidelines. NIH intends to make use of the PRA clearance process to help improve the quality of information that we collect and disseminate, and to ensure that it complies with all applicable guidelines. The standards to which NIH shall adhere include the following:

• Information should be objective in substance and presentation. Objectivity means ensuring that information is accurate, reliable, and unbiased and that information is presented in an accurate, clear, complete, and unbiased manner. If analytic results have been subject to formal, independent, external peer review, the information may generally be presumed to be of acceptable objectivity. However, this presumption is refutable based on a persuasive showing by the petitioner in a particular instance. As described in more detail in Section V, NIH works to maintain objectivity through existing review and clearance procedures, and the peer review of disseminated information.
• Information should be responsive to its intended users, including the public. NIH strives to stay informed of user needs, through user feedback, consultation with advisory committees and peer review groups, and conference participation. Appropriate public access to government information and data play a useful role in improving the overall quality of information disseminated by federal agencies.
• The integrity of information should be protected. As described in more detail in Section V.3, NIH ensures the integrity of its data and information products through the enforcement of rigorous controls that protect against unauthorized access, revision, or corruption. Some of the controls used at NIH include access control, user authentication, encryption, access monitoring, provision of unalterable electronic content, and audit trails.

I. Agency Mission

Founded in 1887, today NIH is one of the world's foremost medical research centers, and the Federal focal point for medical research in the U.S. NIH, comprised of 27 separate Institutes and Centers, is 1 of 8 health agencies of the Public Health Service, which, in turn, is part of the U.S. Department of Health and Human Services (HHS).

Simply described, the goal of NIH research is to acquire new knowledge to help prevent, detect, diagnose, and treat disease and disability, from the most rare genetic disorder to the common cold. The NIH mission is to uncover new knowledge that will lead to better health for everyone. NIH works toward that mission by: Conducting research in its own laboratories; supporting the research of non-Federal scientists in universities, medical schools, hospitals, and research institutions throughout the country and abroad; assisting in the training of research investigators; and promoting communication of medical and health sciences information.

More than 80% of NIH’s funding supports non-Federal researchers working in universities, medical centers, hospitals, and research institutions throughout the country and abroad (collectively referred to as extramural research), and about 10 percent is allocated to in-house research laboratories located on the NIH campus and several off-campus sites (referred to as intramural research). NIH has approximately 18,000 employees, approximately 3,000 of which have doctoral or medical degrees.

It is NIH policy to make available to the public the results and accomplishments derived from the activities that it funds. Therefore, NIH-funded intramural and extramural investigators are expected to make the results and accomplishments of their activities available to the research community and to the public at large, and to effect their timely transfer to industry for commercialization.

1. NIH Policy Issuances

   NIH is organized into Institutes and Centers (ICs), each with its own mission and functions, separate appropriations, and statutory authorities. Although these ICs may have different administrative procedures in place, they operate under the same general NIH policies and requirements. The NIH Policy Manual System is the formal mechanism for issuing NIH policy. The system is comprised of a series of NIH Manual Chapters. It provides an organized, central repository of information that is accessible to all NIH employees. Individual IC's have the flexibility to incorporate the quality and accountability requirements of Federal and NIH guidelines into their own information resource management and administrative practices in the most applicable manner.

2. Responsible Official

   At NIH, the Associate Director for Communications, who is also the Director of the Office of Communications and Public Liaison (OCPL) in the Office of the Director, has the responsibility for implementing the NIH Information Quality Guidelines, while the Office of Science Policy (OSP), will have overall responsibility for implementing the Guidelines, and will work collaboratively with the ICs. OSP is the central office for policy related issues at NIH. As such, they have taken the lead across the NIH for Information Quality, as it relates to NIH programs, issues, and accomplishments to the public and public interest groups and, to a lesser extent, to the scientific community and the medical professions. The OSP, is also the point of contact is between the ICs and the Office of the Assistant Secretary for Public Affairs in the HHS. The OCPL, is the coordinating office or central source for NIH IC matters related to publications, including printing, HHS/PHS/NIH clearance and review procedures, Joint Committee on Printing, U.S. Congress, and Government Printing Office printing and binding regulations, and copyright rules. Among its many activities, the office produces and distributes a number of publications that highlight NIH research results and scientific advances; provides print, radio, and TV coverage of NIH news and activities; produces the NIH Record; and publishes consumer health information, primarily in a newsletter for the press and public entitled The NIH Word on Health. OCPL also supports and coordinates the principal NIH Web site (www.nih.gov) with direct responsibility for several major areas of the NIH home page that address the special needs of healthcare professionals, patients, members of the press, the public, and employees. It manages the NIH Web Coordinating Committee that provides leadership for the design and content of the NIH Web site, including reviewing new Web sites before they are integrated into the structure of the NIH home page; works with other relevant offices and committees in establishing operational standards and guidelines for Web sites at NIH; and manages the responses to electronic mail sent to the NIH home page.

   It is likely that any formal complaint regarding information quality will go first to the IC or Office responsible for originating the information. It is therefore essential that the relevant components of NIH work cooperatively with OCPL and OSP to ensure a timely and appropriate response to any complaints.

   As the lead offices for NIH Information Quality, OCPL and OSP responsibilities include:

   1. Developing policies and procedures to effectively meet the requirements of the OMB Information Quality Guidelines;
   2. Providing information and/or training to NIH staff on their responsibilities in meeting Federal requirements and NIH policies on ensuring the quality of information disseminated to the public;
   3. Assisting in the review of information quality complaints;
   4. Reviewing the proposed IC response for appropriateness, and assisting in finalizing a response;
   5. Establishing a tracking database for complaints, with information on the type of complaint and its disposition and any resolution or corrective action taken;
   6. Submitting an annual report on behalf of NIH to HHS with the number and types of complaints, and the actions taken, in time for the HHS to report to OMB by January 1 (beginning in 2004);
   7. Posting on the OCPL Web site any further clarifications, guidelines, and Frequently Asked Questions (FAQs) about handling NIH information complaints;
   8. Making available examples of typical complaints and appropriate responses collected from IC reports.

II. Scope of Applicability of Guidelines for Agency

The OMB Information Quality Guidelines require NIH to evaluate and identify the types of NIH information that will be subject to the Guidelines. This section identifies the types of information covered by the Guidelines, and also lists the types of information that are exempt. The NIH Office of the General Counsel originally reviewed this information on November 27, 2001 and considered NIH's interpretation to be consistent with the intent of the law.

The pre-dissemination review described in the guidelines only applies to official information (with the NIH imprimatur) that is released on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information. They apply to information in all media — printed, electronic, audiovisual, verbal, and other. The Guidelines focus primarily on the dissemination of substantive information (i.e., reports, studies, summaries) rather than information pertaining to basic agency operations. Information that is disseminated at the request of NIH or with specific NIH approval through a contract or a grant is subject to these Guidelines. Examples are provided below of the kinds of information that the NIH considers to be covered and not covered by the OMB Information Quality Guidelines. Although information that is not covered by the OMB Guidelines are not subject to the new administrative complaint procedures, the information is still subject to the usual NIH internal review procedures for accuracy and quality.

1. NIH Information Covered by the OMB Guidelines

   Scientific research papers, books, journal articles, and similar authoritative materials, unless they have disclaimers alerting the audience that they do not represent official views of the NIH

   Other official reports, brochures, documents, newsletters, fact sheets, electronic documents, and audiovisual productions (i.e., a unified presentation, developed according to a plan or script, containing visual imagery, sound, or both, and used to convey information)

   Editorials, commentaries, letters-to-the-editor, only if NIH staff representing official NIH viewpoints provides them

   Oral information, including speeches, interviews, expert opinions, only if representing NIH's views, official positions, or policies

   Statistical information -- statistical analyses, aggregated information by program, IC, or for NIH, including funding information and histories (by disease, funding mechanism, dollars, and other criteria) -- prepared for public dissemination

   Consensus panel reports and open meetings' proceedings and minutes

2. NIH Information Not Covered by the OMB Guidelines

   National Library of Medicine (NLM) databases or other archival records, Research Portfolio Online Reporting Tools (RePORT), and similar databases

   Documents not authored by the agency and not representing the agency's views, including information authored and distributed by NIH grantees¹

   Information that is limited in dissemination to Government employees or agency contractors or grantees

   Information pertaining to basic agency operations, e.g., information about agency authorities, activities, programs, along with contact information for the public, organizational charts, NIH or IC Directors' Status Reports, solicitations [program announcements (PAs)/requests for applications (RFAs)], receipt and review materials (e.g., summary statements, information for advisory councils or advisory committee members)

   Information intended solely for intra- or interagency use or sharing of Government information

   Responses to requests for agency records under the Freedom of Information Act (FOIA), the Privacy Act, the Federal Advisory Committee Act, or other similar laws

   Information relating solely to correspondence with individuals or persons

   Press releases that support the announcement or give public notice of information that NIH has disseminated elsewhere

   Information intended for public filings, subpoenas, or adjudicative processes involving specific parties (There are well-established procedural safeguards and rights to address the quality of adjudicatory decisions and to provide persons with an opportunity to contest decisions. These guidelines do not impose any additional requirements on agencies during adjudicative proceedings and do not provide parties to such adjudicative proceedings any additional rights of challenge or appeal.)

   Opinions where the agency's presentation makes it clear that what is being offered is personal opinion rather than fact or the agency's views

III. Types of Information Disseminated by NIH to the Public

Each year, NIH components produce 400 or more publications of various types, and about 140,000 static Web pages. All publications that carry the NIH imprimatur, i.e. are considered official NIH publications or releases, must follow NIH policy and procedures for preparation, review, approval, and distribution (see Section V). The types of information disseminated by NIH to the public include the following, however, the OMB guidelines are not directly applicable to all of the information in these categories (See Section II):

1. Program Reviews, Analyses, and Evaluations.

   This category includes research project descriptions [e.g., abstracts of funded grant proposals available through the NIH Computer Retrieval of Information on Scientific Projects (CRISP) database²], bibliographies, collection of abstracts, reviews, and recurring reports. Summaries of research findings are routinely shared with interested parties (e.g., the public, other researchers, the press, Congress). These findings can be released in the form provided by the investigator, or the investigator-supplied information can be used as the basis of a narrative describing research progress in a particular program area. Syntheses of research findings are used for many purposes, including in meeting annual reporting requirements, such as the Government Performance and Results Act (GPRA). Highlights of research findings are posted on the NIH Web site, and can be found in testimonies and speeches by NIH staff in many venues, including annual Appropriations Hearings, presentations on NIH funding opportunities, and literature reviews.

2. Grants and Funding Opportunities.

   The NIH Guide for Grants and Contracts (http://grants.nih.gov/grants/guide/description.htm) is the official document for announcing the availability of NIH funds for biomedical and behavioral research and research training, and disseminating policy and administrative information. Current and past issues of the NIH Guide for Grants and Contracts are available on the NIH Web site, along with other information on grants policy, peer review, award data, research contracts, application forms, and the RePORT database, and links to each of the 27 ICs.

3. Scientific Reports.

   Can be in the form of a book, chapter of a book or textbook, monograph, journal article, proceedings, or the like. These are generally authored or co-authored by NIH staff scientists as part of their official duties, or may be authored by working groups convened by the NIH. Ordinarily first report of any scientific research results or other professional findings is made by publication in a scientific or professional journal; or presentation at a meeting of a professional organization.³

4. Statistical Compendiums.

   Examples of statistical compendiums include annual appropriations by IC, employment data (e.g., numbers of staff and staffing by professional degree), and data books produced by statistical agencies (e.g., Census Bureau, NCHS) under contract to NIH (e.g., Aging World, 65+ in America). Also prominent is the annual table showing research dollars allocated by disease entitled Funding for Research Areas of Interest released by the NIH Budget Office. The estimated spending amounts are self-reported by individual ICs. Although ICs are requested to use consistent methods across years, estimation methods and assumptions across ICs may not be consistent.

5. Guidelines or Authoritative Health Information.

   This type of information is issued after careful review and deliberation of available scientific evidence, usually with the assistance of a panel of outside experts, and is generally associated with a formal meeting or consensus panel specifically convened for the purpose. Prime examples are NIH Consensus Statements and State of the Science Statements issued as part of the NIH Consensus Development Conference program managed by the NIH Office of Medical Applications of Research (see Section V.2.c), and the Report on Carcinogens prepared by the National Toxicology Program at the National Institute of Environmental Health Sciences, NIH (see Section V.2.d).

6. Editorials, Commentaries, Letters-to-the-Editor.

   Only if they are provided by NIH staff representing official NIH viewpoints.

7. Consumer Information.

   NIH provides a number of resources for the general consumer to learn about health conditions, participate in research studies, look up drug information, contact the NIH, find health literature references, and read about special programs. A considerable amount of this information is developed and distributed through IC-established clearinghouses, some of which are required by law. Other sources of consumer information include MedLine Plus, a health database maintained by the NIH's National Library of Medicine; the NIH Research Matters, a newsletter of articles on health maintenance and prevention; A-Z topic index with primary Institute contact; PubMed, a comprehensive database of article titles and abstracts; Clinical Trials database on medical studies around the country; a MEDLINEplus guide to over 9,000 medications; and much more. Other information provided to the public includes Information about NIH, Visitor Information, Job Opportunities, Employee Directory, and FOIA provisions.

8. Science Education Materials and Training Modules.

   NIH provides science education materials as well as training modules for clinical investigators and extramural scientists. The NIH Curriculum Supplement Series are interactive teaching units that combine cutting-edge scientific research discoveries with state-of-the-art instructional materials for grades K-12. Examples of training aids available to extramural researchers include Human Subjects Assurance Training, and various self-instructional guidebooks and videotapes.

9. Press Releases.

   NIH press releases are archived 2 weeks after their release date and made available on the NIH Web site. Interested persons can subscribe to receive these press releases via email.

IV. Types of Dissemination Methods

NIH information is disseminated in many mediums, with the following four being most common:

1. Print --
   publications, books, newsletters, brochures, booklets, pamphlets, and reports.

2. Oral --
   formal speeches, oral presentations, interviews, or commentaries for publication or broadcast; letters-to-the-editor or correspondence likely to result in similar publications.

3. Audio-Visual --
   broadcast scripts, audio or videotapes, and videocasting. The Center for Information Technology (CIT) makes special NIH events, seminars, and lectures available to viewers on the NIH network and the Internet from the VideoCast Web site.

4. Electronic --
   The NIH Web site is the most popular Government Web site after the Internal Revenue Service, and has about 3 million unique visitors per month. The NIH Web site is recognized as one of the most respected and trusted sources for authoritative health information (Consumer Reports, January 2002; Forbes.com review, September 10, 2001; Business Wire, January 29, 2001). The NIH Web site is not just one site, but also a large collection of sites residing on over 150 servers with over 140,000 static pages that are crawled and indexed on public servers. Some areas are updated daily, while others may not be updated for weeks or months.

V. Agency Quality Assurance Policies, Standards, and Processes for Ensuring the Quality of Information Disseminated to the Public

1. Overview

   All NIH documents and audiovisuals must be prepared in accordance with professional and ethical standards, as well as generally accepted standards of good taste. They must be appropriate for dissemination by this Federal agency, and must undergo appropriate review and approval prior to release. NIH must adhere to the laws and regulations applying to publications and audiovisuals, including OMB Information Quality Guidelines, the HHS Printing Handbook, and relevant NIH Manual chapters. NIH efforts to ensure and maximize information quality begin at the preparation stage, and continue through the review and approval stages. Existing NIH policies developed in concert with Federal computer security laws provide appropriate security safeguards to ensure integrity of NIH documents, i.e., ensure that the information is protected from unauthorized access, revision, corruption, or falsification.

   The NIH has many quality control measures embedded in the scientific process to ensure that the information disseminated is of the highest quality. NIH grant applications undergo rigorous scientific review. Scientific journals do not publish articles until they have gone through a similar peer review process. There is a tension inherent in biomedical research between releasing information in a timely fashion and waiting for the peer review process to result in a published article. Sometimes the NIH provides "late breaking news" to the public on research findings prior to publication in scientific journals and prior to peer review by the journals. However, when it does so, there is an internal review process that routinely draws upon external expertise and monitoring/advisory review boards to ensure that information disseminated to the public summarizes the facts as they are currently known, and that appropriate disclaimers are attached.

   The policies and procedures to be followed in the preparation, review, approval, and distribution of NIH information materials, including scientific and professional materials, can be found in NIH Manual Chapter 1183: NIH Publications and Audiovisuals: Preparation, Review, Approval, and Distribution and NIH Manual Chapter 1184: Scientific and Professional Information Presented By NIH Employees: Review, Approval, and Distribution. The procedures currently in place were developed to be sensible, workable, flexible, and timely, and were updated in February 2002 to better articulate OMB, HHS, and NIH information quality guidelines. In the scientific and research context, technical information that has been subjected to formal, independent, external peer review is generally presumed to be of reasonable quality.

   The general principles concerning the responsibilities of the NIH research staff in the collection and recording of data, publication practices, authorship determination, peer review, confidentiality of information, collaborations, human subjects research, and financial conflicts of interest are exemplified in the "Guidelines for the Conduct of Research in the Intramural Research Programs at NIH." These guidelines can be found on the Web (www.nih.gov/news/irnews/guidelines.htm). NIH recognizes the scientific need for replication of findings, and encourages data sharing as appropriate. After publication, the research data, any unique reagents, and any supporting data that form the basis of the communication in question should be made available promptly and completely to all responsible scientists seeking further information. Exceptions may be necessary to maintain the confidentiality of clinical data or if unique materials were obtained under agreements that preclude their dissemination. Investigators should retain research data long enough to allow replication of study results -- in general, 5 to 7 years.

2. NIH Information Review and Approval Policies and Procedures by Type of Information

   The review, approval, and dissemination of substantive scientific information by NIH and/or its ICs require adherence to appropriate clearance procedures set forth in NIH Manual Chapters, internal Web sites, or memos, and are consistent with HHS and OMB guidelines. The originating office is responsible for obtaining the necessary clearances for reproduction and distribution of printed materials and should ensure that written material distributed is appropriate and consistent with HHS policy.⁴

   A document that has obtained publication clearance for paper printing is often posted on the sponsoring IC's Web page for greater accessibility. NIH/IC Web documents derived from IC-approved printed publications should not need additional approvals. NIH/IC Web documents with no print counterpart require content clearance by the appropriate IC office or contact person to ensure that the information observes all applicable requirements governing information for release to the public. These include the requirements provided in NIH Manual Chapter 1183. When IC Web pages are related to more than one IC (e.g., trans-IC publications, special interest groups), the appropriate IC office or contact person for the primary IC responsible for creating the Web page should be notified regarding clearance requirements.

   This section describes NIH procedures and practices in place for review and approval of substantive scientific information that is meant for dissemination primarily to the public, and that NIH considers being subject to the OMB Information Quality Guidelines (see Section II above).

   1. Scientific research papers, books, journal articles, brochures, documents, statistical compendiums, newsletters, electronic documents, audiovisual productions, authoritative health information, and similar materials

      NIH encourages professional dissemination of scientific research and other information on behalf of public health by its employees. Professional and scholarly writing, lecturing, editing, and publishing are an essential part of research, are in the public interest, and bring credit and distinction to NIH and to the employees themselves. In assisting employees to share information about their official and professional activities, NIH seeks to advance scientific knowledge and contribute to professional education. Ordinarily first report of any scientific research results or other professional findings is made by publication in a scientific or professional journal or presentation at a meeting of a professional organization. The choice of the journal or meeting to which reports are offered is the prerogative of the author(s).

      There are many quality control measures embedded in the scientific process to ensure that the information disseminated by NIH employees is of the highest quality. Publications or presentations by NIH employees are expected to meet high standards of quality, make a substantial contribution to the field, and contain sufficient information for the informed audience to assess its validity.

      To ensure and maximize the quality of information disseminated by NIH employees, any non-extemporaneous presentation (written or electronic) by an NIH employee on a subject related to his/her NIH duties must be reviewed and approved through an internal NIH process prior to submitting for publication consideration. With few exceptions, non-extemporaneous oral presentations on health policy or practice, or presentations with policy implications, must also be cleared in advance. Manuscripts intended for publication are customarily subjected to an external peer review process directed by the interested publisher, volume editor, or journal editor.

      Publication or oral presentation of scientific and professional information by individual employees must conform to applicable laws and regulations, including OMB Information Quality Guidelines, and the HHS Standards of Conduct Regulations. Customary professional practices impose certain constraints on the degree to which NIH employees may be identified with the results of research and development work, including that obtained in collaboration with extramural grantees.⁵

      All Institutes have either formal or informal internal operating procedures for identifying printing requirements and tracking publications. These procedures come in a variety of forms such as policy issuances, internal Web sites, memos, or annual requests for printing requirements. In addition, the concept clearance process for new publications is often the vehicle that Institutes use to track the development of a new publication and to identify its attendant printing requirements. The requirements for clearance of prospective publications are contained in NIH Manual Chapter 1183. These requirements state that each prospective publication must be cleared through the Communications Office within the originating NIH component and then be approved by the Office of Communications and Public Liaison, OD/NIH and the Office of the Assistant Secretary for Public Affairs (OASPA), HHS.

      In brief, the NIH Manual Chapter 1183 requires that any official publication (including book, bibliography, chapter of a book or textbook, booklet, brochure, collection of abstracts, fact sheet, house organ, index, leaflet, manual, monograph, newsletter, pamphlet, review, periodical, proceeding, recurring report, statistical compendium, Internet document, audiovisual, or the like), prepared by any NIH component directly or through a contract must be sent for HHS clearance through the Editorial Operations Branch, using form HHS-615, Publication Planning and Clearance Request. This clearance requirement does not apply to publication of articles in journals. The authority to permit the initial publication of articles written by NIH employees in privately published journals, encyclopedias, and textbooks can be delegated according to NIH Manual Chapter 1130 (Delegation of Authority).

      All NIH Audio/Visual projects and exhibits must be cleared through OASPA, whether produced in-house or under contract. To obtain clearance for all NIH audiovisual products, including exhibits, Form HHS 524A must be completed. It can be obtained on the Web (http://www.nih.gov/icd/od/ocpl/resources/audiovisual.htm) and must be filed with the Office of the Assistant Secretary for Public Affairs and approved before actual production may begin. If the cost exceeds $50,000, a written evaluation plan is required. If more than $100,000 is involved, a written evaluation and formal message testing are required. No subsequent change in terms, dollar amounts, conditions, or additions can be made to the product without written approval of OASPA.

      Statistical compendiums, including statistical analyses, aggregated information by program, IC, or for NIH, including funding information and histories (by disease, funding mechanism, dollars, or other criteria), do not require approval by the Office of the Director, NIH. However, the Director of the originating IC is required to determine that the data conform to the accepted quality standards, and if applicable, that the reported statistics be substantially reproducible (see NIH Manual Chapter 1183).

      In general, any writing by an NIH employee on a work-related subject, whether intended for electronic or print publication, or for oral delivery, must be prepared according to accepted NIH standards of quality, reviewed for substantive content, and administratively approved. The purpose of the NIH clearance process is to improve the quality of information, and to ensure the accuracy, objectivity, utility, and validity of information. NIH Manual Chapter 1184, states the policy and procedures to be followed in the review, publication, and distribution of scientific, technical, and other professional manuscripts and speeches by NIH employees. IC Directors (or their delegates) are responsible for establishing and maintaining controls to ensure competent and timely clearance of professional writing and presentations by developing procedures appropriate to the type of information. They are also responsible for maintaining files of requests for approval and actions taken. Individual ICs may determine how best to meet these requirements.

      Written presentations by intramural scientists are reviewed and approved by Laboratory/Branch Chiefs and sometimes by Scientific Directors. The intramural approval process also ascertains that all animal, human subjects, and technology transfer requirements are met, that major press and policy implications are noted, and that at least one supervisory scientist finds the work to be of merit. (See the Intramural Research Sourcebook at www1.od.nih.gov/oir/sourcebook/oversight/pub-clear.htm).

      Materials requiring review in the Office of the Director, NIH, should be approved by a designated review officer within the originating IC, or by a person in a supervisory relationship to the author, prior to submission to the Office of the Director, NIH. No such preliminary review is required for writing by an IC Director. Any statement, commentary, or discussion of Federal policies or practices related to the employee's position or duties that might be construed as reflecting an official position by NIH, HHS, or the Federal Government must be approved in the Office of the Director, NIH.

      For scientific and technical documents, the scientific community recognizes peer review as the primary means of quality control. According to OMB Information Quality Guidelines, material subjected to formal, independent, external peer review may generally be considered to be of acceptable objectivity. However, this presumption of objectivity is refutable based on a persuasive showing to the contrary by a complainant in the particular instance. The single most important determinant of a scientific review group's competence and credibility is its members. Reviewers must have scientific excellence (as demonstrated by their grant and publication records, and academic degrees and honors), and must merit respect in the scientific community. They must possess a wide breadth of expertise, be fair and objective, and should not be influenced by inappropriate personal interests (competition, scientific bias, personal antagonisms, and other irrelevant factors.). Reviewers should review materials for propriety, accuracy, completeness and quality (including objectivity, utility, and integrity).

      Consistent with HHS Standards of Conduct (73.735-705 Writing and Editing), employees are encouraged to engage in outside writing and editing when such activity is not otherwise prohibited. If the writing or editing activity is related to the employee's official duties or other responsibilities and programs of the Federal Government, the employee must (i) make no mention of his or her official title or affiliation with the Department, or (ii) use his or her official title or affiliation with the Department and a disclaimer, or (iii) submit the material for clearance within the operating component, under procedures established by the component. When clearance is denied at any lower level, the employee shall have recourse for review up to the head of the principal operating component. This clearance will show there are no official objections to the activity and the employee may then use his or her official title or affiliation usually without a disclaimer. Except where the requirement for disclaimer is waived as a result of official clearance, disclaimers shall be used in all writing and editing related to the employee's official duties or other responsibilities and programs of the Federal Government: (i) in which the employee identifies himself or herself by official title or affiliation with the Department, or (ii) when the prominence of the employee or the employee's position might lead the public to associate him or her with the Department, even without identification other than name. Disclaimers shall read as follows unless a different wording is approved by the Assistant General Counsel, Business and Administrative Law Division, Office of the General Counsel: "This (article, book, etc.) was (written, edited) by (employee's name) in (his or her) private capacity. No official support or endorsement by (name of operating component or of Department) is intended or should be inferred."

      Normally, the need for a disclaimer is eliminated through the clearance process. However, a disclaimer may still be needed even after official clearance to clarify that the presentation should not be construed as necessarily representing NIH views, and/or to distinguish the status of information (e.g., preliminary, based on partial data set). The Department's regulations (Standards of Conduct) to which the NIH subscribes, require that disclaimers be used in all unofficial writing and editing related to the employee's official duties and/or affiliation with programs of the Federal Government in which the employee's identification with NIH is to be shown, can be inferred, or is well-known.

   2. Oral information, including speeches, interviews, expert opinions, only if representing NIH views, official positions, or policies

      Any statements, comments, or discussion of Federal policies or practices that are relevant to the employee's position or duties, draw conclusions, advocate or oppose professional practices or positions on subjects related to NIH duties, or might otherwise be construed as reflecting an official position by NIH, HHS, or the Federal Government, are covered by the OMB Guidelines, and must be approved in the Office of the Director, NIH.

      An NIH employee may respond orally to questions and requests for information from any source, including the news media, without prior review and approval but must adhere to internal IC guidelines for informing the IC Information Officer, Congressional Liaison Officer, or other appropriate official about the nature of the information to be discussed. An employee may appear as a member of a discussion panel or seminar and on radio, television, and Web broadcasts without prior approval if the appearance does not require a manuscript or written text or statement, and if there is no conflict with NIH Policy as provided in Manual Chapter 1184. An employee should limit his/her statements and responses to subjects about which he/she has official knowledge and should present only official HHS and NIH positions in discussion of policy matters.

      No review or approval is required for nonofficial and private writing, speaking, and publishing by an employee unless his/her NIH employment is likely to be regarded as influencing the content.

      NIH employees are responsible for the statements they make, regardless of whether they have been cleared. If one presents material that requires clearance but that has not been cleared prior to presentation, then the employee must inform the audience of the personal or unofficial nature of his or her views. An example of an appropriate disclaimer follows:

      "This material is presented from my own perspective, and should not be taken as representing the viewpoint of the Department, NIH, or [IC]."

      NIH employees shall not identify themselves as NIH employees in unofficial materials prepared for dissemination to nonprofessional audiences, such as a letter-to-the-editor. These materials must be reviewed prior to presentation in the Office of the Director, NIH, if an employee's identification with NIH is to be shown, can be inferred, or is well known.

   3. NIH Consensus Development Program

      The NIH Office of Disease Prevention (ODP) manages the NIH Consensus Development Conference (CDC) Program, the focal point for evidence-based assessments of medical practice and state of the science on behalf of the medical community and the public. Under this program, ODP organizes major conferences that produce Consensus Statements and State of the Science Statements on controversial issues in medicine important to healthcare providers, patients, and the general public. NIH Consensus Statements and State of the Science Statements are disseminated widely, and more than 120 NIH Consensus Statements and State of the Science Statements have been issued since the program's inception in 1977. Organizationally, ODP is under the Associate Director for Disease Prevention in the Office of the Director, NIH, and works closely with NIH Institutes, Centers, and Offices to assess, translate, and disseminate the results of biomedical research that can be used in the delivery of important health services to the public.

      An NIH Consensus Statement is a report evaluating scientific information on a given biomedical or public health intervention with the purpose of resolving a particular controversial issue in clinical practice. Each NIH Consensus Statement answers a series of four to six questions concerning efficacy, risk, and clinical applications, and recommends directions for future research, and is the product of an NIH Consensus Development Conference. NIH Consensus Statements synthesize new information, largely from recent or ongoing medical research, that has implications for reevaluation of routine medical practices. They do not give specific algorithms or guidelines for practice.

      NIH Consensus Statements are written by broad-based, independent panels of non-Federal, non-advocate individuals knowledgeable in the field of medical or public health science under consideration. The makeup of each panel represents various sectors of professional and community life and typically includes research investigators, healthcare providers, methodologists, and a public representative.

      Following circulation of the draft statement to the conference audience for comment, the panel resolves any conflicting recommendations and releases a revised statement at the end of the conference. The Web site for the Consensus Development program can be found at: consensus.nih.gov.

      If a suggested topic does not have an adequately defined and available base of scientific information, conference planning and implementation may still proceed. However, rather than being designated a Consensus Development Conference, the conference will be designated as a State of the Science Conference. NIH State of the Science Conferences and Workshops generally adhere to the NIH CDC format because the process is useful for evaluating complex issues. Usually, speakers present findings or perspectives on the issue. The public is invited to address questions to the speakers, and policy implications may be discussed. A report of the findings can emerge in one of a variety of formats including publication in a clinical or scientific journal. The Web site for the State of the Science Statements can be found at: consensus.nih.gov.

      Although it is difficult to quantify their impact, the NIH Consensus Statements and State of the Science Statements are intended to influence important public health discussions on topics affecting or broadly applying to a significant number of people. The severity of the problem (morbidity and mortality) and the feasibility of intervention are key considerations. For example, the program has had measured success in influencing reimbursement policy and specialty organization policy, thereby indirectly affecting physician behavior. Each conference is jointly sponsored and administered by one or more ICs of NIH and by ODP. Depending on the topic, other Federal agencies with biomedical components may join in sponsoring a CDC. In conjunction with each conference, the Agency for Health Care Research and Quality (AHRQ) provides a systematic review of the literature on the conference topic through one of its Evidence-Based Practice Centers.

      Both the Consensus Statements and the State of the Science Statements are independent reports of the convened panel; none are policy statements of the NIH or the Federal Government. However, NIH funds and disseminates the Consensus Reports, and considers these Statements to be subject to OMB's higher standard of substantial reproducibility. These Statements meet the OMB's quality standards because of the balanced, rigorous, and systematic procedures that ODP has in place to develop them. The panel makes available the evidence on which the Consensus Statement is based. If consensus cannot be achieved, minority or alternative views are included. The systematic literature review conducted by AHRQ is published with their explicit methods. The reports are peer-reviewed by expert panels, and posted on the Internet for at least one month for public comment.

   4. Health, Safety, and Environmental Information

      To make environmental health research findings more applicable to human risk assessment, NIH works in partnership with the CDC and the EPA to develop better ways to monitor and assess human exposure to specific chemicals. One of our most visible publications is the Report on Carcinogens (RoC), a congressionally mandated document that lists agents, substances, mixtures or exposure circumstances that are known or reasonably anticipated to be human carcinogens, and to which a significant number of persons residing in the United States are exposed. Responsibility for producing this report has been delegated to the National Toxicology Program (NTP) and the Director NTP also serves as the Director, NIEHS, NIH.⁶ The RoC is a composite of Summary Profiles that describes the carcinogenicity, exposure, and regulatory information for each listing with relevant tables and appendices. The NTP follows a published, multi-step process to review and evaluate selected substances and to develop the NTP’s recommendations to the Secretary, HHS, regarding whether or not the substances should be listed in the report.

      Upon review and approval by the Secretary, HHS, and submission to Congress, a notice of the RoC publication, indicating all newly listed or de-listed agents, substances, mixtures or exposure circumstances is published in the Federal Register, NTP newsletters and web pages and other appropriate publications. The current review process was finalized and released to the public on January 11, 2012 [http://ntp.niehs.nih.gov/go/rocprocess].

      It is important to note that the RoC does not present assessments of carcinogenic risks. Listing of substances in this Report, therefore, does not establish that such substances present carcinogenic risks to individuals in their daily lives. Such formal risk assessments are the purview of the appropriate Federal, State, and local health regulatory and research agencies. However, for each effluent, ambient, or exposure standard established by a Federal agency with respect to a listed substance, the RoC is required to state the extent to which, on the basis of available medical, scientific, or other data, the implementation of such standard decreases the risk to public health from exposure to the substance. This requires quantified information on the extent of protection from cancer that the public receives from established Federal standards. Only in a few instances, where studies of long-term human exposures and cancer incidence in restricted environments are available, can risk be estimated with complete confidence.

      NIEHS and NTP procedures conform to accepted NIH scientific practices where quantitative and qualitative scientific conclusions are based on: (1) The best available science and supporting studies, particularly peer-reviewed studies, conducted in accordance with sound and objective scientific practices; and (2) data collected by accepted methods or best available methods (if the reliability of the method and the nature of the decision justifies use of the data). The NIEHS and NTP make every effort to ensure that the presentation and dissemination of information about environmental health is comprehensive, informative, and understandable.

      Scientific results that can directly affect risk assessment and risk management activities are often published in the NIEHS journal Environmental Health Perspectives (EHP), but do not necessarily represent the viewpoints of NIEHS. EHP also publishes perspectives in the form of editorials, commentaries, reviews, and correspondence, as well as workshop summaries. Workshop summaries are reports by expert scientific committees that include reviews of existing information and that summarize research findings on specific topics, present new information, and recommend methods, courses of action, or further research needs for the scientific community. All scientific articles, including workshop summaries, are subject to rigorous peer review. The criteria for publication are weighted toward scientific quality and environmental significance. A submission is assessed according to its originality, scientific merit, and experimental design; the manuscript is evaluated for its conciseness, clarity, and presentation. These standards are thus consistent with the requirement that the presentation of information on risk effects be comprehensive, informative, and understandable. EHP also addresses certain ethical problems during the review process and requires assurances that all human and animal subjects have been treated humanely and with regard for the alleviation of suffering. The review also considers scientific integrity as part of the process.

   5. NIH Clearinghouse Information

      Clearinghouses often serve as the public's point of contact and access to information about IC programs, conferences, and research activities. At NIH, clearinghouses have been contracted to provide varying levels of service, including development and distribution of fact sheets, information packages, and publications; storage of materials; conducting outreach and promotion; and performing training and quality control for the clearinghouse staff. Some clearinghouses respond to inquiries about particular diseases or conditions, ranging from information about available patient and professional education materials to statistical data. Clearinghouses are challenged to ensure accuracy and reliability of information, while continually striving to improve performance and response times. Some clearinghouses also wrestle with how to determine which organizations are worthy of referral when customers need information that is not available at the clearinghouse and how to avoid implying endorsement. Clearinghouse inquiries may also be answered by searching the Combined Health Information Database (CHID), an NIH/CDC database that provides bibliographic references of both NIH and non-NIH materials on various health topics. This database represents a shared data archive, and as such is not covered by the new OMB guidelines.

      The NIH Manual chapter 1183 requires that official materials or information prepared by any NIH component directly or through a contract must be sent for HHS clearance through the Editorial Operations Branch, using form HHS-615, Publication Planning and Clearance Request. This clearance requirement does not apply to publication of articles in journals. Information developed by a clearinghouse for an NIH IC is subject to the OMB Guidelines. See NIH Manual Chapter 1183 for further information regarding this requirement.

      At NIH, there are essentially three types of materials being disseminated through information clearinghouses to the public: (1) Materials produced by NIH staff or contractors that undergo usual NIH review and approval processes; (2) materials produced by NIH grantees that are subject to policies and procedures in the Public Health Service (PHS) Grants Policy Statement; and (3) other materials not produced by NIH but available through libraries, whether in print or in electronic format, with appropriate disclaimers attached. Virtually all NIH ICs direct their clearinghouses to distribute only materials produced by the IC or other NIH ICs or Federal agencies. Non-Federal materials typically undergo careful IC scientific review before they are authorized for dissemination by the clearinghouse, and those materials are accompanied by appropriate disclaimers.

3. Procedures to Ensure the "Integrity" of Information

   NIH has developed World Wide Web (WWW) Guidance (April 15, 1998), which is available on the Internet (http://ocio.nih.gov/docs/policy/guideli2.html). Each IC has a designated IC contact/reviewer for information and approvals related to developing Web pages and operating a new Web server. The list of IC contacts/reviewers is available on the website (www.nih.gov/employee/weblist.htm). NIH/IC Web page creators periodically review material on the Web page to determine whether it is accurate and up to date. Information, particularly time-sensitive information, should be posted as soon as possible. Web page creators are expected to promptly update or remove out-of-date information.

   Unless noted otherwise, it is safe to assume that information posted on public Web sites within the "NIH.GOV" domain is considered to be "in the public domain." As such, others are free to establish links to NIH online resources. In establishing such links, NIH requests that others avoid creating the impression that NIH is endorsing or promoting any particular product or service. In the same vein, any outside link to an external resource from an NIH Web site needs to be examined on a case-by-case basis. In general, the Web developer of each site determines when links to outside entities are justified.

   NIH Web managers are urged to exercise caution when linking to non-NIH, external websites. Professional judgment should be used to weigh the benefits against the possible risks of linking to other resources. In particular, links to sites providing medical and scientific information needs to be on par with the standards used at NIH to ensure the credibility of the information offered there. Steps should be taken to ensure that such links do not give the impression of endorsing the organizations we link to. NIH/IC Web pages containing links to external Web pages not located on NIH servers should include a link to a statement that releases NIH from responsibility for the material included in the external Web page. Again, it is important to avoid giving a user the impression that NIH is endorsing information or a commercial product described in an external site. Disclaimers on copyright, endorsement (general and external links), liability, and medical information are also used, as appropriate, for individual IC Web sites.

   The IC designates a main office or contact person for information and approvals related to Web pages and the operation of Web servers. NIH personnel, contractors, and other authorized users of NIH networks must notify this office or contact person prior to setting up a Web server. Information about appropriate security measures regarding Web Servers is available at: http://ocio.nih.gov/security/sec_policy.html. The IC or the OD Information Office can provide detailed information on required approvals including both NIH and IC-specific policies relating to publication of documents. NIH/IC documents derived from IC-approved printed publications should not need additional approvals (see Section V.2).

   The NIH Center for Information Technology (CIT) is charged with providing, coordinating, and managing information technology for NIH, and with advancing computational science. In terms of computer security, CIT has three distinct objectives: Confidentiality -- ensuring that there is no deliberate or accidental improper disclosure of sensitive automated information; integrity -- protecting against deliberate or accidental corruption of automated information; and availability -- protecting against deliberate or accidental actions that cause automated information resources to be unavailable to users when needed. Information is accorded protection against disclosure, alteration, loss, or destruction based on the degree of sensitivity.

   CIT staff use appropriate safeguards to protect data from improper disclosure by backing up critical data periodically, and, if a security incident occurs, by following proper incident response procedures. In 1994, CIT adopted a security incident response policy and procedures statement that establishes the responsibilities of CIT staff in responding to and reporting computer security problems. Supervisors are responsible for ensuring that employees, both Government and contractor, observe all security requirements, and that employees receive appropriate security training.

   CIT has instituted a structured management control review process that applies throughout the system life cycle. Risk analyses are conducted to strike a balance between an acceptable level of risk and the costs and inconvenience associated with safeguards. A system recertification/accreditation must be conducted at least once every 3 years. Additional information about NIH computer security measures can be found in The Computer Security Handbook of CIT (http://ocio.nih.gov/security/sec_policy.html).

VI. Agency Administrative Complaint Procedures

NIH has developed administrative procedures to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS, and NIH guidelines (See NIH Manual Chapter 1185). Additional guidance on appropriate responses can be obtained from the NIH website (http://ospa.od.nih.gov/infoquality.html), particularly if the complaint involves a policy statement or official position. NIH established this website to advise information consumers of the agency's information quality guidelines, the process to submit a request for correction, information needed by the requestor, and a description of the complaint adjudication process.

The resolution process addresses the valid needs of the complainant without disrupting NIH processes. Complainants should be aware that they bear the "burden of proof" with respect to the necessity for correction, as well as with respect to the type of correction they seek. In making a determination of whether or not to correct information, NIH may reject claims made in bad faith or without justification, and is required to undertake only the degree of correction that is appropriate for the nature and timeliness of the information involved.

1. Responsibility of the Complainant

   To seek a correction of information disseminated by the NIH or its components, an individual should submit or mail the request to the disseminating office (contact information for individual ICs will be made available through the NIH Information Quality website), or submit the request by electronic mail (email) to InfoQuality@mail.nih.gov or InfoQuality@od.nih.gov or mail the complaint to:

   Office of the Director
   Office of Science Policy
   National Institutes of Health
   Building 1, Room 218
   9000 Rockville Pike
   Bethesda, MD 20892

   The request should state that an information quality request for correction is being submitted, and should provide the following information:

   • A detailed description of the specific material that is proposed for correction, including where the material is located, i.e., the publication title, date, and publication number, if any, or the website and web page address (URL), or the presentation, presenter, date and mode of delivery;
   • The specific reasons for believing that the information does not comply with OMB, HHS, or NIH guidelines and is in error, and supporting documentation, if any;
   • Suggested recommendations for what corrective action(s) should be taken;
   • A description of how the person requesting the correction is affected by the information error; and
   • Complete contact information for the requestor, including name, mailing address, telephone number, e-mail address, and organizational affiliation, if any.

2. Determination of Appropriate Response

   Requests for correction should be handled primarily by the originating IC Director or designee (e.g., Scientific Director, Laboratory or Branch Chief). IC Directors are responsible for establishing and maintaining procedures to ensure that requests are properly addressed, that an objective and qualified review of the merits of the request is undertaken, and that an appropriate response is provided in a timely manner. The procedures may include forming a review committee or equivalent, and shall allow for response by the originating and contributing authors, as well as input from the IC Communications Director.

   A complaint about information originating from a division or office within the Office of the Director (OD), NIH, should be addressed by the director of the division or office, with input from the Office of Science Policy and Office of Communications and Public Liaison (OCPL), OD, NIH.

   If more than one IC was involved in releasing the information, the IC of the lead NIH author should take primary responsibility for coordinating a response.

3. Appropriate Responses

   Based on a review of the information provided, the responding office should determine whether a correction or clarification is warranted and if so, what action to take. Agencies may choose not to change claimed defects that are frivolous or unlikely to have substantial future impact. If NIH determines that action is warranted, NIH may respond in any of the following ways:

   • Provide a clarification by personal contact via letter or telephone;
   • Issue a written retraction or clarification, which can be accomplished through a press release, mass mailing, or some other reasonable method that corrects a widely disseminated error or addresses a frequently raised complaint;

   Suspend further dissemination of the information in question;

   • Refer complainant to the underlying data if the data are available in a public archive;
   • Arrange for an independent reanalysis of the data by NIH or a mutually acceptable third party if the data are not publicly available, and the complaint involves "influential scientific or statistical information." Complainants must agree to pay the costs of reanalysis or the process terminates.
   • Work with the grantee institution to respond to the complaint, when it involves research from a grantee. Complaints must be about information derived from a project that is supported in whole or in part with Federal funds under a new or competing continuation grant awarded after April 17, 2000, and that is cited officially by a Federal agency in support of an action that has the force of law, such as a new regulation or administrative order. (If not available elsewhere, these types of data can be obtained from a FOIA Coordinator for the granting IC, see www.nih.gov/icd/od/foia/coord.htm. Additional information about NIH FOIA procedures, including requests, appeals, and fees, is available at www.nih.gov/icd/od/foia/. For NIH guidance on OMB Circular A-110 see grants.nih.gov/grants/policy/a110/a110_guidance_dec1999.htm).

   The response to the complainant will be by letter or email, and should explain the findings of the NIH review of the merits of the complaint, and the actions to be taken, if any. The response should consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response should also describe how the complainant may request reconsideration. NIH will respond to all requests for corrections within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the responsible official will inform the complainant that more time is required and indicate the reason for the delay and an estimated decision date.

   Whether or not corrective action is warranted, the response to the complainant should describe the preparation, clearance, review, and approval process prior to the information being disseminated, including any specifics about the peer review process to support the rigor and objectivity of the review, e.g., the breadth and depth of experts, community input, and message testing. The response should also reiterate any disclaimers that accompanied the information when it was first released.

   Additional guidance on appropriate responses can be sought from the OSP, or OCPL, OD, NIH, particularly if the complaint involves a policy statement or official position.

4. Reporting Requirements

   The IC (or OD office) receiving a request for information correction is required to enter the complaint into the NIH tracking database for this purpose. The IC is encouraged to contact OSP, OD, NIH to discuss the nature of the complaint, and to provide a preliminary assessment of whether the complaint is legitimate. Among the criteria to be used:

   • The information is considered official NIH information, i.e., approved through the NIH clearance process and intended to represent the views of NIH.
   • The information is substantive (i.e., reports, studies, summaries) rather than pertaining to basic agency operations.
   • The information was disseminated on or after October 1, 2002
   • The information is not exempt according to Section C (Applicability) of this chapter.
   • The complainant is someone who may benefit or be harmed by the disseminated information. This includes persons who are seeking to address information about themselves as well as persons who use information.

   If the IC determines that corrective action is warranted, the IC shall forward the following information to the OSP and OCPL, OD, NIH without delay:

   • A copy of the complaint.
   • A list of the relevant contacts within the NIH IC or OD office, including the names of those most knowledgeable about the information in question.
   • A draft letter response with any supporting documentation.

   OSP and OCPL responsibilities include:

   • Assisting with the review of the complaint for legitimacy. If the complaint is later determined not to be legitimate, the IC must provide a clear explanation of the rationale for that determination to the complainant.
   • Reviewing the proposed IC response for appropriateness, and assisting in finalizing the response.
   • Maintaining a tracking database of complaints, including information on their disposition and any resolution or corrective action taken.
   • Submitting an annual report on behalf of NIH to the Department of Health and Human Services (HHS) with the number and types of complaints, and the action taken, in time for the HHS to report to OMB by January 1 (beginning in 2004).
   • Posting on the OSP Web site any further clarifications, guidelines, and Frequently Asked Questions (FAQs) about handling NIH information complaints.
   • Making examples of typical complaints and appropriate responses available to ICs.

5. Appeals

   If NIH denies a request for correction, the complainant may send within 30 days of receipt of the agency's decision a written request for reconsideration. The request should state the reasons for the appeal and may be sent as hard copy or electronically to InfoQuality@od.nih.gov. Requestors should reference the NIH tracking number provided in the NIH response to the original request. If sent by hard copy, requestors should also clearly mark the appeal and the outside envelope, "Information Quality Appeal," and send the appeal to the following address:

   Office of the Director
   Office of Science Policy
   National Institutes of Health
   Building 1, Room 218
   9000 Rockville Pike
   Bethesda, MD 20892

Any office that originally disseminated the information and/or responded to the original complaint should not have responsibility for the resolution of the appeal. If the information in dispute was originally disseminated by the NIH, then an appeal should be addressed to the NIH Director at the address listed above, or sent electronically to InfoQuality@od.nih.gov.

NIH will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason for the delay and an estimated decision date.

VII. Influential Scientific, Financial, and Statistical Information

The OMB Information Quality Guidelines require that "influential" scientific, financial, or statistical information in official Government documents must be based on studies that can be substantially reproduced if the original or supporting data were to be independently reanalyzed using the same methods. "Influential" when used in the phrase "influential scientific, financial, or statistical information" means that the NIH can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important public policies or important private sector decisions, or will have important consequences for specific health practices, technologies, substances, products, or firms." NIH is committed to applying rigorous scientific standards to ensure the accuracy, reliability, and reproducibility of research results.

The reproducibility standard applies to analytic results and not necessarily to the original and supporting data used to produce the analytic results. To facilitate the replication of scientific and other influential information by qualified third parties, NIH continues to encourage the sharing of original data and methods where practicable. After publication, the research data, any unique reagents, and any supporting data that form the basis of any research communication should be made available promptly and completely to any person who seeks further information.

Since the influence and implications of NIH-disseminated information cannot always be fully anticipated, all NIH scientific reports are expected to state clearly how analytic results are generated -- the specific data used, various assumptions, specific analytic methods, statistical procedures, sources of error -- making the analysis sufficiently transparent so as to be capable of being reproduced. NIH advocates the archiving of data where feasible to facilitate the reproducibility of influential information. Exceptions may be necessary to maintain the confidentiality of clinical data or if unique materials were developed or obtained under agreements that preclude their dissemination. Investigators should retain research data long enough to allow replication of study results -- in general, 5 to 7 years. In situations where public access to underlying data is not practicable, NIH shall apply especially rigorous robustness checks to analytic results and document what checks were undertaken.

Examples of the types of information disseminated by NIH that have the potential of being considered influential and that fall within the scope of the OMB Guidelines include:

NIH Consensus Statements

NIH Research Reports

NIH Recommendations about Health Practice or Medical Treatment

NIH Funding for Research Areas of Interest

For scientific and technical documents, the scientific community recognizes peer review as the primary means of quality control. NIH routinely seeks input from qualified peer reviewers of influential materials for propriety, accuracy, completeness, and quality (including objectivity, utility, and integrity) prior to dissemination. Although concerted efforts are made to ensure that influential information be subjected to rigorous peer review and reproducibility specifications, standard operating procedures may be temporarily disrupted under urgent situations, such as when an imminent threat to public health or homeland security is identified.

With respect to health, safety, and environmental information, NIH does not have a mandate to conduct formal risk assessments, which are the purview of the appropriate Federal, State, and local health regulatory and research agencies (see Section V.2.iv). NIH makes every effort to ensure that the presentation and dissemination of information about environmental health is comprehensive, informative, and understandable, and that scientific conclusions are based on: (1) The best available science and supporting studies, particularly peer-reviewed studies, conducted in accordance with sound and objective scientific practices; and (2) data collected by accepted methods or best available methods (if the reliability of the method and the nature of the decision justifies use of the data).

VIII. Special Considerations for Agency Dissemination

Sometimes NIH provides "late breaking news" of urgent import to the public on research findings prior to publication in scientific journals and prior to peer review by journals. Under such special circumstances, NIH may temporarily waive adherence to information quality guidelines. However, when it does so, there is an internal review process that routinely draws upon external expertise and relevant monitoring or advisory boards to ensure that information disseminated to the public summarizes the facts as NIH currently knows them, and that appropriate disclaimers are attached.

IX. References

HHS Standards of Conduct Regulations (45 CFR 73.735-705). Updated October 1, 2000.

Guidelines for the Conduct of Research in the Intramural Research Programs at the National Institutes of Health, 3^rd edition, January 1997 (for a discussion of publication practices and authorship issues.) www.nih.gov/news/irnews/guidelines.htm

The HHS Printing Handbook, September 1998.

NIH Grants Policy Statement, March 1, 2001. grants.nih.gov/grants/policy/nihgps_2001

NIH Instruction and Information Memorandum OER 90-8. NIH Staff (Co-)Authorship of Publications Resulting from NIH Extramural Awards (1184). December 7, 1990.

NIH Manual Chapter 1130 Delegation of Authority, Program: General No. 3, Publish Articles and Results of Scientific Research and No. 4, Availability of Records for Examination or Copying (June 12, 1985) www1.od.nih.gov/oma/manualchapters

NIH Manual Chapter 1183 -- NIH Publications and Audiovisuals: Preparation, Review, Approval, and Distribution, February 27, 2002. www1.od.nih.gov/oma/manualchapters/management/1183

NIH Manual Chapter 1184 -- Scientific and Professional Information Presented By NIH Employees: Review, Approval, and Distribution, February 27, 2002. www1.od.nih.gov/oma/manualchapters/management/1184

NIH Manual Chapter 1185 -- Complaints about NIH Information Quality, forthcoming.

NIH Manual Chapter 6308 -- Acquisition of Printing Requirements at the NIH, February 2, 2002. www1.od.nih.gov/oma/manualchapters/contracts/6308

Office of Management and Budget Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies. Final Guidelines. February 22, 2002 www.whitehouse.gov/omb/fedreg/reproducible2.pdf

OMB, Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies. September 28, 2001. http://www.whitehouse.gov/omb/fedreg/final_information_quality_guidelines.html

OMB, Circular No. A-130, Revised (Transmittal Memorandum No. 4). Management of Federal Information Resources. November 30, 2000.

World Wide Web NIH Guidance. April 15, 1998. http://ocio.nih.gov/docs/policy/guideli2.html

Endnotes

¹ In general, grantees own the data generated by or resulting from a grant-supported project. Special terms and conditions of the award may specify alternative rights, e.g., under a cooperative agreement or if there are shared rights to data. Except as otherwise provided in the terms and conditions of the award, the grantee is free to copyright without NIH approval when publications, data, or other copyrightable works are developed under, or in the course of, work under an NIH grant. For this purpose, "data" means recorded information, regardless of the form or media on which it may be recorded, and includes writings, films, sound recordings, pictorial reproductions, drawings, designs, or other graphic representations, procedural manuals, forms, diagrams, work flow charts, equipment descriptions, data files, data processing or computer programs (software), statistical records, and other research data. Grantees are required to place an acknowledgment of NIH grant support and a disclaimer, as appropriate, on any publication written or published with such support and, if feasible, on any publication reporting the results of, or describing, a grant-supported activity. An acknowledgment shall be to the effect that: "This publication was made possible by Grant Number ________ from _________" or "The project described was supported by Grant Number ________ from ________" and "Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the (name of awarding office or NIH)." For more details, see the NIH Grants Policy Statement.

² RePORTER is a searchable database of federally funded biomedical research projects conducted at universities, hospitals, and other research institutions. The database, maintained by the Office of Extramural Research at NIH, includes projects funded by NIH, Substance Abuse and Mental Health Services (SAMHSA), Health Resources and Services Administration (HRSA), Food and Drug Administration (FDA), Centers for Disease Control and Prevention (CDC), Agency for Healthcare Research and Quality (AHRQ), and Office of Assistant Secretary of Health (OASH). Users, including the public, can use the RePORTER interface to search for scientific concepts, emerging trends and techniques, or identify specific projects and/or investigators.

³ Excluded from this discussion are non-work-related and private writing, speaking, and publishing by an NIH employee, unless the employee's NIH responsibilities are likely to be regarded as influencing the content.

⁴ NIH Manual Chapter 6308 (Acquisition of Printing Requirements at the NIH) sets forth guidelines on how requirements for printing are to be handled by ICs at the NIH, in compliance with Federal printing rules and procedures. Manual Issuance 6308 covers direct acquisition for printing, as well as printing that is a peripheral deliverable in a contract for a larger purpose (e.g., an R&D contract for a study, the results of which are to be published).

⁵ To merit approval for (co-)authorships on publications from extramural awards (including grants, contracts, and other award mechanisms), NIH staff must have played a substantial role beyond normal program officer duties, including the following:
 

• Originating the specific ideas that led to the research activity and manuscript,
• Performing significant portions of the activity, and
• Participating actively in preparing manuscripts.

The conditions allowing NIH staff to be (co-)authors of publications under NIH extramural awards ordinarily arise only from contracts and cooperative agreements, where, by definition, there is substantial programmatic, i.e., scientific-technical, staff involvement. Deviations from these provisions must be approved by IC directors, and only when justified under special circumstances. The Office of Extramural Research, OD, can provide further information and advice on this subject.

⁶ The National Toxicology Program (NTP) was established in 1978 by the Department of Health and Human Services (HHS) to coordinate toxicological testing programs within the Department, strengthen the science base in toxicology; develop and validate improved testing methods; and provide information about potentially toxic chemicals to health regulatory and research agencies, the scientific and medical communities, and the public. NTP's mission is to evaluate agents of public health concern by developing and applying tools of modern toxicology and molecular biology. The NTP is an interagency program consisting of relevant toxicology activities of the National Institutes of Health's National Institute of Environmental Health Sciences (NIH/NIEHS), the Centers for Disease Control and Prevention's National Institute for Occupational Safety and Health (CDC/NIOSH), and the Food and Drug Administration's National Center for Toxicological Research (FDA/NCTR). The NIH's National Cancer Institute (NIH/NCI) was a charter agency; however, the NCI Carcinogenesis Bioassay Program was transferred to the NIEHS in 1981. The NCI remains active in the Program through membership on the NTP Executive Committee.

J. Substance Abuse and Mental Health Services Administration

I. Agency Mission

The Substance Abuse and Mental Health Services Administration (SAMHSA) was established by Congress under Public Law 102-321 on October 1, 1992, to strengthen the Nation's health care capacity to provide prevention, diagnosis, and treatment services for substance abuse and mental illnesses. SAMHSA works in partnership with States, communities and private organizations to address the needs of people with substance abuse and mental illnesses as well as the community risk factors that contribute to these illnesses.

SAMHSA serves as the umbrella under which substance abuse and mental health service Centers are housed, including: the Center for Mental Health Services (CMHS), the Center for Substance Abuse Prevention (CSAP), and the Center for Substance Abuse Treatment (CSAT). SAMHSA also houses the Office of the Administrator, the Office of Applied Studies, and the Office of Program Services.

II. Scope and Applicability of Guidelines for SAMHSA

1. Overview

   SAMSHA will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and SAMSHA guidelines. It is SAMSHA’s policy to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

   The pre-dissemination review described in the guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.

   They apply to information in all media - printed, electronic, audiovisual, verbal, and other. The Guidelines focus primarily on the dissemination of substantive information (i.e., reports, studies, summaries) rather than on the information pertaining to basic agency operations. Information that is disseminated at the request of SAMHSA or with specific SAMHSA approval through a contract or grant is subject to these Guidelines.

2. Covered Information

   Scientific research papers, books, journal articles, and similar materials unless they have disclaimers alerting the audience that they do not represent official views of SAMHSA;

   Other official reports, brochures, documents, newsletters, electronic documents, and audiovisual productions (i.e., a unified presentation, developed according to a plan or script, containing visual imagery, sound, or both, and used to convey information);

   Oral or written information, including speeches, interviews, expert opinions, if representing SAMHSA's official positions, views, or policies;

   Statistical information, including statistical analyses, aggregated information by program, Center or Office, or for SAMHSA as a whole, including funding information and histories (by funding mechanism, dollars, and/or other criteria);

3. Information Not Covered

   1. Documents not authored by the agency and not representing agency's views, including information authored and distributed by SAMHSA grantees¹
   2. Information that is limited in dissemination to Government employees or agency contractors or grantees;
   3. Information pertaining to basic agency operations, e.g., information about agency authorities, activities, programs, along with contact information for the public, organizational charts, SAMHSA or Center Directors' Status Reports, solicitations (for grants or contracts), receipt and review materials (e.g., summary statements, information for advisory councils or advisory committee members);
   4. Information intended solely for intra-or inter-agency use or sharing of Government information;
   5. Responses to requests for agency records under the Freedom of Information Act (FOIA), the Privacy Act, the Federal Advisory Committee Act, or other similar laws;
   6. Information relating solely to correspondence with individuals or persons;
   7. Press releases that support the announcement or give public notice of information that SAMSHA has disseminated elsewhere.
   8. Information for public filings, subpoenas, or adjudicative processes;
   9. Opinions where the agency's presentation makes it clear that what is being offered is personal opinion rather than fact or the agency's views.

III. Types of Information Disseminated

All publications that carry the SAMHSA imprimatur, i.e., that are considered official SAMHSA publications or releases, must follow SAMHSA policy and procedures for preparation, review, approval, and distribution. The types of information disseminated by SAMHSA to the public include the following, however, the OMB guidelines are not directly applicable to all the information in these categories (See Section II):

Scientific Research Papers, Books, Journal Articles, etc. These may be in the form of a book, chapter of a book or textbook, monograph, journal article, proceedings, or the like. These are generally authored or co-authored by SAMHSA staff as part of their official duties and may be authored by working groups convened by SAMHSA. This category also includes grant project description (e.g., abstracts of funded grant proposal available through the NIH Computer Retrieval of Information on Scientific Projects database), bibliographies, collections of abstracts, reviews, and recurring reports. Summaries of grant findings may be released in the form provided by the investigator, or the investigator-supplied information may be used as the basis of a narrative describing a particular program area. These summaries may be used for many purposes, including in meeting annual reporting requirements, such as the Government Performance and Results Act (GPRA). Highlights of findings may be posted on the SAMHSA web site and may be found in testimonies and speeches by SAMHSA staff in many venues, including annual Appropriations Hearings, presentations on SAMHSA funding opportunities, and literature reviews.

Official Reports, Brochures, Documents, and Audiovisual Productions. This category includes:

Summaries and Analyses of Data Collection Activities. Periodically, SAMHSA publishes summaries of its data collection activities, such as the annual Summary of Main findings from the National Household Survey on Drug Abuse and preliminary estimates from the Drug Abuse Warning Network.

Evaluation and Program Findings. SAMHSA sponsors a variety of evaluation and other activities which lead to major publications such as the recently completed National Treatment Improvement Evaluation Study. SAMHSA also publishes information derived from its grant and contract sponsored programs such as the National Estimates of Expenditures for Mental Health and Substance Abuse Treatment.

Grant and Contract Funding Opportunities. The SAMHSA web site is the official vehicle for announcing the availability of SAMHSA funds for discretionary grants and contracts. Current and past SAMHSA funding opportunities are available on the SAMHSA web site, along with other information on grants policy, peer review, award data, contacts, application forms, and CRISP database.

Guidelines. This type of information is issued after careful review and deliberation of available scientific evidence, usually with the assistance of a panel of outside experts and is generally associated with a formal meeting or consensus panel specifically convened for the purpose. Prime examples are the Treatment Improvement Protocols published by the Center for Substance Abuse Treatment, SAMHSA.

Oral or Written Information. This category includes official speeches, editorials, commentaries, Letters-to-the-Editor only if they are provided by SAMHSA staff representing official SAMHSA viewpoints.

Statistical Information. Examples of statistical information published by SAMHSA include annual appropriations by SAMHSA's Centers or Offices, employment data (e.g., numbers of staff and staffing by professional degree), and data books (e.g., the annual Summary of Findings for the National Household Survey on Drug Abuse).

IV. Types of Dissemination Methods

SAMHSA information is disseminated in many media, with the following four being the most common:

Print -- publications, books newsletters brochures, booklets, pamphlets, and reports.

Oral -- formal speeches, oral presentations, interviews, or commentaries for publication or broadcast; letters to the editor or correspondence likely to result in similar publications.

Audio-Visual -- broadcast scripts, audio or videotapes, and videocasting.

Electronic -- The SAMHSA web site at www.samhsa.gov.

V. Quality Assurance Policies, Standards, and Processes for Ensuring the Quality of Information Disseminated to the Public.

1. Overview

   At SAMSHA, the quality assurance process begins at the inception of the information development process. Further, SAMSHA reviews the quality (including the objectivity, utility, and integrity) of information before its is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination.

   All SAMHSA documents and audiovisuals must be prepared in accordance with professional and ethical standards, as well as generally accepted standards of good taste. They must be appropriate for dissemination by SAMHSA and must undergo appropriate review and approval prior to release. SAMHSA adheres to the laws and regulations applying to publications and audiovisual materials, including OMB Information Quality Guidelines, the HHS Printing Handbook, and relevant SAMHSA policy issuances. SAMHSA efforts to ensure and maximize information quality begin at the preparation stage and continue through the review and approval stages. When published electronically, existing SAMHSA policies developed in concert with Federal computer security laws provide appropriate security safeguards to ensure integrity of SAMHSA documents, i.e., that the information is protected from unauthorized access, revisions, corruption, or falsification. Further, SAMHSA is committed to demonstrating in its Paper Reduction Act (PRA) will be collected, maintained, and used in a way that is consistent with OMB, HHS and SAMSHA information quality guidelines.

2. Preparation

   Each publication must be accurate, both in specific details and in general impressions, and meet accepted standards of high quality. The OMB Information Quality Guidelines define quality as including objectivity, utility, and integrity. SAMHSA documents and presentations containing text and summary data must be objective and scientifically sound. Sources should be referenced for the convenience and further information of the reader. Where appropriate, supporting data should have full, accurate, and transparent documentation. Potential error sources affecting data quality should be identified and disclosed to users. Disclaimers should be used to distinguish the status of information (e.g., based on preliminary data or partial data set).

   OMB Information Quality Guidelines hold "influential" scientific, financial, or statistical information to a higher standard of quality, requiring that the results be "substantially reproducible" by qualified third parties were the original or supporting data to be independently re-analyzed using the same methods. Making the data and methods publicly available will assist in determining whether analytical results are capable of being substantially reproduced.

3. Plain Language

   A Government-wide directive requires Federal agencies to use plain language in all communications with the public. Plain language is writing that is geared to the target audience (i.e., a plain language document for a scientific audience may be different from a plain language document for the general public). Plain language is grammatically correct, with accurate word usage. It is clear and expresses exactly what readers need to know without unnecessary words.

4. Layout and Design

   Each publication should be prepared in a pleasing, dignified, and finished format appropriate for its intended use and audience. Use of color and typography must be in accord with Government Printing and Binding Regulations (www.house.gov/jcp/jcpres.pdf).

5. SAMHSA Information Review and Approval Policies and Procedures by Type of Information

   1. Overview

      The purpose of the review process is to improve the quality of SAMHSA documents and to ensure the accuracy and validity of information intended to benefit the targeted audience, such as professional associations and the general public. All materials distributed by SAMHSA must be reviewed for accuracy, propriety, completeness and quality (including objectivity, utility, and integrity). The structure of the review and the types of reviewers will depend on the nature of the information as well as the targeted audience. For scientific and technical documents, peer review provides a level of quality control that is well recognized in the scientific community. According to OMB Information Quality Guidelines, material subject to formal, independent, external peer review can be considered to be of acceptable objectivity. However, this presumption of objectivity is refutable based on a persuasive showing by a complainant in a particular instance. The single most important determinant of a review groups' excellence and credibility are its members. Reviewers must: have appropriate scientific knowledge (as demonstrated by grant and publication records, academic degrees and honors); be respected in the scientific community; have breadth of expertise; be fair and objective; and, not be influenced by inappropriate personal interests (competition, scientific bias, personal antagonisms, or similar irrelevant factors). Because utility is one measure of quality, the review group should include representatives of the targeted audience to the degree practicable.

      Statistical compendia and documents providing "influential scientific, financial, or statistical information" must be reviewed carefully. The Director of the Office of Communications must determine that these documents and the data on which they are based conform to the standards set forth here, and, if applicable, that the reported information and/or statistics reported therein be substantially reproducible.

      A document that has obtained publication clearance for paper printing is often posted on the sponsoring Center's or Office's web page for greater accessibility. Such documents should not need additional approvals. Web documents with no print counterpart require content clearance by the Office of Communications to ensure that the information observes all applicable requirements governing information for release to the public.

      Scientific papers, books, journal articles, brochures, documents, statistical compendia, newsletters, electronic documents, audio-visual productions, and similar materials.

      SAMHSA encourages dissemination of scientific research and other information by its employees. Professional and scholarly writing, lecturing, editing, and publishing are an essential part of research, are in the public interest, and bring credit and distinction to SAMHSA and to the employees themselves. In assisting employees to share information about their official and professional activities, SAMHSA seeks to advance scientific knowledge and contribute to professional education. The first report of any scientific research results or other professional findings may be made by publication in a scientific or professional journal or presentation at a meeting of a professional organization. The choice of the journal or meeting to which reports are offered is the prerogative of the author(s).

      To ensure and maximize the quality of information disseminated by SAMHSA employees, any non-extemporaneous presentation (written or electronic) by a SAMHSA employee on a subject related to his/her SAMHSA duties must be reviewed and approved through an internal SAMHSA process prior to submitting for publication consideration. With few exceptions, non-extemporaneous oral presentation on health policy or practice, or presentations with policy implications, must also be cleared in advance. Manuscripts intended for publication are customarily subjected to an external peer review process directed by the interested publisher, volume editor, or journal editor.

      Publication or oral presentation of scientific and professional information by individual employees must conform to applicable laws and regulations, including OMB Information Quality Guidelines and the HHS Standards of Conduct Regulations.

      Customary professional practices impose certain constraints on the degree to which SAMHSA employees maybe identified with the results of research and development work, including that obtained in collaboration with extra-mural grantees.²

      All SAMHSA components have a formal internal operating procedure for identifying printing requirements and tracking publications. In addition, the concept clearance process for new publications is often the vehicle to track the development of a new publication and to identify its attendant printing requirements. Each prospective publication must be cleared through the originating SAMHSA component and then be approved by the Office of Communications, SAMHSA and the Office of the Assistant Secretary for Public Affairs (OASPA), HHS.

      In brief, any official publication (including book, bibliography, chapter of a book or textbook, booklet, brochure, collection of abstracts, fact sheet, house organ, index, leaflet, manual, monograph, newsletter, pamphlet, review, periodical, proceeding, recurring report, statistical compendium, Internet document, audio-visual or the like) prepared by any SAMHSA component directly or through a contract must be sent by SAMHSA for HHS clearance. This clearance requirement does not apply to publication of articles in journals.

      All audio-visual projects and exhibits must be cleared through OASPA, whether produced in-house or under contract. To obtain clearance for audiovisual products, including exhibits, Form HHS 524A must be completed, approved by the Office of Communications, and filed and approved with the Office of the Assistant Secretary for Public Affairs before production may begin.

      Statistical compendia, including statistical analyses, aggregated information by program, Center, Office or for SAMHSA as a whole, including funding information and histories (by funding mechanism, dollars, or other criteria), require approval by the Director, Office of Communications, SAMHSA. The Director of the originating Center or Office is required to determine that the data conform to accepted scientific and quality standards and that the reported statistics are substantially reproducible.

      In general, any writing by a SAMHSA employee on a work-related subject, whether or not intended for electronic or print publication, or for oral delivery, must be prepared according to accepted SAMHSA standards of quality, reviewed for substantiative content, and administratively approved. The purpose of the clearance process is to improve the quality of information and to ensure the accuracy, objectivity, utility, and validity of information.

      Consistent with HHS Standards of Conduct (73.735-705 Writing and Editing), employees are encouraged to engage in outside writing and editing when such activity is not otherwise prohibited. If the writing or editing activity is related to the employee's official duties or other responsibilities and programs of the Federal Government, the employee must: (1) make no mention of his or her official title or affiliation with the Department, or (2) use his or her official title or affiliation with the Department, or (3) submit the material for clearance within the operating component, under procedures established by the component. When clearance is denied at any lower level, the employee shall have recourse for review up to the head of the principal operating component. This clearance will show there are no official objections to the activity, and the employee may then use his or her official title or affiliation usually without a disclaimer. Except where the requirement for disclaimer is waived as a result of official clearance, disclaimers shall be used in all writing and editing related to the employee's official duties or other responsibilities and programs of the Federal Government: (1) in which the employee identifies himself or herself by official title or affiliation with the Department, or (2) when the prominence of the employee or the employee's position might lead the public to associate him or her with the Department, even without identification other than by name. Disclaimers shall read as follows unless a different wording is approved by the Assistant General Counsel, business and Administrative Law Division, Office of the General Counsel: "This (article, book, etc.) was (written, edited) by (employee's name) in (his or her) private capacity. No official support or endorsement by (name of operating component or of Department) is intended or should be inferred."

      Oral Information including speeches, interviews, expert opinions, only if representing SAMHSA views, official position, or policies

      Any public statements, comments, or discussion of Federal policies or practices that are relevant to the employee's position or duties, draw conclusions, advocate or oppose professional practices or positions on subjects related to SAMHSA duties or that might otherwise be construed as reflecting an official position by SAMHSA, HHS, of the Federal Government, are covered by the OMB Guidelines and must be approved in the Office of Communications, SAMHSA.

      No review or approval is required for non-official and private writing, speaking, and publishing by an employee unless his/her SAMHSA employment is likely to be regarded as influencing the content.

      SAMHSA employees are responsible for the statements they make, regardless of whether or not they have been cleared. If one presents material that requires clearance but that has not been cleared prior to presentation, then the employee must inform the audience of the personal or unofficial nature of his or her views. An example of an appropriate disclaimer follows:

      "This material is presented from my own perspective and should not be taken as representing the viewpoint of the Department, SAMHSA, or the Center for...."

      SAMHSA employees shall not identify themselves as SAMHSA employees in unofficial materials prepared for dissemination to non-professional audiences, such as a letter to the editor. These materials must be reviewed prior to presentation in the Office of Communications, if the employee's identification with SAMHSA is to be shown, can be inferred, or is well known.

      SAMHSA Clearinghouse Information

      Clearinghouses often serve as a public point of contact and provide access to information about SAMHSA programs, conferences, and grant activities. These are often in electronic as well as hard copy format. Clearinghouses have been contracted to provide varying levels of service, including distribution of fact sheets, information packages, and publications; storage of materials; conducting outreach and promotion; and performing training and quality control for the clearinghouse staff. Clearinghouses may respond to inquiries about particular issues, ranging from information about available educational materials to statistical data. Clearinghouses are challenged to ensure accuracy and reliability of information, while continually striving to improve performance and response times. Clearinghouses also must determine which organizations are worthy of referral when customers need information that is not available at the clearinghouse and how to avoid implying endorsement.

   2. Procedures to Ensure the Integrity of Information.

      SAMHSA has developed World Wide Web (www) Guidance which is available on the SAMHSA Intranet to SAMHSA staff. Web page creators must periodically review material on the web page to determine whether or not it is accurate and up-to-date. Information, particularly time-sensitive information, should be posted as soon as possible. Web page creators are expected to promptly update or remove out-of-date information or to notify Division of Information Resources Management that such actions must be taken.

      Unless otherwise noted, it is safe to assume that information posted on public web sites within the "SAMHSA.gov" domain is considered to be "in the public domain." As such, others are free to establish links to SAMHSA online resources. In establishing such links, SAMHSA requests that others avoid creating the impression that SAMHSA is endorsing or promoting any particular product or service. In the same vein, any outside link to an external resource from a SAMHSA web site needs to be examined and approved on a case-by-case basis.

      Web pages containing links to external pages not located on the SAMHSA server should include a link to a statement that releases SAMHSA from responsibility for the material included in the external web page. Again, it is important to avoid giving a user the impression that SAMHSA is endorsing information or a commercial product described in an external site. Disclaimers on copyright, endorsement (general and external links), liability, and medical information are also used, as appropriate, for individual web sites.

      Each Center or Office designates a principal contact for information and approvals related to the development of web pages. SAMHSA personnel, contractors, and other authorized users must notify this contact person prior to setting up a web page on the SAMHSA server.

      The SAMHSA Division of Information Resources (DIRM) is charged with providing, coordinating, and managing information technology for SAMHSA. In terms of computer security, there are three distinct objectives: (i) confidentially -- ensuring that there is no deliberate or accidental improper disclosure of sensitive automated information; (ii) integrity -- protecting against deliberate or accidental corruption of automated information; and, (iii) availability -- protecting against deliberate or accidental actions that cause automated information resources to be unavailable to users when needed. Information is accorded protection against disclosure, alteration, loss, or destruction, based on the degree of sensitivity.

      DIRM staff use appropriate safeguards to protect data from improper disclosure by backing up critical data periodically, and, if a security incident occurs, by following proper incident response procedures. Supervisors are responsible for ensuring that employees, both Government and contractors observe all security requirements and that employees receive appropriate security training.

VI. Agency Administrative Complaint Procedures

SAMSHA has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and SAMSHA guidelines.

At SAMHSA, the Director, Office of Communications (OC) will have overall responsibility for implementing SAMHSA's Information Quality Guidelines and coordinating this effort with other SAMHSA Centers and Offices. All complaints may be sent to:

Charlene Lewis
Office of Communications
Room 8-1023
Substance Abuse and Mental Health Services Administration
1 Choke Cherry Road
Rockville, MD 20857

Alternatively, complaints may be e-mailed to:  charlene.lewis@samhsa.hhs.gov

OC is the central office for communications at SAMHSA. As such, OC takes the lead across SAMHSA for setting communications policy and for communicating information about SAMHSA programs, issues, and accomplishments to the public, public interest groups, and to the scientific community. OC is the communications link between SAMHSA's components and the Office of the Assistant Secretary for Public Affairs (OASPA) in HHS and serves as the coordinating office or central source for matters related to SAMHSA publications, including printing, HHS/SAMHSA clearance and review procedures, Joint Committee on Printing, U.S. Congress, and Government Printing Office printing and binding regulations, and copyright rules. Among its many activities, the office produces and distributes a number of publications that highlight SAMHSA achievements. OC also support and coordinates the principal contents of the SAMHSA web site (www.samhsa.gov).

As the lead office for SAMHSA Information Quality, OC responsibilities include:

Developing policies and procedures to effectively meet the requirements of the OMB Information Quality Guidelines;

Providing information and/or training to SAMHSA staff on their responsibility in meeting Federal requirements and SAMHSA policies on ensuring the quality of information disseminated to the public;

Assisting in the review of information quality complaints;

Reviewing the proposed SAMHSA Center or Office response for appropriateness and assisting in finalizing a response;

Establishing a tracking database for complaints, with information on the type of complaint and its disposition, whether the complaint was deemed inapplicable or frivolous, and any resolution or corrective action taken;

Submitting an annual report on behalf of SAMHSA to HHS with the number and types of complaints, and the actions taken, in time for the HHS to report to OMB by January 1 (beginning in 2004);

Posting on the SAMHSA web site any further clarifications, guidelines, and Frequently Asked Questions (FAQs) about handling SAMHSA information complaints;

Making available examples of typical complaints and appropriate responses collected from SAMHSA Center

1. Responsibility of the Complainant

   To seek a correction of information disseminated by the agency, individuals should follow the procedures described below.

   1. A complaint or request for review and correction of information shall be in written hard copy or electronic form;
   2. it shall be sent to the agency by mail or electronic-mail (e-mail); and
   3. it shall state that an information quality request for correction is being submitted.

   The complaint shall contain

   4. a detailed description of the specific material that needs to be corrected including where the material is located, i.e. the publication title, date, and publication number, if any, or the Web site and Web page address (url), or the speech title, presenter, date and place of delivery; and
   5. the specific reasons for believing the information does not comply with OMB, HHS or SAMHSA guidelines and is in error and supporting documentation, if any;
   6. the specific recommendations for correcting the information;
   7. a description of how the person submitting the complaint is affected by the information error; and
   8. the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.

   Complainants should be aware that they bear the ‘burden of proof’ with respect to the necessity for correction as well as with respect to the type of correction they seek.

2. Responsibility of the Agency

   Based on a review of the information provided, the agency will determine whether a correction is warranted and, if so, what action to take. The agency will respond to the requestor by letter or e-mail. The agency's response will explain the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. The agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

3. Appeals

   If the individual submitting the complaint does not agree with the agency's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal shall state the reasons why the agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the agency response to it, clearly mark the appeal with the words, Information Quality Appeal and send the appeal to the specific agency appeals address.

   The agency official who resolved the original complaint will not have responsibility for the appeal.

4. Appropriate Responses

   The agency will respond to all requests for appeals within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

   SAMHSA may respond in a number of possible ways, including:

   • Issue a written retraction;
   • Suspend further dissemination of the information in question;
   • Reprint the publication with corrections;
   • Refer the complainant to the underlying data;
   • Arrange for an independent re-analysis of the data by SAMHSA or a mutually acceptable third party if the data are not publicly available and the complaint involves "influential scientific or statistical information." Complainants must agree to pay the costs of re-analysis or the process terminates.

VII. Influential Scientific, Financial and Statistical Information

The Information Quality Guidelines require that "influential" scientific, financial, or statistical information in official Government documents must be based on studies that can be substantially reproduced if the original or supporting data were to be independently re-analyzed using the same methods. "Influential" when used in the phrase "influential scientific, financial, or statistical information" means that SAMHSA can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important public policies or important private sector decisions or will have important consequences for specific health practices, technologies, substances, products, firms, etc.

SAMHSA is committed to applying rigorous scientific standards to ensure the accuracy and reliability of research results. For scientific and technical documents, the scientific community recognizes peer review as the primary mean of quality control.

To facilitate the replication of scientific and other influential information by qualified third parties, SAMHSA continues to encourage the sharing of data and methods where practicable. Since the influence and implications of SAMHSA-disseminated information cannot always be truly anticipated, all SAMHSA scientific reports are expected to state clearly how analytic results were generated -- the specific data and sources used, data collection methodologies, various hypotheses and assumptions, specific analytic methods, statistical procedures, sources of bias or measurement error -- making the analysis sufficiently transparent so as to be capable of being reproduced. SAMHSA advocates the archiving of data through the Substance Abuse and Mental Health Data Archive or similar archive of publicly available data to facilitate the reproducibility of influential information.

VIII. References

HHS Standards of Conduct Regulation (45 CFR 73.735-705). Updated October 1, 2000.

HHS Printing Handbook, September 1998.

Office of Management and Budget Guidelines for Ensuring and Maximizing the Quality, Objectivity, utility, and Integrity of Information disseminated by Federal Agencies. Final Guidelines. January 3, 2002. www.whitehouse.gov/omb/fedreg/reproducible.html

Office of Management and Budget Circular No. A-130 Revised (Transmittal Memorandum No. 4) Management of Federal Information Resources. November 30, 2000.

Endnotes

¹ In general, special terms and conditions of a grant award will specify ownership rights to data. For this purpose, "data" means recorded information, regardless of the form or media on which it may be recorded, and includes writings, films, sound recordings, pictorial reproductions, drawings, designs, or other graphic representations, procedural manuals, forms, diagrams, work flow charts, equipment descriptions, data files, data processing or computer programs (software), statistical records, and other research data. Grantees are required to place an acknowledgment of SAMHSA grant support and a disclaimer, as appropriate, on any publication written or published with such support and, if feasible, on any publication reporting the results of, or describing, a grant-supported activity. An acknowledgment shall be to the effect that: "This publication was made possible by Grant Number ____ from _________" or "The project described was supported by Grant Number ___ from ______" and "Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the (name of awarding component or SAMHSA)."

² The conditions allowing SAMHSA staff to be (co-)authors of publications under SAMHSA extramural awards ordinarily arise only from contracts and cooperative agreements, where, by definition, there is substantial programmatic (i.e., scientific-technical) staff involvement. To merit approval for (co-)authorship on publications from extra-mural awards (including grants, contracts, and other award mechanisms), SAMHSA staff must have played a substantial role beyond normal program officer duties, including the following: originating the specific ideas that led to the research activity and manuscript; performing significant portions of the activity, and participating actively in preparing manuscripts.

K. Office of the Secretary

1. Office of the Assistant Secretary for Planning and Evaluation

I. Office Mission

The Assistant Secretary for Planning and Evaluation (ASPE) advises the Secretary of the Department of Health and Human Services on policy development in health, disability, human services, and science, and provides advice and analysis on economic policy. ASPE leads special initiatives, coordinates the Department's evaluation, research and demonstration activities, and manages cross-Department activities such as strategic planning, legislative planning and review of regulations. Integral to this role, ASPE conducts research and evaluation studies, develops policy analyses and estimates the cost and benefits of policy alternatives under consideration by the Department or Congress.

II. Scope and Applicability of Guidelines

ASPE will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and ASPE guidelines.  It is ASPE’s goal to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public.  We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.  The pre-dissemination review described in the guidelines only applies to information disseminated on or after October 1, 2002.  The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information. The guidelines apply to the following kinds of information that the Office of the Assistant Secretary for Planning and Evaluation (ASPE) disseminates: (1) the reports of research and evaluation projects that ASPE sponsors. These reports are usually the results of research that has been conducted for ASPE by contractors under our direction. (2) The final reports of projects supported by grants from ASPE may also be disseminated by ASPE. (3) ASPE staff occasionally write papers for publication and distribution. These reports are formal documents which have gone through several reviews within both within the author's organization and within ASPE.   The Office also produces materials that are not distributed externally. These materials are intended to inform the policy-making process. These materials consist of briefing papers and technical analyses concerning proposed legislative changes, the requirements of new legislation or descriptions of programs or initiatives. While these papers may eventually be distributed externally after the policy-making process is concluded, as required by the Freedom of Information Act, they are not intended to be disseminated widely and consequently are not covered under these guidelines.  Further, the guidelines do not apply to press releases that support the announcement or give public notice of information that ASPE has disseminated elsewhere.

III. Types of Information Disseminated

• Research and evaluation reports
• Annual plans
• Lists of projects in progress or completed
• Project summaries
• Annotated bibliographies
• Lists of sources of data
• Reports to Congress
• Data files for public use
• Tabulations and charts of data relevant to particular topics, such as welfare dependence and child well-being

IV. Types of Dissemination Methods

The Office of the Assistant Secretary for Planning and Evaluation provides both printed copies and electronic copies of the reports it selects for dissemination. The individual components of ASPE have responsibility for the dissemination of the products of their research and evaluation efforts. Each component maintains mailing lists and conducts direct mailings that are limited to the intended audience of the publication. In addition, some components have developed extensive electronic mailing lists that notify subscribers of the availability of new reports on the ASPE website. In some circumstances, a plan for the dissemination of a specific product may be developed to ensure widespread dissemination to a particular audience. Such a plan might include notices to appropriate media outlets, interviews, and presentations at conferences and other meetings.

V. Agency Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Disseminated to the Public

From their inception, research and evaluation projects and any other information developmental efforts  that are undertaken by ASPE are subjected to a quality assurance process. Further, ASPE reviews the quality (including the objectivity, utility, and integrity) of information before it’s disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination. Each project is developed with the knowledge that the purpose, proposed methodology and work plan will have to be defended before a peer review panel. The members of the panel are appointed for at least a year at a time and are technically qualified by their professional training and experience. The process for conducting this peer review has been established for many years and refined as circumstances have changed. Requests for changes in the design of proposed projects are not infrequent. Projects are not approved for funding until the peer review panel is satisfied that the project design and work plan have a reasonable expectation of providing a useful product. All projects involving original large scale data collected from the public undergo an exacting, rigorous multi-level review process in connection with the Paperwork Reduction Act (PRA).  Further, ASPE is committed to demonstrating in its PRA clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and ASPE information quality guidelines.

Reports that are selected for dissemination to the public undergo the following processes: Every report that is received as the result of a grant or contract is reviewed by the project officer and that person’s supervisor. If necessary, a technical expert may be asked to review the report and offer an opinion. A decision is made whether the report may be distributed or not. A further decision is made whether to seek support for a more extensive dissemination effort.

If a more extensive dissemination effort is recommended, a more thorough review process involving other organizations is initiated. This will usually follow formal clearance procedures controlled by the Executive Secretariat in the Office of the Secretary. Offices outside of ASPE, including the Office of the Assistant Secretary for Public Affairs and one or more operating divisions with an interest in the report’s contents are asked to concur in the release of the report and provide comments. Editorial changes may be made as a result of this review.

VI. Agency Administrative Complaint Procedures

ASPE has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and ASPE guidelines. Complaints about the quality or accuracy of the information being disseminated in a report or other document should be addressed in writing, by mail to:

Coordinator of Information Dissemination
Office of the Assistant Secretary for Planning and Evaluation
200 Independence Ave., S.W.
Washington, D.C. 20201 or

2. HHS Office of Inspector General

I. Office Mission

The mission of the Office of Inspector General, as mandated by Public Law 95-452 (as amended), is to protect the integrity of Department of Health and Human Services programs, as well as the health and welfare of the beneficiaries of those programs. The OIG has a responsibility to report both to the Secretary and to the Congress program and management problems and recommendations to correct them. The OIG's duties are carried out through a nationwide network of audits, investigations, inspections and other mission-related functions performed by the OIG components

II. Scope and Applicability of Guidelines for Office

OIG will ensure that disseminated information meets the standards of quality set forth in the OMB, HHS and OIG guidelines. It is OIG’s policy to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public. We strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. We are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

The OMB Information Quality Guidelines require OIG to evaluate and identify the types of OIG information that will be subject to the Guidelines. The pre-dissemination reviewed described in the guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002 regardless of when the agency first disseminated the information. This section identifies the types of information covered, and also lists the types of information that are exempt.

1. Covered Information

   OIG reports prepared for use by Department officials formulating broad program policies which are not subject to review by the affected individual(s) or entity prior to public dissemination.

   Editorials or Open Letters to the public representing position or view of the agency.

2. Information Not Covered

   Documents that are summary in nature and where the source documents are already covered under the Guidelines, this would include the Semiannual Report, the Orange Book and the Red Book.

   Planning documents that represent future work which has not been completed. This includes the annual OIG Work Plan.

   Findings and determinations or agreements made in the course of adjudication. This includes the List of Excluded Individuals and Entities and Corporate Integrity Agreements. Affected parties are subject to and/or can contest exclusion decisions through an adjudicatory process.

   Proprietary information owned by another agency, which the Department does not have authority to release to outside sources, such as the Annual Report of the State Medicaid Fraud Control Units.

   Documents where the subject individual or entity is already afforded an opportunity to comment on the accuracy of the information to ensure a fair, objective and complete report, this includes most Audit reports.

   Advisory Opinions, which are requested of the OIG and are binding only on the requestor.

   The guidelines do not apply to press releases that support the announcement or give public notice of information that OIG has disseminated elsewhere.

   Documents which comply with the Federal Register standards for public notices. This includes Fraud Alerts, Special Advisory Bulletins and Safe Harbor Regulations, which are published in the Federal Register. As such, the public is accorded the administrative procedures for public notices and afforded the opportunity to comment. The safe harbors also are excluded because they serve as OIG policy.

   Guidance issued by the OIG where compliance is voluntary such as those issued under the Compliance Guidance Program. The compliance guidance is based on the Centers for Medicare and Medicaid Services (CMS) regulations and guidelines, any requirements imposed in corporate integrity agreements negotiated by the OIG and input from the affected industry, thus there is an established quality control process through input from the affected entities.

   Information or documents explicitly excluded by the Quality Control Guidelines such as press releases, public filings, subpoenas, or adjudicative processes.

   Procedural and policy manuals that are produced primarily for internal use.

III. Types of Information Disseminated

The types of information published by the OIG is described in Section II.

IV. Types of Dissemination Methods

Hard copy reports and electronic media such as the Internet. Interested persons can subscribe to receive information electronically on recently issued reports, press releases and other documents.

V. Agency Quality Assurance Policies, Standards and Procedures for Ensuring the Quality of Information Dissemination to the Public

The quality assurance process begins at the inception of the information development process.

Information released by OIG is developed from reliable data sources utilizing accepted methods for data collection and is based on thoroughly reviewed analyses and models. The guidelines below describe procedures that OIG employs to assure the quality of its information products. Quality is an encompassing term comprising utility, objectivity, integrity, and reproducibility.

1. Utility

   Utility involves the usefulness of the information to its intended audience. The purpose of OIG documents and information is to identify fraud, waste and abuse and to protect the integrity of HHS programs. The OIG issues findings and recommendations on the efficiency, effectiveness, and vulnerabilities of departmental programs.

2. Objectivity

   Objectivity involves a focus on ensuring that information is accurate, reliable and unbiased and that information products are presented in an accurate, clear, complete and unbiased manner. Objectivity is achieved by using reliable data sources and sound analytical techniques, and preparing information products that use proven methods by qualified individuals that are carefully reviewed. Below is a description of the quality assurance guidelines that are used to ensure objectivity and accuracy of information.

   OIG reviews the quality (including the objectivity, utility, and integrity) of information before its is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination.

   Quality Assurance Process

   The quality assurance procedures is a collaborative, team effort by staff who conduct program inspections and staff who assist, guide, and review written and oral inspection plans and products. Any individual involved with a particular inspection is responsible for helping assure quality of work done and products released. Further, OIG is committed to demonstrating its Paper Reduction Act (PRA) clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS, and OIG information quality guidelines.

   Specific responsibility for conducting and ensuring quality program inspections rests with Regional Inspectors Generals (RIGs). RIGs assign a Team Leader and other team members to each inspection. This team is responsible for doing quality inspection field work -- including design, data collection, data analysis, and written and oral reports. RIGs must certify for final reports that the inspection was done in accordance with OIG procedures and PCIE quality standards.

   If an audit methodology is used to develop the product, the Government Auditing Standards are employed as described below:

   • Government Auditing Standards State that "Each audit organization conducting audits in accordance with these standards should have an appropriate internal quality control system in place and undergo an external quality control review."
   • This control system is to provide reasonable assurance that the audit organization (1) has adopted, and is following, applicable auditing standards (which includes reporting "the views of responsible officials") and (2) has established, and is following, adequate policies and procedures.
   • To determine whether in fact the audit organization's quality control system is operating effectively, the organization is required by standards to have "an external quality control review at least once every 3 years by an organization not affiliated with the organization being reviewed."

   OIG's quality control program ensures that audit work performed meets government auditing standards. It consists of two elements:

   • The OIG System of Quality Control consists of independent report referencing, Headquarter Desk Reviews and Internal Quality Control Reviews. The latter includes review of working papers related to selected reports.
   • External Reviews -- These reviews, also known as peer reviews, are performed by Federal auditors outside the OAS. The external review is required by the Comptroller General's standard on quality control and should be conducted at least once every three years.

   Report Validation

   Report Validation is a process that is used to help ensure that the facts, findings, conclusions, and recommendations contained in OIG reports are accurate, reliable, and supportable by inspection work papers and analysis.

   Report validation is conducted prior to submitting the draft report for IG signature. For objectivity, validation is performed by regional staff that are not a part of the inspection team. To facilitate this process, the draft report is cross indexed to applicable work papers. Work papers must include documentation that inspection teams verified data presented in draft reports by checking it against supporting evidence in the work papers.

   Agency Reviews

   To help assure quality and impact of its inspection results, OIG obtains comments and other input from applicable agency staffs on inspection plans, results, and selected products. Inspection teams obtain such input in a variety of ways, including entrance, exit, and other conferences; and comments on inspection designs, data collection instruments, and draft and final reports. Inspection teams use such meetings and reviews to help identify client needs, identify relevant data for accomplishing inspection purposes, verify accuracy and reliability of data collected, and verify soundness of findings, conclusions, and recommendations developed.

   Reliability of Data Sources

   To maintain credibility, OIG must take reasonable steps to assess the reliability of pre-existing computerized data used as the basis for inspection findings and recommendations. Many inspections either begin with a computerized sample selection or are based entirely on analyzing data extracted from computerized records not under OIG’s direct control. Project staff do not assume that such computer extracts or sample selections are complete or that they accurately reflect the universe of people or transactions being studied.

   Data reliability means the degree to which data extracted from computer records for a program inspection completely and accurately reflect the individuals or transactions being studied. This is a relative concept, one that recognizes that data with errors may still be usable, if the errors are not of a magnitude that would cause a reasonable person to doubt findings or conclusions that are based on the data.

   To provide reasonable assurance of computerized data reliability we:

   • Identify prior reviews by OIG, GAO or by system managers attesting to the computer system and data reliability,
   • Review the data dictionary, if it exists, for the database to assure a full understanding of the relevant data elements’ structure, content, how the elements are derived, and their interrelationships before requesting data extractions,
   • Obtain frequency counts of critical data elements to determine if the data selection criteria are providing the information anticipated,
   • Conduct data accuracy tests to ensure that required data elements have been provided and are in the expected format,
   • Obtain detailed printouts for a sub-sample of records included in the data extract to confirm that the extraction produced the types of records sought and the required information from those records,
   • Obtain source documents (e.g., claim folders) for a sample of extracted records to determine the validity of the data contained in the automated records, if the data reliability is questionable, and include a statement in the Methodology section of draft and final reports describing the extent of reliability testing performed and our confidence in the data used.

   Integrity

   Integrity refers to the security of information from unauthorized access or revision to ensure that the information is not compromised through corruption or falsification. To ensure the integrity of information, OIG has in place rigorous controls that have been identified as representing sound security practices.

   OIG is highly protective of the confidentiality of information it holds through its policies and practices. OIG has in place programs and policies for securing OIG resources as required by the Government Information Security Reform Act (P.L. 106-398, Title X, Subtitle G). OIG is subject to statutory requirements to protect the sensitive information it gathers and maintains on individuals.

   Reproducibility

   If an agency is responsible for disseminating "influential" information, guidelines for dissemination should include a high degree of transparency about data and methods to facilitate its reproducibility by qualified third parties. Information is considered influential if it will have a substantial impact on important public policies or important private sector decisions. Since many of the OIG's Inspection reports have an impact on important public policies, OIG's information that is subject to section 515 should be highly transparent and capable of being reproduced by qualified persons.

   OIG's guidelines call for identification and documentation of data sets used in producing estimates and projections and clear description of the methodology used to produce the analytical results. Some results included in OIG reports are not directly reproducible by the public because the underlying data sets used to produce them are confidential. However, those inspections that are based on publically available data and are made available on request are fully reproducible by the public.

VI. Agency Administrative Complaint Procedure

OIG has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and OIG guidelines.

Requests for review should to be submitted in writing to the OIG at the following address:

Public Affairs Officer
Office of Inspector General
Office of Management and Policy
Room 5541 Cohen Building
Washington D.C. 20201

Alternatively, they can be e-mailed to: DataQuality@hhs.gov

1. Responsibility of the Complainant

   To seek a correction of information disseminated by the agency, individuals should follow the procedures described below.

   1. A complaint or request for review and correction of information shall be in written hard copy or electronic form;
   2. it shall be sent to the agency by mail or electronic-mail(e-mail); and
   3. it shall state that an information quality request for correction is being submitted.

   The complaint shall contain

   4. a detailed description of the specific material that needs to be corrected including where the material is located, i.e. the publication title, date, and publication number, if any, or the website and web page address (url), or the speech title, presenter, date and place of delivery; and
   5. the specific reasons for believing the information does not comply with OMB, HHS or OIG guidelines and is in error and supporting documentation, if any;
   6. the specific recommendations for correcting the information;
   7. a description of how the person submitting the complaint is affected by the information error; and
   8. the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.
   9. Complainants should be aware that they bear the ‘burden of proof’ with respect to the necessity for correction as well as with respect to the type of correction they seek.

2. Responsibility of the Agency

   Based on a review of the information provided, the agency will determine whether a correction is warranted and if, so what action to take. The agency will respond to the requestor by letter or e-mail. The agency’s response will explain the findings of the review and the actions that the agency will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information and the magnitude of the correction. The response will describe how the complainant may request reconsideration. The agency will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

3. Appeals

   If the individual submitting the complaint does not agree with the agency's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the agency's decision. The appeal shall state the reasons why the agency response is insufficient or inadequate. Complainants shall attach a copy of their original request and the agency response to it, clearly mark the appeal and the envelope with the words, "Information Quality Appeal," and send the appeal to:

   Public Affairs Officer
   Office of Inspector General
   Office of Management and Policy
   Room 5541 Cohen Building
   Washington D.C. 20201

   The agency official who resolved the original complaint will not have responsibility for the appeal. The agency will respond to all requests for appeals within 60 working days of receipt. If the request requires more than 60 calendar days to resolve, the agency will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

VII. Influential scientific, financial and statistical information

Addressed in Section V.

VIII. Special Considerations for Dissemination

All OIG inspections are conducted in accordance with the Quality Standards for Inspections issued by the President's Council on Integrity and Efficiency. The following is an outline of these standards.

Qualifications: Individuals assigned to perform inspection work must collectively possess adequate professional proficiency for the task required.

Independence: Individuals performing inspection work must be free from impairments that hinder objectivity. Inspectors must consistently maintain an independent, objective attitude and appearance, and shall be subject to supervisory guidance and review to preclude actual or perceived impairments or bias in conducting inspection work and presenting results.

Due Professional Care: Due professional care will be used in conducting inspection work and in preparing reports of other products.

Quality Control: To ensure quality and to expedite the progress of an inspection, proper supervision will be exercised from the start of an inspection to completion of the final inspection report.

Planning: To ensure adequate planning, inspection work will be coordinated, researched, and designed to achieve the objectives of the inspection.

Data Collection: Information and data obtained about the organization, program analysis activity, or function being inspected should be consistent with inspection objectives and sufficient enough to provide a reasonable basis for reaching conclusions.

Evidence: Evidence supporting inspection conclusions should be competent and relevant and lead a prudent person to the same conclusion as that of the inspectors.

Supporting: All relevant information generated, obtained, and used in Documentation supporting inspections findings, conclusions, and recommendations should be retained.

Timeliness: inspectors should seek to deliver significant information to appropriate management officials in a timely manner.

Fraud and Other: If during or in connection with an inspection, inspectors become Illegal Acts aware of illegal acts, or indications of such acts, they should promptly present such information to their supervisors for review and possible referral to the appropriate investigative office.

Reporting: All inspection reports shall present factual data accurately, fairly, and objectively, and present findings and conclusions in a persuasive manner.

Follow-up: Appropriate follow-up will be performed to assure that any recommendations made to agency officials are adequately considered and appropriately addressed.

All OIG audits are conducted in accordance with Government Auditing Standards and OAS policy. Both (1) afford affected entities the opportunity for corrections and (2) require a stringent quality control program.

1. Auditee Response

   1. Government Auditing Standards

      Government Auditing Standards state that "Auditors should report the views of responsible officials of the audited program concerning auditor's findings, conclusions, and recommendations, as well as corrections planned."

      • Obtaining comments is one of the most effective ways to ensure that a report is fair, complete, and objective.
      • Advance comments should be objectively evaluated and recognized, as appropriate in the report.

   2. OIG Audit Policy

      OIG Audit policy requires that reports give recognition to the views of the auditee.

      • The auditee's formal response to each finding should be included in the final audit report.
      • The official position of the auditee should be in writing and should be signed by the responsible official.
      • The auditee may present new information in formal written comments to the draft report. In these instances, the information should be evaluated prior to incorporating the comments and issuing the final report.
      • The auditee comments would appear in the final report in three places, report summary, individual finding and appendix to report.
      • The OIG regards the absence of a response to audit findings and recommendations as a departure from generally accepted government auditing standards. If the auditee does not provide comments after receiving the draft report, this is stated in the final audit report.
      • OIG Audit policy requires that respond to each relevant auditee comment.

2. Quality Control Program

   1. Government Auditing Standards

      Government Auditing Standards State that "Each audit organization conducting audits in accordance with these standards should have an appropriate internal quality control system in place and undergo an external quality control review."

      This control system is to provide reasonable assurance that the audit organization (1) has adopted, and is following, applicable auditing standards (which includes reporting "the views of responsible officials") and (2) has established, and is following, adequate policies and procedures.

      To determine whether in fact the audit organization's quality control system is operating effectively, the organization is required by standards to have "an external quality control review at least once every 3 years by an organization not affiliated with the organization being reviewed.

   2. OIG Audit Policy

      OIG's Audit quality control program ensures that work performed meets government auditing standards. It consists of two elements:

      1. A System of Quality Control which consists of independent report referencing, Headquarter Desk Reviews and Internal Quality Control Reviews. The latter includes review of working papers related to selected reports.
      2. External Reviews -- These reviews, also known as peer reviews, are performed by Federal auditors outside the OIG. The external review is required by the Comptroller General’s standard on quality control and should be conducted at least once every three years.

IX. Other Agency Specific Policies and Procedures

Office of Evaluation Procedures Manual, Data Analysis -- oig.hhs.gov/organization/OEI/other/doc_m.pdf (currently under revision)

Quality Standards for Federal Offices of Inspector General -- www.ignet.gov/pande/standards/igstds.pdf

Quality Standards for Inspections -- oig.hhs.gov/organization/OEI/other/qsidoc.pdf

Government Auditing Standards

Quality Standards for Federal Offices of Inspector General

MB Circular A-123 (Revised June 21, 1995)

3. Office of the Assistant Secretary for Health

I. Office Mission

The Office of Public Health and Science (OPHS) provides leadership to the nation on public health and science, and communicates on these subjects to the American people. OPHS is led by the Assistant Secretary for Health (ASH), whose chief interest is promoting, protecting, and improving the nation's health. This role encompasses responsibilities as senior advisor to the Secretary for public health and science and director of program offices housing a variety of essential public health activities. The offices in OPHS are: the Immediate Office of the ASH; the Office of the Surgeon General; the Office of HIV/AIDS Policy; the Office of Population Affairs (OPA); the Office of Disease Prevention and Health Promotion (ODPHP); the President's Council on Physical Fitness and Sports; the Office of Minority Health (OMH); the Office on Women's Health (OWH); the Office for Human Research Protections; the Office of Global Health Affairs (OGHA); the Office of Research Integrity; and the Office of Military Liaison and Veterans Affairs. In addition, the Director of the National Vaccine Program Office and the Regional Health Administrators report to the ASH.

II. Scope and Applicability of Guidelines for Agency/Office

Types of Information Disseminated by the Agency to the Public

The pre-dissemination review described in the guidelines only applies to information disseminated on or after October 1, 2002. The administrative mechanism for correction applies to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information.

The purpose of these Guidelines is to provide guidance to OPHS offices about administrative procedures to ensure the quality of information they disseminate to the public.  The Guidelines also provide guidance to the public about how to file a complaint about the quality of the substantive information disseminated by OPHS offices and how OPHS offices should respond to public complaints.   The Guidelines apply to substantive information disseminated by OPHS offices and representing OPHS/HHS views.   Substantive information includes consumer and professional education materials, scientific and technical reports, policy and program recommendations, research findings from sponsored grants that include a dissemination component, and public speeches representing official HHS policy.   The Guidelines do not apply to information that is labeled with a disclaimer as not representing agency views, intra- or inter- agency information, regulations, compliance oversight reports, grant and program announcements, or information describing basic agency operations.

OPHS offices disseminate a variety of public health and science information to the public. The primary types of information disseminated are consumer and professional education and scientific and technical reports. Information disseminated by OPHS is based on science, derived from state of the art knowledge, and peer-reviewed by experts inside and outside government, depending on the nature of the information.

For example, the Office of Disease Prevention and Health Promotion disseminates information about national disease prevention and health promotion goals and objectives such as the Healthy People 2010 document published in November 2000. The Office on Women's Health disseminates consumer education about women's health issues in the form of pocket planners, annual daybooks, and fact sheets such as the 2002 Women's Health Daybook Living Long, Living Well. The Office of Minority Health disseminates information on minority health issues through two nationally distributed newsletters. The Office of the Surgeon General disseminates information on high priority national public health issues in the form of Surgeon General Reports, Calls to Action and National Strategies such as the Surgeon General's Call to Action to Prevent and Decrease Overweight and Obesity 2001.

III. Types of Dissemination Methods

OPHS offices use both print and electronic methods to disseminate information. Offices use Web sites, clearinghouses (telephone information services), print reports, print brochures and newsletters, fact sheets, and a variety of consumer and professional educational materials, such as bookmarks, wallet cards, day planners, and pocket guides. OPHS staff also makes public speeches representing OPHS/HHS public health policy matters. These speeches are presentations of substantive content, not public affairs events.

IV. Agency Quality Assurance Policies, Standards, and Processes for Ensuring the Quality of Information Disseminated to the Public

It is OPHS policy to ensure and maximize the quality, objectivity, utility, and integrity of information that it disseminates to the public according to the standards set forth in the OMB, HHS and OPHS guidelines. OPHS offices strive to provide information that is accurate, reliable, clear, complete, unbiased, and useful. OPHS offices are committed to integrating the principle of information quality into every phase of information development, including creation, collection, maintenance, and dissemination.

The general standard for information disseminated by OPHS offices is the best available public health and science information. Best available is determined by information published in the highest quality peer-reviewed journals, comparison with best practices as established by the relevant discipline, and reviews by expert panels, individual subject experts, external Advisory Committees, Coordinating Committees with subject experts from HHS agencies, Steering Committees, and staff review.  OPHS reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination. For consumer and professional educational materials, appropriateness of the materials for the intended users is a special focus. Appropriateness is determined through staff review, focus groups, user surveys, audience testing, and dissemination of information and materials for public comment. Depending on the subject matter, disseminated information may also be subject to review by legal staff.

Further, staff strives to collect updated, timely information and remain aware of emerging and newly developed data.  In addition, staff is committed to demonstrating in the Paper Reduction Act (PRA) clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and OPHS information quality guidelines.

V. Agency Administrative Complaint Procedures

OPHS has developed administrative mechanisms to allow affected persons to seek and obtain correction of disseminated information that does not comply with OMB, HHS and OPHS guidelines.

1. Responsibility of the Complainant

   To seek a correction of information disseminated by the agency, individuals should follow the procedures described below. Complainants should be aware that they bear the ‘burden of proof’ with respect to the necessity for correction as well as with respect to the type of correction they seek.

   A complaint or request for review and correction of information must be in written hard copy or electronic form; sent to the agency by mail or electronic-mail (e-mail); and state that an information request for correction is being submitted.

   The complaint shall contain

   1. a detailed description of the specific material that needs to be corrected including where the material is located, i.e. the publication title, date, and publication number, if any, or the Web site and Web page address (url), or the speech title, presenter, date and place of delivery; and
   2. the specific reasons for believing the information does not comply with OMB, HHS or OPHS guidelines and is in error and supporting documentation, if any;
   3. the specific recommendations for correcting the information;
   4. a description of how the person submitting the complaint is affected by the information error; and
   5. the name, mailing address, telephone number, e-mail address, and organizational affiliation, if any, of the individual making the complaint.

   Complaints should be addressed and submitted to:

   Executive Officer
   Office of Public Health and Science
   U.S. Department of Health and Human Services
   200 Independence Ave, S.W.
   Washington, D.C. 20201

   Alternatively, complaints may be e-mailed to: hthompson@osophs.dhhs.gov

2. Responsibility of the Agency

   Based on a review of the information provided, the relevant OPHS office will determine whether a correction is warranted and, if so, what action to take. The office will respond to the requestor by letter or e-mail. The office's response will explain the findings of the review and the actions that the office will take, if any. The response will consider the nature and timeliness of the information involved and such factors as the significance of the correction on the use of the information, the magnitude of the correction, and the resource requirements for the correction. The response will describe how the complainant may request reconsideration. The office will respond to all requests for correction within 60 calendar days of receipt. If the request requires more than 60 calendar days to resolve, the office will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

3. Appeals

   If the individual submitting the complaint does not agree with the office's decision (including the corrective action, if any), the complainant may send a written hard copy or electronic request for reconsideration within 30 days of receipt of the office's decision. The appeal shall state the reasons why the office response is insufficient or inadequate. Complainants shall attach a copy of their original request and the office response to it, clearly mark the appeal with the words, "Information Quality Appeal" and send the appeal to the OPHS appeals address.

   The office official who resolved the original complaint will not have responsibility for the appeal.  OPHS will respond to all requests for appeals within 60 calendar days of receipt.  If the request requires more than 60 calendar days to resolve, OPHS will inform the complainant that more time is required and indicate the reason why and an estimated decision date.

   Appeals should be addressed and submitted to:

   Executive Officer
   Office of Public Health and Science
   U.S. Department of Health and Human Services
   200 Independence Ave, S.W.
   Washington, D.C. 20201
   hthompson@osophs.dhhs.gov

VI. Influential Scientific, Financial and Statistical Information

Given the OPHS mission, from time to time, OPHS offices disseminate information that is regarded as influential. In these instances, OPHS adheres to the highest standards of transparency about information sources, methods and analytical techniques. This influential information is in the form of scientific and technical reports. These reports compile, synthesize, and analyze state-of-the-art knowledge about high priority public health issues that have not previously received sufficient attention. The reports include data from published sources and the public domain, as well as expert opinion, consensus, and recommendations. The sources of underlying data are referenced in reports. Typically, the sponsoring or disseminating office does not conduct original research for these reports, although the office may convene Advisory Groups, Steering Committees, Coordinating Committees or similar bodies as well as take public comment. This input may become part of the report.