HHS Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated to the Public. V. Agency Quality Assurance Policies, Standards, and Processes


As described in the HHS Implementation Plan, as one of the HHS operating divisions, we have established a number of quality assurance policies, standards, and processes for ensuring the quality of the information we disseminate to the public. Our documents undergo a rigorous review and clearance evaluation according to pre-established procedures, documented in our regulations and guidances.

Generally, Agency documents are cleared as follows:

  • Document is developed by an individual or team
  • Document is circulated to working group members, and often an editor, and comments are incorporated
  • Document is circulated and cleared by center managers
  • We publish many documents (for example, guidances, proposed rules) for comment by members of the public, and some documents are reviewed by outside advisory committees comprising experts in the subject matter of the document
  • If required by regulation or policy, documents are circulated to and cleared by the Office of the Chief Counsel, Office of Policy, Planning, and Legislation, the Department (HHS), and the Office of Management and Budget (OMB).

In addition to these clearance procedures, we use a number of mechanisms to ensure the quality of the information we disseminate. FDA reviews the quality (including the objectivity, utility, and integrity) of information before it is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance, and dissemination. Quality, as defined in the OMB Guidelines, encompasses (1) utility, the usefulness of the information to its intended users, including the public; (2) objectivity, whether information is being presented in an accurate, clear, complete, and unbiased manner; and (3) integrity, the information is protected from unauthorized access or revision.

  1. Utility

    We only disseminate information that we believe will be useful to the public or a segment of the public. In fact, often we disseminate information because members of the public or the regulated industry have requested it. We develop many guidances as a result of public questions about a specific topic. We also have processes (21 CFR 10.30) by which members of the public can petition us to take certain actions, such as initiating rulemaking or taking specific administrative or enforcement actions. Requests for dissemination of information also can be submitted through petitions.

    We developed our good guidance practice (GGP) policy as a result of public request. Congress later enacted the policy into law, and we codified our GGP policy in our regulations at 21 CFR 10.115. The GGPs describe our procedures for developing, issuing, and using guidance documents and include detailed procedures on how members of the public can suggest areas for guidance development, submit drafts of proposed guidance documents, and request the revision or withdrawal of an existing guidance document. We also maintain a guidance Agenda, which is a list of guidances we are planning to develop in the coming year. We post the list on the Internet and publish it annually in the Federal Register. We publish the Agenda to keep the public up-to-date on guidance development plans and solicit input from the public on what guidances are needed.

    In addition, we are subject to the Freedom of Information Act and the Electronic Freedom of Information Act Amendments (5 U.S.C. 552), which provide for the dissemination of information to members of the public and posting on the Internet certain information that is, or is likely to be, responsive to multiple information requests.

    In accordance with the Regulatory Flexibility Act (5 U.S.C. 602), the General Services Administration publishes a semiannual regulatory agenda describing the regulatory actions being developed. The Secretary welcomes comments on this agenda and suggestions for improvements and initiatives.

  2. Objectivity

    As already mentioned, we have many different systems in place to ensure that the information we disseminate is presented in an accurate, clear, and unbiased manner. We have a strong commitment to writing all our new documents in plain English. We have provided plain English training to many of the employees who write our documents. We also continue to solicit feedback from stakeholders on our efforts to present written information clearly.

    We also take steps to ensure that our regulatory decisions are based on objective information. If we rely on information submitted to us by third parties in support of an application for product approval or in a rulemaking proceeding, we make sure that this information meets the appropriate standards for quality and objectivity. We have a number of regulations and guidances that set standards for the generation of information in support of regulatory decisions. For example, the supporting data may be generated in new research using good laboratory practices (GLPs) 3, in clinical studies subject to Good Clinical Practices (GCPs) 4, in reviews of existing information obtained primarily from peer-reviewed scientific literature, or obtained from surveys based on widely accepted scientific survey techniques. Interpretations of quantitative results of Agency studies are commonly subjected to statistical analyses. Even though we may be unable to fully disclose the information on which our actions are based because it has been submitted by a third party with a proprietary interest in maintaining its confidentiality, we make special efforts to ensure that the information submitted meets these standards for objectivity and is of high quality.

    The methods by which we ensure the objectivity of the information for some of our major regulatory activities are described here.

    1. Product Review Activities

      One key FDA responsibility is the evaluation of data submitted to the Agency in medical and veterinary medical product applications and in food and color additive petitions or notifications. In general, firms that want to market certain products (for example, drugs and medical devices) submit applications to FDA. These applications contain data or information on which the firm relies to claim that its product is safe and effective for its intended uses. We base our decisions about safety and effectiveness primarily on our analyses of the integrity of the submitted data. When we approve a product and post a drug review package on the Internet, we are ensuring that our analyses of the data submitted to us are available for public scrutiny.

      We develop regulations and guidance documents to help ensure that the data submitted to us result from the best available studies, that the studies are conducted in accordance with sound and objective scientific practices, and that the data are collected using scientifically accepted methods. For example, FDA regulations specify the format and content of the clinical studies that are submitted in support of an application to market a new drug product. They specify how the data are to be collected and the types of analyses that are to be performed. In the case of biological products, we have developed guidance on the format and content of reports on clinical studies that are submitted to the Agency. Other FDA guidances provide detailed descriptions of appropriate methodologies, analyses, and procedures.

      Since the early 1990s, we have been involved in an intensive international effort to harmonize technical requirements for the conduct of studies in support of marketing applications and the content and format of applications with the goal of allowing the submission of a common application for marketing around the world. The International Conference on Harmonisation for Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) brings together scientific experts from different countries to develop a consensus on the appropriate requirements. We also are engaged in international activities in the device, food, and animal drug areas. For example, International Cooperation on Harmonisation of Technical Requirements for Registration of Veterinary Products (VICH) is the veterinary counterpart to ICH. The Global Harmonization Task Force (GHTF) is working to harmonize device regulations and guidance. The Codex Alimentarius committees are working to harmonize international food regulations. Many agreements reached are then embodied in regulations issued through notice-and-comment rulemaking and in guidance documents that describe in more detail appropriate ways to comply with the regulations. As a result of these efforts, most of our actions on product approval applications are consistent with international standards for data collection and quality of analysis.

      We also are in the process of developing good review practices (GRPs) for drug reviews with the goal of making our drug product review process consistent across all divisions in the Center for Drug Evaluation and Research. A major emphasis of the GRP project is to ensure that the reviews we make available to the public are consistently formatted and clearly written so interested individuals can access important health and safety information.

      We frequently consult with scientific experts on product approval applications and broader issues. We have 31 standing Advisory Committees 5, whom we routinely consult on whether the data in particular applications are sufficient to support an approval decision. As noted above, we incorporate our approval decisions into drug approval packages and device summaries of safety and effectiveness that contain our analyses of the submitted data. These packages and summaries do not include confidential commercial, trade secret, and other information exempt from disclosure when we place them on the Internet.

    2. Food Safety Activities

      One of FDA's major areas of responsibility is ensuring the safety of the food we eat. Food safety activities include research, risk assessment, inspections, surveillance, compliance, education, and system coordination activities. We must make sure that the information we provide on food safety is presented in an accurate, clear, complete and unbiased manner. This means that the data on which we base our decisions must be collected in an objective manner using sound scientific principles for data collection.

      We collect information to support our food safety activities through many sources, including research, risk assessment, inspection and surveillance, peer-reviewed literature, and advisory committee opinions. We conduct in-house research on a variety of food safety topics and also fund a substantial amount of extramural research every year. The Center for Food Safety and Applied Nutrition (CFSAN) participates in collaborative research on processing and packaging through the National Center for Food Safety and Technology, which is a consortium of government, industry, and academia, and we coordinate food safety research activities through a cooperative program with the University of Maryland. Topics of both internal and external research interest are determined by CFSAN's 3-year research plan. This plan is developed in conjunction with other federal agencies to prioritize our research to inform our most critical food safety efforts and to avoid duplication of effort.

      We also gather information for certain food safety activities through risk assessment. Risk assessments are a very useful tool for evaluating the benefits of pursuing various rulemaking strategies. To date, we have conducted or been involved in four risk assessments related to food safety. We conducted a joint risk assessment with USDA on Salmonella Enteritidis in shell eggs that was published in 1998. More recently, we published risk assessments on the Public Health Impact of Vibrio parahaemolyticus in Raw Molluscan Shellfish and the Relative Risk to Public Health from Foodborne Listeria monocytogenes Among Selected Ready-to-Eat Foods. We also intend to use the risk assessment model developed for USDA by Harvard University on bovine spongiform encephalopathy to determine the risk reduction outcomes of various rulemaking efforts we are considering.

      We gather data for FDA food safety activities through use of surveys designed for specific purposes and advisory committee opinions. The surveys include the Health and Diet Survey and the Total Diet Study, both of which are used in our development of safety and exposure assessments for various compounds. When information is not available through research or literature, we have several advisory committees that are able to render expert opinions on particular matters. These committees include the National Advisory Committee on Microbiological Criteria for Foods, which considers a variety of food safety issues for FDA and USDA, and FDA's Transmissible Spongiform Encephalopathy Advisory Committee (TSEAC), which specifically considers issues related to TSE diseases. Finally, our food safety activities are informed through the participation of FDA scientists in a variety of professional organizations such as Codex Alimentarius, International Commission on Microbiological Specifications for Foods, Institute of Food Technologists, American Society for Microbiology, the International Association for Food Protection, Society for Toxicology, American Chemical Society, the National Academy of Sciences, International Life Sciences Institute, and editorial review boards of several publications including Journal of Food Protection and Journal of Food Science.

    3. Adverse Events Analysis for Medical Products

      Once products are marketed, we continue to monitor their safety after approval and disseminate information about their risks to health care providers, patients, and consumers. We undertake a number of data collection activities to ensure the objectivity of the information we disseminate on medical products.

      1. Human Drug and Biological Products

        The Adverse Event Reporting System (AERS) is an Oracle-based computerized information system designed to support the Agency's postmarketing safety surveillance program for all approved drug and therapeutic biologic products. The structure of the database complies with the international safety reporting guidance (E2B Guidance on Data Elements for Transmission of Individual Case Safety Reports, January 1998 6), including content and format for electronic submission of the reports from the manufacturers. The ultimate goal of AERS is to help reduce the risks associated with medical product use by providing the best available tools for storing and analyzing safety reports. Information from this system is used to support decisions to disseminate information of product safety. By systematizing the submission of data to the Agency, we have greatly improved the quality (and objectivity) of related decisions and information dissemination efforts.

        A separate system is used to monitor the safety of vaccines after approval. The Vaccine Adverse Events Reporting System (VAERS) is a cooperative program for vaccine safety of the FDA and the Centers for Disease Control and Prevention (CDC). VAERS is a postmarketing safety surveillance program, collecting information about adverse events that occur after the administration of U.S. licensed vaccines. Other systems are in place to monitor the quality of manufacturing of drugs and biological products and blood related products.

      2. Medical Devices

        We use a variety of tools to identify problems and safety issues related to medical devices that are approved and being used by health care practitioners and consumers. Tools include both voluntary and mandatory reporting of adverse events; monitoring of product performance through other data sources, such as registries and various research efforts; and the use of both mandatory and voluntary postmarket studies aimed at examining specific safety issues.

        Adverse events related to either product problems or issues associated with the use of the device are reported by both manufacturers and device users. Although both the manufacturing and user communities have mandatory reporting requirements for device-related problems, this surveillance system is virtually a passive system that depends on the reporter to recognize an event and follow through in reporting. This passive surveillance system is augmented by a more active reporting network composed of hospitals and other health care facilities, where reporting is encouraged and supported through educational activities and feedback. Adverse event reports are immediately triaged to quickly identify problems that require urgent attention; all reports are then reviewed by clinical analysts and others with appropriate expertise to decide if further follow-up is needed.

      3. Animal Drugs

        An Adverse Drug Event (ADE) report for veterinary medicinal products consists of either an undesired side effect or the lack of a desired effect associated with drugs administered to animals. Reports may also involve product defects and potential harm posed to persons administering or using animal drugs. For example, in the year 2000, FDA's Center forVeterinary Medicine reviewed 14,497 ADE reports consisting of: 13,757 undesired side effects and lack of desired effect; and 740 product defects.

        Adverse event reports related to animal drugs are maintained in a separate database, the Center for Veterinary Medicine's adverse drug event reporting system (ADERS). This database is used to identify adverse effects not detected during pre-market testing of FDA-approved animal drugs and to monitor the performance of drugs not approved for use in animals. The ADERS depends upon the detection of an adverse clinical event by veterinarians and animal owners, the attribution of the clinical event to the use of a particular drug ("suspect" drug), and the reporting of the ADE either to the manufacturer of the suspected drug or directly to FDA. Data from these ADE reports are reviewed, coded and entered into the computerized ADERS.

        The ADE for veterinary drugs generates current information on the safety and efficacy of veterinary drugs. These data expand the knowledge base used in animal drug approvals and ultimately contribute to reducing the risks associated with veterinary medical products. Summary information from this system is available to support decisions about disseminating information on product safety.

      4. Foods, Including Dietary Supplements, and Cosmetics

        We use several reporting systems to identify problems associated with foods, including dietary supplements, and cosmetics.

        • The Adverse Reaction Monitoring System (ARMS) collects spontaneous reports from consumers and health professionals regarding alleged adverse effects from food products.
        • The Special Nutritional Adverse Event Monitoring System (SN/AEMS) collects spontaneous reports from consumers and health professionals regarding adverse effects from special nutritional.
        • The Cosmetic Adverse Reaction Monitoring System (CARMS) collects spontaneous reports from consumers and health professionals regarding alleged adverse effects from cosmetic products.
        • CFSAN receives adverse event reports linked to the products it regulates through FDA's MedWatch program.
    4. Research to Support the Regulatory Functions of FDA

      FDA conducts peer-reviewed research that supports the regulatory functions of the Agency. To accomplish this mission, the FDA seeks feedback from its stakeholders and partners, including its center scientists, other government agencies, industry, and academia. In addition, the FDA Science Board (SB) provides guidance on specific complex and technical issues as well as emerging issues within the scientific community, in industry, and academia. The SB conducts ongoing reviews of the research programs themselves and how these programs are keeping pace with technical and scientific evolutions in the fields of regulatory science. Their review includes advice on developing an appropriate research agenda and on upgrading FDA's scientific and research facilities to keep pace with these changes. The SB is composed of non-governmental scientists from industry, academia, and consumer organizations. This board is further supplemented with subject matter experts and scientists representing all FDA centers. Research programs can be via collaborations through partnerships with other scientific organizations. Scientific and monetary collaborations include interagency agreements with other government agencies, Cooperative Research and Development Agreements and technology transfer with industry, and grants or informal agreements with academic institutions.

      Each FDA Center and Office also uses several strategies to ensure the quality (including objectivity) of its extramural research and the accuracy of data collected in association with these studies. For example, study protocols may be developed collaboratively by principal investigators and program officials. A project advisory group (PAG) consisting of experts who oversee extramural research projects in a particular field may be appointed.

      To ensure the data are accurate and timely, FDA monitors research progress at the project level on a recurring basis. FDA quality assurance staff assess experiments that fall within the scope of FDA's Good Laboratory practice guidelines for compliance. Additionally, agency scientists publish research manuscripts, book chapters, and abstracts in recognized, peer-reviewed scientific journals. FDA's research findings are also presented at national and international scientific meetings, many of which are sponsored or co-sponsored by FDA. Many FDA scientists also serve on international scientific advisory boards.

  3. Integrity

    We strive to maintain the integrity of the information we collect and use, and protect it against disclosure, alteration, loss, or destruction. We require all of our operating divisions to adhere to a series of Agency guidelines to ensure our data integrity operations. Guidelines include:

    • FDA Staff Manual Guide 3250.17 Data Security\Data Integrity
    • Computer Security Act of 1987
    • Computer Fraud and Abuse Act of 1986
    • Clinger-Cohen Act of 1996
    • Federal Managers Financial Integrity Act
    • Government Information Security Reform Act (GISRA)
    • Office of Management and Budget (OMB) Circular A-130, Appendix III, Management of Federal Information Resources, Revised 2/96
    • HHS Automated Information Systems Security Program Handbook (Release 2.0), 5/94
    • NIST Special Publication 800-14, Generally Accepted Principles and Practices for Securing Information Technology Systems (September 1996)

    All data submitted for inclusion in our systems must be accompanied by information about origin, sensitivity, reliability, and the date of most recent revision. We have systems in place to ensure that data modifications are accomplished in a managed and controlled manner and that all information is protected from unauthorized access, revision, corruption, or falsification.

    Transactions affecting sensitive or valuable information can only be processed if the originating individual or system has been validated as authorized to submit such transactions. Additionally, transactions must be initiated only through source documents or computerized messages in which the originating individual or system is clearly identified. All transactions intended for input into a multi-user production computer system must first be subjected to reasonableness checks, edit checks, and/or validation checks.

    Transactions that fail such checks must either be:

    • Rejected with a notification of the rejection sent to the submitter
    • Corrected and resubmitted or
    • Suspended pending further investigation

    Management has established and maintains preventive and detective security measures that ensure that our information is protected from undetected alteration. All rejected input transactions are placed in a suspense file and listed in exception reports until they are successfully resubmitted for processing. Resubmission and corrections are subject to the same validation procedures that original input transactions receive. Management reviews the reasonableness and accuracy of all changes to internal records. If a client or customer brings record errors to our attention, an investigation of the errors is initiated promptly.

    The Office of Public Affairs provides guidance to the Agency on best communications principles and practices to help assure that public messages are accurate, understandable and consistent with current Agency policies. The quality, objectivity, utility and integrity of information posted on the FDA Internet is assured by the internal clearance process used to clear printed documents.