HHS Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated to the Public. V. Agency Quality Assurance Policies, Standards and Processes for Ensuring the Quality of Information Disseminated to the Public

10/01/2002

HRSA reviews the quality (including the objectivity, utility, and integrity) of information before its is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination.

  1. Program information:

    Program information provided by grantees in the form of application data, reporting requirements, performance reports and progress reports, receives a series of reviews prior to any dissemination to the public. Grantees conduct internal reviews in order to provide data that are accurate, consistent, and complete. The HRSA Office or Bureau conducts a standard review to ensure data quality, completeness, and reliability.

  2. Reports, publications, and other products:

    HRSA publications, audiovisual products, and exhibits are required to be consistent with government-wide and HHS public affairs policies. HRSA has published guidelines which closely follow the regulations described in the HHS Public Affairs Management Manual and Government Printing and Binding Regulations to assist program staff in obtaining appropriate review and clearance of information products. To produce a publication, a complete HHS-615 (Publication Planning and Clearance Request) form must be submitted to, and approved by, ASPA.

    Publications to be produced at HRSA's request under contract are subject to HHS clearance and the Joint Committee on Printing (JCP) printing procedures. An HHS-524 clearance is required for publications to be developed as part of a larger public affairs services contract. The HHS-524 must be approved before a Request for Proposals (RFP) can be issued.

  3. Statistical data disseminated to the public:

    At HRSA, the quality assurance process begins at the inception of the information development process.

    For information disseminated from sample surveys and other research related or evaluative activities, the widely accepted standards of technical and scientific review are utilized to ensure data quality. These standards include, where appropriate, peer review, internal expert review, Institutional Review Board (IRB) review and OMB review.

    For each of the examples provided earlier, the following quality assurance procedures are employed.

    1. The HRSA State Profiles are reviewed by program staff from each Bureau providing input into the profile prior to dissemination. This is an interval review by each Office and Bureau from which data are obtained. The originating office is responsible for ensuring that all necessary internal review, approval, and clearance is obtained prior to dissemination. Where indicators are taken from other sources, that source is clearly indicated and the limitations of the data are described. For the demographic and other data derived from other sources, HRSA provides an explicit list of the source information, the year the data were obtained, citations, and related URLs. For example, for health indicators such as percent of people insured in the State who had private insurance, the source listed is the U.S. Census Bureau, Health Insurance Statistical Tables, and the citation is Table HI-4, Health Insurance Coverage Status and Type of Coverage by State-All Persons, 1998. Each data element contains such a source listing along with definitions.
       
    2. The HIV/AIDS Bureau uses CAREWare, a software package for use by the HIV/AIDS Bureau providers to track clients and services. The software and manual are available online, and the system captures all the data items required for the reports needed. CAREWare contains consistency and edit checks on inputted data for quality assurance. This system was designed specifically to collect the exact data elements and generate complete reports. The HIV/AIDS program staff review all data for verification prior to dissemination. The CARE Act State Profiles data provides, for each element, a source list and year of data, as well as notes relating to limitations, rounding, and restrictions, where appropriate. For example, for the Title II Profile Report, Clients Served by gender, age, race/ethnicity, exposure category, the source is listed as the Annual Administrative Report to HRSA, with a note that Accounts are rounded to the nearest 10. Data are not reported from cells of fewer than six clients.
       
    3. The Maternal and Child Health Bureau Title V annual report uses an electronic reporting package that was developed in collaboration with the states in order to improve accuracy and completeness of the report. This reporting package provides for automatic calculations of ratios, rates, and percentages, and carries data over from year to year, and assures that data used in multiple tables is entered only once. This system has in place a mechanism that warns states when conflicting or unacceptable figures appear as a quality check on the final data. Knowledgeable MCHB program staff monitor and review all information submitted in the grant annual report for completeness, accuracy, and reliability.
       
    4. The HRSA Preview undergoes appropriate internal Division, Bureau, and Office review and approval prior to dissemination. HRSA adheres to all accepted standards applying to the publication of information in the Federal Register. The HRSA Preview undergoes extensive Office/Bureau, and Agency wide review to ensure accuracy, reliability, and comprehensiveness prior to dissemination to the pubic.
       
    5. The Area Resource File provides county level data from a variety of data sources. As a result the data have a number of limitations and users of the file are provided with a disclaimer as to the applicability and detail of the data. The ARF utilizes secondary data only, and the years and detail for which all county level data are available are considerably limited. Each data source conducts its own review prior to the release of information. Knowledgeable staff within the originating office and across offices conduct standard reviews for completeness and reliability of the data.
       
    6. The National Sample Survey of Registered Nurses received extensive review during each phase of operation: survey design, sample selection, fielding of the survey, data editing, and analysis. Clearance was required of all proposed sampling methods, survey procedures, the data collection instrument, and analytic plans by HRSA, HHS, and OMB. This clearance requires a series of expert review. A scientific review, an internal review, and IRB review were conducted to ensure that survey procedures, data collection, editing, and analyses would adhere to the highest standards.

    As in all research activities involving human subjects, it is HRSA's policy that adequate protection of participants be ensured in accordance with the provisions of 45 CFR Part 46. Project plans and procedures must be submitted to the appropriate IRB for approval, and the program must execute all necessary Assurances of Compliance with OPRR. The NSSRN obtained IRB review and approval for all survey methods and procedures.

    For the National Sample Survey of Registered Nurses, widely accepted standards as established by the statistical community are utilized to assure data survey quality. For example, the median design effect for the survey was 1.66, an accepted level by the statistical community for a large national survey. The information disseminated in the report provides extensive detail on the analytic issues (sampling and nonsampling errors, standard error calculation, variance estimation, etc.). As with any sample survey, the results are subject to sampling error. The report provides detail on measures of variability, standard errors, information on the computation of the sampling variance and the design effect.

    Further, HRSA strives to demonstrate in its Paper Reduction Act (PRA) clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS and HRSA information quality guidelines.