The quality assurance process begins at the inception of the information development process.
Information released by OIG is developed from reliable data sources utilizing accepted methods for data collection and is based on thoroughly reviewed analyses and models. The guidelines below describe procedures that OIG employs to assure the quality of its information products. Quality is an encompassing term comprising utility, objectivity, integrity, and reproducibility.
Utility involves the usefulness of the information to its intended audience. The purpose of OIG documents and information is to identify fraud, waste and abuse and to protect the integrity of HHS programs. The OIG issues findings and recommendations on the efficiency, effectiveness, and vulnerabilities of departmental programs.
Objectivity involves a focus on ensuring that information is accurate, reliable and unbiased and that information products are presented in an accurate, clear, complete and unbiased manner. Objectivity is achieved by using reliable data sources and sound analytical techniques, and preparing information products that use proven methods by qualified individuals that are carefully reviewed. Below is a description of the quality assurance guidelines that are used to ensure objectivity and accuracy of information.
OIG reviews the quality (including the objectivity, utility, and integrity) of information before its is disseminated and treats information quality as integral to every step of the development of information, including its creation, collection, maintenance and dissemination.
Quality Assurance Process
The quality assurance procedures is a collaborative, team effort by staff who conduct program inspections and staff who assist, guide, and review written and oral inspection plans and products. Any individual involved with a particular inspection is responsible for helping assure quality of work done and products released. Further, OIG is committed to demonstrating its Paper Reduction Act (PRA) clearance packages that each draft information collection will result in information that will be collected, maintained, and used in a way that is consistent with OMB, HHS, and OIG information quality guidelines.
Specific responsibility for conducting and ensuring quality program inspections rests with Regional Inspectors Generals (RIGs). RIGs assign a Team Leader and other team members to each inspection. This team is responsible for doing quality inspection field work -- including design, data collection, data analysis, and written and oral reports. RIGs must certify for final reports that the inspection was done in accordance with OIG procedures and PCIE quality standards.
If an audit methodology is used to develop the product, the Government Auditing Standards are employed as described below:
- Government Auditing Standards State that "Each audit organization conducting audits in accordance with these standards should have an appropriate internal quality control system in place and undergo an external quality control review."
- This control system is to provide reasonable assurance that the audit organization (1) has adopted, and is following, applicable auditing standards (which includes reporting "the views of responsible officials") and (2) has established, and is following, adequate policies and procedures.
- To determine whether in fact the audit organization's quality control system is operating effectively, the organization is required by standards to have "an external quality control review at least once every 3 years by an organization not affiliated with the organization being reviewed."
OIG's quality control program ensures that audit work performed meets government auditing standards. It consists of two elements:
- The OIG System of Quality Control consists of independent report referencing, Headquarter Desk Reviews and Internal Quality Control Reviews. The latter includes review of working papers related to selected reports.
- External Reviews -- These reviews, also known as peer reviews, are performed by Federal auditors outside the OAS. The external review is required by the Comptroller General's standard on quality control and should be conducted at least once every three years.
Report Validation is a process that is used to help ensure that the facts, findings, conclusions, and recommendations contained in OIG reports are accurate, reliable, and supportable by inspection work papers and analysis.
Report validation is conducted prior to submitting the draft report for IG signature. For objectivity, validation is performed by regional staff that are not a part of the inspection team. To facilitate this process, the draft report is cross indexed to applicable work papers. Work papers must include documentation that inspection teams verified data presented in draft reports by checking it against supporting evidence in the work papers.
To help assure quality and impact of its inspection results, OIG obtains comments and other input from applicable agency staffs on inspection plans, results, and selected products. Inspection teams obtain such input in a variety of ways, including entrance, exit, and other conferences; and comments on inspection designs, data collection instruments, and draft and final reports. Inspection teams use such meetings and reviews to help identify client needs, identify relevant data for accomplishing inspection purposes, verify accuracy and reliability of data collected, and verify soundness of findings, conclusions, and recommendations developed.
Reliability of Data Sources
To maintain credibility, OIG must take reasonable steps to assess the reliability of pre-existing computerized data used as the basis for inspection findings and recommendations. Many inspections either begin with a computerized sample selection or are based entirely on analyzing data extracted from computerized records not under OIG’s direct control. Project staff do not assume that such computer extracts or sample selections are complete or that they accurately reflect the universe of people or transactions being studied.
Data reliability means the degree to which data extracted from computer records for a program inspection completely and accurately reflect the individuals or transactions being studied. This is a relative concept, one that recognizes that data with errors may still be usable, if the errors are not of a magnitude that would cause a reasonable person to doubt findings or conclusions that are based on the data.
To provide reasonable assurance of computerized data reliability we:
- Identify prior reviews by OIG, GAO or by system managers attesting to the computer system and data reliability,
- Review the data dictionary, if it exists, for the database to assure a full understanding of the relevant data elements’ structure, content, how the elements are derived, and their interrelationships before requesting data extractions,
- Obtain frequency counts of critical data elements to determine if the data selection criteria are providing the information anticipated,
- Conduct data accuracy tests to ensure that required data elements have been provided and are in the expected format,
- Obtain detailed printouts for a sub-sample of records included in the data extract to confirm that the extraction produced the types of records sought and the required information from those records,
- Obtain source documents (e.g., claim folders) for a sample of extracted records to determine the validity of the data contained in the automated records, if the data reliability is questionable, and include a statement in the Methodology section of draft and final reports describing the extent of reliability testing performed and our confidence in the data used.
Integrity refers to the security of information from unauthorized access or revision to ensure that the information is not compromised through corruption or falsification. To ensure the integrity of information, OIG has in place rigorous controls that have been identified as representing sound security practices.
OIG is highly protective of the confidentiality of information it holds through its policies and practices. OIG has in place programs and policies for securing OIG resources as required by the Government Information Security Reform Act (P.L. 106-398, Title X, Subtitle G). OIG is subject to statutory requirements to protect the sensitive information it gathers and maintains on individuals.
If an agency is responsible for disseminating "influential" information, guidelines for dissemination should include a high degree of transparency about data and methods to facilitate its reproducibility by qualified third parties. Information is considered influential if it will have a substantial impact on important public policies or important private sector decisions. Since many of the OIG's Inspection reports have an impact on important public policies, OIG's information that is subject to section 515 should be highly transparent and capable of being reproduced by qualified persons.
OIG's guidelines call for identification and documentation of data sets used in producing estimates and projections and clear description of the methodology used to produce the analytical results. Some results included in OIG reports are not directly reproducible by the public because the underlying data sets used to produce them are confidential. However, those inspections that are based on publically available data and are made available on request are fully reproducible by the public.