The purpose of discovery activities is to identify instances where the program is not operating as intended and is out of compliance with the Federal assurances. Discovery activities, however, are not an end in themselves but rather (and most importantly) a means to identifying problems so that they can be addressed. In the CQI waiver cycle this is called Remediation. The focus of remediation is to address and resolve all individual problems uncovered in the course of discovery.
Using Sampling Parameters to Determine a Representative Sample
Three parameters must be set to determine sample size: (1) Confidence Level, (2) Confidence Interval (also called the Margin of Error), and (3) Distribution (of the variable) in the Population. CMS considers a sample to be sufficiently representative if the Confidence Level is .95 or larger and the Confidence Interval (Margin of Error) is +/ 5 percent or less. A Confidence Level of 95 percent ensures that one can be 95 percent certain that the estimate derived from the sample is accurate. A Confidence Interval of +/ 5 percent ensures that the actual population value is within +/ 5 percent points of the estimate provided by the sample. A Distribution value of .5 is always acceptable (discussed in more detail below.)
For most states, determining a credible sample size will involve using a sample size calculator. Many calculators may be found online by searching the term sample size calculator. These calculators will prompt the user for values for Population Size, Confidence Level, and Confidence Interval (Margin of Error).
Many calculators will not prompt the user for a value for the Distribution (of the variable) in the Population, but will assume a .5 value. The best calculators, however, will allow the user to specify the Distribution value. It is not possible here to delve into the conceptual underpinnings of this parameter; suffice it to say that under certain conditions states may vary this parameter (from .5), thereby lowering the recommended sample size, and still have a sample that is considered large enough to be representative. There is a rule of thumb states may follow in varying the Distribution parameter: If the state is drawing a sample for the first time in order to collect data for a performance measure(s), it must assume a 50-50 Distribution and use the .5 value.
However, if the state has collected data on a given performance measure(s) previously, and derived the estimate from the entire population or a representative sample, they may alter the distribution to reflect that previous experience. For example, in Year 1 of data collection the state discovered that 90 percent of service plans addressed participants needs, but 10 percent did not. In Year 2, they would be justified in using a 9010 split (.9 Distribution value) in the sample size calculator. To illustrate the difference in sample size that altering the Distribution can make, let us use the example of a waiver with 3,000 participants. With a Distribution value of .5, the recommended sample size is 341, but with a Distribution of .9 (or .1, its reciprocal), the recommended sample is 133.5
If a state can justify using a Distribution other than the standard .5, it can decrease sample size while simultaneously generating performance measure estimates that are credible and representative.
One error states should avoid is simply choosing sample size based on a percentage of the population (e.g., 10 percent of all waiver participants). This approach will often either yield a sample that is not large enough to deliver credible results or will specify a sample that is larger than necessary. Sample size should always be calculated by specifying the sampling parameters discussed above.
Sample Stratification. Sometimes states are interested in obtaining information on how various subgroups--or strata--are performing. For example, a state may want to monitor the performance of care coordination agencies in their responsibility for conducting timely annual level-of-care determination. So, the state decides to stratify its record review sample by care coordination agency. However, before doing so, states need to consider several implications of a stratification approach. First, if there is a desire to compare subgroups, then it is important that the sample for each subgroup be large enough to be representative of each subgroup. In general, the more subgroups, or strata, the larger the overall sample size will need to be. The state must balance its need for information on subgroups with available resources.
Although CMS elicits information from states on the waiver application about stratification, stratification is not a CMS requirement; it is entirely at the states discretion whether to stratify. However, if stratification is used, it is important that the data be re-weighted so that they represent the entire population; a simple averaging of results from the various strata will not produce a valid estimate.6 A statistician or someone with expertise in sampling statistics should be consulted. Technical assistance is also available from CMS as described later in this appendix.
When states use the same quality improvement strategy (QIS) across multiple waivers or across waivers and state-funded HCBS programs (sometimes referred to as a Global QIS), they may be tempted to draw one sample for all the waivers and programs combined. However, CMS requires that evidence be reported for each waiver separately, and that the samples for each waiver be large enough, on their own, to be representative of a given waivers population/providers. In this context, one might hear the issue referred to as stratifying by waiver. If the state wants to sample across multiple programs, it will have to stratify by program and ensure that the sample size for each program is large enough to be representative.
The rate of compliance is measured through the performance measures as discussed above. CMS expects states to be in compliance with the statutory assurances. If a performance measure indicates that the state achieved less than 100 percent compliance, the state must remediate all instances of non-compliance discovered. While a state may not be in compliance initially, it may come into compliance by taking remedial actions. Compliance can occur by appropriately addressing all detected problems. It is clearly preferable, however, for the state to achieve compliance by preventing problems in the first place (i.e., having a high level of performance initially).
As states design their quality strategies they must build in systematic mechanisms for addressing problems as they are uncovered. In the waiver application, states are required to specify these methods. States should have policies and procedures describing (1) who is responsible for monitoring remediation activities and verifying that problems are appropriately addressed, (2) the explicit expectations for timeframes within which problems should be resolved, and (3) what sanctions may be imposed in the event that corrective action is not taken by the responsible party. It is also important that remediation methods be appropriate to the problems uncovered. Corrective actions will differ depending upon the assurance, subassurance, and/or performance measure for which non-compliance was discovered.
Like discovery evidence, states should also be able to aggregate remediation evidence. The aggregation of remedial activities is a states way of summarizing the types and numbers of actions taken in response to non-compliance with regard to a given performance measure. Aggregated data about remediation actions, in the form of remediation reports, provide the evidence CMS requires to ensure that the state has addressed instances of non-compliance. Remediation reports can also be used by the state, along with discovery reports, to identify and analyze trends related to non-compliance.
Table 3 illustrates what a summary discovery and remediation report might look like for one level-of-care subassurance. The discovery and remediation reports are linked because the number and types of remediation actions necessarily follow from the instances of non-compliance uncovered through the states discovery activities. In this example, over the course of one calendar year, the state uncovered 30 instances where a reevaluation to determine the level of care was not conducted on time (Discovery Results).
|TABLE 3. Fictitious State Waiver Quality Monitoring Report, Generated 1-1-2009
Period of Performance: January 1, 2008 through December 31, 2008
Performance Measure: Number and percent of waiver participants who received an annual re-determination of eligibility within 12 months of their initial level-of-care evaluation or within 12 months of their last annual level-of-care re-determination
|| Compliance Rate
|| Number of Actions
|Reevaluation conducted, still eligible
|Reevaluation conducted, not eligible
|Referred to state-funded program
|Claims from period of ineligibility excluded from Federal Financial Participation(FFP)
|# of remediations completed in < 30 days
|# of remediations completed in 31-60 days
|# of remediations completed in > 60
|Outstanding remediation actions
|Total instances of non-compliance addressed
|* In a record review on 12/22/08, a person was discovered to be 35 days overdue for a re-determination. Due to the holidays, the re-determination had not occurred at the time this report was generated.
Twenty-nine of the 30 instances were addressed through the state conducting a reevaluation on each tardy re-determination (Remediation Actions). In 25 of the 30, the person was found to still be eligible at reevaluation and thus no further action was necessary. However, in 4 cases the person reevaluated was deemed ineligible for the waiver, and in these instances the state took further actions to refer these individuals to the state-funded program and to ensure that no FFP was claimed for waiver services provided during the period of ineligibility.
In this example, the state had a policy that all remediation actions related to late level-of-care re-determinations must be accomplished within 30 calendar days of discovery; as such, the state is also tracking whether appropriate remedial actions are taken within the specified timeframe. In addition, the remediation report identifies the number of outstanding problems not addressed on the date the report was generated.
The example provided in Table 3 is a summary report for one year and might be the type of report a state would submit to CMS as evidence. However, the state should have a data system capable of generating more frequent reports, to enable performance monitoring as the discovery and remediation results come in. A key component of a quality assurance system is the ability to discover problems close to their occurrence, which then enables a quick response--be it fixing the original problem or uncovering remediation actions that are inappropriate and/or untimely.