Interim Evaluation Report: Congressionally Mandated Evaluation of the State Children’s Health Insurance Program. B. Development of the Evaluation Plan


The congressional mandate was broad in scope, and to fulfill it required that a complex set of activities be organized and implemented rapidly. The Office of the Assistant Secretary for Planning and Evaluation (ASPE) in the Department of Health and Human Services (DHHS) was assigned responsibility for the evaluation. The task of translating Congress' comprehensive mandate into a responsive and manageable evaluation plan was challenging. To carry it out, ASPE consulted widely with policy experts and researchers, with the aim of producing a robust specification of the questions the evaluation should address and thorough consideration of methodological issues. This consultation also enabled ASPE to devise a strategy that: built on existing research efforts to the extent possible; avoided duplication of other work; and maximized the potential of the evaluation to add to our knowledge about SCHIP and Medicaid.

Federal and state policymakers and researchers both within and outside the government participated in these early planning efforts and discussions, which formed the basis for ASPE's decisions about the evaluation design and methods. A multi-faceted research strategy ultimately emerged that, while addressing the broad range of issues articulated in the mandate, also met the need for results in the near term. The three-year evaluation project has numerous research components that include both qualitative and quantitative approaches and that bring data and information from multiple sources--including states, health care providers, families, and other interested parties--to bear. While time, resource and legislative constraints necessitated this somewhat complex strategy, the design is expected to produce a uniquely rich assessment of the program.

The evaluation plan ASPE developed included three contractual components: (1) an overall study of the issues identified in the legislative mandate, but focused primarily in ten states; (2) a focus group study in nine states during 2001; and (3) special studies on issues of immediate policy interest, such as the experience of states providing family coverage under SCHIP. Although conducted under separate contracts, the findings from the three different components will be fully integrated by the investigators under the contract for the overall study. 4, 5, 6, 7

The central focus of the overall study is states' programmatic and policy choices and their implications. Data and information for this study will come from three sources: (1) in-depth case studies of ten states; (2) a nationally representative survey, in the same ten states, of parents of SCHIP enrollees and disenrollees, and, in two of the states, of parents of Medicaid enrollees and disenrollees; and (3) a survey of SCHIP administrators in all 50 states, the District of Columbia, and the five U.S. territories. The study will also draw on a national survey of awareness and perceptions of Medicaid and SCHIP among low-income families with uninsured children, using the State and Local Area Integrated Telephone Survey (SLAITS), as well as states' SCHIP and Medicaid administrative files. Finally, as indicated earlier, the overall study will incorporate the findings from the focus group study and the studies of special topics. These data sources are described in Chapter III as part of the discussion on the evaluation design.

4. The contract for the overall study was awarded to Mathematica Policy Research, Inc. (MPR), under contract number HHS-100-01-00002. MPR subcontracted with The Urban Institute and The MayaTech Corporation. The contract began on December 22, 2000, and will end in October 2004.

5. This is the second study of SCHIP to be mandated by Congress. In the legislation creating SCHIP (the Balanced Budget Act of 1997), Congress had already mandated that DHHS submit a report to Congress by December 31, 2001, based on mandated state evaluations. Recognizing this statutory requirement--as well as the need for a more detailed assessment of the performance of the SCHIP programs--the Centers for Medicare & Medicaid Services (CMS), which administers SCHIP, contracted for an evaluation.

The CMS evaluation also is being conducted by MPR (contract number 500-96-0016 [03]). That evaluation is reviewing the literature on the effects of SCHIP; drawing on state data to examine SCHIP enrollment, expenditures, and service use; synthesizing information on program performance from the state evaluations and annual reports; and examining outreach and enrollment processes in eight states using site visits and focus groups. In addition, the study will review trends in the number of uninsured children before and after SCHIP implementation, using data from the Current Population Survey (CPS).

6. Health Systems Research, Inc. (HSR) conducted the focus group study for ASPE (Bellamy et al. 2002).

7. George Washington University and the National Academy for State Health Policy conducted the family coverage study for ASPE (Kaye, Wysen, and Pernice 2001)

View full report


"Interim.pdf" (pdf, 2.84Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®