Feasibility Study for the Evaluation of DHHS Programs Operated under Tribal Self-Governance

Unit of Measurement

Data Available

At the Service Unit level

Number of patients provided services in SU, by age and gender

 

Number and type of Contract Health Services provided by quarter of the fiscal year

 

Percent with Medicare

 

Percent with Medicaid

 

Percent with SCHIP

 

Percent with Private Health Insurance

 

Number of hospital admissions

 

Number of hospital days

 

Number of primary care visits

 

Number of specialist physician visits

 

Number of dental visits

 

Number of prescriptions filled

     

 

     

Percent children under age 5 immunized

 

Percent aged 50+ receiving influenza immunizations

 

Percent of women over 18 with annual Pap smears

 

Percent pregnant women obtaining prenatal care in first trimester

 

Percent of adults screened for diabetes

 

Percent diagnosed with breast cancer surviving 5 years

 

Percent diagnosed with cervical cancer surviving 5 years

 

Percent of births that are low-weight or premature

 

Percent of births that are high-weight

 

Percent of deaths attributable to diabetes

   

For each Service Unit, for all patients with diabetes, three years:

Percent seeing physician at least once in 3 months

 

Percent receiving HbA1c testing once in 3 months

 

Percent receiving dilated eye exam annually

 

Percent of people with diabetes who have diabetic retinopathy

 

Percent of people with diabetes who have had amputation

The availability of administrative and personnel-related data elements was also investigated.  These elements would include staffing information, information on staff credentials, staff turnover, pharmacy information, and payment information.  Some of this information are likely available through other components of the RPMS (e.g., accounts payable, contract health, staff credentials), if these components are in use by the Tribal entities of interest.  

Review of Reporting Requirements for Other DHHS Programs

Current reporting requirements for each of the DHHS programs were also investigated by talking with several program staff and reviewing the documents provided to us by the Tribes visited.  The current reporting requirements are described below.

Tribal Temporary Assistance for Needy Families (TTANF).     Currently, Tribes managing this program are required to provide the standard Federal financial reporting form SF269[5] and electronic submission (preferred) of family-level and individual-level data elements for families receiving TTANF. (Some Tribes may qualify to sample the caseloads on which they report these data.) For the family, these data elements include funding stream, number of family members, type of family for work participation, receiving subsidized housing, receiving medial assistance, receiving food stamps and amount, receiving subsidized child care and amount, child support, and family cash resources. At the individual level, Tribes are required to submit characteristics such as adult and minor child head-of-household characteristic such as date for birth, ethnicity, gender, receipt of disability benefits, marital status, relationship to head of household, parent with minor child in the family, needs of pregnant women, educational level, citizenship, cooperation with child support, employment status, and work participation status. The child characteristics submitted by TTANF grantees include family affiliation, race/ethnicity, gender, receiving disability benefits, relationship to head of household, educational level, amount of unearned income,

Low Income Home Energy Assistance Program.     Currently, Tribes managing this program are required to provide the Household Service Report—Short Format or a letter containing similar information.  This information includes number of household receiving the following types of assistance: heating, cooling, winter/year round crisis, summer crisis, or weatherization.  Tribes are also required to file the SF269.

Community Services Block Grant.     Currently, there is no specific Federal required reporting form beyond the SF269.

Child Care and Development Fund.     Standard Child Care and Development Fund Annual reporting requires the following information: number of families and children receiving services, age breakdown for children receiving services, reasons for needing childcare (e.g., working, in school), number of hours services provided, amount of CCDF subsidy, amount of parent co-payment, poverty status of families receiving services, and financial reporting (SF269).

Native Employment Works.     Current Federal reporting requirements include the SF269 and a Program Report that includes a narrative section that compares achievements for the year to their plan for the year.  It also summarizes significant barriers to implementation, provides explanations for variances with the plan, and describes actions taken.  Grantees must also summarize plans for unobligated funds.  The Program Report also includes a statistical report that provides the following information: number of clients served characteristics of clients served (e.g., age, sex, TANF recipients), number of clients participating in types of NEW activities and services (e.g., classroom training, on-the-job training, counseling), and number of clients with selected outcomes (e.g., GED, unsubsidized employment).

Head Start.     The standard Head Start reports includes information in the broad categories of children enrolled by demographics, staff information by demographics, information on classes/ groups/ centers, volunteer information, and services provided. Head Start also currently has a requirement for extensive outcome measurement. 

Child Welfare Services.     As reported to us, there are no specific reporting requirements.  Each grantee must report how they are progressing toward their 5-year plan.  Tribes are required to file the SF269 also.

Promoting Safe and Stable Families.     Like other Child Welfare programs, we are aware of no specific reporting requirements beyond reporting concerning progress toward planned activities and the SF269.

Family Violence Prevention: Grants for Battered Women’s Shelters.     Narrative or summary reports generally list the number of clients served and the services provided. Current Federal reporting requirements also include the SF269.

Administration on Aging Grants for Native Americans.    The standard report for AoA includes information on full-time/part-time staff; program resources and expenditures, including sources of income other than grants; unduplicated numbers of people that receive support services, congregate meals, home-delivered meals; total numbers of congregate and home-delivered meals; units of supportive services, legal services, at-home services, ombudsmen services, and others.  In addition, grantees must submit the SF269.

SAMHSA Targeted Capacity Expansion Grants.    Current Federal reporting requirements include the SF269 and a quarterly report and specified GPRA measures.  The quarterly and GPRA reports include the following information: grantee information; staffing information; data including number of new clients, services provided, and individual-level information on the clients as required by GPRA[6]; and narrative information about the project such as challenges and successes over the past quarter.

3.3 - Methods for Obtaining Input from Technical Working Group and Others

The Evaluation Feasibility Study is just one component of the DHHS efforts to support effective Tribal Self-Governance.  Since the inception of the Title VI Self-Governance demonstration feasibility study, mandated by Congress, to examine potential new DHHS programs for Tribal Self-Governance, DHHS has actively consulted with the Tribes.  A Title VI Advisory Group, comprised of Tribal Leaders with a commitment to self-governance and their designees, has worked closely with DHHS throughout that earlier process.  In addition, DHHS has made information regarding the Title VI Self-Governance demonstration feasibility study available to all interested persons through its website and through direct mailings to Tribes and other stakeholders.

The current Evaluation Feasibility Study has continued this practice of active consultation and information dissemination.  Specific communication activities that were undertaken included:

• A project description, explaining the project and its objectives, was sent to all Chairpersons or Presidents of Federally-recognized Tribes, with a background letter.  The packet sent included contact information for the DHHS Task Order Managers (TOMs) and for each of the Co-Principal Investigators, as well as the DHHS website address where ongoing information about the study and its progress is maintained.  Interested individuals were encouraged to provide comments and inquiries, through telephone or email to the OASPE Task Order Managers and/or the Co-Principal Investigators.
• The Task Order Managers and Co-Principal Investigators conducted presentations and question-and-answer sessions on the project at national or regional AI/AN meetings, including the Self-Governance meetings in San Diego (November 2002) and Palm Springs (October 2003) and the Alaska Native Health Board meeting in August 2003.
• The Technical Working Group was established and met in February 2003 to review and comment on Draft Reports, the Project Work Plan, and to provide guidance to the project.  Monthly conference calls were conducted with the TWG to discuss interim reports and progress on the project, from March through August 2003.  The Technical Working Group will meet with the project team on December 9-10, 2003 to review and discuss the Draft Final Report on the project.
• Discussion sessions were arranged and held at the National Indian Health Board, Self-Governance, and NCAI conferences to obtain broader Tribal comments and suggestions on the study objectives and preliminary findings.
• At the end of the project, the summary of the Final Report will be mailed to all Tribal Chairpersons with contact information for the OASPE Task Order Managers to encourage further comments and suggestions to DHHS.

The goal of all of these communication and information dissemination activities was to ensure that the project and its development was conducted as an open and ‘transparent’ process and to glean the maximum useful advice and input from the Tribes in the conduct of the study, while seeking to assure a balanced inquiry and lead to valid and objective advice to DHHS. 

3.4 - Methodology for Site Visits and Tribal Data Review

Site visits were made to six Tribes to assess the feasibility of conducting an evaluation of DHHS programs managed under Tribal self-governance.  The site visits focused on determining the extent to which there is historical documentation and knowledgeable individuals who are able to provide background and information on the development and goals of Tribal management of federal programs, the Tribes’ management information systems capabilities, and the availability, sources, and completeness of data on each program managed by Tribes.  The two-day site visits were conducted by a two or three-person team during late June through August 2003. 

While selection of site visit participants depended on availability of willing volunteers, the Technical Working Group and DHHS developed several criteria for selecting Tribes for the study.  These criteria were:

• All Tribes selected should have three or more years of experience with compacting or contracting IHS health programs.
• Preference should be given to Tribes that have experience in managing, under contracts or grants, one or more of the DHHS programs recommended for the Self-Governance Demonstration.
• In addition, the sites selected, to the extent possible, reflect the following:
• Geographic diversity, reflecting the distribution of the AI/AN population;
• Variation in the size of the Tribe’s population; and
• Variation in economic conditions (i.e., average income, employment levels) of the Tribe.

The process developed by OASPE, the Technical Working Group, and the project team for recruitment of Tribes to participate in site visits included:  1) a presentation on the project by the OASPE Project Officer at the Tribal Self-Governance meetings held in late April 2003 in Phoenix; 2) distribution of a letter of invitation to all Tribes to participate, and relevant background materials, at the Tribal Self-Governance meetings; and 3) mailing of the letter of invitation and background materials to all Tribal leaders during the last week in April.  Tribes interested in participating in the site visit were asked to contact the project contact by May 20, 2003 to indicate their interest and/or to obtain additional information.

The site visits were conducted in July and August 2003 to the six Tribes/Tribal organizations listed here:

• Bois Forte Band of Chippewa (Minnesota)
• The Choctaw Nation (Oklahoma)
• Port Gamble S’Klallam Tribe (Washington)
• Yukon-Kuskokwim Corporation (Alaska)
• Hopi Nation (Arizona)
• Little Traverse Bay Band of Odawa Indians (Michigan)

3.5 - Methodology for Small Group Discussions

The Small Group Discussions, conducted by Kauffman and Associates, Inc. (KAI), brought together tribal leaders, experienced tribal program managers and technical experts in self-governance program management to provide feedback and response to preliminary findings and conclusions related to this study.  These discussions provided another means for review and analysis of draft findings and conclusions.  A qualitative analysis of these discussions was conducted to identify “major themes and issues” that emerged across the board.  These major themes informed the study team and the Technical Work Group prior to finalizing reports.

People recruited to participate in the Small Group Discussions included tribal leaders, tribal management and technical staff with direct experience in the administration of Self Governance compacts, including financial managers, MIS directors, legal or regulatory analysts, program administrators and related positions. Sign-up sheets were provided prior to each group to make sure we had the appropriate mix of expertise in each session and adequate space to conduct each session.

The Small Group Discussions occurred between September and November 2003.  Locations for these discussion groups included the annual consumer conference of the National Indian Health Board, September 29-October 2, 2003, in St. Paul, MN; the DHHS and DOI Tribal Self Governance Conference, October 6-10, 2003 in Palm Springs, CA; and. the annual convention of the National Congress of American Indians, November 16-21, 200, in Albuquerque, NM;   At both the NIHB and the DHHS/DOI Self Governance meetings, a separate room or break-out session was provided to conduct these discussions.  Rich and substantial qualitative data was collected during these sessions.  Less effective was the session conducted at the NCAI gathering, where this topic was one of several on a busy agenda.  Comment sheets were distributed but few turned in from the NCAI event.  The majority of comments reflected in this report are from the NIHB and the Self Governance meetings.

Each discussion group involved 10 to 25 individuals representing a mix of interests and experiences from Tribes and Tribal organizations, including both Self-Governance and non-Self-Governance Tribes.  Individuals were recruited through fliers, inserts in conference packets or by appearing on the conference agenda as a workshop option. 

Topic Areas and Prompt Questions:

KAI staff facilitated these discussions.  An overview was provided and a written summary of the Draft Findings and Conclusions distributed for review. The following discussion guide was generally followed, however the flow of conversation generally centered upon three main topics:  (1) reaction to the draft findings and conclusions; (2) omissions in the draft; and (3) best outcomes for this feasibility study.  The following are the questions in the formal Discussion Guide:

1. Facilitator will describe what this feasibility study did and did not do:
   • Facilitator will explain why this study was done. (purpose)
   • Facilitator will describe the difference between an evaluation feasibility study and an actual program evaluation.
   • Q & A regarding this overall study
2. Identification of feasible alternatives for measuring success
   • Reaction to the options proposed for measuring success.
   • What other means exist to evaluate success?
   • How would this differ between IHS and other HHS programs?
3. Facilitator will review the preliminary findings and recommendations
• Do these preliminary findings and recommendations reflect your experiences with SG?  Give examples of why or why not.
• What’s missing?
4. What is the best outcome from your perspective for this feasibility study?
   • How do you see these preliminary findings or recommendations impacting future opportunities to expand SG compacting to other programs of HHS?
   • How can this study help local planning for SG?
   • Any other comments?