Needs Assessment Methodologies in Determining Treatment Capacity for Substance Use Disorders: Environmental Scan Final Report. RESULTS I: REVIEW OF NEEDS ASSESSMENT REPORTS

09/16/2019

Overview of Mandated Needs Assessments

As discussed in the Methods section, mandated needs assessments are those carried out in connection with requirements for various Federal and, in some cases, State programs, one of the two types of reports that we reviewed. They are conducted according to specific guidelines of the program and therefore tend to be more generic than the locally initiated reports, the other type reviewed. The mandated reports are also much more numerous and usually address SUD as one among other types of public health issues. Accordingly, we reviewed a representative selection of several dozen, with a less rigorous approach than for the locally initiated reports, primarily summarizing characteristics of methods and content. The major programs that require this type of needs assessment are identified in Figure 4, on the following page.

FIGURE 4. Federal Programs Requiring Needs Assessments
Program Title Program Administration Organization
Substance Abuse Prevention and Treatment Block Grant SAMHSA[1] States
Community Mental Health Services Block Grant (MHBG) SAMHSA States
Strategic Prevention Framework (SPF) State Incentive Grant (SIG) SAMHSA States
State Targeted Response to the Opioid Crisis Grant (Opioid STR) SAMHSA States
Community Health Needs Assessments (CHNAs) IRS[2] Non-profit hospitals
Community Needs Assessments (CNAs) NPHAB[3] Public health agencies
Delivery System Reform Incentive Payment (DSRIP) CMS[4] States
Community Services Block Grants ACF[5] States
Community Development Block Grants HUD[6] States
Comprehensive State-wide Needs Assessments RSA[7] States
Maternal and Child Health Services Title V Block Grants HRSA[8] States
Primary Care Services Grant Program HRSA States
Ryan White HIV/AIDS HRSA States
  1. Substance Abuse and Mental Health Services Administration
  2. Internal Revenue Service
  3. National Public Health Accreditation Board
  4. Centers for Medicare & Medicaid Services
  5. Administration for Children and Families
  6. Department of Housing and Urban Development
  7. Rehabilitation Services Administration
  8. Health Resources and Services Administration

With the exception of needs assessments conducted for the Substance Abuse Prevention and Treatment Block Grant (SAPTBG) program, those in the above list generally address a wide range of issues and service types, with SUD being discussed to a very limited extent. Here we provide an overview of those that are more likely to contain at least some discussion of SUD treatment needs.

Substance Abuse Prevention and Treatment Block Grant (SAPTBG) and Community Mental Health Services Block Grant (MHBG )

The FY 2018-2019 combined application for the SAPTBG and MHBG requires applicants to conduct a behavioral health needs assessment and plan that focuses on "the strengths, needs, and service gaps for specific populations."[3] The assessment step in this process requires the applicant to "Identify the unmet service needs and critical gaps within the current system" and recommends for that purpose the use of SAMHSA's various data sets (NSDUH, Treatment Episode Data Set [TEDS], etc.). The strengths of this approach are: (1) it requires a data-driven analysis of existing service system capacity; (2) it directs attention to the needs of various subpopulations who often experience disparities in care; and (3) it calls for a detailed plan for how funding will be used to enhance the existing system. However, it lacks some of the features that would result in a needs assessment of the highest quality as described in the Background section on needs assessment methodology.

The major limitation is that these requirements do not provide a definition of need as called for in the literature. Consequently, there is a tendency for applicants to rely on the simple and less useful definition of "unmet need" as the gap between untreated and treated prevalence. Descriptions of gaps in the service system are therefore often very general--for example, this statement from one state application: "There is a need for more services which provide various levels of treatment and which extend to women in underserved areas of the state." Another application from a large state has only a single reference to gaps, citing a task force report on Youth SUD treatment needs that identified three main gaps--standardized assessments, workforce development, and access to care--without further discussion.

A second limitation of the block grant approach is that it does not set out a standard for a description of an optimal service system against which applicants can measure gaps in the capacity of their service system. The application does, however, request--but not require--applicants to provide detailed information in the form of checklists about whether they provide a long list of services, described as "key focus areas that are critical to implementation of provisions related to improving the quality of life for individuals with behavioral health disorders."

Strategic Prevention Framework (SPF) State Incentive Grants (SIGs)

As of January 2017, all 50 states, eight jurisdictions, and 19 tribes have received SPF SIG/TIG funding. SPF SIG grant funding has been used to support a wide range of prevention initiatives such as prescriber education, coordination with law enforcement, production of educational videos, naloxone training, and many others.

Needs assessment for the SPF SIG is the first of the five steps of the SPF (SAMHSA, 2018):

  • Step 1: Assess Needs: What is the problem, and how can I learn more?

  • Step 2: Build Capacity: What do I have to work with?

  • Step 3: Plan: What should I do and how should I do it?

  • Step 4: Implement: How can I put my plan into action?

  • Step 5: Evaluate: Is my plan succeeding?

The website for SAMHSA's Center for the Application of Prevention Technologies (CAPT) provides some very general guidance on conducting needs assessments; this guidance focuses on collecting data to address four questions that are primarily epidemiological, with examples provided for each: What substance use problems and related behaviors are occurring in your community? How often are these problems and related behaviors occurring? Where are these substance use problems and related behaviors occurring? Who is experiencing more of these substance use issues and related behaviors?

As demonstrated by a sample of SPF SIG reports that we reviewed, the SPF approach for conducting needs assessments differs in certain respects from the model that we have utilized in this report. While the SPF emphasizes the collection and analysis of data, the type of data is primarily related to prevalence and patterns of substance use with little attention to analyzing existing service capacity and gaps. Instead, there is an emphasis on assessing a community's "readiness for change," which generally refers to attitudes about substance use and prevention, for which CAPT provides a variety of measurement tools such as stakeholder interview guides.

State Targeted Response to the Opioid Crisis Grants (Opioid STR)

According to the Funding Opportunity Announcement, the purpose of the Opioid STR program is "to address the opioid crisis by increasing access to treatment, reducing unmet treatment need, and reducing opioid overdose related deaths through the provision of prevention, treatment and recovery activities for opioid use disorder." Award amount is determined by a formula based on unmet need for opioid use disorder treatment and drug poisoning deaths. Funding is intended, in part, to "support a comprehensive response to the opioid epidemic using a strategic planning process to conduct needs and capacity assessments." The requirements for the needs assessment are to identify the following:

  • Areas where opioid misuse and related harms are most prevalent.

  • The number and location of opioid treatment providers in the state, including providers that offer opioid use disorder services.

  • All existing activities and their funding sources in the state that address opioid use prevention, treatment, and recovery activities and remaining gaps in these activities.

The state needs assessments follow a prescribed format that calls for a variety of epidemiological and treatment system information, but a limited requirement for specifying gaps and need. With regard to prevention services, for example, grantees are asked simply to "identify any strengths or gaps in services." To estimate treatment needs, grantees are suggested to use the SAMHSA data sources (NSDUH, etc.) without further refinement such as anticipated demand, etc.

Community Health Needs Assessments (CHNAs)

CHNAs originated as a requirement by the Internal Revenue Service (IRS) that non-profit hospitals be able to demonstrate a "community benefit" beyond simply delivering standard medical care. Subsequently, a number of states implemented their own community benefit statutes. This requirement was greatly expanded by the Affordable Care Act (ACA), which added the requirement that non-profit hospitals conduct a needs assessment every three years. With nearly 3,000 non-profit hospitals in the nation,[4] the number of CHNAs that have been produced far exceeds the resources for any practical systematic review. However, we have reviewed a sample of these reports to identify the manner and extent to which they address SUD capacity and gaps.

The basic principles and requirements for a CHNA are described by Rosenbaum (2013). Key features of the requirements are:

  • Involvement by people who represent the "broad interests" of the communities served by non-profit hospitals, including people with "special knowledge of or expertise in public health."

  • A definition of community that encompasses both a significant enough area to allow for population-wide interventions and measurable results and includes a targeted focus to address disparities among subpopulations.

  • Development of an accompanying Community Health Improvement (CHI) Plan detailing activities to be conducted by the hospital to promote public health and address issues identified in the CHNA.

  • An emphasis on transparency, for example a requirement that the report be made widely available to the public, by posting on a hospital organization website or the facility's website.

Community Health Assessments (CHAs)

CHAs are similar to CHNAs but are conducted by different types of organizations for different purposes and with different requirements; whereas CHNAs are conducted by non-profit hospitals to satisfy requirements of the IRS and the ACA, under direction of the Centers for Medicare & Medicaid Services (CMS), CHAs are conducted by county public health boards for purposes of accreditation by the National Association of Public Health Directors (NAPHD), established in 2007 with support from the Robert Wood Johnson Foundation. The IRS and NAPHD have coordinated to encourage hospitals and public health departments to collaborate in producing a single combined CHNA-CHA that satisfies both sets of requirements, and a number of them have done so. Several of these have been conducted by collaborations between hospitals and public health agencies.

NAPHD has very detailed requirements for CHAs, which address the following components:

  • Various sources of data.

  • Population demographics.

  • Health issues identified.

  • Special populations with health issues.

  • Contributing causes of health issues.

  • Description of assets to address health issues.

  • Documented input from stakeholders.

Many of these reports utilize a framework known as Mobilizing for Action through Planning and Partnerships, which was developed jointly by the National Association of County and City Health Officials (NACCHO) and the CDC. The needs assessment component is embedded in the third of six phases, entitled Collecting and Analyzing Data, which involves four types of assessments: Community Themes and Strengths, Local Public Health System, Community Health Status, and Forces of Change. The NACCHO website provides an array of tools and training materials for conducting each of these assessments.

A review of a sample of 58 CHAs that extracted health status indicators reported in the plans identified a total of 1,524 indicators with an average of 29 indicators per health department (Bender, 2017). Categories of indicators included nutrition, physical activity, and obesity; access to health services; maternal, infant, and child health; tobacco prevention and control; social determinants/health equity; injury and violence; substance abuse; mental health; clinical preventive services; chronic disease; environmental quality; reproductive and sexual health; oral health; data; overall health and well-being; organizational capacity; and emergency preparedness.

RELEVANCE OF CHNAS AND CHAS FOR SUD TREATMENT CAPACITY NEEDS ASSESSMENT

CHNAs and CHAs provide a wealth of information that has the potential for uses beyond the primary function of meeting organizational requirements, a potential that is greatly enhanced by their accessibility (particularly in contrast to locally produced needs assessments, which are often not disseminated beyond the agency that produced them). With respect to the issue of SUD needs assessments specifically, a striking finding of our review of a sample of CHNAs and Community Needs Assessments (CNAs) is that substance abuse is almost invariably at or near the top of the list of community concerns. As a basis for the utility aspect of needs assessment--providing guidance for policy making and priority-setting--these reports have both limitations and benefits. The limitation is that with so many areas of public health being addressed, the degree of detail is inevitably constrained. The main benefit is that these reports provide an understanding of where substance use fits in with the broader range of public health problems and resources. While they are not a substitute for a needs assessment that focuses exclusively on substance use treatment capacity, they offer a useful starting point for collecting more granular information.

Review of Locally Initiated Needs Assessments

In this section we provide a narrative and tabular summary of needs assessments that were initiated locally (e.g., at the city, county, or state-level) for specific policy initiatives rather than those that are federally mandated for ongoing policy purposes or for public health accreditation, a total of 40 reports (see Appendix B for a list of the reports). The summary is based on, and organized according to, the research questions that guide this project.

What Questions are Addressed?

TABLE 1. Topic Areas: Supply and Demand
  Number of Reports
Prevalence/frequency related to:
   General substance use 22
   Alcohol 15
   Opioids 9
   Heroin 8
   Methamphetamine 2
   Marijuana 3
   Cocaine 5
   Depression 2
   Suicide 2
   Overdose 5
   Co-Occurring Disorder (COD) 6
   Medical Co-morbidity 4
   Other prevalence/frequency described 23
Unmet need (other than prevalence) 9
Provider organization supply/capacity 6
Utilization/penetration 17
High utilizers 5
Demand (services likely to be requested) 17
Referrals 7
Means of obtainment (how drugs are obtained) 5
Risk factors 4
Expenditures/costs/cost benefit 9
Benefit design/coverage 4
Funding sources 8
Policy changes affecting supply or demand 4
Law enforcement 1

We operationalized this research broadly, as "What topics were addressed by the locally initiated needs assessments?" Tables 1-5 report the results, organized into categories of topics: supply and demand, populations, quality issues, use of technology, and barriers and limitations. We identified approximately 40 different issues and topics addressed by the reports we reviewed, with most addressing multiple topics--that is, individual reports may have addressed more than one of the items in each category. Notably, there were some topics that received relatively limited attention. Whereas recent literature on needs assessment methodology emphasizes the importance of collecting information about network assets and strengths, we identified only a handful that addressed this topic. Only one of the reports addressed law enforcement as a factor affecting service utilization. Other topics that are of current interest in the field but receive less attention in the needs assessments that we reviewed are benefit design, peer involvement, technology, and EBPs. Table 1 presents the number of reports that address topics related to supply and demand including prevalence, utilization, risk factors and issues related to costs.

Table 2 represents questions that were addressed related to issues of diversity and disparities as an aspect of system capacity, indicating that this has not been considered a central issue in the field despite the widespread attention in health care policy.

TABLE 2. Populations
  Number of Reports
Special populations 3
Disparities 7
Diagnostic group differences 3
Regional variation 5
Demographic trends 8
Cultural competency 6

Table 3 identifies questions related to quality that were addressed. It is noteworthy than only a few included measures of clinical quality (outcomes, adherence, and perception of care), as an element in the broader conception of system capacity; however, a number did include qualitative information from service users obtained through focus groups and interviews.

TABLE 3. Quality Issues
  Number of Reports
Access (wait-times, available slots, etc.) 19
Availability of services/network adequacy 16
Outpatient follow-up 7
Outcomes 5
Treatment completion/adherence 1
Satisfaction/perception of care 4
Quality/performance measurement systems 2
Collaboration 10
Leadership 3
Evidence-Based Practices (EBPs) 4
Benchmarks for future progress 3
Transitions by functional level 8
Behavioral health integration 4
System assets/strengths 7
Other quality 3

Table 4 indicates the number of needs assessments that addressed questions related to the use of technology. This category also provides background for the research question about how telehealth policies can be incorporated into needs assessments, which will be addressed in the final report recommendations, for which there is a clear need as only a very few reports address this increasingly important aspect of system capacity.

TABLE 4. Use of Technology
  Number of Reports
Data systems 4
Use of Health Information Technology 3
Use of Electronic Medical Records 2
Use of Telehealth 4
Use of e-prescribing 3
Interoperability among provider organizations 1
Interoperability among public agencies 2
Other technology issues 2

The reports address a wide variety of questions related to barriers and limitations, including many that that are not included in the already extensive list in the protocol, with nearly all the reports (n=39) identifying at least one other barrier or limitation that was not on the list (Table 5). Notably, many more reports identified policies--an area where change may be achieved relatively easily in that additional resources are not required--compared to other barriers such as provide reimbursement that would require additional resources to address.

TABLE 5. Barriers/Limitations
  Number of Reports
Policies 19
Mental health (MH)/SUD bifurcation 4
Agency silos 3
Decentralized (e.g., county) systems 4
Service locations 3
Lack of services 9
Service costs 2
Lack of access (not specified) 5
Lack of qualified workforce 9
Provider reimbursement limitations 9
Stigma 7
Privacy/confidentiality issues 2
Other barriers/limitations 39

What Populations are Addressed?

This section addresses the research question regarding the populations addressed in needs assessment. Whereas earlier SUD treatment needs assessments tended to address the general population, one of the hallmarks of more recent needs assessments, which are largely represented in the reports obtained through Internet searches, is that they drill down to examine availability and accessibility of services for a wide range of specific subgroups, such as demographic subpopulations including age, gender, racial/ethnic and sexual orientation, and persons in a particular status (such as homelessness and incarceration) as shown in Table 6. More recent needs assessments also tend more to focus on users and service capacity of specific substances--particularly opioids. These results are in contrast to the relatively little attention given to issues of disparities indicated in Table 2. As a general explanation for this discrepancy, needs assessments focusing on specific subpopulations generally do not draw comparisons among subgroups but focus only on the capacity issues related to the group of interest.

TABLE 6. Populations/Groups of Focus
  Number of Reports
Substance use disorder (SUD)
   Substance unspecified 8
   Alcohol 7
   Poly drug use 2
   Needle use 1
   Opioids 10
   Prescription drugs 7
   Heroin 5
   Methamphetamines 2
Mental health 11
Co-occurring disorders (CODs) 10
HIV/AIDS 0
Children/adolescents 21
Transition-age youth 6
Adults 19
Older adults 2
Racial/ethnic groups 1
LGBT/LGBTQA 4
Gender 5
Income 5
Marital status 0
Homeless 7
Veterans 1
Adults criminal justice system 10
Youth juvenile justice system 1
Child/youth child welfare 0
Other population or group of focus specified 14
Not described/specified 0

What Workers are Incorporated?

As shown in Table 7, needs assessment reports incorporated a wide variety of workforce categories, although any single category was addressed by only a few reports. Consistent with the findings in relation to other research questions, only a few reports (n=4) address peers as an aspect of workforce capacity. This requires further investigation, however, as a large proportion of the SUD treatment workforce consists of people in recovery; it may be a matter of nomenclature where the term peer is applied differently in SUD than in mental health. This is a question that will be explored with the TAG and addressed in the final report.

TABLE 7. Workforce Categories
  Number of Reports
Psychiatrists 11
   Addiction 2
   Other psychiatrists 2
Physicians (non-psychiatrist) 8
   Licensed buprenorphine physicians 6
   Other physicians 3
Nurses 7
   Licensed Practical Nurses/Licensed Vocational Nurses 1
   Registered Nurse (Diploma, Associate's, and Baccalaureate) 3
   Psychiatric and Mental Health Nurses 1
   Advanced Practice Registered Nurses (Master's and Doctoral) 2
   Nurse Practitioner 2
Physician Assistant 2
Psychologists 9
Social Workers 10
   Associate's and Baccalaureate 1
   Master's 2
   Clinical 3
Counselors 12
   Rehabilitation Counseling 3
   Addiction Counseling 4
   Clinical Mental Health Counseling 1
   Other counselors 4
Peers 4
Other workforce category 1

Needs assessments most frequently examined the workforce in a specific geographic boundary or characteristics related to training, certification, and licensing. Although SUD treatment workforce salaries are frequently cited as a contributing factor to workforce shortages, this was addressed by a negligible number of reports.

TABLE 8. Workforce Distribution, Characteristics and Funding of Salaries
  Number of Reports
Workforce geographic distribution
   By state 3
   By county across the state 3
   By urban/rural 3
   Other geographical classification 9
Workforce characteristics
   Racial/ethnic composition 2
   Competencies/training 7
   Certification 5
   Licensing 5
   Other workforce characteristics 5
Sources of funding for workforce salaries
   Medicaid 4
   Medicare 1
   Private insurance 1
   State funds 1
   Charity/free care 1
   Self-pay 1

What Settings (geographic areas/regions) are Addressed?

In the locally initiated needs assessments reports we have reviewed to date, most have focused on specific states or specific counties and cities. Most have focused on the entire state or county, while only a few have focused on urban, rural, or frontier regions/boundaries of the state or county. One focused on a tribal community within the northern portion of a state.

What Levels of Care are Addressed?

We reviewed how the needs assessments addressed levels of care in connection with service capacity--that is, what kind of services are required to meet the individual-level needs of a population. This is another issue that has been discussed since the 1980s (Ford & Luckey, 1983; Rush, 1990). To assess how it was addressed, we looked for evidence of some standard for a comprehensive mental health system--such as the ASAM levels of care or the SAMHSA Good and Modern Behavioral Health System standards--and included a question about how need was defined in the quality rating section. As with the reporting on treatment needs, this important feature of needs assessment was relatively neglected. Instead, many reports simply provided utilization data of existing services. Especially in the mandated needs assessment, often there was also a list of service gaps but without any discussion as to how these were identified.

What Data Sources and Methods are Used?

Methods in needs assessments consist of three types of activity: (1) methods of data collection; (2) methods of data analysis; and (3) methods for formulating recommendations. The following sections discuss how these activities were conducted in the needs assessments we reviewed. This section also provides an introduction to the question, "What are key pieces of missing data?"--which will be addressed in more detail with recommendations from the TAG in the final report.

METHODS OF DATA COLLECTION

Data collection consists of obtaining information about the "what is" and "what should be" conditions--characteristics of the population, prevalence of the disorder, and characteristics of the service system (number and types of services and workforce, etc.). The most common means of data collection are: obtaining stakeholder perspectives by means of surveys, conducting focus groups and interviews, and holding public meetings; utilizing secondary data such as national data sets (most NSDUH) or administrative data such as claims, and reviewing documents (previous reports, policies, program descriptions, etc.). Most assessments combined stakeholder perspectives with national data sets to estimate prevalence and combined SAMHSA data sets with local health information to estimate capacity/utilization. Sources of data used in the reports are enumerated in Table 9 and Table 10.

Table 9 enumerates the various federal data sources used in the needs assessments. Notably, none used data from Healthy People 2020 and the National Directory of Drug and Alcohol Treatment Facilities. Reasons for this and recommendations for how these sources might be utilized more frequently will be provided in the final report.

TABLE 9. Federal Data Sources
  Number of Reports
SAMHSA National Survey on Drug Use and Health (NSDUH) 12
Behavioral Risk Factor Surveillance System (BRFSS) 4
Youth Risk Behavior Surveillance System (YRBSS) 5
SAMHSA Treatment Episode Data Set (TEDS) 5
SAMHSA Behavioral Health Barometer 1
SAMHSA Buprenorphine Treatment Practitioner Locator 2
SAMHSA National Survey of Substance Abuse Treatment Services (N-SSATS)[a] 1
U.S. Census Bureau Data 13
Bureau of Labor Statistics (BLS) 3
Centers for Disease Control and Prevention (CDC) Data 2
Healthy People 2020 0
National Directory of Drug and Alcohol Treatment Facilities 0
SAMHSA Emergency Department Data 0
SAMHSA National Mental Health Services Survey (N-MHSS) 0
  1. See https://wwwdasis.samhsa.gov/dasis2/nssats.htm.

Table 10 presents the number using various types of secondary state-level data. Notably, although Medicaid and to a lesser extent Medicare claims are widely used in services research, they are not widely utilized for needs assessment despite their potential for assessing service utilization. We will discuss this issue and provide recommendations for how these sources may be incorporated into SUD needs assessment.

TABLE 10. State and Local Secondary Data
  Number of Reports
State Management Information System (MIS) 7
State health database (including Medicaid enrollment) 12
Licensing boards 4
County/City Management Information System (MIS) 1
County/City health database 1
Provider Management Information System (MIS) 1
Medical record/Electronic health record 2
Medicaid claims 2
Existing reports/studies 13
Document review 2
Other data source or method 14
Medicare claims 0
Healthy People 2020 0

Tables 11-15 present sources of information obtained from various groups by means of surveys, focus groups, key informant interviews, meetings and steering committees. It is notable that peer organizations are not included in any form, in contrast to mental health needs assessments, in which this very common. A possible explanation is that peer organizations have not developed in the SUD treatment system to the extent that they have in mental health. We will discuss this issue with the TAG. Also notable is that very few (n=4) needs assessments utilized steering committees, though it is likely that many were guided by collaborations of various kinds that were not identified as steering committees.

TABLE 11. Surveys
  Number of Reports
Any surveys 19
State/local agency leadership/staff 3
Legislators 0
Provider organization leadership 13
Practitioner/workers 6
Peer organizations 0
Advocacy organizations 0
Consumers/patients 6
Family members 2
General public 4
Other groups 7

 

TABLE 12. Focus Groups
  Number of Reports
Any focus groups 8
State/local agency leadership/staff 1
Legislators 0
Provider organization leadership 4
Practitioner/workers 0
Peer organizations 0
Advocacy organizations 2
Consumers/patients 5
Family members 3
Other groups 4

 

TABLE 13. Key Informant Interviews
  Number of Reports
Any key informant interviews 19
State/local agency leadership/staff 11
Legislators 0
Provider organization leadership 12
Practitioner/workers 5
Peer organizations 0
Advocacy organizations 3
Consumers/patients 4
Family members 3
Other groups 15

 

TABLE 14. Meetings
  Number of Reports
Any meetings 4
Community/public 2
State/local agency leadership/staff 3
Legislators 0
Provider organization leadership 3
Practitioner/workers 1
Peer organizations 0
Advocacy organizations 2
Consumers/patients 2
Family members 2

 

TABLE 15. Steering Committees
  Number of Reports
Any steering committee 4
State/local agency leadership/staff 1
Legislators 0
Provider organization leadership 1
Practitioner/workers 1
Peer organizations 0
Advocacy organizations 0
Consumers/patients 1
Family members 1

METHODS OF DATA ANALYSIS

After data collection, the second methodological activity involves analysis of the data. As discussed in the Background section, data analysis in the context of SUD treatment needs assessment primarily involves measuring the gap between what is and what should be--the need for additional service capacity, additional types of services, elimination of redundant or ineffective services, and additional population subgroups served. The simplest approach for measuring the population need for services is prevalence. Similarly, the simplest approach for measuring the supply of services is to conduct an inventory of the existing service system, measuring how many people are served. Though some needs assessments we reviewed go no further than this, these are not very useful for the function of needs assessments to serve as guides for decision making, for several reasons. First, as noted above, there is not a one-to-one correspondence between prevalence of a disorder and need for services. Second, this method does not take into account different treatment needs and types of treatment (for example, residential treatment versus outpatient treatment). Third, it provides little information about the inputs to treatment capacity--for example, specific types of staff or programs that would be necessary to provide adequate treatment.

Most of the reports we reviewed used some combination of quantitative and qualitative data in their analysis of need, going beyond the simplest approach but falling short of a more robust approach. An example is New Jersey's needs assessment for child behavioral health (New Jersey Department of Children and Families, 2016), which draws upon several management information systems (MIS) used for tracking and identifying patterns of service utilization combined with county-based planning and advisory groups established by statute to "identify service and resource gaps and priorities for resource development" and to advise the Department of Children and Families on "the development and maintenance of a responsive, accessible, and integrated system of care." The limitation of utilization data for measuring need is that the population of individuals with unmet need are likely to differ from those who are receiving services--and therefore to require different types of services. Additionally, utilization data do not identify inappropriate use (services received by individuals who do not have a need) unless covariates such as clinical status are examined. In short, this approach does not serve the third form of data analysis: to identify solutions.

METHODS FOR FORMULATING RECOMMENDATIONS AND PRIORITIES

Once the needs assessment has collected data and analyzed it to define the gap between what is and what should be, the final step is to specify and prioritize what is usually a considerable number of possible solutions. Few of the unpublished reports that we reviewed used techniques of planning models, such as those proposed by Green at al. (2016) that use mathematical formulas to estimate a population's service need based on social indicators correlated with substance abuse.

As noted in the background discussion of methodology, planning models of this type that use formulas to estimate capacity needs have received limited attention in recent years, with the exception of some use of simulation models and the Calculating for an Adequate System Tool (CAST) model developed by Green et al. (2016). Where this does take place, however, is with studies that focus on contingency planning related to anticipated policy changes, for example a needs assessment conducted in anticipation of California Medicaid expansion (California DHCS, 2012). For this assessment, researchers developed a complex algorithm that estimated the size and characteristics of the potential expansion population, with upper and lower bounds based on various assumptions, then estimated service utilization for various subgroups of this population, based on comparison with the current Medicaid population adjusted for differences between the two groups. Although this analysis did not attempt to estimate the need for particular categories of services, it did offer recommendations for addressing various gaps and shortcomings in the current system to accommodate the expansion population, as well as to improve the service system for all.

Optimally, needs assessments not only offer recommendations but also prioritization of recommendations to guide decision making. Only a fraction of those we reviewed did so, but one of the most effective means employed by those that did, primarily academic centers, was to provide a cost analysis, as in the report by Shepard et al. (2005). After estimating the number of persons with a clinically significant SUD, they calculated that providing substance abuse treatment and outreach services to them would require an additional cost of approximately $109 million ($17 per capita), of which the state Bureau of Substance Abuse Services would need to fund $31 million ($5 per capita), representing an increase of 42% over its current spending. While the authors acknowledge that this increase is formidable, they suggest that it could be manageable if spread over a period of five years, and that the increase for other payer sources would be proportionately smaller.

EXAMPLE OF DATA SOURCES AND METHODS

The Wisconsin Community Mental Health Needs Assessment stands as an example of an assessment that collected data from a wide range of sources. These included secondary data (LIFE Study reports, Behavioral Risk Factor Surveys, Youth Risk Behavior Surveys, data from the Centers for Disease Control and Prevention (CDC), hospital health assessments, and administrative data from organizations that serve mental health consumers); focus groups with diverse stakeholders including the general public, providers, and referral agents; surveys designed based on focus group feedback including a telephone survey with a sample of the general public, online surveys of providers and referral sources, and in-person surveys with a convenience sample of consumers recruited through provider organizations. While this represents a broader range of sources and a more rigorous methodology than most of the reports we reviewed, it demonstrates the possibilities for SUD treatment data collection. Several reports refer to "data triangulation," but there is little indication that this is done in any systematic way.

Data collected by a variety of federal agencies and made readily available electronically have become a primary source of information for both mandated and locally initiated needs assessment. This includes data related to prevalence, utilization, provider supply, and population characteristics. Given the central role these sources play in current needs assessment methodology, it is important to understand their strengths and limitations. As background for recommendations to address data gaps that will be provided in the final report, the following reviews the strengths and limitations of these sources for purposes of SUD treatment needs assessment.

Are there Best Practices Identified?

This section addresses the research question, "Are there best practices with respect to these needs assessments?" While it may be premature to identify best practices in SUD treatment needs assessment as there are not yet formal criteria to evaluate what constitutes best practice, in the review of needs assessment reports we identified a number of innovative and exemplary practices that might be candidates for these criteria.

Over the years, there have been articles recommending refinements in SUD treatment needs assessment. The aforementioned articles by McAuliffe et al. (2004) propose an improved method of measuring need; Rowan-Szal et al. (2007) discuss an instrument for assessing treatment staff training needs known at the Texas Christian University Program Training Needs; and most recently Green at al. (2016) present a sophisticated research-based methodology for assessing treatment needs and system capacity on a broad scale.

In our review of a sample of needs assessment, we identified practices, methods and approaches (typically more focused than those cited above) that we considered would contribute to the field if adopted more widely. These are discussed briefly below. We will discuss these practices in more detail in the Final Report.[5]

INNOVATIVE AND EXEMPLARY PRACTICES

As part of our quality review, we flagged practices that were innovative or exemplary in some respect and worthy of adoption in the field. In general, these are practices that add to the precision or the utility of a needs assessment. Among the exemplary practices were reports that went beyond offering recommendations to explain the rationale, feasibility, cost or barriers for acting on the recommendations. For example, one report identified various political, social, and demographic factors likely to affect capacity in the future, thus adding to the utility for planning purposes. We also flagged studies that presented more detail about the methods they employed as this helps users of the report to assess the reliability and applicability to a particular situation.

Other practices serve to improve the precision or level of detail in the measurement of need or availability of services, for example by adding a measure of clinical significance, which has been a major challenge in SUD needs assessment. Likewise, several studies adopted strategies for drilling down into data to obtain a fuller understanding, for example by conducting follow-up interviews with survey respondents or obtaining workforce capacity information that included not only numbers but caseload size. We flagged one study that used a simulated patient approach, calling agencies as though a potential client to inquire about availability of special services for special populations (LBGT). This "secret shopper" method has been used by CMS to test network adequacy, and in this case, the method demonstrated inaccuracies in the self-report information represented in N-SSATS.

Practices that enhance utility:

  • Identified forces that affect capacity at multiple (federal, state, county, systems, and organization) levels.

  • Along with strategies for recommendations, also included "rationale" for each strategy with examples from other states and "implementation steps" serving the purpose of the needs assessments as a tool for decision making.

  • Added clinical significance to prevalence estimates in measuring need.

  • Revised service definitions to clarify gaps identified in a previous-year needs assessment.

  • Estimated how new treatment slots should be distributed geographically.

  • Developed a code of ethics to guide the project and explain why IRB approval was not required.

  • Used the Robert Wood Johnson Social Determinants of Health (a set of social and behavioral factors that affect health and health disparities) as an organizing framework.

  • Utilized data visualization software to map clinics and organizations to reveal possible gaps in treatment provision across different regions.

  • Compared the state's needs to those of other comparable states.

  • Provided estimates of additional spending required for a "good" and an "ideal" system of care.

  • Reviewed policy changes that impact behavioral health services with graphic representation of impacts.

Practices that enhance precision or level of detail:

  • Identified comparable states for comparing rates (vs. simply using national averages).

  • Population density and number of overdoses by zip code were used to highlight underserved areas.

  • Created online inventory of inpatient, outpatient, and in-state residential behavioral health services for children and electronic bed tracking system for quantifying usage and determining service needs.

  • Survey asked workers for number of clients per week.

  • Followed up on publicly-available information about providers by contacting them to confirm and elaborate.

  • Described interviewer training and quality control.

  • Described the selection and training of bilingual surveyors (bilingual Navajo speakers) including project goals, concepts, interviewing skills and confidentiality, and how concepts and items could be interpreted into Navajo language.

  • Conducted follow-up interviews with a sub-sample of survey respondents to explore discrepancies between positive clinical screening and perceived need.

Have Any Methods been Validated?

The only needs assessments that provided any information about validation methods were the several academic research reports, which addressed, for example, validity of social indicators in estimating prevalence.

Quality of Substance Use Disorder Needs Assessment Reports

As discussed in the Methods section, we rated the quality of reports based on the clarity and transparency of the information presented about the methods utilized in the needs assessment. Here we summarize the results of the quality ratings. A table showing the ratings by number of reports in each category of quality is provided in Appendix D. The number of reports varies for each quality criteria. This is because we added items as we identified issues in the review process. For the Final Report we will include the entire sample of reports.[6]

Are the Purpose or Questions Clearly Defined and Addressed?

We assessed whether the assessment report clearly specified the research questions the assessment intended to address. As shown in Appendix D, most of the reports (24 out of 29) were judged to have clearly defined and addressed the purpose.

Are Populations Defined Clearly?

Of 33 reports, 26 provided clear explanations for how the population was defined for the purposes of needs assessment, and five provided partial or imprecise explanations.

Is Treatment Need Fully Defined?

As discussed in the section on methodology, defining treatment need is one of the most critical components of a needs assessment; it was also the most problematic aspect of the reports we reviewed. Given the importance, we approached this issue from several perspectives, examining how it was addressed in items on questions and populations and, for a sub-sample of reports, a question in the quality section about how need was defined.

Despite the attention to this complex issue in the literature on SUD needs assessment methodology as far back as the early 1980s (Maddock, et al., 1988; Rush, 1990), very few of the reports we reviewed offer more than a brief explanation of need. In many cases, the extent was limited to survey questions about unmet needs or an estimate of unmet needs defined as untreated prevalence, drawn from data sources such as NSDUH. In the quality ratings, eight of the 16 reports rated on this criterion provided what we judged to be an adequate definition of need, three defined (unmet) need simply as the gap between prevalence and service utilization, and the remaining reports provided no definition. As discussed above, this is inadequate for the purpose of needs assessments to serve as a guide for planning and decision making. We will discuss this issue in more detail in the final report, with recommendations and examples of best practices for how it may be addressed.

Is the Setting Clearly Specified?

Nearly all (n=28) of the reports rated on this criterion (n=32) provided clear definitions of the setting, while four provided partial or less precise explanations.

Are Levels of Care Defined?

Rating on this criterion was mixed: 16 rated as providing clear definitions of levels of care, whereas six were rated as providing less complete definitions and nine as not defining levels of care. Consistent with other findings related to levels of care, this is clearly an area for improvement and will be addressed with recommendations in the final report.

Are Data Methods Described?

This category includes criteria related to survey methodology (when applicable, whether random sampling was employed, whether testing of the instrument was reported), whether statistical methods were described, whether stratification was used in the analysis if applicable, whether data limitations were reported and whether validation of the methodology was reported. Of 24 reports using surveys, six reported random sampling, while it was unclear for seven. Only three reported on whether the instrument had been tested. Statistical methods were described by 15 of 27 reports. Stratification was reported in three of the 15 reports reviewed on this criterion. Limitations related to data were reported by 22 of 31 reports. Related to the research question about whether any needs assessment methods have been validated, only three of 30 reports described validation of any aspects of methodology. Recommendations for improvement in this area will be addressed in the final report, with input from the TAG.

Was a Model System for Gap Comparison Presented?

This criterion is related to the "what should be" condition of a needs assessment. Model system refers to some template such as the ASAM Levels of Care or the SAMHSA Good and Modern Behavioral Health System. Exactly half of 30 reports did reference such a model.

Is there Guidance for Policy Decisions?

This important criterion is related to the fourth essential component of a needs assessment, that it serves the purpose of supporting decision making. Of 34 reports, 28 offered recommendations, and 20 of those addressed priorities, feasibility, and/or strategies for acting on the recommendations.

How is Capacity Measured?

How capacity is measured is a fundamental aspect of needs assessment. We rated methods of measuring capacity based on the reliability of the data source, with the most reliable being an inventory such as number of providers, beds, slots, etc. (provided in eight reports), followed in order of decreasing reliability (due to the relative potential for missing data, selection bias, subjectivity, or imperfect knowledge) by utilization data such as administrative data, claims, etc. (four reports), consumer self-report regarding access, availability, etc. (three reports), key informants (six reports), surveys (four reports).

A number of reports identified the challenge of measuring capacity. While there are various sources of information about the number of different types of facilities and provider organizations, such as SAMHSA's N-SSATS database, and sources of utilization data, such as TEDS, they do not provide readily available information about the actual capacity of a system--how many people it is able to serve. In addition to a lack of data, capacity is challenging to measure because it involves multiple factors--workforce characteristics, available resources, location of services, etc.

Assessing the workforce component of capacity is especially challenging for a variety of reasons: the nomenclature and scope of practice for different occupational categories in the substance use treatment field vary widely from one location to another, and it is difficult to determine the job functions in a particular organization--for example, is a nurse an administrator or a direct care provider? This issue arises especially frequently in connection with Medication-Assisted Treatment (MAT) capacity as there are numerous reports that authorized prescribers and treatment facilities do not operate at full capacity, but the extent of the gap and the reasons for it are unclear. A recent U.S. Government Accountability Office (GAO) report has emphasized the importance of measuring progress and understanding barriers in increasing MAT capacity, which the U.S. Department of Health and Human Services is now actively evaluating (GAO, 2017). The results of these evaluation activities will be informative in understanding issues related to SUD treatment capacity generally.

With respect to workforce capacity it should be noted that a number of reports focus on this topic exclusively, whereas others took a more global approach to measuring capacity and need and did not address it all. We allowed for both when determining which assessments to include in our review--on the grounds that this difference depended on the intended scope of the needs assessment. In the quality ratings we assess whether this scope was adequately defined.

Are Ethical Issues/IRB Approval Addressed?

This quality question was prompted by a needs assessment involving substance use by a vulnerable population, which referenced an earlier study in the 1970s that is regarded as having violated human subjects protection. Five of 32 reports indicated having received IRB approval. We flag this as a topic for discussion: whether locally initiated needs assessments should be subject to the same requirements as research studies and, if so, what the implication of this burden would be and whether there might be less burdensome ways of addressing ethical issues.

Our quality assessment was designed to provide an overview of how effectively the sample of needs assessment reports fulfilled the primary functions of a needs assessment, with an emphasis on the utility. Thus, we did not attempt to judge the appropriateness of the methodology as much as the level of detail that was provided, such that a user of the report would be able to judge the extent to which they might rely on it for a particular purpose. In addition, we included several criteria, such as the question of ethical issues and IRB approval, that are not widely met, but which are present in a few exemplary cases.

Population, Workforce, Setting, and Levels of Care

Clear and appropriate definitions of these components of a treatment system are critically important for the reliability and utility of a needs assessment. A majority, but not all, reports were adequate in these areas. Few needs assessments utilized the ASAM Criteria for defining levels of care; this is a particularly challenging aspect of SUD needs assessment because definitions of levels of care vary so extensively from one locale to another and are often not clearly defined for the service system, making it difficult for a needs assessment to do the same.

Data Collection and Analysis

We examined reports for several aspects of methodology, including the use of surveys and analytic methods. Our focus was less on whether the methodology was appropriate but rather whether it was reported. Needs assessments vary widely in methodological sophistication, which has important implications for how they are used.

Validation

In our quality ratings, we looked specifically for any mention of validation and found only three of the unpublished reports that referenced any type of validation.

Need and Capacity

We examined how need and capacity are defined and measured. In particular, we were interested in how many reports go beyond an estimate of the difference between treated and untreated prevalence as a measure of need, which has limited utility for planning purposes. We extracted this information from a subset of 16 reports; of these five provided no operational definition of need, three defined need simply as estimated untreated prevalence, and eight provided definition based various other sources such as published literature and standard clinical assessments. Only one study estimated the number with a diagnosis that would not need treatment. One study, which focused on accessibility of services for people with disabilities, used the requirements of the Americans for Disabilities Act as a definition of the "what should be" condition.