Needs Assessment Methodologies in Determining Treatment Capacity for Substance Use Disorders: Environmental Scan Final Report. BACKGROUND


Review of Selected Needs Assessment Methodology Literature

The following section provides a brief review of needs assessments methodology literature, focusing on aspects most relevant to SUD treatment capacity, as a framework for clarifying the concepts of needs assessment and capacity. This is followed by a review of selected articles on needs assessment methodology specific to SUD treatment.

What is a Needs Assessment?

Needs assessment emerged as a distinct branch of program evaluation in the 1970s and has evolved to address a broad scope of social issues.

Despite this broad application, the field has generally settled on a fairly standard definition of what constitutes a needs assessment as distinct from other types of evaluation and research. This definition begins with what White & Altschuld (2012) describe as "the classic definition of need: the discrepancy between the normative or 'current' condition and the optimal or 'what should be' state"--a formulation that is often condensed to the gap between "what is" and "what should be." In addition to these three components--a definition of what is, a specification of what should be, and a measurement of the gap between them--there is an essential fourth component: "Needs assessment also includes making judgments with regard to needs and putting them into prioritized order to guide decisions about what to do next" (Altschuld & Watkins, 2014).

This orientation to action is emphasized by a statement of the needs website: "Needs Assessments are used to identify strategic priorities, define results to be accomplished, guide decisions related to appropriate actions to be taken, establish evaluation criteria for making judgments of success, and inform the continual improvement of activities within organizations" and it is this feature that primarily distinguishes needs assessment from other forms of evaluation and social science research (Kaufman, 1986). It is worth examining each of the four components of a needs assessment in some detail with reference to the methodological literature, as they are more complex than they may initially appear.

Describing "What Is" and "What Should Be"

These two elements present several challenges that must be addressed in conducting a needs assessment. In the first place, there are often (as in the case of SUD treatment needs assessment) two dimensions to what is and what should be: the population of people with SUD and the services to provide treatment for them. The definition of each requires different sources of data and methods of analysis, which then must be reconciled to assess their interrelationship --for example, the relationship between the predominance of a specific type of SUD in a local area and the types of services available in that area (McAuliffe & Dunn, 2004b).

A second challenge is related to obtaining input from multiple stakeholders to represent a broad range of perspectives, which has now become a standard and essential method of needs assessment data collection (Stefaniak et al., 2015).

Analysis of data obtained by stakeholder input also presents challenges. Lee & Altschuld (2007b) discuss these in the framework of "discrepancy analysis"--that is, analyzing the difference between what respondents indicate what is and what should be. Challenges for analysis include the following:

  • Differentiating between discrepancies derived from wants versus needs.

  • Determining the size and nature of a discrepancy.

  • Developing discrepancies from multiple sources or methods.

  • Qualitative data (focus group interviews, individual interviews) may not directly lead to discrepancies.

  • Understanding the value of "Not Applicable" responses.

  • Understanding missing data for one or both scales.

  • Deciding whether discrepancies can be determined using total group means for an item or only from the subgroup of individuals completing both scales.

  • Determining whether different needs indices produce similar results.

  • When there is variation of the number of responses for different items.

White & Altschuld (2012) also address this issue in the context of surveys that produce "importance scores"--that is, ratings of what respondents consider most important with respect to both the what is and what should be. The problem is that responses of stakeholder groups may not be equivalent and therefore not comparable due to various factors such as differences in the availability of information. The authors recommend various methods of survey design to minimize these issues to enhance the comparability of different perspectives.

These methodological issues related to discrepancy analysis and importance scores are relevant to the methodology of SUD needs assessments, which often utilize surveys of a wide variety of stakeholders with varying amounts and types of knowledge--for example, members of the general public, who may have limited knowledge of the SUD treatment system but many concerns; providers, who possess a vast store of one kind knowledge; and consumers, who possess extensive knowledge of a different kind. A well-designed survey can minimize the variance that is due to variations in knowledge by designing questions that are based on knowledge available to all (for example, by providing an introductory explanation of the issue), thereby obtaining equivalent opinions from diverse stakeholder groups about what is and what should be.

Defining a Need--Measuring the Gap Between What Is and What Should Be

Once the what is and what should be conditions are properly specified, the next step is to measure the gap between, which constitutes the need. Of the four components of a needs assessment, it is the way in which a need is defined that has the greatest effect in determining the course and outcomes. According to Watkins & Kavele (2014): "How you define needs: (a) clarifies the goals of your assessment; (b) influences how you design your assessment; (c) determines what you measure, and therefore how you measure; and (d) influences what you report, to whom, and in what format," and there is a considerable body of literature addressing the question of how to define need. The authors provide a list of common categories with subtypes of meanings of the term "need" with references to sources: object-focused definitions (e.g., things without which it is impossible to live); goals (e.g., basic drives); deficiencies; gaps (in results, conditions, etc.) human condition (necessities for a full life); and physical or psychological needs. Given this variability, there is considerable potential for ambiguity in practice, the most common of which in needs assessment is a failure to distinguish between need and solutions, whereby need properly defined is the gap between two conditions which may be addressed by a diversity of solutions. Clarity in this distinction is an important consideration for SUD treatment needs assessment methodology.

Providing Recommendations and Setting Priorities

Of the four components of a needs assessment, this may be the most challenging, for two reasons: first, because it requires choosing what in most cases will be a multitude of possible solutions for addressing a need; and second, because a needs assessment typically involves input from multiple stakeholders with diverse perspectives, values, and interests, which must be taken into consideration in offering recommendations and priorities. The following considers these challenges, particularly in reference to SUD treatment capacity needs assessment.

Prioritization is especially critical in SUD needs assessment because the gap between treated and untreated prevalence is large. In the United States, fewer than 20% of individuals with drug use disorders and 10% with alcohol use disorders receive treatment (SAMHSA, 2013). A major reason for this gap is that a large proportion of those with a disorder do not perceive a need for treatment or are skeptical about its effectiveness; a logical recommendation, therefore, would be to initiate a public education campaign or an outreach program. As Edlund et al. (2009) point out, however, if eight out of nine individuals currently do not perceive a need for treatment, then if efforts to increase perceived need were successful for only one out of eight individuals, the overall number of people receiving SUD services could almost double, an increase that would likely overwhelm the capacity of the SUD treatment system. Shepard et al. (2005) estimate that to provide services to all those identified in surveys as needing services would require an increase of 5-10 times current services levels.

Another problem with untreated prevalence as a measure of need for treatment that has been widely debated in the literature since the first national epidemiological surveys in the 1990s is that prevalence as estimated by diagnostic criteria in national surveys may not correspond to need for treatment--either because informal supports are adequate for many individuals' level of distress or because their condition is transitory (Spitzer, 1998). Analyzing data from a nationally representative longitudinal study, the National Epidemiologic Survey of Alcohol and Related Conditions, Sareen et al. (2013) found that 49.8% of those with a diagnosis who did not receive treatment had remitted after three years.

Experts therefore generally indicate that for recommendations to be feasible they must incorporate some form of prioritization. As noted by Ciarlo & Tweed in reference to mental health planning "to permit effective planning, policy makers must decide exactly which needy group(s) of persons are likely targets for Only then can the selection and implementation of an appropriate indirect needs assessment model to estimate those needs logically follow" (1992). McAuliffe, for example, suggests that data obtained from population surveys that define need for treatment according to diagnostic criteria should be supplemented with measures of clinical significance to prioritize treatment for subgroups of the SUD population that have the greatest need--especially as he notes, a significant proportion of those identified as having a SUD at a point-in-time may not need treatment at all (due to spontaneous recovery) or at least not in a specialty setting (as opposed to routine primary care, 12-step programs, etc.).

There are many other ways of prioritizing, however, and the process of choosing among them is likely to be complex and frequently politicized. In the first place, prioritization may focus on the population (subgroups with greatest need) or on the service system (filling gaps in the continuum of care). Second, stakeholders may differ in their priorities depending on their perspective: law enforcement may prioritize individuals at risk of incarceration, neighborhood associations may prioritize public intoxication, advocacy groups may prioritize the homeless or racial/ethnic groups, policy makers may prioritize the uninsured, etc. Third, prioritization of population subgroups is likely also to require prioritization of components of the service system, as subgroups will vary in the type of service appropriate for each.

Review of Selected Substance Use Disorder Treatment Needs Assessment Methodology Literature

Technical reports on methodology for SUD services needs assessment are relatively limited in number, and most are several decades old. The number is somewhat larger if mental health needs assessment methodology is included, but the time frame for these is similar. An additional limitation, especially in the more recent literature, is that most of these reports focus on techniques for measuring treated and untreated prevalence, but few (with the exception of several discussed below) discuss methods for determining the appropriate number and capacity of service systems--for example, how many inpatient beds, outpatient slots, workforce categories, etc.

A key set of papers on SUD needs assessment are those by McAuliffe & colleagues in the 1990s and early 2000s (McAuliffe, 1978; McAuliffe, 1990; McAuliffe, Breer et al., 1991; Breer, McAuliffe et al., 1996; McAuliffe, LaBrie et al., 1999; McAuliffe, Woodworth et al., 2002; McAuliffe, Labrie et al., 2003; McAuliffe, 2004a; McAuliffe & Dunn, 2004b). The earliest of the reports (McAuliffe, Breer et al., 1991), one of several on a needs assessment project in Rhode Island, is also among the most rigorous of those identified in the environmental scan. Conducted prior to the widespread availability of national prevalence surveys such as the National Survey on Drug Use and Health (NSDUH), which many later reports rely upon, McAuliffe & colleagues developed estimates of treated and untreated prevalence in the Rhode Island population by conducting a telephone survey of Rhode Island residents, which they tested for reliability and validity. They also estimated the number of additional treatment slots needed and how these should be distributed across treatment modalities and geographical areas by reviewing drug use patterns of those who said they wanted treatment, by interviewing providers and agency officials to identify system gaps, and by examining areas' social indicators. Finally, they calculated the relative cost-effectiveness of different treatments.

More recently, Balenko & Peugh (2005) conducted a study of treatment needs of prison inmates in which they used a framework for estimating treatment needs derived from the ASAM Patient Placement Criteria and other client matching protocols that assessed drug use severity, drug-related behavioral consequences, and other social and health problems. The results indicated high levels of drug involvement, but considerable variation in severity/recency of use and health and social consequences, consistent with the suggestion by McAuliffe cited above, that many persons with SUD diagnoses may require little or no treatment.

Defining "Network Capacity" in the Context of SUD Treatment

Network capacity may be defined in a variety of ways depending on the context, but the methodological literature provides a framework for an operational definition specific to needs assessment. This formulation guides the review of SUD needs assessment reports presented in this report.

Essential to a determination of what is and what should be is an understanding of what is meant by network capacity (used here synonymously with "system capacity"), which may be defined narrowly or more broadly. The narrow definition refers to the number of available treatment slots, such as hospital beds, workforce size, number of clinics. Measurement of capacity in this definition consists of an inventory or census, conducted within a defined geographic boundary. An example of an inventory, with the United States as the boundary, is the SAMHSA National Survey of Substance Abuse Treatment Services (N-SSATS). This report assumes a broader definition of network capacity, one that takes into consideration two other aspects of a service system besides the supply: continuity and quality.

Continuity refers to a balanced continuum of care such as represented by the ASAM Criteria or the SAMHSA Good and Modern Behavioral Health System. ASAM's criteria is a system for individualized treatment planning that represents SUD treatment as a continuum marked by four broad levels of service and an early intervention level. Within the five broad levels of care, decimal numbers are used to further express gradations of intensity of services. These levels of care provide a standard scale representing the continuum of recovery-oriented addiction services.[1] The SAMHSA Continuum of Care[2] is a structure consisting of four components, Promotion, Prevention, Treatment and Recovery, each of which includes a set of interventions or services.

Quality is a multi-dimensional construct that expands the definition of network capacity. An example of a framework for assessing capacity in terms of quality is the Institute of Medicine's (IOM's) "Six Aims" (IOM, 2011) for a system in which there is not only an adequate supply of services but those services also have the attributes of being Safe, Effective, Patient-centered, Timely, Efficient, and Equitable. Examples of how each of the six aims might be addressed in SUD treatment needs assessments are:

  • Safe: Is the workforce adequately trained to deal with emergencies?

  • Effective: Does the system include a full complement of evidence-based services?

  • Patient-centered: Is there leadership and a management structure that can support a culture of patient-centeredness?

  • Timely: Are programs adequately staffed such that long waiting times are avoided?

  • Efficient: Is there an adequate data infrastructure in place to monitor utilization and resource allocation?

  • Equitable: Are there adequate quality improvement processes in place to identify and address disparities?

These are only a few of many aspects of capacity related to the six aims that a needs assessment might address. Few, if any, needs assessments address all six aims, but our review identifies which of these dimensions they do address and describes approaches and best practices in the context of the project's research questions.

Defining "Needs Assessment" in the Context of SUD Treatment

The "what is" condition: Based on the four components of needs assessment, specification of "what is" calls for measures of current system capacity as described above--an inventory of existing services, available slots, number of people receiving services, number and configuration of staffing, etc.--as represented, for example, by the N-SSATS and also a count of the number of people with met and unmet treatment needs as represented by the NSDUH.

The "what should be" condition: The second component, "What should be" implies specification of some goal or standard for those factors included under "what is." It is this component that distinguishes a needs assessment from other types of research and reports (epidemiological studies, for example) which include only the first component, a measurement of "what is."

The specification of "what should be," though essential for a needs assessment, may be more formal and quantifiable or less formal and qualitative, as shown in the SUD needs assessments we review. Examples of more formal definitions are the standards represented by the ASAM Criteria or the SAMHSA Continuum of Care described above. Less formal specifications, which are more typical of the needs assessments reviewed, generally involve some combination of stakeholder input, comparison with national statistics, expert opinion and the like. The less formal approach of SUD needs assessment is explained to some extent by the fact that the SUD treatment system is less formally organized than, for example, health care, for which the Health Resources and Services Administration (HRSA) is able to develop research-based formulas for the number of patients that can be served by specific practitioner types as discussed below.

Measurement of the gap: This component of a SUD needs assessment also may be more or less formal. An example of a very formal approach is the HRSA method of designating Health Professional Shortage Areas (HPSA) and Medically Underserved Areas/Populations (MUA/P), as determined by the Index of Medical Underservice (IMU), the calculation of which is based on four weighted criteria (population-to-provider ratio; percent of population below the federal poverty level (FPL); percent of the population over age 65; infant mortality rate). IMU can range from zero to 100, where zero represents the completely underserved and 100 represents completely served. MUA/Ps are those with an IMU of 62.0 or less. In this example, a standard of "what should be" is set by setting an ideal standard of number of specific workforce professionals to the population levels, adjusted for poverty, age, and infant mortality. Notably, however, the ideal of the population being completely served is then adjusted downward by the 62.0 cut-off score, representing what is practical to achieve. This is analogous to the case of SUD treatment, where the ideal of treatment for all with a diagnosed disorder is neither feasible nor necessary, as discussed in the Background section of this report.

Guidance for decision making: The formality of the HRSA method of gap measurement is not always possible or even necessary for purposes of guiding policy decisions related to SUD treatment, and in many cases the standard is more subjective or ad hoc. A major challenge in decision making with respect to SUD treatment needs is the very large gap between those who need treatment (based on most definitions of need) and those who receive it. As noted in the Background section, fewer than 8% of those with alcohol abuse or dependence receive treatment. Reducing this gap by even a few percentage points would require an enormous expansion of capacity and a corresponding commitment of resources beyond that which is feasible in most circumstances. For this reason, prioritization of recommendations is especially important in needs assessment for SUD treatment. Few of the needs assessments that we reviewed included anything more than the most cursory process of prioritizing recommendations. We suggest that methods of doing so would be an important target for improvement of SUD needs assessment methodology, and that the methods would be prime candidates for designation as best practices, a topic we will address in more detail in the final report with input from the TAG.

Challenges in Defining Need

The challenge of defining need for SUD treatment was addressed at length by the TAG in the June 11, 2018, meeting. Key points include:

  • Defining need based on diagnosis alone (e.g., the DSM criteria used in the NSDUH) is problematic. The DSM criteria for SUD consist of a certain number of symptoms (2 out of a possible 11 in DSM-IV) displayed in the past 12 months; however, many people--perhaps as many as half--who meet these criteria at some point in the 12 months, recover. Using diagnosis alone would result in an overestimate of service capacity requirements.
  • The TAG discussed various approaches to addressing the limitations of diagnosis alone--notably, supplementing diagnosis with some measure of clinical significance or functioning. Yet this presents the challenge of determining appropriate cut-off points: At what point does substance use become clinically significant? How much and what kind of functional impairment indicates a need for treatment? The BRFSS, which includes questions about functioning (though not specific to SUD) suggests one possible approach.
  • An additional challenge in measuring need is that many of those identified by some objective criteria--even a more restrictive definition that supplements diagnosis with clinical significance or functioning--do not themselves perceive a need or do not wish to receive treatment. A planning model that ignores motivation would therefore result in excess capacity. One approach would be to combine treatment capacity with outreach and public education, though it would then be necessary to ensure that resources were adequate to meet the increased demand.