Needs Assessment Methodologies in Determining Treatment Capacity for Substance Use Disorders: Environmental Scan Final Report. METHODS


Following the review of selected literature on needs assessment methodology, we conducted three activities to address the research questions (separated primarily because they address different types of questions that draw on different data sources): (1) retrieval and review of a convenience sample of SUD treatment capacity needs assessment reports, to address questions related to content and practices in the field; (2) description of some needs assessments from other fields that might be used as models for SUD treatment, to address the research question, "Are there needs assessments from other provider types...that could be applied to the SUD treatment workforce?"; and (3) a separate scan, based on a review of state policy documents, to address the research question, "How do states and managed care organizations operationalize network capacity."

Methods for Review of Substance Use Disorder Needs Assessment Reports

Search Strategy

The scan of SUD treatment capacity needs assessments utilizes modified methods of systematic literature review to locate representative examples of SUD treatment needs assessment reports in the published and gray literature to address the research questions related to practices in the field. Systematic reviews require careful search strategies to ensure that all relevant reports are identified, and they typically entail methods for rating the quality of individual reports (primarily the degree to which they minimize the risk of bias) to assess the overall strength of the evidence summarized from the reports.

A challenge of this approach is that a majority of SUD needs assessment reports are not published in peer reviewed journals and therefore do not adhere to the consistent standards of format and quality that apply to the peer reviewed reports that are the typical source for systematic reviews in the health care field. Moreover, accessibility of unpublished needs assessment, especially those produced less recently (prior to widespread use of the Internet) is limited. Though we believe that the convenience sample of SUD network capacity needs assessments that we were able to retrieve, given the resources for this project, is representative of current practice in the field, it is undoubtedly a small portion of those produced over the years.

White and Altschuld (2012) comment on this challenge:

The lack of sources is not surprising since many assessments are never circulated beyond the organization, probably because they are used for internal purposes with no thought to publishing. Sharing them with the evaluation community would go a long way to help practitioners as well as build the body of professional knowledge. Toward that end more needs assessment research should be disseminated. This starts with greater emphasis on writing and publication by university students, professional associations, practitioners, and those supporting needs assessments.

We have addressed these limitations of the systematic review approach in two ways--first by using more flexible inclusion/exclusion criteria and second by using a more complex data extraction process.

Inclusion/Exclusion Criteria

Our inclusion/exclusion criteria are based on the concept of needs assessment derived from the methodological literature as described previously. That is, to be included a study of SUD must include: (1) a description of "what is" at both the population and system level (for example, prevalence of a disorder and an inventory of available services); (2) some operational definition of what should be (for example, the numbers and characteristics of individuals who should receive treatment) and some model of services necessary to treat the identified population; (3) some measure of the gap between what is and what should be (for example, a reduction in the number or percent of individuals with an unmet need, or the cost of additional services required to address the need).

We initially considered a fourth requirement, consistent with the definition of needs assessment: that the report include specific recommendations or priorities for action. Without knowing in advance how many reports this might exclude, we decided on a more relaxed standard--to include any report that: (a) is self-identified as a needs assessment; (b) meets the other three criteria; and (c) at least provides information suitable for informing policy and decision making, whether or not in the form of explicit recommendations. To assess the utility of reports for this purpose, however, we rated each as to whether it includes: (1) suggestions or recommendations for narrowing the gap; (2) prioritized recommendations; (3) strategies for acting on the recommendations; and (4) the feasibility of accomplishing recommended actions.

Excluded Reports: An example of a report that would be excluded based on these criteria is an epidemiological study providing estimates of treated and untreated prevalence--that is, a measure of need--but without some measure of existing or desired system capacity. Another example would be a description of a service system that may identify areas for needed expansion or improvement but does not include information about the characteristics and size of the population served by the system.

Sources and Search Terms

We initially cast a wide net, testing various sources (e.g., databases including PubMed, Webofscience, Google Scholar, relevant journals such as the Journal of Substance Abuse, etc.) and a variety of search terms. We were quickly able to narrow this strategy down to a few sources and search terms as we found that any more added very little to sensitivity (identifying a larger number of relevant reports) or specificity (excluding non-relevant reports). Accordingly, we adopted two strategies with search terms: one for unpublished reports and the other for published (peer reviewed) articles. For unpublished reports we simply entered Google search terms "[state] needs assessment" with the name of each state. This typically produced reports in the range of 150-200 per state. We then scanned the summary description in the Google entry to determine whether it merited closer review to determine eligibility for inclusion. We did not establish a time frame for two reasons: (1) because the availability of information on the Internet is self-limiting to the past 10-15 years; and (2) because the focus of the environmental scan is primarily current, rather than historical, practice.

For peer reviewed literature we used PubMed; after testing various combinations of Medical Subject Headings (MeSH) search terms, we settled on "Substance-Related Disorders/therapy"[MeSH] AND "Health Services Needs and Demand"[MeSH] as providing the best balance between specificity and sensitivity. (We found that the MeSH term "needs assessment" was not useful in that it resulted primarily in articles related to instruments for assessing the health care of individuals.)

FIGURE 1. Flow Diagram, Adapted from PRISMA
FIGURE 1, Flow Diagram with Four Steps and 2 paths: Step 1) Identification - path 1 = Reports identified through PubMed Search (n=862); path 2 = Reports identified through Internet search (n=7920). Step 2) Screening - 82 path 1 reports screened; 289 path 2 reports screened. Step 3) Eligibility - 7 path 1 reports reviewed, 33 path 2 reports reviewed. Step 4) Included - 40 total reports (paths 1 and 2) included in the quantitative synthesis.
SOURCE: Moher, Liberati, Tetzlaff & Altman, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med, 6(7): e1000097. doi:10.1371/journal.pmed1000097. For more information, visit

For published articles, we reviewed abstracts of those identified initially by the search terms, and for those that seemed appropriate we reviewed the entire article. For those identified in the Internet search, we reviewed the document and retrieved any that addressed SUD, including those that also addressed mental health.

After screening out studies that did not meet the inclusion criteria, we classified those that were included into two categories: "mandated" and "locally initiated" needs assessments. Mandated needs assessments, described in detail below, are those conducted to meet requirements of various broader programs, such as federal grants to the states. Locally initiated needs assessments, on the other hand, are typically standalone initiatives conducted for a specific policy purpose or single research project that meet the inclusion criteria.

Using this search strategy, for locally initiated needs assessments we obtained a purposeful convenience sample of 33 reports from the grey literature (Internet) and seven from the peer-reviewed literature that reflect a diverse range of factors of interest for the environmental scan: geographical diversity (a range of states, and both rural and urban areas), scale (national, state, local areas) methodological variety (complex statistical models as well as basic approaches using available data), and a variety of topic areas (e.g., both general and vulnerable populations, general capacity issues as well as specific areas such as workforce capacity, various type of substance use).

The flow diagram above, adapted from the PRISMA recommendations for reporting of systematic reviews, presents the number of reports screened and included.

For the mandated reports, given their more generic nature, we did not aim for a specific sample but instead scanned a number of them in each category using the review approach described below.

Review Methods for Mandated and Locally Initiated Needs Assessments

We adopted different strategies for reviewing these two categories of needs assessments. For the mandated needs assessments, we provide a general overview of the methodologies and content based on review of a sample of each type and review of the guidance or requirements provided by the relevant authorizing organization. For the locally initiated needs assessment we conducted a more detailed analysis with data extraction based on the research questions presented above.

This differential approach is based on three considerations about the mandated reports--first because the sheer number would exceed the resources of this project to analyze individually, second their generic nature in being conducted according to specified requirements thereby limits the value in reviewing large numbers of them, and third because they are less focused on specific policy questions related to system capacity and therefore provide less knowledge relevant to the research questions that would contribute to the advance of SUD needs assessment methodology.

Procedures for Extracting Information to Address Research Questions Related to SUD Needs Assessment Practices and Methods

Content and Methods: Each of the research questions was broken down into a set of individual items. In this context, we use the term methods in the broad sense of approaches taken in conducting the needs assessment (e.g., how data were collected, how levels of care were defined, how the service system was defined, etc.). Using a coding instrument based on the research questions listed in the introduction, we extracted information related to the content and methods of the reports and to the quality based on a set of criteria. Questions related to content and methods are addressed in the review are presented in abbreviated form in Figure 2. The protocol (Needs Assessment Data Extraction Form), with questions in full and individual items may be found in Appendix E. The remaining research questions listed in the introduction involve recommendations that will be addressed in the final report.

FIGURE 2. Abbreviated Review Questions Related to Content and Practices
  1. What questions are addressed?
  2. What populations are addressed?
  3. What workers are incorporated?
  4. What settings are incorporated?
  5. What levels of care are incorporated?
  6. What data sources/methods are used?
  7. Are there best practices identified?
  8. Have any methods been validated?

Using the number of responses to each item we were then able to quantify this information in order to provide an overview of practices in the field as represented by this sample of needs assessments.

It should be emphasized that the research questions focus more on the issues addressed by SUD needs assessments and the method employed, rather than specifics of the findings. That is, we were not attempting to summarize data about, for example, prevalence or numbers of people receiving services as in a systematic review of research literature, but rather were extracting information about the kinds of data that were presented and how they were obtained and analyzed in order to assess current practices in the field.

Quality Ratings: In addition to extracting information about the content and practices of SUD needs assessment, the protocol included criteria for rating the quality of needs assessment. This approach is adapted from the methods of systematic reviews, where quality is a matter of the extent to which the reader is able to assess the strengths and weaknesses of the study based on the relative clarity and transparency of information provided about how the study was conducted (Moher, et al., 2009). In this framework, quality is assessed based on the extent of information provided in the reports about the methods employed: for example, if a survey was conducted, does the report indicate whether it had been tested, how the sample was defined, etc. Questions related to quality are presented in abbreviated form in Figure 3 and are presented in their entirety in the protocol (Appendix E).

FIGURE 3. Abbreviated Review Questions Related to Quality
  1. Are questions/purposes clearly expressed and addressed?
  2. Are methods for identifying the population described?
  3. Are criteria for classifying workforce presented (if applicable)?
  4. Are levels of treatment need clearly specified?
  5. Is the setting/geographical area clearly identified?
  6. Is survey methodology described (if applicable)?
  7. What types of recommendations and priorities are presented?
  8. Are ethical issues/Institutional Review Board (IRB) requirements discussed?

The purpose of the quality assessment is not to judge the quality of individual studies but rather to gain a sense of current practices in SUD capacity needs assessment overall. The criteria are based on some of the research questions and are generally designed to indicate how successfully reports represent the essential components of a needs assessment: defining "what is" and "what should be" with sufficient clarity and specificity that that the gap between them is measurable and providing guidance for decision making and priority-setting. Thus, the focus of the quality ratings is upon utility--to what extent does a needs assessment provide the information necessary to serve as a guide for decision making?

In the following sections these questions are addressed as a general overview of the mandated needs assessments and then in a more detailed way, using the Data Extraction Form, for the locally initiated SUD-specific needs assessment reports.