Measures of Material Hardship: Final Report. Creating Summary Measures of Material Hardship

04/01/2004

Many researchers have combined measures of material hardship to create a composite or summary measure (e.g., Beverly, 1999a; Federman et al., 1996; Mayer & Jencks, 1989; Rector et al., 1999). These composite measures may be formed within a dimension (e.g. insufficient food, number of housing problems) or across dimensions (combining housing problems and insufficient food). Composite or summary measures provide additional information on the concurrence of various hardships, but are at risk of obscuring detail seen in the individual components.

The key argument for constructing a summary measure is that it shows the extent of overall material hardship. A composite measure could rank hardship severity for families that are housing insecure, but not food insecure, and families that are food insecure, but not housing insecure. Individual measures provide no way to order these two groups of families. Still, collapsing the data in this way sacrifices information. If these types of families need different supports it may be important to distinguish them.

Furthermore, the process for constructing a summary measure is judgmental. Overall rankings vary depending on the items contained within a dimension, the dimensions included, and the weights used to combine dimensions. In the absence of a standard approach, comparisons may be difficult and disagreements about the extent of hardship likely.

A final disadvantage to composite measures is that they may have less face validity than individual measures. For example, insufficient money to pay the rent, or having the telephone disconnected for non-payment, are clearer on their face than a concept of overall material hardship. Aggregation within a dimension is more likely to have face validity than aggregation across dimensions. For example, "number of housing problems" is a summary measure but is nevertheless a clear concept to many. The justification for constructing a summary measure is strongest when there is a well-defined central idea or construct. The summary measure of food security, for example, was developed after extensive theoretical and empirical work on the interrelated concepts of food security, food insecurity, and hunger. Many researchers now use this measure; however, this measure continues to be controversial, with some critics questioning whether food insecurity measures provide useful information about hunger.

The disadvantages of summary measures of material hardship are mitigated if the individual components are presented as well. This is the approach generally taken in the literature. Typically the relationships among the indicators also are analyzed, and the discussion of key findings is based both on the composite and individual measures. Presenting both types of measures provides a sense of the extent to which findings based on the composite measure are robust to the approach used to construct the composite.

Technical Issues in Creating Summary Measures

Two general approaches are used for combining multiple indicators: indexes and scales. Indexes, the most commonly used approach to combining material hardship measures, are created using logic and judgment about what constitutes a "reasonable" set of indicators. In contrast, scales attempt to identify a set of "factors or principal components that are assumed to represent an underlying dimension of well-being that is not perfectly reflected by any single indicator" (Bauman, 2002a). An overview of Bauman's (2002a) summary of the comparative advantages and disadvantages of indexes, scales, and separate indicators is presented in Exhibit 2.4.

Both approaches must determine whether the summary measure should be continuous or categorical. For example, a continuous measure of material hardship might be constructed so as to take on values between 0 and 100. A categorical measure has a discrete number of hardship levels, with as few as two. An example of a categorical hardship measure is the USDA food security scale, which has three levels (food secure, food insecure-without hunger, food insecure-with hunger).

With either a categorical or continuous measure, analysts also may choose to set a threshold level, with hardship defined according to whether a family is above or below the threshold (analogous to the way poverty is typically defined). A threshold may be established based on judgment, or based on the distribution or correlation with an external measure. While a threshold provides a convenient summary, its use may entail the loss of valuable information. For example, two households might meet a threshold for being housing insecure, but one might be much worse off than another (e.g., homeless versus missing a rent payment). This drawback can be mitigated by providing more detailed results.

Exhibit 2.4
Bauman (2002a) Summary of Advantages and Disadvantages of Indexes, Scales, and Separate Indicators
Indexes Scales Subjective Evaluation of Separate Indicators
Advantages Disadvantages Advantages Disadvantages Advantages Disadvantages
  • Simple to construct  only need a "reasonable" set of indicators of a specific contract to be put together.
  • Improves measurement over the use of a single indicator in that it lowers measurement error (assuming index is well-constructed).
  • The index approach established by Mayer and Jencks (1989) has been validated and used as a model for indexes in other research (e.g., (Mirowsky & Ross, 1999; Short & Shea, 1995).
  • Lack of agreement on a set of criteria for choosing items.
  • There may be an indeterminate number of items to be included in an index.
  • Different numbers of items can result in different scores and weights for different components of the index.
  • Even if researchers agree to use a standard set of items, there is still the challenge of producing acceptable and valid weights.
  • Using an index throws away potentially useful information about the severity of the items used.
  • Allows researchers to identify a "latent class" or underlying definition of a construct (e.g., material hardship).
  • One strategy allows researchers to look at correlations between indicators to arrive at a set of factors (i.e., principle components) that are assumed to represent an underlying dimension of well-being not perfectly reflected by any single indicator. This avoids may of the disadvantages associated with an index.
  • Intercorrelations between indicators may be attenuated by substitution or selective applicability.
  • Relationships between variables are causal, or related to causal processes. This makes interpreting the model more complicated.
  • Avoids problems of weighting and ignoring severity, as found when developing indexes.
  • Avoids problems of ignoring causal relationships.
  • Face validity of specific measures.
  • Analyst must make subjective evaluations about the reliability and validity of the individual items and take into account offsetting errors when examining a set of unsummarized indicators.
  • May present difficulties in ranking people consistently  that is, we don't know when someone is definitely "worse off" when the set of indicators changes.

Summary Measures of Material Hardship Based on Logic and Judgment (Indexes)

When developing indexes of material hardship, indicators or dimensions chosen by the researcher are often combined simply by summation, with each component receiving the same weight. Weights are sometimes assigned based on the perceived importance or relevance to respondents of each component. Low frequency of a given hardship in the population indicates the degree to which a component is a necessity, in that the higher the proportion of households with a particular item (or that do not experience a particular hardship), the greater the extent to which the item may be deemed to be a necessity. Thus, lacking a refrigerator may be given greater weight than lacking an automatic dishwasher.

Mayer and Jencks (1989) created an index that was weighted according to the separate indicators' relative importance to the families that experience hardship. Weights were developed by regressing respondents' answers to a question on how families felt about their standard of living on the researchers' 10 hardship measures. Nonetheless, Mayer and Jencks ultimately reported their results in terms of unit weights (measured on the total number of hardships a respondent reported) because it was easier to interpret.

Despite the ad hoc nature of this approach, it is possible to use statistical techniques to validate indexes and similar summary measures to provide confidence in the soundness of the approach. In the paper mentioned above by Mayer and Jencks (1989) their approach has been validated (Bauman, 1998) and has been the model of indexes used in other research (Mirowsky & Ross, 1999; Short & Shea, 1995).

Exhibit 2.5 lists example studies that illustrate this general approach; however, it is important to note that there are numerous other studies that have used this approach.

Exhibit 2.5
Examples of Summary Measures Based on Logic and Judgment
(Mayer & Jencks, 1989). Based on two telephone surveys of Chicago households, the authors construct an index using the total number of hardships reported per family, out of eight dichotomous hardships relating to food, housing, and medical care. Each component is weighted equally in the total (i.e., unit weighted).(3) The value of their summary measure therefore varies from zero to eight.

(Federman et al., 1996). Using SIPP data, the authors construct an index that is the total number of deprivations reported out of nine dichotomous indicators relating to food, housing, utilities, and appliances. Each component is implicitly weighted equally.

(Rector et al., 1999). The authors use SIPP data to construct a composite hardship measure based on a combination of specific hardship indicators and income in relation to the poverty threshold. Specifically, the authors define persons to have "overall material hardship" if they live in households with incomes below 200 percent of the official poverty threshold, and they have one or more "substantial" hardships or three or more "moderate" problems.

(Beverly, 1999a). Using SIPP data, the author constructs a primary hardship index as the sum of six equally-weighted dichotomous indicators relating to food security, housing, utilities, and medical need. The author also defines a threshold: a family is in hardship if it experiences any one of the individual indicators.

(Martinez & Ruiz-Huerta, 2000). Using data for Spain from the European Community Household Panel survey, the authors aggregate 20 dichotomous hardship indicators into four dimensions: maintenance (measures of current financial strain); durable goods; housing conditions; and lifestyle (e.g., ability to save, ability to buy furniture). The authors use weighted sums to combine the 20 items into four dimensions, where the weights are based on the proportion of individuals not lacking an item. The indexes are constructed to vary between 0 and 100. The authors construct a total hardship index by taking a weighted sum of the four dimensional indexes, where the weights are based on the average weight within each dimension. The authors also construct a basic hardship index, analogous to the total index but including only items lacked by less than half the population.

(Martinetti, 2000). Using data from 1994 survey of Italian households, Martinetti uses "fuzzy sets" theory to combine individual hardship indicators--some dichotomous, some categorical--into five dimensions: housing, health, education, social interactions, and psychological conditions. In combining the indicators within dimensions, the author uses different approaches for each dimension, including: weight averaging with weights based on the frequency of the hardship; weight averaging with unit weights; and taking the union of dichotomous indicators. Martinetti also constructs an overall hardship index combining the five dimensions.

(Muffels & Fourarge, 2003). Using data for 12 countries from the European Community Household Panel survey, the authors aggregate 21 dichotomous hardship indicators into a total hardship index. The 21 indicators reflect health conditions, financial stress, housing conditions, and possession of durables. The indicators are combined into a total hardship index using a weighted sum, where the weights for each indicator are based on the proportion of individuals not deprived by that indicator.

Summary Measures of Material Hardship Based on Statistical Approaches (Scales)

An alternative to a judgment-based approach to creating summary measures of hardship is to use statistical methods to select indicators, group indicators into dimensions, and to create weights. Methods such as cluster analysis, correspondence analysis, latent class analysis and factor analysis can be used to aggregate indicators into groups (based on their mutual correlations), with weights determined by the statistical model. Weights also may be based on rarity or on correlation with an external measure of the construct. Items that are found to represent different dimensions are dropped from the scale. The statistical or modeling approach, however, does not eliminate the need to make assumptions and subjective decisions.

Exhibit 2.6 lists example studies that illustrate this general approach; numerous other studies have been done.

Exhibit 2.6
Percentage of Total Examples of Summary Measures Based on Statistical Approaches
(Bickel, Nord, Price, Hamilton, & Cook, 2000). The authors provide a methodology for measuring household food security using 18 indicators collected via survey (or using a set of 6 indicators). A statistical approach known as a Rasch model was used to select the 18 indicators from a larger set, and to develop a scale that translates the number of affirmative responses into an equal interval scale that can be manipulated mathematically (e.g., mean scale scores can be computed). The scale scores are also used to assign households to one of four different levels of food security (food secure; food insecure without hunger; food insecure with hunger, moderate; and food insecure with hunger, severe). The food security scale is an example of aggregation within a particular dimension of material hardship, not an aggregation across dimensions.

(Bauman, 2002a). Using SIPP data, the author attempted to construct a material hardship scale using latent class analysis. Bauman examined the relationship between the latent classes and poverty to see whether a natural ordering of classes existed, and tested the degree to which the classes captured the information about poverty in the individual indicators. Based on his analysis, the author did not find strong support for a scale summarizing material hardship.

(Layte, Maitre, Nolan, & Whelan, 1999). Using data from the European Community Household Panel survey, the authors use factor analysis to cluster 25 dichotomous indicators of material hardship into five distinct groups: basic lifestyle deprivation (e.g., food, clothing), secondary lifestyle deprivation (e.g., car, telephone), housing facilities, housing deterioration, and environmental problems (e.g., noise, vandalism). The authors calculate a value for each household in each group by summing the number of indicators on which the household is deprived. The authors also construct an overall hardship index as the unit-weighted sum of the 25 indicators.

(Gundersen, 1996). In contrast to the other studies summarized in this section, the author uses a model-based approach to create a summary measure of hardship. Gunderson begins by developing an axiomatically-derived model of hardship, which he refers to as a "well-being evaluation function." Using data from the American Housing Survey, the author applies the model to develop "housing evaluation functions." Each function uses a different functional form to aggregate three indicators of housing quality (adequacy, comfort, and neighborhood) into an overall measure of housing hardship. (This is an example of aggregation within a dimension rather than across dimensions.)

View full report

Preview
Download

"report.pdf" (pdf, 4Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®