In addition to identifying what constructs should be measured, researchers must select reliable and valid measures. In separate works, Beverly (2001), Citro & Michael (1995), Bauman (1998), and Moore (1997) suggest criteria researchers may use when selecting existing measures or developing new measures of material hardship. (Exhibit 2.3 provides a complete list of each set of proposed criteria.)
While Beverly (2001) and Bauman (1998) are the only authors that propose criteria specifically for material hardship measurement, the criteria proposed by Citro & Michael (1995) and Moore (1997) for selecting poverty measures and indicators of child well-being, respectively, also have considerable applicability to material hardship measurement.
Bauman, Citro & Michael, and Moore's criteria have several common threads:
- "Public acceptability," "Face validity," and "Clear and comprehensible"
In recommending criteria, Citro & Michael, Bauman, and Moore all include criteria that direct researchers to select measures that will produce data that are easily understood and "broadly acceptable" to both researchers and the general public.
- "Statistical feasibility," "Common interpretation," "Forward looking," "Relationship to explanatory variables"
Researchers should consider, in advance, how the data collected using a specific measure will be used. Citro & Michael caution researchers to select measures that are "logically consistent" with each other and will allow for comparisons (e.g., across time periods and groups). Similarly, Bauman and Moore both note that when selecting measures researchers should pay close attention to whether they will serve as good predictors of poverty or material hardship, have a consistent interpretation across population subgroups, and provide effective baseline data for future trends.
- "Predicting long-term negative outcome[s]," "Positive outcomes," "Comprehensive coverage"
Selected measures should be related to well-established outcomes both negative and positive (e.g., educational attainment, food insecurity). They also should assess well-being across an "array of outcomes, behavior and processes" (Moore, 1997).
- "Reliability," "Consistency over time," "Operational feasibility"
Measures should be reliable both in the short- and long-term. That is, they should consistently provide data that measures the same construct.
Beverly's (2001) criteria are somewhat narrower in focus and specifically relate to how measures of material hardship should be defined and operationalized. In her presentation at the Roundtable Meeting, Beverly focused her list of criteria on three specific recommendations. Material hardship measures should:
- Measure objective, rather than subjective conditions.
Both objective and subjective measures may be used to describe material need. In the latter case, subjective measures are self-assessments or evaluations of material need, whereas objective measures capture facts about specific conditions or circumstances. As suggested by the definitional difference between material well-being and overall well-being, assessing material need is best done with objective indicators of living conditions, rather than subjective self-assessments or evaluations of these conditions. Researchers present at the Roundtable Meeting also noted that there are fundamental differences between actual experiences and perceptions of actual experiences and that direct measures of material hardship should focus on actual experiences.
However, most survey questions related to material need will always have an element of subjectivity due to the fact that the respondent reports the information. In some cases, this may lead to either "false positives" or underreporting. For example, in face-to-face interviews during the WES some interviewers noted that they saw mothers become visibly uncomfortable when they reported that their children had insufficient food. This led researchers to be concerned that the mothers might be underreporting food insufficiency in their household due to embarrassment or discomfort with admitting that they cannot provide for their children.
- Use direct measures of need that focus on consumption and short-term outcomes.
Direct measures should focus on consumption outcomes associated with material hardship, rather than mediating factors. For example, determining whether a person has health insurance captures data about the resources available to address unmet medical needs (indirect). In contrast, a question about whether an unmet need exists provides a direct assessment of a particular health outcome.
- To the extent possible, indicate the cause of hardship.
It is important to know why families experience hardship. For example, a hardship measure that reports hunger could mean that there are not enough resources to obtain necessary food, or that the respondent has reduced food intake for some other reason (e.g. a desire to lose weight). Measures of failure to obtain needed medical or dental care are even more vulnerable to this difficulty in interpretation. In the absence of information on "cause" it is difficult to know whether the hardship is "real" or the result of individual choices or preferences.
Additionally, in their presentations at the Roundtable Meeting, Connie Citro and Sondra Beverly both emphasized that hardship measures ought to be demonstrably linked to poor outcomes or well-being, and not to mediating factors. For example, it was pointed out that "lack of health insurance" is a mediating factor, not a direct hardship.
|The Committee on National Statistics/Panel on Poverty and Family Assistance
(Citro & Michael, 1995)
|Recommendations for Measures of Material Hardship (Beverly, 2001)||Four Evaluation Techniques Used to Evaluate Hardship Measures (Bauman, 1998)||Criteria for Indicators of Child Well-being (Moore, 1997)|
|Public Acceptability: "A sensible cut-off. Some rationale that has face validity understandable and broadly acceptable."
Statistical Feasibility. "The measure must be logically consistent[and] allow for reasonable comparative analyses across time, place, types of families, and population groups."
Operational Feasibility. "Implies that data can be collected that will in fact measure the prevalence of the conditions underlying the concept of poverty."
|Indicators of material hardship should assess consumption of the following goods and services: food, housing, utilities, medical, clothing, and consumer durables.
Measures of material hardship should reflect very basic standards of material adequacy.
To the extent possible, indicators should measure the severity of hardship.
The core set of hardship measures should capture objective, rather than subjective conditions.
The core set of hardship measures should consist of direct, rather than indirect, indicators.
To the extent possible, hardship measures should indicate the cause of hardship.
The core set of hardship indicators should include composite indexes of hardship as well as separate measures.
Reasonable relationship to explanatory variables that are used to predict poverty.
Utility in predicting a long-term negative outcome whose relation to family poverty has been well established (e.g., high school drop out).
Reliability, measured by tracking measures over time and assessing sensitivity to sample selection and attrition bias.
|Comprehensive coverage. Indicators should assess well-being across a broad array of outcomes, behavior and processes.
Children of all ages. Age-appropriate indicators are needed at every age. Clear and comprehensible. The public should easily and readily understand indicators.
Positive outcomes. Indicators should assess positive as well as negative aspects of well-being.
Depth, breadth, and duration. Indicators are needed that assess dispersion across given measures of well-being, children's duration in status, and cumulative risk factors experienced by children.
Common interpretation. Indicators should have the same meaning in varied population subgroups.
Consistency over time. Indicators should have the same meaning across time.
Forward-looking. Indicators should be collected now that anticipate the future and provide baseline data for subsequent trends.
Reflective of Social Goals. Some indicators should allow us to track progress in meeting national, state and local goals.