Due to data shortcomings, the study authors made many decisions regarded as “second best.” Reviewers were concerned about the extent to which findings from the analysis were sensitive to these decisions and specifically those regarding:
- Weighting methods
- Decisions for inclusion of carriers and states in the analysis
Consequently, to determine the effect of different weighting mechanism and inclusion rules, we simulated an extensive set of alternative rules for weighting and inclusion. NORC conducted these simulations in March 2012
We tested four methods for weighting:
- Original method – NAIC basis with MEPS small group adjustment. This was the original proposed method. Enrollment data from the NAIC was the basis for the initial carrier weights (Table 6, Initial Carrier Weight). MEPS-IC was used to adjust for enrollment distributions by plan type for the small group market as described above (Table 6, MEPS-IC Distribution). We found no comparable data that would allow us to make a similar adjustment for the individual market; hence, we assumed a uniform distribution in that market (i.e., no adjustment was made). This approach made no attempt to adjust weights within carrier on the basis of filing enrollment data.
- Alternative method – Filing enrollment as basis for weights. We used the number of covered lives from filings as the basis for weighting instead of the NAIC data, and no adjustments were made to reflect market differences as in the original method. Results differ significantly from the original method when using this alternative method, primarily due to a large number of observations being excluded due to missing enrollment information (and thus not being able to derive a survey weight). Although the percentage of observations that fell out of the analysis was not large, lost observations were systematic, not random, and distributed non-uniformly across states. States with prior-approval regulations were more likely to have enrollment data. Large carriers were less likely to have enrollment data. HMO plans were more likely to have enrollment information. Thus, this approach would have yielded large non-response bias in the resulting estimates and was therefore rejected.
- Alternative method – Original method with an adjustment based on filing enrollment. We used the number of covered lives from filings where available in the calculation of weights in addition to the original method. Results were similar to the original method. When using this method, not only are MEPS data on enrollment in the small group market used to allocate weights within a carrier’s business in the small group market, but enrollments from the filings are used as a within-carrier adjustment factor when available. Weights in the individual market for a carrier are divided equally among filings and then the adjustment factor based on enrollments from the filings is applied. As this approach comes closest to representing both the population and withincarrier distributions, this is the final method chosen and is fully described above.
- Alternative method – Original method minus the MEPS adjustment. Rather than using MEPS-IC data to distribute enrollment weights within a carrier/year, we assigned equal weights to all plans with filings. This method made little difference in national trends relative to the original method. Given this approach did not yield weights consistent with independent data on product distribution, it was rejected.
- Alternative method – Alternate method 3 with adjustment for PPO. In some states for the small group market, not all products were represented in the filings. Review of the individual filings suggested that carriers may have reported products as indemnity rather than as PPO. Given this potential reporting error, state product adjustments were derived assuming redefining indemnity and PPO filings as indemnity/PPO and deriving the state product adjustments using collapsed indemnity/PPO distributions. Under this approach, weights tended to be larger for indemnity filings, but resulting estimates reflected those obtained from Method 3. Given the similarity of estimates to those from Method 3 and given the lack of information on the true status of filings and the inconsistency in the indemnity/PPO confusion across state, this approach was not used for this analysis.
Methods 1, 3, 4, and 5 are all similar and based on the values shown in Initial Carrier Weight column of Table 6. They differ in the number and types of adjustments made to these weights. Method 3 uses MEP-IC and filing information, while method 4 ignores this information, and Method 5 uses collapsed indemnity/PPO sizes to derive the adjustment. Method 2, on the other hand, uses different information (filings enrollment data versus the NAIC enrollment data) from the other four methods as the basis of the weights. The primary reason for not considering Method 2 is that many observations are lost due to insufficient information in the filings. We opted to use Method 3 fully described above as it used the most information in determining the weights.
Inclusion and Exclusion Scenarios
This study examined differences between measures of interest for 2011 and earlier years for the small group and individual markets at both the state and national level. Because data were sparse in some states, there was concern that: a) some carriers received undue influence in both the state and national statistics for a given year; and b) inconsistency across years in the set of states for which filings were available may have affected study findings. Thus, potential sparse data exclusion rules to address these issues were identified and the impact on state and/or national estimates was determined. Ideally, we would like to include all data so as to provide tabulations that are as complete as possible relative to the avaialable information. Table 8 and the following detail the effects of different exclusion rules and utilizes weights created without a national level single filer adjustment:
- States with just one year of data in specific markets: Here the concern is that states with only one year of filings could adversely affect national level across-year comparions if measures for that state differ largely from the average of measures across the other states. As only one year of state data is available, there are no across-year state level comparisons that could be made. An exclusion rule would be to exclude states with just one year of data for a market.
- With this rule, one state-year (HI 2009) is excluded in the individual market and two stateyears (MA 2011, WI 2011) are excluded in the small group market. In both individual and small group markets, there were no statistically significant changes in premium increases each year, although the estimated national level of the 2011 rate change was 0.7 percentage points lower under this exclusion rule. Given no significant impact on the annual estimates were observed, it was determined not to use this exclusion rule.
- States with just one filing within a market for a given year: Here the concern is two-fold: a) that years with only one filing for a given state could adversely affect state-level comparions between 2011 and earlier years if the carrier submitting the one filing does not appropriately represent the full population of filings for that state-year; and b) the impact of these states adversely affects the national level estimate for the year. An exclusion rule would be to exclude state-years with just one filing for a market.
- With only one filing for a state-year, variance estimates for that state-year cannot be derived and thus this year cannot be compared to other years. Thus the consideration is solely at the national level. With this rule, thirteen state-years (CA 2009, HI 2009, ID 2011, KS 2010, MD 2009, MI 2010, NE 2009, OH 2008, OK 2008, RI 2008, RI 2009, RI 2010, SD 2008) are excluded in the individual and twelve (AL 2009, HI 2011, IA 2008, IA 2011,ID 2008, ID 2011, MN 2009, NE 2010, PA 2009, PA 2010, VA 2009, WA 2009) in the small group market. In the individual market, the estimated national premium increase changes significantly for two of the four years. The majority of these excluded filings are from carriers that constitute a small proportion (<50%) of the total state member-months from the 2010 NAIC. These same filings also tend to have larger premium increases, indicating their inclusion could introduce bias into the national estimates. However, one of the thirteen excluded filings in the individual market and two of the twelve in the small group market are filings for which the carrier constitutes >50% of the state member-months based on the 2010. To make use of all available data but to avoid undue influence of single-filer states, the exclusion rule was modified to include state-years with just one filing for a market but to adjust the weighting methodology so as to allow these single filers to represent only themselves. The estimates from this scenario are shown as 2a in Table 8, and show similar results to those from Option 2. Tests comparing the results for Option 2 and Option 2a found no significant differences (not shown). As this approach accounts for the undue impact of single filers and yet utilizes all the data, this is the scenario chosen for implementation.
- Combination of states with one year of data and states with one filing for the year in a given market: An exclusion rule would be to first exclude state-years with just one filing for a market and then exclude states with just one year of data for a market.
- This rule adds one additional state-year (KS 2009) in the individual market and four additional state-years (IA 2011, MA 2011, NE 2011, WI 2011) to the small group market than that described in scenario #2. The results are similar to scenario #2 for both the individual market and the small group market. In the individual market, two additional filings were excluded beyond those noted in scenario #2. The small group market, however, added 27 additional filings to the exclusion list beyond those noted in scenario #2
- States missing filings in 2008, 2009, 2010, or 2011: As with alternative 1, here the concern is that states with one or more years with missing data could adversely affect national level acrossyear comparions if measures for those states differ largely from the average of measures across the states that have data for every year. Simlar to alternative 1, if data are missing for a year, that year cannot be used for across-year state level comparisons. An exclusion rule would be to exclude states with no filing data in a market for one or more of the three years.
- With this rule seven states (California, Hawaii, Illinois, Kansas, Nebraska, Rhode Island, and South Dakota) in the individual market and nine states (Connecticut, Hawaii, Iowa, Massachusetts, Nebraska, Rhode Island, Virginia, Washington, and Wisconsin) in the small group market are excluded. These exclusions have no significant impact on estimated premium increases in the small group market, but yield significant differences in the individual market for 2009 and 2010. Not surprisingly, the exclusion list of states is similar to those listed in scenarios #2 and # 3, thus producing comparable estimates.
Table 8: Sensitivity Analysis of Average National Level Premium Increase
† Data for 2011 are incomplete.
* National level estimate is significantly different from Full Database by year by market at p < .05.
Our decision was to use scenario #2a. We believed that having one filing represent an entire state gave undue weight to one filing and can have an undue effect on national level estimates, but also wanted to include all data in the estimates. Therefore, the weighting methodology was modified to allow single filers to represent only themselves.
State Reporting – In reporting figures for individual states and markets, we do not display figures if filings constitute less than 50 percent of state enrollment for the year. However, all filings are included in the calculation of national figures, including states where enrollment was insufficient for state reporting.