As discussed in previous sections, NORC took multiple steps to ensure quality in the data collection phase of this project. Research analysts extracting and entering data on filings used an Access database for all steps of the data collection process, linking each step and creating meta-data on the process. Screen shot images were saved for every query to ensure that every available filing (at the time of query) was downloaded for review. During data entry, wherever possible, drop-down menus were used to ensure standardized capture of information. NORC reviewed a random sample of 100 filings, discussed in the prior section, for errors in data entry, and an error rate of approximately one percent was calculated. Subsequent meta-analysis of these errors found many were variables omitted when they were available in the filing or entered despite conflicting evidence (i.e. of another value) in the filing – both of these types are fundamentally due to judgment calls in reconciling information from different parts of the same filing document. A single filing pdf document may contain correspondence between the filing carrier and state regulators that spans several months, but individual pages are often undated, and it is not always clear which part of the filing indicates its final disposition.
Every effort was made to capture data for variables that were key to the analysis. If a key variable was noted as missing from the filing, additional investigation was done to confirm whether the data was truly missing from the filing. After building the final database, we performed additional checks on outliers. NORC re-examined observations with a premium rate increase of less than -10 percent or greater than 20 percent to look for and correct any data entry errors. Similarly, NORC also checked the observations showing an approved premium rate increase in excess of the requested rate increase to ensure the accuracy of the captured rate increase data.