This study compared alternatives for implementing the National Institutes of Health (NIH) Research, Condition, and Disease Categorization (RCDC) System. RCDC is an information technology and document categorization system that uses the Collexis platform to automate and standardize the classification of research projects into specific research and disease categories, and to assist in reporting research expenditures to Congress and the public. The evaluators assessed the relative suitability of the current leading categorization packages by comparing them to the current RCDC/Collexis implementation. The categorization packages were evaluated using business-related criteria, functionality criteria, technical criteria and usability criteria. The evaluators also assessed the levels of agreement between the RCDC system and human raters, using 2005 and 2006 validity feedback provided by subject matter experts within the NIH.
Three of the five alternative categorization packages tested scored higher than Collexis on the business-related criteria, and scored equal to or higher than Collexis on the functionality criteria. In addition, two of the three scored higher on the technical criteria. Collexis scored highest, however, on the usability criteria. The median overlap fraction across categories was approximately 37%. The evaluators recommended further analysis of the validity and categorization process, however, before an alternative categorization package should be considered more feasible for NIH's research categorization.
Report Title: Alternatives for Implementing the National Institutes of Health's Research, Condition, and Disease Categorization System