Development of a National Adult Protective Services Data System: Namrs Pilot Final Report (volume 2). 5. Namrs Pilot Case Loader

09/25/2015

The NAMRS Pilot Case Loader was an application that does processing on the Case Component XML files that were uploaded by state users. These files had to be validated against the NAMRS Case Component XSD file. State users should have validated the file against the XSD before they uploaded it to the NAMRS Pilot Website (although the file was validated as part of processing). If the file was very large the state user could zip the file and upload the zipped file.

Function

The Case Component Loader performed the functions on all files waiting to be processed:

  • Downloaded the file from NAMRS Pilot Storage to the local temp space.

  • Unzipped the file if the file was zipped.

  • Validated the file against the XSD. If it was not valid, saved an error message for the user and stopped.

  • Ran the data validation rules against the file (and performed actions on rule violations) and built a list of warning messages (which could be quite long) for the user. If the file was not valid (i.e., it had no investigations left at the end of validation), then added an error message for the user and stopped. After the file had been validated and any invalid date had been removed (the actions mentioned above), a new version of the file without any invalid tags/data was saved.

  • Loaded the data from the valid case XML file produced by the previous step into the database. This was the longest part of the process.

  • Calculated the basic counts that were displayed on the website.

  • Built the PDF files for the summary counts and frequency counts.

  • Saved the computed Key Indicators to the database.

The Case XML Loader was a "console application" that could be run from the command line by executing the following command: Namrs.WebJob.CaseXmlLoader.exe -- it did not take any parameters.

The Case XML Loader was built as a Microsoft Console Application. The site was written in C# and used Entity Framework and LINQ for all database access. The site used several open-source components and frameworks, including DotNetZip, iTextSharp, log4net, RazorEngine.

This application ran on an Azure VM. (It ran on the same VM that hosted the NAMRS Pilot Database.) It was executed as a Scheduled Task on the VM. It ran once a minute every day. So on average, it started processing an XML file within 30 seconds of the state user uploading the file. The scheduler was set so that it only ran one instance of the loader at a time.

The validation rules applied at the time of data entry of Agency Component data, validation rules applied at the time of data entry of Key Indicator data, and the validation rules applied on the Case Component XML file along with the action taken when invalid data were found--were all included as Appendix F in Volume 1.

Access

Since the Case XML Loader was a console application, it was not accessed by any other process (although one could say that it was accessed by the Task Scheduler, which runs it every minute).

The Case XML Loader accessed the following:

  • NAMRS Pilot Database--The loader queried the database to find which Case Component XML files were awaiting processing, to save the Case Component data from the XML files into the database, to save other statistics and computed counts into the database, etc.

  • NAMRS Pilot Storage--The loader downloaded the Case XML file from NAMRS Pilot Storage into a local disk so that it could be easily accessed (and unzipped if necessary).

  • NAMRS Pilot Email--The loader would send an email to the state user when it was finished processing the file.

Security

The NAMRS Pilot Case Component XML Loader was a command line application and did not accept any incoming connections or parameters. It ran on the same VM that hosted the NAMRS Pilot Database, so all the same security measures for that VM were in place for this process.

Configuration

The NAMRS Pilot Case XML Loader was configured using the NAMRS.WebJob.CaseXmlLoader.exe.config file, located in the root of the App. All parameters were documented in the config file. The configuration parameters were similar to the parameters for the website, although there were not as many.

View full report

Preview
Download

"NAMRSpilot-V2.pdf" (pdf, 1.83Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®