NovaStar / Core Data Filing Process / Overview

• Introduction
• Interaction Between Programs

---

Introduction

This documentation describes the core data filing process that is run to load data into the NovaStar system and modify existing data.

A NovaStar system runs a variety of automated processes, including:

1. Single command line programs that start, perform tasks, and exit, for example:
   • data (re)validation
2. Scheduled programs where command line programs are run repeatedly on a schedule, for example:
   • data import and data export programs
3. Background system processes that run continuously, for example:
   • nsrecdata and other data collection programs
   • system processes that ensure that various NovaStar processes are running

Core data filing process steps are executed by the above software and involve programs that modify the NovaStar database contents, including data collection and data import. All of these programs run the same core data filing sequence, which is shown in the following diagram. See the Data data model documentation for background on data reports.

NovaStar Core Data Processes (see full-size image)

Each major data filing process step is described in separate documentation. Data records are typically inserted/updated at the end after all steps have been completed. TODO Is this true? I show the following as discrete but I assume that insert/update occurs at the end.

• Raw value insert/update - received from data collection or import
• Calibration - calculates raw/scaled values from calibration data
• Scaled value insert/update - computed from calibration
• Validation - evaluates data values and trends to assign data flags
• Rated values - computed from scaled values and ratings
• Alarms - computed from scaled values and alarm triggers

Interaction Between Programs

The above core data processes are executed in sequence by various NovaStar software applications, which use shared software layers (libraries and web services), as shown in the following figure.

NovaStar Core Data Filing Layers (see full-size image)

Most programs process data for specific stations and points and data for a point in a NovaStar system are typically only loaded in one way (e.g., ALERT data collection or import program). Consequently, the in-memory data for different points typically only exist in one program that is concerned with that point's data.

However, it is possible for the Administrator to edit data, and it may be necessary to reload historical data (e.g., from a log file or by requerying a web service after a network outage). In these cases, the real-time system may be inserting data for a point while historical data are also inserted or updated. Consequently, the data filing process re-queries needed input data in order to use calibrations and ratings (with effective times), and neighboring data reports to evaluate trends. TODO is this true - caching may be difficult.

TODO explain that if a data edit is made in the Administrator, then programs such as nsdbrecalibrate must be run (does the legacy and new Administrator do this?). Explain that manual data inserts/updates should be followed by running nsdbrecalibrate, etc. in sequence.