NovaStar Reference / Data Formats / NovaStar Custom Specification

• Overview
• Specification File Details
  • Message Types
  • [General] Properties
  • [MessageType.General] Properties
  • [MessageType.ColumnN] Properties
  • Endianness Property Values
  • Column Name Property Values
  • Column Type Property Values
  • PointOrder Property
  • Date/time Format
    • Encode Date and Time as Two Integers
    • Encode Date and Time as One Integer
    • Encode Date and Time as Text using ASCII Format
  • Error Checking
• Standard NovaStar Properties
• Examples
  • Message Type: Sensor
    • Sensor Binary Encoding
    • Sensor ASCII Encoding using Hexadecimal
  • Message Type: MultiSensor
    • MultiSensor Binary Encoding
    • MultiSensor ASCII Encoding using Hexadecimal
  • nsrecdata Implementation Example
• Limitations
• Importing Data into NovaStar
• Exporting Data from NovaStar

---

Overview

The initial focus of this specification is binary-encoded transmissions (Encoding=Binary). ASCII encoding has been prototyped but requires additional testing.

The NovaStar Custom Specification format is used with the nsrecdata program to define custom data report formats. It is intended to support "flat" messages encoded in a line of text or a sequence of binary values, which is typical for data collection. Multiple messages can be included in a "transmission" and one or more data reports can be included in a message, as illustrated below:

  Transmission (`nsrecdata` is typically configured to listen for pushed transmissions)
    Header (optional header, for example for Iridium)
    Message
      Mesage type
      Data values
      End of message (optional, for example \n for ASCII encoding)
    Message
      Mesage type
      Data values
      End of message

Using this data format and specification file is more complex than the simple nsrecdata -G#,#,# approach, which is limited to timestamp, point identifier, and value in each message.

The specification focuses on data payload and does not include specific handling of communication layers. In contrast, the ALERT2 specification is complex and includes data elements that represent communication and data packaging layers. Consequently, the specification is suitable for common applications that do not require the communication properties of ALERT2 messages.

Data transmissions over IP (or data loaded from a file) consist of one or more data messages, each of which contain encoded data. The specification file indicates the encoding format for data in messages and allows for custom message formats for different hardware. The goal of the design is to allow the specification file to define data formats without requiring additional software enhancements.

A message specification is typically defined to optimize communication bandwidth and minimize latency. For example, messages for single sensor (a "sensor" message type) contain the timestamp, point identifier, and value and are appropriate for periodic transmissions and cases where bandwidth throughput and cost is not an issue.

If multiple sensor values are available at a station, using a "multi sensor" message will transmit the timestamp with multiple values and therefore save bandwidth. However, sending "multi sensor" messages too frequently, such as when some sensor values do not change, can waste bandwidth and single sensor messages may be better. These design issues should be evaluated based on the requirements for each implementation and the frequency that each message type is used. The specification allows multiple message types to be defined, which allows mixing of messages from a transmitter to optimize performance.

The message type at the start of each message indicates to the decoding software such as nsrecdata how to decode the message, using specification file properties for each message type. Multiple message types can be transmitted in sequence, typically corresponding to different station configurations and transmissions for different conditions, including weather events and station status or summary check-in reports.

Use the nsrecdata -Gcustomspec --customspec=file command parameters to indicate that a custom specification file is used. The specification file typically exists in the /usr/ns/cus folder on the NovaStar server in production systems. The specification file format is described in the remainder of this document. Version control can also be used to track changes in the specification file. It is typical to name the file with .cfg extension indicating that it is a configuration file.

Specification File Details

The specification file (used with nsrecdata --customspec command parameter) is a text file that follows the conventions of an INI file, using # for comments. The format uses [Section] to group properties into a section, and Property = Value syntax to define properties. Property values can be surrounded by double quotes if necessary to ensure that whitespace and other characters are included. This format is easily parsed by programming languages, which can search for sections and properties in a section. An equivalent format, such as JSON, may be implemented in the future.

Currently, the specification file must be edited with a text editor and copied to the /usr/ns/cus folder. Restart the nsrecdata process to read the file if the specification file changes.

The different file sections are used to specify related blocks of specification properties, as described in the following documentation.

General constraints for the specification include:

• A data transmission's encoding must be consistent throughout the transmission. The following encodings are typical:
  • Binary (Encoding=Binary) - initial focus of this specification
  • ASCII delimited (for example: Encoding=ASCII,Delimiter=,) - prototyped but not fully tested
  • ASCII hex - protototyped but not fully tested
• All times are reported in UTC (not local time). Support for time zone can be added; however, this requires coordination of configuration on stations and NovaStar database. Using UTC ensures a common clock.
• Global general properties that apply throughout the specification, such as global default property values, are specified in the [General] section.
• Properties for each message type are specified using a section name that starts with the message type and a period:
  • This allows the specification for multiple message types to be described in one specification file.
  • General properties for a message type are specified in a section similar to [MessageType.General], for example [Sensor.General].
  • Column properties for a message type are specified in a section for each numbered column similar to [MessageType.Column1], for example [Sensor.Column1].
  • Default column properties for a message type are specified in a section with wildcard similar to [MessageType.Column*], for example [Sensor.Column*]. For example, a multi-sensor message type may have a variable number of data values and each data column may have similar message format.

Message Types

Message types allow defining multiple data encodings based on requirements. When properly configured, software such as nsrecdata requires minimal knowledge about the messages because the specification file defines the software behavior. Although standard specifications are recommended for consistency, the design allows for highly-customized formats to meet specific requirements. It is recommended that messages types are consitent within a system, for example use the same message types for all stations, if possible.

The message type is assumed to be transmitted as a one byte unsigned integer at the start of the message. This value is cross-referenced with the MessageTypeNumber property in the specification, which then allows all other specification properties to be determined. Consequently, new message types with different data configurations and encodings can generally be enabled without changing the software.

The following table summarizes standard message types that are recommended for use and are described in this documentation. The text MessageType is used for documentation and readability (e.g., MessageType=Sensor) whereas the numeric MessageTypeNumber is used internally by software to cross-reference specification properties (e.g., MessageTypeNumber = 1 (e.g., MessageTypeNumber = 1)). The initial implementation of the specification focuses on optimizing sensor and multi-sensor data reports, which requires consideration of how timestamp, point identifier, and other data are handled. See the message general properties for an explanation of how MessageType is used.

The following message types are recommended based on common use in a system, which is to define single sensor and multi-sensor message types consistent with system requirements, communication bandwidth optimization, and station logger programming.

It is recommended that message types and their specification are defined as simply as possible. For example, use a 4-byte integer for data values for all columns rather than mixing types. A simple specification helps to minimize configuration errors.

MessageType Property Values

MessageType	MessageTypeNumber	Description
Sensor	1	Single sensor data report, typically with NovaStar point identifier included.
MultiSensor	2	Multi-sensor data report, which may or may not include the NovaStar point identifier. If not included, it must be determined from the NovaStar database based on data number.
		Other message types can be added as necessary, for example to receive and log station metadata.

The specification is flexible to allow including point identifier and other properties in the messages, or looking up point properties from the database. If multiple message variants are needed for a system, specific message types can be defined, for example:

• Variants on Sensor:
  • Sensor - simple message with point identifier included in the message (this is the focus of the initial specification)
  • SensorNoPointId - look up the point identifier from the database rather than including in the message (smaller message size, but more configuration in the database)
• Variants on MultiSensor:
  • MultiSensor - may or may not include point identifier in the message data (message without point identifier is the focus of the initial specification)
  • MultiSensorWithPointId - define message types that include point identifier in the message data (larger message size, but less configuration in the database)
  • MultiSensorNoPointId - look up the point identifier from the database rather than including in the message (smaller message size, but more configuration in the database)

[General] Properties

General properties provide general information and defaults for message column properties. Recognized properties are listed in the following table. Property names are case-specific to ensure exact match lookup; however, property values are case independent unless otherwise indicated.

[General] Section Properties

Property	Description	Default
Delimiter	The delimiter used to separate columns when used with Encoding=ASCII. This property is the default for all message types. Supported values are: None - no delimiter, which requires that Hex format be used for Encoding=ASCII to handle fixed-format width column data , (or other non-whitespace character) - messages will have variable length defined by delimited values	None - for Encoding=Binary , - comma, for Encoding=ASCII
Encoding	The overall encoding of the transmission and messages within the transmission: Binary - data from the transmitting station are written in binary form and require decoding in order to view in human-readable form. Binary encoding is the initial focus this specification. Log messages print binary data in hex and decimal format for troubleshooting. ASCII - data from the transmitting station are written in ASCII (text) form, typically using delimited format but may contain hexadecimal codes. ASCII encoding has been prototyped but not fully tested for production use.	ASCII (in anticipation of ASCII delimited format being the default due to ease of use)
Endianness	The order of bytes when Encoding=Binary and the order of Hex bytes in message data for multi-byte values for Encoding=ASCII. This property is the default for all message types. Supported values are: Big Little See the Endianness Properties documentation for more information. Note that the order of data bytes in messages may be different than the endianness of the encoding or decoding operating systems.	Big - results in more readable left to right messages
Header	If specified, each transmission is assumed to start with a header that is read before messages are decoded. Possible values are: None Iridium - read Iridium binary header (see the "MAN0025 Iridium Short Burst Data Service Developers Guide).	None
MessageSeparator	The character(s) that indicate the end of one message and start of another message. Encoding=ASCII data messages that are separated by Linux line feed (newline) that can be read with software "read line" functions. Any empty messages due to duplicate newline characters are ignored. None - no message separator LF - which translates to software \n character and is the same as the ASCII LF character. Carriage return CR (\r) characters are ignored. See also the Error Checking section.	None for Encoding=Binary NL for Encoding=ASCII

[MessageType.General] Properties

These properties describe general properties for a message type, for example in a [Sensor.General] section. The properties can be used as defaults so that fewer properties need to be specified for columns. Property names are case-specific to ensure exact match lookup; however, property values are case independent unless otherwise indicated.

[MessageType.General] Section Properties

Property	Description	Default
Delimiter	The delimiter used to separate columns for the message, will override the [General] section Delimiter property. Supported values are: None - no delimiter, which requires that Hex format be used for fixed-format width column data , (or other non-whitespace character) - messages will have variable length defined by delimited values	Most specific of: Global default if specified None
Endianness	The order of bytes when Encoding=Binary and the order of Hex bytes in message data for multi-byte values for Encoding=ASCII. This property is the default for encoding columns for the message and will override the [General] section Endianness property. Supported values are: Big Little See the Endianness Properties documentation for more information. Note that the order of data bytes in messages may be different than the endianness of the encoding or decoding operating systems.	Most specific of: Global default Endianness if specified Big
Format	The use of this property is being evaluated. The format of the data value, as the default for all columns for a message type: ASCII - ASCII (text) characters, limited to printable characters, requires that Delimiter is specified Hex - hexadecimal values in sequence with two characters for each byte, for example 4 hex values for a 4-byte integer	Not required for Encoding=Binary Hex - if Delimiter=None ASCII - if Delimiter is not None
MessageType required	The message type as indicated in the Message Types documentation. The message type must agree with the first part of the section, for example MessageType = Sensor for section [Sensor.General]. The property facilitates processing by software and improves readability of the specification file.	None - must be specified.
MessageTypeNumber required	The message type number as indicated in the Message Types documentation. The number used in the specification must be consistent with the software that encodes the messages. The MessageTypeNumber is transmitted in messages as an integer and is critical for interpreting the specification.	None - must be specified.

[MessageType.ColumnN] Properties

These properties describe each column for a message type. Property names are case-specific to ensure exact match lookup; however, property values are case independent unless otherwise indicated. The following are typical:

• [Sensor.Column1], [Sensor.Column2], etc.:
  • Define single sensor fixed columns at the start of messages, for example for the message type, date/time, point identifier, and data value columns.
• [MultiSensor.Column1], [MultiSensor.Column2], etc.:
  • Define multi-sensor fixed columns at the start of messages, for example for the message type, date/time, and station identifier columns.
• [MultiSensor.Column*]:
  • Define multi-sensor data columns after fixed columns, which will be processed in sequence until the end of the message is detected, for example for the point identifier or order and data value columns.

Message columns are separated by a delimiter if the general message property Delimiter is specified. If Delimiter=None, the columns must be formatted using an encoding that does not require a delimiter:

• For Encoding=Binary and no delimiter:
  • The columns must contain values of the correct type and endianness.
• For Encoding=ASCII and no delimiter:
  • A data column with type of Integer4 must be represented by 4 bytes as 4 hex values, each of which is a 2-character string.
  • A hex value of 00 corresponds to a decimal value of 0 and a hex value of ff corresponds to an unsigned one byte integer decimal value of 255.
  • The Endianness of the data message indicates the order of the hex values so that they can be properly decoded into the software's in-memory variables.

Decoding software should determine column properties as follows:

1. Use the number section (e.g., [MultiSensor.Column1]).
2. Use the wildcard default for the message type (e.g., [MultiSensor.Column*]).
3. Use the appropriate [General] property.

The following are recognized column properties, which must be handled by NovaStar decoding software in order to file data reports. Property names are case-specific to ensure exact match lookup; however, property values are case independent unless otherwise indicated.

[ColumnN] and [Column*] Section Properties

Property	Description	Default
DateFormat	A string that allows decoding text or integer date into its parts, for example using format YYYYMMDD, case-sensitive. This format is applied after initial decoding occurs. For example, a column identified with Name=ReportDate may contain a binary or ASCII integer that must be decoded into a date using DateFormat. See also DateTimeFormat and TimeFormat.	YYYYMMDD
DateTimeFormat	A string that allows decoding text or integer date into its parts, for example using format YYYYMMDDhhmmss, case-sensitive. This format is applied after the initial decoding occurs. For example, a column identified with Name=ReportDateTime may contain a binary or ASCII integer that must be decoded into date and time using DateTimeFormat. See also DateFormat and TimeFormat.	YYYYMMDDhhmmss
Description	A description of the column, for information purposes to avoid confusion.
Divisor	Can use if point calibration multiplier is not used. An integer divisor (must be positive) to be applied to the value after decoding, for example to convert a transmitted integer value into floating point value for the NovaStar database. A divisor is necessary if a measured floating point value is encoded as an integer in the message. The divisor provides a way to indicate how many digits after the decimal point are encoded. For example, air temperature data with a range -NNN.NN to NNN.NN would require the station to multiply all values by 100 to create an integer and then use Divisor=100 to recreate the floating point number. For example, a measured floating point value of 25.12 would become an integer 2512 for transmission and then would be converted to 25.12 to load into the database. Note that the divisor is different than the NovaStar "calibration" adder and multiplier factors, which are used to convert raw measurement into scaled (engineering unit) values. The NovaStar point table does not include a specific column for this value and therefore the divisor must be specified as one of: an integer - however, this only works for simple systems with no variability in divisor for different station's points ${ns.point.description:Divisor} - use the property named Divisor that is defined in the point description with Divisor=value - currently not supported ${ns.point:data_parameter} - use the NovaStar point table data_parameter column, where the database point is determined from PointOrder (data_parameter is typically used for station polling) - recommended and may be required in the future ${ns.point.sensor_id} - use the NovaStar point table sensor_id, where the data is determined from PointOrder (sensor_id is typically used for a data import) - not recommended and may not be supported in the future ${ns.point:data_position} - use the NovaStar point table data_position column, where the database point is determined from the PointOrder (data_position is typically used for a data import) - not recommended and may not be supported in the future	1 (no change in the transmitted value) See the PointOrder Property documentation for more information.
Endianness	The order of Hex bytes as transmitted, which will be decoded appropriately for the decoding software.	Most specific of: Global Endianness property value if specified Big
Format	The use of this property is being evaluated. The format of the data value: ASCII - ASCII (text) characters, limited to printable characters, requires that Delimiter is specified Hex - hexadecimal values in sequence with two characters for each byte, for example 4 hex values for a 4-byte integer	Most specific of: Not required for Encoding=Binary Format from [MessageType.General] property Hex - if Delimiter=None ASCII - if Delimiter is not None
Name required	The column name, which corresponds to known values when decoding and allows the software to load data into the NovaStar database. See the Column Name Property Values table below for more information.	None - must be specified.
PointOrder	Point order for MultiStation message type, used to look up PointNumId using NovaStar data because the message does not contain the PointNumId. Specify as: ${ns.point:data_position} - use the NovaStar point table data_position column value, where 1 corresponds to the first data value decoded - recommended and may be required in the future ${ns.point:data_parameter} - use the NovaStar point table data_parameter column value, where 1 corresponds to the first data value decoded - not recommended and may not be supported in the future ${ns.point.sensor_id} - use the NovaStar point table sensor_id column value, where 1 corresponds to the first data value decoded - not recommended and may not be supported in the future See the PointOrder Property documentation for more information.	None - must be specified.
TimeFormat	A string that allows decoding text or integer time into its parts, for example using format hhmmss, case-sensitive. This format is applied after initial decoding. For example, a column identified with Name=ReportTime may contain a binary or ASCII integer that must be decoded into a time using TimeFormat. See also DateFormat and DateTimeFormat.	hhmmss
Type required	The column type, which indicates the primitive data type for the data value in the column. Initial decoding will occur using Format and Endianness and additional formats may be specified to allow further decoding, such as DateFormat, TimeFormat, and DateTimeFormat to decode the date/time. See the Column Type Property Values table below for available options.	None - must be specified.
ValueCount required for Binary format	Count of data value columns, necessary for binary format messages to know how many values are part of a message. ASCII delimited messages use the MessageSeparator property to specified the message separator character (typically a newline) and the ValueCount is not needed.	Must be specified for Binary format.

Endianness Property Values

The following table lists possible values for the Endianness property, which indicate how encoded multi-byte hex values should be decoded. Property names are case-specific to ensure exact match lookup; however, property values are case independent unless otherwise indicated. The Endianness property can be defined in the [General] section to apply for all message types or can be defined in message type sections for granular control of encoding.

It is recommended that transmitting software use the same endianness for all encoding in order to avoid complex specifications that may be prone to errors.

Endianness Property Values

Endianness	Description
Big	Multi-byte hex values in the message are shown in left to right order with the byte on the left being the most significant. This is similar to the order of decimal numbers. The left-most byte contains the sign, if used. This endianness is used for the default in this specification because it is often used for network data transmissions and reads left to right when viewed in text form.
Little	Multi-byte hex values in the message are shown in left to right order with the byte on the left being the least significant. The right-most byte contains the sign, if used.

Endianness is important when formatting integers in byte representations. Endianness indicates the order of bytes for multi-byte values. See Endianness on Wikipedia. Encoding is also impacted by whether an integer is signed. Most computers use two's complement (see Wikipedia article) to indicate sign. Two's complement signed values do not result in only the signed bit being set and all bytes that form a value must be shifted appropriately when decoding the value.

• Big endian format numbers are shown with the most significant byte at the lowest (leftmost) address when written as text.
• Little endian format numbers are shown with the least significant byte at the lowest (leftmost) address when written as text.
• The 8 bits within a byte are always represented with the most significant bit on the left, regardless of whether big or little endian
• Normal numbers are written with most significant digits on the left, similar to big-endian format.
• The leftmost bit in the most significant byte is used for the sign for data types that support sign, such as signed integers.

An unsigned one byte integer is able to store a maximum value of 255, as shown below.

1 1 1 1 1 1 1 1
| | | | | | | |
| | | | | | | +- =   1
| | | | | | +--- =   2
| | | | | +----- =   4
| | | | +------- =   8
| | | +--------- =  16
| | +----------- =  32
| +------------- =  64
+--------------- = 128
           Total = 255

The next significant byte for multi-byte values will continue a similar pattern with larger numbers 256, 512, etc. Rather than representing a byte in binary representation, a two-character hexadecimal value can be used. For example ff in an unsigned one-byte integer is equivalent to 8-bits set to 1 or decimal 255.

A four byte unsigned integer has a value range of 0 to 4,294,967,295, whereas a signed integer has a value range of -2,147,483,648 to 2,147,483,647.

A decimal value of 25000 has a big-endian hexadecimal value of 000061a8, meaning that the byte values from left to right are (00, 00, 61, a8).

The same number in little-endian format is the reverse order (a8, 61, 00, 00).

The decoding computer system must ensure that data value formatting is decoded consistent with the endianness of the in-memory integer.

Column Name Property Values

The following table lists possible values for column Name property. Property names are case-specific to ensure exact match lookup; however, property values are case independent unless otherwise indicated. The property value is interpreted by software to know how to file data in the NovaStar database.

Name Property Values

Name	Description
PointNumId	The NovaStar point numeric identifier, which corresponds to the NovaStar point table point_numid column. Once matched, the point table id is used for NovaStar data table column point_id.
ReportDateTime	The date and time corresponding to the data measurement, used for the NovaStar database data* table report_time. The report time can alternatively be provided using ReportDate and ReportTime. See the Date/time Format section.
ReportDate	The date corresponding to the data measurement, used with ReportTime to create the report time when ReportDateTime is not specified. See the Date/time Format section.
ReportTime	The time (for a date) corresponding to the data measurement, used with ReportDate to create the report time when ReportDateTime is not specified. See the Date/Time Format section.
StationNumId	The station numerical identifier, for MultiSensor message type, used to match the NovaStar station table numid in order to determine associated point table records.
ValueRaw	Raw sensor value, which will be loaded into the NovaStar data table raw_val value. The scaled_val value is calculated using the point's calibration data.
ValueScaled	Scaled sensor value, which will be loaded into NovaStar data table scaled_val value. The raw_val value is calculated using the point's calibration data.

Column Type Property Values

The following table lists possible values for column Type property, which defines the encoded column value. For example, the type may indicate that a 4-byte integer is used for the data value, represented as 4 hex codes. Integer types are generally signed by default unless indicated as unsigned, wheres characters are typically unsigned by default. The primitive type is shown for comparison (see C data types). Property names are case-specific to ensure exact match lookup; however, property values are case independent unless otherwise indicated.

Integers are often the primary types for transmitting data. Transmitted floating point values can be multiplied by a factor on the station, transmitted, and then divided by the factor during decoding, using one of the following approaches:

1. Use the point's calibration multiplication factor to convert transmitted integer to floating point value (e.g., scale by 100 on station and by .01 in calibration).
2. Specify the Divisor in the custom specification to convert the integer to a floating point number with digits after the decimal point (e.g., 100 if the value was scaled by 100 on the station).

Decoding software may store all decoded values in memory as long, double, or other suitable data type that is ensured to not lose precision, given that consuming software generally is not constrained to small types.

Type Property Values

Type	Description	Primitive Type	Range
Char	Single 1-byte character. Not currently supported.	char	See ASCII character set.
Char[N]	Array of 1-byte characters. Use Char[] for ASCII format where strings are of variable length. The string should contain a trailing NULL character if less than the array size. Not currently supported.	char[]	See ASCII character set, any length is supported.
Char2	Unicode 2-byte character - Not currently supported.	Depends on software language.
Float4	IEEE 754 4-byte signed floating point number. Not currently supported.	float	Approximately +- 10^38
Float8	IEEE 754 8-byte signed floating point number. Not currently supported.	double	Approximately +- 10^308
Integer1	1-byte signed integer.	byte, signed char	-128 to 127
UInteger1	1-byte unsigned integer.	unsigned char	0 to 255
Integer2	2-byte signed integer.	short	-32,768 to 32,767
UInteger2	2-byte unsigned integer.	unsigned short	0 to 65,535
Integer4	4-byte signed integer.	integer	-2,147,483,648 to 2,147,483,647
UInteger4	4-byte unsigned integer.	unsigned integer	0 to 4,294,967,295
Integer8	8-byte signed integer.	long	-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
UInteger8	8-byte unsigned integer.	unsigned long	0 to 18,446,744,073,709,551,615
	Other types can be added as necessary.

PointOrder Property

The specification for multi-sensor messages typically includes the station identifier (Name=StationNumId column property) and column properties necessary to decode multiple data columns for the station. A point identifier is necessary to file the data from the message column for the appropriate point in the database. The Divisor must also be specified if the data value is scaled in the message. Including a point identifier and divisor for each data value in the message would increase the size of the message and therefore an alternative strategy is typically used, as described below.

1. The order of data transmitted for the station must be known, for example 1) rain sensor bucket tip count, 2) water level, 3) air temperature, 4) battery. This configuration occurs at the station.
2. These values correspond to the data columns in a message, which are counted after the fixed columns (e.g., after MessageTypeNumber, ReportDate, ReortTime, StationNumId). The data position is counted as 1, 2, 3, 4, ... , which is different than the column number in a message (due to other columns at the start of a message).
3. The NovaStar database record for each point should be configured accordingly:
   1. Use a consistent combination of the point table data_position and optionally data_parameter to hold the message data position and divisor. The sensor_id is also an option but is not currently recommended for use and may be prohibited in the future.
   2. For example, decide that data_position will be used for the data position (1, 2, 3, ...).
   3. There is no database column for "divisor", which is why one of the above columns must be used instead. For example, use data_parameter for the divisor value and set accordingly for the data value for the point. For example, set to 100 if the data value was multiplied by 100 before transmitting. Alternatively, use the point's calibration multiplication factor, which will result in the factor representing the transmission scaling as well as any other scaling that is required.
4. Define the specification to describe the above. For example, if data column configurations are the same for all data columns, the specification file might look like the following:

[MultiSensor.General]

MessageType = MultiSensor
MessageTypeNumber = 2

[MultiSensor.Column1]

Description= "Message type number"
Name = MessageTypeNumber
Type = Integer1

[MultiSensor.Column2]

Description = "Report date"
Name = ReportDate
Type = Integer4
DateFormat = MMDDYYYY

[MultiSensor.Column3]

Description = "Report time"
Name = ReportTime
Type = Integer4
TimeFormat = hhmmss

[MultiSensor.Column4]

Description = "NovaStar station numeric identifier"
Name = StationNumId
Type = Integer4

[MultiSensor.Column*]

Description = "Point data raw value(s)"
Name = ValueRaw
Type = Integer4
# A value of data_parameter=1 in the database corresponds to the first data value column in the message.
PointOrder = ${ns.point:data_position}
# After the point is found using 'PointOrder', use the 'data_parameter' for the divisor value, if `Divisor` is used.
Divisor = ${ns.point:data_parameter}

Decoding software such as nsrecdata should handle the above configuration as follows, for each data column that is processed:

1. Select all points that match the given StationNumId.
2. For the data column of interest (for example 1 for the first data position), match the PointOrder value. For example, for the above example, find a station's point that has data_position=1. This point will be used to file the first data value.
3. If Divisor is used, once the point has been found, determine the divisor from the data_parameter column, which is applied to compute the data value to insert into the database.
4. File the data report.

If no matching point is found, the data report will not be filed. A warning should be logged and the data should be written to the log file in case it needs to be reloadd later. If a sensor is removed from the transmitting station, the data position will shift to the left and NovaStar database data_position values will need to be adjusted accordingly.

Date/time Format

Data reports typically have a report time in UTC that is the time of measurement or the inteval-ending time of a computed interval statistic value (e.g., mean over an interval). The NovaStar system expects data messages to include a time and therefore if only date is used for a report, the time parts will default to zero. The following examples illustrate options for encoding date and time for common cases.

Encode Date and Time as Two Integers

Separate integers can be used to encode date and time. For example the date/time 2022-02-08T01:02:03 can be represented with integers 20220208 and 10203. If Format = Hex (specified in the global or message General properties), the following specification properties should be used (for example). The integers as hex codes are first decoded to integers. The parts of the integer are then parsed into the report date and time based on the DateFormat and TimeFormat properties.

[Sensor.Column2]

Description = "Report date"
Name = ReportDate
Type = Integer4
DateFormat = YYYYMMDD

[Sensor.Column3]

Description = "Report time"
Name = ReportTime
Type = Integer4
TimeFormat = hhmmss

Encode Date and Time as One Integer

One integer can be used to encode the date and time. For example the date/time 2022-02-08T01:02:03 can be represented as a fourteen-digit integer 2022020810203. If Format = Hex (specified in the global or message General properties), the following specification properties should be used (for example). The integer as hex codes is first decoded to a long integer. The parts of the long integer are then parsed into the report date and time based on the DateTimeFormat.

[Sensor.Column2]

Description = "Report date and time"
Name = ReportDateTime
Type = Integer8
DateTimeFormat = YYYYMMDDhhmmss

Encode Date and Time as Text using ASCII Format

Text can be used to encode the date and time. For example the date/time 2022-02-08T01:02:03 can be transmitted as shown. The original string is first decoded into characters (simple copy). Then the text is parsed to determine the date and time parts. A space can be used in the DateTimeFormat to ignore the T between date and time.

[Sensor.Column2]
Description = "Report date and time"
Name = ReportDateTime
Type = Char[]
Format = ASCII
DateTimeFormat = "YYYY-MM-DD hh-mm-ss"

Error Checking

Error checking is important to ensure that data are properly decoded and filed in the system. The NovaStar system implements error checking for each point and will mark out of range values as questionable. It is more difficult to detect values that are in range but are extreme, such as rapidly increasing water level. In these cases, the system tends to report all data so that real-time emergency management decisions have as much data as possible, and quality control may be required on data when time is available.

The code that parses this specification should implement basic checks where possible, including:

• message length:
  • ASCII delimited transmissions should use MessageSeparator to control the end of one message and the start of the next message
  • binary transmissions should use ValueCount to indicate the number of data values to read, and binary messages must adhere
• decoded numerical values should be valid numbers
• detect extra data columns that are not configured, such as extra columns in ASCII delimited messages

Problems in data should be indicated in the log file using clear messaging such as ERROR or WARNING.

Standard NovaStar Properties

The custom specification allows integration of various data collection technologies with NovaStar. The configuration of the transmitting station coupled with the specification file allows data in messages to be inserted into NovaStar. The Column Name Property Values table describes how message column data is mapped to NovaStar database table values.

Examples

The following examples illustrate contents of messages and specification file contents for the corresponding message type. A specification file should contain as many sections necessary to define general properties and properties for required message types.

General

General properties provide general information and defaults for message and column properties.

The following is used for binary encoding using big-endian byte order:

[General]

Encoding = Binary
Endianness = Big

The following is used for ASCII encoding using comma-delimited data and big-endian byte order. Note that comma is the default delimiter for ASCII encoding and big endian is also the default.

[General]

Encoding = ASCII
Delimiter = ","
Endianness = Big

Message Type: Sensor

Sensor Binary Encoding

The following specification file properties would be used for Sensor message type that transmits data similar to the following. If necessary, include an appropriate Header property in the main [General] section.

MessageTypeNumber = 01 = 1 = Sensor

ValueCount = 1

ReportDate 4052021 = 04/05/2021

ReportTime 111940 = 11:19:40

PointNumId 99910, a precipitation point for station 99910

ValueRaw = 1, raw count of 1 tip

# Example of specification file contents for 'Sensor' message type.
# These properties can be appended to a specification file after the [General] section.

[Sensor.General]

MessageType = Sensor
MessageTypeNumber = 1

[Sensor.Column1]

Description= "Message type number"
Name = MessageTypeNumber
Type = Integer1

[Sensor.Column2]

Description= "Data value count"
Name = ValueCount
Type = Integer1

[Sensor.Column3]

Description = "Report date"
Name = ReportDate
Type = Integer4
DateFormat = MMDDYYYY

[Sensor.Column4]

Description = "Report time"
Name = ReportTime
Type = Integer4
TimeFormat = hhmmss

[Sensor.Column5]

Description = "NovaStar point numeric identifier"
Name = PointNumId
Type = Integer4

[Sensor.Column6]

Description = "Point raw data value from sensor"
Name = ValueRaw
Type = Integer4
# Optional Divisor will vary between points so get from the database.  Can also use point calibration.
Divisor = ${ns.point:data_parameter}

Sensor ASCII Encoding using Hexadecimal

The use of Format=Hex is being evaluated.

The following specification file properties would be used for Sensor message type that transmits data similar to the following (spaces have been added for illustration but are not included in the actual message):

01 00 3D D4 35 00 01 B5 44 00 01 86 46 00 00 00 01

MessageTypeNumber = 01 = 1 = Sensor

ReportDate 00 3D D4 35 = 4052021 = 04/05/2021

ReportTime 00 01 B5 44 = 111940 = 11:19:40

PointNumId 00 01 86 46 = 99910, a precipitation point for station 99910

ValueRaw 00 00 00 01 = 1, raw count of 1 tip

# Example of specification file contents for 'Sensor' message type.
# These properties can be appended to a specification file after the [General] section.

[Sensor.General]

MessageType = Sensor
MessageTypeNumber = 1
Delimiter = None
Format = Hex

[Sensor.Column1]

Description= "Message type number"
Name = MessageTypeNumber
Type = Integer1

[Sensor.Column2]

Description = "Report date"
Name = ReportDate
Type = Integer4
DateFormat = MMDDYYYY

[Sensor.Column3]

Description = "Report time"
Name = ReportTime
Type = Integer4
TimeFormat = hhmmss

[Sensor.Column4]

Description = "NovaStar point numeric identifier"
Name = PointNumId
Type = Integer4

[Sensor.Column5]

Description = "Point raw data value from sensor"
Name = ValueRaw
Type = Integer4
# Optional Divisor will vary between points so get from the database.  Can also use point calibration.
Divisor = ${ns.point:data_parameter}

Message Type: MultiSensor

MultiSensor Binary Encoding

The following configuration file properties can be used for MultiSensor message type that transmits data similar to the following. If necessary, include an appropriate Header property in the main [General] section.

MessageType = 2 = MultiSensor

ValueCount = 5

ReportDate = 4292021 = 04/29/2021

ReportTime = 171250 = 17:12:50

StationNumId = 7470, Washout Gulch at Hwy 14 station setup in the WET GCP

ValueRaw = 11, raw count of 11 tips for point 747

ValueRaw = 177, master stage 1.77 ft for point 7473

ValueRaw = 266, remote stage of 2.66 ft for point 7474

ValueRaw = 1226, master battery of 12.26 volts for point 7475

ValueRaw = 1222, remote battery of 12.22 volts for point 7476

# Example of specification file contents for `MultiSensor` message type.
# These properties can be appended to a specification file after the [General] section.

[MultiSensor.General]

MessageType = MultiSensor
MessageTypeNumber = 2

[MultiSensor.Column1]

Description= "Message type number"
Name = MessageTypeNumber
Type = Integer1

[MultiSensor.Column2]

Description= "Data value count"
Name = ValueCount
Type = Integer1

[MultiSensor.Column3]

Description = "Report date"
Name = ReportDate
Type = Integer4
DateFormat = MMDDYYYY

[MultiSensor.Column4]

Description = "Report time"
Name = ReportTime
Type = Integer4
TimeFormat = hhmmss

[MultiSensor.Column5]

Description = "NovaStar station numeric identifier"
Name = StationNumId
Type = Integer4

[MultiSensor.Column*]

Description = "Point data raw value(s)"
Name = ValueRaw
Type = Integer4
# A value of data_position=1 in the database corresponds to the first data value column in the message.
PointOrder = ${ns.point:data_position}
# If Divisor is used, after the point is found using 'PointOrder', use the 'data_parameter' for the divisor value.
# Divisor is not needed if the point uses the NovaStar calibration to scale the data value.
Divisor = ${ns.point:data_parameter}

MultiSensor ASCII Encoding using Hexadecimal

The use of Format=Hex is being evaluated.

The following configuration file properties can be used for MultiSensor message type that transmits data similar to the following.

02 00 41 7D B5 00 02 9C F2 00 00 1D 2E 00 00 00 0B 00 00 00 B1 00 00 01 0A 00 00 04 CA 00 00 04 C6

MessageType 02 = 2 = MultiSensor

ReportDate 00 41 7D B5 = 4292021 = 04/29/2021

ReportTime 00 02 9C F2 = 171250 = 17:12:50

StationNumId 00 00 1D 2E = 7470, Washout Gulch at Hwy 14 station setup in the WET GCP

ValueRaw 00 00 00 0B = 11, raw count of 11 tips for point 747

ValueRaw 00 00 00 B1 = 177, master stage 1.77 ft for point 7473

ValueRaw 00 00 01 0A = 266, remote stage of 2.66 ft for point 7474

ValueRaw 00 00 04 CA = 1226, master battery of 12.26 volts for point 7475

ValueRaw 00 00 04 C6 = 1222, remote battery of 12.22 volts for point 7476

# Example of specification file contents for `MultiSensor` message type.
# These properties can be appended to a specification file after the [General] section.

[MultiSensor.General]

MessageType = MultiSensor
MessageTypeNumber = 2
Delimiter = None
Format = Hex

[MultiSensor.Column1]

Description= "Message type number"
Name = MessageTypeNumber
Type = Integer1

[MultiSensor.Column2]

Description = "Report date"
Name = ReportDate
Type = Integer4
DateFormat = MMDDYYYY

[MultiSensor.Column3]

Description = "Report time"
Name = ReportTime
Type = Integer4
TimeFormat = hhmmss

[MultiSensor.Column4]

Description = "NovaStar station numeric identifier"
Name = StationNumId
Type = Integer4

[MultiSensor.Column*]

Description = "Point data raw value(s)"
Name = ValueRaw
Type = Integer4
# A value of data_position=1 in the database corresponds to the first data value column in the message.
PointOrder = ${ns.point:data_position}
# If Divisor is used, after the point is found using 'PointOrder', use the 'sensor_id' for the divisor value.
Divisor = ${ns.point:sensor_id}

nsrecdata Implementation Example

This section provides a checklist for implementing the custom specification using nsrecdata. Following this checklist will ensure that implementation is consistent with the NovaStar software and database design. The goal is to configure the NovaStar database to run a data collection command line similar to:

nsrecdata 0.0.0.0:5002 -Gcustomspec -d4 -c0 -S7 --customspec=/usr/ns/cus/nsrecdata-customspec.cfg

1. Configure software on transmitting stations:
   1. The previous sections illustrate how data messages can be defined. Use a consistent approach throughout the system.
   2. Implement software on the station consistent with the message format:
      1. If integers are used for transmission, handle the conversion of floating point measurements to integers using a scale factor. The scale factor during decoding can be handled either using NovaStar point calibration factor (in which case the scale factor will include transmission float/integer conversion) or configure to use the Divisor in the specification file (in which case the the scale factor must be an integer and is processed independent of the calibration factor).
      2. For each station, the order of data columns (1+) in a message must be mapped to NovaStar points (below). Note that the data columns are those with raw or scaled value. Keep track of the order because it is used below.
2. Create a custom specification file to allow ingest of the data:
   1. For example, name: nsrecdata-customspec.cfg
   2. Save in the /usr/ns/cus folder on the NovaStar machine.
   3. Set the PointOrder property in the configuration file for data columns to use: PointOrder = ${ns.point:data_position}
   4. Handle scale factor if station is configured to scale flointing point numbers to integers, using one of the following approaches:
      1. If the point's calibration is used to handle scaling, do not use Divisor in the custom specification file.
      2. Set the Divisor property in the configuration file for data columns to use: Divisor = ${ns.point:data_parameter}
   5. Remember to set the main [General] properties for Encoding, Delimiter, and Endianness.
3. Configure the station type data in the NovaStar database:
   1. What should this be set to? Is there a standard station type for custom data format stations, or reuse an existing station type?
   2. Can this be similar to stations that use -G #,#,#?
4. Configure the station in the NovaStar database:
   1. Set the station's station type to the type defined above.
5. Configure point data in the NovaStar database:
   1. There is currently no way to edit in the legacy Administrator so SQL must be used. The new Administrator will allow editing and will ensure that users see appropriate entry fields. The legacy Administrator does not provide an interface to edit -G customspec or --customspec options in the configuration table.
   2. Set the Data Position (data_position) value for the point to value 1+, consistent with the column in the message for the data.
   3. Handle scaling using one of the following approaches if the transmitted raw value was converted from float to integer using a multiplication factor:
      1. Set the point's calibration multiplier factor as appropriate, for example .01 if the floating point value was multipled by 100 to create an integer for transmission. The multiplier must take into account other calibration scaling if any is implemented.
      2. Set the Divisor (point's data_parameter) value for the point to the scale value that was applied to data value in the message. For example, set to 100 if the original raw value was scaled by 100 to convert floating point value to integer for transmission. If no scaling was done, set the value to 1 (which is the default if Divisor is not specified). Using this approach allows the transmission scaling factor to be separate from other calibration factor.
6. Enable the data ingest process:
   1. Note that the legacy Administator does not allow editing custom specification configuration. Use SQL. The new Administrator is being developed to edit the configuration.
   2. Add a configuration table entry with the following values:
      1. Name: alert_portN (where N is a number that is not already in use)
      2. Description: Example of custom data push to NovaStar
      3. Port name: :5002 (where the port number is distinct from other ports being used)
      4. Format: customspec (will result in -Gcustomspec)
      5. Custom specification file: /usr/ns/cus/nsrecdata-customspec.cfg (will result in --customspec=/usr/ns/cus/nsrecdata-customspec.cfg)
      6. Do not file data: false
      7. Source: specify a line number that is unique from other data feeds
      8. Connect type: listen
      9. The remaining options similar to nsrecdata ALERT2 command parameters are optional. For example, set debug level and timeout.

Limitations

• The format has been developed for use with the nsrecdata program. Support in other software may be added over time.
• The Encoding=Binary format is the initial focus of development. Other features will be fully enabled over time.
• Binary support for floating point values has not been implemented. Use integers with scale (divisor) or use NovaStar calibration factor to scale.
• The encoded format specification is often controlled by limitations of the communication network, for example bandwidth and cost limitations. This can result in messages that are terse and depend on additional configuration data. For example, the MultiSensor message type does not include point identifiers in the message and relies on the NovaStar database to provide the point identifiers based on the column position of data values.

Importing Data into NovaStar

Use the nsrecdata program with a custom specification file to import data into NovaStar.

Exporting Data from NovaStar

The specification file is currently not used to export data. Data can be exported from NovaStar in various formats by various software and web services.