Rule Engine Overview

NDU Rule Engine is a highly customizable and configurable system for complex event processing. With rule engine you are able to filter, enrich and transform incoming messages originated by IoT devices and related assets. You are also able to trigger various actions, for example, notifications or communication with external systems.

Key Concepts

Rule Engine Message

Rule Engine Message is a serializable, immutable data structure that represent various messages in the system. For example:

Rule Engine Message contains the following information:

Rule Node

Rule Node is a basic component of Rule Engine that process single incoming message at a time and produce one or more outgoing messages. Rule Node is a main logical unit of the Rule Engine. Rule Node can filter, enrich, transform incoming messages, perform action or communicate with external systems.

Rule Node Relation

Rule Nodes may be related to other rule nodes. Each relation has relation type, a label used to identify logical meaning of the relation. When rule node produces the outgoing message it always specifies the relation type which is used to route message to next nodes.

Typical rule node relations are “Success” and “Failure”. Rule nodes that represent logical operations may use “True” or “False”. Some specific rule nodes may use completely different relation types, for example: “Post Telemetry”, “Attributes Updated”, “Entity Created”, etc.

Rule Chain

Rule Chain is a logical group of rule nodes and their relations. For example, the rule chain below will:

image

Tenant administrator is able to define one Root Rule Chain and optionally multiple other rule chains. Root rule chain handles all incoming messages and may forward them to other rule chains for additional processing. Other rule chains may also forward messages to different rule chains.

For example, the rule chain below will:

image

Message Processing Result

There are three possible results of message processing: Success, Failure and Timeout. The message processing attempt is marked as “Success” when the last rule node in the processing chain successfully process the message. The message processing attempt is marked as “Failure” if one of the rule nodes produce “Failure” of message processing, and there is no rule nodes to handle that failure. The message processing attempt is marked as “Timeout” when overall time of processing exceed configurable threshold.

See diagram below and let’s review the possible scenarios:

image

If the “Transformation” script fails, the message is not marked as “Failed”, because there is a “Save to DB” node connected with “Failure” relation. If the “Transformation” script is successful, it will be pushed to “External System” with the REST API call. If the external system is overloaded, the REST API call may “hang” for some time. Let’s assume the overall timeout for message pack processing is 20 seconds. Let’s ignore Transformation script execution time because it is < 1ms. So, if the “External System” will reply within 20 seconds, the message will be successfully processed. Similar, if “Save to DB” call will succeed, the message will be successfully processed. However, if the external system will not reply within 20 seconds, the message processing attempt will be marked as “timed-out”. Similar, if “Save to DB” call will fail, the message will be marked as failed.

Rule Engine Queue

Rule Engine subscribe to queues on startup and polls for new messages. There is always “Main” topic that is used as a main entry point for new incoming messages. You may configure multiple queues using NDU.yml or environment variables. Once configured, you may put message to the other topic using “Checkpoint” node. This automatically acknowledges corresponding message in the current topic.

The definition of the queue consists of the following parameters:

Queue submit strategy

Rule Engine service constantly polls messages for specific topic and once the Consumer returns a list of messages it creates the TbMsgPackProcessingContext object. Queue submit strategy controls how messages from TbMsgPackProcessingContext are submitted to rule chains. There are 5 available strategies:

See this guide for an example of submit strategy use case.

Queue processing strategy

Processing Strategy controls how failed or timed out messages are re-processed. There are 5 available strategies:

All “RETRY*” strategies support important configuration parameters:

See this guide for an example of processing strategy use case.

Default queues

There are three default queues configured: Main, HighPriority and SequentialByOriginator. They differ based on submit and processing strategy. Basically, rule engine process messages from Main topic and may optionally put them to other topics using “Checkpoint” rule node. Main topic simply ignores failed messages by default. This is done for backward compatibility with previous releases. However, you may reconfigure this at your own risk. Note that if one message is not processed due to some failure in your rule node script, it may prevent next messages from being processed. We have designed specific dashboard to monitor Rule Engine processing and failures.

The HighPriority topic may be used for delivery of alarms or other critical processing steps. The messages in HighPriority topic are constantly reprocessed in case of failure until the message processing succeeds. This is useful if you have an outage of the SMTP server or external system. The Rule Engine will retry sending the message until it is processed.

The SequentialByOriginator topic is important if you would like to make sure that messages are processed in correct order. Messages from the same entity will be processed with the order they arrive to the queue. Rule Engine will not submit new message to the rule chain until the previous message for the same entity id is acknowledged.

Predefined Message Types

List of the predefined Message Types is presented in the following table:

Message TypeDisplay NameDescriptionMessage metadataMessage payload
POST_ATTRIBUTES_REQUEST Post attributes Request from device to publish client side attributes (see attributes api for reference) deviceName - originator device name,
deviceType - originator device type		 key/value json:
{
  "currentState": "IDLE"
}
POST_TELEMETRY_REQUEST Post telemetry Request from device to publish telemetry (see telemetry upload api for reference) deviceName - originator device name,
deviceType - originator device type,
ts - timestamp (milliseconds)		 key/value json:
{
  "temperature": 22.7
}
TO_SERVER_RPC_REQUEST RPC Request from Device RPC request from device (see client side rpc for reference) deviceName - originator device name,
deviceType - originator device type,
requestId - RPC request Id provided by client		 json containing method and params:
{
  "method": "getTime",
  "params": { "param1": "val1" }
}
RPC_CALL_FROM_SERVER_TO_DEVICE RPC Request to Device RPC request from server to device (see server side rpc api for reference) requestUUID - internal request id used by sustem to identify reply target,
expirationTime - time when request will be expired,
oneway - specifies request type: true - without response, false - with response		 json containing method and params:
{
  "method": "getGpioStatus",
  "params": { "param1": "val1" }
}
ACTIVITY_EVENT Activity Event Event indicating that device becomes active deviceName - originator device name,
deviceType - originator device type		 json containing device activity information:
{
  "active": true,
  "lastConnectTime": 1526979083267,
  "lastActivityTime": 1526979083270,
  "lastDisconnectTime": 1526978493963,
  "lastInactivityAlarmTime": 1526978512339,
  "inactivityTimeout": 10000
}
INACTIVITY_EVENT Inactivity Event Event indicating that device becomes inactive deviceName - originator device name,
deviceType - originator device type		 json containing device activity information, see Activity Event payload
CONNECT_EVENT Connect Event Event produced when device is connected deviceName - originator device name,
deviceType - originator device type		 json containing device activity information, see Activity Event payload
DISCONNECT_EVENT Disconnect Event Event produced when device is disconnected deviceName - originator device name,
deviceType - originator device type		 json containing device activity information, see Activity Event payload
ENTITY_CREATED Entity Created Event produced when new entity was created in system userName - name of the user who created the entity,
userId - the user Id		 json containing created entity details:
{
  "id": {
    "entityType": "DEVICE",
    "id": "efc4b9e0-5d0f-11e8-8559-37a7f8cdca74"
  },
  "createdTime": 1526918366334,
  ...
  "name": "my-device",
  "type": "temp-sensor"
}
ENTITY_UPDATED Entity Updated Event produced when existing entity was updated userName - name of the user who updated the entity,
userId - the user Id		 json containing updated entity details, see Entity Created payload
ENTITY_DELETED Entity Deleted Event produced when existing entity was deleted userName - name of the user who deleted the entity,
userId - the user Id		 json containing deleted entity details, see Entity Created payload
ENTITY_ASSIGNED Entity Assigned Event produced when existing entity was assigned to customer userName - name of the user who performed assignment operation,
userId - the user Id,
assignedCustomerName - assigned customer name,
assignedCustomerId - Id of assigned customer		 json containing assigned entity details, see Entity Created payload
ENTITY_UNASSIGNED Entity Unassigned Event produced when existing entity was unassigned from customer userName - name of the user who performed unassignment operation,
userId - the user Id,
unassignedCustomerName - unassigned customer name,
unassignedCustomerId - Id of unassigned customer		 json containing unassigned entity details, see Entity Created payload
ADDED_TO_ENTITY_GROUP Added to Group Event produced when entity was added to Entity Group. This Message Type is specific to NDU PE. userName - name of the user who performed assignment operation,
userId - the user Id,
addedToEntityGroupName - entity group name,
addedToEntityGroupId - Id of entity group		 empty json payload
REMOVED_FROM_ENTITY_GROUP Removed from Group Event produced when entity was removed from Entity Group. This Message Type is specific to NDU PE. userName - name of the user who performed unassignment operation,
userId - the user Id,
removedFromEntityGroupName - entity group name,
removedFromEntityGroupId - Id of entity group		 empty json payload
ATTRIBUTES_UPDATED Attributes Updated Event produced when entity attributes update was performed userName - name of the user who performed attributes update,
userId - the user Id,
scope - updated attributes scope (can be either SERVER_SCOPE or SHARED_SCOPE)		 key/value json with updated attributes:
{
  "softwareVersion": "1.2.3"
}
ATTRIBUTES_DELETED Attributes Deleted Event produced when some of entity attributes were deleted userName - name of the user who deleted attributes,
userId - the user Id,
scope - deleted attributes scope (can be either SERVER_SCOPE or SHARED_SCOPE)		 json with attributes field containing list of deleted attributes keys:
{
  "attributes": ["modelNumber", "serial"]
}
ALARM Alarm event Event produced when an alarm was created, updated or deleted All fields from original Message Metadata
isNewAlarm - true if a new alram was just created
isExistingAlarm - true if an alarm is existing already
isClearedAlarm - true if an alarm was cleared 		json containing created alarm details:
{
  "tenantId": {
     ...
  },
  "type": "High Temperature Alarm",
  "originator": {
     ...
  },
  "severity": "CRITICAL",
  "status": "CLEARED_UNACK",
  "startTs": 1526985698000,
  "endTs": 1526985698000,
  "ackTs": 0,
  "clearTs": 1526985712000,
  "details": {
    "temperature": 70,
    "ts": 1526985696000
  },
  "propagate": true,
  "id": "33cd8999-5dac-11e8-bbab-ad47060c9431",
  "createdTime": 1526985698000,
  "name": "High Temperature Alarm"
}
REST_API_REQUEST REST API Request to Rule Engine Event produced when user executes REST API call requestUUID - the unique request id,
expirationTime - the expiration time of the request		 json with request payload

Rule Node Types

All available rule nodes are grouped in correspondence with their nature:

Configuration

Each Rule Node may have specific configuration parameters that depend on the Rule Node Implementation. For example, “Filter - script” rule node is configurable via custom JS function that process incoming data. “External - send email” node configuration allows to specify mail server connection parameters.

Rule Node configuration window may be opened by double-clicking on the node in the Rule Chain editor:

image

Test JavaScript functions

Some rule nodes have specific UI feature that allow users to test JS functions. Once you click on the Test Filter Function you will see the JS Editor that allows you to substitute input parameters and verify the output of the function.

image

You can define:

After pressing Test output will be returned in right Output section.

Rule Engine Statistics

NDU Team have prepared the “default” dashboard for Rule Engine statistics. This dashboard is automatically loaded for each tenant. The statistics collection is enabled by default and is controlled via configuration properties.

You may notice insights about errors in processing and what causes them on the dashbaord below:

image

## Debugging NDU provides ability to review incoming and outgoing messages for each Rule Node. To enable debug, user need to ensure that “Debug mode” checkbox is selected in the main configuration window (see first image in the Configuration section).

Once debug is enabled, user is able to see incoming and outgoing messages info as long as corresponding relation types. See image below for a sample debug messages view:

image
image

Import/Export

You are able to export your rule chain to JSON format and import it to the same or another NDU instance.

In order to export rule chain, you should navigate to the Rule Chains page and click on the export button located on the particular rule chain card.

image

Similar, to import the rule chain you should navigate to the Rules Chains page and click on the big “+” button in the bottom-right part of the screen and then click on the import button.

Architecture

To learn more about internals of the rule engine, see architecture page.

Tutorials

NDU authors have prepared several tutorials to help you get started with designing rule chains by example: