A Unified Data Platform for IoT transport protocol interoperability, part two – Normalizing data
This multi-part series addresses the need for a single data interoperability model that unifies data exchange across multiple transport protocols (including SMTP, MQTT, CoAP, DDS, BACnet, and others) via an extension layer. Such a layer must be simple and extensible to enable plug and play interoperability and universal adoption amongst humans, machines, and applications in the IoT.
Editor’s note: Read part one at http://iotdesign.embedded-computing.com/articles/a-unified-data-platform-for-iot-transport-protocol-interoperability-part-one/.
A Unified Data Platform can normalize the structure of data to eliminate barriers to personal productivity and provide a unifying standard of interoperability for machines and businesses. An entity-centric data service and message payload protocol (such as BEAM and OASIS’ Open Data Protocol (OData)) can standardize create, read, update, and delete (CRUD) operations across machine data sources. A managed runtime environment (for example, BEAMrt and .NET Common Language Runtime) can processes portable code to control machines and render human-to-machine interfaces. The runtime can abstract away details of the underlying hardware, and allows programs to execute the same way on any machine, from appliance to gateway to server.
Together a single data service (BEAM Service), a unified message payload protocol (BEAM Protocol), and a universal runtime can provide the infrastructure to create, manage, and share data objects between interoperable machines and automation systems (Figure 6). This infrastructure can break down data silos, eliminate complex system integrations, and unify information spaces using only metadata.
The data service can be added to a chipset (such as Qualcomm, Broadcom, Marvell) or operating system (such as Windows, Linux, ARM mbed), which is then integrated into machines. The service can route inbound messages to machine resources, including the universal runtime, to process unified message payloads. These payloads are highly structured and can be transported via several communication protocols including HTTP/REST, AMQP, MQTT, and CoAP. The universal runtime can create the content of a data store by processing object events contained within certain payloads.
Email “Message,” “Sales Order,” and “Coffee Machine” can each be defined as an Entity, where an Entity is a category of like things or objects capable of an independent existence and unique identity. An entity-centric Unified Message Payload Protocol can eliminate data silos by enabling attribute values of any new or updated object (for example, email message, sales order, coffee machine) to be transported as serialized, machine-readable object events within a semantically precise message body (Figure 7).
Object Events are an extension of the Entity Attribute Value (EAV) data model, and can include event types that support the creation, update, and deletion of any object. Serialized object events can replace the fragmented payload structures of the EDI, SMTP, and M2M examples by relaying the following normalized information:
- The value of the Customer attribute of a new Sales Order object (4F19…) of the “oem.com” domain is 0540… (the identifier for “SmartMart.com”)
- The value of the Reference Number attribute of the new Sales Order object is 45678
- The value of the Sales Order attribute of a new Sales Order Item object is 4F19…
- The value of the Item attribute of the new Sales Order Item object is 6685…
- The value of the Sender attribute of a new email Message object (4BA7…) of the “Alice.Smith@email.com” domain is 92C3… (Tim.Johnson@SmartMart.com)
- The value of the Subject attribute of the new Message object is “Your Order is Shipped”
- The new value of the Status attribute of Coffee Machine object (DCE1…) is 2 (Brewing)
- The new value of the Temperature attribute of the Coffee Machine object is 120
Unifying data and metadata models
The same Unified Message Payload Protocol can support CRUD operations for entities that define interoperable automation (Figure 8).
Domains (like SmartMart.com) can own entity objects (like Sales Orders, Items, and Customers) that can be shared with other Domains (like Alice.Smith@email.com) through Memberships and Subscriptions. Machines (such as cloud servers) can manage these objects within data stores on behalf of Domains and utilize Connections (e.g. IP address) and Sessions (e.g. web socket) to transport object events, query requests, and view responses to other Machines (like smartphones). Machines can utilize Triggers to automatically generate and transport object events from processed object events – for example, a defined Trigger can cause object events representing a new email message to be generated when an object event reflects a change in a Sales Order’s Status to “Shipped.”
An Entity Relationship (ER) Data Model, as described by Peter Chen in 1976 , can define the relationships between entities (such as Customer and Sales Order). A metadata-driven data platform (for example, ControlBEAM, Microsoft Dynamics CRM) extends the utilization of this data model beyond customer relationship management (CRM) to managing any entity relationship.
The ControlBEAM ER Data Model (Figure 9) illustrates how entity objects can be defined as data within an object dataset, where the dataset represents an entity (for example, Sales Order), each row in the dataset represents an object (for example, Order #13579), and each column represents an attribute (e.g. Customer).
Datasets can persist within data stores and can be manifested as two-dimensional arrays, database tables, delimited strings, and even as port pins in a microcontroller. Values in an object dataset can be serialized in a delimited string for transport.
Every object dataset can have a minimal set of attributes, including an “Object Identifier” that enables an object to be uniquely identified. A Machine can generate a globally unique identifier (GUID) for each object it creates and transports in the form of object events. These GUID values are used in relationship attributes to define relationships between objects. For example (in Figure 7), the “Sales Order” attribute of the “Sales Order Item” entity is set to the GUID value of the Sales Order object identifier (4F19...).
Every object dataset can also include an “Owner Domain” attribute, which identifies the Domain object (for instance, oem.com) via its GUID value (such as A4CC...) that owns an entity object (like Sales Order #45678).
Datasets and their unique attributes can also be referenced by GUID values as they are also objects themselves. For example, a Sales Order object dataset can be referenced by the GUID of the Sales Order entity (78BD...) and a dataset column can be referenced by the GUID (9E76...) of the attribute it represents (for instance, Customer).
The “entity” entity
Each object dataset represents a specific entity, and every entity and its unique attributes can be defined as data (called metadata) within object datasets. Each object (row) in an “Entity” dataset can represent an entity, including the “Entity” dataset itself, which enables the data platform to examine and modify its own structure (or, reflection). Further, each object in an Attribute dataset (Figure 10) can represent an attribute of an entity (such as a column within an object dataset), including the attributes of the Attribute dataset itself (i.e. reflection).
The Attribute dataset includes objects (relationship attributes) that define the relationships between entities (the “Attribute” entity is related to the “Entity” entity, and the “Membership” entity is related to the “Domain” entity). The same Unified Message Payload Protocol can support CRUD operations for these metadata objects.
Data ownership domains
Each object created in a Domain dataset can represent a person, business, organization, or cloud service such as Outlook.com (or fictitious email.com). Domains can own other entity objects, including other Domain objects (Figure 11). For example, mailbox domain “Tim.Johnson” of host domain “SmartMart” of registrar domain “com” has the full name “Tim.Johnson@SmartMart.com.” And mailbox domain “Alice.Smith” of host domain “email.com” has the full name “Alice.Smith@email.com.”
A domain can assume multiple roles. For example, host domain “SmartMart.com” can be both a customer and a vendor within the retail supply chain; mailbox domain “Tim.Johnson@SmartMart.com” can be an employee, sales rep, and customer of “SmartMart.com.” The various roles of domains can be represented as entities and defined as objects within the Entity dataset (Figure 12). Each of these “domain role” entities has unique attributes that can supplement the attributes of the Domain entity.
Member domains and their roles and rights
The role (or entity) of a domain in relation to another domain can be defined as an object in a Membership dataset (Figure 13). For example, domain “SmartMart.com” has the role of “Vendor” in a membership of domain “Alice.Smith@email.com”. A “Machines” subscription provides this “member domain” access to object events associated with machines it sold to her.
- Domain “Alice.Smith@email.com” has a mirrored “Customer” role in a membership of “SmartMart.com” that provides her data access to sales orders and customer preferences within a “Sales” subscription
- Domain “Tim.Johnson@SmartMart.com” has “Employee” and “Sales Rep” memberships with domain “SmartMart.com”
- Domains “Tim.Johnson@SmartMart.com” and “Alice.Smith@email.com” have mirrored “Contact” memberships with each other
- The Object Identifiers (for example, 32E3...) of mirrored memberships can be shared to reflect interdependency of the two objects
Regardless of role, a domain can always be identified by the same GUID value within relationship attributes of any entity. For example, the object identifier (92C3...) of domain “Tim.Johnson@SmartMart.com” can be used to identify him as the “Sales Rep” of a Sales Order and the “Sender” of an email Message (Figure 14).
A subscription that’s assigned to memberships can provide data access to the member domain based on the entities that are included in the subscription. For example, a “Sales” subscription has Customer, Sales Rep, Item, Sales Order, and Sales Order Item entities defined within Subscription Entity objects (Figure 15).
Constructing human-readable portable apps from machine-readable metadata
The collection of entities within a subscription can form a portable App. When the member domain is a person, the portable App can be rendered through a user interface that includes human-readable names of the subscription entities and their attributes. These human-readable names can be derived from a composite of Term objects that reflect entity relationships (Figure 16). For example, the “Sales Order Item” entity name can be generated from the Terms “Sales,” “Order,” and “Item.” The same “Sales” Term object can be used for naming other related entities and attributes (such as “Sales Rep,” ”Sales Order”). Translated Term objects can enable a user interface to be rendered in the human-readable language designated by the member domain.
Machine connections and sessions
Machines manage all of these entity objects on behalf of domains and utilize Connections and Sessions to transport object values in unified message payloads to other machines. Connection objects can define the address of machines connected within a domain’s network (Figure 17). For example, a cloud server, smartphone, gateway, and coffee machine have connections to domain “Alice.Smith@email.com.” Another cloud server has a connection to domain “SmartMart.com.”
A machine can utilize a connection’s address to direct a session request to the connection’s machine, which can create a session object and return the session’s object identifier (in other words, a token) to the initiating machine. The two machines can utilize the session token to exchange object events, queries, and views with each other.
In the example Session dataset, a row representing a session provides Alice Smith access to her information objects by exchanging unified message payloads between her cloud server and her smartphone. A second session enables the cloud server of “SmartMart.com” to send object events representing email messages from member domain “Tim.Johnson@SmartMart.com” to Alice’s cloud server. A third session enables her home gateway to forward state changes (or object events) to her cloud server that it receives from her coffee machine (a fourth session).
Reading data using metadata-defined object queries (the “R” in CRUD)
While object events can standardize create, update, and delete operations across machine data sources, metadata-defined object queries can standardize read operations.
An Object Query dataset can be transported as a request within a unified message payload to a machine resource (for example, runtime), which can return an Object View response based on the query metadata. An Object View response can include the state (in other words, current attribute values) of a machine object or related information objects maintained within a data store.
In an example, an Object Query that is nested within the Body attribute of a Message object (Figure 20) can be transported to a runtime on a cloud server machine to retrieve current attribute values (or its state) associated with a Sales Order object that’s maintained within a relational database (Figure 18).
Defining automation through triggers and actions
Trigger and Action objects associated with entities can define the automated actions to occur if an object event meets a specific condition. For example, an event of one machine object can trigger an event of another machine object (interoperable machines) within a home management system. WHEN the Time of a Clock is 6:00 AM, THEN set the Status of a Coffee Machine to Brewing (Figure 19).
In a second example, an event of one information object can trigger events of another information object within a warehouse management system. WHEN the Status of a Sales Order is Shipped, THEN create a Message to the Customer of the Sales Order that includes a metadata-defined query of the Sales Order and related information.
A hybrid of these examples can illustrate how object events within one system can trigger object events within another system (interoperable systems). For example, WHEN the Heater of the Ultra Coffee Machine is Failing, THEN create a Sales Order to ship a new Ultra Coffee Machine to the Owner of the failing machine.
A portable App can include datasets of trigger and action objects associated with a subscription’s entities. The App’s datasets can be loaded in-memory for use by a universal runtime. When processing object events, the runtime can create, update, and delete datasets within a data store, and can generate new object events from the object events it processes that meet a trigger’s condition. These new events are also processed by the runtime, or can be transported to another machine for processing by its runtime.
In an example, a defined Trigger object (Figure 19) can cause object events representing a new email Message to be generated when a Sales Order object event reflects a change in the Status attribute value to “Shipped” (Figure 20).
As defined by the Trigger’s Action objects, the value of the new Message’s Sender attribute is set to the Sales Order’s Sales Rep attribute value (92C3...), which is the object identifier for domain “Tim.Johnson@SmartMart.com”; the value of the Message’s Owner Domain is set to the Sales Order’s Customer attribute value (3A45...), which is the object identifier for domain “Alice.Smith@email.com”; and the value of the Message’s Body attribute is set to a nested Object Query dataset.
Part three addresses portable app frameworks supporting interoperable information exchange between humans and machines.
9. The Entity-Relationship Model - Toward a Unified View of Data, Peter Chen, March 1976