FIWARE NGSIv2 support in the Onesait Platform

26/04/2021 Jesus Fernández Gómez-Pimpollo

What is FIWARE?

FIWARE is a European Union-driven platform for the development and global deployment of Future Internet applications. It is based on an open, public and free architecture and on a set of specifications that enable developers, service providers, enterprises and other organizations to develop products that meet their needs.

FIWARE provides a set of Open Source components that can be used along with other developments to build solutions both quickly and cost-effectively.

It provides an API for component integration (FIWARE NGSI), which provides interoperability, extension and replication capabilities.

FIWARE NGSI support in the Onesait Platform

The Onesait Platform provides native compatibility for applications developed on the FIWARE platform, allowing applications that use the NGSI standard to connect so they can interoperate with each other, and also to use the platform’s other capabilities (dashboards, analytical capabilities, KPI calculation, data API-fication, etc.).

Native support for the NGSI protocol is achieved by incorporating three components of the open FIWARE architecture into the Onesait Platform architecture:

Orion Context Broker
Cygnus
STH-Comet (Short Time History)

This results in the following component diagram in the Platform:

The functionalities covered by each FIWARE component incorporated in the Onesait Platform are the following ones:

Orion Context Broker:

The main component of any FIWARE solution, it is responsible for managing the context information of the systems and applications connected to the Platform, allowing them to send, modify, query and subscribe to notifications about any data relating to the system’s current state.

Information management is done through entities, which are representations of objects modeled ontologically as Data Models.

It implements the OMA NGSI specification by exposing a RESTful API of the NGSI v2 standard. Through this API, different applications (context providers) connected to the broker can integrate with each other by exchanging information.

The context information is persisted in the Platform’s Semantic Datahub, so that at every moment it reflects the system’s updated state.

Cygnus:

The context information managed by the Orion Context Broker is useful to know at any moment the state of the system, as any change is reflected immediately in it. In a data-centric platform such as the Onesait Platform, it is also essential to store the entire history of changes in any entity managed by the Orion Context Broker, because this allows making analytical processes, machine learning, building dashboards with historical data series, calculating KPIs, offering open data to citizens…

Within the FIWARE project, Cygnus is the connector in charge of providing historical persistence to the context data managed by the Orion Context Broker.

Based on Apache Flume, Cygnus provides a distributed process that allows to obtain all the changes in the context information from the Orion Context Broker, and to communicate them to different destinations:

Short Time Historic: Historical database for NGSI applications and systems.
Digital Broker: Onesait Platform’s native Broker, it stores the information in the semantic datahub, where this information is ready to be exploited by the different platform modules (API Manager, notebooks, dataflows, flow engine, BPM, dashboards, reports…) and to trigger all the processes associated in the platform to the corresponding ontology.

STH-Comet (Short Time Historic):

It provides applications and systems with REST NGSI interface, historical context data in the form of time series. It allows to calculate and return data, both «raw» and aggregated by attributes, as well as to temporally delimit the query range.

The historical information is persisted in the Platform’s Semantic Datahub by the corresponding Cygnus agent, so that STH-Comet is exclusively dedicated to answer requests via REST and to solve queries on the database.

NGSIv2 Compatibility

As mentioned above, Onesait Platform’s architecture incorporates the fundamental elements of the FIWARE scheme to enable applications and systems developed under the NGSIv2 standard to connect to the Platform to interoperate with each other, and also so that the information they produce can be managed for use with the Onesait Platform’s own capabilities.

In more detail, the integration diagram of these FIWARE components in the Onesait Platform is described below:

For applications and systems using NGSI v2 REST protocol, either through direct connection (context providers) or through an IoT Agent-type FIWARE intermediary using REST, MQTT, AMQP or other protocol in Ultralight 2.0, JSON or other formats, the REST interface is kept as is.

This REST interface, exposed from the acquisition layer of the platform, is the brokers’ own:

Orion Context Broker: Platform context manager.
Short Time Historic Comet: Context’s historical query manager.

For security purposes, these two brokers are not exposed directly, but through an NGINX reverse proxy, configured to intercept the requests and delegate to the platform’s Identity Manager the checking of the OAuth 2 token of the REST requests addressed to the Orion Context Broker and the STH Comet, and verify whether the client to which the token corresponds is valid and has permission to access the entity (ontology) on which the request is made or not.

Orion Context Broker, as the Platform’s context manager, handles NGSI v2 requests to create, query, modify and delete entities, as well as to manage subscriptions and notifications to changes in the context. It keeps all the information updated in the Platform’s Semantic Datahub, where a database for the context is created in the MongoDB instance.

Cygnus, as a context change persistence bus, has a dual task:

Maintain, for query from the STH Comet broker, the Short Time Historic with the context history in the Semantic Datahub of the Platform, where a database for the STH has been created in the MongoDB instance.
Notify the Platform’s main broker (Onesait Platform’s Semantic Broker) of the changes in the context so that it can transfer them to the ontologies corresponding to the entities. This way, all the information will be available in the Platform, ready to be exploited by its different capabilities: API manager, KPI engine, reporting engine, Dashboards engine, analytical Notebooks, Machine Learning, Dataflows, etc.

To do this, Cygnus, based on Apache Flume, subscribes to Orion Context Broker, to receive notifications of changes in the context and generate events that, once processed and prepared, are finally sent to the corresponding destination (STH in MongoDB and Digital Broker).

Short Time Historic Comet, as a historical query engine on the context, handles requests to query historical time series on the STH database previously fed by Cygnus.

Model compatibility between FIWARE and the Onesait Platform

NGSI is an open standard published by OMA (Open Mobile Alliance) to provide an interoperability protocol through centralized management and exchange of the so-called context information. The first version dates back to 2013 and it evolved in 2016 to its second version, NGSI v2, converting the standard to RESTful, simplifying operations, using native JSON format data types, supporting GeoJSON and Datetime data types and enriching the operations supported by the first version.

The protocol is based on the management of entities with their corresponding attributes and on the exchange between applications of the information of these entities.

An entity is any object of interest for an application, and it is defined by a set of attributes or properties:

The set of all entities existing at a given time, with their corresponding attributes updated to the actual value, is known as context.

Therefore, the main element in an NGSI v2 solution is the context broker or manager, to which all applications (context providers) are connected to keep updated all the entities they work with in the real world. The applications interoperate with each other by sharing the same context updated in the broker through the operations it provides to have access to the context.

The operations contemplated by the NGSI v2 standard allow to:

Create new entities in the context.
Modify attributes of an entity to update the context.
Consult and filter existing entities in the context.
Subscribe to changes in the context to be notified.
Consult all types of existing entities in the context and the detail of their attributes.
Batch insertions, updates and deletions.

Onesait Platform provides support for the NGSI v2 standard, including the above-mentioned set of pieces of the FIWARE ecosystem in its architecture. This eases the transparent integration of systems and applications developed according to the NGSI v2 standard with others that connect to the platform through any other module of the acquisition layer or through the interoperability layer.

Transparent integration, both between NGSI v2 applications and with the rest of the platform’s utilities, is possible thanks to the semantics of the used entities, which provides a univocal interpretation of the data independent of the applications.

For this purpose, entities belonging to the same entity type are modeled as ontologies through their corresponding Data Model. The platform provides an extensible set of templates with different Data Models following standards from fields such as IoT, Smart Cities, Social Media, GSMA….

The Data Models provided in the Platform to create ontologies that model entities are configurable and extensible, operable from the Control Panel. Thanks to this, with an adequate data governance, all entities of the same type comply with the requirements and restrictions previously established when designing the Data Model from which the developers will create the ontologies.

The ontologies in the Platform have physical storage for the data, both in real time and historical, in the chosen database.

All changes in the context that affect entities belonging to the type of data modeled by an ontology are immediately reflected in the ontology database. This provides seamless integration with other applications connected to the Platform that use such ontology, regardless of the broker or platform module they use to connect. For example: Digital Broker, Dataflow, Digital Twin Broker, API Manager, Portal Open Data (CKAN), etc.

NGSIv2 security in Onesait Platform

The FIWARE pieces integrated in the Platform (Orion Context Broker and STH Comet) that expose interfaces to the applications are all secured using the same model as the rest of the Onesait Platform components.

As they are REST interfaces, security is provided by means of the OAuth 2 protocol using the Platform’s own Identity Manager.

The security architecture is the one proposed by FIWARE. The brokers are not directly exposed to the applications, but instead NGINX is used as a reverse proxy acting as an intermediary. This proxy intercepts REST requests and redirects them first to a PEP-Proxy (plugin in Onesait Platform’s Identity Manager) that extracts the OAuth 2 token from the request, checks its validity in the configured OAuth 2 server (Identity Manager) and verifies whether the token is valid and whether the user it belongs to has privileges on the accessed resource (entity, present in the request path) and the access mode (http verb of the request). In case of a valid request, the PEP-Proxy returns a response with code 204 (No Content) and the NGINX reverse proxy lets the request pass to the broker. Otherwise, a code 401 (Unauthorized) is returned if the OAuth 2 token is invalid or 403 (Forbidden) if the user does not have access to the resource, and the NGINX reverse proxy does not let the request pass to the broker.

The Platform Identity Manager has been extended with a plugin that acts as FIWARE’s PEP-Proxy, analyzing the token received in each request to the brokers and verifying whether the token is valid and whether the operation on the requested entity is allowed for the user to whom the token belongs.

The security management of the applications that connect to the Orion Context Broker and to the STH Comet is performed from the platform’s control panel itself, following Onesait Platform’s standard guidelines:

If the application manages application roles, a Realm is registered for the application. The application roles, users and role(s) associated with each user are registered in this Realm. For example, say this is a bicycle rental management system with three types of sites connected to the platform: Rental Stations, Stores and Workshops. Then, we can consider each one of those as a role, and each specific site will belong to one or more roles.

The application is registered as a Platform project, to which the following are associated:
- Users: they can be associated one by one, or in block by associating a Realm (with its corresponding users) to the project.
- Resources: elements registered in the platform (Ontologies, Notebooks, Dataflows, APIs in API Manager, etc.). For each resource, the permission that each user in the project or each application role has over it, must be specified, and also whether a Realm has been associated to the project and whether permissions by role are to be established.
Once the application is registered as a project in the control panel of the platform, any users associated to it can identify themselves using the Identity Manager’s REST OAuth 2 API, to request an access token. This token will be included in the NSGI v2 REST requests through the X-Auth-Token header, so that the Identity Manager PEP-Proxy performs the access control tasks for the requested operation.

The Onesait Platform Identity Manager PEP-Proxy can provide three security levels:

Authentication:

Verifying that the OAuth 2 token included in the X-Auth-Token header is valid in the Identity Manager and belongs to an authenticated user in the Platform.

Authorization based on user permissions:

In this case the PEP-Proxy parses the NGSIv2 REST request extracting:

X-Auth-Token header: OAuth token 2. It allows validating that this is an authenticated user, and also knowing the user’s identity.
Resource accessed in the REST request path: it allows to know the entity (ontology) on which the request is made.
HTTP verb: access mode to the resource. It allows to know the type of operation requested: Add, Query, Modify, or Delete.

With this information, the PEP-Proxy within the Identity Manager determines whether the user is associated to the application (project registered in the control panel) and whether, within this project, the user has the access requested in the request to the entity (ontology).

Advanced authorization based on application roles:

It is identical to the authorization based on user permissions, except that, in this case, the project registered in the control panel for the application is associated to a Realm with roles that are the application’s own, and the permissions on each entity (ontology) have been established at the role level and not at the nominal user level, because the users have been profiled in the Realm with these roles from the control panel.

Thus, by installing a simple FIWARE-compliant plugin, the platform’s native security is used to provide native NGSI v2 applications with access control to information in the Context Broker and in the Short Time Historic.