Onesait Platform Technologies (Part II)

26/06/2020 Jesus Fernández Gómez-Pimpollo

As we said in the previous entry, we have grouped the technologies used to build the Onesait Platform into five blocks:

Core technologies.
Communication technologies.
Technologies based on Open Source products.
Storage technologies.
Deployment and execution technologies.

This second entry will deal with the last three blocks of technology.

Technologies based on Open Source products

One of the principles of the Onesait Platform is not to reinvent the wheel – there are many open source products that cover very interesting capabilities that can be incorporated into the Platform. Therefore, the Onesait Platform’s strategy for these cases is to carry out integration tasks to incorporate them into the set of offered functionalities.

Gravitee Api Manager: integrated with the Platform’s API Manager, it is deployed as an independent module. It provides better control over the exposed REST APIs, allowing to establish usage plans, access control that is integrated with the platform’s OAuth2 server, information pre- and post-processing, usage statistics and monetization.

Streamsets: this is the Dataflow component, which allows the visual construction of pipelines, integrated within the control panel, for information processing. From a source, they allow the information to be transformed and stored in different destinations. A set of visual components have been developed and they can be inserted in the information processing pipelines, so that this can be information whose origin or destination are the Onesait Platform ontologies. It is deployed as an independent module to which the platform and configuration components are added from a Streamsets Docker image.

Apache Zeppelin: it is the Notebooks component that supports the analytical part. It is based on a web interface, integrated within the control panel, from which developers and data scientists can implement, execute and train algorithms and graphically view the results. Its operation is based on interpreters, which allow the development of algorithms in Spark, Python, Java, Scala, etc. as well as access to different sources of information via JDBC, REST or ontologies in the Platform from Onesait Platform’s own interpreters. It is deployed as an independent module to which interpreters and configuration are added from a Zeppelin Docker image.

Node-RED: it is the Platform’s flow engine, which allows for the creation and execution of visual flows. It has a web interface, integrated into the control panel, from which flows of many different types can be orchestrated. It is deployed as an independent module that can internally start and manage multiple Node-RED instances. The types of flows can be grouped in the following categories:
- Information flows: they provide access to the platform’s ontologies through Onesait Platform’s own Node-RED components, so that the information can have an ontology as its origin and/or its destination, once it has been treated or enriched in the flow.
- Digital Twins orchestration: they provide integration with the Platform’s Digital Twin Broker, so that an entire external system managed as a Digital Twin can be abstracted and orchestrated in the flow engine as if it is only another node, by means of a Node-RED component of the Onesait Platform. This allows to visually orchestrate like that the communications and events between different Digital Twins.
- Microservices Orchestration: they provide integration with the microservices managed from the Onesait Platform, so that each microservice can be abstracted and orchestrated in the flow engine as another node, once more through a Node-RED component of the Platform. This allows communications and events between different microservices to be visually orchestrated.
- Notebooks and Dataflows Orchestration: by means of the Onesait Platform’s own Node-RED components, they allow invoking Notebooks and Dataflows from the flow engine.

Camunda: it provides the BPM engine of the Platform. It allows the execution of business flows in which it is possible to access ontologies, as well as to invoke microservices managed by the Onesait Platform. It is deployed as an own module that internally can start multiple Camunda tenants.

Drools: it is the component for rule execution, which allows the creation and management of DRL rules from the control panel so that these can be executed integrated with the platform when events occur in an ontology, or on demand through an API REST. It is deployed as an own module of the Onesait Platform that instances and manages the different Drools Containers.

CesiumJS: This is the library that provides support for viewing and editing geospatial layers. It is integrated in Javascript APIs that, from the georeferenced information stored in the ontologies, allow the geographic visualization of the information on a WebGL-compatible browser. It has total integration with the Platform’s Gadgets, which allows the incorporation of this visualization in Dashboards.

Jasper Reports: allows the creation of Reports from information stored in the Platform’s ontologies, so that it is possible to generate each report with the updated information, either on demand or periodically, so that it reflects the current status.

NGINX: it provides the reverse proxy function. All HTTP and REST services provided by the Onesait Platform are registered in NGINX and are only accessible from outside through the secure port exposed by NGINX. It is deployed as an independent module to which all the Platform’s configuration is registered from an NGINX image.

Storage technologies

These are the different storage technologies that support the persistence of the business information in the ontologies for its querying and exploitation.

In all cases, Onesait Platform provides query support in SQL even if the database engine in question does not support it, thus abstracting the developers of the specific query languages of each database management system.

MongoDB/MongoDB Atlas: they allow for the use of MongoDB or MongoDB Atlas as the management system for ontology storage. A MongoDB collection is associated to each ontology of the platform.

External databases: they allow the use of relational database management systems, and ease the integration with legacy systems that have past information that must be exploited in the Platform. A table or view is associated to each ontology. Currently, connectors are available with the following databases:
- Oracle
- MySQL
- MariaDB
- PostgreSQL
- PostGIS, supporting geographic queries.

ElasticSearch: it provides support to ontologies, storing information in ElasticSearch indexes. Each ontology is associated to one ElasticSearch index.

Hive, Impala and Kudu: database engines that are suitable for analytical scenarios where the information volume is huge.

Kafka: thanks to the filepersistence of the messages processed by Kafka, it can also be used as a support for the storage of ontologies, acting as a persistence engine.

Deployment and CaaS technologies

They are part of the Platform’s CI/CD cycle, automating the devices,’ construction and testing, as well as their deployment in previous and productive environments.

Jenkins: provides automated support for compilation, construction of artefacts throughout all the platform, versioning, repository publishing and environment updating.

Ansible, Terraform and Helm: are the technologies that ease the automation of installation tasks in the different compatible container platforms (Kubernetes, Rancher, Openshift). They reduce the installation time of both the platform and all its associated software to a few minutes.

Kubernetes, Rancher and Openshift provide a container-centric management environment. They ease and automate the orchestration of a cluster of machines, networks and storage, so that users do not have to do it. This way, they provide the environment in which all Onesait Platform modules can run once they have been containerized. Rancher 2 and Openshift are supported by Kubernetes and extend it with their own tools and web interfaces. They facilitate the management of containers in a centralized manner, also offering business support.

Well then, these are the technologies we use in the Onesait Platform as of 2020. We hope you have found them interesting, and if you have not yet read the technologies of the first two modules, we encourage you to do so.

We hope you found them interesting, and if you have any question, leave us a comment.