Onesait Platform Release 3.0.0-jailbreak

This first quarter of 2021 brings us a new major version of Onesait Platform, full of new features and visual redesigns. We have therefore reached version 3.0, which the Community has decided to call «JailBreak», following the versioning policy we have defined.

This release completes what we had in mind in the Onesait Platform Roadmap for this Q1:

And what’s new in this new major version? Let’s have a look at it:

Engine

This distribution of the Platform supports the development of microservices and all kinds of applications. To help in this type of development, the Platform allows visual development with a LowCode approach incorporating a centralized web console that supports management, configuration, development and deployment.

In this version we have incorporated important features:

New UX for the Control Panel

After months of hard work, in this version you will see a new graphic design of the Platform’s Control Panel, adapted to the Onesait Design System (ODS)’s new version 3.

We are still working on the complete adaptation to this new graphic design. In future versions you will see a new UI for the main screens.

New Ontology type: Non-persistent or control ontology

So far, the concept of ontology has represented the entities managed by the Platform and persisted in one of the Platform repositories, although in many scenarios ontologies are used as «Control Ontologies» that allow us to trigger processes such as:

• Triggering a Rule.
• Executing a Flow from the FlowEngine.
• Executing a DataFlow Pipeline.
• Communicating with the Kafka Broker for insertion and notification.
• Etcetera.

That’s why we have created the concept of Non Persistent Ontology/Control Ontology, which is a new type of ontology that will not be persisted, but it will trigger all the processes that can be launched in the Platform after the insertion of an ontology. It can be created from the «Advanced Settings» option when creating an ontology:

Centralized Log integrated in Control Panel

In the previous release, we integrated Graylog as a solution to centralize the logs of the Platform, and in turn to be used by the solutions built on the Platform. In this release, we have integrated Graylog’s powerful UI with the Control Panel, allowing access to administrator users. Its REST API has also been made available, as you can see here.

If you want to know more about this, we tell you about it in this post.

Integrating Keycloak as Advanced Identity Manager

Although this functionality was completed in the previous Q4, we have included functionalities that you have been requesting.

Thus, the Platform includes an Identity Manager built on top of Spring Cloud Security that covers typical security use cases such as authentication, authorization, realms, plugins, etc. We have integrated Keycloak as an alternative Identity Manager for advanced scenarios, mapping Keycloak realms with platform verticals, Keycloak Clients with Platform Realms and the Platform ConfigDB as the default Provider (although LDAP can also be used).

You can find more information about the integration in this guide: Support for Keycloak as Advanced Identity Manager.

Platform’s Kafka Broker updated to Kafka Confluent Version 6.1.1

We have updated the version of Kafka, Zookeeper and KSQL (now KsqlDB) to Confluent version 6.1.1, meaning that, with this update, we now have Kafka version 2.7.0.

Adjustments to the deployment of microservices on Onesait DevOps

We have adjusted the deployment of microservices on Onesait DevOps so that the user of this functionality does not need to configure anything but the essential.

You can have a look to the use of this functionality here.

New configuration options in integration with Elasticsearch

We have increased the configuration possibilities for ontologies defined on Elasticsearch allowing the configuration of Shards and Replicas for each ontology, as well as the configuration of Templates (to store audit data, logs, timeseries, etc.). Elasticsearch recommends creating several indexes per time block, especially if the data is more or less homogeneous per time block – for example, creating an index per month, or per week, or per day, etc.).

We explain it in more detail in this post.

Intelligence

This distribution of the Platform supports the development of systems that use the Platform’s Intelligence capabilities, either with its AI capabilities, ingest from different sources, analytics, visualization, etc.

This quarter, we have worked on:

New tool for data lineage

We keep on expanding our Data Governance capabilities (and there’s more to come yet!), incorporating this functionality that allows you to query the relationships of an element (Ontology, Dashboard, etc.) with all the other elements of the Platform, which will help to give a global view of the impact that any one change in any of the elements of the lineage may have. A new menu option, DATA GOVERNANCE, has been created with this functionality:

This allows us to show the traceability over the selected ontology:

We explain its use in this post.

New version of the Rules Engine

We have extended the functionalities of the Rules Engine according to the requests you have been sending us (you can see that we listen to you carefully). Thus, in this version, rules can be defined through decision tables that can be loaded through the UI.

Besides, a utility is available in the engine to query ontologies, which will allow us to cross-reference the input data to the rule with other context conditions. All this is explained in this guide.

New version of the Python client library

We have published a new version of the Python client library (1.4) that includes several new features, among these the «BaseModelService» class that supports training, retraining, deployment and inference of models in a simple way. You can see an example of its use here.

Wrapper to create models on Notebooks

In this version of the Platform, in collaboration with the Onesait Data team, we have included improvements in the Python client, allowing among other functionalities to:

• Transparently store/retrieve model files generated by trainings in the File Repository.
• Use ontology data as model data input.
• Management of a meta-information ontology of the different associated models and retrieval of the most accurate model.

Beyond that, we have prepared a complete tutorial on its use in Notebooks for Model creation as you can see in this guide.

Favorite Gadgets in Dashboards

This new feature will allow each user to assign favorite gadgets, so that they can then quickly add those to their Dashboards.

These favorite gadgets can be represented as drag-and-drop components and dragged on top of your Dashboards, in order to add them to your Dashboards.

Updated to the DataFlow-support component

In this release, we have updated the Streamsets version from 3.10 to 3.18.1, which includes several improvements in the nodes of the Streamsets libraries.

This new version includes the orchestrator library, which will allow us to schedule the execution of flows of the Dataflow instances that we have deployed in the Platform, including the Cron Scheduler:

We explain it in more detail in this article about it.

Improvements in the creation of ontologies on external databases

Up until this release, you could only generate ontologies from external databases that came directly with the connection (and, therefore, forcing you to include the database and/or schema, depending on the database type, in the connection url). With the new version of the Platform, you can choose the database and/or schema (as applicable in each database) before selecting the table.

You can find more information about this in this post.

Things

This distribution of the Platform supports the development of IoT systems, both in Cloud and Edge.

In this quarter, we have made progress in:

Dashboard Broker

Integrated with Prometheus and with Grafana, the IoT/Edge HUB version included in this release now provides a tool to monitor the IoT/Edge Broker based on EQMX.v.4.0. With an already-packaged, complete set of KPIs, you can now easily view parameters such as: number of connected clients, number of exchanged messages, current concurrency of message reception per second, load of Erlang cluster nodes, etc.

Project creation wizard

You can now create IoT/Edge Computing projects in Things through an intuitive wizard. It will guide you through associating and registering devices, connecting container registries, editing edge device operating commands, and setting Gitlab/Gitea repositories for configuration and codebase.

Improvements in Landscape

In this new release, we have improved the visibility of device landscape availability, providing information about control and enterprise communication channels, Device-2-Cloud and Cloud-2-Device information. Unique identification (urn) has been included following the RFC-8141 standard. QRs (urls for accessing device information) are also provided, and a set of context hints have been added providing help in device handling.

Connectors for DataFlow and FlowEngine

We have added the capacity to define bi-directional connectors to Data Flow and FlowEngine. This allows for automatic ingestion of device measurements and routing of commands using simple ontology models. Transports from these two connectors are performed securely via TLS> = 1.2 and user secrets of configurable cryptographic algorithms.

Docker Registry.v.2 Browser

It allows to browse the image registry integrated with IoT/Edge HUB for Edge containers. You can also check versions and sizes, parse different layers and automatically select tags to be included in your deployment projects. Image loading and versioning status can be easily verified.

KMS-Vault

We have added a completely new section for the lifecycle management of keys and secrets. Thanks to this new module, it is possible to generate a large number of cryptographic tokens, store them securely in HMS (hardware and software) and ensure regulatory compliance in the protection of distributed digital assets. Things uses this module to protect devices and communications.

DevOps

Within this line of work we include all the tools, utilities and capabilities of the Platform that help in Development and Operation.

Helm Charts for Onesait Platform CE

Helm allows you to package the Onesait Platform deployment to be deployed in Kubernetes, allowing you to «template» its files or manifests, such as Deployment, Service, PersistentVolumeClaim, Ingress, Secret, etc., in templates or Helm Templates and package them in a Chart. In this version, we have released the code of a Helm chart to be able to deploy the Onesait Platform Community Edition on k8s. You can find the code in github and also in Artifact HUB:

Stack for Advanced Platform Monitoring

Another new feature of this version, as related to Operation, is the integration of advanced monitoring tools on the different modules of the Platform (or on modules external to the Platform). To do this, we have chosen the stack formed by:

• Telegraf: as a metrics collector agent from different sources (Mysql, Mongo, Redis, Apache, Cassandra, etc.).
• InfluxDB: as a TimeSeries database in charge of storing the metrics sent by the Telegraf agent and Grafana, as a tool for visualizing InfluxDB metrics.

Certification of the Platform under OpenJDK 15

Another addition to the Platform in this first Q of 2021 is the certification of the Platform under OpenJDK 15, on demand and in those environments where required. This version is the Open Source implementation of Java SE version 15, it is offered with GNU open source license, but the commercial version with support (GA) is also offered since September 2020.

Migration from Platform-integrated Elasticsearch to OpenDistro for ElasticSearch

The free version of Elasticsearch provided by Elastic, which is its developer, lacks many basic features that are necessary to do a production deployment. For example, it does not have security or SSL support for communications, etc. Therefore, we have integrated into the OpenDistro Platform, which is a distribution of Elasticsearch with Apache 2.0 license that does include security, SSL support and SQL support, and it is also maintained by Amazon and is the version used by AWS.

New TRAINING section in documentation

In this release, we have created new documentation for the TRAINING sessions, creating a basic, an advanced and an operation training, containing a presentation to guide the training and a set of video tutorials to explain each topic.

These manuals are available to users with an active Onesait Platform subscription.

Onesait Platform Community

This line of work includes the tasks we carry out that are related to the Platform Community, of which the Open Source version of Onesait Platform forms part, and also to the Platform’s different communication channels.

Throughout this first quarter of 2021, we have worked on:

Curitiba Smart Hack

We participated in the realization of this event in Brazil, providing logistical support on the Platform, as well as conducting a two-and-a-half-hour training workshop in Portuguese, which was attended by more than 100 participants:

Onesait Platform Webcast

This quarter we have conducted two new webcasts on the Platform:

One with the new features of the Platform’s 2.3.0 release:

Another on the management of Digital Twins in the Onesait Platform::

If you want to keep up to date with our webcasts, do not hesitate to join our Meetup community, where there are already more than 700 subscribers.

Community Channels

We keep on growing on social media and our web channels. Throughout the quarter, we have reached the following milestones:

• We exceeded 1,000 users in our Onesait Platform CloudLab environment, achieving more than 100 new users in this period.
• We grew by more than 35 new subscribers on our YouTube channel, surpassing 130 total subscribers, totaling more than 210 hours of viewing and achieving more than 12,000 impressions during this quarter.
• We reached 70 clones of our Platform’s Community version on GitHub , with up to 244 developers following the project.
• We continue writing in DZonewith two new publications (Making a Dashboard About COVID-19 in Portugal and Map Viewers: Main Open Source Libraries) and more than 48,000 reads of our articles.
• We maintain the pace of publication in our Blog, with 49 new posts and more than 21,000 quarterly reads.
• Finally, we continue to grow on Twitter, reaching 284 followers, with 290 tweets published, 145 retweets received and reaching almost 52,000 impressions.

CloudLab Environment Update

We have updated with the quarter’s version (version 2.3-immortal). As always, you can download it, free of charge, from our GitHub repository.

✍🏻 Author(s)