Release 4.3.0-Quest for Onesait Platform

We start this 2023 with the last pending task we had from 2022’s Q4: the launch of the latest iteration of the Onesait Platform, which reaches version 4.3.0 and which The Community™ has decided to call «Quest», following our version policy.

As usual, if you are interested in finding out what we have been working on throughout 2022, be sure to take a look at our Roadmap, without forgetting that we can find more information about the Platform’s capabilities in the Developer Portal.

Having said that, let’s see what’s new with this new version.

Engine

In this distribution, we apply agile and low-code strategies for the development and integration of business solutions that are based on standards, compatible with different specific cloud vendors, and deployable on IaaS, SaaS or On Premise infrastructures:

• Scalability and stability of any infrastructure.
• Data-centric.
• Integrated end-to-end security.

Throughout this fourth period of the year, we have worked on the following points:

Use of Java 17 (LTS) as the Platform’s default JVM

Up until now, the Platform images used Java 8 by default, although the Platform was also certified for Java 12 and Java 15.

In this version, we have adopted OpenJDK 17 as the default JVM. This version has been chosen because it is the latest LTS version of Oracle Java SE, so it guarantees support.

Support to Google Cloud Storage in the File Repository

The File Repository is a component of the Platform that allows uploading and sharing files from the Platform itself, both from the Control Panel and from a REST API.

Until now, this component was able to work with GridFS when we had MongoDB installed, or with MinIO for installations using this distributed file system.

In this version 4.3.0, Google Cloud Storage is supported as another of the repositories on which to store the files managed by the File Repository. This scenario is very interesting for those Platform Projects that are deployed on GCP and that can use these native services.

Besides, the management user interfaces have been unified, so that from the same section you can select the storage mechanism (typically each Project will use only one):

You can find more information about it in this article on the Development Portal.

Queries Profiler UI: Dashboard for Query profiling

In release 4.2 of the Onesait Platform, we incorporate a system of metrics for queries to have more information about the queries that are executed in the system (these metrics are stored in the QueryMetrics Entity).

In this version, a new user interface has been created (built as a Dashboard) that allows a complete profiling of the queries that are executed within the platform, including the slowest queries, those that give errors, etc.

You can access this interface from the Tools > Queries Profiler UI menu:

We tell you about it in detail in this guide of the Development Portal.

Config Manager

The first version of this Architecture accelerator has been generated, which contemplates:

• The multi-language module to manage the languages and literals per language of each application:

• The master data manager of an application:

Git editor for functions in the Serverless Manager

We have created a visual component (on Vue) that allows editing of files in git.

This component has been enabled in the platform’s serverless module to facilitate the creation of functions. Thanks to this improvement, you will be able to upload, edit and select the necessary files for the execution of a function during its creation or edition.

If you are more interested in knowing how this tool works, you can check the documentation that we have prepared in this regard.

Support encrypt/decrypt attributes over Hashicorp Vault

The HashiCorp Vault is a popular tool used to securely manage sensitive information. HashiCorp Vault comes with several pluggable components called secrets engines and authentication methods that allow it to integrate with external systems. The purpose of these components is to manage and protect your secrets in dynamic infrastructures (for example, database credentials, passwords, or API keys).

In the Platform, there is a functionality that allows encrypting and decrypting attributes of an Entity, automatically, from a master key stored in the Platform. With this integration, the Vault manages this master key.

You can find more information about the integration in this entry that we have prepared in the Development Portal.

GIS Library

We have developed GIS library focused on the representation of geospatial data. For its design, it was based on all the work of gathering the needs of the different products and projects including map viewers.

For this first version, we use OpenLayers, but the library is designed in such a way that it allows the updating of OpenLayers versions without great impact, as well as to contemplate the use of other different libraries in the future (CesiumJS, Leaflet, etc.)

Besides, it has a number of basic default geoprocessings, with which to perform certain analyses (influence zones, proximity, intersection).

We have prepared a more detailed post with the features of this library.

Architecture Framework Update

This quarter, we have updated the Architecture reference framework with information referring to:

• Centralized Swagger Catalog (Microservices)
• Kafka Connect, CDC and Sink Connector
• R2DBC (Reactive Relational Database Connectivity)
• Schema Registry

GeoServer as a GIS server

In addition to generating a GIS library for the Products and Projects, in this version we have worked on the implementation of a GIS server which, according to Onesait’s reference framework for GIS, the chosen server has been GeoServer.

Specifically, we have installed GeoServer version 2.22.0 on a version 13 PostgreSQL and the version 3.1 PostGIS extension.

The server is running and serving layers in different OGC formats: WMS, TMS, WMS-C, WMTS, WFS, WCS and WPS.

We tell you about it in detail in this entry on the Development Portal.

INTELLIGENCE

We virtualize and simplify access to data between organizations, departments and systems, achieving a 360º vision of business entities and facilitating their exploitation through real-time rules, execution of complex algorithms or traditional analysis:

• Unique multi-repository data model.
• Collaborative environment for Data Scientists.
• Access, management and consumption of APIs.

We have focused our work this quarter on the following functionalities:

Integration of Presto as a multi-repository query engine

In release 3.1.0 of the Platform, we had integrated Presto+MinIO as support for DataLake-type storage for migration scenarios from Hadoop.

In this release, we have supported Presto as a multi-repository SQL query engine, which will allow you to make analytical queries on all the Entities/Ontologies of the Platform regardless of the repository where they are stored (this, for example, will allow you to do JOINs between a PostgreSQL and a MongoDB, or between a MinIO and an Oracle).

To do this, we have created a new type of Entity: Presto Entity.

This new type of Entity will allow users to connect to the different catalogs registered in Presto by the Platform manager who is creating PRESTO entities.

Once the PRESTO entities are created, you can make JOINs between them, transparently to the repository:

We explain it in greater detail in this guide at the Developer Portal.

DataClass Concept (Data Governance)

The DataClass functionality, associated with Data Governance, allows associating both to Entities (at a general level) and to their attributes. These rules allow defining validation rules and data format, thus allowing preprocessing to be executed before inserting the data.

This preprocessing will allow you to define error messages that will be inserted into the user’s audit entity, thus facilitating the visualization of failed data and load statistics on a dashboard. Rules can be created in Groovy and JavaScript.

Rules can be associated to an attribute of an Entity/Ontology, in which case they will be launched for that attribute:

Or at the Entity level, in which case all the attributes of the Entity can be used in the rule.

You can find more information in this article at the Development Portal.

REST communication support in the Dashboard Engine

The Dashboard Engine allows the creation of Dashboards from Gadgets built on DataSources, which are translated into queries on the Platform’s Entities/Ontologies. At the execution level, the browser establishes a communication with the Dashboard Engine to pass parameters and retrieve the information.

In previous versions the communication could be WebSockets or XHR in Fallback mode. For the application type, WebSockets communication is perfect because it establishes a secure channel that it then maintains between the client and the server. However, in some scenarios, this communication poses problems for peripheral security elements such as WAFs.

For this reason, a new REST communication channel has been implemented in the Dashboard Engine.

Once this Dashboard Engine work mode is activated, the operation is transparent to both the dashboard’s developer and its end user.

Destination Component in the Platform’s DataFlow CRUD

A new component has been developed for the DataFlow, that allows to perform a CRUD on one or several destination entities.

With this component, we will be able to upload data to the Platform from other sources in a simple way.

You can check the documentation that we have prepared in this regard at this link.

CENTER

Center manages the deployment and operation of our Solutions in our preferred cloud, and it provides tools to facilitate the migration to the Cloud of our Legacies:

• Simplify migrations to the Cloud.
• Centralizes the DevSecOps ecosystem.
• Manages the deployment of reusable functional accelerators.

After the initial version of the Control Center, this quarter we have focused on these lines of work:

Support to Spring Boot applications and Onesait Architecture applications

In the new release of the Center, the Spring Boot category has been added to the Assessment. This allows adding to a project, this type of applications, that are more modern and in line with the trend of decomposition of the application into microservices, from the initial inventory phase.

You can add pre-existing applications, in which case you need to indicate some information and the path to the project’s source code in Git. For a new application, the Center will archetype the new Spring Boot project as a build-ready Maven project. In this case, to create the project archetype, you can use the Onesait Architecture Initializr or the Spring.io Initializr.

We explain it in detail in this entry on the Development Portal.

ODS CLI support

Onesait Design System (ODS) provides the developers with a CLI (command line interface) tool to build Front application archetypes on top of this technology.

In this release, we have included integration with this tool in the Onesait Platform Center so that, in the Assessment phase. these Front application archetypes can be generated and included in the cycle of containerization and deployment in the Cloud managed by the Onesait Platform Center.

Infrastructure diagram

Using this new diagram, from the Center, you can easily model the infrastructure of a solution, so that the Center is in charge of orchestrating calls to Terraform and Ansible to deploy the required infrastructure:

In this link, you can find a video with an example of deploying some VMs and networks on GCP.

Integrated Git file editor

A Center project is associated with a Git repository where all project-related code is stored and versioned. To make it easier to edit the project files, a Git editor has been integrated into the project itself, and is accessible from the project tabs:

Once you are in the section, you can select a branch of your repository:

And automatically the directory tree of the editor is loaded; from there you can choose any file to edit:

We explain this greater detail in this article that we have prepared in the Development Portal.

THINGS

This distribution of the Platform supports the development of IoT systems, both in the Cloud and in the Edge fields.

Ansible-based deployment

The deployment of services in the field is a main pillar in node management. For this reason, new remote orchestration systems have been included, improving and expanding the capabilities of Onesait Edge.

Until now, remote deployment was based on the use of docker-compose as an orchestrator for Edge services. In this release, Onesait Edge incorporates deployment methods by updating the IoT agent (iota), allowing installation using Ansible.

Red Hat Operating Systems Certification

We have worked with the Red Hat team for Onesait Edge deployment certification in their operating systems, both Cloud and Edge.

Deployment over Openshift

The deployment of Onesait Edge in Openshift has been validated, making some changes with the Helm used in Kubernetes, adapting them to the peculiarities of Openshift. Some of the images used have been modified for the use of systems certified by Red Hat (OpenJDK, Node. js etc).

RHEL for Edge support

Red Hat has a lightweight operating system for remote field deployment.

From Onesait Edge, complete integration and validation with Rhel for Edge has been carried out: base image installation with IoT agent (IoTA), orchestration through Podman, hardware and software validation (specific programs, modems from several manufacturers).

Field accelerators

The ease to perform complex operations in the field and implement artificial intelligence, is an essential requirement for Onesait Edge, which is why the system has been integrated with Nvidia Jetson accelerators. Each Nvidia Jetson is a complete System on Module (SOM) that includes GPU, CPU, memory, power management, high-speed interfaces, sensors, etc.

These Jetson machines are compatible with the most popular AI frameworks on the market: TensorFlow, PyTorch, Caffe, Keras, MXNet and with SDKs for new developments. This integration made with Jetson systems allows Onesait Edge not only to execute more complex AI in the field, but also to evolve it.

The equipment used has been the Jetson Nano, whose main capabilities are:

• NVIDIA CUDA® architecture: Maxwell™ GPU 128 cores.
• Quad-core ARM® Cortex®-A57 MPCore CPU.
• AI: High performance AI computing with GPU processing.
• Ideal for smart city applications with edge AI.
• PoE GbE 15 W support for cameras.

DEMONSTRATORS

Here we tell you about the latest demonstrators that we have developed on the Platform, and that have served to teach clients some of the Platform’s capabilities.

Demonstrator of ingestion, correlation and action on systems

This demonstrator is responsible for the detection, notification and solution of problems in multiple remote systems mounted on a topological network with the information given by the raw logs of these systems.

To do this, we use modules such as DataFlow to ingest the logs, Kafka:

FlowEngine is used to launch correction processes on remote systems:

Graph Entities to model the service topology:

And Dashboards for information exploitation:

The complete detail on the demonstrator is explained in this link.

IBM Maximo Integration Demonstrator

This demonstrator allows for the exploitation of data stored in IBM Maximo from the Platform. In the demonstrator, it was decided to connect with the Maximo database; to do this the DataFlow module was used to connect with the Maximo tables, extract the information and store it in the Platform entities:

Subsequently, it was exploited in a Dashboard:

You can find the full scope of the demonstrator in this link.

Cloud Migration Demonstrator with Center

In this demonstrator, we start with a traditionally-deployed legacy application and we have to containerize and template it in a Kubernetes cluster through Charts Helm:

Finally, it is deployed on OpenShift on Google Cloud Platform:

We have more information about this demonstrator in this article on the Development Portal.

EXPERIENCE

We connect research and user-centered design with the construction and development of interfaces using the latest technologies. In this fourth Q, we have worked on:

Conversion of Onesait Design System components to Web Components

We have made the approach, design, adaptation and development of ODS (Onesait Design System) components to Web Components, including a storybook with the available Web Components and a migration policy for existing systems.

As the first project built on these Web Components, a Chatbot has been created:

Evolutionaries and ODS maintenance

• ODS XR, creation of a reusable onboarding scenario for virtual reality projects and adaptation of the ODS, and creation of interaction resources for a VR projector.
• Component evolution and support: accessibility improvements, style evolution, component design fixes such as Steps, tabs, popover, dropdown, Action menu, etc.
• Libraries and methodology, adaptation of libraries and documentation of new processes, covers and file organization in Figma.
• Support and documentation, evolution of the documentation by components in more detail.
• Design tokens approach for interactions and animations.

COMMUNITY

Onesait is committed to Open Source technologies to create a community and ensure independence. Throughout this fourth quarter of 2022, we have:

• Reached 1,250 users in the Onesait Platform CloudLab, our free and experimental environment.
• Reached 175 subscribers on our YouTube channel, adding +830 total viewing hours and exceeding 100,000 impressions.
• Our Community version of the Platform on GitHub exceeds +1200 clones, and with a total of 258 developers currently following the project.
• The Blog (here!) adds a total of +320,000 readings of our articles, increasing this quarter by +65,000 readings.
• Surpassed +340,000 total impressions on Twitter, with almost 170,000 visits to our profile, +900 retweets and almost +1,400 likes from our tweets.
• Published 4.3.0-CE release: in the coming weeks, the 4.3.0 Community version of the Platform will be available in our Github repository.

• Updated the CloudLab instance with version 4.2.0-Predator.

✍🏻 Author(s)