Training course (part II): DataFlow

19/08/2020 Fco. Javier López Acevedo

Last May, the «Intelligence and Data» team held an online training course on the Onesait Platform, of which we told you about here on the blog (part I and part II).

Due to the interest generated by the course, we have decided to create a number of explanatory videos with the content taught during the course, so that, over the next few weeks, we will be bringing you the different videos of the course.

The course is organized with the following sections:

Ontologies
DataFlow
FlowEngine
Notebooks
APIs and Tokens
SQL
Introduction to MongoDB
Datasources
Dashboards and Gadgets
My Files, Web Projects, My Projects
JasperReports

Important: As the course videos were recorded in Spanish, we have added subtitles both in Spanish and English for your convenience.

2.- DataFlow

To continue learning about the Onesait Platform, in this module we are going to study the DataFlow tool, which allows ETL processes to be carried out both in streaming and in batch. Therefore, it is quite important to know this tool within the Platform as it open endless doors.

This video will give you a tour of the following points:

What is the DataFlow?
How it works: general knowledge about the operation of the tool.
Types of stages and connectors: shows and explains some of the options available for each of the stages that make up an ETL.
How to activate and define alerts and rules within the flow, in order to establish data quality controls on our ETL processes.
Monitoring: brief explanation about the monitoring screen and the opportunities it provides to monitor our flows.
To end the video, a practical example will be made in which all the aforementioned points will be seen in more detail, in which data in .csv format will be extracted from a URL, a number of transformations will be made to that data, and they will be finally loaded in an ontology of the Platform.