New Features

Improvements to the DataFlow module

Until now, everything related to DataFlow has been done with internal users of Dataflow Streamsets Data Collector. From version 6.1.0 of Onesait Platform we have implemented SSO with DataFlow, so that at the Streamsets level, it will be the same users of the Platform who register all the operations carried out, instead of doing so with the application’s own users.

This allows us, among other things, to have more control over who executes what. For example, we can see both in the logs and in the execution history who executes each process:

Historical: each execution reflects the user who launched it.
Logs: reflect the user who launched the process.
Pipeline Manager: shows users of the Platform.

For the end user of Onesait Platform, this change is completely transparent. For the user, the operation of the DataFlow module and the development and execution of pipelines will be identical to what it has been until now.

How to configure it?

For Platform deployments with DataFlow, the configuration changes slightly.

To facilitate the configuration of DataFlow, the following properties have been added to the standard configuration file ‘/etc/sdc/sdc.properties’:

onesaitplatform.im.enabled=true
dpm.app.security.path=controlpanel/streamsets/auth
onesaitplatform.im.base.url=http://controlpanelservice:18000
onesaitplatform.controlpanel.api.base=http://controlpanelservice:18000
onesaitplatform.acl.get.url=controlpanel/api/dataflows/pipelines
onesaitplatform.admin.user.token=xxxxxxxxxx

PropertyUse
onesaitplatform.im.enabledEnables the use of SSO. In deployments of version 6.1.0 and later, ‘true’ is mandatory.
dpm.app.security.pathURL for accessing the Platform’s security service. Specifically, it is the ULR for internal access to the Control Panel.
onesaitplatform.controlpanel.api.baseURL of access to the intermediate services necessary for DataFlow (mainly ACLs). Specifically, it is the URL for internal access to the Control Panel.
onesaitplatform.acl.get.urlPath for access to the ACL validation service. The default value does not change.
onesaitplatform.admin.user.tokenAPI Key token of an administrator user of the Platform. This data changes with each installation.

This is the default configuration, which works on all common installations. The only property that needs to be changed on each installation is the ‘onesaitplatform.admin.user.token’ property.

Next steps

This improvement will allow us in the future to configure the Onesait Platform nodes (sources, destinations and processors) to be able to perform user-dependent internal operations.


Header Image: Dan Roizer at Unsplash.

✍🏻 Author(s)

Leave a Reply

Your email address will not be published. Required fields are marked *