Improvements to the DataFlow module
Until now, everything related to DataFlow has been done with internal users of Dataflow Streamsets Data Collector. From version 6.1.0 of Onesait Platform we have implemented SSO with DataFlow, so that at the Streamsets level, it will be the same users of the Platform who register all the operations carried out, instead of doing so with the application’s own users.
This allows us, among other things, to have more control over who executes what. For example, we can see both in the logs and in the execution history who executes each process:
For the end user of Onesait Platform, this change is completely transparent. For the user, the operation of the DataFlow module and the development and execution of pipelines will be identical to what it has been until now.
How to configure it?
For Platform deployments with DataFlow, the configuration changes slightly.
To facilitate the configuration of DataFlow, the following properties have been added to the standard configuration file ‘/etc/sdc/sdc.properties’:
onesaitplatform.im.enabled=true
dpm.app.security.path=controlpanel/streamsets/auth
onesaitplatform.im.base.url=http://controlpanelservice:18000
onesaitplatform.controlpanel.api.base=http://controlpanelservice:18000
onesaitplatform.acl.get.url=controlpanel/api/dataflows/pipelines
onesaitplatform.admin.user.token=xxxxxxxxxx
| Property | Use |
| onesaitplatform.im.enabled | Enables the use of SSO. In deployments of version 6.1.0 and later, ‘true’ is mandatory. |
| dpm.app.security.path | URL for accessing the Platform’s security service. Specifically, it is the ULR for internal access to the Control Panel. |
| onesaitplatform.controlpanel.api.base | URL of access to the intermediate services necessary for DataFlow (mainly ACLs). Specifically, it is the URL for internal access to the Control Panel. |
| onesaitplatform.acl.get.url | Path for access to the ACL validation service. The default value does not change. |
| onesaitplatform.admin.user.token | API Key token of an administrator user of the Platform. This data changes with each installation. |
This is the default configuration, which works on all common installations. The only property that needs to be changed on each installation is the ‘onesaitplatform.admin.user.token’ property.
Next steps
This improvement will allow us in the future to configure the Onesait Platform nodes (sources, destinations and processors) to be able to perform user-dependent internal operations.
Header Image: Dan Roizer at Unsplash.
✍🏻 Author(s)
