MinIO for a file repository

29/11/2021 Jesus Fernández Gómez-Pimpollo

MinIO is a cloud object storage engine compatible with the API of Amazon S3.

Offers very interesting features like:

High data availability in Active-Active configurations, with as many MinIO server distribution instances as necessary to guarantee high availability.
High performance with read and write speeds of 183 Gb/ s and 171 Gb/ s.
Active data replication, since each server maintains at least one copy of the data. You can configure how many copies of the data each server keeps.
Sharing, versioning and locking of objects.
Native deployment in public and private Cloud, based on kubernetes.
Data security and encryption.

With MinIO, we have a secure storage for our information, where each user has their own root directory, known as a «bucket», from which they can organize their information with a classic subdirectory structure and with the advantage of being able to manage different versions of the same object and share them with other users.

In the Platform, MinIO has been incorporated as a repository to offer users storage of all types of files, both structured and unstructured. This provides the Platform with two new functionalities:

File repository both for processing on the Platform (Notebooks, Dataflows, etc.) and for backing up information from applications via API S3.
Persistence engine for the creation of Datalakes, by storing files with structured information and using Presto as an SQL query engine to said files.

All users of the Platform with roles of «Developer», «Analytics» and «Administrator», can use MinIO in an integrated way in the Control Panel, accessing through the menu option Dev Tools> My Object Store.

In this screen you can perform total management of the user’s own files, as well as those files to which you have given authorization by other users. It includes features such as:

File Upload.
File Update
File Delition
Sharing Files with other users.
Copy download url via API S3.
Download/update files of other users to whom I have been authorized.
Visualize the structure of user’s bucket (directories,subdirectories and files).

Platform administrators are treated like other users in terms of having their own bucket in MinIO and being able to share files. But for the purposes of monitoring and managing the MinIO server, they have another menu option in which they have full control over the MinIO console and from which they can manage: users, buckets, security policies…

An administrator will be able to access the MinIO management console through Dev Tools> Object Store Management.