Integration with Vault HashiCorp

15/02/202317/02/2023 Fco. Javier López Acevedo

The Onesait Platform has the functionality of being able to encrypt and decrypt attributes of an Entity, automatically from a master key stored in the Platform. Well, since version 4.3.0-Quest, we have integrated Vault HashiCorp so that the encryption and decryption of entity data is delegated to it, so that it is the Vault that manages the data encryption key:

To see an example, let’s look at this Entity that has the attributes «file» and «stages» tagged as encryptable:

When querying the data, we will see plain data, since the Vault is being used below to encrypt (and decrypt):

However, at rest, this information will be encrypted:

What is HashiCorp Vault

HashiCorp’s Vault is a popular tool used to securely manage sensitive information in today’s applications.

As you can see in the image below, HashiCorp Vault has several pluggable components called «secrets engines» and «authentication methods» that allow it to integrate with external systems. The purpose of these components is to manage and protect secrets in dynamic infrastructures (for example, database credentials, passwords, or API keys).

What does a Vault solve?

Before we look at the Vault in detail, we must understand the problem it tries to solve: the management of sensitive information.

Most applications need to access sensitive data to work. For example, an application may have a username/password configured somewhere to connect to its database, or it may need API keys to integrate with other service providers such as payment gateways, logistics, etc.

Database credentials and API keys are two examples of sensitive information that we need to store and make available to our applications securely.

A simple solution is to store those credentials in a configuration file, and read them at boot time, although the problem with this approach is obvious: whoever has access to this file, shares the same database privileges as our application.

We can also encrypt these files, making things a little more difficult, although, since our application will know how to decrypt them, it will only give a false sense of security.

Today’s applications and cloud environments add additional complexity: distributed services, multiple databases, messaging systems, etc., have sensitive information scattered everywhere, increasing the risk of a security breach.

This is where a Vault fits in.

HashiCorp Vault addresses the problem of managing sensitive information (secret in Vault slang). In this context, «manage» means that Vault controls all aspects of sensitive information: its generation, storage, use, and last but not least, its revocation.

HashiCorp offers two versions of Vault: the open-source version and a paid version, which includes technical support on different SLAs and additional features, such as a HSM (Hardware Security Module) support.

HashiCorp Vault Architecture

Vault’s architecture is simple. Its main components are:

A persistence backend: for the storage of all the secrets.
An API server: which manages client requests and performs operations with the secrets.
A number of secret engines: one for each type of secret supported.

By delegating all secret management to Vault, we can mitigate some security issues:

Our apps no longer have to store them, just request them from Vault when needed, then throw them away.
We can use short-lived secrets, thereby limiting the «window of opportunity» in which an attacker can use a stolen secret.
Vault encrypts all data with an encryption key before writing it to the vault. This encryption key is encrypted by another key: the master key, which is only used at startup.

A key point in the Vault implementation is that it does not store the master key on the server. This means that not even Vault can access the saved data after boot. At this point, a Vault instance is said to be in a «sealed» state. Once «unsealed», Vault will be ready to accept API requests.

Authentication

To access the secrets in the Vault, the client must authenticate using one of the supported methods. The easiest method uses tokens (strings that are sent on each API request using a special HTTP header).

When first installed, Vault automatically generates a «root token». This token is the equivalent of the root superuser on Linux systems, so its use should be kept to a minimum. As a best practice, we should use this root token only to create other less privileged tokens, and then revoke it (this is not a problem, however, since we can later generate another root token using unseal keys).

Vault also supports other authentication mechanisms such as LDAP, JWT or TLS Certificates. All of these mechanisms are based on the basic token mechanism: once Vault validates our client, it will provide a token that we can use to access other APIs.

Tokens have some associated properties. The main ones are:

Whether it can be renewed or not.
Maximum usage count.
Unless otherwise stated, tokens created by Vault will form a parent-child relationship. A child token can have, at most, the same privilege level as its parent, although it is usual to create a child token with restrictive policies.
Another key point about this relationship: when we invalidate a token, all child tokens and their descendants are also invalidated.

Associated Policies

Policies define exactly which secrets a client can access and what operations they can perform on them. Let’s see what a simple policy looks like:

path "secret/accounting" {
    capabilities = [ "read" ]
}

Here we have used the HCL (HashiCorp Configuration Language) syntax to define our policy. Vault also supports JSON for this purpose.

Policies in Vault are «denied by default». A token associated with this example policy will have access to the secrets stored in secret/accounting and nothing else. At creation time, a token can be associated with multiple policies. This is very useful because it allows us to create and test smaller policies, and then apply them as needed.

Las políticas en Vault son «denegadas por defecto». Un token asociado a esta política de ejemplo tendrá acceso a los secretos almacenados en secret/accounting y nada más. En el momento de la creación, se puede asociar un token a varias políticas. Esto es muy útil porque nos permite crear y probar políticas más pequeñas y luego aplicarlas según sea necesario.

If we update a given policy, all associated tokens will be affected immediately.

The policies described so far are also called ACL Policies. Vault also supports two additional policy types: EGP and RGP Policies, only available in paid versions.

When available, they allow us to factor additional attributes such as time of day, multi-factor authentication, customer network origin, etc. into our policies. For example, you can define a policy that allows access to a certain secret only during business hours.

Types of secrets

Vault supports a number of different secret types that serve different use cases:

Key-Value: simple static key-value pairs.
Dynamically generated credentials: generated by Vault at the request of a client.
Cryptographic keys: used to perform cryptographic functions with customer data.

Each type of secret is defined by the following attributes:

A mount point: which defines its REST API prefix.
A set of exposed operations: through the corresponding API.
A set of configuration parameters.

A given secret instance can be accessed via a path, similar to a directory tree on a file system. The first component of this path corresponds to the mount point where all secrets of this type are located.

For example, the string secret/my-application corresponds to the path where we can find key-value pairs for my-application.

Key-Value Secrets

Key-value secrets are pairs available under a given path.

For example, we can store the pair «foo=bar» in the path /secret/my-application and later, we can use the same path to retrieve the same pair or pairs: multiple pairs can be stored in the same path.

Vault supports three types of key-value secrets:

Non-versioned key pairs: where updates replace existing values.
Versioned key pairs: retaining up to a configurable number of old versions.
Cubbyhole: a special type of unversioned key pairs whose values are circumscribed to a given access token.

Key-value secrets are static by nature, so there is no concept of expiration associated with them. In these cases, credential updates are a semi-manual process, typically requiring someone to acquire new credentials and use the Vault command line or user interface to enter the new values.

The main use case for this type of secret is to store credentials to access external systems, such as API keys.

Dynamically generated secrets

Dynamic secrets are generated on the fly by Vault when requested by an application.

Vault supports several types of dynamic secrets, including the following:

Database credentials.
SSH key pairs.
X.509 certificates
AWS credentials.
Google Cloud service accounts.
Active Directory accounts.

They all follow the same usage pattern. First, we configure the secrets engine with the necessary details to connect to the associated service. Next, we define one or more roles, which describe the creation of the secret itself.

Let’s take the database secrets engine as an example:

First, we need to configure Vault with all the user database connection details, including the credentials of a pre-existing user with administrator privileges to create new users.
Next, we create one or more roles (Vault roles, not database roles) containing the actual SQL statements used to create a new user. These typically include not only the user create statement, but also all grant statements needed to access schema objects (tables, views, etc.).
When a client accesses the corresponding API, Vault will create a new temporary user in the database using the statements provided, and will return their credentials. The client can then use those credentials to access the database during the period defined by the time-to-live attribute of the requested role.
Once a credential reaches its expiration time, Vault will automatically revoke any privileges associated with this user. A customer can also ask Vault to renew those credentials. The refresh process will only occur if supported by the specific database driver and allowed by the associated policy.

Cryptographic keys

This engine handles cryptographic functions like encryption, decryption, signing, etc. All of these operations use cryptographic keys generated and stored internally by Vault. Unless explicitly told to do so, Vault will never expose a given cryptographic key.

The associated API allows customers to send Vault data in plain text and receive an encrypted version of them. The opposite is also possible: we can send encrypted data and receive back the original text.

Currently, there is only one such engine: the Transit engine. This engine supports the most popular key types, such as RSA and ECDSA, and also supports converged encryption. When this mode is used, a given plaintext value always results in the same ciphertext.

For example, we can use this mode to encrypt credit card numbers in a transaction log table. With converged encryption, every time we insert a new transaction, the encrypted value of the credit card would be the same, allowing normal SQL queries to be used for reporting, searching, etc.

Header image: Vault HashiCorp.