Secret Management

Securely managing sensitive data and credentials in complex data projects is crucial.

Use secrets to avoid confidential information in model deployments and data source connections.

In traditional approaches, using credentials in an ML build involved including them in the Python code or using environment variables. Both options pose significant security risks.

Using the Qwak Secret Service, you can easily and securely store your credentials and pass them to your Python code with full confidentiality.

🚧

Secret naming conventions

Secret names follow the valid DNS format, which should be all lowercase, a minimum of 3 characters and up to 100 characters without underscores.

Creating Secrets via UI

Secrets may be created via the application by visiting the Secrets page under Account Settings to view, create or delete credentials.

Creating Secrets via CLI

Secret names may be up to 36 characters and only contain letters, numbers and dash ("-"), and should start with a letter.

Create secrets via the Qwak CLI with the following commands:

qwak secrets set --name <aws-api-key> --value <the_value_of_the_key>
qwak secrets set --name <aws-api-secret> --value <the_value_of_the_secret>

Retrieving Credentials in a build

You may need the credentials during a build process. For example, retrieving a pre-trained model or data that was not stored in Qwak's Feature Store. To retrieve the credentials, import the SecretServiceGrpcClient and use it to retrieve the secret:

from qwak import QwakModel
from qwak.clients.secret_service import SecretServiceClient

class TestModel(QwakModel)
	# ...
    
  def build():
      secret_service = SecretServiceClient()
      aws_api_key = secret_service.get_secret('aws_api_key')
      aws_secret_key = secret_service.get_secret('aws_secret_key')

❗️

Protecting Secrets

Avoid printing or logging secret values as the model stdout is visible on the build logs.

Retrieving Credentials in the Feature Store

Qwak's Feature Store integrates with the Secret Service to enable secure access to data sources. Once you've created a new secret, use it in the data source definition to ensure secure authorized access.

Connecting to Snowflake Using Secrets

First we need to create new secrets with the user name and password.

qwak secrets set --name qwak_secret_snowflake_user --value <snowflake_user>
qwak secrets set --name qwak_secret_snowflake_password --value <secured_password_1234>

Define the Snowflake data source using the secret names:

from qwak.feature_store.sources.data_sources import SnowflakeSource

snowflake_source = SnowflakeSource(
    name='my-snowflake-datasource',
    description='An example snowflake data source',
    date_created_column='DATE_COLUMN',
    host='<SnowflakeAddress/DNS:port>',
    username_secret_name='qwak_secret_snowflake_user', # secret name, stored in the Secret Service
    password_secret_name='qwak_secret_snowflake_password', # secret name, stored in the Secret Service
    database='snowflake_db_name',
    schema='snowflake_schema_name',
    warehouse='snowflake_data_warehouse_name'
)