Building a Qwak Model
At Qwak, a build is an ML equivalent of a software build — a trained and serialized model instance. Builds are the actual deployable entities in the Qwak ecosystem.
The Build Lifecycle
A Qwak build process is composed mainly of the following steps:
- Creating the model's virtual environment
- Executing the build function
- Running unit and integration tests
- Serializing the model
- Building a serving docker image
- Pushing it to Qwak's model registry
In the following sections we will see how to create a customize the Qwak build process.
Creating a Basic Build
Create a Qwak-based model in your local environment. You can use an example of a Qwak-based model by running the following command:
qwak models init \
--example titanic \
<dest>
<dest>
- Qwak based model local directory path.
Build the Qwak-based model using the following command:
qwak models build \
--model-id <model-id> \
<uri>
<model-id>
- Qwak model id, Which exists in one of you projects.
<uri>
- Qwak based model URI.
Creating a Remote Build
If you build your model on your local machine, available resources may be limited.
By using a remote build you will gain the following benefits:
- A stable network (within the cluster).
- Configurable RAM size (via the CLI)
- Configurable CPU / GPU usage (via the CLI)
- Fewer local dependencies. There will be no need for Docker in the localhost.
Right now, all builds are remote by default. If you want to switch to a local build add the --no-remote
option to the build command.
Controlling the Available Resources
While using a remote build, you can control the number of CPUs, amount of available RAM, or get access to a GPU using the following options:
- CPU limit - Configure the CPU limit by using the following flag
--cpus
- RAM limit - Configure the memory limit values by using the following flag
--memory
The configuration units are defined by k8s docs.
- GPU type - Configure the GPU type by using the following flag
--gpu-type
(available types:NVIDIA_K80
,NVIDIA_V100
) - GPU amount - Configure the amount of GPU's by using the following flag
--gpu-amount
The configuration units are defined by AWS docs (EC2 Accelerated Computing - P2, P3).
(available amounts per type: NVIDIA_K80 - 1 / 8 / 16, NVIDIA_V100 - 1 / 4 / 8).
Remote Build Logs
The remote build will run asynchronously and therefore you will not see the logs on the screen while the build is being performed. There are two options for viewing the logs:
- Qwak UI (Build page):
![qwak_build.png 1815](https://files.readme.io/96f88ef-qwak_build.png)
- Follow the logs by using the following command:
qwak models builds logs \
-b <build-id> \
--follow
<build-id>
- The build ID that you received when executing the build remotely.
Using Multiple AWS Profiles
The AWS profile that you wish to use while interacting with the Docker registry, can be explicitly configured by using the --aws-profile
flag. For example:
qwak models build \
--model-id <model-id> \
--aws-profile <profile-name> \
<uri>
<model-id>
- Model ID that the build should be related to.
<profile-name>
- The AWS profile name, as shown in ~/.aws/config
.
<uri>
- Qwak-based model URI.
If you choose not to specify which AWS profile you wish to use, the default AWS profile will be used.
Using Custom AWS IAM Role
In some cases, a model needs to access external services during its build process.
If a build needs to access external AWS resources, a custom AWS IAM role can be passed to the Qwak build process.
The IAM role should be created with the following trust policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account-id>:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"ArnLike": {
"aws:PrincipalArn": "arn:aws:iam::<account-id>:role/eksctl-Qwak-EKS-Cluster-nodegroup-*"
}
}
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account-id>:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"ArnLike": {
"aws:PrincipalArn": "arn:aws:iam::<account-id>:role/qwak-eks-base*"
}
}
}
]
}
The IAM role ARN can be passed directly to a build using the --iam-role-arn flag. For example:
qwak models build \
--model-id <model-id> \
--remote \
--iam-role-arn arn:aws:iam::<account-id>:role/<role-name> \
<uri>
The Supplied IAM role ARN role can be accessed from the model code using the following code example:
from boto3
session = boto3.Session(profile_name='user-provided-role')
# Example for an S3 client:
s3 = session.client('s3')
Tagging a Build
Tags can be attached to specific builds for later identification:
qwak models build \
--model-id <model-id> \
-T <tag_1> -T <tag_2> \
<uri>
<model-id>
- model id which the build should be related to.
<tag>
- Tag name as string.
<uri>
- Qwak based model URI.
Build config is a yaml file that describes the build command configuration. It is often useful when integrating a build command into a CI/CD pipeline.
Generating a Config File from a Command
To generate a config file from a command that you already use, specify the --outconf
flag, which will print to the console a ready config based on the options you specified. For example assume you run the following command:
qwak models build \
--model-id test_model \
-T tag_1 -T tag_2 \
-E OS=WINDOWS \
-E VERSION=3.9 \
~/model
Here is an example of the output:
build_env:
docker:
base_image: qwak-base:0.0.18-cpu
build_args: {}
env_vars: ["OS=WINDOWS", "VERSION=3.9"]
no_cache: true
params: []
local:
aws_profile: null
no_push: true
python_env:
conda:
conda_file: conda.yml
git_credentials: null
git_credentials_secret: null
poetry: null
qwak_sdk_extra_index_url: null
virtualenv: null
remote:
is_remote: false
resources:
cpus: 2.0
memory: 4Gi
build_properties:
branch: main
build_id: f139b71a-533a-11ec-a934-367dda8b746f
model_id: test_model
model_uri:
git_branch: master
git_credentials: null
git_credentials_secret: null
main_dir: main
uri: ~/model
tags:
- tag1
- tag2
To save the configuration into redirection , For example: qwak models build --model-id test_model ~/model --out-conf > config.yml
Customizing Your Build
Custom Docker Image
You can use a custom Docker image for the build process. The custom base image will also be used for the serving image.
When executing a build, the image should be specified using the --base-image
flag:
qwak models build \
--model-id <model> \
--base-image <image-id> \
<dest>
<model-id>
- The model ID associated with this build.
<image-id>
- Docker image ID.
<uri>
- Qwak-based model URI.
The custom docker image should be based on: public.ecr.aws/w8k8y6b6/qwak-base:0.0.18-cpu
or public.ecr.aws/w8k8y6b6/qwak-base:0.0.6-gpu
(in case of models requiring a GPU).
The Docker image entry point shouldn't be changed.
Environment Variables
Configuring environment variable in build and serving environment, The environment variables should specified with the flag -E
when executing a build:
qwak models build \
--model-id <model> \
-E <key>=<value> -E <key>=<value> \
<dest>
<model-id>
- model id which the build should be related to.
<key>
- Environment variable key.
<value>
- Environment variable value.
<uri>
- Qwak based model URI.
Customizing the Main Directory
Given the following Qwak-based model directory structure:
qwak_based_model/
├── main/
├── tests/
You can change the main/
directorie’s name. For example, changing its name to iris_classifier/
:
qwak_based_model/
├── iris_classifier/
├── tests/
To configure the name in the main directory, use a build config as follows:
build_properties:
model_uri:
main_dir: <new-directory-name>
So for the above example, the build config would be:
build_properties:
model_uri:
main_dir: iris-classifier
Installing packages using a pypi repositories
During the build process you can download and use packages from private repositories.
Create an environment variable PIP_EXTRA_INDEX_URL .
qwak models build --env-vars PIP_EXTRA_INDEX_URL=https://USERNAME:PASSWORD@JFROG_ARTIFACTORY_URL
You can also send this data not in clear text, by creating local environment variables:
export JFROG_USERNAME=<USERNAME>
export JFROG_PASSWORD=<PASSWORD>
qwak models build --env-vars PIP_EXTRA_INDEX_URL=https://${JFROG_USERNAME}:${JFROG_PASSWORD}@JFROG_ARTIFACTORY_URL
Updated over 1 year ago