Defining Automations
Save time and improve efficiency with automated model workflows
Overview
JFrog ML offers built-in automations for various recurring actions in the model lifecycle. These automations are meant to replace external orchestration tools such as Airflow for example.
In the JFrog ML platform, you can easily define automations for model retraining or batch model executions on either time based or trigger based.
Configuring automations
The automation name must be unique throughout your models within the JFrog ML environment.
We now create an instance of the Automation class and configure it:
from qwak.automations import Automation, ScheduledTrigger, QwakBuildDeploy,\
BuildSpecifications, BuildMetric, ThresholdDirection, DeploymentSpecifications
test_automation = Automation(
name="retrain_my_model",
model_id="my-model-id",
trigger=ScheduledTrigger(cron="0 0 * * 0"),
action=QwakBuildDeploy(
build_spec=BuildSpecifications(git_uri="https://github.com/org_id/repository_name.git#dir_1/dir_2",
git_access_token_secret="token_secret_name",
git_branch="main",
main_dir="main",
tags=["prod"],
env_vars=["key1=val1", "key2=val2", "key3=val3"]),
deployment_condition=BuildMetric(metric_name="f1_score",
direction=ThresholdDirection.ABOVE,
threshold="0.65"),
deployment_spec=DeploymentSpecifications(number_of_pods=1,
cpu_fraction=2.0,
memory="2Gi",
variation_name="B")
)
)
Scheduler Timezone
The default timezone for the cron scheduler is UTC.
Selecting an instance type
You can specify the purchase_option
parameter of the BuildSpecifications
to select between on-demand and spot instances. Available values: spot
and ondemand
. For example:
...
build_spec=BuildSpecifications(
git_uri="https://github.com/org_id/repository_name.git#dir_1/dir_2",
git_access_token_secret="token_secret_name",
git_branch="main",
main_dir="main",
tags=["prod"],
env_vars=["key1=val1", "key2=val2", "key3=val3"],
purchase_option="ondemand"
),
...
spot
is the default value if you don't specify anything.
Triggering automations
Automations may be configured to either compare the model's performance with a pre-defined threshold and deploy the model when the evaluation results pass.
We can trigger the automation in two ways:
Schedule-based triggers
Triggering an automation based on an interval name or a cron expression.
In the case of an interval configuration, the trigger configuration would look like this:
from qwak.automations import ScheduledTrigger
# Valid values: Daily, Weekly, Hourly
ScheduledTrigger(interval="Daily")
Metric-based triggers
To retrain the model based on production performance metrics, we can use the MetricBasedTrigger
.
In this case, we need to specify a SQL query which should return the metric value from JFrog ML model Analytics:
from qwak.automations import MetricBasedTrigger
MetricBasedTrigger(
name='metric_name',
metric=SqlMetric(sql_query='SQL_QUERY'),
direction=ThresholdDirection.ABOVE,
threshold="0.7"
)
Notifications
JFrog ML supports configuring notifications in case of an error or success using either Slack Webhooks or a custom webhook configuration
Slack Webhook
from qwak.automations import SlackNotification
test_automation = Automation(
name="my_automation",
model_id="my_model",
trigger=ScheduledTrigger(cron="0 0 * * 0"),
action=QwakBuildDeploy(
build_spec=BuildSpecifications(...),
deployment_condition=BuildMetric(...),
deployment_spec=DeploymentSpecifications(...)
),
on_error=SlackNotification(webhook="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"),
on_success=SlackNotification(webhook="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX")
)
It will send a slack message when the automation is finished - containing the execution time, the model, the automation name, and the final status of the automation.
You can define alerting notifications for either one of the
on_error
andon_success
triggers, or for both.on_success
will trigger after a successful Build, if the deployment threshold was not met and after a successful Build and Deploy, if the threshold was met.
Custom Webhook
Another option is to be notified to a custom webhook, using the following configuration:
from qwak.automations import CustomWebhook
on_success=CustomWebhook(
url="http://my.api/endpoint/v1",
http_method="GET", # defaults to "GET"
data={"a1": "b1", "a2": "b2"},
headers={"content-type": "application/json"}
)
The definition above will call the API (defined by the url) above with the following parameters:
- Execution time (
execution_date
) - Model ID (
model_id
) - Automation name (
automation
) - Execution Status (
status
) - Build ID - (
build_id
)
In addition to the fields explicitly defined in the data
field.
If the HTTP method defined is GET
- the data
field plus the JFrog ML parameters above will be embedded as request parameters. In all other methods - in the body to attached to the request.
Registering the Automation
Now, we can register the automation using the JFrog ML CLI:
qwak automations register -p .
In the command above, we specified the name of the JFrog ML environment (--environment
) and the directory containing the automation definitions (-p
). In this case, the current working directory.
Updated 2 months ago