Defining Automations

Overview

JFrog ML offers built-in automations for various recurring actions in the model lifecycle. These automations are meant to replace external orchestration tools such as Airflow for example.

In the JFrog ML platform, you can easily define automations for model retraining or batch model executions on either time based or trigger based.

Configuring automations

❗️

The automation name must be unique throughout your models within the JFrog ML environment.

We now create an instance of the Automation class and configure it:

from qwak.automations import Automation, ScheduledTrigger, QwakBuildDeploy,\
    BuildSpecifications, BuildMetric, ThresholdDirection, DeploymentSpecifications

test_automation = Automation(
    name="retrain_my_model",
    model_id="my-model-id",
    trigger=ScheduledTrigger(cron="0 0 * * 0"),
    action=QwakBuildDeploy(
        build_spec=BuildSpecifications(git_uri="https://github.com/org_id/repository_name.git#dir_1/dir_2",
                                       git_access_token_secret="token_secret_name",
                                       git_branch="main",
                                       main_dir="main",
                                       tags=["prod"],
                                       env_vars=["key1=val1", "key2=val2", "key3=val3"]),
        deployment_condition=BuildMetric(metric_name="f1_score",
                                         direction=ThresholdDirection.ABOVE,
                                         threshold="0.65"),
        deployment_spec=DeploymentSpecifications(number_of_pods=1,
                                                 cpu_fraction=2.0,
                                                 memory="2Gi",
                                                 variation_name="B")
    )
)

📘

Scheduler Timezone

The default timezone for the cron scheduler is UTC.

Selecting an instance type

You can specify the purchase_option parameter of the BuildSpecifications to select between on-demand and spot instances. Available values: spot and ondemand. For example:

... 
build_spec=BuildSpecifications(
  git_uri="https://github.com/org_id/repository_name.git#dir_1/dir_2",
  git_access_token_secret="token_secret_name",
  git_branch="main",
  main_dir="main",
  tags=["prod"],
  env_vars=["key1=val1", "key2=val2", "key3=val3"],
  purchase_option="ondemand"
),
 ...

spot is the default value if you don't specify anything.

Triggering automations

Automations may be configured to either compare the model's performance with a pre-defined threshold and deploy the model when the evaluation results pass.

We can trigger the automation in two ways:

Schedule-based triggers

Triggering an automation based on an interval name or a cron expression.

In the case of an interval configuration, the trigger configuration would look like this:

from qwak.automations import ScheduledTrigger

# Valid values: Daily, Weekly, Hourly
ScheduledTrigger(interval="Daily")

Metric-based triggers

To retrain the model based on production performance metrics, we can use the MetricBasedTrigger.

In this case, we need to specify a SQL query which should return the metric value from JFrog ML model Analytics:

from qwak.automations import MetricBasedTrigger

MetricBasedTrigger(
    name='metric_name',
    metric=SqlMetric(sql_query='SQL_QUERY'),
    direction=ThresholdDirection.ABOVE,
    threshold="0.7"
)

Notifications

JFrog ML supports configuring notifications in case of an error or success using either Slack Webhooks or a custom webhook configuration

Slack Webhook

from qwak.automations import SlackNotification


test_automation = Automation(
    name="my_automation",
    model_id="my_model",
    trigger=ScheduledTrigger(cron="0 0 * * 0"),
    action=QwakBuildDeploy(
        build_spec=BuildSpecifications(...),
        deployment_condition=BuildMetric(...),
        deployment_spec=DeploymentSpecifications(...)
    ),
    on_error=SlackNotification(webhook="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"),
    on_success=SlackNotification(webhook="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX")
)

It will send a slack message when the automation is finished - containing the execution time, the model, the automation name, and the final status of the automation.

📘

You can define alerting notifications for either one of the on_error and on_success triggers, or for both. on_success will trigger after a successful Build, if the deployment threshold was not met and after a successful Build and Deploy, if the threshold was met.

Custom Webhook

Another option is to be notified to a custom webhook, using the following configuration:

from qwak.automations import CustomWebhook

on_success=CustomWebhook(
    url="http://my.api/endpoint/v1",
    http_method="GET", # defaults to "GET"
    data={"a1": "b1", "a2": "b2"},
    headers={"content-type": "application/json"}
)

The definition above will call the API (defined by the url) above with the following parameters:

• Execution time (execution_date)
• Model ID (model_id)
• Automation name (automation)
• Execution Status (status)
• Build ID - (build_id)

In addition to the fields explicitly defined in the data field.
If the HTTP method defined is GET - the data field plus the JFrog ML parameters above will be embedded as request parameters. In all other methods - in the body to attached to the request.

Registering the Automation

Now, we can register the automation using the JFrog ML CLI:

qwak automations register -p .

In the command above, we specified the name of the JFrog ML environment (--environment) and the directory containing the automation definitions (-p). In this case, the current working directory.