There are a few options to manage the lifecycle of batch executions.
To check the current status of an execution, use the following command:
qwak models execution status --execution-id <execution-id>
from qwak.models.executions.logic.client import BatchJobManagerClient from qwak.models.executions.logic.results import ExecutionStatusResult batch_job_manager_client = BatchJobManagerClient() status_response: ExecutionStatusResult = batch_job_manager_client.get_execution_status("<execution-id>") status = status_response.status
execution_id is returned when an execution is created, and is also visible in the UI.
To cancel an execution, use the following command:
qwak models execution cancel --execution-id <execution-id>
from qwak.models.executions.logic.client import BatchJobManagerClient batch_job_manager_client = BatchJobManagerClient() batch_job_manager_client.cancel_execution("<execution-id>")
In some cases, the execution is a single step in a larger workflow orchestration. If the speed of execution is critical, use the warmup option.
The warmup option allows you to allocate the resources for execution before the execution starts. The resources are raised and kept running until the execution itself starts. This is especially relevant when a lot of resources are needed, or when reducing the running time by even 5 minutes is critical.
from qwak.models.executions.logic.client import BatchJobManagerClient from qwak.models.executions.logic.executions_config import ExecutionConfig # execution configuration execution_spec = ExecutionConfig.Execution( model_id=<model-id>, bucket=<bucket-name>, destination_bucket=<destination-bucket-name>, source_folder=<source-folder-path>, destination_folder=<destination-folder-path>, access_token_name=<access_token_name>, access_secret_name=<access-secret-name>, build_id=<alternate-build-id> ) warmup_spec = ExecutionConfig.Warmup( timeout=0 # warmup timeout in seconds ) batch_job_manager_client = BatchJobManagerClient() execution_config = ExecutionConfig(execution=execution_spec, warmup=warmup_spec) batch_job_manager_client = BatchJobManagerClient() batch_job_manager_client.start_warmup_job(execution_config)
from qwak.inference.clients import BatchInferenceClient # You can also set QWAK_MODEL_ID environment variable instead of passing it batch_inference_client = BatchInferenceClient(model_id="<model-id>") batch_inference_client.warmup( executors=<number-of-pods>, cpus=<number-of-cpus>, memory=<memory-amount>, timeout=<timeout-for-warmup>, build_id=<alternate-build-id>)
For each execution there are two types of logs.
- Execution Report: Contains the initial request, status updates, as well as the cancel and failed requests.
- Model Logs: These are available once the execution advances to the stage during which the files are processed.
To view both log types, use the following command:
qwak models execution report --execution-id <execution-id>
from qwak.models.executions.logic.client import BatchJobManagerClient from qwak.models.executions.logic.results import GetExecutionReportResult execution_report: GetExecutionReportResult = batch_job_manager_client.get_execution_report(<execution-id>) report_records = execution_report.records model_logs = execution_report.model_logs
In some cases you might want to output logs from the model itself in order to better understand the model processing behavior. In order to make the logs available, you need to use the Qwak Logger in your model's code.
from qwak.tools.logger import get_qwak_logger logger = get_qwak_logger() logger.info("your message here")
Updated 12 months ago