Managing Dependencies
JFrog ML supports a variety of Python frameworks to manage model dependencies.
Supported Python Versions
When building and managing your Python projects, different tools have varying levels of support for Python versions. Below is a summary of the supported Python versions for each tool:
- Poetry supports Python versions: 3.8 - 3.11
- Conda supports Python versions: 3.8 - 3.11
- requirements.txt (pip) supports only Python 3.9
.qwakignore file
Occasionally, we may want to exclude a file from the JFrog ML build but keep it in the repository with the model code. In such cases, we should add the .qwakignore
file to the root directory of our project.
In the file, we define the patterns to match files to exclude from the model build.
For example, suppose we have the following file structure:
.qwakignore
main/
__init__.py
model.py
README.md
tests/
test_model.py
research/
paper_a.pdf
paper_b.pdf
if we want to exclude the entire research
directory and the README.md
file from the build, our .qwakignore
file may contain:
research
README.md
Hidden files
By default, JFrog ML disregards hidden files. Hidden files are files or directories whose names start with a dot (.) in Unix-like operating systems, or they may have the "Hidden" attribute set in Windows. These files are typically used to store configuration data or hold temporary information.
Suppose you have a directory with files and subdirectories, including a hidden file named
.config_file
. JFrog ML, following its default behavior, will exclude this file from processing when triggering a remote build.
Incorporating Python Dependencies from .whl
Files
.whl
FilesQwak facilitates the use of Python dependencies packaged as .whl
files through requirements.txt
and conda.yaml
for managing dependencies. It's important to note that Poetry does not support dependencies from .whl
files.
- Preparing Your
.whl
Files:
First, ensure your .whl
file(s) are either uploaded with your model code or fetched from external storage. For instructions on uploading additional dependencies, refer to the Qwak CLI documentation (qwak models build --help
). Below is an example directory structure for your model, where main
is uploaded by default and the dep
directory, containing the pandas dependency in a .whl
file, is included via the --dependency-required-folders dep
option in the Qwak command.
The wheel file has to be uploaded as part of additional dependencies folder, and not as part of the main model folder.
/qwak/model_dir/
.
├── main # Main directory containing core code
│ ├── __init__.py # An empty file that indicates this directory is a Python package
│ ├── model.py # Defines the Credit Risk Model
│ └── conda.yaml # Conda environment configurationdata
│
├── dep # Additional dependency directory added with --dependency-required-folders
│ └── pandas-2.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
│
├── tests # Empty directory reserved for future test
│ └── ... # Future tests
|
└──
- Configuring Dependency Management Files:
Conda: Include the .whl
file in your conda.yaml as follows:
name: test_model
channels:
- defaults
- conda-forge
dependencies:
- python=3.9
- pip:
- "/qwak/model_dir/dep/pandas-2.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
Requirements.txt: Directly reference the .whl
file path relative to the requirements file location:
# requirements file located in main model folder
./../deps/wheel_test-0.1-py3-none-any.whl
# requirements file located in model dir
./deps/wheel_test-0.1-py3-none-any.whl
- Using the Dependency in Your Code:
Once the dependency is properly configured, you can import and use it in your Python code as usual:
import pandas as pd
Updated 13 days ago