Prediction Input & Output Adapters

Adapters help you customize input and output formats of your ML models.
JFrog ML uses adapters validate the model input and output formats and perform relevant type conversions.

This document lists to wide variety of available input and output adapters.

🚧
Adapters Support
Please note that Input and Output Adapters are currently supported only in Real-time and Streaming deployed models, and not in Batch ones. We're actively working to extend support for Adapters across all deployment types.

Adapter Types

Image

In the model file, we have to import the ImageInputAdapter

import numpy as np
import pandas as pd
from qwak.model.adapters import ImageInputAdapter

@qwak.api(analytics=False, input_adapter=ImageInputAdapter())
def predict(self, input_data) -> pd.DataFrame:

Now, in the predict function, we get a list of arrays containing the RGB properties of the image pixels. If we pass a 28px x 28px image, we get an array with shape (28, 28, 3). Of course, if we pass a grayscale image, we will get a (28, 28, 1) array.

If you trained your model using grayscale pictures but pass RGB values in production, remember to convert the input to grayscale. For example like this:

@qwak.api(analytics=False, input_adapter=ImageInputAdapter())
def predict(self, input_data) -> pd.DataFrame:
    def rgb2gray(rgb):
        return np.dot(rgb[...,:3], [0.2989, 0.5870, 0.1140])

    result = []

    for image in input_data:
        gray = rgb2gray(image)
        gray = gray / 255.0
        
        prediction_input = (np.expand_dims(gray, 0))
        prediction = self.probability_model.predict(prediction_input)
        result.append(prediction[0])
    
    return pd.DataFrame(result)

File

We can pass the image as a file data stream and load it as a file inside the predict function. Of course, sending a file works with any data format, not only images. However, in this example, we will use the same model as in the ImageInputAdapter example, but with a file adapter.

Before we start, we have to add the Pillow library to the model dependencies and import the Image class and the input adapter

import numpy as np
import pandas as pd
from PIL import Image
from qwak.model.adapters import FileInputAdapter

Now, we can change the input_adapter parameter in the qwak.api decorator:

@qwak.api(analytics=False, input_adapter=FileInputAdapter())
def predict(self, file_streams) -> pd.DataFrame:

In the predict function, we will iterate over the files in the file_stream, load them as images, convert them to grayscale and resize them to the size required by the trained model. After that, we can pass the image data to the model and get the prediction.

result = []
for fs in file_streams:
    im = Image.open(fs).convert(mode="L").resize((28, 28))

    prediction_input = (np.expand_dims(im, 0))
    prediction = self.probability_model.predict(prediction_input)
    result.append(prediction[0])

return pd.DataFrame(result)

String

If we want to pass a single sentence to the ML model, we can use the StringInputAdapter.

First, we have to import it

from qwak.model.adapters import StringInputAdapter

Now, we can configure the predict function to use the input adapter:

@qwak.api(analytics=False, input_adapter=StringInputAdapter())
def predict(self, texts) -> pd.DataFrame:

The texts variable will contain a list of string values. We can iterate over it and pass them to the model.
For example, if we added the StringInputAdapter to our example Pytorch text classifier, it would look like this:

@qwak.api(analytics=False, input_adapter=StringInputAdapter())
def predict(self, texts) -> pd.DataFrame:
    text_pipeline = lambda x: self.vocab(self.tokenizer(x))

    responses = []
    for text in texts:
        with torch.no_grad():
            text = torch.tensor(text_pipeline(text))
            output = self.model(text, torch.tensor([0]))
            responses.append(output.argmax(1).item() + 1)

    return pd.DataFrame.from_dict({'label': responses, 'text': texts})

JSON

If you want to use your model in a front-end application, you will probably send JSON to the server. We can handle the JSON automatically by using the JsonInputAdapter.

We must import the adapter first and configure the predict function

from qwak.model.adapters import JsonInputAdapter

@qwak.api(analytics=False, input_adapter=JsonInputAdapter())
    def predict(self, json_objects) -> pd.DataFrame:

Then, we can iterate over the json_objects and pass the text to the model:

@qwak.api(analytics=False, input_adapter=JsonInputAdapter())
def predict(self, json_objects) -> pd.DataFrame:
    text_pipeline = lambda x: self.vocab(self.tokenizer(x))

    responses = []
    for json in json_objects:
        with torch.no_grad():
            text = torch.tensor(text_pipeline(json['text']))
            output = self.model(text, torch.tensor([0]))
            responses.append(output.argmax(1).item() + 1)

    return pd.DataFrame.from_dict({'label': responses, 'text': json_objects})

To send a request to the deployed model, we must remember to specify the Content-Type: application/json.

Proto

If you use the protobuf library in your software, you may also want to use it for communication with your ML model. It is common to use protobuf for both input and output formats, so our example will show both.

Let's assume that we have the following protobuf definition of the input data

syntax = "proto3";

package qwak.demo;

option java_multiple_files = true;
option java_package = "com.qwak.ai.demo";

message ModelInput {

  int32 f1 = 1;

  int32 f2 = 2;
}

and the output data:

syntax = "proto3";

package qwak.demo;

option java_multiple_files = true;
option java_package = "com.qwak.ai.demo";

message ModelOutput {

  float prediction = 1;
}

We have to generate the protobuf classes for both the client application and the ML code. Of course, the ML code uses Python implementation. We will store the Python files in the input_pb and the output_pb files in the qwak_proto_demo directory.

In the model class, we will have to import the protobuf class and the input adapter:

from qwak.model.adapters import ProtoInputAdapter, ProtoOutputAdapter

from .qwak_proto_demo.input_pb import ModelInput
from .qwak_proto_demo.output_pb import ModelOutput

In the next step, we configure the input and output adapter as a decorator of the predict function:

@qwak.api(
   analytics=False,
   input_adapter=ProtoInputAdapter(ModelInput),
   output_adapter=ProtoOutputAdapter(),
)
def predict(self, input) -> ModelOutput:
    ...
    return ModelOutput(prediction=prediction_from_the_model)

In our implementation, we use the ParseFromString function to read a protobuf message, so remember to serialize your classes using the SerializeToString function!

message = ModelInput(f1=1, f2=2).SerializeToString()

TF Tensor

If we have all of the preprocessing code running as a separate service, we can pass a Tensorflow tensor directly to the model using a TfTensorInputAdapter.

In this case, we import the adapter and configure the predict function's decorator:

from qwak.model.adapters import TfTensorInputAdapter

@qwak.api(analytics=False, input_adapter=TfTensorInputAdapter())
def predict(self, tensor) -> pd.DataFrame:

To pass a tensor to a deployed model, we must send a JSON representation of the tensor. For example, if we used curl, the request would look like this:

curl -i –header "Content-Type: application/json" –request POST –data '{"instances": [1]}' qwak_rest_url

Multi Input

The MultiInputAdapter supports Automatic input format detection.

Sometimes we want to deploy a single model with multiple different input adapters. We could create a copy of the model, change the input adapter, and deploy multiple copies. However, we can also use a MultiInputAdapter to handle various input formats with a single model.

from qwak.model.adapters import DefaultOutputAdapter, DataFrameInputAdapter, ImageInputAdapter, MultiInputAdapter


@qwak.api(
        analytics=False,
        input_adapter=MultiInputAdapter([ImageInputAdapter, DataFrameInputAdapter]),
        output_adapter=DefaultOutputAdapter(),
    )

To use the MultiInputAdapter adapter, we must pass a list of adapters to its constructor. The MultiInputAdapter parses the data using the first compatible parser!

In our example, if a given input can be parsed as an Image, we will get an image in the predict function. If not, we will get a Pandas dataframe. If all parsers fail, the model returns an error.

Be careful with the following adapter configuration:

input_adapter=MultiInputAdapter([JsonInputAdapter, DataFrameInputAdapter]),

The JsonInputAdapter will successfully parse a JSON representation of a DataFrame!

Numpy

A NumpyInputAdapter can automatically parse a JSON array as a Numpy array and reshape it to the desired structure. When we configure the NumpyInputAdapter, we have to specify the content type and its shape:

from qwak.model.adapters import NumpyInputAdapter, NumpyOutputAdapter

@qwak.api(
    analytics=False,
    input_adapter=NumpyInputAdapter(
        shape=(2, 2), enforce_shape=False, dtype="int32"
    ),
    output_adapter=NumpyOutputAdapter(),
    )
def predict(self, input):

If we configure the input adapter as in the example above, and send the following value to the model: [[5,4,3,2]], we will get a result equivalent to running np.array([[5, 4, 3, 2]], dtype=np.int32).reshape(2, 2).

The NumpyOutputAdapter converts the returned output array directly to JSON without changing its structure. For example, if the model returns this Numpy array: np.array([[5, 4, 3, 2]], dtype=np.int32).reshape(2, 2), it will get converted to: [[5, 4], [3, 2]].
Starting from Sdk version 0.9.87 it will return numpy binary format .

Default Output

With DefaultOutputAdapter we can return multiple result formats from a single model. The adapter will automatically detect the type of the returned value.

from qwak.model.adapters import DefaultOutputAdapter, ImageInputAdapter

@qwak.api(  
        analytics=False,  
        input_adapter=ImageInputAdapter(),  
        output_adapter=DefaultOutputAdapter(),  
    )
def predict(self, input):

Note that the DefaultOutputAdapter doesn't work with Protobuf objects! To automatically detect the output type when your code returns DataFrames, JSONs, and Protobuf objects, you need to use the AutodetectOutputAdapter.

Json Output

With JsonOutputAdapter we can return Dict results, but the result has to be iterable


from qwak.model.adapters import ProtoInputAdapter, AutodetectOutputAdapter

@qwak.api(
        analytics=False,
        input_adapter=ProtoInputAdapter(ModelInput),
        output_adapter=AutodetectOutputAdapter(),
    )
    def predict(self, df) -> JsonOutputAdapter:
        ...
        return [{"result": ...}]

Auto Detect Output

Automatic output format detection with Protobuf support

This adapter works like the DefaultOutputAdapter, but it can also handle Protobuf classes:


from qwak.model.adapters import ProtoInputAdapter, AutodetectOutputAdapter

@qwak.api(
        analytics=False,
        input_adapter=ProtoInputAdapter(ModelInput),
        output_adapter=AutodetectOutputAdapter(),
    )
    def predict(self, df) -> ModelOutput:
        ...
        return [ModelOutput(prediction=result)]

Data Frame Based Adapters

Pandas Dataframe is supported with requested orient.

import pandas as pd
from qwak.model.adapters import DataFrameInputAdapter, DataFrameOutputAdapter

  @qwak.api(
      analytics=False,
      input_adapter=DataFrameInputAdapter(input_orient="split"),
      output_adapter=DataFrameOutputAdapter(output_orient="records"),
  )
  def predict(self, df) -> pd.DataFrame:
      # ... your prediction logic here ...

      # Constructing a DataFrame with the inference results
      predictions_df = pd.DataFrame({'prediction': result})

      return predictions_df

Note that the you can just choose the Data frame adapters with default value like below:


from qwak.model.adapters import DataFrameInputAdapter, DataFrameOutputAdapter

	@qwak.api(
        analytics=False,
        input_adapter=DataFrameInputAdapter(),
        output_adapter=DataFrameOutputAdapter(),
    )
  def predict(self, df) -> pd.DataFrame:
      # ... your prediction logic here ...

      # Constructing a DataFrame with the inference results
      predictions_df = pd.DataFrame({'prediction': result})

      return predictions_df

In this case for the DataFrameInputAdapter will try to automatically recognize the type of the input. For DataFrameOutputAdapter the output will be orient by records

All Supported Adapters

The following is a list of all input and output adapters provided by JFrog ML:

Output Adapters

DataFrameOutputAdapter
DefaultOutputAdapter
AutodetectOutputAdapter
JsonOutputAdapter
ProtoOutputAdapter
TfTensorOutputAdapter
NumpyOutputAdapter

Input Adapters

DataFrameInputAdapter
FileInputAdapter
ImageInputAdapter
JsonInputAdapter
ProtoInputAdapter
StringInputAdapter
TfTensorInputAdapter
NumpyInputAdapter
MultiInputAdapter

🚧Adapters Support

Adapter Types

Image

File

String

JSON

Proto

TF Tensor

Multi Input

Numpy

Default Output

Json Output

Auto Detect Output

Data Frame Based Adapters

All Supported Adapters

Output Adapters

Input Adapters

🚧
Adapters Support