Altair SmartWorks Analytics

 

Basic Serving of TensorFlow Models

This help page describes how TensorFlow models are served.

Requests Format

TensorFlow models require request data to be formatted in a way that is slightly different than the other model types. In particular, you should use the jsonData parameter for the input. You can then format your request data in the following way:

 

{

  "jsonData": <input_data>,

  "meta": {

            "method": <method>,

            "batch_size": <batch_size>,

            "dtypes": <dtypes>

          }

}

 

 

where:

  • input_data – This is the data to be passed as input to the model. It can be given in one of two formats. The format must match the format that is used for the dtypes, if given:

    • array input style – pass your array directly. For example:

    •  

      input_data = tf.random.uniform([224, 224, 3], minval=0.0, maxval=1.0).numpy().tolist()

       

       
    • dictionary input style – pass your data within a dictionary. The key corresponds to the name of the layer to pass the data to and the value corresponds to the data itself. For example:

    •  

      image_data = tf.random.uniform([224, 224, 3], minval=0.0, maxval=1.0).numpy().tolist()

      input_data = {"input_image": image_data}

       

     

    This format is the best way to use models that have multiple inputs. You can pass data to multiple input layers at the same time using the layer names.

  • method – The method with which to execute model inference. There are two options:

  • batch_size – An integer specifying the batch size to use at inference. Only valid for the predict method.

  • dtypes – For all inputs, the tf.convert_to_tensor() function will be applied; by default, the dtype parameter is set to None. You can change this by passing your own data types via the dtypes input here.  It can be given in one of two formats. The format must match the format that is used for the jsonData:

    • array – pass an array of data types. The order of the dtypes in the array should exactly match the order of the inputs given in jsonData. For example:

    •  

      dtypes = ["uint8"]

       

       

    • dictionary – pass a dictionary of data types. The key corresponds to the name of the layer where the data type should be applied to the input and the value corresponds to the data type itself. For example:

    •  

      dtypes = {"input_image": "uint8"}

       

 

Notes:

  • For models saved using the tf API, only the array input style is supported

  • For models saved using the tf API, multi-input models are not supported

Preprocessing

After reaching the model endpoint in JSON format, each input to your model will be converted to a TensorFlow Tensor via the tf.convert_to_tensor() function. If you have passed a list or dictionary of dtypes, then they will be passed to the tf.convert_to_tensor() function for each individual input.

Predictions

After preprocessing, the Tensor is passed directly to the model for inference, based on the method e.g.:

 

if method == "call":

    preds = model(tensor)

elif method == "predict":

    model.predict(tensor, batch_size)

 

 

Known Issues

TensorFlow models that handle large data, such as in the cases of video processing, super resolution, etc., may not be reliable to deploy. This due to the high load of transferring such large data over the network via the REST API.