Altair SmartWorks Analytics

 

Model Registry

PySpark models can be registered in two ways:

  • Training the model within the platform using the Spark AutoML Node and then registering the model through the MLflow UI.

  • Training the model outside the platform, zipping it, uploading it to the platform, and then registering it through the MLOps Node UI.

Training a PySpark Model in SmartWorks Analytics

You can train your own PySpark Models in the platform and have them managed (versioning, organization, etc) via the Model Registry.

Steps:

  1. Train your model using the Spark-based AutoML Node in the SmartWorks platform.

  2. Navigate to the MLflow UI where your model is tracked – it’s the “Connection Profile“ you set when you configured the AutoML node.

  3. Click into your model to view it

    • If you scroll down to the Tags section, you’ll see some information about your model, such as createdBy, packageVersion, type, and modelMetadata

    • If you wish to add your own custom tags to this model, you can edit the modelMetadata JSON to do so. See the Training a PySpark Model Outside SmartWorks Analytics section below for details on how the JSON should be formatted

  4. Register your model by clicking on the Register Model button in the MLflow UI.

  5. Once all of the above steps are complete, your model will be visible in the Model Registry of the MLOps Node with all of the appropriate information

Training a PySpark Model Outside of SmartWorks Analytics

You can also bring in your own PySpark models that you’ve trained outside the platform to leverage SmartWorks Analytics' Model Registry and Model Serving features.

Steps:

  1. Train your PySpark model outside of the SmartWorks platform and then save it.

  2. Package the metadata and stages folders of your model into a single zip file.

  3. Upload this zip file to the SmartWorks Analytics library .

  4. In the MLOps Node, click the + Register New Model button.

  5. Set the Model Type to PYSPARK.

  6. Browse and select your zip file to set the Model File Location.

  7. Fill out the rest of the settings for your model and then click the Save button to register it.