Adding a New Model Definition

Teradata’s Analytic Ops (AOPS) framework provides an easy-to-use web-based user interface (UI) and a command line interface (CLI) to handle end-to-end pipelining of data science workflows.

In this tutorial, you will complete the following tasks related to defining a new Model definition in Git. Note: these examples use python, but the same can be done with R.

• Add a New Project

• Using CLI to Clone an Existing Project

• Using CLI to Add a New Model

• Copy Example Notebook

• Define the Model Configuration

• Define the Training File

• Define the Evaluation File

• Define the Scoring File

• Using CLI to validate the model files

• Commiting the code

In order to install AnalyticOps Accelerator CLI follow see installation instructions.

Before adding a new model, a project must be set up and configured in your local machine. This can be achieved by creating a new project or cloning an existing one. In both cases, a git repository to host the project is required.

Add a New Project

Prior to adding a new project locally, it must be created in the UI as explained in the User Guide: Create a New Project. When done, it can be added locally using the CLI:

1. Open a shell terminal and navigate to the path where you want your project to reside.

   cd <path to project dir>
   

2. Initialize the project files and structure with the command init:

   aoa init
   

3. Then the CLI will prompt the user to select a project to initialize locally. Type the index for the newly created project.

   Available projects:
   -------------------
   [0] (1dc14dba-49f9-4e31-8c84-abbb2177e14e) Aoa Demo Models
   [1] (23e1df4b-b630-47a1-ab80-7ad5385fcd8d) New Project
   Select project by index (current selection: none): _
   

Using CLI to Clone an Existing Project

To clone an existing project using the CLI:

1. Open a terminal and navigate to the path where you want your project to reside.

   cd <path to project dir>
   

2. Initialize the project files and structure with the command clone:

   aoa clone
   

3. Then the CLI will prompt the user to select a project to initialize locally. Type the index for the selected project.

   Available projects:
   -------------------
   [0] (1dc14dba-49f9-4e31-8c84-abbb2177e14e) Aoa Demo Models
   [1] (23e1df4b-b630-47a1-ab80-7ad5385fcd8d) Existent Project
   Select project by index (current selection: none): _
   

Using CLI to Add a New Model

To add a new model using the CLI:

1. Open a terminal and navigate to the path where your AnalyticsOps project resides.

   cd <path to project dir>
   

2. Use the add command from the CLI.

   aoa add
   

3. When prompted, enter the desired model name and model description.

   Enter model name: _
   Enter model description: _
   

4. Choose the desired language, by typing the index number from the list.

   Supported languages:
   --------------------
   [0] R
   [1] python
   [2] sql
   Select model language by index: _
   

5. Choose the desired template, by typing the index number from the list.

   Supported templates:
   --------------------
   [0] empty (default)
   [1] pyspark
   [2] sklearn
   Select template type by index (or leave blank for the default one):  _
   

6. As a result, a message will be shown with the id and name of the newly creted model. E.g.

   Creating model structure for model: (f7627d7a-3a4b-430b-953c-fc4fa738b642) demo
   

7. The files and folder structure of this new model will be under a path with the pattern <path to project dir>/model_definitions/<model id>

Copy Example Notebook

To keep all documents and code when developing a model in the same location, it is recommended to save or copy all your notebooks under the path <path to project dir>/model_definitions/<model id>/notebooks

1. As an example, a notebook can be copied in the terminal with:

   cp <origin path dir>/example.ipynb <path to project dir>/model_definitions/<model id>/notebooks
   

Define the Model Configuration

Once the model is defined, it can be formatted for AnalyticOps Accelerator.

1. Create or edit the existing requirements.txt file with all the model’s dependencies and versions in the folder <path to project dir>/model_definitions/<model id>/model_modules. E.g.

   xgboost==0.90
   scikit-learn==0.22.2
   shap==0.36.0
   matplotlib>=2.2.2
   teradataml>=17.0.0.2
   nyoka>=4.3.0
   aoa==4.1.4
   

2. Create or edit the existing <path to project dir>/model_definitions/<model id>/config.json file with all the model’s configuration parameters and values in json format. E.g.

   {
     "hyperParameters": {
       "eta": 0.2,
       "max_depth": 6
     }
   }
   

Define the Training File

1. Open and edit the file <path to project dir>/model_definitions/<model id>/model_modules/training.py. The contents will be something like:

   def train(data_conf, model_conf, **kwargs):
       """Python train method called by AOA framework
   
       Parameters:
       data_conf (dict): The dataset metadata
       model_conf (dict): The model configuration to use
   
       Returns:
       None:No return
   
       """
   
       hyperparams = model_conf["hyperParameters"]
   
       # load data & engineer
   
       print("Starting training...")
   
       # fit model to training data
   
       print("Finished training")
   
       # export model artefacts to models/ folder
   
       print("Saved trained model")
   

2. Add all the necessary imports at the beginning of the file. E.g.

   from xgboost import XGBClassifier
   from sklearn.preprocessing import MinMaxScaler
   from sklearn.pipeline import Pipeline
   from nyoka import xgboost_to_pmml
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import joblib
   import os
   
   
   def train(data_conf, model_conf, **kwargs):
       """Python train method called by AOA framework
   
       Parameters:
       data_conf (dict): The dataset metadata
       model_conf (dict): The model configuration to use
   
       Returns:
       None:No return
   
       """
   
       hyperparams = model_conf["hyperParameters"]
   
       # load data & engineer
   
       print("Starting training...")
   
       # fit model to training data
   
       print("Finished training")
   
       # export model artefacts to models/ folder
   
       print("Saved trained model")
   

3. Replace the comment # load data & engineer with the code to load the data and do all the necessary engineering before the training step. E.g.

   from xgboost import XGBClassifier
   from sklearn.preprocessing import MinMaxScaler
   from sklearn.pipeline import Pipeline
   from nyoka import xgboost_to_pmml
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import joblib
   import os
   
   
   def train(data_conf, model_conf, **kwargs):
       hyperparams = model_conf["hyperParameters"]
   
       # Create context to connect to Vantage
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       feature_names = ["NumTimesPrg", "PlGlcConc", "BloodP", "SkinThick", "TwoHourSerIns", "BMI", "DiPedFunc", "Age"]
       target_name = "HasDiabetes"
   
       # read training dataset from Teradata and convert to pandas
       train_df = DataFrame(data_conf["table"])
       train_df = train_df.select([feature_names + [target_name]])
       train_pdf = train_df.to_pandas()
   
       # split data into X and y
       X_train = train_pdf.drop(target_name, 1)
       y_train = train_pdf[target_name]
   
       print("Starting training...")
   
       # fit model to training data
   
       print("Finished training")
   
       # export model artefacts to models/ folder
   
       print("Saved trained model")
   

   Note: the function train and all it’s arguments must be present in the function definition. As shown in the example above, the data_conf parameter is a python dictionary that holds all the info for the dataset that can be established on the UI. Similarly, the model_conf parameter holds all the info established on the configuration file config.json and overridden on the UI. The env vars AOA_CONN_HOST, AOA_CONN_USERNAME and AOA_CONN_PASSWORD are passed by the system from the connection selected in the UI when executing the task.

4. Replace the comment # fit model to training data with the code required to train the model. E.g.

   from xgboost import XGBClassifier
   from sklearn.preprocessing import MinMaxScaler
   from sklearn.pipeline import Pipeline
   from nyoka import xgboost_to_pmml
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import joblib
   import os
   
   
   def train(data_conf, model_conf, **kwargs):
       hyperparams = model_conf["hyperParameters"]
   
       # Create context to connect to Vantage
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       feature_names = ["NumTimesPrg", "PlGlcConc", "BloodP", "SkinThick", "TwoHourSerIns", "BMI", "DiPedFunc", "Age"]
       target_name = "HasDiabetes"
   
       # read training dataset from Teradata and convert to pandas
       train_df = DataFrame(data_conf["table"])
       train_df = train_df.select([feature_names + [target_name]])
       train_pdf = train_df.to_pandas()
   
       # split data into X and y
       X_train = train_pdf.drop(target_name, 1)
       y_train = train_pdf[target_name]
   
       print("Starting training...")
   
       # fit model to training data
       model = Pipeline([('scaler', MinMaxScaler()),
                         ('xgb', XGBClassifier(eta=hyperparams["eta"],
                                               max_depth=hyperparams["max_depth"]))])
       # xgboost saves feature names but lets store on pipeline for easy access later
       model.feature_names = feature_names
       model.target_name = target_name
   
       model.fit(X_train, y_train)
   
       print("Finished training")
   
       # export model artefacts to models/ folder
   
       print("Saved trained model")
   

5. Replace the comment # export model artefacts to models/ folder with the code to export all the model’s artefacts. E.g.

   from xgboost import XGBClassifier
   from sklearn.preprocessing import MinMaxScaler
   from sklearn.pipeline import Pipeline
   from nyoka import xgboost_to_pmml
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import joblib
   import os
   
   
   def train(data_conf, model_conf, **kwargs):
       hyperparams = model_conf["hyperParameters"]
   
       # Create context to connect to Vantage
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       feature_names = ["NumTimesPrg", "PlGlcConc", "BloodP", "SkinThick", "TwoHourSerIns", "BMI", "DiPedFunc", "Age"]
       target_name = "HasDiabetes"
   
       # read training dataset from Teradata and convert to pandas
       train_df = DataFrame(data_conf["table"])
       train_df = train_df.select([feature_names + [target_name]])
       train_pdf = train_df.to_pandas()
   
       # split data into X and y
       X_train = train_pdf.drop(target_name, 1)
       y_train = train_pdf[target_name]
   
       print("Starting training...")
   
       # fit model to training data
       model = Pipeline([('scaler', MinMaxScaler()),
                         ('xgb', XGBClassifier(eta=hyperparams["eta"],
                                               max_depth=hyperparams["max_depth"]))])
       # xgboost saves feature names but lets store on pipeline for easy access later
       model.feature_names = feature_names
       model.target_name = target_name
   
       model.fit(X_train, y_train)
   
       print("Finished training")
   
       # export model artefacts
       joblib.dump(model, "artifacts/output/model.joblib")
   
       # we can also save as pmml so it can be used for In-Vantage scoring etc.
       xgboost_to_pmml(pipeline=model, col_names=feature_names, target_name=target_name, pmml_f_name="artifacts/output/model.pmml")
   
       print("Saved trained model")
   

   Note: the path used to store the artefacts must be in any case artifacts/output/. In the above example, the main artefact is stored as model.joblib, and optionally, as an exportable model in PMML format as model.pmml.

6. Add at the end of the train function the code required for monitoring. E.g.

   from xgboost import XGBClassifier
   from sklearn.preprocessing import MinMaxScaler
   from sklearn.pipeline import Pipeline
   from nyoka import xgboost_to_pmml
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import joblib
   import os
   
    
   def train(data_conf, model_conf, **kwargs):
       hyperparams = model_conf["hyperParameters"]
   
       # Create context to connect to Vantage
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       feature_names = ["NumTimesPrg", "PlGlcConc", "BloodP", "SkinThick", "TwoHourSerIns", "BMI", "DiPedFunc", "Age"]
       target_name = "HasDiabetes"
   
       # read training dataset from Teradata and convert to pandas
       train_df = DataFrame(data_conf["table"])
       train_df = train_df.select([feature_names + [target_name]])
       train_pdf = train_df.to_pandas()
   
       # split data into X and y
       X_train = train_pdf.drop(target_name, 1)
       y_train = train_pdf[target_name]
   
       print("Starting training...")
   
       # fit model to training data
       model = Pipeline([('scaler', MinMaxScaler()),
                         ('xgb', XGBClassifier(eta=hyperparams["eta"],
                                               max_depth=hyperparams["max_depth"]))])
       # xgboost saves feature names but lets store on pipeline for easy access later
       model.feature_names = feature_names
       model.target_name = target_name
   
       model.fit(X_train, y_train)
   
       print("Finished training")
   
       # export model artefacts
       joblib.dump(model, "artifacts/output/model.joblib")
   
       # we can also save as pmml so it can be used for In-Vantage scoring etc.
       xgboost_to_pmml(pipeline=model, col_names=feature_names, target_name=target_name, pmml_f_name="artifacts/output/model.pmml")
   
       print("Saved trained model")
   
       from xgboost import plot_importance
       model["xgb"].get_booster().feature_names = feature_names
       plot_importance(model["xgb"].get_booster(), max_num_features=10)
       save_plot("feature_importance.png")
   
       feature_importance = model["xgb"].get_booster().get_score(importance_type="weight")
       stats.record_training_stats(train_df,
                          features=feature_names,
                          predictors=[target_name],
                          categorical=[target_name],
                          importance=feature_importance,
                          category_labels={target_name: {0: "false", 1: "true"}})
   

   Note: to enable monitoring capabilities for the model, the method stats.record_training_stats must be called with all the parameters required as shown in the above example.

7. The resulting file should be:

   from xgboost import XGBClassifier
   from sklearn.preprocessing import MinMaxScaler
   from sklearn.pipeline import Pipeline
   from nyoka import xgboost_to_pmml
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import joblib
   import os
   
   
   def train(data_conf, model_conf, **kwargs):
      hyperparams = model_conf["hyperParameters"]
   
      create_context(host=os.environ["AOA_CONN_HOST"],
                     username=os.environ["AOA_CONN_USERNAME"],
                     password=os.environ["AOA_CONN_PASSWORD"],
                     database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
      feature_names = ["NumTimesPrg", "PlGlcConc", "BloodP", "SkinThick", "TwoHourSerIns", "BMI", "DiPedFunc", "Age"]
      target_name = "HasDiabetes"
   
      # read training dataset from Teradata and convert to pandas
      train_df = DataFrame(data_conf["table"])
      train_df = train_df.select([feature_names + [target_name]])
      train_pdf = train_df.to_pandas()
   
      # split data into X and y
      X_train = train_pdf.drop(target_name, 1)
      y_train = train_pdf[target_name]
   
      print("Starting training...")
   
      # fit model to training data
      model = Pipeline([('scaler', MinMaxScaler()),
                        ('xgb', XGBClassifier(eta=hyperparams["eta"],
                                              max_depth=hyperparams["max_depth"]))])
      # xgboost saves feature names but lets store on pipeline for easy access later
      model.feature_names = feature_names
      model.target_name = target_name
   
      model.fit(X_train, y_train)
   
      print("Finished training")
   
      # export model artefacts
      joblib.dump(model, "artifacts/output/model.joblib")
   
      # we can also save as pmml so it can be used for In-Vantage scoring etc.
      xgboost_to_pmml(pipeline=model, col_names=feature_names, target_name=target_name,
                      pmml_f_name="artifacts/output/model.pmml")
   
      print("Saved trained model")
   
      from xgboost import plot_importance
      model["xgb"].get_booster().feature_names = feature_names
      plot_importance(model["xgb"].get_booster(), max_num_features=10)
      save_plot("feature_importance.png")
   
      feature_importance = model["xgb"].get_booster().get_score(importance_type="weight")
      stats.record_training_stats(train_df,
                                  features=feature_names,
                                  predictors=[target_name],
                                  categorical=[target_name],
                                  importance=feature_importance,
                                  category_labels={target_name: {0: "false", 1: "true"}})
   

Define the Evaluation File

1. Open and edit the file <path to project dir>/model_definitions/<model id>/model_modules/evaluation.py. The contents will be something like:

   def evaluate(data_conf, model_conf, **kwargs):
   """Python evaluate method called by AOA framework
   
       Parameters:
       data_conf (dict): The dataset metadata
       model_conf (dict): The model configuration to use
   
       Returns:
       None:No return
   
       """
   
       # dump results as json file evaluation.json to models/ folder
       print("Evaluation complete...")
   

2. Add all the necessary imports at the beginning of the file. E.g.

   from sklearn import metrics
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import os
   import joblib
   import json
   import numpy as np
   import pandas as pd
   
   
   def evaluate(data_conf, model_conf, **kwargs):
   """Python evaluate method called by AOA framework
   
       Parameters:
       data_conf (dict): The dataset metadata
       model_conf (dict): The model configuration to use
   
       Returns:
       None:No return
   
       """
   
       # dump results as json file evaluation.json to models/ folder
       print("Evaluation complete...")
   

3. Define the function and load the model artifact, previously stored in the training phase. E.g.

   from sklearn import metrics
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import os
   import joblib
   import json
   import numpy as np
   import pandas as pd
   
   
   def evaluate(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       # dump results as json file evaluation.json to models/ folder
       print("Evaluation complete...")
   

   Note: the function evaluate and all it’s arguments must be present in the function definition. The path used to store the artefacts used in the training artifacts/output/ seen in Define the Training File has now become artifacts/input/ in the evaluation phase and cannot be changed.

4. Create the connection context using teradataml package. E.g.

   from sklearn import metrics
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import os
   import joblib
   import json
   import numpy as np
   import pandas as pd
   
   
   def evaluate(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       # dump results as json file evaluation.json to models/ folder
       print("Evaluation complete...")
   

   Note: the parameters data_conf, model_conf and the env vars for the connection are as seen in Define the Training File.

5. Load and process the data that will be used to evaluate the model. E.g.

   from sklearn import metrics
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import os
   import joblib
   import json
   import numpy as np
   import pandas as pd
   
   
   def evaluate(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       # Read test dataset from Teradata
       # As this is for demo purposes, we simulate the test dataset changing between executions
       # by introducing a random sample. Note that the sampling is performed in Teradata!
       test_df = DataFrame(data_conf["table"]).sample(frac=0.8)
       test_pdf = test_df.to_pandas()
   
       X_test = test_pdf[model.feature_names]
       y_test = test_pdf[model.target_name]
   
       # dump results as json file evaluation.json to models/ folder
       print("Evaluation complete...")
   

6. Score the data to evaluate the model. E.g.

   from sklearn import metrics
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import os
   import joblib
   import json
   import numpy as np
   import pandas as pd
   
   
   def evaluate(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       # Read test dataset from Teradata
       # As this is for demo purposes, we simulate the test dataset changing between executions
       # by introducing a random sample. Note that the sampling is performed in Teradata!
       test_df = DataFrame(data_conf["table"]).sample(frac=0.8)
       test_pdf = test_df.to_pandas()
   
       X_test = test_pdf[model.feature_names]
       y_test = test_pdf[model.target_name]
   
       print("Scoring")
       y_pred = model.predict(test_pdf[model.feature_names])
   
       # dump results as json file evaluation.json to models/ folder
       print("Evaluation complete...")
   

7. Generate and store the metrics of the evaluation. E.g.

   from sklearn import metrics
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import os
   import joblib
   import json
   import numpy as np
   import pandas as pd
   
   
   def evaluate(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       # Read test dataset from Teradata
       # As this is for demo purposes, we simulate the test dataset changing between executions
       # by introducing a random sample. Note that the sampling is performed in Teradata!
       test_df = DataFrame(data_conf["table"]).sample(frac=0.8)
       test_pdf = test_df.to_pandas()
   
       X_test = test_pdf[model.feature_names]
       y_test = test_pdf[model.target_name]
   
       print("Scoring")
       y_pred = model.predict(test_pdf[model.feature_names])
   
       y_pred_tdf = pd.DataFrame(y_pred, columns=[model.target_name])
       y_pred_tdf["PatientId"] = test_pdf["PatientId"].values
   
       evaluation = {
           'Accuracy': '{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)),
           'Recall': '{:.2f}'.format(metrics.recall_score(y_test, y_pred)),
           'Precision': '{:.2f}'.format(metrics.precision_score(y_test, y_pred)),
           'f1-score': '{:.2f}'.format(metrics.f1_score(y_test, y_pred))
       }
   
       with open("artifacts/output/metrics.json", "w+") as f:
           json.dump(evaluation, f)
   
       metrics.plot_confusion_matrix(model, X_test, y_test)
       save_plot('Confusion Matrix')
   
       metrics.plot_roc_curve(model, X_test, y_test)
       save_plot('ROC Curve')
   
       # dump results as json file evaluation.json to models/ folder
       print("Evaluation complete...")
   

   Note: the outputs of the evaluation phase must be stored in any case under artifacts/output/.

8. Finally, generate and record the stats for monitoring. E.g.

   from sklearn import metrics
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import os
   import joblib
   import json
   import numpy as np
   import pandas as pd
   
   
   def evaluate(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       # Read test dataset from Teradata
       # As this is for demo purposes, we simulate the test dataset changing between executions
       # by introducing a random sample. Note that the sampling is performed in Teradata!
       test_df = DataFrame(data_conf["table"]).sample(frac=0.8)
       test_pdf = test_df.to_pandas()
   
       X_test = test_pdf[model.feature_names]
       y_test = test_pdf[model.target_name]
   
       print("Scoring")
       y_pred = model.predict(test_pdf[model.feature_names])
   
       y_pred_tdf = pd.DataFrame(y_pred, columns=[model.target_name])
       y_pred_tdf["PatientId"] = test_pdf["PatientId"].values
   
       evaluation = {
           'Accuracy': '{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)),
           'Recall': '{:.2f}'.format(metrics.recall_score(y_test, y_pred)),
           'Precision': '{:.2f}'.format(metrics.precision_score(y_test, y_pred)),
           'f1-score': '{:.2f}'.format(metrics.f1_score(y_test, y_pred))
       }
   
       with open("artifacts/output/metrics.json", "w+") as f:
           json.dump(evaluation, f)
   
       metrics.plot_confusion_matrix(model, X_test, y_test)
       save_plot('Confusion Matrix')
   
       metrics.plot_roc_curve(model, X_test, y_test)
       save_plot('ROC Curve')
   
       # xgboost has its own feature importance plot support but lets use shap as explainability example
       import shap
   
       shap_explainer = shap.TreeExplainer(model['xgb'])
       shap_values = shap_explainer.shap_values(X_test)
   
       shap.summary_plot(shap_values, X_test, feature_names=model.feature_names,
                         show=False, plot_size=(12, 8), plot_type='bar')
       save_plot('SHAP Feature Importance')
   
       feature_importance = pd.DataFrame(list(zip(model.feature_names, np.abs(shap_values).mean(0))),
                                         columns=['col_name', 'feature_importance_vals'])
       feature_importance = feature_importance.set_index("col_name").T.to_dict(orient='records')[0]
   
       predictions_table="TMP_{}".format(data_conf["predictions"]).lower()
       copy_to_sql(df=y_pred_tdf, table_name=predictions_table, index=False, if_exists="replace", temporary=True)
   
       stats.record_evaluation_stats(test_df, DataFrame(predictions_table), feature_importance)
   

   Note: to enable monitoring capabilities for the model, the method stats.record_evaluation_stats must be called with all the parameters required as shown in the above example.

9. The resulting file should be:

   from sklearn import metrics
   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   from aoa.util.artefacts import save_plot
   
   import os
   import joblib
   import json
   import numpy as np
   import pandas as pd
   
   
   def evaluate(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       # Read test dataset from Teradata
       # As this is for demo purposes, we simulate the test dataset changing between executions
       # by introducing a random sample. Note that the sampling is performed in Teradata!
       test_df = DataFrame(data_conf["table"]).sample(frac=0.8)
       test_pdf = test_df.to_pandas()
   
       X_test = test_pdf[model.feature_names]
       y_test = test_pdf[model.target_name]
   
       print("Scoring")
       y_pred = model.predict(test_pdf[model.feature_names])
   
       y_pred_tdf = pd.DataFrame(y_pred, columns=[model.target_name])
       y_pred_tdf["PatientId"] = test_pdf["PatientId"].values
   
       evaluation = {
           'Accuracy': '{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)),
           'Recall': '{:.2f}'.format(metrics.recall_score(y_test, y_pred)),
           'Precision': '{:.2f}'.format(metrics.precision_score(y_test, y_pred)),
           'f1-score': '{:.2f}'.format(metrics.f1_score(y_test, y_pred))
       }
   
       with open("artifacts/output/metrics.json", "w+") as f:
           json.dump(evaluation, f)
   
       metrics.plot_confusion_matrix(model, X_test, y_test)
       save_plot('Confusion Matrix')
   
       metrics.plot_roc_curve(model, X_test, y_test)
       save_plot('ROC Curve')
   
       # xgboost has its own feature importance plot support but lets use shap as explainability example
       import shap
   
       shap_explainer = shap.TreeExplainer(model['xgb'])
       shap_values = shap_explainer.shap_values(X_test)
   
       shap.summary_plot(shap_values, X_test, feature_names=model.feature_names,
                         show=False, plot_size=(12, 8), plot_type='bar')
       save_plot('SHAP Feature Importance')
   
       feature_importance = pd.DataFrame(list(zip(model.feature_names, np.abs(shap_values).mean(0))),
                                         columns=['col_name', 'feature_importance_vals'])
       feature_importance = feature_importance.set_index("col_name").T.to_dict(orient='records')[0]
   
       predictions_table="TMP_{}".format(data_conf["predictions"]).lower()
       copy_to_sql(df=y_pred_tdf, table_name=predictions_table, index=False, if_exists="replace", temporary=True)
   
       stats.record_evaluation_stats(test_df, DataFrame(predictions_table), feature_importance)
   

Define the Scoring File

1. Open and edit the file <path to project dir>/model_definitions/<model id>/model_modules/scoring.py. The contents will be something like:

   def score(data_conf, model_conf, **kwargs):
   """Python score method called by AOA framework batch mode
   
       Parameters:
       data_conf (dict): The dataset metadata
       model_conf (dict): The model configuration to use
   
       Returns:
       None:No return
   
       """
   
   
   # Uncomment this code if you want to deploy your model as a Web Service (Real-time / Interactive usage)
   # class ModelScorer(object):
   #    def __init__(self, config=None):
   #        self.model = joblib.load('models/iris_knn.joblib')
   #
   #    def predict(self, data):
   #        return self.model.predict([data])
   #
   

2. Add all the necessary imports at the beginning of the file. E.g.

   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   
   import os
   import joblib
   import pandas as pd
   import numpy as np
   
   
   def score(data_conf, model_conf, **kwargs):
   """Python score method called by AOA framework batch mode
   
       Parameters:
       data_conf (dict): The dataset metadata
       model_conf (dict): The model configuration to use
   
       Returns:
       None:No return
   
       """
   
   
   # Uncomment this code if you want to deploy your model as a Web Service (Real-time / Interactive usage)
   # class ModelScorer(object):
   #    def __init__(self, config=None):
   #        self.model = joblib.load('models/iris_knn.joblib')
   #
   #    def predict(self, data):
   #        return self.model.predict([data])
   #
   

3. For batch scoring, define the function and load the model artifact, previously stored in the training phase. E.g.

   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   
   import os
   import joblib
   import pandas as pd
   import numpy as np
   
   
   def score(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
   # Uncomment this code if you want to deploy your model as a Web Service (Real-time / Interactive usage)
   # class ModelScorer(object):
   #    def __init__(self, config=None):
   #        self.model = joblib.load('models/iris_knn.joblib')
   #
   #    def predict(self, data):
   #        return self.model.predict([data])
   #
   

   Note: the function score and all it’s arguments must be present in the function definition, this function will be called when doing batch scoring. The path used to store the artefacts used in the training artifacts/output/ seen in Define the Training File has now become artifacts/input/ in the scoring phase and cannot be changed.

4. Create the connection context using teradataml package. E.g.

   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   
   import os
   import joblib
   import pandas as pd
   import numpy as np
   
   
   def score(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
   # Uncomment this code if you want to deploy your model as a Web Service (Real-time / Interactive usage)
   # class ModelScorer(object):
   #    def __init__(self, config=None):
   #        self.model = joblib.load('models/iris_knn.joblib')
   #
   #    def predict(self, data):
   #        return self.model.predict([data])
   #
   

   Note: the parameters data_conf, model_conf and the env vars for the connection are as seen in Define the Training File.

5. Load and process the data that will be scored by the model. E.g.

   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   
   import os
   import joblib
   import pandas as pd
   import numpy as np
   
   
   def score(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       features_tdf = DataFrame(data_conf["table"])
      
       # convert to pandas to use locally
       features_df = features_tdf.to_pandas()
   
   # Uncomment this code if you want to deploy your model as a Web Service (Real-time / Interactive usage)
   # class ModelScorer(object):
   #    def __init__(self, config=None):
   #        self.model = joblib.load('models/iris_knn.joblib')
   #
   #    def predict(self, data):
   #        return self.model.predict([data])
   #
   

6. Score the data. E.g.

   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   
   import os
   import joblib
   import pandas as pd
   import numpy as np
   
   
   def score(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       features_tdf = DataFrame(data_conf["table"])
      
       # convert to pandas to use locally
       features_df = features_tdf.to_pandas()
   
       print("Scoring")
       y_pred = model.predict(features_df[model.feature_names])
   
   # Uncomment this code if you want to deploy your model as a Web Service (Real-time / Interactive usage)
   # class ModelScorer(object):
   #    def __init__(self, config=None):
   #        self.model = joblib.load('models/iris_knn.joblib')
   #
   #    def predict(self, data):
   #        return self.model.predict([data])
   #
   

7. Process and save the scored result (predictions). E.g.

   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   
   import os
   import joblib
   import pandas as pd
   import numpy as np
   
   
   def score(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       features_tdf = DataFrame(data_conf["table"])
      
       # convert to pandas to use locally
       features_df = features_tdf.to_pandas()
   
       print("Scoring")
       y_pred = model.predict(features_df[model.feature_names])
   
       print("Finished Scoring")
   
       # create result dataframe and store in Teradata
       y_pred = pd.DataFrame(y_pred, columns=[model.target_name])
       y_pred["PatientId"] = features_df["PatientId"].values
       copy_to_sql(df=y_pred, table_name=data_conf["predictions"], index=False, if_exists="replace")
   
   # Uncomment this code if you want to deploy your model as a Web Service (Real-time / Interactive usage)
   # class ModelScorer(object):
   #    def __init__(self, config=None):
   #        self.model = joblib.load('models/iris_knn.joblib')
   #
   #    def predict(self, data):
   #        return self.model.predict([data])
   #
   

8. Generate and record the stats for monitoring. E.g.

   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   
   import os
   import joblib
   import pandas as pd
   import numpy as np
   
   
   def score(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       features_tdf = DataFrame(data_conf["table"])
      
       # convert to pandas to use locally
       features_df = features_tdf.to_pandas()
   
       print("Scoring")
       y_pred = model.predict(features_df[model.feature_names])
   
       print("Finished Scoring")
   
       # create result dataframe and store in Teradata
       y_pred = pd.DataFrame(y_pred, columns=[model.target_name])
       y_pred["PatientId"] = features_df["PatientId"].values
       copy_to_sql(df=y_pred, table_name=data_conf["predictions"], index=False, if_exists="replace")
   
       predictions_tdf = DataFrame(data_conf["predictions"])
   
       stats.record_scoring_stats(features_tdf, predictions_tdf)
   
   # Uncomment this code if you want to deploy your model as a Web Service (Real-time / Interactive usage)
   # class ModelScorer(object):
   #    def __init__(self, config=None):
   #        self.model = joblib.load('models/iris_knn.joblib')
   #
   #    def predict(self, data):
   #        return self.model.predict([data])
   #
   

   Note: to enable monitoring capabilities for the model, the method stats.record_scoring_stats must be called with all the parameters required as shown in the above example.

9. For RESTful scoring, define the class and the init method to load the model artifact, previously stored in the training phase. E.g.

   from teradataml import create_context
   from teradataml.dataframe.dataframe import DataFrame
   from teradataml.dataframe.copy_to import copy_to_sql
   from aoa.stats import stats
   
   import os
   import joblib
   import pandas as pd
   import numpy as np
   
   
   def score(data_conf, model_conf, **kwargs):
       model = joblib.load('artifacts/input/model.joblib')
   
       create_context(host=os.environ["AOA_CONN_HOST"],
                      username=os.environ["AOA_CONN_USERNAME"],
                      password=os.environ["AOA_CONN_PASSWORD"],
                      database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
   
       features_tdf = DataFrame(data_conf["table"])
      
       # convert to pandas to use locally
       features_df = features_tdf.to_pandas()
   
       print("Scoring")
       y_pred = model.predict(features_df[model.feature_names])
   
       print("Finished Scoring")
   
       # create result dataframe and store in Teradata
       y_pred = pd.DataFrame(y_pred, columns=[model.target_name])
       y_pred["PatientId"] = features_df["PatientId"].values
       copy_to_sql(df=y_pred, table_name=data_conf["predictions"], index=False, if_exists="replace")
   
       predictions_tdf = DataFrame(data_conf["predictions"])
   
       stats.record_scoring_stats(features_tdf, predictions_tdf)
   
   
   # Add code required for RESTful API
   class ModelScorer(object):
   
       def __init__(self, config=None):
           self.model = joblib.load('artifacts/input/model.joblib')
   
           from prometheus_client import Counter
           self.pred_class_counter = Counter('model_prediction_classes',
                                             'Model Prediction Classes', ['model', 'version', 'clazz'])
   

   Note: the class ModelScorer and all it’s methods must be present as it will be called when doing RESTful scoring.

10. Next, define the predict method. E.g.

    from teradataml import create_context
    from teradataml.dataframe.dataframe import DataFrame
    from teradataml.dataframe.copy_to import copy_to_sql
    from aoa.stats import stats
    
    import os
    import joblib
    import pandas as pd
    import numpy as np
    
    
    def score(data_conf, model_conf, **kwargs):
        model = joblib.load('artifacts/input/model.joblib')
    
        create_context(host=os.environ["AOA_CONN_HOST"],
                       username=os.environ["AOA_CONN_USERNAME"],
                       password=os.environ["AOA_CONN_PASSWORD"],
                       database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
    
        features_tdf = DataFrame(data_conf["table"])
       
        # convert to pandas to use locally
        features_df = features_tdf.to_pandas()
    
        print("Scoring")
        y_pred = model.predict(features_df[model.feature_names])
    
        print("Finished Scoring")
    
        # create result dataframe and store in Teradata
        y_pred = pd.DataFrame(y_pred, columns=[model.target_name])
        y_pred["PatientId"] = features_df["PatientId"].values
        copy_to_sql(df=y_pred, table_name=data_conf["predictions"], index=False, if_exists="replace")
    
        predictions_tdf = DataFrame(data_conf["predictions"])
    
        stats.record_scoring_stats(features_tdf, predictions_tdf)
    
    
    # Add code required for RESTful API
    class ModelScorer(object):
    
        def __init__(self, config=None):
            self.model = joblib.load('artifacts/input/model.joblib')
    
        def predict(self, data):
            return self.model.predict([data])
    

    Note: the method predict from class ModelScorer and all it’s arguments must be present as it will be called when doing RESTful scoring.

11. The resulting file should be:

    from teradataml import create_context
    from teradataml.dataframe.dataframe import DataFrame
    from teradataml.dataframe.copy_to import copy_to_sql
    from aoa.stats import stats
    
    import os
    import joblib
    import pandas as pd
    import numpy as np
    
    
    def score(data_conf, model_conf, **kwargs):
        model = joblib.load("artifacts/input/model.joblib")
    
        create_context(host=os.environ["AOA_CONN_HOST"],
                       username=os.environ["AOA_CONN_USERNAME"],
                       password=os.environ["AOA_CONN_PASSWORD"],
                       database=data_conf["schema"] if "schema" in data_conf and data_conf["schema"] != "" else None)
    
        features_tdf = DataFrame(data_conf["table"])
    
        # convert to pandas to use locally
        features_df = features_tdf.to_pandas()
    
        print("Scoring")
        y_pred = model.predict(features_df[model.feature_names])
    
        print("Finished Scoring")
    
        # create result dataframe and store in Teradata
        y_pred = pd.DataFrame(y_pred, columns=[model.target_name])
        y_pred["PatientId"] = features_df["PatientId"].values
        copy_to_sql(df=y_pred, table_name=data_conf["predictions"], index=False, if_exists="replace")
    
        predictions_tdf = DataFrame(data_conf["predictions"])
    
        stats.record_scoring_stats(features_tdf, predictions_tdf)
    
    
    # Add code required for RESTful API
    class ModelScorer(object):
    
        def __init__(self, config=None):
            self.model = joblib.load('artifacts/input/model.joblib')
    
        def predict(self, data):
            return self.model.predict([data])
    

Using CLI to validate the model files

1. In order to validate all the files previously created, just execute the model with the command run from the CLI. When prompted, type the index of the model to validate.

   > aoa run
   Available models:
   -----------------
   [0] PySpark PIMA Prediction
   [1] Python Demand Forecasting
   [2] Python STO Forecasting
   [3] R Diabetes Prediction
   [4] Python Diabetes Prediction
   Select model by index: _
   

2. Then it will prompt to select the mode, and type the index of the mode you want to validate.

   Available modes:
   ----------------
   [0] Train
   [1] Evaluate
   [2] Score (Batch)
   Select mode by index: _
   

3. The next step will prompt to select the dataset (for the Score mode, it will be the dataset template). Type the index of the dataset you want to use for this execution.

   Available datasets:
   -------------------
   [0] Demand Forecast Evaluate
   [1] Demand Forecast Train
   [2] PIMA Diabetes
   [3] PIMA Diabetes Evaluate
   [4] PIMA Diabetes Train
   [5] STO Synthetic Evaluate
   [6] STO Synthetic Train
   Select dataset by index: _
   

4. Finally, the CLI will prompt to select the dataset connection (you can create local connections with the CLI by using the command aoa connection add, check the manual). Type the index of the dataset you want to use for this execution.

   Available connections:
   ----------------------
   [0] Demo Connection
   [1] Vantage Connection
   Select connection by index: _
   

5. The CLI will run your model and it will fail if there is any bug or error. If not, it will exit successfully, thus meaning your model file has been validated. You should repeat these steps for all 3 modes and/or files: training, evaluation and scoring.

Commiting the code

1. To commit your code to the configured repository, in order to make it available to the AOA system, start by adding the files you want to commit.

   git add <path to project dir>/model_definitions/<model id>
   

   Note: this will add all the files and folders under the specified model folder.

2. Now create the commit and set a commit message.

   git commit -m "First commit for model <model id>"
   

3. Finally, push the commit to the remote repository at the specified branch.

   git push origin master
   

   Note: this example pushes to the remote origin and branch master