Using AnalyticOps, you can organize models within projects and store them in a common model repository. You can also train models, import models, evaluate and compare them for champion model selection, deploy models and monitor their performance.
The models module organizes and facilitates creation and management of models within the selected project. This chapter covers the following models related details:
Types of Models¶
AnalyticOps allows you to work with two types of models:
Git models: Using Github you add models to a given repository by following several steps, such as creating the required directory structure, editing configuration files, generating a model ID, etc.
Bring your own models (BYOM): Bring your own models (BYOM) support lets you import models that you develop using an external application as well as open-source programming languages such as Python or R, and PMML models. The purpose is to import, govern and deploy pre-trained Python and R models.
Create a Git Model¶
Comment: Need to discuss what should be added in this section
Define a BYOM Model¶
AnalyticOps allows you to define BYOM models and import versions.
To define a BYOM model:
Double-click a Project from the List of Projects.
The List of Models for the selected project displays in the Work area and the Models option highlights in the Navigation bar.
Click the Define BYOM Model button. The Define BYOM Model dialog displays.
In the Basic tab, set the properties:
Specifies the BYOM model name.
Specifies the description of the BYOM model.
Lets you select the format of the BYOM model: PMML, ONNX, H2O, Python, R, Custom
Link to BYOM
An optional property that allows linking to BYOM model workspace. It can be a GIT or DS workspace.
In the Advanced tab, set the properties:
An optional property that allows adding further details about the BYOM model.
An optional property that allows adding automation configuration about the BYOM model.
Click Save to save the defined BYOM model.
Note: Clicking Save & Import Versions will move you directly towards the Import Versions dialog. For details, see Model Versions.
The BYOM model adds to the List of Models.
Edit a BYOM Model¶
AnalyticOps allows you to edit an existing BYOM model. To edit an existing BYOM:
View List of Models¶
To view models added to a project:
Double-click a Project from the List of Projects. The List of Models for the selected project displays in the Work area and the Models option highlights in the Navigation bar.
The List of Models displays the following properties for each model.
Specifies the auto-generated model ID.
Specifies the model name.
Specifies the description of the project.
Shows the model type as GIT and BYOM.
Specifies the programming language of the model.
Lets you know the creation date of model.
View Model Versions
Allows you to view the model versions for the selected model. For details, see Model Versions.
Model Monitoring Alerts¶
AnalyticOps allows you to monitor the performance of models and model versions against specific key metrics. In addition to metric gathering and tracking, the model monitoring involves alerting based on these metric values.
AnalyticOps is using Prometheus alert manager to provide the support of alerting in UI. Prometheus is a very easy-to-use yet powerful tool that allows you to make the alerting rules very sophisticated and keep the alerting rule complexity encapsulated for the end-user.
The below section describes how to enable alerts for a model and how to define new alert configurations. For more details on monitoring alert, see Monitoring Alerts.
You can enable/disable alerts for a model added to the Models list. By default, the alerts are disabled for all models.
To enable alerts for a model:
In the Models list, select the Enable Alert option for the model. The alerts for the selected model enable. A few default alerts are added to each model, however you can define new alert configurations for each model. For details, see Configure Alerts.
You can disable the alerts for a model anytime by again selecting the Enable Alert option for the model.
Enabling alerts will start generating alerts if the defined rules in alert configuration become true. To view the active alerts and their details, see Monitoring Alerts.
You can define new alert configurations for a selected model. The alert configuration allows you to create rules by defining threshold values for specific metrics. If the metric values reaches to the defined threshold value, the alert will generate.
To configure an alert:
Select a model in the list and select Configure Alerts from the Actions menu.
The Alert Configurations list displays the list of alerts defined for the model.
Click the Create Rule button. The Create Alert Configuration dialog displays.
In the Create Alert Configuration dialog, set the properties:
Specifies the name of the alert configuration.
Specifies the read-only model ID for which the alert is defined.
Lets you define the summary of the alert.
Lets you add the description for the alert.
Specifies the condition expression for the alert by defining threshold values for metrics.
Reset to Default allows you to reset the expression to the default expression.
Lets you select the time duration for which the condition should hold true in order to generate the alert.
Click Save Configuration. The alert configuration adds to the list.
Edit an Alert Configuration¶
To edit an existing alert configuration:
Select an alert configuration from the list. The Edit Rule button enables.
Click the Edit Rule button. The Edit Alert Configuration dialog displays where you can modify the selected configuration.
After modifying the desired properties, click Save Configuration. The alert configuration saves with all the latest modifications.
Delete an Alert Configuration¶
To delete an alert configuration:
Archive a Model¶
The archiving feature allows you to hide a model from the list to better organize your models. You can view the archived models and unarchive them if required.
Note: The Archiving feature is provided with multiple modules including projects, models, model versions, dataset templates, datasets, and connections.
To archive a model:
Select a model in the list. The Actions button enables.
Click the Actions button. The Actions menu displays.
Click Archive Model. The model archives and hides from the current List of Models. A confirmation message displays on the top.
To view an Archived model, click the Show Archived option on the top. The archived model displays in the list along with an Archived label.
To Un-archive a model: