Supported ML Models

The package currently supports various Scikit-Learn objects. It can also embed gradient boosting regression models from XGBoost, and LightGBM. Finally, it supports Sequential Neural Networks from PyTorch and Keras. In Mixed Integer Formulations, we briefly outline the MIP formulations used for the various ML models.

Scikit-learn

The following table lists the name of the models supported, the name of the corresponding object in the Python framework, and the function that can be used to insert it in a SCIP model.

Supported ML models of scikit-learn
Machine Learning Model	Scikit-learn object	Functions to insert
Ordinary Least Square	`LinearRegression` `Ridge` `ElasticNet` `Lasso`	`add_linear_regression_constr`
Partial Least Square	`PLSRegression` `PLSRegression`	`add_pls_regression_constr`
Logistic Regression	`LogisticRegression`	`add_logistic_regression_constr`
Neural-network	`MLPRegressor` `MLPClassifier`	`add_mlp_regressor_constr` `add_mlp_classifier_constr`
Decision tree	`DecisionTreeRegressor` `DecisionTreeClassifier`	`add_decision_tree_regressor_constr` `add_decision_tree_classifier_constr`
Gradient boosting	`GradientBoostingRegressor` `GradientBoostingClassifier`	`add_gradient_boosting_regressor_constr` `add_gradient_boosting_classifier_constr`
Random Forest	`RandomForestRegressor` `RandomForestClassifier`	`add_random_forest_regressor_constr` `add_random_forest_classifier_constr`
Support Vector Machines	`SVR` `SVC` `LinearSVR` `LinearSVC`	`add_support_vector_regressor_constr` `add_support_vector_classifier_constr`
Centroid Clustering	`KMeans` `MiniBatchKMeans`	`add_centroid_cluster_constr`
Pipeline	`Pipeline`	`add_pipeline_constr`
MultiOutput	`MultiOutputClassifier` `MultiOutputRegressor`	`add_multi_output_classifier_constr` `add_multi_output_regressor_constr`

PyTorch

In PyTorch, only torch.nn.Sequential objects are supported.

They can be embedded in a SCIP model with the function pyscipopt_ml.torch.add_sequential_constr().

Currently, only five types of layers are supported:

Linear layers,

ReLU layers,

Sigmoid layers,

Tanh layers

Softmax layers

Softplus layers

In the case of the final layer being an activation function used for classification, e.g. Softmax, simply set output_type==”classification” when inserting the predictor constraint. The result is that the class with highest value is assigned value 1 and all other classes are assigned value 0. Essentially, explicitly modelling the final activation function for classification purposes is unnecessary from a MIP perspective as the maximum value is preserved after the function is applied.

Keras

For Keras, only keras.Model and keras.Sequential are supported.

They can be embedded in a SCIP model with the function pyscipopt_ml.keras.add_keras_constr().

The supported layer types and activation functions are the same as in torch (see above). This support holds for the classification case when the final layer is an unsupported activation function, e.g. softmax. Please read the above explanation in the PyTorch section, and in such use cases set output_type=”classification” when inserting the predictor constraint.

ONNX

For ONNX we also support standard feed forward neural networks. These must be provided in the ModelProto format.

They can be embedded in a SCIP model with the function pyscipopt_ml.onnx.add_onnx_constr().

The supported layer types and activation functions are the same as in torch and keras (see above). The classification trick for the final layer is not done for ONNX models, so be warned that there will be a performance difference for imported classification models.

XGBoost

Models for XGBoost’s Scikit-Learn interface can be embedded in a SCIP model. The following table lists the name of the models supported, the name of the corresponding object in the Python framework, and the function that can be used to insert it in a SCIP model.

Supported ML models of xgboost
XGBoost object	Function to insert
`xgboost.XGBRegressor`	`add_xgbregressor_constr`
`xgboost.XGBClassifier`	`add_xgbclassifier_constr`
`xgboost.XGBRFRegressor`	`add_xgbregressor_rf_constr`
`xgboost.XGBRFClassifier`	`add_xgbclassifier_rf_constr`

Currently only “gbtree” boosters are supported.

LightGBM

Models for LightGBM’s Scikit-Learn interface can be embedded in a SCIP model. The following table lists the name of the models supported, the name of the corresponding object in the Python framework, and the function that can be used to insert it in a SCIP model.

Supported ML models of lightgbm
LightGBM object	Function to insert
`lightgbm.LGBMRegressor`	`add_lgbregressor_constr`
`lightgbm.LGBMClassifier`	`add_lgbclassifier_constr`

Currently “gbdt” and “rf” boosters are supported.