twistml.evaluation package¶
Submodules¶
twistml.evaluation.evaluation module¶
- Contains functions to evaluate regression and classification
methods
The given methods can be any machine learning algorithms that adhere to sklearn’s estimator pattern - this includes sklearn’s pipelines. The evaluate_ functions will perform a set number of cross validations over given parameters and return certain metrics (MSE and MAPE for regression, ROC-AUC for binary classification).
<routine listings>
<see also>
<notes>
<references>
<examples>
| Author: | Matthias Manhertz |
|---|---|
| Copyright: |
|
| Licence: | MIT |
-
class
twistml.evaluation.evaluation.EvalBinClassResult(auc, aucstd, f1, f1std)¶ Bases:
tuple-
auc¶ Alias for field number 0
-
aucstd¶ Alias for field number 1
-
f1¶ Alias for field number 2
-
f1std¶ Alias for field number 3
-
-
class
twistml.evaluation.evaluation.EvalRegrResult(mse, msestd, mape, mapestd, auc, aucstd, f1, f1std)¶ Bases:
tuple-
auc¶ Alias for field number 4
-
aucstd¶ Alias for field number 5
-
f1¶ Alias for field number 6
-
f1std¶ Alias for field number 7
-
mape¶ Alias for field number 2
-
mapestd¶ Alias for field number 3
-
mse¶ Alias for field number 0
-
msestd¶ Alias for field number 1
-
-
twistml.evaluation.evaluation.cross_validate(X, y, method, cv_params, **kwargs)¶ Runs cross validation on method and returns fitted GridSearchCV instance.
- X : array_like, shape = (n_samples, n_features)
- The feature vectors
- y : array-like, shape (n_samples,)
- The target values
- method : sklearn.base.BaseEstimator
- An estimator object that is used to compute the predictions. Has to provide fit and predict. Can be a Pipeline.
- cv_params : dict or list of dicts
- The GridSearchCV param_grid as described in sklearn docs: Dictionary with parameters names (string) as keys and lists of parameter settings to try as values, or a list of such dictionaries, in which case the grids spanned by each dictionary in the list are explored. This enables searching over any sequence of parameter settings.
- n_runs : int
- Number of times the cross validation will be repeated to get averages and standard deviations for the metrics.
- **kwargs : dict of keyword arguments
- These arguments will be passed to GridSearchCV. See the sklearn docs for details on which arguments are available.
- GridSearchCV
- The fitted GridSearchCV instance.
-
twistml.evaluation.evaluation.evaluate_binary_classification(X, y, method, cv_params, n_runs, use_coeff=False, force_dense=False, relative_test_size=0.2, return_auc=True, return_F1=True, **kwargs)¶ Runs cross validation on the given classification method. Returns evaluation metrics.
- X : array_like, shape = (n_samples, n_features)
- The feature vectors
- y : array-like, shape (n_samples,)
- The target values
- method : sklearn.base.BaseEstimator
- An estimator object that is used to compute the predictions. Has to provide fit and predict. Can be a Pipeline.
- cv_params : dict or list of dicts
- The GridSearchCV param_grid as described in sklearn docs: Dictionary with parameters names (string) as keys and lists of parameter settings to try as values, or a list of such dictionaries, in which case the grids spanned by each dictionary in the list are explored. This enables searching over any sequence of parameter settings.
- n_runs : int
- Number of times the cross validation will be repeated to get averages and standard deviations for the metrics.
- use_coeff : bool, optional
- To generate sensible AUC values, we need confidence values for the predictions. For linear SVMs these can be efficiently obtained by multplying the coefficients-vector w with the test data. For most other estimators we will use the predict_proba method. Setting use_coeff to True, means the coef_ will be used instead of predict_proba(). (default is False)
- force_dense : bool, optional
- Some estimators cannot handle sparse matrices (I’m looking at you, GradientBosstingClassifier). Setting this to True, will force conversion of sparse data to np.arrays. (default is False)
- relative_test_size : float, optional
- Determines the portion of data points to be kept out of training and held over as test set. (Default is 0.2, which implies 80% of the data are used for training and 20% for testing.)
- **kwargs : dict of keyword arguments
- These arguments will be passed to GridSearchCV. See the sklearn docs for details on which arguments are available.
- EvalBinClassResult : named tuple
- The named tuple has the fields auc and aucstd containing the average of the area under the roc curve and the respective standard deviation over the given number of runs.
-
twistml.evaluation.evaluation.evaluate_regression(X, y, method, cv_params, n_runs, force_dense=False, relative_test_size=0.2, simulate_classification=False, **kwargs)¶ Runs cross validation on the given regression method. Returns evaluation metrics.
- X : array_like, shape = (n_samples, n_features)
- The feature vectors
- y : array-like, shape (n_samples,)
- The target values
- method : sklearn.base.BaseEstimator
- An estimator object that is used to compute the predictions. Has to provide fit and predict. Can be a Pipeline.
- cv_params : dict or list of dicts
- The GridSearchCV param_grid as described in sklearn docs: Dictionary with parameters names (string) as keys and lists of parameter settings to try as values, or a list of such dictionaries, in which case the grids spanned by each dictionary in the list are explored. This enables searching over any sequence of parameter settings.
- n_runs : int
- Number of times the cross validation will be repeated to get averages and standard deviations for the metrics.
- force_dense : bool, optional
- Some estimators cannot handle sparse matrices (I’m looking at you, GradientBosstingClassifier). Setting this to True, will force conversion of sparse data to np.arrays. (default is False)
- relative_test_size : float, optional
- Determines the portion of data points to be kept out of training and held over as test set. (Default is 0.2, which implies 80% of the data are used for training and 20% for testing.)
- **kwargs : dict of keyword arguments
- These arguments will be passed to GridSearchCV. See the sklearn docs for details on which arguments are available.
- EvalRegrResult : named tuple
- The named tuple has the fields mse, msestd, mape and mapestd, containing the average of the mean squared error, the average of the mean absolute percentage error and their respective standard deviations over the given number of runs.
Will cause a divide by zero error in MAPE calculation if the targets y contain any zero entries.
-
twistml.evaluation.evaluation.mean_average_percentage_error(y_true, y_pred)¶ Calculates the mean absolute percentage error.
This will cause divide by zero error, if y_true contains zeroes.
-
twistml.evaluation.evaluation.selective_roc_auc_score(y_true, y_pred, lower_bound=0.2, upper_bound=0.8)¶ Ignore the data points for which the predictions are most uncertain. Return roc_auc_score for the rest.
This was an idea that I have not (yet) gotten to work with real data.
Module contents¶
<package summary>
<extended summary>
<module listings>
| Author: | Matthias Manhertz |
|---|---|
| Copyright: |
|
| Licence: | MIT |