2 ISLP Ch9 Q5
(ns assignment.islp-9-5
(:require
;plotting
[scicloj.kindly.v4.kind :as kind]
;[aerial.hanami.common :as hc]
[aerial.hanami.templates :as ht]
;[scicloj.metamorph.ml.viz :as ml-viz]
;[scicloj.noj.v1.vis.hanami.templates :as vht]
[scicloj.noj.v1.vis.hanami :as hanami]
;maths
[fastmath.random :as r]
[fastmath.stats :as stats]
;datasets
[tablecloth.api :as tc]
[tablecloth.pipeline :as tcm]
[tech.v3.dataset.metamorph :as dsm]
[tech.v3.datatype.functional :as dfn]
;machine learning
[scicloj.metamorph.core :as morph]
[scicloj.metamorph.ml :as ml]
[scicloj.metamorph.ml.classification :as mlc]
[scicloj.metamorph.ml.gridsearch :as grid]
[scicloj.metamorph.ml.loss :as loss]
[scicloj.ml.smile.classification]
;interop
;[libpython-clj2.require :refer [require-python]]
;[libpython-clj2.python :refer [py. py.. py.-] :as py]
[scicloj.sklearn-clj.ml]))- We have seen that we can ft an SVM with a non-linear kernel in order to perform classification using a non-linear decision boundary. We will now see that we can also obtain a non-linear decision boundary by performing logistic regression using non-linear transformations of the features.
- Generate a data set with n = 500 and p = 2, such that the observations belong to two classes with a quadratic decision boundary between them.
2.1 Generate Data
(defn generate-data [len]
(let [rng (r/rng :isaac 0)
x1 (map #(- % 0.5)
(r/->seq (r/distribution :uniform-real {:rng rng}) len))
x2 (map #(- % 0.5)
(r/->seq (r/distribution :uniform-real {:rng rng}) len))
y-int (map #(> (- (Math/pow %1 2) (Math/pow %2 2)) 0) x1 x2)
y (map {false 0 true 1} y-int)]
{:x1 x1 :x2 x2 :y y}))(def data
(tc/dataset (generate-data 500)))(tc/head data)_unnamed [5 3]:
| :x1 | :x2 | :y |
|---|---|---|
| 0.42144632 | -0.13972611 | 1 |
| -0.13155138 | 0.12495964 | 1 |
| -0.06103313 | 0.28605361 | 0 |
| 0.18657268 | -0.04808715 | 1 |
| -0.35271109 | -0.22763962 | 1 |
(tc/info data)_unnamed: descriptive-stats [3 11]:
| :col-name | :datatype | :n-valid | :n-missing | :min | :mean | :max | :standard-deviation | :skew | :first | :last |
|---|---|---|---|---|---|---|---|---|---|---|
| :x1 | :float64 | 500 | 0 | -0.49734823 | 0.00563017 | 0.49996755 | 0.28662407 | -0.03626498 | 0.42144632 | 0.37330638 |
| :x2 | :float64 | 500 | 0 | -0.49909375 | 0.00879174 | 0.49911608 | 0.28692970 | -0.03754993 | -0.13972611 | 0.33195477 |
| :y | :int64 | 500 | 0 | 0.00000000 | 0.50200000 | 1.00000000 | 0.50049675 | -0.00802416 | 1.00000000 | 1.00000000 |
- Plot the observations, colored according to their class labels. Your plot should display X1 on the x-axis, and X2 on the y-axis.
2.2 Plot data
^kind/vega-lite
(let [plot (tc/rows data :as-maps)]
{:data {:values plot}
:mark "circle"
:encoding {:x {:field :x1 :type "quantitative"}
:y {:field :x2 :type "quantitative"}
:color {:field :y :type "nominal"}}})(comment
;wont plot when rendered
(-> data
(hanami/plot ht/point-chart
{:X :x1
:Y :x2
:MSIZE 75
:COLOR "y"
:CTYPE "nominal"})))
- Fit a logistic regression model to the data, using X1 and X2 as predictors.
2.3 Logistic regression
(def response :y)(def regressors
(tc/column-names data (complement #{response})))2.3.1 Model task
Pipeline-vanilla in case want to do more in the ingestion.
(def pipeline-vanilla
(morph/pipeline
(dsm/categorical->number [response])
(dsm/set-inference-target response)))Add model context to ingestion.
(defn- create-model-pipeline
[pipeline-fn model-type params]
(morph/pipeline
pipeline-fn
{:metamorph/id :model}
(ml/model (merge {:model-type model-type} params))))#'assignment.islp-9-5/create-model-pipeline2.3.1.1 Logistic model
(defn logistic-pipe-fn
[pipeline-fn params]
(create-model-pipeline pipeline-fn :smile.classification/logistic-regression params))2.3.2 Evaluate chain
(defn train-test [dataset]
(tc/split->seq dataset :bootstrap {:seed 123 :repeats 20}))(defn train-val [dataset]
(let [ds-split (train-test dataset)]
(tc/split->seq (:train (first ds-split)) :kfold {:seed 123 :k 5})))2.3.2.1 Hyperparameter-grid
(comment
(ml/hyperparameters :smile.classification/logistic-regression))(defn generate-hyperparams [model-type]
(case model-type
:logistic (take 100
(grid/sobol-gridsearch
(ml/hyperparameters :smile.classification/logistic-regression)))))2.3.2.2 Work-horse
(defn evaluate-pipe [pipe data]
(ml/evaluate-pipelines
pipe
data
stats/cohens-kappa
:accuracy
{:other-metrices [{:name :mathews-cor-coef
:metric-fn stats/mcc}
{:name :accuracy
:metric-fn loss/classification-accuracy}]
:return-best-pipeline-only false
:return-best-crossvalidation-only true}))(defn evaluate-model [dataset split-fn model-type model-fn pipeline-fn]
(let [data-split (split-fn dataset)
pipelines (map (partial model-fn pipeline-fn) (generate-hyperparams model-type))]
(evaluate-pipe pipelines data-split)))change for different models to test
(def model-type-fns
{:logistic logistic-pipe-fn})(defn evaluate-models [dataset split-fn pipeline-fn]
(mapv (fn [[model-type model-fn]]
(evaluate-model dataset split-fn model-type model-fn pipeline-fn))
model-type-fns))(comment
; Alternative: if want to expand `model-type-fns` to simplify `evaluate-models`
(def model-type-fns
{:logistic [logistic-pipe-fn pipeline-vanilla]})
(defn evaluate-model [dataset split-fn model-type model-and-pipeline]
(let [[model-fn pipeline-fn] model-and-pipeline
data-split (split-fn dataset)
pipelines (map (partial model-fn pipeline-fn) (generate-hyperparams model-type))]
(evaluate-pipe pipelines data-split)))
(defn evaluate-models [dataset split-fn]
(mapv (fn [[model-type model-and-pipeline]]
(evaluate-model dataset split-fn model-type model-and-pipeline))
model-type-fns))
(def logistic-models (evaluate-models data train-test)))2.3.3 View model/s
(defn best-models [eval]
(->> eval
flatten
(map
#(hash-map :summary (ml/thaw-model (get-in % [:fit-ctx :model]))
:fit-ctx (:fit-ctx %)
:timing-fit (:timing-fit %)
:metric ((comp :metric :test-transform) %)
:other-metrices ((comp :other-metrices :test-transform) %)
:other-metric-1 ((comp :metric first) ((comp :other-metrices :test-transform) %))
:other-metric-2 ((comp :metric second) ((comp :other-metrices :test-transform) %))
:params ((comp :options :model :fit-ctx) %)
:pipe-fn (:pipe-fn %)))
(sort-by :metric)))2.3.4 Get model/s
(comment
(def logistic-models (evaluate-models data train-test pipeline-vanilla))
(def logistic-model
(-> logistic-models
best-models
reverse))
(-> logistic-model first :summary)
(-> logistic-model first :metric)
(-> logistic-model first :other-metrices)
(-> logistic-model first :params)
;=>
;{:model-type :smile.classification/logistic-regression,
; :lambda 79.31055172413794,
; :tolerance 1.0E-9,
; :max-iterations 9478}
(-> logistic-model first :fit-ctx :model :options))(def params
{:model-type :smile.classification/logistic-regression,
:lambda 79.31055172413794,
:tolerance 1.0E-9,
:max-iterations 9478})(def logistic-model
(first (evaluate-pipe
(map (partial logistic-pipe-fn pipeline-vanilla) params)
(train-test data))))
- Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be linear.
2.4 Visualize model fit
Function to get the best model’s training data.
(defn model->data [model]
(let [processed-data (-> model first :fit-ctx :model :model-data :smile-df-used)
keys-vec (-> data
(morph/pipe-it (-> model first :pipe-fn))
keys)
without-y (vec (remove #(= :y %) keys-vec))
with-y-at-end (conj without-y :y)
data-map (zipmap with-y-at-end processed-data)]
(tc/dataset data-map)))(def predictions
(-> (model->data logistic-model)
(morph/transform-pipe
(-> logistic-model first :pipe-fn)
(-> logistic-model first :fit-ctx))
:metamorph/data
:y
vec))(def data-predict
(tc/add-or-replace-column (model->data logistic-model) :y predictions))^kind/vega-lite
(let [plot (tc/rows data-predict :as-maps)]
{:data {:values plot}
:mark "circle"
:encoding {:x {:field :x1 :type "quantitative"}
:y {:field :x2 :type "quantitative"}
:color {:field :y :type "nominal"}}})2.4.1 Model performance
Functions to view performance on full data.
(defn preds
[model]
(-> data
(morph/transform-pipe
(-> model first :pipe-fn)
(-> model first :fit-ctx))
:metamorph/data
:y
(->> (map #(long %))
vec)))(defn actual
[model]
(-> data
(morph/fit-pipe
(-> model first :pipe-fn))
:metamorph/data
:y
vec))(defn evaluate-predictions
"Evaluates predictions against actual labels, returns confusion map and metrics."
[preds actual]
(let [conf-map (mlc/confusion-map->ds (mlc/confusion-map preds actual :none))
kappa (stats/cohens-kappa preds actual)
mcc (stats/mcc preds actual)]
{:confusion-map conf-map
:cohens-kappa kappa
:mcc mcc}))^kind/dataset
(evaluate-predictions (preds logistic-model) (actual logistic-model)){:confusion-map _unnamed [3 3]:
| :column-name | 0 | 1 |
|---|---|---|
| column-name | 0 | 1 |
| 0 | 235 | 14 |
| 1 | 174 | 77 |
, :cohens-kappa 0.2499082334540928, :mcc 0.3246722267851866}
- Now fit a logistic regression model to the data using non-linear functions of X\(_1\) and X\(_2\) as predictors (e.g. \(X_1^2\), \(X_1*X_2\), \(log(X_2)\), and so forth)
2.5 Non-linear regressors
Making pipelines (like vanilla) to apply to evaluate-models
(def pipeline-squared
(morph/pipeline
(tcm/add-or-replace-columns {:x1-sq (fn [row]
(map #(Math/pow % 2) (:x1 row)))
:x2-sq (fn [row]
(map #(Math/pow % 2) (:x2 row)))})
(dsm/categorical->number [response])
(dsm/set-inference-target response)))(def pipeline-interact
(morph/pipeline
(tcm/add-or-replace-columns {:x1-x2 (fn [ds] (dfn/* (:x1 ds)
(:x2 ds)))})
(dsm/categorical->number [response])
(dsm/set-inference-target response)))(def pipeline-combined
(morph/pipeline
(tcm/add-or-replace-columns {:x1-x2 (fn [ds] (dfn/* (:x1 ds)
(:x2 ds)))
:x1-sq (fn [row]
(map #(Math/pow % 2) (:x1 row)))
:x2-sq (fn [row]
(map #(Math/pow % 2) (:x2 row)))})
(dsm/categorical->number [response])
(dsm/set-inference-target response)))2.5.1 Set the model case name and respective model pipeline
(def model-type-fns
{:logistic logistic-pipe-fn})(comment
(def pipe-squared
(evaluate-models data train-test pipeline-squared))
(def pipe-interact
(evaluate-models data train-test pipeline-interact))
(def pipe-combined
(evaluate-models data train-test pipeline-combined)))Don’t want to run the full evaluate models Squared
(def params
{:model-type :smile.classification/logistic-regression,
:lambda 0.001,
:tolerance 0.07894736863157896,
:max-iterations 3747})(def pipe-squared
(first (evaluate-pipe
(map (partial logistic-pipe-fn pipeline-squared) params)
(train-test data))))(def logistic-squared
(-> pipe-squared best-models reverse))(-> logistic-squared first :fit-ctx :model :feature-columns)[:x1 :x2 :x1-sq :x2-sq]Interact
(def params
{:model-type :smile.classification/logistic-regression,
:lambda 41.37989655172413,
:tolerance 1.0E-9,
:max-iterations 4268})(def pipe-interact
(first (evaluate-pipe
(map (partial logistic-pipe-fn pipeline-interact) params)
(train-test data))))(def logistic-interact
(-> pipe-interact best-models reverse))(-> logistic-interact first :fit-ctx :model :feature-columns)[:x1 :x2 :x1-x2]Combined
(def params
{:model-type :smile.classification/logistic-regression,
:lambda 0.001,
:tolerance 0.07894736863157896,
:max-iterations 3747} )(def pipe-combined
(first (evaluate-pipe
(map (partial logistic-pipe-fn pipeline-combined) params)
(train-test data))))(def logistic-combined
(-> pipe-combined best-models reverse))(-> logistic-combined first :fit-ctx :model :feature-columns)[:x1 :x2 :x1-x2 :x1-sq :x2-sq]^kind/dataset
(evaluate-predictions (preds logistic-squared) (actual logistic-squared)){:confusion-map _unnamed [3 3]:
| :column-name | 0 | 1 |
|---|---|---|
| column-name | 0 | 1 |
| 0 | 240 | 9 |
| 1 | 0.000 | 251 |
, :cohens-kappa 0.9639942390782525, :mcc 0.964619715510437}
^kind/dataset
(evaluate-predictions (preds logistic-interact) (actual logistic-interact)){:confusion-map _unnamed [3 3]:
| :column-name | 0 | 1 |
|---|---|---|
| column-name | 0 | 1 |
| 0 | 191 | 58 |
| 1 | 123 | 128 |
, :cohens-kappa 0.27674061760756985, :mcc 0.28657584572643147}
^kind/dataset
(evaluate-predictions (preds logistic-combined) (actual logistic-combined)){:confusion-map _unnamed [3 3]:
| :column-name | 0 | 1 |
|---|---|---|
| column-name | 0 | 1 |
| 0 | 249 | 0.000 |
| 1 | 5 | 246 |
, :cohens-kappa 0.9800012799180852, :mcc 0.98019731389307}
- Apply this model to the training data in order to obtain a predicted class label for each training observation. Plot the observations, colored according to the predicted class labels. The decision boundary should be obviously non-linear. If it is not, then repeat (a)–(e) until you come up with an example in which the predicted class labels are obviously non-linear.
2.6 Non-linear logistic visualization
(def data-plot
(tc/add-or-replace-column data :y (preds logistic-interact)))^kind/vega-lite
(let [plot (tc/rows data-plot :as-maps)]
{:data {:values plot}
:mark "circle"
:encoding {:x {:field :x1 :type "quantitative"}
:y {:field :x2 :type "quantitative"}
:color {:field :y :type "nominal"}}})(def data-plot
(tc/add-or-replace-column data :y (preds logistic-squared)))^kind/vega-lite
(let [plot (tc/rows data-plot :as-maps)]
{:data {:values plot}
:mark "circle"
:encoding {:x {:field :x1 :type "quantitative"}
:y {:field :x2 :type "quantitative"}
:color {:field :y :type "nominal"}}})
- Fit a support vector classifer to the data with X1 and X2 as predictors. Obtain a class prediction for each training observation. Plot the observations, colored according to the predicted class labels.
2.7 Support vector visualization
Many issues with sklearn. One of which, major parameter “C” cost, does not work.
(comment
;failed a lot, checked and found "c" cant be param, strange
(def svm-pipe
(morph/pipeline
(dsm/categorical->number [response])
(dsm/set-inference-target response)
{:metamorph/id :model}
(ml/model {:model-type :sklearn.classification/svc
:kernel "poly"
:degree 7
;:c 0.001 ;doesnt work with c???
:predict-proba? false})))
(def ds-split
(tc/split->seq data :bootstrap {:seed 123 :repeats 20}))
(evaluate-pipe [svm-pipe] ds-split))2.7.1 SVM context pipeline
Like the logistic pipeline context.
(defn svm-pipe-fn
[pipeline-fn params]
(create-model-pipeline pipeline-fn :sklearn.classification/svc params))(comment ;nil
(ml/hyperparameters :sklearn.classification/svc))2.7.2 Set the model case name and respective model pipeline
(defn generate-hyperparams [model-type]
(case model-type
:svm-linear (grid/sobol-gridsearch
{:kernel "linear"
:predict-proba? false})
:svm-rbf (grid/sobol-gridsearch
{:predict-proba? false})
:svm-sigmoid (grid/sobol-gridsearch
{:kernel "sigmoid"
:predict-proba? false})
:svm-poly (grid/sobol-gridsearch
{:kernel "poly"
:degree (grid/linear 1 4 4 :int32)
:predict-proba? false})))(def model-type-fns
{:svm-linear svm-pipe-fn
:svm-rbf svm-pipe-fn
:svm-sigmoid svm-pipe-fn
:svm-poly svm-pipe-fn})2.7.3 Collect all four SVM models.
(def svm-models (evaluate-models data train-test pipeline-vanilla))2.7.4 Extract first for the `:svm-linear.
(def svm-linear
(-> svm-models first best-models reverse))(-> svm-linear first :fit-ctx :model :options){:model-type :sklearn.classification/svc,
:kernel "linear",
:predict-proba? false}^kind/dataset
(evaluate-predictions (preds svm-linear) (actual svm-linear)){:confusion-map _unnamed [3 3]:
| :column-name | 0 | 1 |
|---|---|---|
| column-name | 0 | 1 |
| 0 | 159 | 90 |
| 1 | 121 | 130 |
, :cohens-kappa 0.1564049256356948, :mcc 0.15762023273027012}
(def data-plot
(tc/add-or-replace-column data :y (preds svm-linear)))^kind/vega-lite
(let [plot (tc/rows data-plot :as-maps)]
{:data {:values plot}
:mark "circle"
:encoding {:x {:field :x1 :type "quantitative"}
:y {:field :x2 :type "quantitative"}
:color {:field :y :type "nominal"}}})
- Fit a SVM using a non-linear kernel to the data. Obtain a class prediction for each training observation. Plot the observations,colored according to the predicted class labels
2.8 Non-linear kernels
Already made in the last question, now extract from the svm-models collection.
(def svm-rbf
(-> svm-models second best-models reverse))(-> svm-rbf first :fit-ctx :model :options){:model-type :sklearn.classification/svc, :predict-proba? false}(def data-plot
(tc/add-or-replace-column data :y (preds svm-rbf)))^kind/vega-lite
(let [plot (tc/rows data-plot :as-maps)]
{:data {:values plot}
:mark "circle"
:encoding {:x {:field :x1 :type "quantitative"}
:y {:field :x2 :type "quantitative"}
:color {:field :y :type "nominal"}}})^kind/dataset
(evaluate-predictions (preds svm-rbf) (actual svm-rbf)){:confusion-map _unnamed [3 3]:
| :column-name | 0 | 1 |
|---|---|---|
| column-name | 0 | 1 |
| 0 | 236 | 13 |
| 1 | 1 | 250 |
, :cohens-kappa 0.943988349576712, :mcc 0.9450781610261636}
(def svm-sigmoid
(-> svm-models rest second best-models reverse))(-> svm-sigmoid first :fit-ctx :model :options){:model-type :sklearn.classification/svc,
:kernel "sigmoid",
:predict-proba? false}(def data-plot
(tc/add-or-replace-column data :y (preds svm-sigmoid)))^kind/vega-lite
(let [plot (tc/rows data-plot :as-maps)]
{:data {:values plot}
:mark "circle"
:encoding {:x {:field :x1 :type "quantitative"}
:y {:field :x2 :type "quantitative"}
:color {:field :y :type "nominal"}}})^kind/dataset
(evaluate-predictions (preds svm-sigmoid) (actual svm-sigmoid)){:confusion-map _unnamed [3 3]:
| :column-name | 0 | 1 |
|---|---|---|
| column-name | 0 | 1 |
| 0 | 127 | 122 |
| 1 | 98 | 153 |
, :cohens-kappa 0.11964785914365761, :mcc 0.120203484336723}
(def svm-poly
(-> svm-models last best-models reverse))(-> svm-poly first :fit-ctx :model :options){:model-type :sklearn.classification/svc,
:kernel "poly",
:degree 2,
:predict-proba? false}(def data-plot
(tc/add-or-replace-column data :y (preds svm-poly)))^kind/vega-lite
(let [plot (tc/rows data-plot :as-maps)]
{:data {:values plot}
:mark "circle"
:encoding {:x {:field :x1 :type "quantitative"}
:y {:field :x2 :type "quantitative"}
:color {:field :y :type "nominal"}}})^kind/dataset
(evaluate-predictions (preds svm-poly) (actual svm-poly)){:confusion-map _unnamed [3 3]:
| :column-name | 0 | 1 |
|---|---|---|
| column-name | 0 | 1 |
| 0 | 248 | 1 |
| 1 | 3 | 248 |
, :cohens-kappa 0.984000255995904, :mcc 0.984031744507912}
- Comment on your results.
Linear and Sigmoid kernels in our SVM models perform poorly on certain polynomial relationships. We can model those relationships “linearly” by creating polynomial regressors in our model. Another way is to use different SVM kernel functions. Scikit-learn’s default kernel function–RBF–provided a nice non-linear option. Also, the poly kernel was able to identify that degree two is the appropriate power for our data. RBF and poly performed similarly on our data, while poly is the best choice with a Kappa score of 0.98.
source: src/assignment/islp_9_5.clj