2 Exploratory Data Analysis

(ns assignment.eda
  (:require
    [aerial.hanami.templates :as ht]
    [clojure.math.combinatorics :as combo]
    [fastmath.stats :as stats]
    [scicloj.kindly.v4.kind :as kind]
    [scicloj.ml.dataset :as ds]
    [scicloj.noj.v1.vis.hanami :as hanami]))

Load data

(defonce liver-disease
         (ds/dataset "data/bupa.csv"
                     {:key-fn (fn [colname]
                                (-> colname
                                    (clojure.string/replace #"\.|\s" "-")
                                    clojure.string/lower-case
                                    keyword))}))

2.1 Tables and nature of data

First seven of data

(ds/head liver-disease 7)

data/bupa.csv [7 7]:

:mcv	:alkphos	:sgpt	:sgot	:gammagt	:drinks	:selector
85	92	45	27	31	0.0	1
85	64	59	32	23	0.0	2
86	54	33	16	54	0.0	2
91	78	34	24	36	0.0	2
87	70	12	28	10	0.0	2
98	55	13	17	17	0.0	2
88	62	20	17	9	0.5	1

Descriptive statistics of columns

(ds/info liver-disease)

data/bupa.csv: descriptive-stats [7 11]:

:col-name	:datatype	:n-valid	:min	:mean	:max	:standard-deviation	:skew	:first	:last
:mcv	:int16	345	65.0	90.15942029	103.0	4.44809597	-0.38843312	85.0	98.0
:alkphos	:int16	345	23.0	69.86956522	138.0	18.34767034	0.75366677	92.0	99.0
:sgpt	:int16	345	4.0	30.40579710	155.0	19.51230891	3.06349864	45.0	57.0
:sgot	:int16	345	5.0	24.64347826	82.0	10.06449375	2.29307245	27.0	45.0
:gammagt	:int16	345	5.0	38.28405797	297.0	39.25461617	2.86609356	31.0	65.0
:drinks	:float64	345	0.0	3.45507246	20.0	3.33783526	1.54381943	0.0	20.0
:selector	:int16	345	1.0	1.57971014	2.0	0.49432233	-0.32438319	1.0	1.0

2.1.1 What is column :selector?

(set (:selector liver-disease))

#{1 2}

:selector takes two value only, 0 and 1. Let’s see descriptive statistics of each group.

(ds/info (tech.v3.dataset/filter liver-disease #(= (:selector %) 1)))

data/bupa.csv: descriptive-stats [7 11]:

:col-name	:datatype	:n-valid	:min	:mean	:max	:standard-deviation	:skew	:first	:last
:mcv	:int16	145	78.0	90.63448276	99.0	3.86906139	-0.13253898	85.0	98.0
:alkphos	:int16	145	23.0	71.97931034	138.0	18.59079408	0.62377127	92.0	99.0
:sgpt	:int16	145	10.0	31.20689655	103.0	15.77792786	1.88282131	45.0	57.0
:sgot	:int16	145	5.0	22.78620690	57.0	7.73806088	1.14900370	27.0	45.0
:gammagt	:int16	145	5.0	31.54482759	203.0	33.22502807	2.83802686	31.0	65.0
:drinks	:float64	145	0.0	3.54137931	20.0	3.92972268	1.80063118	0.0	20.0
:selector	:int16	145	1.0	1.00000000	1.0	0.00000000		1.0	1.0

(ds/info (tech.v3.dataset/filter liver-disease #(= (:selector %) 2)))

data/bupa.csv: descriptive-stats [7 11]:

:col-name	:datatype	:n-valid	:min	:mean	:max	:standard-deviation	:skew	:first	:last
:mcv	:int16	200	65.0	89.8150	103.0	4.80481072	-0.40881586	85.0	96.0
:alkphos	:int16	200	37.0	68.3400	134.0	18.06199263	0.86583251	64.0	69.0
:sgpt	:int16	200	4.0	29.8250	155.0	21.84491650	3.31719988	59.0	53.0
:sgot	:int16	200	8.0	25.9900	82.0	11.28880354	2.36380090	32.0	43.0
:gammagt	:int16	200	8.0	43.1700	297.0	42.51846894	2.80462310	23.0	203.0
:drinks	:float64	200	0.0	3.3925	12.0	2.84166661	0.84994446	0.0	12.0
:selector	:int16	200	2.0	2.0000	2.0	0.00000000		2.0	2.0

The data looks similar in terms of summary statistics per column in either selector equals 1 or 2, above. Three columns I am focusing on are, mean, standard deviation, and skew. Mean, standard deviation, and skew are similar for both groups in :mcv and :alkphos. The other four columns deviate in some way from the other :selector groups in either mean, standard deviation, and/or skew.

2.2 Plots

(def cols-of-interest
  (remove #{:selector} (ds/column-names liver-disease)))

2.2.1 Linearity with response and histogram of regressor.

^kind/vega
(let [dataset liver-disease
      make-plot (fn [field]
                  (-> dataset
                      (hanami/vconcat {}                    ;can switch to hconcat
                                      [(hanami/plot dataset ht/point-chart
                                                    {:X field :XSCALE {:zero false}
                                                     :Y :drinks :YSCALE {:zero false}})
                                       (hanami/histogram dataset field {:nbins 20})])))]
  (->> (map make-plot cols-of-interest) (hanami/hconcat {} {})))

(comment                                                    ;make plot alternative were I can change height and width but not nbins
  (let [dataset liver-disease]
    (-> dataset
        (hanami/hconcat {}
                        [(hanami/plot dataset ht/point-chart
                                      {:HEIGHT 125 :WIDTH 175
                                       :X      :mcv :XSCALE {:zero false}
                                       :Y      :drinks :YSCALE {:zero false}})
                         (hanami/plot dataset ht/bar-chart
                                      {:HEIGHT 125 :WIDTH 175
                                       :X      field :YAGG :count})]))))

2.2.2 Pairs-plots.

^kind/vega
(let [data (ds/rows liver-disease :as-maps)
      column-names cols-of-interest]
  {:data   {:values data}
   :repeat {:column column-names
            :row    column-names}
   :spec   {:height   100 :width 100
            :mark     "circle"
            :encoding {:x {:field {:repeat "column"} :type "quantitative" :scale {:zero false}}
                       :y {:field {:repeat "row"} :type "quantitative" :scale {:zero false}}}}})

(let [combos (combo/combinations cols-of-interest 2)]
  (for [[x y] combos]
    (assoc {} [x y] (stats/correlation (get liver-disease x) (get liver-disease y)))))

({[:mcv :alkphos] 0.04410299969572895}
 {[:mcv :sgpt] 0.14769504803185451}
 {[:mcv :sgot] 0.1877651531698029}
 {[:mcv :gammagt] 0.2223144936927525}
 {[:mcv :drinks] 0.3126795966511246}
 {[:alkphos :sgpt] 0.07620760528013328}
 {[:alkphos :sgot] 0.1460565465048546}
 {[:alkphos :gammagt] 0.13314039838436367}
 {[:alkphos :drinks] 0.10079605839698665}
 {[:sgpt :sgot] 0.7396748677149187}
 {[:sgpt :gammagt] 0.5034352544029691}
 {[:sgpt :drinks] 0.20684793492452233}
 {[:sgot :gammagt] 0.5276259280455446}
 {[:sgot :drinks] 0.2795877736826015}
 {[:gammagt :drinks] 0.3412239586693896})

We can see pearson correlation between each pair of variables of interest. Correlations closer to |1| represent variables that have a strong relationship to each other. For example, Looking at both the pairs-plot and the correlations, we see :sgpt :sgot are the most related variables and are related in the positive direction. Below we focus on the correlation of regressors on the response only.

(let [combos (combo/combinations cols-of-interest 2)]
  (for [[x y] combos
        :when (or (= :drinks x) (= :drinks y))]
    (assoc {} [x y] (stats/correlation (get liver-disease x) (get liver-disease y)))))

({[:mcv :drinks] 0.3126795966511246}
 {[:alkphos :drinks] 0.10079605839698665}
 {[:sgpt :drinks] 0.20684793492452233}
 {[:sgot :drinks] 0.2795877736826015}
 {[:gammagt :drinks] 0.3412239586693896})

(comment                                                    ;another way to make histograms for diagonals
  (let [data (ds/rows liver-disease :as-maps)
        column-names cols-of-interest]
    {:data   {:values data}
     :repeat {:column column-names}
     :spec   {:mark     "bar"
              :encoding {:x {:field {:repeat "column"} :type "quantitative"}
                         :y {:aggregate "count"}}}})

  (stats/correlation-matrix (ds/columns liver-disease) :spearman))

2.2.3 Box-plots.

^kind/vega
(let [data (ds/rows liver-disease :as-maps)
      column-names (remove #{:selector} (ds/column-names liver-disease))]
  {:data   {:values data}
   :repeat {:column column-names}
   :spec   {:width    60 :mark "boxplot"
            :encoding {:y {:field {:repeat "column"} :type "quantitative" :scale {:zero false}}}}})

Looking at the box-plots, the circle points are outliers, viz. points outside Q1 - 1.5IQR or Q3 + 1.5IQR. Below we count the number of outliers per column.

(let [columns cols-of-interest]
  (for [column columns]
    (assoc {} column (count (stats/outliers (get liver-disease column))))))

({:mcv 2} {:alkphos 9} {:sgpt 28} {:sgot 24} {:gammagt 27} {:drinks 5})

(comment
  ; created permutation of columns of interest. Wanted combinations, below
  (for [x cols-of-interest
        y (rest cols-of-interest)
        :when (not= x y)]
    [x y]))

source: src/assignment/eda.clj