[Julia] 配列に対して一律の演算をする場合

配列のすべての要素に一律の演算をした場合があります．
Juliaでは，その際に以下のように記載することで演算が可能になります．

まずは，平均を取るのにmean関数を使うので，Statisticsパッケージをインポートします．

julia> using Statistics

この例では，1, 2, 3, 4, 6, 7, 8, 9 という平均が5になる配列（data）を用います．

mean関数を使って，この配列の平均μを取ると5になります．

配列 data の各要素から一律にμを引くと標準化されます．この時，以下の例では .- μ としており，演算子の前に . （ドット）を付けます．

標準化した配列の平均値は0になっていることがわかります．

julia> data = [1, 2, 3, 4, 6, 7, 8, 9]

8-element Array{Int64,1}:

 1

 2

 3

 4

 6

 7

 8

 9

julia> μ = mean(data)

5.0

julia> standardized_data = data .- μ

8-element Array{Float64,1}:

 -4.0

 -3.0

 -2.0

 -1.0

  1.0

  2.0

  3.0

  4.0

julia> μ2 = mean(standardized_data)

0.0

一律の演算を関数適用する場合は，以下の sin.()のように引数のカッコの前に . （ドット）を挿入します．

julia> rad = [0.523599, 0.785398, 1.5708]

3-element Array{Float64,1}:

 0.523599

 0.785398

 1.5708  

julia> sin_angle = sin.(rad)

3-element Array{Float64,1}:

 0.5000001943375613

 0.7071066656470943

 0.9999999999932537

ロジスティック回帰

ロジスティック回帰についてのメモです．内容はPythonで学ぶあたらしい統計学の教科書 ，機械学習のエッセンスを参考にしています．

ロジスティック回帰は，確率分布に二項分布を用いて，リンク関数にロジッド関数を用いた一般線形化モデルで，主に二値分類に用いられるアルゴリズムです．説明変数は複数あっても構わず，連続型・カテゴリがたが混在していても支障はありません．

ここで，ラベルの値は0 or 1とします．与えられた特徴量のサンプル${\bf x} \in {\mathcal{R}}^{d}$に対して，ラベル$y$が1になる確率を$P(Y = 0 | X = x)＄で表します．
ロジスティック回帰は次式を前提としたモデルです．

$P(Y = 1 | X = x)  = \sigma (w_{0} + \sum_{j = 1}^{d} x_{j} w_{j}) = \sigma ({\bf{w}}^{T} {\bf{\tilde{x}}}^{T})$

$\sigma$の中は${\bf{x}}$の線形関数となっており，このアイデアは，サンプル$x$がラベル1に属する確からしさは線形関数の値が大きいほど大きくなるという仮定です．ここで，線形関数であることから，その値は$- \infty$から$\infty$の間をとることになります．その線形関数の値にシグモイド関数を作用させることで確率として扱うことができるようになります．
ここで，$\sigma$はシグモイド関数で

$\sigma(\eta) = \cfrac{1}{1 + e^{-\eta}}$

で定義されます．

特徴量行列$X$とラベルのベクトル${\bf{y}}$が与えられたときに，その$X$から${\bf{y}}$が生じる確率を考えます．
行列$X$のi番目のサンプルがラベル$y_{i}$に分類される確率を全て掛け算したものとして考えるこができて，以下のようになります．

$P(y | x) =  \prod_{k=1}^{n} [ \sigma ({\bf{\tilde{x_{k}}}} {\bf{w}})^{y_{k}} (1 - \sigma({\bf{\tilde{x_{k}}}}) {\bf{w}}))^{1-y_{k}} ]$

この確率を最大化することを考えます（このままでは考えにくいので，対数をとってマイナスをつけると下式のようになります）．

$E({\bf{w}}) = - \log P ( {\bf{y}} | X) = - \sum_{k = 1}^{n} [ y_{k} \log \sigma ( {\bf{w}}^{T} {\tilde{\bf{x_{k}}}}) + (1 - y_{k}) \log ( 1 - \sigma ({\bf{w}}^{T} {\tilde{x_{k}}})) ]$

この$E({\bf{w}})$の最適値をニュートン法で求めます．つまり$\nabla E({\bf{w}}) = 0$の解をニュートン法で求めることになります．

これをPythonで実装すると以下のようになります．
ファイル名：logisticreg.py

import numpy as np

from scipy import linalg

THRESHMIN = 1e-10

def sigmoid(x):

    return 1 / (1 + np.exp(-x))

class LogisticRegression:

    def __init__(self, tol=0.001, max_iter=3, random_seed=0):

        self.tol = tol

        self.max_iter = max_iter

        self.random_state = np.random.RandomState(random_seed)

        self.w_ = None

    def fit(self, X, y):

        self.w_ = self.random_state.randn(X.shape[1] + 1)

        Xtil = np.c_[np.ones(X.shape[0]), X]

        diff = np.inf

        w_prev = self.w_

        iter = 0

        while diff > self.tol and iter < self.max_iter:

            yhat = sigmoid(np.dot(Xtil, self.w_))

            r = np.clip(yhat * (1 - yhat),

                        THRESHMIN, np.inf)

            XR = Xtil.T * r

            XRX = np.dot(Xtil.T * r, Xtil)

            w_prev = self.w_

            b = np.dot(XR, np.dot(Xtil, self.w_) -

                       1 / r * (yhat - y))

            self.w_ = linalg.solve(XRX, b)

            diff = abs(w_prev - self.w_).mean()

            iter += 1

    def predict(self, X):

        Xtil = np.c_[np.ones(X.shape[0]), X]

        yhat = sigmoid(np.dot(Xtil, self.w_))

        return np.where(yhat > .5, 1, 0)

ファイル名：logisticreg_wdbc.py

import logisticreg

import csv

import numpy as np

n_test = 100

X = []

y = []

with open("wdbc.data") as fp:

    for row in csv.reader(fp):

        if row[1] == "B":

            y.append(0)

        else:

            y.append(1)

        X.append(row[2:])

y = np.array(y, dtype=np.float64)

X = np.array(X, dtype=np.float64)

y_train = y[:-n_test]

X_train = X[:-n_test]

y_test = y[-n_test:]

X_test = X[-n_test:]

model = logisticreg.LogisticRegression(tol=0.01)

model.fit(X_train, y_train)

y_predict = model.predict(X_test)

n_hits = (y_test == y_predict).sum()

print("Accuracy: {}/{} = {}".format(n_hits, n_test, n_hits / n_test))

実行すると以下のようになります（実行する際には，logisticreg_wdbc.pyとwdbc.dataが同じディレクトリにある必要があります）．

$ python3 logisticreg_wdbc.py

Accuracy: 97/100 = 0.97

Juliaで実装すると以下のようになります．
ファイル名：logisticreg.jl

module logisticreg

using LinearAlgebra

using Random

using Statistics

THRESHMIN = 1e-10

sigmoid(x) = 1 ./ (1 .+ ℯ.^(-x))

mutable struct LogisticRegression

    tol

    max_iter

    random_seed

    w_

    function LogisticRegression(tol, max_iter=3, random_seed=0)

        new(tol, max_iter, random_seed, Nothing)

    end

end

function fit(s::LogisticRegression, X, y)

    Random.seed!(s.random_seed)

    s.w_ = randn(size(X)[2] + 1)

    Xtil = hcat(ones(size(X)[1]), X)

    diff = Inf

    w_prev = s.w_

    iter = 0

    while diff > s.tol && iter < s.max_iter

        yhat = sigmoid(Xtil * s.w_)

        r = clamp.(yhat .* (1 .- yhat), THRESHMIN, Inf)

        XR = Xtil' .* r'

        XRX = (Xtil' .* r') * Xtil

        w_prev = s.w_

        b = XR * (Xtil * s.w_ .- (1 ./ r) .* (yhat - y))

        s.w_ = XRX \ b

        diff = mean(abs.(w_prev - s.w_))

        iter = iter + 1

    end

end

function predict(s::LogisticRegression, X)

    Xtil = hcat(ones(size(X)[1]), X)

    yhat = sigmoid(Xtil * s.w_)

    return [ifelse(x .> .5, 1, 0) for x in yhat]

end

end

ファイル名：logisticreg_wdbc.jl

include("logisticreg.jl")

using .logisticreg

using CSV

n_test = 100

dataframe = CSV.read("wdbc.data", header=false)

row,col=size(dataframe)

X = Float64[dataframe[r,c] for r in 1:row, c in 3:col]

y = Float64[ifelse(dataframe[r,2] == "B", 0, 1) for r in 1:row]

y_train = y[1:end-n_test]

X_train = X[1:row-n_test, :]

y_test = y[end-n_test+1:end]

X_test = X[end-n_test+1:end, :]

model = logisticreg.LogisticRegression(0.01)

logisticreg.fit(model, X_train, y_train)

y_predict = logisticreg.predict(model, X_test)

n_hits = sum(y_test .== y_predict)

println("Accuracy: $n_hits/$n_test = $(n_hits / n_test)")

実行すると以下のようになります（実行する際には，logisticreg_wdbc.pyとwdbc.dataが同じディレクトリにある必要があります）．

$ Julia logisticreg_wdbc.jl
Accuracy: 97/100 = 0.97

[Jupyter Notebook] matplotlibのインライン展開

matplotlibで描画をしようとした際に，インライン展開されない場合は，以下のように

%matplotlib inline

と書くことで，インライン展開されます．

例えば，matplotlibでグラフを示そうとしているのに，図が現れず

<Figure size 640x480 with 1 Axes>

というような表示のみが返ってきているときはインライン展開されていません．このような時は描画する命令の前に，上記のように

%matplotlib inline

と書くとグラフが表示されるようになるはずです．

[Python] 主成分分析

主成分分析に関するメモです．

主成分分析を行うには scikit-learn パッケージを使用して，sklearn.decomposition の PCA でインスタンスを生成します．
以下の例では，Davis データを用いて主成分分析を行っています．

Davisデータ（Davis.csv）はJupyter Notebookの保存されているディレクトリと同じディレクトリに保存されているものとします．
Davisデータの読み込みには pandas パッケージの pd.read_csv を使用します．
データ配列の第1, 2列の各行がデータ点${\bf{x_{i}}} = ( w_{i}, h_{i} )$に対応しています（$x_{i}$は$i$番目の人の体重[kg]，$h_{i}$は身長[cm]に対応）．

パッケージの読み込みを行います．

>>> import numpy as np

>>> import matplotlib.pyplot as plt

>>> import pandas as pd

sklearn の PCA を使います．

>>> from sklearn.decomposition import PCA

pandasを使ってデータ読み込みます．読みこむDavis.csvファイル*は，REPLを実行しているディレクトリにあるものとしているので，必要に応じてパスの書き換えが必要です．

>>> dat = pd.read_csv('Davis.csv').values

身長の単位を[m]に変換し，対数の値を計算します．

>>> logdat = np.log(np.c_[dat[:,1],dat[:,2]/100].astype('float'))

データのプロットを行います．

>>> plt.plot(logdat[:,0], logdat[:,1], '.'); plt.show()

[<matplotlib.lines.Line2D object at 0x11df9dac8>]

読み込んだデータに対して主成分分析を行います．

>>> pca = PCA()>>> pca.fit(logdat)
PCA(copy=True, iterated_power='auto', n_components=None, random_state=None,

  svd_solver='auto', tol=0.0, whiten=False)

>>> pca.components_

array([[ 0.99672116,  0.08091309],

       [ 0.08091309, -0.99672116]])

>>> 

上記のコードの pca.components_ は主成分です．

インデックス 11 のデータは外れ値として除去することにします．

>>> clean_logdat = np.delete(logdat, 11, axis=0)

外れ値（インデックス 11）を除去したデータに主成分分析を行います．

>>> pca = PCA() 

>>> pca.fit(clean_logdat) 

PCA(copy=True, iterated_power='auto', n_components=None, random_state=None,

  svd_solver='auto', tol=0.0, whiten=False)

>>> pca.components_

array([[ 0.97754866,  0.21070979],

       [-0.21070979,  0.97754866]])

>>> 

----------
* 上記のコードで読み込む.csvファイル（Davis.csv）の中身
sex,weight,height,repwt,repht
M,77,182,77,180
F,58,161,51,159
F,53,161,54,158
M,68,177,70,175
F,59,157,59,155
M,76,170,76,165
M,76,167,77,165
M,69,186,73,180
M,71,178,71,175
M,65,171,64,170
M,70,175,75,174
F,166,57,56,163
F,51,161,52,158
F,64,168,64,165
F,52,163,57,160
F,65,166,66,165
M,92,187,101,185
F,62,168,62,165
M,76,197,75,200
F,61,175,61,171
M,119,180,124,178
F,61,170,61,170
M,65,175,66,173
M,66,173,70,170
F,54,171,59,168
F,50,166,50,165
F,63,169,61,168
F,58,166,60,160
F,39,157,41,153
M,101,183,100,180
F,71,166,71,165
M,75,178,73,175
M,79,173,76,173
F,52,164,52,161
F,68,169,63,170
M,64,176,65,175
F,56,166,54,165
M,69,174,69,171
M,88,178,86,175
M,65,187,67,188
F,54,164,53,160
M,80,178,80,178
F,63,163,59,159
M,78,183,80,180
M,85,179,82,175
F,54,160,55,158
M,73,180,NA,NA
F,49,161,NA,NA
F,54,174,56,173
F,75,162,75,158
M,82,182,85,183
F,56,165,57,163
M,74,169,73,170
M,102,185,107,185
M,64,177,NA,NA
M,65,176,64,172
F,66,170,65,NA
M,73,183,74,180
M,75,172,70,169
M,57,173,58,170
M,68,165,69,165
M,71,177,71,170
M,71,180,76,175
F,78,173,75,169
M,97,189,98,185
F,60,162,59,160
F,64,165,63,163
F,64,164,62,161
F,52,158,51,155
M,80,178,76,175
F,62,175,61,171
M,66,173,66,175
F,55,165,54,163
F,56,163,57,159
F,50,166,50,161
F,50,171,NA,NA
F,50,160,55,150
F,63,160,64,158
M,69,182,70,180
M,69,183,70,183
F,61,165,60,163
M,55,168,56,170
F,53,169,52,175
F,60,167,55,163
F,56,170,56,170
M,59,182,61,183
M,62,178,66,175
F,53,165,53,165
F,57,163,59,160
F,57,162,56,160
M,70,173,68,170
F,56,161,56,161
M,84,184,86,183
M,69,180,71,180
M,88,189,87,185
F,56,165,57,160
M,103,185,101,182
F,50,169,50,165
F,52,159,52,153
F,55,155,NA,154
F,55,164,55,163
M,63,178,63,175
F,47,163,47,160
F,45,163,45,160
F,62,175,63,173
F,53,164,51,160
F,52,152,51,150
F,57,167,55,164
F,64,166,64,165
F,59,166,55,163
M,84,183,90,183
M,79,179,79,171
F,55,174,57,171
M,67,179,67,179
F,76,167,77,165
F,62,168,62,163
M,83,184,83,181
M,96,184,94,183
M,75,169,76,165
M,65,178,66,178
M,78,178,77,175
M,69,167,73,165
F,68,178,68,175
F,55,165,55,163
M,67,179,NA,NA
F,52,169,56,NA
F,47,153,NA,154
F,45,157,45,153
F,68,171,68,169
F,44,157,44,155
F,62,166,61,163
M,87,185,89,185
F,56,160,53,158
F,50,148,47,148
M,83,177,84,175
F,53,162,53,160
F,64,172,62,168
F,62,167,NA,NA
M,90,188,91,185
M,85,191,83,188
M,66,175,68,175
F,52,163,53,160
F,53,165,55,163
F,54,176,55,176
F,64,171,66,171
F,55,160,55,155
F,55,165,55,165
F,59,157,55,158
F,70,173,67,170
M,88,184,86,183
F,57,168,58,165
F,47,162,47,160
F,47,150,45,152
F,55,162,NA,NA
F,48,163,44,160
M,54,169,58,165
M,69,172,68,174
F,59,170,NA,NA
F,58,169,NA,NA
F,57,167,56,165
F,51,163,50,160
F,54,161,54,160
F,53,162,52,158
F,59,172,58,171
M,56,163,58,161
F,59,159,59,155
F,63,170,62,168
F,66,166,66,165
M,96,191,95,188
F,53,158,50,155
M,76,169,75,165
F,54,163,NA,NA
M,61,170,61,170
M,82,176,NA,NA
M,62,168,64,168
M,71,178,68,178
F,60,174,NA,NA
M,66,170,67,165
M,81,178,82,175
M,68,174,68,173
M,80,176,78,175
F,43,154,NA,NA
M,82,181,NA,NA
F,63,165,59,160
M,70,173,70,173
F,56,162,56,160
F,60,172,55,168
F,58,169,54,166
M,76,183,75,180
F,50,158,49,155
M,88,185,93,188
M,89,173,86,173
F,59,164,59,165
F,51,156,51,158
F,62,164,61,161
M,74,175,71,175
M,83,180,80,180
M,81,175,NA,NA
M,90,181,91,178
M,79,177,81,178

[Python] 交差検証法によるテスト誤差の推定

K重交差検証法によってテスト誤差を推定する際のメモです．

K重交差検証法の計算プロセスは以下のようになります．

• 学習方法と損失：チューニングパラメータ$\lambda$をもつアルゴリズム$\mathcal{A}$，損失$l ({\bf{z}}, h)$
• 入力：データ${\bf{z}}_{1}, {\bf{z}}_{2}, \cdots, {\bf{z}}_{n}$

1. 入力データを要素数がほぼ等しいK個のグループ$D_{1}, D_{2}, \cdots, D_{K}$に分割する．
2. $k = 1, 2, \cdots, K$として，以下を反復する．
1. $h_{\lambda, k} = \mathcal{A} (D^{(k)}, \lambda)$を計算する．ここで，$D^{(k)} = D \ D_{k}$.
2. $\widehat{Err} ( h_{\lambda, k}) = \cfrac{1}{m}\sum_{x \in D_{k}} l ({\bf{z}},  h_{\lambda, k})$を計算する．
3. $\widehat{Err} ( \mathcal{A}; \lambda) = \cfrac{1}{K}\sum_{k = 1}^{K} \widehat{Err} ( h_{\lambda, k} )$

回帰関数は決定木によって推定することにします．よって，推定には，sklearn.tree の DecisionTreeRegressor を使用します．

パッケージの読み込み

>>> import numpy as np

>>> import scipy as sp

>>> import matplotlib.pyplot as plt

>>> from sklearn.tree import DecisionTreeRegressor

設定は，データ数100個，10重CV(Cross Validation)とします．

>>> n = 100; K = 10 

データの生成を行います．データは区間[-2, 2]上の一様分布とします．

>>> x = np.random.uniform(-2, 2, n)

>>> y = np.sin(2 * np.pi * x) / x + np.random.normal(scale = 0.5, size = n)

データのグループ分けと描画を行います．

>>> cv_idx = np.tile(np.arange(K), int(np.ceil(n/K)))[: n]

>>> maxdepths = np.arange(2, 10)

>>> cverr = np.array([])

>>> for mp in maxdepths:

...     cverr_lambda = np.array([])

...     for k in range(K):

...             tr_idx = (cv_idx!=k) 

...             te_idx = (cv_idx==k)

...             cvx = x[tr_idx]; cvy = y[tr_idx]   

...             dtreg = DecisionTreeRegressor(max_depth=mp)

...             dtreg.fit(np.array([cvx]).T, cvy)               

...             ypred = dtreg.predict(np.array([x[te_idx]]).T)  

...             cl = np.append(cverr_lambda, np.mean((y[te_idx]-ypred)**2/2))

...     cverr = np.append(cverr, np.mean(cl))

... 

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

.

. *1 （省略）

.

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

>>> plt.scatter(maxdepths, cverr,c='k')  # cv誤差のプロット

<matplotlib.collections.PathCollection object at 0x122d2cc18>

>>> plt.xlabel("max depth"); plt.ylabel('cv error')

Text(0.5, 0, 'max depth')

Text(0, 0.5, 'cv error')

>>> plt.show()

上記のコードの解説

• 決定木の深さの候補は maxdepths = np.arange(2, 10) で設定
• CVのためのデータ分割は cvx = x[tr_idx]; cvy = y[tr_idx] で行っている
• 決定木で推定するのは dtreg.fit(np.array([cvx]).T, cvy)で行っている．
• 予測は ypred = dtreg.predict(np.array([x[te_idx] - ypred)**2/2) で行っている

上記の例のJupyter Notebookファイルは，GitHubのStatistics.ipynbの"Estimation of test error by cross validation method"で見ることができます．

----------
REPLの実行で省略した *1 部分は，以下の様に表示されます．

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=2, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=4, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=5, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=6, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=7, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=8, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')

DecisionTreeRegressor(criterion='mse', max_depth=9, max_features=None,

           max_leaf_nodes=None, min_impurity_decrease=0.0,

           min_impurity_split=None, min_samples_leaf=1,

           min_samples_split=2, min_weight_fraction_leaf=0.0,

           presort=False, random_state=None, splitter='best')