07-how-random-forests-really-work.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously I've shown how to create a [linear model and neural net from scratch](https://www.kaggle.com/code/jhoward/linear-model-and-neural-net-from-scratch), and used it to create a solid submission to Kaggle's [Titanic](https://www.kaggle.com/competitions/titanic/) competition. However, for *tabular* data (i.e data that looks like spreadsheet or database tables, such as the data for the Titanic competition) it's more common to see good results by using ensembles of decision trees, such as Random Forests and Gradient Boosting Machines.\n",
    "\n",
    "In this notebook, we're going to learn all about Random Forests, by building one from scratch, and using it to submit to the Titanic competition! That might sound like a pretty big stretch, but I think you'll be surprised to discover how straightforward it actually is.\n",
    "\n",
    "We'll start by importing the basic set of libraries we normally need for data science work, and setting numpy to use our display space more efficiently:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:24.640765Z",
     "iopub.status.busy": "2022-05-23T04:37:24.640339Z",
     "iopub.status.idle": "2022-05-23T04:37:25.174055Z",
     "shell.execute_reply": "2022-05-23T04:37:25.172992Z",
     "shell.execute_reply.started": "2022-05-23T04:37:24.640663Z"
    }
   },
   "outputs": [],
   "source": [
    "from fastai.imports import *\n",
    "np.set_printoptions(linewidth=130)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll create `DataFrame`s from the CSV files just like we did in the \"*linear model and neural net from scratch*\" notebook, and do much the same preprocessing (so go back and check that out if you're not already familiar with the dataset):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:26.152841Z",
     "iopub.status.busy": "2022-05-23T04:37:26.152518Z",
     "iopub.status.idle": "2022-05-23T04:37:26.205303Z",
     "shell.execute_reply": "2022-05-23T04:37:26.204112Z",
     "shell.execute_reply.started": "2022-05-23T04:37:26.152806Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "titanic.zip: Skipping, found more recently modified local copy (use --force to force download)\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "iskaggle = os.environ.get('KAGGLE_KERNEL_RUN_TYPE', '')\n",
    "\n",
    "if iskaggle: path = Path('../input/titanic')\n",
    "else:\n",
    "    import zipfile,kaggle\n",
    "    path = Path('titanic')\n",
    "    kaggle.api.competition_download_cli(str(path))\n",
    "    zipfile.ZipFile(f'{path}.zip').extractall(path)\n",
    "\n",
    "df = pd.read_csv(path/'train.csv')\n",
    "tst_df = pd.read_csv(path/'test.csv')\n",
    "modes = df.mode().iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One difference with Random Forests however is that we don't generally have to create *dummy variables* like we did for non-numeric columns in the linear models and neural network. Instead, we can just convert those fields to *categorical variables*, which internally in Pandas makes a list of all the unique values in the column, and replaces each value with a number. The number is just an index for looking up the value in the list of all unique values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:26.941371Z",
     "iopub.status.busy": "2022-05-23T04:37:26.941013Z",
     "iopub.status.idle": "2022-05-23T04:37:26.9692Z",
     "shell.execute_reply": "2022-05-23T04:37:26.968016Z",
     "shell.execute_reply.started": "2022-05-23T04:37:26.941337Z"
    }
   },
   "outputs": [],
   "source": [
    "def proc_data(df):\n",
    "    df['Fare'] = df.Fare.fillna(0)\n",
    "    df.fillna(modes, inplace=True)\n",
    "    df['LogFare'] = np.log1p(df['Fare'])\n",
    "    df['Embarked'] = pd.Categorical(df.Embarked)\n",
    "    df['Sex'] = pd.Categorical(df.Sex)\n",
    "\n",
    "proc_data(df)\n",
    "proc_data(tst_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll make a list of the continuous, categorical, and dependent variables. Note that we no longer consider `Pclass` a categorical variable. That's because it's *ordered* (i.e 1st, 2nd, and 3rd class have an order), and decision trees, as we'll see, only care about order, not about absolute value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:27.728358Z",
     "iopub.status.busy": "2022-05-23T04:37:27.728077Z",
     "iopub.status.idle": "2022-05-23T04:37:27.733013Z",
     "shell.execute_reply": "2022-05-23T04:37:27.732245Z",
     "shell.execute_reply.started": "2022-05-23T04:37:27.728328Z"
    }
   },
   "outputs": [],
   "source": [
    "cats=[\"Sex\",\"Embarked\"]\n",
    "conts=['Age', 'SibSp', 'Parch', 'LogFare',\"Pclass\"]\n",
    "dep=\"Survived\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even although we've made the `cats` columns categorical, they are still shown by Pandas as their original values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:28.64687Z",
     "iopub.status.busy": "2022-05-23T04:37:28.645995Z",
     "iopub.status.idle": "2022-05-23T04:37:28.657163Z",
     "shell.execute_reply": "2022-05-23T04:37:28.656275Z",
     "shell.execute_reply.started": "2022-05-23T04:37:28.646831Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      male\n",
       "1    female\n",
       "2    female\n",
       "3    female\n",
       "4      male\n",
       "Name: Sex, dtype: category\n",
       "Categories (2, object): ['female', 'male']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Sex.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However behind the scenes they're now stored as integers, with indices that are looked up in the `Categories` list shown in the output above. We can view the stored values by looking in the `cat.codes` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:29.606313Z",
     "iopub.status.busy": "2022-05-23T04:37:29.605974Z",
     "iopub.status.idle": "2022-05-23T04:37:29.613586Z",
     "shell.execute_reply": "2022-05-23T04:37:29.612892Z",
     "shell.execute_reply.started": "2022-05-23T04:37:29.606276Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    0\n",
       "2    0\n",
       "3    0\n",
       "4    1\n",
       "dtype: int8"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Sex.cat.codes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we create a Random Forest or Gradient Boosting Machine, we'll first need to learn how to create a *decision tree*, from which both of these models are built.\n",
    "\n",
    "And to create a decision tree, we'll first need to create a *binary split*, since that's what a decision tree is built from.\n",
    "\n",
    "A binary split is where all rows are placed into one of two groups, based on whether they're above or below some threshold of some column. For example, we could split the rows of our dataset into males and females, by using the threshold `0.5` and the column `Sex` (since the values in the column are `0` for `female` and `1` for `male`). We can use a plot to see how that would split up our data -- we'll use the [Seaborn](https://seaborn.pydata.org/) library, which is a layer on top of [matplotlib](https://matplotlib.org/) that makes some useful charts easier to create, and more aesthetically pleasing by default:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:31.229486Z",
     "iopub.status.busy": "2022-05-23T04:37:31.228601Z",
     "iopub.status.idle": "2022-05-23T04:37:32.45072Z",
     "shell.execute_reply": "2022-05-23T04:37:32.449794Z",
     "shell.execute_reply.started": "2022-05-23T04:37:31.229441Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApkAAAFNCAYAAABL6HT2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmj0lEQVR4nO3de7xdZX3n8c+XcLHcREq4lJBCNUrB8RqpttZiUUFHjW1F46VCy0yGKWjbqSI4rbXVVMSOU6rQNOMgeCOmaiU6qUixeEHQhBaBcKkZQAjXACJgHWjCb/5YK7A52UlOkrXOPif5vF+v8zp7PevZa/+Onjx8z3rWWk+qCkmSJKlLO4y6AEmSJG17DJmSJEnqnCFTkiRJnTNkSpIkqXOGTEmSJHXOkClJkqTOGTI15SRZkORPOjjOuUk+0EVNkjTRkqxIcuSo65A2xJCpTiR5cZLvJPlxkvuSXJrkBX18VlWdWFXv7+PYWyPJzUleNuo6JG0bho0pSY5P8m2Aqjq8qi7ZxDEOTlJJduyxVGkof+m01ZLsCXwF+K/AYmBn4FeBh7fgWAFSVY92WuRWSrJjVa0ZdR2SNJk4NmpjPJOpLjwdoKrOr6q1VfXTqvpaVV0FkOR9ST69rvPYv6yTXJJkfpJLgX8D3pNk+eAHJPnDJEva149Ncye5LsmrB/rtmOSeJM9rt/8uyZ3tGdZvJjl8PD9Qe7bg0iT/M8l9wPuSPDXJ15Pc237GZ5Ls1fb/FDAT+HKSh5Kc0ra/sD3De3+S7zu1Jakrg2c6kxyRZHmSB5LcleQjbbdvtt/vb8emFyXZIckfJ/lhkruTfDLJkweO+7Z2371J/mTM57wvyeeTfDrJA8Dx7Wdf1o5zdyT5WJKdB45XSX4vyQ+SPJjk/e14ellb7+LB/tp2GDLVhX8F1iY5L8krkzxlC47x28A8YA/go8Azkswa2P9m4LND3nc+8KaB7aOBe6rqn9vtfwBmAfsC/wx8ZjNq+iXgxva984EAHwR+DvhF4CDgfQBV9dvALcBrqmr3qjojyYHA/wE+AOwNvBP4QpLpm1GDJI3HmcCZVbUn8FSaWSWAl7Tf92rHpsuA49uvlwK/AOwOfAwgyWHA2cBbgAOAJwMHjvmsOcDngb1oxtS1wB8C+wAvAo4Cfm/Me44Bng+8EDgFWNh+xkHAM3niOK5thCFTW62qHgBeDBTwv4DVSZYk2W8zDnNuVa2oqjVV9WPgAtpBpw2bhwJLhrzvs8Brk+zabj8hjFbVOVX1YFU9TBMInz34F/sm3F5VH21r+mlVrayqi6rq4apaDXwE+LWNvP+twNKqWlpVj1bVRcBy4FXj/HxJ+lJ7hvD+JPfTBMBh/h14WpJ9quqhqrp8I8d8C/CRqrqxqh4CTgPmtrNLrwe+XFXfrqpHgPfSjO2DLquqL7Xj2k+r6oqqurwdK28G/pb1x8YPVdUDVbUCuAb4Wvv5P6Y5GfDccf8voinDkKlOVNV1VXV8Vc2g+av054C/2oxD3Dpm+7M8/pftm4EvVdW/DfnclcB1wGvaoPna9r0kmZbk9CT/t53Wubl92z5bUlOSfZMsSnJbe7xPb+JYPw8cO+Y/EC+mOTsgSePxuqraa90X658hXOcEmkuXrk+ybPAyoiF+DvjhwPYPae7R2K/d99jY14679455/9ix8elJvtJemvQA8BesPzbeNfD6p0O2d99IvZqiDJnqXFVdD5xLEzYBfgLsOtBl/2FvG7P9NWCfJM+hCZvDpsrXWTdlPge4tg2e0ITTOcDLaKZ8Dm7bM44fY1hNH2zbntVOSb11zLHG9r8V+NTgfyCqareqOn2cny9J41JVP6iqN9Fc3vMh4PNJdmP9cQngdpo/gteZCayhCX53ADPW7UjyM8DPjv24Mdt/A1wPzGrHxvcw/nFW2zBDprZakkOT/FGSGe32QTShb910zZXAS5LMbKeqT9vUMdu7FT8PfJjmesaLNtJ9EfAKmrvbB8PoHjR3uN9LE3L/YjN+rGH2AB6iuYD+QOBdY/bfRXN90zqfpjnDenR7VvVJSY5c97+TJHUlyVuTTG+fzHF/27wWWA08yhPHpvOBP0xySJLdacbGzw2Mu69J8svtzTh/xqYD4x7AA8BDSQ6lGYslQ6Y68SDNTTLfTfITmnB5DfBHAO21iJ8DrgKuoHnc0Xh8luYs5N9t7BEZVXUHcBnwy+3nrPNJmmmg24BreTz0bqk/A54H/Jjmhp4vjtn/QeCP26nxd1bVrTRnUt9DM9DfShNM/XcnqWvHACuSPERzE9Dcqvp/7XT3fODSdmx6IXAO8CmaO89vAv4f8HaA9prJt9P88X4Hzfh+Nxt/JN07aWaOHqS5Lv9zG+mr7Uiqhp1JlyRJ27v2TOf9NFPhN424HE0xnlGRJEmPSfKaJLu213T+JXA1j984KY2bIVOSJA2aQ3Nz0O00zxmeW057ags4XS5JkqTOeSZTkiRJnTNkStIkkGSvdk3o65Nc164xvXeSi9o1ny8aXLI1yWlJVia5IcnRo6xdkoaZctPl++yzTx188MGjLkPSNuaKK664p6pGtq58kvOAb1XVx9vnE+5K8/ir+6rq9CSnAk+pqne360ufDxxBs0LLPwJPr6q1Gzq+Y6ekPmxs7NxxoovZWgcffDDLly8fdRmStjFJfrjpXr199p7AS4DjAdo1ox9JMgc4su12HnAJ8G6aGzMWVdXDwE1JVtIEzss29BmOnZL6sLGx0+lySRq9X6B5YP8nkvxLko+3j4/Zr11sYN2iA/u2/Q/kietHr2rbJGnSMGRK0ujtSLOa1N9U1XOBnwCnbqT/sGX+1rv2Kcm8JMuTLF+9enU3lUrSOBkyJWn0VgGrquq77fbnaULnXUkOAGi/3z3Q/6CB98+geabhE1TVwqqaXVWzp08f2eWmkrZThkxJGrGquhO4Nckz2qajgGuBJcBxbdtxwAXt6yXA3CS7JDmE5oHZ35vAkiVpk3q98SfJMcCZwDTg41V1+pj9TwY+Dcxsa/nLqvpEnzVJ0iT1duAz7Z3lNwK/Q3MiYHGSE4BbgGMBqmpFksU0QXQNcNLG7iyXpFHoLWQmmQacBbycZmpnWZIlVXXtQLeTgGur6jVJpgM3JPlMe2elJG03qupKYPaQXUdtoP98YH6fNUnS1uhzuvwIYGVV3diGxkU0j90YVMAeSQLsDtxH81e5JEmSprA+Q+Z4HrHxMeAXaS5Yvxr4/ap6tMeaJEmSNAH6DJnjecTG0cCVNCtWPAf4WPtQ4iceyMdwSJIkTSl9hszxPGLjd4AvVmMlcBNw6NgD+RgOSZKkqaXPu8uXAbPax2vcBswF3jymzy00F7V/K8l+wDNo7qrUZjjllFO488472X///TnjjDNGXY4kSVJ/IbOq1iQ5GbiQ5hFG57SP3Tix3b8AeD9wbpKraabX311V9/RV07bqzjvv5Lbbbht1GZKk7dgtf/4fRl2CtsLM917d+TF7fU5mVS0Flo5pWzDw+nbgFX3WIEmSpInnij+SJEnqnCFTkiRJnTNkSpIkqXOGTEmSJHXOkClJkqTOGTIlSZLUOUOmJEmSOmfIlCRJUucMmZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkdc6QKUmSpM4ZMiVJktS5HUddwER5/rs+OeoSerPHPQ8yDbjlnge32Z/zig+/bdQlSJKkzeCZTEmSJHXOkClJkqTOGTIlSZLUOUOmJEmSOmfIlCRJUucMmZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkda7XkJnkmCQ3JFmZ5NQh+9+V5Mr265oka5Ps3WdNkiRJ6l9vITPJNOAs4JXAYcCbkhw22KeqPlxVz6mq5wCnAd+oqvv6qkmSJEkTo88zmUcAK6vqxqp6BFgEzNlI/zcB5/dYjyRJkiZInyHzQODWge1Vbdt6kuwKHAN8YQP75yVZnmT56tWrOy9UkiRJ3eozZGZIW22g72uASzc0VV5VC6tqdlXNnj59emcFSpIkqR99hsxVwEED2zOA2zfQdy5OlUvajiW5OcnV7Y2Qy9u2vZNclOQH7fenDPQ/rb2p8oYkR4+uckkars+QuQyYleSQJDvTBMklYzsleTLwa8AFPdayTXt0591Yu8uePLrzbqMuRdLWeWl7M+TsdvtU4OKqmgVc3G7T3kQ5Fzic5lKjs9ubLSVp0tixrwNX1ZokJwMXAtOAc6pqRZIT2/0L2q6/AXytqn7SVy3bup/MesWoS5DUjznAke3r84BLgHe37Yuq6mHgpiQraW62vGwENUrSUL2FTICqWgosHdO2YMz2ucC5fdYhSVNAAV9LUsDfVtVCYL+qugOgqu5Ism/b90Dg8oH3Dr2xMsk8YB7AzJkz+6xdktbTa8iUJI3br1TV7W2QvCjJ9RvpO64bK9uguhBg9uzZG7rxUpJ64bKSkjQJVNXt7fe7gb+nmf6+K8kBAO33u9vum3NjpSSNhCFTkkYsyW5J9lj3GngFcA3NzZLHtd2O4/EbJJcAc5PskuQQYBbwvYmtWpI2zulySRq9/YC/TwLNuPzZqvpqkmXA4iQnALcAxwK0N1EuBq4F1gAnVdXa0ZQuScMZMiVpxKrqRuDZQ9rvBY7awHvmA/N7Lk2StpjT5ZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkdc6QKUmSpM4ZMiVJktQ5Q6YkSZI6Z8iUJElS5wyZkiRJ6pwhU5IkSZ0zZEqSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjpnyJQkSVLnDJmSJEnqnCFTkiRJnes1ZCY5JskNSVYmOXUDfY5McmWSFUm+0Wc9kiRJmhg79nXgJNOAs4CXA6uAZUmWVNW1A332As4GjqmqW5Ls21c9kiRJmjh9nsk8AlhZVTdW1SPAImDOmD5vBr5YVbcAVNXdPdYjSZKkCdJnyDwQuHVge1XbNujpwFOSXJLkiiRv67EeSZIkTZDepsuBDGmrIZ//fOAo4GeAy5JcXlX/+oQDJfOAeQAzZ87soVRJkiR1qc8zmauAgwa2ZwC3D+nz1ar6SVXdA3wTePbYA1XVwqqaXVWzp0+f3lvBkiRJ6kafIXMZMCvJIUl2BuYCS8b0uQD41SQ7JtkV+CXguh5rkiRJ0gTobbq8qtYkORm4EJgGnFNVK5Kc2O5fUFXXJfkqcBXwKPDxqrqmr5okSZI0Mfq8JpOqWgosHdO2YMz2h4EP91mHJEmSJpYr/kiSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjpnyJQkSVLnDJmSJEnqnCFTkiRJnTNkSpIkqXOGTEmaBJJMS/IvSb7Sbu+d5KIkP2i/P2Wg72lJVia5IcnRo6takjbMkClJk8PvA9cNbJ8KXFxVs4CL222SHAbMBQ4HjgHOTjJtgmuVpE0yZErSiCWZAfxH4OMDzXOA89rX5wGvG2hfVFUPV9VNwErgiAkqVZLGzZApSaP3V8ApwKMDbftV1R0A7fd92/YDgVsH+q1q29aTZF6S5UmWr169uvOiJWljDJmSNEJJXg3cXVVXjPctQ9pqWMeqWlhVs6tq9vTp07e4RknaEjuOugBJ2s79CvDaJK8CngTsmeTTwF1JDqiqO5IcANzd9l8FHDTw/hnA7RNasSSNg2cyJWmEquq0qppRVQfT3NDz9ap6K7AEOK7tdhxwQft6CTA3yS5JDgFmAd+b4LIlaZM8kylJk9PpwOIkJwC3AMcCVNWKJIuBa4E1wElVtXZ0ZUrScIZMSZokquoS4JL29b3AURvoNx+YP2GFSdIWcLpckiRJnTNkSpIkqXOGTEmSJHXOkClJkqTOGTIlSZLUOUOmJEmSOtdryExyTJIbkqxMcuqQ/Ucm+XGSK9uv9/ZZjyRJkiZGb8/JTDINOAt4Oc0yaMuSLKmqa8d0/VZVvbqvOiRJkjTx+jyTeQSwsqpurKpHgEXAnB4/T5IkSZNEnyHzQODWge1VbdtYL0ry/ST/kOTwHuuRJEnSBOlzWckMaasx2/8M/HxVPZTkVcCXgFnrHSiZB8wDmDlzZsdlSpIkqWt9nslcBRw0sD0DuH2wQ1U9UFUPta+XAjsl2WfsgapqYVXNrqrZ06dP77FkSZIkdaHPkLkMmJXkkCQ7A3OBJYMdkuyfJO3rI9p67u2xJkmSJE2A3qbLq2pNkpOBC4FpwDlVtSLJie3+BcDrgf+aZA3wU2BuVY2dUpckSdIU0+c1meumwJeOaVsw8PpjwMf6rEGSJEkTzxV/JEmS1DlDpiRJkjpnyJQkSVLnDJmSJEnq3EZv/EnyIOs/QP0xVbVn5xVJkiRpyttoyKyqPQCS/DlwJ/ApmpV83gLs0Xt1kiRJmpLGO11+dFWdXVUPtqv0/A3wW30WJkmSpKlrvCFzbZK3JJmWZIckbwHW9lmYJEmSpq7xhsw3A28A7mq/jm3bJEmSpPWMa8WfqroZmNNvKZIkSdpWjOtMZpKnJ7k4yTXt9rOS/HG/pUmSJGmqGu90+f8CTgP+HaCqrgLm9lWUJE1FSS4eT5skbQ/GNV0O7FpV30sy2Lamh3okacpJ8iRgV2CfJE+hedQbwJ7Az42sMEkaofGGzHuSPJX2wexJXg/c0VtVkjS1/BfgD2gC5RU8HjIfAM4aUU2SNFLjDZknAQuBQ5PcBtxE80B2SdruVdWZwJlJ3l5VHx11PZI0GYw3ZP6wql6WZDdgh6p6sM+iJGkqqqqPJvll4GAGxteq+uTIipKkERlvyLwpyVeBzwFf77EeSZqyknwKeCpwJY8vWFGAIVPSdme8IfMZwGtops3/d5KvAIuq6tu9VSZJU89s4LCqqlEXIkmjNt6Hsf8UWAwsbu+cPBP4BjCtx9okaaq5BtifbejGyOe/y5OwU9kVH37bqEvQdmy8ZzJJ8mvAG4FXAstolpmUJD1uH+DaJN8DHl7XWFWvHV1JkjQa4wqZSW6iucZoMfCuqvpJn0VJ0hT1vi15U/uczW8Cu9CMy5+vqj9NsjfNtfAHAzcDb6iqH7XvOQ04gebaz3dU1YVbW7wkdWm8ZzKfXVUP9FqJJE1xVfWNLXzrw8CvV9VDSXYCvp3kH4DfBC6uqtOTnAqcCrw7yWE0q64dTvNszn9M8vSqWruhD5CkibbRkJnklKo6A5ifZL0L2avqHb1VJklTTJIHaRetAHYGdgJ+UlV7bux97Y1CD7WbO7VfBcwBjmzbzwMuAd7dti+qqodpnv6xEjgCuKyrn0WSttamzmRe135f3nchkjTVVdUeg9tJXkcT/jYpyTSa1YKeBpxVVd9Nsl9V3dEe+44k+7bdDwQuH3j7qrZNkiaNjYbMqvpy+/KqqvqXzT14kmNo7kSfBny8qk7fQL8X0AyYb6yqz2/u50jSZFRVX2qnucfTdy3wnCR7AX+f5Jkb6Z4hbevNNiWZB8wDmDlz5njKkKTOjPeazI8kOQD4O5opmhWbekP7V/lZwMtp/spelmRJVV07pN+HAC9alzSlJfnNgc0daJ6buVnPzKyq+5NcAhwD3JXkgPYs5gHA3W23VcBBA2+bAdw+5FgLaZYEZvbs2T67U9KE2mE8narqpTTXBa0GFia5Oskfb+JtRwArq+rGqnoEWERzHdFYbwe+wOODpyRNVa8Z+DoaeJDh494TJJnensEkyc8ALwOuB5YAx7XdjgMuaF8vAeYm2SXJIcAs4Hvd/RiStPXG/ZzMqroT+Osk/wScArwX+MBG3nIgcOvA9irglwY7JDkQ+A3g14EXjLcWSZqMqup3tvCtBwDntTM7OwCLq+orSS6jWQTjBOAW4Nj2c1YkWQxcC6wBTvLOckmTzXifk/mLNA9ifz1wL81ZyT/a1NuGtI2drvkr4N1VtTYZ1v2xz/e6IkmTXpIZwEeBX6EZ774N/H5VrdrY+6rqKuC5Q9rvBY7awHvmA/O3tmZJ6st4z2R+AjgfeEVVrXfdzwaM55qh2cCiNmDuA7wqyZqq+tJgJ68rkjRFfAL4LO0ZR+CtbdvLR1aRJI3IJkNmO33zf6vqzM089jJgVnu90G00Dw5+82CHqjpk4HPOBb4yNmBK0hQyvao+MbB9bpI/GFUxkjRKm7zxp73O52eT7Lw5B66qNcDJNHeNX0dzjdGKJCcmOXGLqpWkye2eJG9NMq39eivNJUaStN0Z73T5D4FLkywBHlu3vKo+srE3VdVSYOmYtgUb6Hv8OGuRpMnqd4GPAf+T5prM7wBbejOQJE1p4w2Zt7dfOwB7bKKvJG2v3g8cV1U/AkiyN/CXNOFTkrYr4wqZVfVnfRciSduAZ60LmABVdV+S9e4al6TtwXgfYfRPDFm1oqp+vfOKJGnq2iHJU8acyRz384glaVsy3sHvnQOvnwT8Fs0DgCVJj/sfwHeSfJ7mD/M34LMsJW2nxjtdfsWYpkuTfKOHeiRpyqqqTyZZTrOKWYDfrKprR1yWJI3EeKfL9x7Y3IHmIer791KRJE1hbag0WEra7o13uvwKHr8mcw1wM3BCHwVJkiRp6ttoyEzyAuDWdSvzJDmO5nrMm/EvdUmSJG3Aplb8+VvgEYAkLwE+CJwH/Jh2LXFJkiRprE1Nl0+rqvva128EFlbVF4AvJLmy18okSZI0ZW3qTOa0JOuC6FHA1wf2+ew3SZIkDbWpoHg+8I0k9wA/Bb4FkORpNFPmkiRJ0no2GjKran6Si4EDgK9V1bo7zHcA3t53cZIkSZqaNjnlXVWXD2n7137KkSRJ0rZgU9dkSpIkSZvNkClJkqTOGTIlSZLUOUOmJEmSOmfIlCRJUucMmZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkdc6QKUmSpM71GjKTHJPkhiQrk5w6ZP+cJFcluTLJ8iQv7rMeSZIkTYxNrl2+pZJMA84CXg6sApYlWVJV1w50uxhYUlWV5FnAYuDQvmqSJEnSxOjzTOYRwMqqurGqHgEWAXMGO1TVQ1VV7eZuQCFJkqQpr8+QeSBw68D2qrbtCZL8RpLrgf8D/O6wAyWZ106nL1+9enUvxUqSJKk7fYbMDGlb70xlVf19VR0KvA54/7ADVdXCqppdVbOnT5/ebZWSJEnqXJ8hcxVw0MD2DOD2DXWuqm8CT02yT481SZIkaQL0GTKXAbOSHJJkZ2AusGSwQ5KnJUn7+nnAzsC9PdYkSZKkCdDb3eVVtSbJycCFwDTgnKpakeTEdv8C4LeAtyX5d+CnwBsHbgSSJEnSFNVbyASoqqXA0jFtCwZefwj4UJ81SNJkl+Qg4JPA/sCjwMKqOjPJ3sDngIOBm4E3VNWP2vecBpwArAXeUVUXjqB0SdogV/yRpNFbA/xRVf0i8ELgpCSHAacCF1fVLJrnCp8K0O6bCxwOHAOc3T6bWJImDUOmJI1YVd1RVf/cvn4QuI7mkW9zgPPabufRPIWDtn1RVT1cVTcBK2meTSxJk4YhU5ImkSQHA88FvgvsV1V3QBNEgX3bbuN6DrEkjZIhU5ImiSS7A18A/qCqHthY1yFt69006UIWkkbJkClJk0CSnWgC5meq6ott811JDmj3HwDc3baP6znELmQhaZQMmZI0Yu3zgv83cF1VfWRg1xLguPb1ccAFA+1zk+yS5BBgFvC9iapXksaj10cYSZLG5VeA3wauTnJl2/Ye4HRgcZITgFuAYwHaZw4vBq6luTP9pKpaO+FVS9JGGDIlacSq6tsMv84S4KgNvGc+ML+3oiRpKzldLkmSpM4ZMiVJktQ5Q6YkSZI6Z8iUJElS5wyZkiRJ6pwhU5IkSZ0zZEqSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjrn2uXSduSUU07hzjvvZP/99+eMM84YdTmSpG2YIVPajtx5553cdtttoy5DkrQdcLpckiRJnTNkSpIkqXO9hswkxyS5IcnKJKcO2f+WJFe1X99J8uw+65EkSdLE6C1kJpkGnAW8EjgMeFOSw8Z0uwn4tap6FvB+YGFf9UiSJGni9Hkm8whgZVXdWFWPAIuAOYMdquo7VfWjdvNyYEaP9UiSJGmC9BkyDwRuHdhe1bZtyAnAP/RYjyRJkiZIn48wypC2GtoxeSlNyHzxBvbPA+YBzJw5s6v6JEmS1JM+z2SuAg4a2J4B3D62U5JnAR8H5lTVvcMOVFULq2p2Vc2ePn16L8VKkiSpO32GzGXArCSHJNkZmAssGeyQZCbwReC3q+pfe6xFkiRJE6i36fKqWpPkZOBCYBpwTlWtSHJiu38B8F7gZ4GzkwCsqarZfdUkSZKkidHrspJVtRRYOqZtwcDr/wT8pz5rkLbELX/+H0ZdQi/W3Lc3sCNr7vvhNvszznzv1aMuQZKEK/5IkiSpB4ZMSZIkdc6QKUmSpM4ZMiVJktQ5Q6YkSZI6Z8iUJElS5wyZkiRJ6pwhU5IkSZ0zZEqSJKlzhkxJkiR1zpApSZKkzvW6drmkyWWfJz0KrGm/S5LUH0OmtB1557PuH3UJGiLJOcCrgbur6plt297A54CDgZuBN1TVj9p9pwEnAGuBd1TVhSMoW5I2yulySRq9c4FjxrSdClxcVbOAi9ttkhwGzAUOb99zdpJpE1eqJI2PIVOSRqyqvgncN6Z5DnBe+/o84HUD7Yuq6uGquglYCRwxEXVK0uYwZErS5LRfVd0B0H7ft20/ELh1oN+qtk2SJhVDpiRNLRnSVkM7JvOSLE+yfPXq1T2XJUlPZMiUpMnpriQHALTf727bVwEHDfSbAdw+7ABVtbCqZlfV7OnTp/darCSNZciUpMlpCXBc+/o44IKB9rlJdklyCDAL+N4I6pOkjfIRRpI0YknOB44E9kmyCvhT4HRgcZITgFuAYwGqakWSxcC1wBrgpKpaO5LCJWkjDJmSNGJV9aYN7DpqA/3nA/P7q0iStp7T5ZIkSeqcIVOSJEmdM2RKkiSpc72GzCTHJLkhycokpw7Zf2iSy5I8nOSdfdYiSZKkidPbjT/tWrpnAS+nea7bsiRLquragW73Ae/g8eXSJEmStA3o80zmEcDKqrqxqh4BFtGsufuYqrq7qpYB/95jHZIkSZpgfYZM19eVJEnaTvUZMse9vu4mD+T6u5IkSVNKnyFz3Ovrborr70qSJE0tfYbMZcCsJIck2RmYS7PmriRJkrZxvd1dXlVrkpwMXAhMA85p19w9sd2/IMn+wHJgT+DRJH8AHFZVD/RVlyRJkvrX69rlVbUUWDqmbcHA6ztpptElSZK0DXHFH0mSJHXOkClJkqTOGTIlSZLUOUOmJEmSOmfIlCRJUucMmZIkSeqcIVOSJEmdM2RKkiSpc4ZMSZIkdc6QKUmSpM4ZMiVJktQ5Q6YkSZI6Z8iUJElS5wyZkiRJ6pwhU5IkSZ0zZEqSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjpnyJQkSVLnDJmSJEnqnCFTkiRJnTNkSpIkqXO9hswkxyS5IcnKJKcO2Z8kf93uvyrJ8/qsR5K2FZsaXyVp1HoLmUmmAWcBrwQOA96U5LAx3V4JzGq/5gF/01c9krStGOf4Kkkj1eeZzCOAlVV1Y1U9AiwC5ozpMwf4ZDUuB/ZKckCPNUnStmA846skjVSfIfNA4NaB7VVt2+b2kSQ9kWOnpElvxx6PnSFttQV9SDKPZjod4KEkN2xlbduifYB7Rl1EX/KXx426hG3JNv27wp8OG1bG5ee7LKNnjp3d2ab/PTh2dmqb/l3pY+zsM2SuAg4a2J4B3L4FfaiqhcDCrgvcliRZXlWzR12HJj9/V7YJjp0d8d+Dxsvflc3X53T5MmBWkkOS7AzMBZaM6bMEeFt7l/kLgR9X1R091iRJ24LxjK+SNFK9ncmsqjVJTgYuBKYB51TViiQntvsXAEuBVwErgX8DfqeveiRpW7Gh8XXEZUnSE6Rqvct4NAUlmddOjUkb5e+K9Dj/PWi8/F3ZfIZMSZIkdc5lJSVJktQ5Q+YkkeQdSa5L8pmejv++JO/s49ia2pIcmeQro65D2hKOnRoVx85N6/MRRto8vwe8sqpuGnUhkjSFOHZKk5RnMieBJAuAXwCWJPnvSc5JsizJvySZ0/Y5PsmXknw5yU1JTk7y39o+lyfZu+33n9v3fj/JF5LsOuTznprkq0muSPKtJIdO7E+sriU5OMn1ST6e5Jokn0nysiSXJvlBkiPar++0vzPfSfKMIcfZbdjvnzQZOXZqazl29suQOQlU1Yk0D1J+KbAb8PWqekG7/eEku7Vdnwm8mWbd4vnAv1XVc4HLgLe1fb5YVS+oqmcD1wEnDPnIhcDbq+r5wDuBs/v5yTTBngacCTwLOJTmd+XFNP8fvwe4HnhJ+zvzXuAvhhzjv7Ph3z9pUnHsVEccO3vidPnk8wrgtQPXAD0JmNm+/qeqehB4MMmPgS+37VfT/OMAeGaSDwB7AbvTPEfvMUl2B34Z+LvksSWkdunh59DEu6mqrgZIsgK4uKoqydXAwcCTgfOSzKJZgnCnIcfY0O/fdX0XL20lx05tKcfOnhgyJ58Av1VVT1hjOMkvAQ8PND06sP0oj/9/eS7wuqr6fpLjgSPHHH8H4P6qek6nVWsy2NTvx/tp/mP7G0kOBi4Zcoyhv3/SFODYqS3l2NkTp8snnwuBt6f9UznJczfz/XsAdyTZCXjL2J1V9QBwU5Jj2+MnybO3smZNDU8GbmtfH7+BPlv7+yeNimOn+uLYuYUMmZPP+2lOxV+V5Jp2e3P8CfBd4CKa60iGeQtwQpLvAysAL1DePpwBfDDJpTRLEQ6ztb9/0qg4dqovjp1byBV/JEmS1DnPZEqSJKlzhkxJkiR1zpApSZKkzhkyJUmS1DlDpiRJkjpnyNSU1q5XvCLJVUmubB+8LEnaCMdOTQRX/NGUleRFwKuB51XVw0n2AXYecVmSNKk5dmqieCZTU9kBwD1V9TBAVd1TVbcneX6SbyS5IsmFSQ5I8uQkNyR5BkCS85P855FWL0mj4dipCeHD2DVlJdkd+DawK/CPwOeA7wDfAOZU1eokbwSOrqrfTfJy4M+BM4Hjq+qYEZUuSSPj2KmJ4nS5pqyqeijJ84FfBV5KM1B+AHgmcFG7hOw04I62/0XtusNnAa45LGm75NipieKZTG0zkrweOAl4UlW9aMj+HWj+Uj8EeFVVXTXBJUrSpOPYqb54TaamrCTPSDJroOk5wHXA9PbCdpLslOTwdv8ftvvfBJyTZKeJrFeSJgPHTk0Uz2Rqymqnez4K7AWsAVYC84AZwF8DT6a5JOSvaP4KvwA4oqoeTPIR4MGq+tOJr1ySRsexUxPFkClJkqTOOV0uSZKkzhkyJUmS1DlDpiRJkjpnyJQkSVLnDJmSJEnqnCFTkiRJnTNkSpIkqXOGTEmSJHXu/wNqeVkkNDm5xAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 792x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "fig,axs = plt.subplots(1,2, figsize=(11,5))\n",
    "sns.barplot(data=df, y=dep, x=\"Sex\", ax=axs[0]).set(title=\"Survival rate\")\n",
    "sns.countplot(data=df, x=\"Sex\", ax=axs[1]).set(title=\"Histogram\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that (on the left) if we split the data into males and females, we'd have groups that have very different survival rates: >70% for females, and <20% for males. We can also see (on the right) that the split would be reasonably even, with over 300 passengers (out of around 900) in each group.\n",
    "\n",
    "We could create a very simple \"model\" which simply says that all females survive, and no males do. To do so, we better first split our data into a training and validation set, to see how accurate this approach turns out to be:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:32.688753Z",
     "iopub.status.busy": "2022-05-23T04:37:32.688257Z",
     "iopub.status.idle": "2022-05-23T04:37:32.908453Z",
     "shell.execute_reply": "2022-05-23T04:37:32.907399Z",
     "shell.execute_reply.started": "2022-05-23T04:37:32.688718Z"
    }
   },
   "outputs": [],
   "source": [
    "from numpy import random\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "random.seed(42)\n",
    "trn_df,val_df = train_test_split(df, test_size=0.25)\n",
    "trn_df[cats] = trn_df[cats].apply(lambda x: x.cat.codes)\n",
    "val_df[cats] = val_df[cats].apply(lambda x: x.cat.codes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(In the previous step we also replaced the categorical variables with their integer codes, since some of the models we'll be building in a moment require that.)\n",
    "\n",
    "Now we can create our independent variables (the `x` variables) and dependent (the `y` variable):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:33.154552Z",
     "iopub.status.busy": "2022-05-23T04:37:33.153932Z",
     "iopub.status.idle": "2022-05-23T04:37:33.164771Z",
     "shell.execute_reply": "2022-05-23T04:37:33.163736Z",
     "shell.execute_reply.started": "2022-05-23T04:37:33.154505Z"
    }
   },
   "outputs": [],
   "source": [
    "def xs_y(df):\n",
    "    xs = df[cats+conts].copy()\n",
    "    return xs,df[dep] if dep in df else None\n",
    "\n",
    "trn_xs,trn_y = xs_y(trn_df)\n",
    "val_xs,val_y = xs_y(val_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the predictions for our extremely simple model, where `female` is coded as `0`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:33.932259Z",
     "iopub.status.busy": "2022-05-23T04:37:33.931549Z",
     "iopub.status.idle": "2022-05-23T04:37:33.940681Z",
     "shell.execute_reply": "2022-05-23T04:37:33.93969Z",
     "shell.execute_reply.started": "2022-05-23T04:37:33.932218Z"
    }
   },
   "outputs": [],
   "source": [
    "preds = val_xs.Sex==0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use mean absolute error to measure how good this model is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:34.580131Z",
     "iopub.status.busy": "2022-05-23T04:37:34.579605Z",
     "iopub.status.idle": "2022-05-23T04:37:34.588669Z",
     "shell.execute_reply": "2022-05-23T04:37:34.58794Z",
     "shell.execute_reply.started": "2022-05-23T04:37:34.580088Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.21524663677130046"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import mean_absolute_error\n",
    "mean_absolute_error(val_y, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we could try splitting on a continuous column. We have to use a somewhat different chart to see how this might work -- here's an example of how we could look at `LogFare`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:36.079566Z",
     "iopub.status.busy": "2022-05-23T04:37:36.078932Z",
     "iopub.status.idle": "2022-05-23T04:37:36.428338Z",
     "shell.execute_reply": "2022-05-23T04:37:36.427385Z",
     "shell.execute_reply.started": "2022-05-23T04:37:36.079515Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAo8AAAE9CAYAAABqeoiYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABE/UlEQVR4nO3deXxU9b3/8ddnJhvZgISw74soKojiimuh1bZS609t7Xrb2lpbF7pX216t7b21ve2vt1rttVzR2+W2Krb0J9VqxX0XcAFZhMi+JpBAQkL2z++PmWAIARKSmXNm8n4+HnmQOXPmnHdknHz4rubuiIiIiIh0RiToACIiIiKSOlQ8ioiIiEinqXgUERERkU5T8SgiIiIinabiUUREREQ6TcWjiIiIiHRaRtAB2howYICPHj066BgikmaWLFmy091Lgs6RCPrcFJFEONznZqiKx9GjR7N48eKgY4hImjGzDUFnSBR9bopIIhzuc1Pd1iIiIiLSaSoeRURERKTTVDyKiIiISKepeBQRERGRTlPxKCIiIiKdpuJRJI28/PLLQUcQEZE0p+JRJE3ceeed3HTTTdx5551BRxERkTSm4lEkDdTW1vLQQw8B8NBDD1FbWxtwIhERSVcqHkXSwLXXXnvYxyIiIj1FxaNIiluyZAnr1q074Ni6det44403AkokIiLpTMWjSIr79a9/3eHxO+64I8lJRIL34KJNfOF/FvH/3tyCuwcdRyQtqXgUSXGzZ8/u8PgNN9yQ5CQiwdq+p44fLljOi6U7mX3/mzyzujzoSCJpScWjSIqbOnUqY8aMOeDYmDFjmDp1akCJRILxH4+toqnFeXT2OQwqzGbu8+uO/CIR6TIVjyJp4K677jrsY5F0t6+hmb8v3cYnTxvJuJJ8PnfWGF4o3cmKrVVBRxNJOyoeRdJAbm4ul19+OQCXX345ubm5AScSSa4lGyppaG7h/IklAHzytJFkRIyH39oacDKR9JMRdAAR6RnXXXcdp5xyCmeeeWbQUUSS7qV3d5IRMU4dXQRA39xMThrRj5fX7go4mUj6UcujSBpR4Si91Uvv7mLKiH7kZb/XJnLWuGKWbd5NVV1jgMlE0o+KRxERSWnVdY0s27KHs8YVH3D8zHEDaHFYtK4ioGQi6UnFo4iIpLTlW6tobnFOHtX/gONTR/YjKyPCy++q61qkJyW0eDSzfmb2kJmtMrOVZqY+NRER6VFryvYCMHFQwQHHczKjnDSiH4s3VAYRSyRtJbrl8XbgMXc/FpgCrEzw/UREpJdZs6OaguwMhvTNOei5E4b25Z3t1TS3aLcZkZ6SsOLRzAqBc4G5AO7e4O67E3U/ERHpnVbvqGb8oHzM7KDnjh9ayL7GZtbt3BtAMpH0lMiWx7FAOXCfmb1hZveYWV4C7yciIr1QadleJgzM7/C544cVArFxkSLSMxJZPGYAJwP/5e5TgRrgxvYnmdnVZrbYzBaXl2sfUhER6byKmgZ27m3gmHbjHVuNK8knKyOi4lGkByWyeNwMbHb3V+OPHyJWTB7A3ee4+zR3n1ZSUpLAOCIikm5W76gGYMIhisfMaISJgwq0TaFID0pY8eju24FNZjYxfmgGsCJR9xMRkd6ndab1obqtITbucfnWPbhr0oxIT0j0bOvrgf81s6XAScBPEnw/ERHpRTZX1JKVEWFw4cEzrVtNHFxAZW0jO/c2JDGZSPpK6N7W7v4mMC2R9xARkd5rc+U+hvfrQyRy8EzrVuPjrZKlZXspKchOVjSRtKUdZkREksTMLjKzd8ys1MwOmkAYP+d8M3vTzJab2bPJzphqNlfWMqx/n8OeM64kXjyWa7kekZ6g4lFEJAnMLArcBXwQmAR8wswmtTunH/Ab4CPufjxwRbJzpprNlfsY3j/3sOcM6ZtDXlaUd8tUPIr0BBWPIiLJcRpQ6u5r3b0BuB+4pN05nwT+6u4bAdy9LMkZU0ptQxO7ahoYfoSWRzNj3MB83lXLo0iPUPEoIpIcw4BNbR5vjh9r6xigv5k9Y2ZLzOyzHV1I6+PGbN29D+CIxSPA+JJ8StXyKNIjVDyKiCRHRzM62q8dkwGcAnwYuBD4VzM75qAXaX1cADZVdr54HDcwn2176thb35ToWCJpT8WjiEhybAZGtHk8HNjawTmPuXuNu+8EngOmJClfytm8v3g8/JhHeG/GtcY9inSfikcRkeRYBEwwszFmlgVcCTzc7pz/B5xjZhlmlgucDqxMcs6UsbmylqxohJL8Iy+/M3ZAHgDrd9UkOpZI2kvoOo8iIhLj7k1mdh3wOBAF7nX35WZ2Tfz5u919pZk9BiwFWoB73P3t4FKH2+bKfQztl3PYNR5bjSjKxQw27KpNQjKR9KbiUUQkSdz9UeDRdsfubvf458DPk5krVW3fU8eQvkce7wiQkxllSGGOWh5FeoC6rUVEJCXtqKpjUGHnd4wZWZzLRrU8inSbikcREUk57k5ZVT2DDrOndXuji/NYr+JRpNtUPIqISMrZXdtIQ3MLA7tQPI4qzmPn3not1yPSTSoeRUQk5ZRV1wN0qdt6VHFsSZ8NGvco0i0qHkNu1apVQUeQFPLyyy8HHUEkKXZU1QF0qdu6tXjUuEeR7lHxGGLz58/nmmuuYf78+UFHkRRw5513ctNNN3HnnXcGHUUk4fYXjwVd67YGNO5RpJtUPIZUfX39/iLgzjvvpL6+PuBEEma1tbU89NBDADz00EPU1uqXo6S31m7rgV3ots7PzmBAfpa6rUW6ScVjSP3oRz+iubkZgObmZn784x8HnEjC7Nprrz3sY5F0s6Oqjr59MsnJjHbpdaOK87TWo0g3qXgModLSUl588cUDjr3wwgusXbs2oEQSZkuWLGHdunUHHFu3bh1vvPFGQIlEEq+razy2GqW1HkW6TcVjCN13330dHr/33nuTnERSwa9//esOj99xxx1JTiKSPDu6uMZjq9HFeWzdU0ddY3MCUon0DioeQ+iqq67q8PgXvvCFJCeRVDB79uwOj99www1JTiKSPGVVdQzswmSZVq0zrjdVqPVR5GipeAyhsWPHMn369AOOnX322YwdOzagRBJmU6dOZcyYMQccGzNmDFOnTg0okUhitbQ45XvruzRZplXrjOsN6roWOWoqHkPq5ptvJhqNDQSPRqP867/+a8CJJMzuuuuuwz4WSSdVdY00Njsl+V0vHkfHWx41aUbk6Kl4DKns7GwuvfRSAC699FKys7v+ISm9R25uLlOmTAHgpJNOIjc3N+BEIomzc29smZ7i/Kwuv7ZfbhaFORlqeRTpBhWPIdXU1LR/xvVLL71EU5P2YpVDq6+v5+233wZg2bJlWhdU0trOvQ0ADDiKlkeA0QO0XI9Id6h4DKn58+dTWVkJQEVFhXaZkcPSuqDSm7S2PB5t8TiyKFctjyLdoOIxhHbt2sXcuXOpq4ttv1VXV8fcuXOpqKgIOJmEkdYFld5mV7zl8Wi6rSG2XM+W3ftobG7pyVgivYaKxxB66qmnaGk58EOtpaWFJ598MqBEEmZaF1R6m11764kY9M89uuJxZHEuzS3O1t37ejiZSO+g4jGEZsyYQSRy4F9NJBJhxowZASWSMNO6oNLblO9toCgvi2jEjur1o4piE8rUdS1ydDKCDiAHKyoq4qqrrtrfdZ2Tk8NVV11FUVFR0NEkYAsWLGDhwoUHHS8sLKSqquqAx7fffvsB58ycOZNZs2YlPKNIou3aW3/U4x2hzVqPWihc5KioeAypSy+9lPnz57N161aKior2L9sjvdvChQspXbOGcWNGHnB89IihwNADjnnTezOu3123EUDFo6SFnXvrj3q8I8DAgmyyMyJs1IxrkaOi4jGkMjIyuPHGG7nhhhu48cYbycjQX5XEjBszkl/+6KYuveYbN9+WoDQiyberpoEp/fsd9esjEWNkUS7r1W0tclRUkYTY5MmTmTdvHiUlJUFHEREJjZ3V3eu2hljX9UYVjyJHRRNmQk6Fo4jIe/Y1NFPT0NytbmuAUcW5bKyoxd17KJlI76HiUUREUkbrAuFHs691W6OKc9nX2Ex5tXZjEukqFY8iIpIydtV0b4HwViNbl+vRjGuRLlPxKCIiKaOiJtZSWJTX3W7r2HI963dqxrVIV6l4FBGRlFFZ0wgc/e4yrYb160M0YmxUy6NIl6l4FBGRlFFZG+u27m7xmJURYWi/HO0yI3IUVDyKiEjK2F3bSMSgIKf7K82NKsrTmEeRo6DiUUQkSczsIjN7x8xKzezGDp4/38z2mNmb8a+bg8gZZpW1DfTLzSJylPtatzWyOJcN2mVGpMu0SLiISBKYWRS4C3g/sBlYZGYPu/uKdqc+7+4XJz1githd20i/3Mweudaoolx21zayZ18jffv0zDVFegO1PIZceXl50BFEpGecBpS6+1p3bwDuBy4JOFPKqaxt6PZ4x1atM66104xI16h4DLGlS5dyxRVXsHTp0qCjiEj3DQM2tXm8OX6svTPN7C0z+4eZHZ+caKmjsraR/j3V8ljcutajuq5FukLFY0g1NTXxgx/8AIAf/OAHNDU1BZxIRLqpo0F67ffGex0Y5e5TgF8Df+vwQmZXm9liM1vc23ondsfHPPaE/QuFq+VRpEtUPIbUvHnzqKqqAqCqqop58+YFnEhEumkzMKLN4+HA1rYnuHuVu++Nf/8okGlmA9pfyN3nuPs0d59WUlKSyMyhE+u27pmWx7zsDAbkZ2vSjEgXJbR4NLP1ZrYsPmtwcSLvlU527drFnDlzDjg2Z84cKioqAkokIj1gETDBzMaYWRZwJfBw2xPMbLCZWfz704h9Ru9KetKQqmtspq6xpcdaHiHWda2WR5GuScZs6wvcfWcS7pM2HnjgAdwP7M1ydx588EGuueaagFJJsixYsICFCxd2+FxpaSl4C9+4+bYuXfPddRvAIsyePbvD52fOnMmsWbO6nFU6z92bzOw64HEgCtzr7svN7Jr483cDlwNfMbMmYB9wpbf/MOjFemqB8LZGFefy8ruqz0W6Qkv1hND69es7PL527drkBpFALFy4kNJ3VjJmaPFBz40ZVAhAy749XbrmmMH9AGiuLjvouXVbY784VTwmXrwr+tF2x+5u8/2dwJ3JzpUq3tuasOeW1RlVlMf8N7ZQ19hMTma0x64rks4SXTw68E8zc+C37j7nSC8QuOaaa3jttdc6PC69w5ihxdz25eQs9XfTb/+elPuIdNfueMtjT3dbu8PmylrGDyzoseuKpLNET5iZ7u4nAx8ErjWzc9uf0JtnDR7K2LFjOeussw44Nn36dMaOHRtQIhGR4FXWxlse83qu5XFkfLme9Ts17lGksxJaPLr71vifZcB8Yovktj+n184aPJzWZXpaff/73w8oiYhIOCRkzGPrcj3a41qk0xJWPJpZnpkVtH4PfAB4O1H3SzePPPIIGRmxUQUZGRk88sgjAScSEQnWe93WPdfyWJSXRX52Bhu1XI9IpyWy5XEQ8IKZvQW8Bjzi7o8l8H5pY9euXcydO3f/wuBNTU3MnTtXS/WISK9WWdtIblaU7Iyem9hiZowZkMfanSoeRTorYcVjfP/WKfGv49393xN1r3Tz1FNP0dLScsCxlpYWnnzyyYASiYgEryf3tW5rXEke75bt7fHriqQr7TATQjNmzCASOfCvJhKJMGPGjIASiYgEb3dtY492WbcaPzCfrXvqqKnXNrAinaHiMWQWLFjArbfeSlFREfGNJjAzioqKuPXWW1mwYEHACUVEgpG4lsd8ANap61qkU7RIeMgsXLiQNWtKGTl6DIX9ig54bs2aUkCLOYtI77SntpFh/fr0+HXHD4wVj6VlezlhWN8ev75IulHxGEIjR4/hplt/ctDx2275XgBpRETCobK2ISHd1iOLc4lGjHfLNe5RpDPUbS0iIqHX0uLs2deYkG7r7IwoI4tyKdWkGZFOUfEoIiKhV1XXSIv37NaEbY0ryVfLo0gnqXgUEZHQ2781YQK6rQHGDcxj/c5amppbjnyySC+n4lFEREIvEVsTtjWuJJ+G5hY2Ve5LyPVF0omKRxERCb1EbE3YVuuMay0WLnJkKh5FRCT0Kmtau60T1/IIUKpxjyJHpOJRRERCL9Hd1n37ZFJSkK2WR5FOUPEoIiKht7u2kYhBQU7iliceV5KnGdcinaDiUUREQi+2QHgWkYgl7B7jSvIpLduLuyfsHiLpQMWjiIiE3u7axoRNlmk1fmA+VXVN7NzbkND7iKQ6FY8iIhJ6lbUNCRvv2GrCwAIAVu+oTuh9RFKdikcREQm9ytrGhC0Q3urYIbHicdV2FY8ih6PiUUREQm93fMxjIg3Iz2ZAfjartlUl9D4iqU7Fo4iIhF6s2zqxLY8Axw0pUMujyBGoeBQRkVCra2ymrrEl4S2PAMcOLmD1jmrtcS1yGIlbMEukl1uwYAELFy7s8utKS0vx5kZu+u3fE5DqYGu37sKiVcyePbvLr505cyazZs1KQCqR9yR6gfC2jh1cSH1TC+t31TA+PoFGRA6k4lEkQRYuXMialcsYXZzTpdeNKgTIoGXfnoTkam90/wzAaSxb06XXrd9VB6DiURLuva0JE99t3TppZuW2ahWPIoeg4lEkgUYX53DrrFFBx0iIWxZsCDqC9BK74y2Pyei2Hj8wn8yosWJbFbOmDE34/URSkcY8iohIqFXWxlse8xLf8pidEeWYQQW8vSU5Lf8iqUjFo4iIhFoyxzwCnDisL0s379E2hSKHoOJRRERC7b1u68S3PAKcMKwve/Y1srlyX1LuJ5JqNOYxYO1n5JaWltLizm23fO+gczesX0vEbP+sWM10FUktZnYRcDsQBe5x958e4rxTgVeAj7v7Q0mMGEqVtY3kZkXJzogm5X4nDusLwLItexhRlJuUe4qkEhWPAVu4cCGr3lnNwGEjASgaPByAmvrGg84dMGQEABV76yjbshHQTFeRVGFmUeAu4P3AZmCRmT3s7is6OO9nwOPJTxlOydjXuq2JgwvIiBjLtuzhQycOSdp9RVKFiscQGDhsJJ+49rtdes2f7/pZgtKISIKcBpS6+1oAM7sfuARY0e6864G/AKcmN1547altTFqXNUBOZpSJgwtYtlmTZkQ6ojGPIiLJMQzY1Obx5vix/cxsGHApcHcSc4VeslseAU4a0Y+3Nu2mpUWTZkTaU/EoIpIc1sGx9pXJr4DvunvzYS9kdrWZLTazxeXl5T2VL7R2J7nlEWDqyP5U1zexpmxvUu8rkgpUPIqIJMdmYESbx8OBre3OmQbcb2brgcuB35jZR9tfyN3nuPs0d59WUlKSoLjhEUTL48kj+wHwxsbKpN5XJBWoeBQRSY5FwAQzG2NmWcCVwMNtT3D3Me4+2t1HAw8BX3X3vyU9aYi0tDh79jUmZWvCtsYMyKN/biavq3gUOYgmzIiIJIG7N5nZdcRmUUeBe919uZldE39e4xw7UFXXSIsnZ2vCtsyMqSP78/rG3Um9r0gqUPGYBO3XcmyrtLSUpuaWLs+eLtuykYpoZP+aj+1pDUiR8HH3R4FH2x3rsGh0988lI1PYJXNrwvZOHtmPp1aVUVnTQP+85BavImHW6eLRzPoAI939nQTmSUsLFy5k+cp3KCwZetBz2f0GkQ3U1B28ruPh5BXH1h7btLP6oOeqymPDqFQ8ikiqq9y/u0zyi7fTxxYD8Nr6Ci48fnDS7y8SVp0qHs1sFvALIAsYY2YnAT9y948kMFtaKSwZyplXXJOUe708T71fIolkZn8B7gX+4e4tQedJZ7uTvK91W5OH9yU7I8Kra1U8irTV2QkzPyS2wO1uAHd/ExidiEAiIingv4BPAmvM7KdmdmzQgdJVZU282zrJE2YAsjOinDKqP6+s3ZX0e4uEWWeLxyZ311L7IiKAuy90908BJwPrgSfM7CUz+7yZJb/KSWNBdlsDnD6mmJXbq9hT27WhRSLprLPF49tm9kkgamYTzOzXwEsJzCUiEmpmVgx8Dvgi8AZwO7Fi8okAY6Wd3bWNRCNGYU4w8zvPGFuEO7yyTq2PIq06WzxeDxwP1AN/AvYAX0tQJhGRUDOzvwLPA7nALHf/iLs/4O7XA/nBpksvlbUN9O2TiVlHG/Qk3tSR/cnLivL8mvTfyUeks474TzkziwIPu/tM4PuJjyQiEnr3xJfd2c/Mst293t2nBRUqHQWxNWFbWRkRzhxXzLOry3H3wIpYkTA5YstjfI/VWjPrm4Q8IiKp4N86OPZy0lP0AkFsTdjeeceUsKliH+t31QaaQyQsOjuIpA5YZmZPADWtB939hoSkEhEJITMbDAwD+pjZVKC1GaqQWBe29LDK2kaG9csJNMO5x8T2D39udTljBuQFmkUkDDpbPD4S/xIR6c0uJDZJZjjwyzbHq4HvBREo3e2ubeD4oYWBZhhVnMeYAXk8uaqMfzlrdKBZRMKgU8Wju//uaG8QHzO5GNji7hcf7XVEuutw20QmQmlpKd5Yxy0LNiTtnsm0flcdVlV6yC0yEyHobTfjn4W/M7PL3P0vgQXpRWLd1sGvfvSBSYO498V1VNU1UpgTfB6RIHV2h5kJwG3AJGB//4G7j+3Ey2cDK4l164gEZuHChaxZ/gYj85uTcr8RmUAmtOyrT8r9km1kLkA99RsWJ+V+G/dGgWC33TSzT7v7H4HRZvaN9s+7+y87eJkcpbrGZuoaWwJb47GtDxw/iN8+t5Zn3innI1MO3mpWpDfpbLf1fcAtwH8CFwCf572xPodkZsOBDwP/Dhz0QSuSbCPzm/neyVVBx5Cj8JPXQ/Hvz9YBb1qOJwkqA9yasL2TRvRnQH42jy/fruJRer3OFo993P1JMzN33wD80MyeJ1ZQHs6vgO8ABd3IKCISCu7+2/iftwadpTcIcmvC9qIR4wPHD2L+61uoqW8iLzuYRctFwqCzi4TXmVmE2D6u15nZpcDAw73AzC4Gytx9yRHOu9rMFpvZ4vJyLcIqIuFnZv9hZoVmlmlmT5rZTjP7dNC50s3ugLcmbO+SKUPZ19jMEyt2BB1FJFCdLR6/RmwZihuAU4BPA/9yhNdMBz5iZuuB+4H3mdkf25/k7nPcfZq7TyspKelsbhGRIH3A3auAi4HNwDHAt4ONlH4q4/tJ988LvuUR4NTRRQztm8Pf3twSdBSRQB22eDSzfwK4+yLgenff7O6fd/fL3P2Vw73W3W9y9+HuPhq4EnjK3fUvcxFJB63VzIeAP7t7RZBh0lWYxjwCRCLGJVOH8fyanZRV1wUdRyQwR2p5bNsUeEUig4iIpJAFZrYKmAY8aWYlxDZTkB60Z1+s5THI7Qnbu+KU4TS3OA8u2hR0FJHAHKl49J64ibs/ozUeRSRduPuNwJnANHdvJLbz1iXBpko/lTUN5GZFyc6IBh1lv7El+Zw9fgB/enUjzS098itSJOUcabrYWDN7mNiyPK3f7+fuH0lYMhGRcDuO2HqPbT9Hfx9UmHRUWdsYmi7rtj59xiiu+eMSnlixg4tOGBx0HJGkO1Lx2PZf0r9IZBARkVRhZn8AxgFvAq2rzjsqHnvU7tqGUHVZt5p53EBGFedy19OlXHj8IMyOuOyxSFo5bPHo7s8mK4iISAqZBkxyd/VbJlBsa8LwtTxmRCN89fxxfPcvy3hmdTkXTDzsynUiaaez2xMu4+Dxj3uI7Vn9b+6+q6eDiYiE2NvAYGBb0EHS2e7aRob26xN0jA5dOnU4v36qlJ/9YxXnjB9ARrRzK9+5O397cwsPLdnM2vIaivKy+OTpI/nYtBFkdvIaIkHr7Dv1H8AjwKfiXwuA54HtwP8kJJmISHgNAFaY2eNm9nDrV9Ch0k1YWx4BsjIi/ODDx7FqezW/e3lDp15TXl3PF3+3mK8/8Bbbdtdx5rhiohHj+/Pf5ku/X0xdY/ORLyISAp3dX2m6u09v83iZmb3o7tO1q4KI9EI/DDpAumtpcfbsawzF1oSHcuHxgzl/Ygm/ePwdzhpXzHFDDr3/+uPLt3PTX5ext76Jmy+exOfOGk0kYrg7//vqRn7wt7f55oNvcecnp2oMpYReZ1se883s9NYHZnYakB9/2NTjqUREQiw+Hnw9kBn/fhHweqCh0kxVXSMtHp6tCTtiZvzHZZMpyMng6j8sZuvufQedU13XyLfnvcWX/7CEIX1zeOT6s/nC2WOIRGz/NT59xii+feFEHlm2jb8v1UgICb/OFo9fBO4xs3Xx7QbvAb5oZnnAbYkKJyISRmb2JeAh4LfxQ8OAvwUWKA2FbWvCQxlYmMNvP3MKlTWNXHLXi/x96Vbqm5rZXdvAn17dyAf+8zn+8vpmrrtgPPO/Op0Jgwo6vM6Xzx3LlOF9+eHDy6mua0zyTyHSNZ3qto5vT3iimfUFzN13t3n6wUQEExEJsWuB04BXAdx9jZlpym0Pat2aMMwtj62mjuzPX796Fl/54xKu+9MbBzw3ZUQ/7vzkyZwyqv9hr5ERjfCjS07gkrte5Pcvb+DaC8YnMrJIt3R2tnVf4Bbg3PjjZ4EfufueBGYLnQULFrBw4cIuv660tJT6xmZennd3AlIdrKp8K6W7o8yePbvLr505cyazZs1KQCqRtFLv7g2tY9PiC4Vr2Z4etDtk+1ofyTGDCvjn18/jmXfKWLG1ikjEOGNsESeP7N/pMYxTRvTjvGNKmPvCOj4/fTS5WZ2dliCSXJ19Z95LbGmKj8Uffwa4D/g/iQgVVgsXLmTpilVk9e1iA0OfIqJ9oKY+OV0R0cISmoBVWyq69LqGPWUAKh5FjuxZM/se0MfM3g98ldgqFNJDKmvi3dYhnjDTXjRizDhuEDOOG3TU17j+feO5/O6X+cuSzXzmzNE9F06kB3W2eBzn7pe1eXyrmb2ZgDyhl9V3IIPP+0TQMRJi+7N/DjqCSKq4EbgKWAZ8GXiU2Fhw6SGp1G3dk04Z1Z/jhhTy59c28ekzRmnmtYRSZyfM7DOzs1sfmNl04OBpZSIivYC7txCbIPNVd7/c3f+7M7vNmNlFZvaOmZWa2Y0dPH+JmS01szfNbHHbz93eZndtI9GIUZjTu7puzYxPnjaCFduqWLalV40MkxTS2eLxGuAuM1sfn219J7F/bYuI9BoW80Mz2wmsAt4xs3Izu7kTr40CdwEfBCYBnzCzSe1OexKY4u4nAV+gF7dmVtY20K9PZq9sebtk6jByMiM8uHhT0FFEOtSp4tHd33L3KcBkYLK7TwXel9BkIiLh8zVgOnCquxe7exFwOjDdzL5+hNeeBpS6+1p3bwDuBy5pe4K7723TgplHL56Es7u2kX4pNN6xJxXmZDLjuEH8Y9l2mppbgo4jcpAubaTp7lXuXhV/+I0E5BERCbPPAp9w93WtB9x9LfDp+HOHMwxo25S0OX7sAGZ2qZmtIrYl7Be6nThFVdQ0UJyXHXSMwMyaPIRdNQ28srZrEx9FkqE7u7D3vr4EEentMt19Z/uD7l4OHKmZrKPPzINaFt19vrsfC3wU+HGHFzK7Oj4mcnF5efmRU6egipqG0C8QnkjnTxxIXlaUvy/dGnQUkYN0p3jstd0pItJrNRzlcxBraRzR5vFw4JCVgbs/B4wzswEdPDfH3ae5+7SSkpIj3DY1VdQ2UJTXu2Zat5WTGWXGcYN4fPl2mlv061bC5bDFo5lVm1lVB1/VwNAkZRQRCYsph/lMPPEIr10ETDCzMWaWBVwJPNz2BDMbb/EZImZ2MpAF7ErAzxFq7k5lTe8uHgFmHDeQytpG3tq8O+goIgc47BoI7t7xJpwiIr2Qu0e78domM7sOeByIAve6+3Izuyb+/N3AZcBnzayR2HJoH+/MEkDppqquiaYWT5ndZRLlvGNKiBg8vaqMk0cefntDkWTqXQtoiYgEyN0fJbageNtjd7f5/mfAz5KdK2wqa2IjAHp7y2O/3CxOGdWfp1aV8c0PTAw6jsh+3RnzKCIi0uN2qXjc7/yJA1m+tYodVXVBRxHZT8WjiIiEiloe3/O+YwcCsa5rkbBQ8SgiIqFSEd/XurePeQQ4dnABQ/rm8JSKRwkRFY8iIhIqFfGWx+J8FY9mxgXHDuSF0p3UNzUHHUcEUPEoIiIhU1nTQHZGhD6ZRz25Pa28b+JAahuaeW2ddpuRcFDxKCIioVIRX+MxvuRlr3fW+GKyohGefSc9dxOS1KPiUUREQqWipkHjHdvIzcrg1DH9eX7NQTtjigRCxaOIiIRKRW2Dxju2c86EEt7ZUc32PVqyR4Kn4lFEREKlUi2PBzl3QmwP8+fWqOtagqfiUUREQmWX9rU+yHFDCigpyOa51SoeJXgqHkVEJDQam1uormtS8diOmXHOhAG8ULqT5pZet925hEzK7229YMECFi5cmJR7lZaW0lDfxPZn/5yU+yVbw+4ySvdVMHv27KTdc+bMmcyaNStp9xORcKtsXSBcxeNBzp1Qwl9f38LbW/YwZUS/oONIL5byxePChQt58+2VNOcWJf5mkQLoA431jYm/VxD69GcPsGTtjqTcLlobW7NMxaOItGpdILxIYx4PcvaEAQA8t7pcxaMEKuWLR4Dm3CL2HfuhoGNIF/VZ9WjS77mjqp5n1muXhlS0o6qekUGHkISr0L7WhzQgP5sThhXy/JqdXD9jQtBxpBfTmEcREQmNyppYz46Kx46dM6GE1zdWUl2Xpj1gkhLSouVRpLMGFWZz/uj6oGPIUXipIjvoCJIEFTWx/z/752UGnCSczp1Qwn898y4vvbuLC48fHHQc6aXU8igiIqFREW951DqPHTtlVH/ysqI8r/UeJUAqHkVEJDQqaxsozMkgM6pfTx3Jyohw5rhinlutrQolOPq/U0REQkMLhB/ZORNK2FhRy/qdNUFHkV5KxaOIiIRGpYrHIzr3GG1VKMFS8SgiIqGhlscjG12cy4iiPuq6lsCoeBQRkdCorGnQZJkjiG1VWMLL7+6koakl6DjSC6l4FBGRUHB3KmrV8tgZ504ooaahmdc3VgYdRXohFY8iIhIKNQ3NNDS1qHjshLPGFxONGM+8o3GPknwJKx7NLMfMXjOzt8xsuZndmqh7iYhI6quMb03YX8XjERXmZHL6mCIWrtwRdBTphRLZ8lgPvM/dpwAnAReZ2RkJvJ+IiKSwXfHisVjFY6d8YNIgSsv2srZ8b9BRpJdJWPHoMa3v6Mz4lyfqfiIiktp2Vse2JhyQr60oO2PmpEEAPLFCrY+SXAkd82hmUTN7EygDnnD3Vzs452ozW2xmi8vLNXZDRKS32rk3XjwWqHjsjOH9czl+aCGPL98edBTpZRJaPLp7s7ufBAwHTjOzEzo4Z467T3P3aSUlJYmMIyIiIdZaPKrbuvM+dOIQXt+4my279wUdRXqRpMy2dvfdwDPARcm4n4iIpJ6dexsoyMkgJzMadJSUcfHkIQA8snRrwEmkN0nkbOsSM+sX/74PMBNYlaj7iYhIaivfW0+Jxjt2yajiPKaM6MeCt7YFHUV6kUS2PA4BnjazpcAiYmMe/57A+4mIhJqZXWRm75hZqZnd2MHznzKzpfGvl8xsShA5g7Kzul6TZY7CrMlDWLZlD2t2VAcdRXqJRM62XuruU919sruf4O4/StS9RETCzsyiwF3AB4FJwCfMbFK709YB57n7ZODHwJzkpgzWzr31DCjQeMeu+ujUYWRGjQcWbQo6ivQS2mFGRCQ5TgNK3X2tuzcA9wOXtD3B3V9y99b95l4hNtmw19i5t0Etj0dhQH42M48bxF/f2EJ9U3PQcaQXUPEoIpIcw4C2TUOb48cO5SrgHx09kY5LnDU0tbBnX6OKx6P08VNHUFHTwGNva9keSTwVjyIiyWEdHOtw4wQzu4BY8fjdjp5PxyXOdtVogfDuOHdCCWMH5PHfz6/FXftxSGKpeBQRSY7NwIg2j4cDB62vYmaTgXuAS9x9V5KyBW5ndWxrwgH5GvN4NCIR44vnjOXtLVW8/G6vedtIQFQ8iogkxyJggpmNMbMs4Erg4bYnmNlI4K/AZ9x9dQAZA6PdZbrv/5w8jAH52fzqyTVqfZSEUvEoIpIE7t4EXAc8DqwEHnT35WZ2jZldEz/tZqAY+I2ZvWlmiwOKm3Tl8eJR6zwevZzMKLNnTuC1dRUsXFkWdBxJYxlBBxAR6S3c/VHg0XbH7m7z/ReBLyY7VxiUV2vMY0+48tQR3PfiOn7y6ErOHj+APlnarUd6nloeRUQkcOXV9RTkZKjY6abMaIQfX3IC63bW8LPHtKmbJIaKRxERCdyOqjoGarxjj5g+fgCfnz6a/3lpPX97Y0vQcSQNqdtaREQCV1Zdz6DCnKBjpI3vXnQsK7dV8a15b5ERNS6ePDToSJJG1PIoIiKB21FVp+KxB+VkRpnz2WlMHt6X6/70Bv/6t7f3z2gX6S61PIqISKDcnbLqenVb97DCnEz+fPUZ3PboKv7wygbuX7SR08YUMWFgAVkZERqaWtjX0ExlbUP8q5H6pmYKczIZMyCPs8YN4KITBlOUp7U35UAqHkVEJFB79jXS0NTCQLU89rjsjCg//MjxfPbMUTywaBPPr9nJvMWbaGpxsjMi5GRG6Z+bRf+8TCYMzCcnM8ru2gaWbKjk70u38eO/r+CzZ41i9owJ5GapZJAYvRNERCRQO6pi3alqeUycsSX53PSh47ipk+e7Oyu2VXHP8+v47bNr+efyHdz96VOYOLggoTklNWjMo4iIBKqsug5AYx5DxMw4fmhf/vPjJ/HnL51BTX0Tl9/9Eq+tqwg6moSAikcREQlUa8vjoEK1PIbRmeOKmX/tdAYWZPOF/1nE21v2BB1JAqZua+lVNu6N8pPXC4OOIUdh494oE4IOIQmxoyrW8jiwQC2PYTWsXx/++MXTuew3L/GF/1nEIzecQ4mGGfRaKh6l15g5c2ZS71daWoo37mN0cXr+Qly/qw7L7MP48eOTcr8JJP/vUJJDu8ukhiF9+zD3c6dy6W9e5IY/v8Efv3g60YgFHUsCoOJReo1Zs2Yxa9aspN1v9uzZNJat4dZZo5J2z2S6ZcEGMgeO5/bbbw86iqQ47S6TOo4bUsiPLzmBbz+0lLkvrOXqc8cFHUkCoDGPIiISqO1aIDylXH7KcC48fhC/+OdqSsuqg44jAVDxKCIigdq2u46h/foEHUM6ycz4t4+eSJ/MKLcuWIG7Bx1JkkzFo4iIBKapuYWy6jqG9lXLYyopKcjm6zMn8PyanTyxYkfQcSTJVDyKiEhgdlTX0+IwRC2PKedTZ4xiwsB8/u2RldQ1NgcdR5IoLSbM1FVX0rhlRdAxpIusuhIYFHQMEQnQ1t37ABiilseUkxmNcMus4/n03FeZ+8I6rr0gOSsvSPDU8igiIoFpLR415jE1nT1hAO+fNIjfPF1KRU1D0HEkSdKi5TGnoD8+bFLQMaSLcqrXBx1BRAK2bU9sgXC1PKau71w4kQ+s3MFvn32Xmz50XNBxJAnU8igiIoHZtnsfBTkZFORkBh1FjtKEQQV89KRh/O7l9fv3KZf0puJRREQCs3VPHUP7qss61c2eMYHGZuc3T78bdBRJAhWPIiISmG179jGkn7qsU93oAXlcfvJw/vTqxv3jWCV9qXgUEZHAbNtdxxC1PKaF62eMx3HufLo06CiSYCoeRUQkEPsamtlV08AwtTymheH9c7ny1JE8uGgTmypqg44jCaTiUUREArG5MlZgjCjKDTiJ9JRrLxhPJGLc/uSaoKNIAql4FBGRQGysUPGYbgb3zeEzZ4zir69vprSsOug4kiAqHkVEJBCtXZsjVTymla+eP44+mVF++cTqoKNIgqh4FBGRQGys2EduVpTivKygo0gPKs7P5qpzxvLosu0s27wn6DiSACoeRUQkEBsrahnRPxczCzqK9LAvnjOGfrmZ/Pyf7wQdRRJAxaOIiARic2WtxjumqcKcTL56/jieW13OK2t3BR1Helha7G0dra2gz6pHg44hXRStrQAGBR0jodbvquOWBRuCjpEQ63fVMWFg0ClSi5ldBNwORIF73P2n7Z4/FrgPOBn4vrv/Ivkpk8Pd2VhRy1njBgQdRRLks2eOZu4L6/j54+/w0DVnqoU5jaR88Thz5syk3au0tJSa2losmp7jc7y5gbzcXMaPH5+kOw5K6t9fsh3tz1ZaWsq+2lpyspOz129dfSN9juLvfcLA5P7/l+rMLArcBbwf2AwsMrOH3X1Fm9MqgBuAjyY/YXLtqmmgtqGZkUVaIDxd5WRGuWHGBL4//22eWlXGjOPSu7GgN0n54nHWrFnMmjUrKfdasGABCxcu7PLrSktLqW9sprBkaAJSHayqfCvZmdGjKgJnzpyZtP+e6e5o35uHe5+VlpaCtzBuzKguXfPddRvAIod8T+jvPSlOA0rdfS2Amd0PXALsLx7dvQwoM7MPBxMxebRMT+/wsWkjuOf5dfz7Iys5e8IAsjOiQUeSHpDyxWMyHW0xMHv2bDbtrObMK65JQKqDvTzvbkYMKOD2229Pyv2kZx3ufTZ79my8qZ5f/uimLl3zGzffhmVk6z0RrGHApjaPNwOnB5QlcOt31gAwqljFYzrLjEa4ZdYkPnffIua+sI6vnp+sni1JJBWPSVJVvpWX592dtHsxYGJS7iXJ9+66jXzj5tu6/JrxEyYkKJF0UkcDvvyoLmR2NXA1wMiRI7uTKTBry2vIiBijivOCjiIJdv7EgVx4/CB+/WQpl5w0jGH9NFQh1al4TILDjQsrLS2ltnYfWdnZXbpmQ309ubl9Ou6GHDBRY9HSVEd/r2+99dYhz58yZQoA4ydM0HsieJuBEW0eDwe2Hs2F3H0OMAdg2rRpR1WABu3d8r2MLMolM6pFP3qDf714EjN/+Sw/XrCCuz9zStBxpJtUPCbB4boh249vKy0tpcWdUaPHHnTuhvVriZjtLxg1Tq336ei99K1vfYvFixcfdO6pp57Kz3/+82RFkyNbBEwwszHAFuBK4JPBRgrOu+V7GVuSH3QMSZLh/XO5/n0T+Pnj7/DI0m18ePKQoCNJNySseDSzEcDvgcFACzDH3TXgqp32xcDs2bNZs6a0w3ONWOGocWvS1oknnthh8XjCCScEkEYOxd2bzOw64HFiS/Xc6+7Lzeya+PN3m9lgYDFQCLSY2deASe5eFVTuRGhucdbvrOWCY7XWU29y9blj+eeKHdz016VMHdmPoeq+TlmJbHlsAr7p7q+bWQGwxMyeaLcshbTT2rW4fPlympqa9h/PyMjg+OOPV9ejHGTWrFncd999HR6XcHH3R4FH2x27u83324l1Z6e1zZW1NDS3MG6AWh57k8xohNs/fhIfvuN5vv7Am/zpS2cQjWjtx1SUsMEm7r7N3V+Pf18NrCQ221AOY9asWXz0ox89oHAEaGpq4rLLLlNBIAcpKiriK1/5ygHHvvKVr1BUVBRQIpHDW1sem2k9bqAmy/Q2owfk8cOPHM+r6yq46+mOe9kk/JIyUtnMRgNTgVeTcb9Ud8cdd3R4/Fe/+lVyg0jKuOyyyyguLgaguLiYyy67LOBEIof2bvleAMaq5bFXuvyU4Xz0pKH88onVPLJ0W9Bx5CgkvHg0s3zgL8DXOhq3Y2ZXm9liM1tcXl6e6Dgp4etf/3qHx7/2ta8lN4ikjIyMDG655RYAbrnlFjIyNBdOwmv1jmqK87Lon5eeu3XJ4ZkZP71sMtNG9efrD77Jkg2VQUeSLkpo8WhmmcQKx/919792dI67z3H3ae4+raSkJJFxUsa55557UJdjUVER5557bkCJJBVMnjyZefPmMXny5KCjiBzWym3VTBpaGHQMCVBOZpQ5n53G0L45fOn3iyktqw46knRBwopHi+2APhdY6e6/TNR90tXvfve7wz4W6Yj+ASZh19Tcwjs7qjluiIrH3q4oL4v7Pn8aETM+/ttXWLU9rRYVSGuJbHmcDnwGeJ+ZvRn/+lAC75dWCgoKOO+88wA4//zzKSgoCDiRiEj3rd1ZQ0NTC5NUPAowZkAeD375DDKjEa6c8wpvb9kTdCTphIQNjHL3F+h4Oy7ppFtvvZXHHnuMiy66KOgoIiI9YsXWWOuSWh6l1diSfB748hl88r9f5eO/fZnbr5zKzEmDgo4lh6F9oUJOhaOIpJOV26rIikYYW6JleuQ9o4rz+MtXzmJsST5f+sNi/vu5tbin5M6bvYKKx5DTDHQRSScrtlVxzOB87WktBxncN4cHv3wmHzxhMP/+6Epu/MsyGppago4lHdD/vSG2dOlSrrjiCpYuXRp0FBGRbmtpcZZu3sMJQ/sGHUVCqk9WlDs/cTLXv288DyzexCf++xW27dkXdCxpR8VjSDU1NXHbbbcB8NOf/vSgHWdERFLN2p172bOvkZNH9Q86ioRYJGJ88wMT+fUnprJqWxUfvuMFnlutXrgwUfEYUvPnz6eyMrZwakVFBfPnzw84kYhI97QuBn2KikfphFlThvLw9WdTkp/Nv9z3Gr98YjXNLRoHGQYqHkNo165dzJ07l7q6OgDq6uqYO3cuFRUVAScTETl6SzZU0i83k7EDNFlGOmdcST5/u3Y6l508nDueXMNn732V8ur6oGP1eioeQ+ipp56ipeXAQcItLS08+eSTASUSEem+JRsqOWVkf2J7SIh0Tp+sKL+4Ygr/cflkFq+v5IO3P8c/l28POlavpuIxhGbMmEEkcuBfTSQSYcaMGQElEhHpnsqaBt4tr9F4RzlqH5s2ggXXn82gwhyu/sMSvjXvLarqGoOO1SupeAyhoqIirrrqKnJycgDIycnhqquuOmi/axGRVPFC6U4AzhxXHHASSWXHDCpg/lenc/37xjP/jS188FfP88w7ZUHH6nVUPIbUpZdeur9YLCoq4tJLLw04kYjI0Xt2dTl9+2QyZXi/oKNIisvKiPDND0zkoWvOJDszwufuW8SX/7CYzZW1QUfrNVQ8hlRGRgY33ngjADfeeCMZGQnbSVJEJKHcnedWl3P2hAFEIxrvKD1j6sj+/GP2OXz7wok8t3onM/7vs/z47yvYuVcTahJNxaOIiCTUqu3VlFXXc94xJUFHkTSTnRHl2gvGs/Cb53Hx5KHc9+I6zvnZ0/zssVVU1jQEHS9tqXgMKS0SLiLp4okVOzBDxaMkzLB+ffi/H5vCE984j/dPGsTdz77L2T97ip88upKyqrqg46UdFY8hpUXCRSQduDt/e3MLp40uYlBhTtBxJM2NK8nnjk9M5fGvncvMSYO45/m1nP2zp7npr8vYsKsm6HhpQ8VjCGmRcBFJF8u3VrG2vIZLThoWdBTpRY4ZVMDtV07l6W+dz+XThvOXJZu54BfPcP2f32Dltqqg46U8FY8hpEXCRSRd/O2NLWRGjQ+eMDjoKNILjSrO4yeXnsgL372AL50zlqdW7uCDtz/PF/5nEW9t2h10vJSl4jGEtEi4iKSDmvomHly8ifdPGkT/vKyg40gvNrAwh5s+dBwv3TiDb77/GN7YWMkld73I7PvfYFOFlvjpKhWPIaRFwkUkHTy0ZDNVdU1cdfbYoKOIANA3N5PrZ0zgue9cwLUXjOOxt7cz45fPctujK9mzT7vVdJaKx5DSIuEiksoamlqY+8I6po7sxynaklBCpiAnk29feCxPf+t8Zk0eypzn1/K+XzzD/766geYWDzpe6Kl4DCktEi4iqex3L61nY0UtN8yYEHQUkUMaGl/iZ8F1ZzOuJJ/vz3+bD9/xPC/Ft9OUjql4DLHJkyczb948Jk+eHHQUEZFO21FVxx1PruH8iSVcMHFg0HFEjuiEYX154Mtn8JtPncze+iY+ec+rfOn3iyktqw46WiipeAy5khItqisiqaOpuYUb/vwGTS3OzRdPCjqOSKeZGR86cQgLv3Ee37loIi+V7uT9//kc1/7pdVZs1fI+bakvVEREeoS7c+uCFby6roL/e8UUxpbkBx1JpMtyMqN89fzxXHnqSOa+sJbfvbSBR5Zu46QR/fjYtBFcdMJginr56gEqHkVEpNsamlr44YLl/OnVjXz53LFcdsrwoCOJdEtRXhbfvvBYvnTOWP7y+hYeXLSJ781fxvfmL+O4IYWcPqaIcQPzGV2cS0lBNnlZGeRmRcmIRIhEIGJGNGJEzIgYRCOGmQX9Y/UIFY8iIkliZhcBtwNR4B53/2m75y3+/IeAWuBz7v560oN20ZubdnPz/3ubpZv38JXzx/GdCycGHUmkx/TLzeKqs8fwhemjeXtLFc+tKeeld3fywKJN7Gts7tK1zCAzGmFgQTaDCnMYXJjD0H45TBhYwPhB+YwfmE9hTmaCfpKeo+JRRCQJzCwK3AW8H9gMLDKzh919RZvTPghMiH+dDvxX/M/Q2bOvkWdXlzNv8SaeX7OTAfnZ/NenTuaDJw4JOppIQpgZJw7vy4nD+3LtBeNpaXHKqutZv6uGipoG9tY3sa+hmaYWx91pbnGa3WlpcVocmlucFnfqm1oor65n+546Vm6vYuHKHdQ3vber3JC+OYwfmM+EgQVMGJTPhIGxorJfbni6ylU8iogkx2lAqbuvBTCz+4FLgLbF4yXA793dgVfMrJ+ZDXH3bYkI1PoLrin+S62pJfaLrqnFqWtsprquib31TVTXNVJWVc/WPXVs3FXD0i17WFteA8Cgwmy++f5j+Nz00RSkQIuJSE+JRIzBfXMY3DenW9dpbnE2V9ayZsdeVpdVUxr/88+vbTygZbM4L4vh/fswrH8fhvbtw6DCHPr2yaSwTyZ9418FORlkZUTIjEbIjFr8zwjRSM92l6t4FBFJjmHApjaPN3Nwq2JH5wwDeqx4/Mofl/DkyjKaWlro6lrIZjCkMIfjh/Xl0pOGcca4Yk4e2b/HfzGJ9CbRiDGqOI9RxXnMnDRo//GWFmfL7n2Ulu1lTVk175bVsHXPPlZtr+apVWXUNbYc5qoHihh8/8OTuOrsMT2SOVTF45IlS3aa2Yagc4TMAECrlUpn6f3SsVFBBwA6qrDal2+dOQczuxq4Ov5wr5m9081s0Mn3znrg5R64WTekwns8FTJCauRMhYyQAjm/+FP4YtdyHvJzM1TFo7trUcN2zGyxu08LOoekBr1fQm0zMKLN4+HA1qM4B3efA8zpyXCp8t5JhZypkBFSI2cqZITel1OLhIuIJMciYIKZjTGzLOBK4OF25zwMfNZizgD2JGq8o4jI0QpVy6OISLpy9yYzuw54nNhSPfe6+3Izuyb+/N3Ao8SW6SkltlTP54PKKyJyKCoew69Hu6Yk7en9EmLu/iixArHtsbvbfO/AtcnOFZcq751UyJkKGSE1cqZCRuhlOS32WSUiIiIicmQa8ygiIiIinabiMcTM7CIze8fMSs3sxqDzSHiZ2b1mVmZmbwedRVJLKrx3zGyEmT1tZivNbLmZzQ46U0fMLMfMXjOzt+I5bw0606GYWdTM3jCzvwed5VDMbL2ZLTOzN81scdB5DiW+mP9DZrYq/h49M+hMbZnZxPh/w9avKjP7WreuqW7rcIpvZbaaNluZAZ9ot5WZCABmdi6wl9juJCcEnUdSRyq8d8xsCDDE3V83swJgCfDRsH0exvcmz3P3vWaWCbwAzHb3VwKOdhAz+wYwDSh094uDztMRM1sPTHP3UK+faGa/A55393viKynkuvvugGN1KF5bbAFOd/ejXldbLY/htX8rM3dvAFq3MhM5iLs/B1QEnUNSTyq8d9x9m7u/Hv++GlhJbOedUPGYvfGHmfGv0LXQmNlw4MPAPUFnSXVmVgicC8wFcPeGsBaOcTOAd7tTOIKKxzA71DZlIiK9lpmNBqYCrwYcpUPx7uA3gTLgCXcPY85fAd8BOr+/XTAc+KeZLYnvqhRGY4Fy4L74MIB7zCwv6FCHcSXw5+5eRMVjeHVqmzIRkd7CzPKBvwBfc/eqoPN0xN2b3f0kYrsDnWZmoRoKYGYXA2XuviToLJ0w3d1PBj4IXBsfYhE2GcDJwH+5+1SgBgjlHIV4l/pHgHndvZaKx/Dq1DZlIiK9QXwM4V+A/3X3vwad50jiXZfPABcFm+Qg04GPxMcT3g+8z8z+GGykjrn71vifZcB8YsO5wmYzsLlNC/NDxIrJMPog8Lq77+juhVQ8hldntjITEUl78Ykoc4GV7v7LoPMcipmVmFm/+Pd9gJnAqkBDtePuN7n7cHcfTez3ylPu/umAYx3EzPLik6OIdwN/AAjdigDuvh3YZGYT44dmAKGayNXGJ+iBLmtQ8Rha7t4EtG5lthJ40N2XB5tKwsrM/gy8DEw0s81mdlXQmSQ1pMh7ZzrwGWKtZK3LjXwo6FAdGAI8bWZLiTUAPOHuoV0KJ+QGAS+Y2VvAa8Aj7v5YwJkO5Xrgf+N/7ycBPwk2zsHMLJfY6i090mqvpXpEREREpNPU8igiIiIinabiUUREREQ6TcWjiIiIiHSaikcRERER6TQVjyIiIiLSaSoeJSnM7PtmttzMlsaX2Ti9B675ETPrkZX8zWzvkc8SEUm+nvh8MrPzzWxPm6WOFvZENumdMoIOIOnPzM4ELgZOdvd6MxsAZHXytRnxNS8P4u4Po4XTRUQ663l3v7grLzjcZ7D0Xmp5lGQYAux093oAd9/p7lvNbH28kMTMppnZM/Hvf2hmc8zsn8DvzexVMzu+9WJm9oyZnWJmnzOzO82sb/xakfjzuWa2ycwyzWycmT1mZkvM7HkzOzZ+zhgze9nMFpnZj5P830NEpFvM7CQzeyXemzPfzPrHj58aP/aymf3czA65K4uZnWZmL5nZG/E/J8aPf87M5pnZAuCf8d1e7o1/Xr5hZpck6ceUkFLxKMnwT2CEma02s9+Y2XmdeM0pwCXu/kli+69+DMDMhgBD3X1J64nuvgd4C2i97izgcXdvBOYA17v7KcC3gN/Ez7md2Eb2pwLbu/0Tiogk1++B77r7ZGAZcEv8+H3ANe5+JtDc7jXntOm2/j6xrRPPdfepwM0cuDPKmcC/uPv7gO8T28bwVOAC4OfxLQOll1K3tSScu+81s1OAc4h98DzQibGKD7v7vvj3DwJPEPtw/Bgwr4PzHwA+DjxNbL/W35hZPnAWMC+2NS4A2fE/pwOXxb//A/Czrv5cIiJBMLO+QD93fzZ+6HfEPuf6AQXu/lL8+J+IDRlqdUC3tZmNAH5nZhMABzLbnPuEu1fEv/8A8BEz+1b8cQ4wktjWudILqXiUpHD3ZuAZ4BkzWwb8C9DEe63fOe1eUtPmtVvMbJeZTSZWIH65g1s8DNxmZkXEWi2fAvKA3e5+0qFiHd1PIyISSnbkUw7wY+Bpd7/UzEYT+4xuVdPmewMuc/d3uhdP0oW6rSXhzGxi/F+2rU4CNgDriRV68F4r4KHcD3wH6Ovuy9o/6e57gdeIdUf/3d2b3b0KWGdmV8RzmJlNib/kRWItlACf6vIPJSISkPhQnUozOyd+6DPAs+5eCVSb2Rnx41d2eIH39AW2xL//3GHOexy43uJdOGY29aiCS9pQ8SjJkE+sa2SFmS0FJgE/BG4Fbjez5zl4bE57DxH7IHzwMOc8AHw6/merTwFXmdlbwHKgdaD3bOBaM1tE7ANURCSscs1sc5uvbxDrvfl5/DP1JOBH8XOvAuaY2cvEWgz3HOa6/0Gsx+ZFIHqY835MrEt7aXwCjiYZ9nLmrp47ERGRdGBm+fGeGOJjy4e4++yAY0ma0ZhHERGR9PFhM7uJ2O/3DRy+O1rkqKjlUUREREQ6TWMeRURERKTTVDyKiIiISKepeBQRERGRTlPxKCIiIiKdpuJRRERERDpNxaOIiIiIdNr/B4oHLL+rB6ZBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 792x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_fare = trn_df[trn_df.LogFare>0]\n",
    "fig,axs = plt.subplots(1,2, figsize=(11,5))\n",
    "sns.boxenplot(data=df_fare, x=dep, y=\"LogFare\", ax=axs[0])\n",
    "sns.kdeplot(data=df_fare, x=\"LogFare\", ax=axs[1]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [boxenplot](https://seaborn.pydata.org/generated/seaborn.boxenplot.html) above shows quantiles of `LogFare` for each group of `Survived==0` and `Survived==1`. It shows that the average `LogFare` for passengers that didn't survive is around `2.5`, and for those that did it's around `3.2`. So it seems that people that paid more for their tickets were more likely to get put on a lifeboat.\n",
    "\n",
    "Let's create a simple model based on this observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:38.68033Z",
     "iopub.status.busy": "2022-05-23T04:37:38.680007Z",
     "iopub.status.idle": "2022-05-23T04:37:38.68637Z",
     "shell.execute_reply": "2022-05-23T04:37:38.685259Z",
     "shell.execute_reply.started": "2022-05-23T04:37:38.680295Z"
    }
   },
   "outputs": [],
   "source": [
    "preds = val_xs.LogFare>2.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and test it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:40.107202Z",
     "iopub.status.busy": "2022-05-23T04:37:40.106891Z",
     "iopub.status.idle": "2022-05-23T04:37:40.114544Z",
     "shell.execute_reply": "2022-05-23T04:37:40.113805Z",
     "shell.execute_reply.started": "2022-05-23T04:37:40.107167Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.336322869955157"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_absolute_error(val_y, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is quite a bit less accurate than our model that used `Sex` as the single binary split.\n",
    "\n",
    "Ideally, we'd like some way to try more columns and breakpoints more easily. We could create a function that returns how good our model is, in order to more quickly try out a few different splits. We'll create a `score` function to do this. Instead of returning the mean absolute error, we'll calculate a measure of *impurity* -- that is, how much the binary split creates two groups where the rows in a group are each similar to each other, or dissimilar.\n",
    "\n",
    "We can measure the similarity of rows inside a group by taking the standard deviation of the dependent variable. If it's higher, then it means the rows are more different to each other. We'll then multiply this by the number of rows, since a bigger group has more impact than a smaller group:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:40.630562Z",
     "iopub.status.busy": "2022-05-23T04:37:40.62976Z",
     "iopub.status.idle": "2022-05-23T04:37:40.636571Z",
     "shell.execute_reply": "2022-05-23T04:37:40.635377Z",
     "shell.execute_reply.started": "2022-05-23T04:37:40.630514Z"
    }
   },
   "outputs": [],
   "source": [
    "def _side_score(side, y):\n",
    "    tot = side.sum()\n",
    "    if tot<=1: return 0\n",
    "    return y[side].std()*tot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we've got that written, we can calculate the score for a split by adding up the scores for the \"left hand side\" (lhs) and \"right hand side\" (rhs):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:41.160896Z",
     "iopub.status.busy": "2022-05-23T04:37:41.160555Z",
     "iopub.status.idle": "2022-05-23T04:37:41.166645Z",
     "shell.execute_reply": "2022-05-23T04:37:41.165803Z",
     "shell.execute_reply.started": "2022-05-23T04:37:41.160861Z"
    }
   },
   "outputs": [],
   "source": [
    "    \n",
    "def score(col, y, split):\n",
    "    lhs = col<=split\n",
    "    return (_side_score(lhs,y) + _side_score(~lhs,y))/len(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, here's the impurity score for the split on `Sex`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:42.530883Z",
     "iopub.status.busy": "2022-05-23T04:37:42.529881Z",
     "iopub.status.idle": "2022-05-23T04:37:42.542981Z",
     "shell.execute_reply": "2022-05-23T04:37:42.541924Z",
     "shell.execute_reply.started": "2022-05-23T04:37:42.530826Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.40787530982063946"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score(trn_xs[\"Sex\"], trn_y, 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and for `LogFare`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:43.657577Z",
     "iopub.status.busy": "2022-05-23T04:37:43.656642Z",
     "iopub.status.idle": "2022-05-23T04:37:43.665306Z",
     "shell.execute_reply": "2022-05-23T04:37:43.66471Z",
     "shell.execute_reply.started": "2022-05-23T04:37:43.657534Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.47180873952099694"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score(trn_xs[\"LogFare\"], trn_y, 2.7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we'd expect from our earlier tests, `Sex` appears to be a better split.\n",
    "\n",
    "To make it easier to find the best binary split, we can create a simple interactive tool (note that this only works in Kaggle if you click \"Copy and Edit\" in the top right to open the notebook editor):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:44.159662Z",
     "iopub.status.busy": "2022-05-23T04:37:44.159112Z",
     "iopub.status.idle": "2022-05-23T04:37:44.20316Z",
     "shell.execute_reply": "2022-05-23T04:37:44.202647Z",
     "shell.execute_reply.started": "2022-05-23T04:37:44.159612Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cca27efa8e1643e990264062f595df43",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Dropdown(description='nm', options=('Age', 'SibSp', 'Parch', 'LogFare', 'Pclass'), value…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def iscore(nm, split):\n",
    "    col = trn_xs[nm]\n",
    "    return score(col, trn_y, split)\n",
    "\n",
    "from ipywidgets import interact\n",
    "interact(nm=conts, split=15.5)(iscore);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try selecting different columns and split points using the dropdown and slider above. What splits can you find that increase the purity of the data?\n",
    "\n",
    "We can do the same thing for the categorical variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:45.542149Z",
     "iopub.status.busy": "2022-05-23T04:37:45.541373Z",
     "iopub.status.idle": "2022-05-23T04:37:45.587574Z",
     "shell.execute_reply": "2022-05-23T04:37:45.58671Z",
     "shell.execute_reply.started": "2022-05-23T04:37:45.542101Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d6078c0b725e464998e37b5c00c5de08",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Dropdown(description='nm', options=('Sex', 'Embarked'), value='Sex'), IntSlider(value=2,…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "interact(nm=cats, split=2)(iscore);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That works well enough, but it's rather slow and fiddly. Perhaps we could get the computer to automatically find the best split point for a column for us? For example, to find the best split point for `age` we'd first need to make a list of all the possible split points (i.e all the unique values of that field)...:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:47.225685Z",
     "iopub.status.busy": "2022-05-23T04:37:47.225336Z",
     "iopub.status.idle": "2022-05-23T04:37:47.234728Z",
     "shell.execute_reply": "2022-05-23T04:37:47.233526Z",
     "shell.execute_reply.started": "2022-05-23T04:37:47.225646Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.42,  0.67,  0.75,  0.83,  0.92,  1.  ,  2.  ,  3.  ,  4.  ,  5.  ,  6.  ,  7.  ,  8.  ,  9.  , 10.  , 11.  , 12.  ,\n",
       "       13.  , 14.  , 14.5 , 15.  , 16.  , 17.  , 18.  , 19.  , 20.  , 21.  , 22.  , 23.  , 24.  , 24.5 , 25.  , 26.  , 27.  ,\n",
       "       28.  , 28.5 , 29.  , 30.  , 31.  , 32.  , 32.5 , 33.  , 34.  , 34.5 , 35.  , 36.  , 36.5 , 37.  , 38.  , 39.  , 40.  ,\n",
       "       40.5 , 41.  , 42.  , 43.  , 44.  , 45.  , 45.5 , 46.  , 47.  , 48.  , 49.  , 50.  , 51.  , 52.  , 53.  , 54.  , 55.  ,\n",
       "       55.5 , 56.  , 57.  , 58.  , 59.  , 60.  , 61.  , 62.  , 64.  , 65.  , 70.  , 70.5 , 74.  , 80.  ])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nm = \"Age\"\n",
    "col = trn_xs[nm]\n",
    "unq = col.unique()\n",
    "unq.sort()\n",
    "unq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and find which index of those values is where `score()` is the lowest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:49.070221Z",
     "iopub.status.busy": "2022-05-23T04:37:49.069942Z",
     "iopub.status.idle": "2022-05-23T04:37:49.150766Z",
     "shell.execute_reply": "2022-05-23T04:37:49.14992Z",
     "shell.execute_reply.started": "2022-05-23T04:37:49.070191Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = np.array([score(col, trn_y, o) for o in unq if not np.isnan(o)])\n",
    "unq[scores.argmin()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on this, it looks like, for instance, that for the `Age` column, `6` is the optimal cutoff according to our training set.\n",
    "\n",
    "We can write a little function that implements this idea:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:50.094341Z",
     "iopub.status.busy": "2022-05-23T04:37:50.093867Z",
     "iopub.status.idle": "2022-05-23T04:37:50.175498Z",
     "shell.execute_reply": "2022-05-23T04:37:50.174679Z",
     "shell.execute_reply.started": "2022-05-23T04:37:50.094305Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6.0, 0.478316717508991)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def min_col(df, nm):\n",
    "    col,y = df[nm],df[dep]\n",
    "    unq = col.dropna().unique()\n",
    "    scores = np.array([score(col, y, o) for o in unq if not np.isnan(o)])\n",
    "    idx = scores.argmin()\n",
    "    return unq[idx],scores[idx]\n",
    "\n",
    "min_col(trn_df, \"Age\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try all the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:51.295064Z",
     "iopub.status.busy": "2022-05-23T04:37:51.294528Z",
     "iopub.status.idle": "2022-05-23T04:37:51.577522Z",
     "shell.execute_reply": "2022-05-23T04:37:51.57682Z",
     "shell.execute_reply.started": "2022-05-23T04:37:51.29503Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Sex': (0, 0.40787530982063946),\n",
       " 'Embarked': (0, 0.47883342573147836),\n",
       " 'Age': (6.0, 0.478316717508991),\n",
       " 'SibSp': (4, 0.4783740258817434),\n",
       " 'Parch': (0, 0.4805296527841601),\n",
       " 'LogFare': (2.4390808375825834, 0.4620823937736597),\n",
       " 'Pclass': (2, 0.46048261885806596)}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = cats+conts\n",
    "{o:min_col(trn_df, o) for o in cols}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to this, `Sex<=0` is the best split we can use.\n",
    "\n",
    "We've just re-invented the [OneR](https://link.springer.com/article/10.1023/A:1022631118932) classifier (or at least, a minor variant of it), which was found to be one of the most effective classifiers in real-world datasets, compared to the algorithms in use in 1993. Since it's so simple and surprisingly effective, it makes for a great *baseline* -- that is, a starting point that you can use to compare your more sophisticated models to.\n",
    "\n",
    "We found earlier that our OneR rule had an error of around `0.215`, so we'll keep that in mind as we try out more sophisticated approaches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a decision tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can we improve our OneR classifier, which predicts survival based only on `Sex`?\n",
    "\n",
    "How about we take each of our two groups, `female` and `male`, and create one more binary split for each of them. That is: find the single best split for females, and the single best split for males. To do this, all we have to do is repeat the previous section's steps, once for males, and once for females.\n",
    "\n",
    "First, we'll remove `Sex` from the list of possible splits (since we've already used it, and there's only one possible split for that binary column), and create our two groups:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:52.822448Z",
     "iopub.status.busy": "2022-05-23T04:37:52.821912Z",
     "iopub.status.idle": "2022-05-23T04:37:52.829494Z",
     "shell.execute_reply": "2022-05-23T04:37:52.82831Z",
     "shell.execute_reply.started": "2022-05-23T04:37:52.822405Z"
    }
   },
   "outputs": [],
   "source": [
    "cols.remove(\"Sex\")\n",
    "ismale = trn_df.Sex==1\n",
    "males,females = trn_df[ismale],trn_df[~ismale]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's find the single best binary split for males...:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:54.372844Z",
     "iopub.status.busy": "2022-05-23T04:37:54.372319Z",
     "iopub.status.idle": "2022-05-23T04:37:54.585566Z",
     "shell.execute_reply": "2022-05-23T04:37:54.584693Z",
     "shell.execute_reply.started": "2022-05-23T04:37:54.372801Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Embarked': (0, 0.3875581870410906),\n",
       " 'Age': (6.0, 0.3739828371010595),\n",
       " 'SibSp': (4, 0.3875864227586273),\n",
       " 'Parch': (0, 0.3874704821461959),\n",
       " 'LogFare': (2.803360380906535, 0.3804856231758151),\n",
       " 'Pclass': (1, 0.38155442004360934)}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{o:min_col(males, o) for o in cols}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and for females:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:55.970401Z",
     "iopub.status.busy": "2022-05-23T04:37:55.969784Z",
     "iopub.status.idle": "2022-05-23T04:37:56.171762Z",
     "shell.execute_reply": "2022-05-23T04:37:56.170711Z",
     "shell.execute_reply.started": "2022-05-23T04:37:55.970348Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Embarked': (0, 0.4295252982857327),\n",
       " 'Age': (50.0, 0.4225927658431649),\n",
       " 'SibSp': (4, 0.42319212059713535),\n",
       " 'Parch': (3, 0.4193314500446158),\n",
       " 'LogFare': (4.256321678298823, 0.41350598332911376),\n",
       " 'Pclass': (2, 0.3335388911567601)}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{o:min_col(females, o) for o in cols}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the best next binary split for males is `Age<=6`, and for females is `Pclass<=2`.\n",
    "\n",
    "By adding these rules, we have created a *decision tree*, where our model will first check whether `Sex` is female or male, and depending on the result will then check either the above `Age` or `Pclass` rules, as appropriate. We could then repeat the process, creating new additional rules for each of the four groups we've now created.\n",
    "\n",
    "Rather than writing that code manually, we can use `DecisionTreeClassifier`, from *sklearn*, which does exactly that for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:57.022761Z",
     "iopub.status.busy": "2022-05-23T04:37:57.022405Z",
     "iopub.status.idle": "2022-05-23T04:37:57.201325Z",
     "shell.execute_reply": "2022-05-23T04:37:57.200214Z",
     "shell.execute_reply.started": "2022-05-23T04:37:57.022724Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n",
    "\n",
    "m = DecisionTreeClassifier(max_leaf_nodes=4).fit(trn_xs, trn_y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One handy feature or this class is that it provides a function for drawing a tree representing the rules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:57.253844Z",
     "iopub.status.busy": "2022-05-23T04:37:57.253469Z",
     "iopub.status.idle": "2022-05-23T04:37:57.274531Z",
     "shell.execute_reply": "2022-05-23T04:37:57.27374Z",
     "shell.execute_reply.started": "2022-05-23T04:37:57.253805Z"
    }
   },
   "outputs": [],
   "source": [
    "import graphviz\n",
    "\n",
    "def draw_tree(t, df, size=10, ratio=0.6, precision=2, **kwargs):\n",
    "    s=export_graphviz(t, out_file=None, feature_names=df.columns, filled=True, rounded=True,\n",
    "                      special_characters=True, rotate=False, precision=precision, **kwargs)\n",
    "    return graphviz.Source(re.sub('Tree {', f'Tree {{ size={size}; ratio={ratio}', s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:58.207169Z",
     "iopub.status.busy": "2022-05-23T04:37:58.206306Z",
     "iopub.status.idle": "2022-05-23T04:37:59.286087Z",
     "shell.execute_reply": "2022-05-23T04:37:59.284841Z",
     "shell.execute_reply.started": "2022-05-23T04:37:58.207112Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"593pt\" height=\"358pt\"\n",
       " viewBox=\"0.00 0.00 592.50 358.40\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 354.4)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-354.4 588.5,-354.4 588.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<path fill=\"#f5ceb2\" stroke=\"black\" d=\"M352,-339C352,-339 227,-339 227,-339 221,-339 215,-333 215,-327 215,-327 215,-283 215,-283 215,-277 221,-271 227,-271 227,-271 352,-271 352,-271 358,-271 364,-277 364,-283 364,-283 364,-327 364,-327 364,-333 358,-339 352,-339\"/>\n",
       "<text text-anchor=\"start\" x=\"254.5\" y=\"-323.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Sex ≤ 0.5</text>\n",
       "<text text-anchor=\"start\" x=\"250\" y=\"-308.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.47</text>\n",
       "<text text-anchor=\"start\" x=\"236.5\" y=\"-293.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 668</text>\n",
       "<text text-anchor=\"start\" x=\"223\" y=\"-278.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [415, 253]</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<path fill=\"#7ebfee\" stroke=\"black\" d=\"M268.5,-199C268.5,-199 152.5,-199 152.5,-199 146.5,-199 140.5,-193 140.5,-187 140.5,-187 140.5,-143 140.5,-143 140.5,-137 146.5,-131 152.5,-131 152.5,-131 268.5,-131 268.5,-131 274.5,-131 280.5,-137 280.5,-143 280.5,-143 280.5,-187 280.5,-187 280.5,-193 274.5,-199 268.5,-199\"/>\n",
       "<text text-anchor=\"start\" x=\"167\" y=\"-183.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Pclass ≤ 2.5</text>\n",
       "<text text-anchor=\"start\" x=\"171\" y=\"-168.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.38</text>\n",
       "<text text-anchor=\"start\" x=\"157.5\" y=\"-153.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 229</text>\n",
       "<text text-anchor=\"start\" x=\"148.5\" y=\"-138.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [59, 170]</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M270.58,-270.95C259.76,-252.05 246.09,-228.17 234.52,-207.96\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"237.55,-206.2 229.54,-199.26 231.47,-209.68 237.55,-206.2\"/>\n",
       "<text text-anchor=\"middle\" x=\"222.98\" y=\"-219.69\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>2</title>\n",
       "<path fill=\"#eb9e67\" stroke=\"black\" d=\"M426.5,-199C426.5,-199 310.5,-199 310.5,-199 304.5,-199 298.5,-193 298.5,-187 298.5,-187 298.5,-143 298.5,-143 298.5,-137 304.5,-131 310.5,-131 310.5,-131 426.5,-131 426.5,-131 432.5,-131 438.5,-137 438.5,-143 438.5,-143 438.5,-187 438.5,-187 438.5,-193 432.5,-199 426.5,-199\"/>\n",
       "<text text-anchor=\"start\" x=\"333\" y=\"-183.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Age ≤ 6.5</text>\n",
       "<text text-anchor=\"start\" x=\"329\" y=\"-168.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.31</text>\n",
       "<text text-anchor=\"start\" x=\"315.5\" y=\"-153.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 439</text>\n",
       "<text text-anchor=\"start\" x=\"306.5\" y=\"-138.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [356, 83]</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M308.42,-270.95C319.24,-252.05 332.91,-228.17 344.48,-207.96\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"347.53,-209.68 349.46,-199.26 341.45,-206.2 347.53,-209.68\"/>\n",
       "<text text-anchor=\"middle\" x=\"356.02\" y=\"-219.69\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>3</title>\n",
       "<path fill=\"#40a0e6\" stroke=\"black\" d=\"M119,-62.5C119,-62.5 12,-62.5 12,-62.5 6,-62.5 0,-56.5 0,-50.5 0,-50.5 0,-21.5 0,-21.5 0,-15.5 6,-9.5 12,-9.5 12,-9.5 119,-9.5 119,-9.5 125,-9.5 131,-15.5 131,-21.5 131,-21.5 131,-50.5 131,-50.5 131,-56.5 125,-62.5 119,-62.5\"/>\n",
       "<text text-anchor=\"start\" x=\"26\" y=\"-47.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.06</text>\n",
       "<text text-anchor=\"start\" x=\"12.5\" y=\"-32.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 120</text>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-17.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [4, 116]</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.78,-130.96C151.11,-111.98 124,-88.24 102.46,-69.37\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"104.62,-66.61 94.79,-62.66 100.01,-71.88 104.62,-66.61\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>4</title>\n",
       "<path fill=\"#fffdfb\" stroke=\"black\" d=\"M268,-62.5C268,-62.5 161,-62.5 161,-62.5 155,-62.5 149,-56.5 149,-50.5 149,-50.5 149,-21.5 149,-21.5 149,-15.5 155,-9.5 161,-9.5 161,-9.5 268,-9.5 268,-9.5 274,-9.5 280,-15.5 280,-21.5 280,-21.5 280,-50.5 280,-50.5 280,-56.5 274,-62.5 268,-62.5\"/>\n",
       "<text text-anchor=\"start\" x=\"179.5\" y=\"-47.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.5</text>\n",
       "<text text-anchor=\"start\" x=\"161.5\" y=\"-32.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 109</text>\n",
       "<text text-anchor=\"start\" x=\"157\" y=\"-17.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [55, 54]</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;4 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M211.54,-130.96C212.1,-113.15 212.79,-91.14 213.37,-72.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.88,-72.76 213.69,-62.66 209.88,-72.54 216.88,-72.76\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<path fill=\"#88c4ef\" stroke=\"black\" d=\"M414.5,-62.5C414.5,-62.5 316.5,-62.5 316.5,-62.5 310.5,-62.5 304.5,-56.5 304.5,-50.5 304.5,-50.5 304.5,-21.5 304.5,-21.5 304.5,-15.5 310.5,-9.5 316.5,-9.5 316.5,-9.5 414.5,-9.5 414.5,-9.5 420.5,-9.5 426.5,-15.5 426.5,-21.5 426.5,-21.5 426.5,-50.5 426.5,-50.5 426.5,-56.5 420.5,-62.5 414.5,-62.5\"/>\n",
       "<text text-anchor=\"start\" x=\"326\" y=\"-47.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.41</text>\n",
       "<text text-anchor=\"start\" x=\"317\" y=\"-32.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 21</text>\n",
       "<text text-anchor=\"start\" x=\"312.5\" y=\"-17.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [6, 15]</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>2&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M367.72,-130.96C367.3,-113.15 366.78,-91.14 366.35,-72.9\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"369.84,-72.57 366.11,-62.66 362.84,-72.74 369.84,-72.57\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<path fill=\"#ea995f\" stroke=\"black\" d=\"M572.5,-62.5C572.5,-62.5 456.5,-62.5 456.5,-62.5 450.5,-62.5 444.5,-56.5 444.5,-50.5 444.5,-50.5 444.5,-21.5 444.5,-21.5 444.5,-15.5 450.5,-9.5 456.5,-9.5 456.5,-9.5 572.5,-9.5 572.5,-9.5 578.5,-9.5 584.5,-15.5 584.5,-21.5 584.5,-21.5 584.5,-50.5 584.5,-50.5 584.5,-56.5 578.5,-62.5 572.5,-62.5\"/>\n",
       "<text text-anchor=\"start\" x=\"475\" y=\"-47.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.27</text>\n",
       "<text text-anchor=\"start\" x=\"461.5\" y=\"-32.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 418</text>\n",
       "<text text-anchor=\"start\" x=\"452.5\" y=\"-17.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [350, 68]</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>2&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M406.48,-130.96C428.3,-111.98 455.6,-88.24 477.29,-69.37\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"479.76,-71.86 485.01,-62.66 475.16,-66.58 479.76,-71.86\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.sources.Source at 0x7f50444bd6d0>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "draw_tree(m, trn_xs, size=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that it's found exactly the same splits as we did!\n",
    "\n",
    "In this picture, the more orange nodes have a lower survival rate, and blue have higher survival. Each node shows how many rows (\"*samples*\") match that set of rules, and shows how many perish or survive (\"*values*\"). There's also something called \"*gini*\". That's another measure of impurity, and it's very similar to the `score()` we created earlier. It's defined as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:37:59.76912Z",
     "iopub.status.busy": "2022-05-23T04:37:59.76879Z",
     "iopub.status.idle": "2022-05-23T04:37:59.776055Z",
     "shell.execute_reply": "2022-05-23T04:37:59.774896Z",
     "shell.execute_reply.started": "2022-05-23T04:37:59.769086Z"
    }
   },
   "outputs": [],
   "source": [
    "def gini(cond):\n",
    "    act = df.loc[cond, dep]\n",
    "    return 1 - act.mean()**2 - (1-act).mean()**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What this calculates is the probability that, if you pick two rows from a group, you'll get the same `Survived` result each time. If the group is all the same, the probability is `1.0`, and `0.0` if they're all different:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:38:01.166595Z",
     "iopub.status.busy": "2022-05-23T04:38:01.166139Z",
     "iopub.status.idle": "2022-05-23T04:38:01.177753Z",
     "shell.execute_reply": "2022-05-23T04:38:01.176747Z",
     "shell.execute_reply.started": "2022-05-23T04:38:01.166562Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.3828350034484158, 0.3064437162277842)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gini(df.Sex=='female'), gini(df.Sex=='male')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how this model compares to our OneR version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:38:02.057241Z",
     "iopub.status.busy": "2022-05-23T04:38:02.056429Z",
     "iopub.status.idle": "2022-05-23T04:38:02.068392Z",
     "shell.execute_reply": "2022-05-23T04:38:02.067471Z",
     "shell.execute_reply.started": "2022-05-23T04:38:02.057192Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2242152466367713"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_absolute_error(val_y, m.predict(val_xs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a tiny bit worse. Since this is such a small dataset (we've only got around 200 rows in our validation set) this small difference isn't really meaningful. Perhaps we'll see better results if we create a bigger tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:38:02.926983Z",
     "iopub.status.busy": "2022-05-23T04:38:02.926216Z",
     "iopub.status.idle": "2022-05-23T04:38:02.981361Z",
     "shell.execute_reply": "2022-05-23T04:38:02.98003Z",
     "shell.execute_reply.started": "2022-05-23T04:38:02.926945Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"1175pt\" height=\"708pt\"\n",
       " viewBox=\"0.00 0.00 1174.50 707.60\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 703.6)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-703.6 1170.5,-703.6 1170.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<path fill=\"#f5ceb2\" stroke=\"black\" d=\"M627.5,-698C627.5,-698 502.5,-698 502.5,-698 496.5,-698 490.5,-692 490.5,-686 490.5,-686 490.5,-642 490.5,-642 490.5,-636 496.5,-630 502.5,-630 502.5,-630 627.5,-630 627.5,-630 633.5,-630 639.5,-636 639.5,-642 639.5,-642 639.5,-686 639.5,-686 639.5,-692 633.5,-698 627.5,-698\"/>\n",
       "<text text-anchor=\"start\" x=\"530\" y=\"-682.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Sex ≤ 0.5</text>\n",
       "<text text-anchor=\"start\" x=\"525.5\" y=\"-667.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.47</text>\n",
       "<text text-anchor=\"start\" x=\"512\" y=\"-652.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 668</text>\n",
       "<text text-anchor=\"start\" x=\"498.5\" y=\"-637.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [415, 253]</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<path fill=\"#7ebfee\" stroke=\"black\" d=\"M476,-591C476,-591 360,-591 360,-591 354,-591 348,-585 348,-579 348,-579 348,-535 348,-535 348,-529 354,-523 360,-523 360,-523 476,-523 476,-523 482,-523 488,-529 488,-535 488,-535 488,-579 488,-579 488,-585 482,-591 476,-591\"/>\n",
       "<text text-anchor=\"start\" x=\"374.5\" y=\"-575.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Pclass ≤ 2.5</text>\n",
       "<text text-anchor=\"start\" x=\"378.5\" y=\"-560.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.38</text>\n",
       "<text text-anchor=\"start\" x=\"365\" y=\"-545.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 229</text>\n",
       "<text text-anchor=\"start\" x=\"356\" y=\"-530.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [59, 170]</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M518.49,-629.78C503.87,-619.33 487.59,-607.71 472.47,-596.91\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"474.38,-593.97 464.21,-591.01 470.31,-599.67 474.38,-593.97\"/>\n",
       "<text text-anchor=\"middle\" x=\"468.32\" y=\"-611.97\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<path fill=\"#eb9e67\" stroke=\"black\" d=\"M779,-591C779,-591 663,-591 663,-591 657,-591 651,-585 651,-579 651,-579 651,-535 651,-535 651,-529 657,-523 663,-523 663,-523 779,-523 779,-523 785,-523 791,-529 791,-535 791,-535 791,-579 791,-579 791,-585 785,-591 779,-591\"/>\n",
       "<text text-anchor=\"start\" x=\"667.5\" y=\"-575.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">LogFare ≤ 3.31</text>\n",
       "<text text-anchor=\"start\" x=\"681.5\" y=\"-560.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.31</text>\n",
       "<text text-anchor=\"start\" x=\"668\" y=\"-545.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 439</text>\n",
       "<text text-anchor=\"start\" x=\"659\" y=\"-530.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [356, 83]</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>0&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M614.36,-629.78C630.02,-619.24 647.47,-607.49 663.64,-596.61\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"665.62,-599.49 671.96,-591.01 661.71,-593.69 665.62,-599.49\"/>\n",
       "<text text-anchor=\"middle\" x=\"667.17\" y=\"-611.84\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<path fill=\"#40a0e6\" stroke=\"black\" d=\"M254.5,-484C254.5,-484 147.5,-484 147.5,-484 141.5,-484 135.5,-478 135.5,-472 135.5,-472 135.5,-428 135.5,-428 135.5,-422 141.5,-416 147.5,-416 147.5,-416 254.5,-416 254.5,-416 260.5,-416 266.5,-422 266.5,-428 266.5,-428 266.5,-472 266.5,-472 266.5,-478 260.5,-484 254.5,-484\"/>\n",
       "<text text-anchor=\"start\" x=\"161\" y=\"-468.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Age ≤ 28.5</text>\n",
       "<text text-anchor=\"start\" x=\"161.5\" y=\"-453.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.06</text>\n",
       "<text text-anchor=\"start\" x=\"148\" y=\"-438.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 120</text>\n",
       "<text text-anchor=\"start\" x=\"143.5\" y=\"-423.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [4, 116]</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M349.64,-522.92C326.17,-511.57 299.79,-498.8 275.84,-487.21\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"277.32,-484.04 266.79,-482.84 274.27,-490.34 277.32,-484.04\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<path fill=\"#fffdfb\" stroke=\"black\" d=\"M471.5,-484C471.5,-484 364.5,-484 364.5,-484 358.5,-484 352.5,-478 352.5,-472 352.5,-472 352.5,-428 352.5,-428 352.5,-422 358.5,-416 364.5,-416 364.5,-416 471.5,-416 471.5,-416 477.5,-416 483.5,-422 483.5,-428 483.5,-428 483.5,-472 483.5,-472 483.5,-478 477.5,-484 471.5,-484\"/>\n",
       "<text text-anchor=\"start\" x=\"369\" y=\"-468.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">LogFare ≤ 2.7</text>\n",
       "<text text-anchor=\"start\" x=\"383\" y=\"-453.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.5</text>\n",
       "<text text-anchor=\"start\" x=\"365\" y=\"-438.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 109</text>\n",
       "<text text-anchor=\"start\" x=\"360.5\" y=\"-423.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [55, 54]</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M418,-522.78C418,-513.69 418,-503.7 418,-494.15\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"421.5,-494.01 418,-484.01 414.5,-494.01 421.5,-494.01\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<path fill=\"#45a3e7\" stroke=\"black\" d=\"M110,-368.5C110,-368.5 12,-368.5 12,-368.5 6,-368.5 0,-362.5 0,-356.5 0,-356.5 0,-327.5 0,-327.5 0,-321.5 6,-315.5 12,-315.5 12,-315.5 110,-315.5 110,-315.5 116,-315.5 122,-321.5 122,-327.5 122,-327.5 122,-356.5 122,-356.5 122,-362.5 116,-368.5 110,-368.5\"/>\n",
       "<text text-anchor=\"start\" x=\"21.5\" y=\"-353.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.11</text>\n",
       "<text text-anchor=\"start\" x=\"12.5\" y=\"-338.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 53</text>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-323.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [3, 50]</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M157.09,-415.75C139.86,-402.71 120.17,-387.8 103.12,-374.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"105.07,-371.98 94.98,-368.73 100.84,-377.56 105.07,-371.98\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<path fill=\"#3c9ee5\" stroke=\"black\" d=\"M250,-368.5C250,-368.5 152,-368.5 152,-368.5 146,-368.5 140,-362.5 140,-356.5 140,-356.5 140,-327.5 140,-327.5 140,-321.5 146,-315.5 152,-315.5 152,-315.5 250,-315.5 250,-315.5 256,-315.5 262,-321.5 262,-327.5 262,-327.5 262,-356.5 262,-356.5 262,-362.5 256,-368.5 250,-368.5\"/>\n",
       "<text text-anchor=\"start\" x=\"161.5\" y=\"-353.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.03</text>\n",
       "<text text-anchor=\"start\" x=\"152.5\" y=\"-338.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 67</text>\n",
       "<text text-anchor=\"start\" x=\"148\" y=\"-323.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [1, 66]</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>2&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M201,-415.75C201,-404.01 201,-390.76 201,-378.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"204.5,-378.73 201,-368.73 197.5,-378.73 204.5,-378.73\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<path fill=\"#cbe5f8\" stroke=\"black\" d=\"M399.5,-368.5C399.5,-368.5 292.5,-368.5 292.5,-368.5 286.5,-368.5 280.5,-362.5 280.5,-356.5 280.5,-356.5 280.5,-327.5 280.5,-327.5 280.5,-321.5 286.5,-315.5 292.5,-315.5 292.5,-315.5 399.5,-315.5 399.5,-315.5 405.5,-315.5 411.5,-321.5 411.5,-327.5 411.5,-327.5 411.5,-356.5 411.5,-356.5 411.5,-362.5 405.5,-368.5 399.5,-368.5\"/>\n",
       "<text text-anchor=\"start\" x=\"306.5\" y=\"-353.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.49</text>\n",
       "<text text-anchor=\"start\" x=\"297.5\" y=\"-338.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 59</text>\n",
       "<text text-anchor=\"start\" x=\"288.5\" y=\"-323.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [25, 34]</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M395.42,-415.75C387.04,-403.42 377.53,-389.42 369.11,-377.02\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"371.99,-375.04 363.48,-368.73 366.2,-378.97 371.99,-375.04\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<path fill=\"#f6d5bd\" stroke=\"black\" d=\"M548.5,-368.5C548.5,-368.5 441.5,-368.5 441.5,-368.5 435.5,-368.5 429.5,-362.5 429.5,-356.5 429.5,-356.5 429.5,-327.5 429.5,-327.5 429.5,-321.5 435.5,-315.5 441.5,-315.5 441.5,-315.5 548.5,-315.5 548.5,-315.5 554.5,-315.5 560.5,-321.5 560.5,-327.5 560.5,-327.5 560.5,-356.5 560.5,-356.5 560.5,-362.5 554.5,-368.5 548.5,-368.5\"/>\n",
       "<text text-anchor=\"start\" x=\"455.5\" y=\"-353.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.48</text>\n",
       "<text text-anchor=\"start\" x=\"446.5\" y=\"-338.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 50</text>\n",
       "<text text-anchor=\"start\" x=\"437.5\" y=\"-323.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [30, 20]</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>5&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M442.15,-415.75C451.11,-403.42 461.28,-389.42 470.29,-377.02\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"473.26,-378.88 476.31,-368.73 467.6,-374.76 473.26,-378.88\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<path fill=\"#e99659\" stroke=\"black\" d=\"M779,-484C779,-484 663,-484 663,-484 657,-484 651,-478 651,-472 651,-472 651,-428 651,-428 651,-422 657,-416 663,-416 663,-416 779,-416 779,-416 785,-416 791,-422 791,-428 791,-428 791,-472 791,-472 791,-478 785,-484 779,-484\"/>\n",
       "<text text-anchor=\"start\" x=\"681\" y=\"-468.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Age ≤ 20.5</text>\n",
       "<text text-anchor=\"start\" x=\"681.5\" y=\"-453.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.24</text>\n",
       "<text text-anchor=\"start\" x=\"668\" y=\"-438.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 320</text>\n",
       "<text text-anchor=\"start\" x=\"659\" y=\"-423.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [275, 45]</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>8&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M721,-522.78C721,-513.69 721,-503.7 721,-494.15\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"724.5,-494.01 721,-484.01 717.5,-494.01 724.5,-494.01\"/>\n",
       "</g>\n",
       "<!-- 18 -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>18</title>\n",
       "<path fill=\"#f1bc96\" stroke=\"black\" d=\"M1005.5,-484C1005.5,-484 898.5,-484 898.5,-484 892.5,-484 886.5,-478 886.5,-472 886.5,-472 886.5,-428 886.5,-428 886.5,-422 892.5,-416 898.5,-416 898.5,-416 1005.5,-416 1005.5,-416 1011.5,-416 1017.5,-422 1017.5,-428 1017.5,-428 1017.5,-472 1017.5,-472 1017.5,-478 1011.5,-484 1005.5,-484\"/>\n",
       "<text text-anchor=\"start\" x=\"910\" y=\"-468.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">SibSp ≤ 0.5</text>\n",
       "<text text-anchor=\"start\" x=\"912.5\" y=\"-453.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.43</text>\n",
       "<text text-anchor=\"start\" x=\"899\" y=\"-438.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 119</text>\n",
       "<text text-anchor=\"start\" x=\"894.5\" y=\"-423.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [81, 38]</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;18 -->\n",
       "<g id=\"edge18\" class=\"edge\">\n",
       "<title>8&#45;&gt;18</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M791.25,-524.07C818.44,-511.71 849.6,-497.54 877.21,-485\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"878.77,-488.13 886.42,-480.81 875.87,-481.76 878.77,-488.13\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<path fill=\"#f1b992\" stroke=\"black\" d=\"M697.5,-368.5C697.5,-368.5 590.5,-368.5 590.5,-368.5 584.5,-368.5 578.5,-362.5 578.5,-356.5 578.5,-356.5 578.5,-327.5 578.5,-327.5 578.5,-321.5 584.5,-315.5 590.5,-315.5 590.5,-315.5 697.5,-315.5 697.5,-315.5 703.5,-315.5 709.5,-321.5 709.5,-327.5 709.5,-327.5 709.5,-356.5 709.5,-356.5 709.5,-362.5 703.5,-368.5 697.5,-368.5\"/>\n",
       "<text text-anchor=\"start\" x=\"604.5\" y=\"-353.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.43</text>\n",
       "<text text-anchor=\"start\" x=\"595.5\" y=\"-338.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 55</text>\n",
       "<text text-anchor=\"start\" x=\"586.5\" y=\"-323.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [38, 17]</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>9&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M696.85,-415.75C687.89,-403.42 677.72,-389.42 668.71,-377.02\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"671.4,-374.76 662.69,-368.73 665.74,-378.88 671.4,-374.76\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<path fill=\"#e89050\" stroke=\"black\" d=\"M856,-376C856,-376 740,-376 740,-376 734,-376 728,-370 728,-364 728,-364 728,-320 728,-320 728,-314 734,-308 740,-308 740,-308 856,-308 856,-308 862,-308 868,-314 868,-320 868,-320 868,-364 868,-364 868,-370 862,-376 856,-376\"/>\n",
       "<text text-anchor=\"start\" x=\"758\" y=\"-360.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Age ≤ 32.5</text>\n",
       "<text text-anchor=\"start\" x=\"758.5\" y=\"-345.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.19</text>\n",
       "<text text-anchor=\"start\" x=\"745\" y=\"-330.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 265</text>\n",
       "<text text-anchor=\"start\" x=\"736\" y=\"-315.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [237, 28]</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>9&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M745.15,-415.75C752.39,-405.79 760.42,-394.73 767.99,-384.32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"770.89,-386.28 773.93,-376.13 765.22,-382.16 770.89,-386.28\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<path fill=\"#e99356\" stroke=\"black\" d=\"M781,-269C781,-269 665,-269 665,-269 659,-269 653,-263 653,-257 653,-257 653,-213 653,-213 653,-207 659,-201 665,-201 665,-201 781,-201 781,-201 787,-201 793,-207 793,-213 793,-213 793,-257 793,-257 793,-263 787,-269 781,-269\"/>\n",
       "<text text-anchor=\"start\" x=\"678.5\" y=\"-253.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Age ≤ 24.75</text>\n",
       "<text text-anchor=\"start\" x=\"683.5\" y=\"-238.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.22</text>\n",
       "<text text-anchor=\"start\" x=\"670\" y=\"-223.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 181</text>\n",
       "<text text-anchor=\"start\" x=\"661\" y=\"-208.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [158, 23]</text>\n",
       "</g>\n",
       "<!-- 11&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>11&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M774.27,-307.78C767.36,-298.11 759.73,-287.43 752.52,-277.32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"755.24,-275.11 746.58,-269.01 749.54,-279.18 755.24,-275.11\"/>\n",
       "</g>\n",
       "<!-- 17 -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>17</title>\n",
       "<path fill=\"#e78946\" stroke=\"black\" d=\"M921,-261.5C921,-261.5 823,-261.5 823,-261.5 817,-261.5 811,-255.5 811,-249.5 811,-249.5 811,-220.5 811,-220.5 811,-214.5 817,-208.5 823,-208.5 823,-208.5 921,-208.5 921,-208.5 927,-208.5 933,-214.5 933,-220.5 933,-220.5 933,-249.5 933,-249.5 933,-255.5 927,-261.5 921,-261.5\"/>\n",
       "<text text-anchor=\"start\" x=\"832.5\" y=\"-246.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.11</text>\n",
       "<text text-anchor=\"start\" x=\"823.5\" y=\"-231.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 84</text>\n",
       "<text text-anchor=\"start\" x=\"819\" y=\"-216.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [79, 5]</text>\n",
       "</g>\n",
       "<!-- 11&#45;&gt;17 -->\n",
       "<g id=\"edge17\" class=\"edge\">\n",
       "<title>11&#45;&gt;17</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M821.41,-307.78C829.86,-295.79 839.41,-282.24 847.92,-270.17\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"851,-271.88 853.9,-261.69 845.28,-267.84 851,-271.88\"/>\n",
       "</g>\n",
       "<!-- 13 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>13</title>\n",
       "<path fill=\"#e88d4c\" stroke=\"black\" d=\"M704,-161C704,-161 588,-161 588,-161 582,-161 576,-155 576,-149 576,-149 576,-105 576,-105 576,-99 582,-93 588,-93 588,-93 704,-93 704,-93 710,-93 716,-99 716,-105 716,-105 716,-149 716,-149 716,-155 710,-161 704,-161\"/>\n",
       "<text text-anchor=\"start\" x=\"592.5\" y=\"-145.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">LogFare ≤ 2.18</text>\n",
       "<text text-anchor=\"start\" x=\"606.5\" y=\"-130.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.16</text>\n",
       "<text text-anchor=\"start\" x=\"593\" y=\"-115.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 114</text>\n",
       "<text text-anchor=\"start\" x=\"584\" y=\"-100.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [104, 10]</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;13 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>12&#45;&gt;13</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M698.85,-200.75C691.61,-190.79 683.58,-179.73 676.01,-169.32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"678.78,-167.16 670.07,-161.13 673.11,-171.28 678.78,-167.16\"/>\n",
       "</g>\n",
       "<!-- 16 -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>16</title>\n",
       "<path fill=\"#eb9f69\" stroke=\"black\" d=\"M853.5,-153.5C853.5,-153.5 746.5,-153.5 746.5,-153.5 740.5,-153.5 734.5,-147.5 734.5,-141.5 734.5,-141.5 734.5,-112.5 734.5,-112.5 734.5,-106.5 740.5,-100.5 746.5,-100.5 746.5,-100.5 853.5,-100.5 853.5,-100.5 859.5,-100.5 865.5,-106.5 865.5,-112.5 865.5,-112.5 865.5,-141.5 865.5,-141.5 865.5,-147.5 859.5,-153.5 853.5,-153.5\"/>\n",
       "<text text-anchor=\"start\" x=\"760.5\" y=\"-138.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.31</text>\n",
       "<text text-anchor=\"start\" x=\"751.5\" y=\"-123.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 67</text>\n",
       "<text text-anchor=\"start\" x=\"742.5\" y=\"-108.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [54, 13]</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;16 -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>12&#45;&gt;16</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M747.15,-200.75C756.11,-188.42 766.28,-174.42 775.29,-162.02\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"778.26,-163.88 781.31,-153.73 772.6,-159.76 778.26,-163.88\"/>\n",
       "</g>\n",
       "<!-- 14 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>14</title>\n",
       "<path fill=\"#e99254\" stroke=\"black\" d=\"M625,-53.5C625,-53.5 527,-53.5 527,-53.5 521,-53.5 515,-47.5 515,-41.5 515,-41.5 515,-12.5 515,-12.5 515,-6.5 521,-0.5 527,-0.5 527,-0.5 625,-0.5 625,-0.5 631,-0.5 637,-6.5 637,-12.5 637,-12.5 637,-41.5 637,-41.5 637,-47.5 631,-53.5 625,-53.5\"/>\n",
       "<text text-anchor=\"start\" x=\"536.5\" y=\"-38.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.21</text>\n",
       "<text text-anchor=\"start\" x=\"527.5\" y=\"-23.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 50</text>\n",
       "<text text-anchor=\"start\" x=\"523\" y=\"-8.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [44, 6]</text>\n",
       "</g>\n",
       "<!-- 13&#45;&gt;14 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>13&#45;&gt;14</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M622.3,-92.82C615.17,-82.84 607.35,-71.89 600.19,-61.86\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"603.02,-59.81 594.36,-53.71 597.33,-63.88 603.02,-59.81\"/>\n",
       "</g>\n",
       "<!-- 15 -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>15</title>\n",
       "<path fill=\"#e78946\" stroke=\"black\" d=\"M765,-53.5C765,-53.5 667,-53.5 667,-53.5 661,-53.5 655,-47.5 655,-41.5 655,-41.5 655,-12.5 655,-12.5 655,-6.5 661,-0.5 667,-0.5 667,-0.5 765,-0.5 765,-0.5 771,-0.5 777,-6.5 777,-12.5 777,-12.5 777,-41.5 777,-41.5 777,-47.5 771,-53.5 765,-53.5\"/>\n",
       "<text text-anchor=\"start\" x=\"676.5\" y=\"-38.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.12</text>\n",
       "<text text-anchor=\"start\" x=\"667.5\" y=\"-23.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 64</text>\n",
       "<text text-anchor=\"start\" x=\"663\" y=\"-8.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [60, 4]</text>\n",
       "</g>\n",
       "<!-- 13&#45;&gt;15 -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>13&#45;&gt;15</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M669.7,-92.82C676.83,-82.84 684.65,-71.89 691.81,-61.86\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"694.67,-63.88 697.64,-53.71 688.98,-59.81 694.67,-63.88\"/>\n",
       "</g>\n",
       "<!-- 19 -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>19</title>\n",
       "<path fill=\"#f6d5bd\" stroke=\"black\" d=\"M1005.5,-368.5C1005.5,-368.5 898.5,-368.5 898.5,-368.5 892.5,-368.5 886.5,-362.5 886.5,-356.5 886.5,-356.5 886.5,-327.5 886.5,-327.5 886.5,-321.5 892.5,-315.5 898.5,-315.5 898.5,-315.5 1005.5,-315.5 1005.5,-315.5 1011.5,-315.5 1017.5,-321.5 1017.5,-327.5 1017.5,-327.5 1017.5,-356.5 1017.5,-356.5 1017.5,-362.5 1011.5,-368.5 1005.5,-368.5\"/>\n",
       "<text text-anchor=\"start\" x=\"912.5\" y=\"-353.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.48</text>\n",
       "<text text-anchor=\"start\" x=\"903.5\" y=\"-338.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 60</text>\n",
       "<text text-anchor=\"start\" x=\"894.5\" y=\"-323.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [36, 24]</text>\n",
       "</g>\n",
       "<!-- 18&#45;&gt;19 -->\n",
       "<g id=\"edge19\" class=\"edge\">\n",
       "<title>18&#45;&gt;19</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M952,-415.75C952,-404.01 952,-390.76 952,-378.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"955.5,-378.73 952,-368.73 948.5,-378.73 955.5,-378.73\"/>\n",
       "</g>\n",
       "<!-- 20 -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>20</title>\n",
       "<path fill=\"#eda877\" stroke=\"black\" d=\"M1154.5,-368.5C1154.5,-368.5 1047.5,-368.5 1047.5,-368.5 1041.5,-368.5 1035.5,-362.5 1035.5,-356.5 1035.5,-356.5 1035.5,-327.5 1035.5,-327.5 1035.5,-321.5 1041.5,-315.5 1047.5,-315.5 1047.5,-315.5 1154.5,-315.5 1154.5,-315.5 1160.5,-315.5 1166.5,-321.5 1166.5,-327.5 1166.5,-327.5 1166.5,-356.5 1166.5,-356.5 1166.5,-362.5 1160.5,-368.5 1154.5,-368.5\"/>\n",
       "<text text-anchor=\"start\" x=\"1061.5\" y=\"-353.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">gini = 0.36</text>\n",
       "<text text-anchor=\"start\" x=\"1052.5\" y=\"-338.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 59</text>\n",
       "<text text-anchor=\"start\" x=\"1043.5\" y=\"-323.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = [45, 14]</text>\n",
       "</g>\n",
       "<!-- 18&#45;&gt;20 -->\n",
       "<g id=\"edge20\" class=\"edge\">\n",
       "<title>18&#45;&gt;20</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M998.74,-415.75C1017.24,-402.59 1038.41,-387.53 1056.67,-374.54\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1058.71,-377.38 1064.83,-368.73 1054.65,-371.68 1058.71,-377.38\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.sources.Source at 0x7f503417fdf0>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = DecisionTreeClassifier(min_samples_leaf=50)\n",
    "m.fit(trn_xs, trn_y)\n",
    "draw_tree(m, trn_xs, size=25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:38:03.953946Z",
     "iopub.status.busy": "2022-05-23T04:38:03.953608Z",
     "iopub.status.idle": "2022-05-23T04:38:03.965301Z",
     "shell.execute_reply": "2022-05-23T04:38:03.964474Z",
     "shell.execute_reply.started": "2022-05-23T04:38:03.953912Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18385650224215247"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_absolute_error(val_y, m.predict(val_xs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like this is an improvement, although again it's a bit hard to tell with small datasets like this. Let's try submitting it to Kaggle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:38:04.158937Z",
     "iopub.status.busy": "2022-05-23T04:38:04.158557Z",
     "iopub.status.idle": "2022-05-23T04:38:04.183949Z",
     "shell.execute_reply": "2022-05-23T04:38:04.182841Z",
     "shell.execute_reply.started": "2022-05-23T04:38:04.158902Z"
    }
   },
   "outputs": [],
   "source": [
    "tst_df[cats] = tst_df[cats].apply(lambda x: x.cat.codes)\n",
    "tst_xs,_ = xs_y(tst_df)\n",
    "\n",
    "def subm(preds, suff):\n",
    "    tst_df['Survived'] = preds\n",
    "    sub_df = tst_df[['PassengerId','Survived']]\n",
    "    sub_df.to_csv(f'sub-{suff}.csv', index=False)\n",
    "\n",
    "subm(m.predict(tst_xs), 'tree')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When I submitted this, I got a score of 0.765, which isn't as good as our linear models or most of our neural nets, but it's pretty close to those results.\n",
    "\n",
    "Hopefully you can now see why we didn't really need to create dummy variables, but instead just converted the labels into numbers using some (potentially arbitary) ordering of categories. For instance, here's how the first few items of `Embarked` are labeled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:40:33.30121Z",
     "iopub.status.busy": "2022-05-23T04:40:33.300898Z",
     "iopub.status.idle": "2022-05-23T04:40:33.310952Z",
     "shell.execute_reply": "2022-05-23T04:40:33.309927Z",
     "shell.execute_reply.started": "2022-05-23T04:40:33.301175Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    S\n",
       "1    C\n",
       "2    S\n",
       "3    S\n",
       "4    S\n",
       "Name: Embarked, dtype: category\n",
       "Categories (3, object): ['C', 'Q', 'S']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Embarked.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...resulting in these integer codes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:41:09.343513Z",
     "iopub.status.busy": "2022-05-23T04:41:09.34246Z",
     "iopub.status.idle": "2022-05-23T04:41:09.351674Z",
     "shell.execute_reply": "2022-05-23T04:41:09.350699Z",
     "shell.execute_reply.started": "2022-05-23T04:41:09.343456Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2\n",
       "1    0\n",
       "2    2\n",
       "3    2\n",
       "4    2\n",
       "dtype: int8"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Embarked.cat.codes.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's say we wanted to split into \"C\" in one group, vs \"Q\" or \"S\" in the other group. Then we just have to split on codes `<=0` (since `C` is mapped to category `0`). Note that if we wanted to split into \"Q\" in one group, we'd need to use two binary splits, first to separate \"C\" from \"Q\" and \"S\", and then a second split to separate \"Q\" from \"S\". For this reason, sometimes it can still be helpful to use dummy variables for categorical variables with few levels (like this one).\n",
    "\n",
    "In practice, I often use dummy variables for <4 levels, and numeric codes for >=4 levels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The random forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can't make the decision tree much bigger than the example above, since some leaf nodes already have only 50 rows in them. That's not a lot of data to make a prediction.\n",
    "\n",
    "So how could we use bigger trees? One big insight came from Leo Breiman: what if we create lots of bigger trees, and take the average of their predictions? Taking the average prediction of a bunch of models in this way is known as [bagging](https://link.springer.com/article/10.1007/BF00058655).\n",
    "\n",
    "The idea is that we want each model's predictions in the averaged ensemble to be uncorrelated with each other model. That way, if we average the predictions, the average will be equal to the true target value -- that's because the average of lots of uncorrelated random errors is zero. That's quite an amazing insight!\n",
    "\n",
    "One way we can create a bunch of uncorrelated models is to train each of them on a different random subset of the data. Here's how we can create a tree on a random subset of the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:51:20.332256Z",
     "iopub.status.busy": "2022-05-23T04:51:20.331981Z",
     "iopub.status.idle": "2022-05-23T04:51:20.338451Z",
     "shell.execute_reply": "2022-05-23T04:51:20.337562Z",
     "shell.execute_reply.started": "2022-05-23T04:51:20.332229Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_tree(prop=0.75):\n",
    "    n = len(trn_y)\n",
    "    idxs = random.choice(n, int(n*prop))\n",
    "    return DecisionTreeClassifier(min_samples_leaf=5).fit(trn_xs.iloc[idxs], trn_y.iloc[idxs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create as many trees as we want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:51:23.113575Z",
     "iopub.status.busy": "2022-05-23T04:51:23.113271Z",
     "iopub.status.idle": "2022-05-23T04:51:23.392216Z",
     "shell.execute_reply": "2022-05-23T04:51:23.391179Z",
     "shell.execute_reply.started": "2022-05-23T04:51:23.113547Z"
    }
   },
   "outputs": [],
   "source": [
    "trees = [get_tree() for t in range(100)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our prediction will be the average of these trees' predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:51:24.13079Z",
     "iopub.status.busy": "2022-05-23T04:51:24.130425Z",
     "iopub.status.idle": "2022-05-23T04:51:24.27515Z",
     "shell.execute_reply": "2022-05-23T04:51:24.274245Z",
     "shell.execute_reply.started": "2022-05-23T04:51:24.130754Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2272645739910314"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_probs = [t.predict(val_xs) for t in trees]\n",
    "avg_probs = np.stack(all_probs).mean(0)\n",
    "\n",
    "mean_absolute_error(val_y, avg_probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is nearly identical to what `sklearn`'s `RandomForestClassifier` does. The main extra piece in a \"real\" random forest is that as well as choosing a random sample of data for each tree, it also picks a random subset of columns for each split. Here's how we repeat the above process with a random forest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:51:24.302789Z",
     "iopub.status.busy": "2022-05-23T04:51:24.302442Z",
     "iopub.status.idle": "2022-05-23T04:51:24.586127Z",
     "shell.execute_reply": "2022-05-23T04:51:24.584999Z",
     "shell.execute_reply.started": "2022-05-23T04:51:24.302754Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18834080717488788"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rf = RandomForestClassifier(100, min_samples_leaf=5)\n",
    "rf.fit(trn_xs, trn_y);\n",
    "mean_absolute_error(val_y, rf.predict(val_xs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can submit that to Kaggle too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:51:26.868302Z",
     "iopub.status.busy": "2022-05-23T04:51:26.867535Z",
     "iopub.status.idle": "2022-05-23T04:51:26.902301Z",
     "shell.execute_reply": "2022-05-23T04:51:26.901088Z",
     "shell.execute_reply.started": "2022-05-23T04:51:26.868252Z"
    }
   },
   "outputs": [],
   "source": [
    "subm(rf.predict(tst_xs), 'rf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I found that gave nearly an identical result as our single tree (which, in turn, was slightly lower than our linear and neural net models in the previous notebook)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One particularly nice feature of random forests is they can tell us which independent variables were the most important in the model, using `feature_importances_`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-23T04:53:08.654508Z",
     "iopub.status.busy": "2022-05-23T04:53:08.654163Z",
     "iopub.status.idle": "2022-05-23T04:53:08.906439Z",
     "shell.execute_reply": "2022-05-23T04:53:08.904724Z",
     "shell.execute_reply.started": "2022-05-23T04:53:08.654445Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaQAAAD4CAYAAACjd5INAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXvUlEQVR4nO3dfZRddX3v8fcnEBMgGJ7URkADGFKFAEJAQZGAd3GraBFFBK2CFRHX1WWv197mLlqkIq23topwqxgfCvb6QFGxFqxgxQQUBCcaCJGHSzC2wScCNhqehPC9f8wOjsNk5mQyM2efmfdrrVk5Z+/f3vuzz8rkk/07e86kqpAkqdumdTuAJElgIUmSWsJCkiS1goUkSWoFC0mS1ArbdjtAr9ptt91q7ty53Y4hST1l+fLl66rqaUOts5BGae7cufT19XU7hiT1lCQ/3tw6p+wkSa1gIUmSWsFCkiS1gu8hSVKXPfroo6xdu5aHH36421HGzMyZM9ljjz2YPn16x9tYSJLUZWvXrmXHHXdk7ty5JOl2nK1WVdx3332sXbuWvfbaq+PtnLKTpC57+OGH2XXXXSdFGQEkYdddd93iKz6vkEZp5T3rmbv4ym7HmLTWfOC4bkeQJtRkKaNNRnM+XiFJklrBKyRJapmxnn0ZacbhiCOO4Prrrx/TY46GV0iSNMW1oYzAQpKkKW/WrFkALF26lKOOOoqTTjqJfffdl8WLF/PZz36Www47jAULFrB69WoATjvtNM4880yOPPJI9t13X6644ooxydETU3ZJNgIr6c97G3BqVT24mbHnABuq6m8nLqEkTQ4333wzt912G7vssgt77703p59+OjfddBMf+chHuPDCCzn//PMBWLNmDcuWLWP16tUcffTR3HXXXcycOXOrjt0rV0gPVdVBVbU/8BvgzG4HkqTJ6NBDD2XOnDnMmDGDffbZh2OPPRaABQsWsGbNmifGnXTSSUybNo158+ax9957c/vtt2/1sXulkAa6DngOQJI3Jbklyc1J/nHwwCRvTfK9Zv2XkmzfLH9tklub5dc2y/ZLclOSFc0+503oWUlSC8yYMeOJx9OmTXvi+bRp03jssceeWDf4tu6xuG29pwopybbAy4CVSfYDzgKOqaoDgXcNscmXq+rQZv1twFua5WcD/7VZ/ofNsjOBj1TVQcBCYO0Qxz8jSV+Svo0Prh/LU5OknnLZZZfx+OOPs3r1au6++27mz5+/1fvsifeQgO2SrGgeXwd8Cngb8MWqWgdQVfcPsd3+Sd4P7ATMAq5qln8HuDjJPwFfbpbdAJyVZA/6i+z/Dd5ZVS0BlgDMmDOvxuC8JOlJeuEHw+fPn89RRx3Fz3/+cy666KKtfv8IeqeQHmquXJ6Q/uvDkUrhYuBVVXVzktOARQBVdWaSFwDHASuSHFRVn0tyY7PsqiSnV9U1Y3saktQ+GzZsAGDRokUsWrToieVLly594vHgdS960Yv48Ic/PKY5emrKbpBvAicl2RUgyS5DjNkR+GmS6cAbNi1Msk9V3VhVZwPrgD2T7A3cXVUXAF8FDhj3M5AkPaFXrpCepKpWJTkPWNbcFv4D4LRBw/4CuBH4Mf23je/YLP9gc9NC6C+2m4HFwB8leRT4GfC+cT8JSepBF1988bjstycKqapmbWb5JcAlg5adM+Dxx4CPDbHdq4fY3V83X5I04apqUn3AatWWv83ey1N2kjQpzJw5k/vuu29U/4i30abfh7SlNzr0xBWSJE1me+yxB2vXruXee+/tdpQxs+k3xm6JTJZGnmgLFy6svr6+bseQpJ6SZHlVLRxqnVN2kqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFawkCRJrWAhSZJawUKSJLWChSRJagULSZLUCv4+pFFaec965i6+ctyPs+YDx437MSSpDbxCkiS1goUkSWoFC0mS1AoWkiSpFVpTSEk2jME+FiVZn2RF8/VvY5FNkjT+JuNddtdV1Su2ZIMk21bVY+MVSJI0stZcIQ0lyUFJvpvkliSXJ9m5WX5os+yGJB9Mcusw+zgsyfVJftD8Ob9ZflqSy5L8C3B1kh2SfDrJ95qxx0/QaUqSaHkhAZ8B/qyqDgBWAu9tlv8DcGZVHQ5sHLTNkQOm7M4CbgdeUlXPB84G/mrA2MOBU6vqGOAs4JqqOhQ4Gvhgkh0G7jjJGUn6kvRtfHD9GJ+qJE1trZ2ySzIb2KmqljWLLgEuS7ITsGNVXd8s/xwwcIrud6bskuwJXJJkHlDA9AFjv1FV9zePjwX+MMl7muczgWcBt20aXFVLgCUAM+bMq60/S0nSJq0tpGFkC8efC3yrqk5IMhdYOmDdA4P2+5qqumPr4kmSRqO1U3ZVtR74ZZIjm0VvBJZV1S+BXyd5YbP85BF2NRu4p3l82jDjrgLemSQASZ4/quCSpFFp0xXS9knWDnj+IeBU4KIk2wN3A29u1r0F+ESSB+i/4hnuDZ2/oX/K7t3ANcOMOxc4H7ilKaU1/O5UoCRpHKWq994KSTKrqjY0jxcDc6rqXROZYcaceTXn1PPH/Th+uKqkySTJ8qpaONS6Nl0hbYnjkvwv+vP/mOGn4iRJPaAnC6mqLgUu7XYOSdLY6clCaoMFu8+mz+k0SRozrb3LTpI0tVhIkqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFawkCRJrWAhSZJawUKSJLWChSRJagULSZLUChaSJKkVLCRJUitYSJKkVrCQJEmt0JOFlGRjkhVJbk1yWZLtt3J/c5PcOlb5JElbricLCXioqg6qqv2B3wBndrJRkm3HN5YkabR6tZAGug54TpJXJrkxyQ+S/FuSZwAkOSfJkiRXA59J8owklye5ufk6otnPNkk+kWRVkquTbNe1M5KkKainC6m54nkZsBL4NvDCqno+8AXgfw4YeghwfFW9HrgAWFZVBwIHA6uaMfOAv6+q/YD/BF4zxPHOSNKXpO/ee+8dp7OSpKmpV6ewtkuyonl8HfApYD5waZI5wFOAHw0Y/9Wqeqh5fAzwJoCq2gisT7Iz8KOq2rTP5cDcwQetqiXAEoCFCxfWGJ6PJE15vVpID1XVQQMXJLkQ+FBVfTXJIuCcAasf6GCfjwx4vBFwyk6SJlBPT9kNMhu4p3l86jDjvgm8HSDJNkmeOt7BJEkjm0yFdA5wWZLrgHXDjHsXcHSSlfRPze03AdkkSSNIlW+FjMbChQurr6+v2zEkqackWV5VC4daN5mukCRJPcxCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFawkCRJrWAhSZJawUKSJLWChSRJagULSZLUChaSJKkVLCRJUitYSJKkVrCQJEmtYCFJklph224H6FUr71nP3MVXPmn5mg8c14U0ktT7tvgKKcm0JE8djzCSpKmro0JK8rkkT02yA/BD4I4kfzq+0SRJU0mnV0jPq6pfAa8CvgY8C3jjeIWSJE09nRbS9CTT6S+kf66qR4Eat1SSpCmn00L6OLAG2AG4NsmzgV+NV6iRJDkryaoktyRZkeQFST6Z5HnN+g2b2e6FSW5strktyTkTGlyStFkd3WVXVRcAFwxY9OMkR49PpOElORx4BXBwVT2SZDfgKVV1egebXwKcVFU3J9kGmD+eWSVJnRu2kJK8e4TtPzSGWTo1B1hXVY8AVNU6gCRLgfdUVV/z/O+Ao4FfAidX1b3A04GfNtttpP8GDZorpX2A3YE9gb+pqk9M3ClJkkaasttxhK9uuBrYM8mdST6a5KghxuwAfL+qDgaWAe9tln+Y/jsEL0/ytiQzB2xzAHAccDhwdpJnDt5pkjOS9CXp2/jg+jE9KUma6oa9Qqqqv5yoIJ2qqg1JDgGOpP8K6NIkiwcNexy4tHn8f4EvN9u+L8lngWOB1wOnAIuacf9cVQ8BDyX5FnAY8JVBx14CLAGYMWeeN3VI0hjq9OeQ9miuKn6R5OdJvpRkj/EOtzlVtbGqllbVe4F3AK8ZaZMB266uqo8BLwUOTLLr4DGbeS5JGked3mX3D8BXgWfS/z7LvzTLJlyS+UnmDVh0EPDjQcOmASc2j18PfLvZ9rgkaZbPAzYC/9k8Pz7JzKagFgHfG/PwkqTN6vSz7J5WVQML6OIkfzIOeToxC7gwyU7AY8BdwBnAFweMeQDYL8lyYD3wumb5G4EPJ3mw2fYNVbWx6aibgCvp/6Hfc6vqJxNwLpKkRqeFtC7JHwGfb56fAtw3PpGGV1XLgSOGWLVowJhZzcO/GLTtycPs+s6qOmOrA0qSRqXTKbs/Bk4Cfkb/bdMnAm8er1CSpKmn0yukc4FTq+qXAEl2Af6W/qLqeVV1TrczSNJU12khHbCpjACq6v4kzx+nTD1hwe6z6fN3H0nSmOl0ym5akp03PWmukPzlfpKkMdNpqfwdcH2SL9L/8zknAeeNWypJ0pTT6YerfiZJH3AMEODVVfXDcU0mSZpSOp52awrIEpIkjYtO30OSJGlcWUiSpFawkCRJrWAhSZJawUKSJLWChSRJagULSZLUChaSJKkVLCRJUitYSJKkVvATu0dp5T3rmbv4yjHZ1xp/jYUkeYUkSWoHC0mS1AoWkiSpFSwkSVIrTNpCSnJCkkry+93OIkka2aQtJOAU4NvAyd0OIkka2aQspCSzgBcBb6EppCTTknw0yaokVyT5WpITm3WHJFmWZHmSq5LM6WJ8SZqSJmUhAa8Cvl5VdwL3JzkYeDUwF1gAnA4cDpBkOnAhcGJVHQJ8GjhvqJ0mOSNJX5K+jQ+uH/eTkKSpZLL+YOwpwPnN4y80z6cDl1XV48DPknyrWT8f2B/4RhKAbYCfDrXTqloCLAGYMWdejVd4SZqKJl0hJdkVOAbYP0nRXzAFXL65TYBVVXX4BEWUJA1hMk7ZnQh8pqqeXVVzq2pP4EfAOuA1zXtJzwAWNePvAJ6W5IkpvCT7dSO4JE1lk7GQTuHJV0NfAp4JrAVuBT4O3Aisr6rf0F9i/zvJzcAK4IgJSytJAibhlF1VLRpi2QXQf/ddVW1opvVuAlY261cAL5nAmJKkQSZdIY3giiQ7AU8Bzq2qn3U5jySpMaUKaairJ0lSO0ypQhpLC3afTZ+/x0iSxsxkvKlBktSDLCRJUitYSJKkVrCQJEmtYCFJklrBQpIktYKFJElqBQtJktQKFpIkqRUsJElSK1hIkqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFYYt0JKsjHJigFfi7dg20VJrtjK4y9NsnCU216c5MStOb4kactsO477fqiqDhrH/W9Wkm26cVxJ0uhN+JRdkjVJ/irJDUn6khyc5Kokq5OcOWDoU5NcnuSHSS5KMq3Z/mPNdquS/OWg/Z6d5NvAawcsn5bkkiTvT7JNkg8m+V6SW5K8rRmTJP+nOdaVwNMn6OWQJDXGs5C2GzRl97oB6/6jqg4HrgMuBk4EXgi8b8CYw4D/ASwA9gFe3Sw/q6oWAgcARyU5YMA2D1fVi6vqC83zbYHPAndW1Z8DbwHWV9WhwKHAW5PsBZwAzG+O9VbgiKFOKMkZTRn23XvvvaN5TSRJm9GtKbuvNn+uBGZV1a+BXyd5OMlOzbqbqupugCSfB14MfBE4KckZ9GefAzwPuKXZ5tJBx/k48E9VdV7z/FjggAHvD80G5gEvAT5fVRuBnyS5ZqjQVbUEWAKwcOHCGuH8JUlboFt32T3S/Pn4gMebnm8qycH/4FdzNfMe4KVVdQBwJTBzwJgHBm1zPXB0kk1jAryzqg5qvvaqqqs3czxJ0gRq823fhyXZq3nv6HXAt4Gn0l8665M8A3jZCPv4FPA14LIk2wJXAW9PMh0gyb5JdgCuBU5u3mOaAxw9PqckSdqc8Zyy2y7JigHPv15VHd/6DdwAfID+93WuBS6vqseT/ABYBdwNfGeknVTVh5LMBv4ReAMwF/h+kgD3Aq8CLgeOoX8K8U5g2RbklCSNgVQ5UzUaCxcurL6+vm7HkKSekmR5c2Pak7R5yk6SNIVYSJKkVrCQJEmtYCFJklrBQpIktYKFJElqBQtJktQKFpIkqRUsJElSK1hIkqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFSwkSVIrWEiSpFbYttsBetXKe9Yzd/GV3Y4hSRNqzQeOG7d9e4UkSWoFC0mS1AoWkiSpFSZ1ISU5K8mqJLckWZHkBd3OJEka2qS9qSHJ4cArgIOr6pEkuwFP6XIsSdJmTOYrpDnAuqp6BKCq1lXVT5IckmRZkuVJrkoyJ8nsJHckmQ+Q5PNJ3trV9JI0xUzmQroa2DPJnUk+muSoJNOBC4ETq+oQ4NPAeVW1HngHcHGSk4Gdq+oTg3eY5IwkfUn6Nj64fiLPRZImvUk7ZVdVG5IcAhwJHA1cCrwf2B/4RhKAbYCfNuO/keS1wN8DB25mn0uAJQAz5syr8T4HSZpKJm0hAVTVRmApsDTJSuC/Aauq6vDBY5NMA54LPATsAqydwKiSNOVN2im7JPOTzBuw6CDgNuBpzQ0PJJmeZL9m/X9v1p8CfLqZ3pMkTZDJfIU0C7gwyU7AY8BdwBn0T7ldkGQ2/ed/fpJHgdOBw6rq10muBf4ceG9XkkvSFDRpC6mqlgNHDLFqHfCSIZY/d8C27x6vXJKkoU3aKTtJUm+xkCRJrTBpp+zG24LdZ9M3jh/DLklTjVdIkqRWsJAkSa1gIUmSWsFCkiS1goUkSWoFC0mS1AoWkiSpFVLlb1EYjSS/Bu7odo5R2I3+j0/qNb2Yuxczg7knUi9mhq3L/eyqetpQK/zB2NG7o6oWdjvElkrSZ+6J0YuZwdwTqRczw/jldspOktQKFpIkqRUspNFb0u0Ao2TuidOLmcHcE6kXM8M45famBklSK3iFJElqBQtJktQKFtIIkvxBkjuS3JVk8RDrk+SCZv0tSQ7uRs7BOsj9+0luSPJIkvd0I+NgHWR+Q/Ma35Lk+iQHdiPnYB3kPr7JvCJJX5IXdyPnYCPlHjDu0CQbk5w4kfk2k2Wk13pRkvXNa70iydndyDlYJ691k31FklVJlk10xiHyjPRa/+mA1/nW5u/ILlt10KryazNfwDbAamBv4CnAzcDzBo15OfCvQIAXAjf2SO6nA4cC5wHv6ZHMRwA7N49f1kOv9Sx++37tAcDtvZB7wLhrgK8BJ7Y9M7AIuKLbr+8ocu8E/BB4VvP86W3PPGj8K4Frtva4XiEN7zDgrqq6u6p+A3wBOH7QmOOBz1S/7wI7JZkz0UEHGTF3Vf2iqr4HPNqNgEPoJPP1VfXL5ul3gT0mOONQOsm9oZrvWmAHoA13EnXydxvgncCXgF9MZLjN6DRz23SS+/XAl6vq36H/+3OCMw62pa/1KcDnt/agFtLwdgf+Y8Dztc2yLR0z0dqYaSRbmvkt9F+ZdltHuZOckOR24Ergjyco23BGzJ1kd+AE4KIJzDWcTv+OHJ7k5iT/mmS/iYk2rE5y7wvsnGRpkuVJ3jRh6YbW8fdjku2BP6D/Py5bxY8OGl6GWDb4f7edjJlobcw0ko4zJzma/kJqw3sxHeWuqsuBy5O8BDgX+C/jHWwEneQ+H/izqtqYDDV8wnWS+fv0f1bahiQvB74CzBvvYCPoJPe2wCHAS4HtgBuSfLeq7hzvcJuxJf+GvBL4TlXdv7UHtZCGtxbYc8DzPYCfjGLMRGtjppF0lDnJAcAngZdV1X0TlG04W/RaV9W1SfZJsltVdfNDNTvJvRD4QlNGuwEvT/JYVX1lQhI+2YiZq+pXAx5/LclHe+S1Xgusq6oHgAeSXAscCHSrkLbk7/XJjMF0HeBNDcN90V/YdwN78ds39vYbNOY4fvemhpt6IfeAsefQjpsaOnmtnwXcBRzR7bxbmPs5/PamhoOBezY9b3PuQeMvpvs3NXTyWv/egNf6MODfe+G1Bp4LfLMZuz1wK7B/mzM342YD9wM7jMVxvUIaRlU9luQdwFX033Xy6apaleTMZv1F9N999HL6/6F8EHhzt/Ju0knuJL8H9AFPBR5P8if030Xzq83tt9uZgbOBXYGPNv9rf6y6/EnJHeZ+DfCmJI8CDwGvq+a7uVs6zN0qHWY+EXh7ksfof61P7oXXuqpuS/J14BbgceCTVXVrmzM3Q08Arq7+K7ut5kcHSZJawbvsJEmtYCFJklrBQpIktYKFJElqBQtJktQKFpIkqRUsJElSK/x//m0ShhO2Xx0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.DataFrame(dict(cols=trn_xs.columns, imp=m.feature_importances_)).plot('cols', 'imp', 'barh');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that `Sex` is by far the most important predictor, with `Pclass` a distant second, and `LogFare` and `Age` behind that. In datasets with many columns, I generally recommend creating a feature importance plot as soon as possible, in order to find which columns are worth studying more closely. (Note also that we didn't really need to take the `log()` of `Fare`, since random forests only care about order, and `log()` doesn't change the order -- we only did it to make our graphs earlier easier to read.)\n",
    "\n",
    "For details about deriving and understanding feature importances, and the many other important diagnostic tools provided by random forests, take a look at [chapter 8](https://github.com/fastai/fastbook/blob/master/08_collab.ipynb) of [our book](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what can we take away from all this?\n",
    "\n",
    "I think the first thing I'd note from this is that, clearly, more complex models aren't always better. Our \"OneR\" model, consisting of a single binary split, was nearly as good as our more complex models. Perhaps in practice a simple model like this might be much easier to use, and could be worth considering. Our random forest wasn't an improvement on the single decision tree at all.\n",
    "\n",
    "So we should always be careful to benchmark simple models, as see if they're good enough for our needs. In practice, you will often find that simple models will have trouble providing adequate accuracy for more complex tasks, such as recommendation systems, NLP, computer vision, or multivariate time series. But there's no need to guess -- it's so easy to try a few different models, there's no reason not to give the simpler ones a go too!\n",
    "\n",
    "Another thing I think we can take away is that random forests aren't actually that complicated at all. We were able to implement the key features of them in a notebook quite quickly. And they aren't sensitive to issues like normalization, interactions, or non-linear transformations, which make them extremely easy to work with, and hard to mess up!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you found this notebook useful, please remember to click the little up-arrow at the top to upvote it, since I like to know when people have found my work useful, and it helps others find it too. (BTW, be sure you're looking at my [original notebook here](https://www.kaggle.com/jhoward/how-random-forests-work) when you do that, and are not on your own copy of it, otherwise your upvote won't get counted!) And if you have any questions or comments, please pop them below -- I read every comment I receive!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "133.002px",
    "width": "196.553px"
   },
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}