06-why-you-should-use-a-framework.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Introduction and set up","metadata":{}},{"cell_type":"markdown","source":"If you've finished going through my [Linear model and neural net from scratch\n](https://www.kaggle.com/code/jhoward/linear-model-and-neural-net-from-scratch) notebook, then now is a good time to look at how to do the same thing using a library, instead of doing it from scratch. We'll use fastai and PyTorch. The benefits of using these libraries is:\n\n- Best practices are handled for you automatically -- fast.ai has done thousands of hours of experiments to figure out what the best settings are for you\n- Less time getting set up, which means more time to try out your new ideas\n- Each idea you try will be less work, because fastai and PyTorch will do the many of the menial bits for you\n- You can always drop down from fastai to PyTorch if you need to customise any part (or drop down from the fastai Application API to the fastai mid or low tier APIs), or even drop down from PyTorch to plain python for deep customisation.\n\nLet's see how that looks in practice. We'll start by doing the same library setup as in the \"from scratch\" notebook:","metadata":{}},{"cell_type":"code","source":"from pathlib import Path\nimport os\n\niskaggle = os.environ.get('KAGGLE_KERNEL_RUN_TYPE', '')\nif iskaggle:\n    path = Path('../input/titanic')\n    !pip install -Uqq fastai\nelse:\n    import zipfile,kaggle\n    path = Path('titanic')\n    kaggle.api.competition_download_cli(str(path))\n    zipfile.ZipFile(f'{path}.zip').extractall(path)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-05-16T21:26:52.397584Z","iopub.execute_input":"2022-05-16T21:26:52.398131Z","iopub.status.idle":"2022-05-16T21:27:20.193935Z","shell.execute_reply.started":"2022-05-16T21:26:52.398019Z","shell.execute_reply":"2022-05-16T21:27:20.193065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll import the fastai tabular library, set a random seed so the notebook is reproducible, and pick a reasonable number of significant figures to display in our tables:","metadata":{}},{"cell_type":"code","source":"from fastai.tabular.all import *\n\npd.options.display.float_format = '{:.2f}'.format\nset_seed(42)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:27:20.196442Z","iopub.execute_input":"2022-05-16T21:27:20.196812Z","iopub.status.idle":"2022-05-16T21:27:23.389486Z","shell.execute_reply.started":"2022-05-16T21:27:20.196763Z","shell.execute_reply":"2022-05-16T21:27:23.388493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prep the data","metadata":{}},{"cell_type":"markdown","source":"We'll read the CSV file just like we did before:","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(path/'train.csv')","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:27:23.390719Z","iopub.execute_input":"2022-05-16T21:27:23.390946Z","iopub.status.idle":"2022-05-16T21:27:23.41585Z","shell.execute_reply.started":"2022-05-16T21:27:23.390919Z","shell.execute_reply":"2022-05-16T21:27:23.414775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When you do everything from scratch, every bit of feature engineering requires a whole lot of work, since you have to think about things like dummy variables, normalization, missing values, and so on. But with fastai that's all done for you. So let's go wild and create lots of new features! We'll use a bunch of the most interesting ones from this fantastic [Titanic feature engineering notebook](https://www.kaggle.com/code/gunesevitan/titanic-advanced-feature-engineering-tutorial/) (and be sure to click that link and upvote that notebook if you like it to thank the author for their hard work!)","metadata":{}},{"cell_type":"code","source":"def add_features(df):\n    df['LogFare'] = np.log1p(df['Fare'])\n    df['Deck'] = df.Cabin.str[0].map(dict(A=\"ABC\", B=\"ABC\", C=\"ABC\", D=\"DE\", E=\"DE\", F=\"FG\", G=\"FG\"))\n    df['Family'] = df.SibSp+df.Parch\n    df['Alone'] = df.Family==1\n    df['TicketFreq'] = df.groupby('Ticket')['Ticket'].transform('count')\n    df['Title'] = df.Name.str.split(', ', expand=True)[1].str.split('.', expand=True)[0]\n    df['Title'] = df.Title.map(dict(Mr=\"Mr\",Miss=\"Miss\",Mrs=\"Mrs\",Master=\"Master\")).value_counts(dropna=False)\n\nadd_features(df)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:27:23.417899Z","iopub.execute_input":"2022-05-16T21:27:23.418374Z","iopub.status.idle":"2022-05-16T21:27:23.608237Z","shell.execute_reply.started":"2022-05-16T21:27:23.41833Z","shell.execute_reply":"2022-05-16T21:27:23.606987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we discussed in the last notebook, we can use `RandomSplitter` to separate out the training and validation sets:","metadata":{}},{"cell_type":"code","source":"splits = RandomSplitter(seed=42)(df)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:27:23.609777Z","iopub.execute_input":"2022-05-16T21:27:23.610261Z","iopub.status.idle":"2022-05-16T21:27:23.617051Z","shell.execute_reply.started":"2022-05-16T21:27:23.610212Z","shell.execute_reply":"2022-05-16T21:27:23.616183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now the entire process of getting the data ready for training requires just this one cell!:","metadata":{}},{"cell_type":"code","source":"dls = TabularPandas(\n    df, splits=splits,\n    procs = [Categorify, FillMissing, Normalize],\n    cat_names=[\"Sex\",\"Pclass\",\"Embarked\",\"Deck\", \"Title\"],\n    cont_names=['Age', 'SibSp', 'Parch', 'LogFare', 'Alone', 'TicketFreq', 'Family'],\n    y_names=\"Survived\", y_block = CategoryBlock(),\n).dataloaders(path=\".\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:27:23.618653Z","iopub.execute_input":"2022-05-16T21:27:23.619435Z","iopub.status.idle":"2022-05-16T21:27:23.706067Z","shell.execute_reply.started":"2022-05-16T21:27:23.619384Z","shell.execute_reply":"2022-05-16T21:27:23.705359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here's what each of the parameters means:\n\n- Use `splits` for indices of training and validation sets:\n\n      splits=splits,\n    \n- Turn strings into categories, fill missing values in numeric columns with the median, normalise all numeric columns:\n    \n      procs = [Categorify, FillMissing, Normalize],\n    \n- These are the categorical independent variables:\n    \n      cat_names=[\"Sex\",\"Pclass\",\"Embarked\",\"Deck\", \"Title\"],\n    \n- These are the continuous independent variables:\n    \n      cont_names=['Age', 'SibSp', 'Parch', 'LogFare', 'Alone', 'TicketFreq', 'Family'],\n    \n- This is the dependent variable:\n    \n      y_names=\"Survived\",\n\n- The dependent variable is categorical (so build a classification model, not a regression model):\n\n      y_block = CategoryBlock(),","metadata":{}},{"cell_type":"markdown","source":"## Train the model","metadata":{}},{"cell_type":"markdown","source":"The data and model together make up a `Learner`. To create one, we say what the data is (`dls`), and the size of each hidden layer (`[10,10]`), along with any metrics we want to print along the way:","metadata":{}},{"cell_type":"code","source":"learn = tabular_learner(dls, metrics=accuracy, layers=[10,10])","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:27:23.707298Z","iopub.execute_input":"2022-05-16T21:27:23.707789Z","iopub.status.idle":"2022-05-16T21:27:23.726787Z","shell.execute_reply.started":"2022-05-16T21:27:23.707741Z","shell.execute_reply":"2022-05-16T21:27:23.725748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You'll notice we didn't have to do any messing around to try to find a set of random coefficients that will train correctly -- that's all handled automatically.\n\nOne handy feature that fastai can also tell us what learning rate to use:","metadata":{}},{"cell_type":"code","source":"learn.lr_find(suggest_funcs=(slide, valley))","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:27:23.728203Z","iopub.execute_input":"2022-05-16T21:27:23.731489Z","iopub.status.idle":"2022-05-16T21:27:25.934408Z","shell.execute_reply.started":"2022-05-16T21:27:23.731434Z","shell.execute_reply":"2022-05-16T21:27:25.933704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The two colored points are both reasonable choices for a learning rate. I'll pick somewhere between the two (0.03) and train for a few epochs:","metadata":{}},{"cell_type":"code","source":"learn.fit(16, lr=0.03)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:27:25.935368Z","iopub.execute_input":"2022-05-16T21:27:25.935688Z","iopub.status.idle":"2022-05-16T21:27:28.235979Z","shell.execute_reply.started":"2022-05-16T21:27:25.93566Z","shell.execute_reply":"2022-05-16T21:27:28.235073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We've got a similar accuracy to our previous \"from scratch\" model -- which isn't too surprising, since as we discussed, this dataset is too small and simple to really see much difference. A simple linear model already does a pretty good job. But that's OK -- the goal here is to show you how to get started with deep learning and understand how it really works, and the best way to do that is on small and easy to understand datasets.","metadata":{}},{"cell_type":"markdown","source":"## Submit to Kaggle","metadata":{}},{"cell_type":"markdown","source":"One important feature of fastai is that all the information needed to apply the data transformations and the model to a new dataset are stored in the learner. You can call `export` to save it to a file to use it later in production, or you can use the trained model right away to get predictions on a test set.\n\nTo submit to Kaggle, we'll need to read in the test set, and do the same feature engineering we did for the training set:","metadata":{}},{"cell_type":"code","source":"tst_df = pd.read_csv(path/'test.csv')\ntst_df['Fare'] = tst_df.Fare.fillna(0)\nadd_features(tst_df)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:31:03.526802Z","iopub.execute_input":"2022-05-16T21:31:03.527206Z","iopub.status.idle":"2022-05-16T21:31:03.559518Z","shell.execute_reply.started":"2022-05-16T21:31:03.527162Z","shell.execute_reply":"2022-05-16T21:31:03.558458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"But we don't need to manually specify any of the processing steps necessary to get the data ready for modeling, since that's all saved in the learner. To specify we want to apply the same steps to a new dataset, use the `test_dl()` method:","metadata":{}},{"cell_type":"code","source":"tst_dl = learn.dls.test_dl(tst_df)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:31:03.826524Z","iopub.execute_input":"2022-05-16T21:31:03.827754Z","iopub.status.idle":"2022-05-16T21:31:03.854977Z","shell.execute_reply.started":"2022-05-16T21:31:03.827684Z","shell.execute_reply":"2022-05-16T21:31:03.85406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can use `get_preds` to get the predictions for the test set:","metadata":{}},{"cell_type":"code","source":"preds,_ = learn.get_preds(dl=tst_dl)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:31:04.505947Z","iopub.execute_input":"2022-05-16T21:31:04.506524Z","iopub.status.idle":"2022-05-16T21:31:04.586309Z","shell.execute_reply.started":"2022-05-16T21:31:04.506482Z","shell.execute_reply":"2022-05-16T21:31:04.585385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, let's create a submission CSV just like we did in the previous notebook...","metadata":{}},{"cell_type":"code","source":"tst_df['Survived'] = (preds[:,1]>0.5).int()\nsub_df = tst_df[['PassengerId','Survived']]\nsub_df.to_csv('sub.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:31:05.603891Z","iopub.execute_input":"2022-05-16T21:31:05.604749Z","iopub.status.idle":"2022-05-16T21:31:05.619653Z","shell.execute_reply.started":"2022-05-16T21:31:05.604676Z","shell.execute_reply":"2022-05-16T21:31:05.618707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"...and check that it looks reasonable:","metadata":{}},{"cell_type":"code","source":"!head sub.csv","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:31:07.007378Z","iopub.execute_input":"2022-05-16T21:31:07.008492Z","iopub.status.idle":"2022-05-16T21:31:07.819079Z","shell.execute_reply.started":"2022-05-16T21:31:07.008411Z","shell.execute_reply":"2022-05-16T21:31:07.817493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ensembling","metadata":{}},{"cell_type":"markdown","source":"Since it's so easy to create a model now, it's easier to play with more advanced modeling approaches. For instance, we can create five separate models, each trained from different random starting points, and average them. This is the simplest approach of [ensembling](https://machinelearningmastery.com/tour-of-ensemble-learning-algorithms/) models, which combines multiple models to generate predictions that are better than any of the single models in the ensemble.\n\nTo create our ensemble, first we copy the three steps we used above to create and train a model, and apply it to the test set:","metadata":{}},{"cell_type":"code","source":"def ensemble():\n    learn = tabular_learner(dls, metrics=accuracy, layers=[10,10])\n    with learn.no_bar(),learn.no_logging(): learn.fit(16, lr=0.03)\n    return learn.get_preds(dl=tst_dl)[0]","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:31:14.094621Z","iopub.execute_input":"2022-05-16T21:31:14.095657Z","iopub.status.idle":"2022-05-16T21:31:14.10358Z","shell.execute_reply.started":"2022-05-16T21:31:14.095606Z","shell.execute_reply":"2022-05-16T21:31:14.102546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we run this five times, and collect the results into a list:","metadata":{}},{"cell_type":"code","source":"learns = [ensemble() for _ in range(5)]","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:31:14.618981Z","iopub.execute_input":"2022-05-16T21:31:14.620018Z","iopub.status.idle":"2022-05-16T21:31:24.138634Z","shell.execute_reply.started":"2022-05-16T21:31:14.61996Z","shell.execute_reply":"2022-05-16T21:31:24.137567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We stack this predictions together and take their average predictions:","metadata":{}},{"cell_type":"code","source":"ens_preds = torch.stack(learns).mean(0)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:32:57.494424Z","iopub.execute_input":"2022-05-16T21:32:57.495125Z","iopub.status.idle":"2022-05-16T21:32:57.499817Z","shell.execute_reply.started":"2022-05-16T21:32:57.49506Z","shell.execute_reply":"2022-05-16T21:32:57.498927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, use the same code as before to generate a submission file, which we can submit to Kaggle after the notebook is saved and run:","metadata":{}},{"cell_type":"code","source":"tst_df['Survived'] = (ens_preds[:,1]>0.5).int()\nsub_df = tst_df[['PassengerId','Survived']]\nsub_df.to_csv('ens_sub.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T21:33:20.950176Z","iopub.execute_input":"2022-05-16T21:33:20.951163Z","iopub.status.idle":"2022-05-16T21:33:20.960718Z","shell.execute_reply.started":"2022-05-16T21:33:20.951112Z","shell.execute_reply":"2022-05-16T21:33:20.959421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At the time of writing, this submission is well within the top 25% of entries to the competition.\n\n(A lot of submissions to this competition use additional external data, but we have restricted ourselves to just using the data provided. We'd probably do a lot better if we used external data too. Feel free to give that a try, and see how you go. Note that you'll never be able to get to the top of the leaderboard, since a lot of folks in this competition have cheated, by downloading the answers from the internet and uploading them as their submission. In a real competition that's not possible, because the answers aren't public, but there's nothing stopping people from cheating in a tutorial/practice competition like this one. So if you're ready for a real challenge, take a look at the [competitions page](https://www.kaggle.com/competitions/) and start working on a real competition!)","metadata":{}},{"cell_type":"markdown","source":"## Final thoughts","metadata":{}},{"cell_type":"markdown","source":"As you can see, using fastai and PyTorch made things much easier than doing it from scratch, but it also hid away a lot of the details. So if you only ever use a framework, you're not going to as fully understand what's going on under the hood. That understanding can be really helpful when it comes to debugging and improving your models. But do use fastai when you're creating models on Kaggle or in \"real life\", because otherwise you're not taking advantage of all the research that's gone into optimising the models for you, and you'll end up spending more time debugging and implementing menial boiler-plate than actually solving the real problem!\n\nIf you found this notebook useful, please remember to click the little up-arrow at the top to upvote it, since I like to know when people have found my work useful, and it helps others find it too. (BTW, be sure you're looking at my [original notebook](https://www.kaggle.com/jhoward/why-you-should-use-a-framework) here when you do that, and are not on your own copy of it, otherwise your upvote won't get counted!) And if you have any questions or comments, please pop them below -- I read every comment I receive!","metadata":{}}]}