{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Homework 2 - Python for data analysis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*By Todd Gureckis and Brenden Lake*. Code shared under the [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) license. Credit given where inspiration was obtained from others!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Check the class schedule for information about homework due dates. \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The goal of Homework 2 is to make sure you understand enough python for the types of data analysis that we will be doing the rest of the semester. At this point you should be pretty comfortable with the Jupyter notebook environment. If you need a refresher on python there are a number of excellent resources you can explore on your own:\n", "\n", "- Google for Education's [Beginner Python course](https://developers.google.com/edu/python/). Jump to section \"Python Intro\"\n", "- Corey Schafer's [Python Programming Beginner Tutorials](https://www.youtube.com/playlist?list=PL-osiE80TeTskrapNbzXhwoFUiLCjGgY7) on Youtube. These are great if you prefer to watch videos. Also can skip the first video on installing python because you are using Jupyter and there is nothing to install!\n", "\n", "In the homework I will give you some data, give examples of how to use a key programming construct in the analysis of this data, and then give you a chance to try to perform an analysis using what you have learned." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Importing libraries" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you have seen a few times now, before we start we often need to import some additional python libraries. This is one of the key features of python because there are really amazing libraries that will let you do almost anything! Also when you get a bit more advanced you can write your own libraries. \n", "\n", "There are several common ways of importing. Let's say we want to import a package `foo` that defines a function `widget`:\n", "\n", "* `import foo` will import the `foo` package; any reference to modules/classes/functions will need to be prefixed with `foo.`; e.g. `foo.widget()`\n", "* `import foo as bar` will import the `foo` package with the alias `bar`; any reference to modules/classes/functions will need to be prefixed with `bar.`; e.g. `bar.widget()` This is the preferred method especially when the name of the package is long.\n", "* `from foo import widget` can be used to import a specific module/class/function from `foo` and it will be available as `widget()`\n", "* `from foo import *` will import every item in `foo` into the current namespace; this is bad practice, don't do it.\n", "\n", "As you have seen from past notebooks we often want to import `pandas`, `numpy`, and `seaborn`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline \n", "\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Problem 1 (2 points)
\n", " Ooops I forgot to import our favorite plotting library `seaborn`. Import it below with the shorthand name `sns` and run the code so that this library is available to you later!\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# put your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are many great libraries for python. How do you learn about them? Check out [pypi.org](https://pypi.org) which lets you search for different libraries. Of course not all the libraries are installed on our class JupyterHub but all of the [standard ones are](https://docs.python.org/3/library/). One example of a standard library is the `math` library. This library implements a bunch of simple math functions ([full list](https://docs.python.org/3/library/math.html)). For example, `math.sqrt(x)` will take the square root of `x`. Try to execute the folowing cell:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "math.sqrt(2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Oops!
\n", " Did this not work? Don't worry that is expected!\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Problem 2 (2 points)
\n", " What is wrong with the previous cell? Why did it not run? Write the code necessary so it works. You can copy the command again below. Keep in mind the advice above about libraries!\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# put your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Getting Data, Dealing with Lists and Dictionaries" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we explored in the labs so far, the most common way to read in data for analysis is from a CSV (comma separate values) file. This is like a simple version of an excel spreadsheet. Indeed you can export files from Excel as a CSV and also from Google Sheets.\n", "\n", "```\n", "df = pd.read_csv(\"http://myurl.edu/mydata.csv\")\n", "```\n", "\n", "However there are many ways to get data into Python. For example, another approach is to use what is known as a data API (or application programming interfaces). Very simply APIs are ways of getting access to data using code. Many websites including Twitter, Yelp, Strava, etc.. provide APIs that let you access and process at least parts of the data on their websites. Often companies release python libraries to make this easier.\n", "\n", "In the rest of the homework we are going to explore one example of this using the [Petfinder.com](https://petfinder.com) website. Petfinder is a website that helps people locate pets for adoption. It maintains a searchable database of pets from many different shelters and makes the data on the pets searchable. There is also a cool python library call `petpy` which help you to interact with the Petfinder website.\n", "\n", "To get started let's import the `petpy` library. Here I am not using the `import X as Y` syntax because petpy is pretty easy to type. I am also importing a few other libraries for displaying images in the Jupyter notebook and downloading files over the internet." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install petpy" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import petpy\n", "from IPython.display import Image\n", "import requests" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To access Petfinder you have to signup on their website. I have already done this for you so you can use my creditials for class:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "MYKEY = 'nygvq6mFXsSNN6g3BLHOE702FNfEmyQIUsHc7brGQTo1bg5wwV'\n", "MYSECRET = 'aEYDUYD9JnCksb2uoIUtkGu9pviy4tJdruZ3v4jx'\n", "pf = petpy.Petfinder(key=MYKEY, secret=MYSECRET)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The variable `pf` now contains an API interface to the Petfinder website." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animal_types = pf.animal_types()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The new variable we created must contain something but honestly I have no idea what it is right at first. So I will put it in a cell by itself so I can see what it contains." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animal_types" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A cool, ok so `animal_types` looks like a [python dictionary](https://www.w3schools.com/python/python_dictionaries.asp) which you learned about in a prior lab session. How do I know this? Well I notice that the entire output of the cell above is wrapped in `{ }` which is how you denote a dictionary in python. For example:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "my_dictionary = {\"pasta\": \"a type of italian food\", \"pizza\": \"another type of italian food\"}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the cell above, I just created a new variable called `my_dictionary` which has two *keys* ('pasta' and 'pizza') and two corresponding values/descriptions. I can lookup the description of each of the key like this" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(my_dictionary['pasta'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(my_dictionary['pizza'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(my_dictionary['salad'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Oops, that was expected.. 'salad' wasn't defined in our original dictionary so it can't be looked up (we got an exception called 'KeyError' because the key 'salad' could not be found in the dictionary.\n", "\n", "The idea with dictionaries is that you can store stuff and easily look it up by name.\n", "\n", "Getting back to these pets! What are the keys of this dictionary?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animal_types.keys()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok so there is one key, `types`. This must just mean that the API wraps everything up so lets look inside that key. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animal_types['types']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok this is weird but this contains a [python list](). How do I know that? Because it is wrapped in `[ ]`. How many things are in the list?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "len(animal_types['types'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "ok, what is in the first slot or position of the list?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animal_types['types'][3]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ah! Another dictionary (indicated by the `{ }`). But now I can kind of understand what is going on. What are the keys of this dictionary?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animal_types['types'][0].keys()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok, so the keys are things that are describing a type of pet! We have a field called `name` and one called `coats` which sounds fascinating, and `colors` and `genders`. Cool!\n", "\n", "So what is the first type of animal in this list?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animal_types['types'][0]['name']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok, so it is good-boy pups. What are the possible colors?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animal_types['types'][0]['colors']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Really cool stuff. I think a Merle color dog might really suit me! Although I don't know that that is actually. Let me consult wikipedia really quick!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import wikipedia\n", "wikipedia.summary(\"merle long coat\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Isn't python so cool? You can grab wikipedia entiries with one line just by typing `import wikipedia`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok, so let's challenge our understanding of working with lists and dictionaries. What are the possible colors for cats according to the petfinder website?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Try to access the different colors of cats here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Did you get stuck? Give it your best shot but if you really get fustrated you can run the cell below to reveal my solution." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Uncomment and run next line for a sample solution\n", "# %load hw2-hint1.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Problem 3 (2 points)
\n", " What are the colors for the \"small and furry\" animals (mice, gerbils, etc...)?\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# put your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## For loops and other ways to repeat things" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So far we have kind of been playing with dictionaries and list by accessing individual elements. However sometimes we want to know more general stuff like what is the name of all the types of animals? To do this we kind of want to use the methods we developed above to look inside this data frame but to repeat this for each element within the structure. This brings us to the dreaded `for` loop. Many students are afraid of for loops because they can get a bit complex. However, they are also a super critial tool for working with data effectively and open up so many more interesting types of analyses to you." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The most common use for a `for` loop in data analysis is to repeat at step for each record of some data. For instance you might want to print out the value of each item in a list:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "my_list = ['a','b','c','d']\n", "for item in my_list:\n", " print(item)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "you can use features of list to reverse the printout." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "my_list = ['a','b','c','d']\n", "for item in my_list[::-1]:\n", " print(item)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "or each item in a dictionary:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "weapon_strengths = {\n", " 'tomahawk': 20,\n", " 'katana': 60,\n", " 'hand of god': 100,\n", "}\n", "\n", "for weapon in weapon_strengths.keys():\n", " print(f\"I have a {weapon}!\")\n", "\n", "print(\"\\n---\\n\")\n", "\n", "for weapon, strength in weapon_strengths.items():\n", " print(f\"My {weapon} has a strength of {strength}!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This second example shows how you can repeat the printing command for each key in the dictionary `weapon_strengths.keys()` or all the items (like a list): `weapon_strengths.items()`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lets try it out for the animals! What are the types of animals in the Petfinder system?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for animal in animal_types['types']:\n", " print(animal['name'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here I am repeating the same code for each entry in the `animal_types['types']` list. Each entry in that list is first assigned to the variable `animal` and then I look inside that dictionary and access the `name` key." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Problem 4 (2 points)
\n", " Write a for loop that print out the name of each animal category and the coats they have. Your output should look like this:\n", "
\n",
    "Dog: ['Hairless', 'Short', 'Medium', 'Long', 'Wire', 'Curly']\n",
    "Cat: ['Hairless', 'Short', 'Medium', 'Long']\n",
    "Rabbit: ['Short', 'Long']\n",
    "Small & Furry: ['Hairless', 'Short', 'Long']\n",
    "Horse: []\n",
    "Bird: []\n",
    "Scales, Fins & Other: []\n",
    "Barnyard: ['Short', 'Long']\n",
    "
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# put your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So far we have been exploring repeating loops for each entry in a dataset that we obtained via an API. However, sometimes we just want to repeat things a few times in order to do other stuff. For instance, as you will see you can use python and `for` loops to draw simple pictures!\n", "\n", "In order to use `for` loops in this way we have to *create data*. One of the easiest ways to create data is the `range()` command which creates a list with a sequence of numbers." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "list(range(5))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "list(range(10))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So if we wanted to repeat the same thing 5 times we could do this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for x in range(5):\n", " print(\"hi\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also place `for` loops inside another `for` loop if you tab is over appropriately. If this happens you can repeat things in more complex ways. For instance you can print out a square like this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i in range(10):\n", " for j in range(10):\n", " print(\"*\", end = ' ')\n", " print(\"\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Problem 5 (4 points)
\n", " Modify the code for the square above to write a nested for loop that prints out a triangle with height 5 and width 5. Your output should look like this:\n", "
\n",
    "* \n",
    "* * \n",
    "* * * \n",
    "* * * * \n",
    "* * * * * \n",
    "
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`for` loops are really interesting when combined with `if` statement. This means while you are repeating the loop you can skip thing or do something different. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i in range(10):\n", " for j in range(10):\n", " if i < 2 or i > 7:\n", " print(\"*\", end = ' ')\n", " else:\n", " print(\" \", end = ' ')\n", " print(\"\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For example the code above prints out the stars for each square only if the row (i) is less than 2 or greater than 7 (otherwise it prints a empty space instead due to the `else`). Think about this one for a bit:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i in range(10):\n", " for j in range(10):\n", " if (i < 3 or i > 6) or (j < 3 or j > 6):\n", " print(\"*\", end = ' ')\n", " else:\n", " print(\" \", end = ' ')\n", " print(\"\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Problem 6 (4 points)
\n", " Modify your code for the triangle to write a nested for loop that prints out a diamond with height 7 and width 7. You'll probably want to combined nested loops with if/else statements. Your output should look like this:\n", "
\n",
    "      *\n",
    "    * * *\n",
    "  * * * * *\n",
    "* * * * * * *\n",
    "  * * * * * \n",
    "    * * *\n",
    "      *\n",
    "
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "
\n", " BONUS Problem 1 \"The Wave\"(4 points)
\n", " \n", " \n", " \n", "Create a list called 'people' which is full of zeros, except the first number is a 1. Then write a simple program that steps through each element of the vector. If the \"person\" to the left is standing (i.e., is set to 1.0) then have this current person stand up too. If the current person is already standing then have them sit back down. On each pass through print the contents of the people list. Bonus points for making the wave go forwards and backwards (i.e., left to right then back right to left).\n", " \n", "
\n",
    "people = [1,0,0,0,0,0,0,0,0,0]\n",
    "for each number \"i\"\" in between 0 (the start) and the length of people (minus 1):\n",
    "    if person \"i\" is standing up:\n",
    "        make the person one to the right of person \"i\" stand up\n",
    "        make person \"i\" sit down\n",
    "        print people\n",
    "
\n",
    "also remember,\n",
    "
\n",
    "len(people)   # gives you the length of the \"stadium\" of people        \n",
    "people[0] # this is how  you select individual \"people\"\n",
    "people[1]\n",
    "people[2]\n",
    "
\n",
    "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Enough of this programming! As a reward for completing these tough new problems, we'll look at some cute pets using `for` loops. The petfinder API has a function which will return the set of pets currently available for adoption. This takes a number of options but by default will return 20 pet listings." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animals = pf.animals(return_df=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This returns a pandas dataframe that contains lots of information about the pet including if it is declawed, house trained, etc..." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animals.columns" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "animals.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from skimage import io\n", "\n", "def show_animal(animals, number, ax):\n", " record=animals.iloc[number]\n", " if record.photos:\n", " if record.photos[0]:\n", " if 'medium' in record.photos[0]:\n", " photo = record.photos[0]['medium']\n", " img=io.imread(photo)\n", " ax[number].set_axis_off()\n", " ax[number].imshow(img)\n", " \n", "\n", "fig, axes = plt.subplots(4, 5, figsize=(12, 12))\n", "ax = axes.ravel()\n", "\n", "for i in range(20): # check out my for loop!\n", " show_animal(animals,i, ax)\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The previous cell requires a bit of [matplotlib](https://matplotlib.org) magic to create a grid of images but the key is that there is a for loop that steps through and displays each image. Try to understand this code and compare it to the loops described above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Random numbers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In many aspects of analysis you need to generate random numbers. You need this for a few reasons\n", "1. to create data from a distibution to check your intuitions about randomness (simulation)\n", "2. to create cognitive models that have \"noise\" in them so that their behavior is not deterministic\n", "3. to test your code by providing \"fake\" data that looks realistic\n", "\n", "One of the useful libraries for this is numpy's `random` submodule. The full list of available functions in this library is [here](https://docs.scipy.org/doc/numpy-1.14.0/reference/routines.random.html) and reading through can give you a good sense of what is possible." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy.random as npr" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`randn()` gives you a random number of a standard normal distribution (one with the mean = 0 and standard deviation equal to 1.0:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.randn()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "you can get more than one like this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.randn(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "lets get even more (1000) and plot them" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sns.displot(npr.randn(1000))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "if you want the numbers to be uniform you can use `rand()` which draws random numbers from 0 to 1 (but not including exactly 1.0)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.rand()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.rand(1000).max()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "as you can see the maximum value even of 1000 random `rand()` numbers is less than 1.0. while for `randn()` it can be much higher:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.randn(1000).max()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`randint()` will only include the integers between the given range." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.randint(0,10,10)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.randint(0,100,10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "you can also choose a random element of an array or list so that only certain values can be chosen:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.choice(np.array([3,6,1,4]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "notice that even in 100 draws you never see 0 or 9 because they are not in the given array." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.choice(np.array([3,6,1,4]),100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also use random to randomly \"shuffle\" a list (meaning changing the order of things like you would a deck of playing cards)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cards = np.arange(10)\n", "print(cards)\n", "npr.shuffle(cards)\n", "print(cards)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The second list is the same as the first just randomly shuffled." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are lots of distributions you can draw random numbers from:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Chi-square**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sns.displot(npr.chisquare(2,10000))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**lognormal**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sns.displot(npr.lognormal(0,1,10000))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**F** \n", "you'll see this again when we get to ANOVA!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sns.displot(npr.f(2,16,10000))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `cumsum()` command computes the cumulative sum of a list. Meaning it is the sum created by adding each entry to the list to the values of all the ones that come before." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "np.array([1,2,3,4]).cumsum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So the result is `array([1, 2+1, 3+2+1, 4+3+2+1])`. Now we can start adding up random numbers. For example, " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.randn(20).cumsum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is sometimes called a \"random walk\" because you are taking small steps either up or down each time you add a number (the value from `npr.randn()`) but how far you move depends on the current sum. You can get a sense of this by plotting the sum as it evolves:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.plot(npr.randn(200).cumsum())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Try running the above cell a few times to get a sense of the range of random walks that are possible.\n", "\n", "You can also plot a few random walks on the same plot together:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i in range(5):\n", " plt.plot(npr.randn(200).cumsum(),alpha=0.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's change the code to plot even more of these... maybe like 500!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i in range(500):\n", " plt.plot(npr.randn(200).cumsum(),alpha=0.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's interesting... the sum start at zero (makes sense) and then kind of grows to get bigger as time goes on. Not all paths get bigger of course but some randomly add up a bunch of positive or negative numbers and tend to drift further and further from the starting point." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now instead of computing the sum, let's compute the average at each step. So for each step we need to divide by the number of things we have added together so far." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "npr.randn(50).cumsum()/(np.arange(1,50+1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice we needed to use `np.arange(1,50+1)` because we want to start with value 1 for the first number (`np.arange(50)` would count from 0 to 49 and we would then be dividing by zero on the first sum.\n", "\n", "Let's plot a bunch of these random averages:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i in range(50):\n", " plt.plot(npr.randn(50).cumsum()/(np.arange(50)+1), alpha=0.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's interesting, you can see that early on the average is quite variable but as you add more and more numbers the averages seem to converge around a particular value. As it turns out you are looking at the sampling distribution of the mean for different values of N. If you took a slice out of the plot at say 10 sums and made a histogram of the averages you would get a histogram of the sampling distribution of the mean." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sns.displot(npr.randn(10,50).sum(axis=0)/10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Interestingly the \"envelope\" of the sampling distribution of the mean seems to be shrinking according to a particular mathematical relationship which is $1/\\sqrt N$ where $N$ is the number of numbers you have added to the sum.\n", "\n", "In fact we can plot the sqrt(N) on the same plot to see the relationship" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for i in range(50):\n", " plt.plot(npr.randn(50).cumsum()/(np.arange(1,50+1)), alpha=0.2)\n", "plt.plot(1.0/np.sqrt(np.arange(1,50+1)),'k-')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, the thick black line is $1/\\sqrt N$ which is how the variance of the sampling distribution changes as the sample size increases! (remember the formula for the standard error of the mean is the standard devision of the sample divided by the square root of N). Now you can see why!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Problem 7 (4 points)
\n", " Modify the code above to plot the sampling distribution for number drawn from a random uniform distribution and from the chi-sq distribution (see above, you have all the ingredients you need). Does the shape of the curves look that same? Why is that?\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Pandas and Data Frames" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A DataFrame is like a dictionary where the keys are column names and the values are Series that share the same index and hold the column values. The first \"column\" is actually the shared Series index (there are some exceptions to this where the index can be multi-level and span more than one column but in most cases it is flat)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "names = pd.Series(['Alice', 'Bob', 'Carol'])\n", "phones = pd.Series(['555-123-4567', '555-987-6543', '555-245-6789'])\n", "dept = pd.Series(['Marketing', 'Accounts', 'HR'])\n", "\n", "staff = pd.DataFrame({'Name': names, 'Phone': phones, 'Department': dept}) # 'Name', 'Phone', 'Department' are the column names\n", "staff" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note above that the first column with values 0, 1, 2 is actually the shared index, and there are three series keyed off the three names \"Department\", \"Name\" and \"Phone\".\n", "\n", "Like `Series, DataFrame` has an index for rows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff.index" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`DataFrame` also has an index for columns:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff.columns" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff.values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The index operator actually selects a column in the DataFrame, while the .iloc and .loc attributes still select rows (actually, we will see in the next section that they can select a subset of the DataFrame with a row selector and column selector, but the row selector comes first so if you supply a single argument to .loc or .iloc you will select rows):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff['Name'] # Acts similar to dictionary; returns a column's Series" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff.loc[2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can get a transpose of the DataFrame with the .T attribute:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff.T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also access columns like this, with dot-notation. Occasionally this breaks if there is a conflict with a UFunc name, like 'count':" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff.Name" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can add new columns. Later we'll see how to do this as a function of existing columns:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff['Fulltime'] = True\n", "staff.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use `.describe()` to get summary statistics:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use `.quantile()` to get quantiles:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = pd.DataFrame([2, 3, 1, 4, 3, 5, 2, 6, 3])\n", "df.quantile(q=[0.25, 0.75])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use `.drop()` to remove rows. This will return a copy with the modifications and leave the original untouched unless you include the argument `inplace=True`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff.drop([1])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Note that because we didn't say inplace=True,\n", "# the original is unchanged\n", "staff" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are many ways to construct a DataFrame. For example, from a Series or dictionary of Series, from a list of Python dicts, or from a 2-D NumPy array. There are also utility functions to read data from disk into a DataFrame, e.g. from a .csv file or an Excel spreadsheet. We'll cover some of these later.\n", "\n", "Many DataFrame operations take an axis argument which defaults to zero. This specifies whether we want to apply the operation by rows (axis=0) or by columns (axis=1).\n", "\n", "You can drop columns if you specify `axis=1`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff.drop([\"Fulltime\"], axis=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another way to remove a column in-place is to use `del`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "del staff[\"Department\"]\n", "staff" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can change the index to be some other column. If you want to save the existing index, then first add it as a new column:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff['Number'] = staff.index\n", "staff" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Now we can set the new index. This is a destructive\n", "# operation that discards the old index, which is\n", "# why we saved it as a new column first.\n", "staff = staff.set_index('Name')\n", "staff" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alternatively you can promote the index to a column and go back to a numeric index with `reset_index()`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "staff = staff.reset_index()\n", "staff" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Problem 8 (2 points)
\n", " Create a data frame from the dictionary called `example_data` below. Then generate a summary of the data. Calculate the sum of all the visits (i.e., the total number of visits).\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "example_data = {'animal': ['cat', 'cat', 'snake', 'dog', 'dog', 'cat', 'snake', 'cat', 'dog', 'dog'],\n", " 'age': [2.5, 3, 0.5, np.nan, 5, 2, 4.5, np.nan, 7, 3],\n", " 'visits': [1, 3, 2, 3, 2, 3, 1, 1, 2, 1],\n", " 'priority': ['yes', 'yes', 'no', 'yes', 'no', 'no', 'no', 'yes', 'no', 'no']}" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Put your code to create the DataFrame here\n", "# Generate a summary of the data\n", "# Calculate the sum of all the visits (the total number of visits)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Turning in homeworks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you are finished with this notebook. Save your work in order to turn it in. To do this select *File*->*Download As...*->*PDF*.\n", "\n", "\n", "\n", "Homeworks will be submitted according to the instructions provided in class." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }