{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "tags": [ "hide_input" ] }, "outputs": [], "source": [ "from IPython.core.display import HTML, Markdown, display\n", "\n", "import numpy as np\n", "import numpy.random as npr\n", "import pandas as pd\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "import scipy.stats as stats\n", "import statsmodels.formula.api as smf\n", "import pingouin as pg\n", "import math\n", "\n", "import ipywidgets as widgets\n", "\n", "import ipywidgets as widgets\n", "from IPython.display import display, Markdown, clear_output\n", "import numpy as np\n", "import pandas as pd\n", "import os\n", "from datetime import datetime\n", "from ipycanvas import Canvas, hold_canvas\n", "\n", "from sdt_exp import *\n", "\n", "# Enable plots inside the Jupyter Notebook\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Signal detection: Part 2" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "popout" ] }, "source": [ "Authored by *Todd Gureckis*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the previous lab you learned some of the basic analysis steps involved in signal detection analysis. In particular, we learned how to compute a statistic we called d' (dprime) and c (bias/criterion). These variables are commonly used in studies of perception to understand the performance of a subject on a task independent of their response bias (here meaning tendency to give a particular response). \n", "\n", "In the second part of the lab we will analyze data from a simple detection experiment you will have just conducted on yourselves. You will get to put your python knowledge into practice by reading in the data analyzing it. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The Experiment Instructions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this experiment you job will be similar to that of a doctor. There are going to be small \"clusters\" of points which we might think of as a tumor or some other biological measurement. You can use the interaction element below to examine a few of these \"clusters\".\n", "\n", "Try using the widget below to slide between a lot of dots within a cluster and a small number. Sweep back and forth and get an idea of what a \"cluster\" could look like. The dots are spread out to a certain degree as you can see more clearly with the large number of dots. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "cf233898cd984ee59a899376ec154617", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(IntSlider(value=36, description='dots', max=75, min=1, step=5), Output()), _dom_classes=…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "@widgets.interact(dots=(1,75, 5))\n", "def draw(dots):\n", " return draw_stimulus(include_source=True, n_background_dots=0, n_source_dots=dots, source_sd=10, source_border=20, width=400, height=400)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The challenge is that these dots are going to be hidden in a background of \"noise\" which is other dots unrelated to the cluster. Here is an example of the background with no cluster present:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9fd75a4ba3f24ab0a49e4dec3b0dd5ed", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Canvas(height=425, layout=Layout(grid_area='canvas'), width=425)" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "draw_stimulus(include_source=False, n_background_dots=30*30, n_source_dots=0, source_sd=10, source_border=20, width=400, height=400)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can run the above cell many times to see different example patterns when the cluster is absent.\n", "\n", "Next here is an example of a very large cluster hidden in the background noise:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5ac6e759393c4f94bbb4994d782594b8", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Canvas(height=425, layout=Layout(grid_area='canvas'), width=425)" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "draw_stimulus(include_source=True, n_background_dots=30*30, n_source_dots=60, source_sd=10, source_border=20, width=400, height=400)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From this example you can see a very clear \"cluster\" or tumor in the stimulus. Here is one where there is a cluster/tumor but it is a little less obvious:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e19c073a2f1d4f698be16dbb311bc4ad", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Canvas(height=425, layout=Layout(grid_area='canvas'), width=425)" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "draw_stimulus(include_source=True, n_background_dots=30*30, n_source_dots=50, source_sd=10, source_border=20, width=400, height=400)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, here is one where it is pretty hard to tell that there is a cluster at all (oh but there is one involving only 20 dots!):" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "940b9669d0514fceb3a8451f4385b1b2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Canvas(height=425, layout=Layout(grid_area='canvas'), width=425)" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "draw_stimulus(include_source=True, n_background_dots=30*30, n_source_dots=20, source_sd=10, source_border=20, width=400, height=400)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Your job in the experiment will be to view stimuli such as the ones above and judge if you think there is a cluster/tumor present in the image. It can often be hard to tell because even in the random dots there can be little clusters of dots that group together. Just try to do your best. Notice that there is a red \"absent\" button and a green \"present\" button. You should press \"absent\" if you think the cluster/tumor is not there, and press \"present\" if you think it is there. There are 100 trials overall.\n", "\n", "Before running the next cell set the subject number to your NYU netid (I will anonymize this later but will let me check that everyone has done the experiment)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Run the Experiment!" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "df7fd6f194e34af28c26f18b52c7c0b1", "version_major": 2, "version_minor": 0 }, "text/plain": [ "GridBox(children=(Button(description='Present', layout=Layout(grid_area='button0', width='auto'), style=Button…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "subject_number = 'bl1611' # set your subject number here\n", "exp = Detection_Experiment(subject_number)\n", "exp.start_experiment()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Getting the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After you run the experiment, the data will be output to a .csv file tied to your subject number. Also it is available as as pandas data frame in the current kernel environment as `exp.trials`:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
signal_presentsignal_typetrial_numbutton_positionsubject_number
01600leftbl1611
11101leftbl1611
21102leftbl1611
31153leftbl1611
4004leftbl1611
\n", "
" ], "text/plain": [ " signal_present signal_type trial_num button_position subject_number\n", "0 1 60 0 left bl1611\n", "1 1 10 1 left bl1611\n", "2 1 10 2 left bl1611\n", "3 1 15 3 left bl1611\n", "4 0 0 4 left bl1611" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp.trials.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Experiment design\n", "\n", "In the experiment, you saw a number of trials where field of random colored and positioned dots appear. On some trials a small \"cluster\" of points was added to the stimulus. The number of dots in the cluster was manipulated across conditions so that it was more or less obvious that there was a cluster. In some cases the number of dots within the cluster was so small that it was hard to detect it, but other cases it might have been more visually obvious. This is very similar to the type of visual diagnosis that a doctor or airport screener might perform (deciding of a tumor or a gun is present in an image). Your task was to indicate if it was a \"stimulus/cluster present\" trial or a \"stimulus/cluster absent\" trial. The computer recorded your response and your reaction time.\n", "\n", "An overview of the situation is shown here:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The things we manipulated!\n", "\n", "There were actually four different types of clusters shown on any given \"stimulus present\" trial. One was a **small** stimulus made up of 10 dots, one was a medium made up of 15 dots, one type of made up of 25, and finally one was made up of 60 dots:\n", "\n", "
\n", "\n", "
\n", "\n", "As the number of dots increases the general visibility of the cluster becomes more clear and easier to detect.\n", "\n", "In total there were 100 stimulus absent trials and 25 of each of the four types of stimulus present trials. The purpose of this was so that a perfect subject with robot-quality vision would say \"yes\" half the time in the experiment so there is no overall bias toward one response.\n", "\n", "In this lab we are interested in analyzing the effect of strength of the stimulus cluster (i.e., number of dots) on the two signal detection variables we learned about in the previous lecture and lab." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analysis Steps\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Remember from the previous lab that our model for signal detection looks something like this:\n", "\n", "\n", "and we showed how we can compute $d'$ and $c$ by computing only the number of false alarms and hits and subjecting those to a particular mathematical transformation. In this case we actually have four stimulus conditions (10, 15, 25, or 60). So our goal will be to compute the hits and false alarms **separately** for the two different types of stimuli. This will mean that each person in our experiment will be described by eight numbers:\n", "\n", "- A d' (d-prime) under (10, 15, 25, 60) visibility cluster conditions\n", "- A c under (10, 15, 25, 60) visibility cluster conditions\n", "\n", "Once we have this we want to perform a simple hypothesis test. Is performance, measured by d' higher in the high visibility conditions? We have strong reason to suspect this is the case but it'll be a nice check of our understanding if we can related the abstract calculation we performed in the previous lab to answer this simple quesiton.\n", "\n", "Answering this question will push our skills at using pandas to organize our data, and using pingouin to compute a t-test." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When computing d-prime with real data, sometimes we run into issues if the hit rate is perfect (1.0) or the false alarm rate is perfect (0.0). D-prime is undefined in these cases, and it's likely unrealistic anyway to assume that the rates are really perfect. Thus, to keep the calculations numerically stable, please use `ppf_smooth` (see below) as a replacement for `stats.norm.ppf`. This new function uses uses 0.01 as the minimum rate, and\n", "0.99 as the maximum. (There are somewhat more elegant ways to do this if you are publishing your analyses, see this [link](https://stats.stackexchange.com/questions/134779/d-prime-with-100-hit-rate-probability-and-0-false-alarm-probability), but this will be okay for our purposes here)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def ppf_smooth(rate):\n", " # Replacement for stats.norm.ppf that uses 0.01 as the minimum rate, and\n", " # 0.99 as the maximum\n", " #\n", " # Input: \n", " # rate : either hit or false alarm rate\n", " assert(rate >= 0. and rate <=1.) # rate must be between 0. and 1.\n", " rate = max(rate,0.01)\n", " rate = min(rate,0.99)\n", " return stats.norm.ppf(rate)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 1
\n", " If we compute one $d'$ for each participant for each stimulus conditon and want to compare performance in two different stimulus conditions, what type of t-test should we most likely use?\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your plan for the analysis as a markdown cell.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 2
\n", " To begin we want to read in the data from each participant into a pandas data frame using `pd.read_csv()` and the used `pd.concat()` to combine them. After you read in the data, inspect the columns and check the data for any inconsistencies that might affect our analysis (e.g., subjects who didn't perform any trials, etc...).\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your code here.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As a reminder the following show a simple for loop (called a list comprehension) that lets you get the filenames for the lab data set. You can use this to find the file names that you need to read in with `pd.read_csv()`. If you need a reminder refer back to your mental rotation lab which should be on your Jupyterhub node still!" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "ename": "FileNotFoundError", "evalue": "[Errno 2] No such file or directory: 'sdt1-data'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;31m# this is an example list comprehension which reads in the all the files.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;31m# the f.startswith() part just gets rid of any junk files in that folder\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mfilenames\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'sdt1-data/'\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mf\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mf\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlistdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'sdt1-data'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'.'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'sdt1-data'" ] } ], "source": [ "# this is an example list comprehension which reads in the all the files.\n", "# the f.startswith() part just gets rid of any junk files in that folder \n", "filenames=['sdt1-data/'+f for f in os.listdir('sdt1-data') if not f.startswith('.')]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Stop and share
\n", " Did you find any interesting or important points about the data that you want to share with the class? Let's get everyone on the same page so we get similar results so if you find something important (good or bad) please take a moment and share!\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 3
\n", " Now that we have the analysis dataframe, let's perform our planned SDT analysis on a single subject. In the cell below, create a new dataframe by subselecting from the main frame with the data from a single subject.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your code here.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 4
\n", " Referring back to the first notebook from the lab, compute the hits and false alarms for this subject for each stimulus present condition. One note about the false alarms: remember that false alarms are trials where the stimulus was absent but the subject said it was present. There is only one type of these false larms trials in the experiment. So when you compute the statistics, you really will create five numbers for this one participant (The hits for each of the stimulus present conditions, and an overall false alarm rate). What are these numbers for this subject? Remember also that if you take the `mean()` of a column of 0/1 values it will tell you the proportion of those trials which are 1.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your code here.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 5
\n", " Ok now compute the d' and c values for this subject. Refer back to the previous notebook for reference if you need the equations and python code. What can you tell from looking at your computed values? If for some reason you can't compute these values in your experiment (e.g., if the subject has no hits or no false alarms) then try question 4 again with a different subject.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your code here.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 6
\n", " Ok, so now you have performed the desired analysis for one subject. You have computed eight numbers for that subject described above. Now we want to repeat this for each subject. There are two ways to do this. One is using the pandas groupby() function. This is somewhat of the advanced mode version but if you do this you can probably perform the analysis in only 5-6 lines of code. An alternative approach is to generalize the code you just wrote to use a for loop that iterates over each subject in the data frame. Which ever way you choose you might find it helpful to refer to the Forty For Loops notebook I provided which shows some example templates for using for loops to analyze individual subject data in pandas dataframes. The end result is that you want a dataframe containing the d' and c' for each subject for each stimulus condition.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your code here.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 7
\n", " Now that you have your summary statistics computed, our next step is to ask if the conditions are different on either the d' or c measure. For now concentrate on the easiest and hardest conditions to see if there is a effect in the most dissimilar conditions. Then compare the hardest and second hardest conditions. Conduct the appropriate t-test to tell if these conditions are different. The chapter reading from this week has all the code you need. However you might find it helpful to refer to the pingouin documentation for the t-test function.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your code here.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 8 (BONUS)
\n", " This question is a bonus but asks you to verify the assumptions of the chosen t-test hold. Usually this involves looking to see that the distribution of scores you are analyzing are roughly \"normally\" distributed. The textbook showed a few example way to check including the histogram, the qqplot, and the Shapiro-Wilks test. Perform this test on your data. If the data seem very non-normal you might consider switching your t-test to the non-parametric alternative discussed in the chapter (the wilcoxon or Mahn-Whitney test).\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your code here.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 9
\n", " Ok great job! You have your data, you have your t-test, you have everything you need to answer the following questions: 1. Is performance higher in the low number of dots or high number of dots condition? How do you know? How would you report the statistical evidence in favor of your conclusion in a paper? Is the criterion different between the conditions? How do you know? Overall was performance high or low in this task?\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your Markdown text here.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Question 10
\n", " The final question asks you to consider limitations to this experiment design. What other analyses might be interesting to perform on this data? If you could change the experiment is there another type of question you would ask?\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " Your Answer
\n", " Enter your Markdown text here.\n", "
" ] } ], "metadata": { "celltoolbar": "Tags", "kernel_info": { "name": "python3" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" }, "nteract": { "version": "0.15.0" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 1 }