Worksheet 3

Questions are below. My solutions will be available after the tutorials are all finished. The whole point of these worksheets is for you to use your lecture notes to figure out what to do. In tutorial, the TAs are available to guide you if you get stuck. Once you have figured out how to do this worksheet, you will be prepared to tackle the assignment that depends on it.

If you are not able to finish in an hour, I encourage you to continue later with what you were unable to finish in tutorial.

Questions are respectively on choosing rows and columns, joins, and the one-sample \(t\). There is a lot here, so you should plan to work at a good pace. You don’t have to do these three questions in order. If you want practice on a certain thing, start with that one.

Student stats

A survey of students at the University of California Davis asked respondents:

• the gender they identify as (called Sex in the data set)
• hours of TV they watched in the last week
• hours they spent doing non-academic things on a computer in the last week (playing games, on social media etc. This survey was taken some years ago, and so pre-dates smartphones.)
• how many hours of sleep they got last night
• where they prefer to sit in their classes (front, middle, back of classroom)
• how many alcoholic drinks they consumed in the last week (defined as US standard drinks)
• their height (inches)
• their mother’s height (inches)
• their father’s height (inches)
• the number of hours of exercise they got in the last week
• their GPA
• their area of study, classified as Liberal Arts or NonLib (“not arts” meaning science, math, engineering etc).

Students were allowed to not respond to any of the items, so there are some missing values, labelled NA in the dataframe you are about to read in.

The data are in http://ritsokiguess.site/datafiles/UCDavis1.csv.

1. Read in and display (some of) the data.

Solution

As usual:

my_url <- "http://ritsokiguess.site/datafiles/UCDavis1.csv"
davis <- read_csv(my_url)

Rows: 173 Columns: 12
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): Sex, Seat, class
dbl (9): TV, computer, Sleep, alcohol, Height, momheight, dadheight, exercis...

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

davis

Give the dataframe a suitable name. I called it davis, because it is a bunch of information about students at University of California-Davis.

\(\blacksquare\)

2. Find the mean and standard deviation of sleep times (for all the students taken together).

Solution

A summarize. Use your name for the dataframe, if it is different from mine:

davis %>% summarize(mean_sleep = mean(Sleep), sd_sleep = sd(Sleep))

You can use whatever names you like for the summaries. My recent habit is to use longer and more descriptive names for the summaries than I did in the past (like, when I wrote the lecture notes).

\(\blacksquare\)

3. For the students that sit in each part of the classroom (separately), find the number of students and their mean sleep time. (Hint: the column is called Seat with a capital S; the S being uppercase matters.)

Solution

This is a group-by and summarize, grouping by the categorical variable Seat first, and then computing the required summaries. We are finding something else other than the counts (the mean sleep time), so we have to use the n() idea here:

# summary(davis)
davis %>% 
  group_by(Seat) %>% 
  summarize(n_student = n(), mean_sleep = mean(Sleep))

There were two students who didn’t say where they liked to sit, so their Seat is missing.

Extra:

There seems to be a tendency for students that sit nearer the front of the class to sleep less. This is a bit hard to read from this table, because the values of Seat come out in alphabetical order, rather than a more easily interpretable order like “front, middle, back”.¹

Having seen this, a graph might give us a hint about whether this is a real trend or more likely just chance:

ggplot(davis, aes(x = Seat, y = Sleep)) + geom_boxplot()

The median sleep times (for the students with non-missing Seat) seem to be exactly equal, and there is a lot of variability, so the trend we saw is probably just chance.

\(\blacksquare\)

4. For each of the students this data set, display how much time they have spent watching TV and on the computer.

Solution

These two things are columns in the dataframe, and choosing columns is select:

davis %>% select(TV, computer)

It is rather annoying that the column computer has a lowercase C, and this matters as far as R is concerned.²

\(\blacksquare\)

5. Several of the columns are heights. Display all of these columns, without naming any columns.

Solution

The first thing is to find out exactly what it is that these height column names have in common. They are Height, momheight, and dadheight. (They are the student’s height and the heights of their parents.) What they have in common is that they end with height:

davis %>% select(ends_with("height"))

Given what we were just saying about uppercase and lowercase, you might be surprised that this worked: after all, the one that is the student’s height starts with an uppercase H, which does not match what I put in the ends_width. It turns out that select-helpers are not case-sensitive, unless you make them so:

davis %>% select(ends_with("height", ignore.case = FALSE))

The option is the rather cumbersome double negative “don’t ignore case”, ie. pay attention to the case. This gets only the two columns whose names literally end in height, not the student’s height whose name is (ends with) Height with a capital H, and this no longer matches because we are paying attention to uppercase/lowercase.

Another way to get all three columns, if you are worried about upper and lower case, is to match on the ends of the names that are the same including case:

davis %>% select(ends_with("eight"))

Fortunately, there are no columns ending in weight here, so this gets only the columns we want.

Or you could say that these three columns are the only ones that contain height, which is also true:

davis %>% select(contains("height"))

or if you really want to grapple with regular expressions, even this:

davis %>% select(matches("height$"))

Inside matches, my regular expression searches for columns that match with the letters height at the end of the column name, so that eg height_in_cm wouldn’t match.³

\(\blacksquare\)

6. Display only the students who sit at the back of the classroom.

Solution

This is choosing rows, specifically the rows for which Seat is Back, so it needs filter:

davis %>% filter(Seat == "Back")

Don’t forget the two equals signs, which is how R tests that something is equal to something else: “give me all the rows for which it is true that Seat is equal to Back”.

Confusing aside: Also, Back is a literal piece of text, so it has to be in quotes; filter(Seat == Back) would mean “give me all the rows for which the column Seat is equal to whatever is stored in the variable Back. Can you make sense of this?

Back <- "Front"
davis %>% filter(Seat == Back) 

By the way, because this is so confusing, don’t do something like what I just did! Your work colleagues will hate it, and when you come back to your code in six months, you will hate yourself!

End of confusing aside.

\(\blacksquare\)

7. Display all the students that either slept 5 hours or less or watched over 30 hours of TV (or both).

Solution

This is either/or, so needs |:

davis %>% filter(Sleep <= 5 | TV > 30)

To be precise, “5 hours or less” is less than or equal, and “over” is strictly greater than.

To convince yourself that you have the right ones, scroll through your results and say why each of the students is there. My first ten students are all there because they slept 5 hours or less (the 8th one also watched over 30 hours of TV). In the next ten, the first student got more than 5 hours of sleep but is there because they (claimed to have) watched 100 hours of TV. Students 24 and 25 got lots of sleep but also watched lots of TV. And so on.

\(\blacksquare\)

8. For the students who are 70 or more inches tall, what are the mean heights of their mother and their father? Hint: the first time you do this, your answers will probably be missing. Why is that? Hint: to get the answers you were expecting, search for the na.rm option. What does that do?

Solution

In this kind of problem, where we want summaries for only some of the observations, the first step is to select the observations we want, and the second step is to summarize the observations we selected:

davis %>% filter(Height >= 70) %>% 
  summarize(mom_mean = mean(momheight), dad_mean = mean(dadheight))

OK, so those were both missing. To find out why, step back to the filter (run it without the summarize):

davis %>% filter(Height >= 70) 

Page down almost to the end to see that the 40th student didn’t give an answer for their mother’s and father’s heights. As far as R is concerned, the mean of a set of numbers including a missing value is itself missing, on the basis that the missing value could be anything.

Here, it seems reasonable to calculate the means of the non-missing parental heights.⁴ There are a couple of ways to do that, which you probably haven’t seen before. The first includes the missing values, but removes them when calculating the mean:⁵

davis %>% filter(Height >= 70) %>% 
  summarize(mom_mean = mean(momheight, na.rm = TRUE), dad_mean = mean(dadheight, na.rm = TRUE))

When searching for an R function or option, it usually helps to search with r first. When I searched for “r na.rm”, I got several hits, any of which would be good. For example, https://www.statology.org/na-rm/ says clearly how to calculate the mean without including any missing values, which is exactly what we want to do.

A second way removes the missing momheights and dadheights first before trying to calculate the means:

davis %>% filter(Height >= 70) %>% 
  drop_na(momheight, dadheight) %>% 
  summarize(mom_mean = mean(momheight), dad_mean = mean(dadheight))

To search for this kind of approach, you can try something like “r omit rows with missing values”. The third answer here gives the Tidyverse answer (the one you want). You’ll see a lot of discussion of the old-fashioned na.omit, but the fourth example here shows the “tidy” way to do it.

The answer, both ways, is the same. If you run the first two lines of this (and not the third one), you’ll see that there are now only 41 students chosen, and the one with the missing momheight and dadheight is no longer there. So when you run mean, you will not run into any missing values.

Extras:

Extra 1: when you feed drop_na more than one column, it removes any rows that contain missing values in any of the columns. For example, consider this dataframe:

d

What does this do?

d %>% drop_na(y, z)

There is only the first row left, because the second row has a missing value for y and the third has a missing value for z. The values 5 for z and 7 for y, even though they are good data values, are not included because the rows they were in had missing values for something else. The moral of this story is that if you use drop_na on several columns, you stand to lose quite a lot of your data. The default for drop_na is to drop rows with any missing values at all. The original davis dataframe had 173 rows, but if we drop all the rows with any missing values:

davis %>% drop_na()

we are down to 150 rows, having lost almost 15% of our data. Some people prefer to “impute” missing data, which is to replace missing values with an estimate based on the values of other variables. For example, a student with a missing Height whose parents are both tall is probably tall themselves.

Extra 2: on that note, you would expect taller students to have taller parents. How do the values we calculated above compare with the mean heights of all the parents?

davis %>% 
  summarize(mom_mean = mean(momheight, na.rm = TRUE), dad_mean = mean(dadheight, na.rm = TRUE))

An inch or two less.

Extra 3: You’ll remember group-by and summarize from numerical summaries. Is there a way to use that here? Sort of:

davis %>% 
  group_by(Height >= 70) %>% 
  summarize(n = n(), mom_mean = mean(momheight, na.rm = TRUE), dad_mean = mean(dadheight, na.rm = TRUE))

The answers in the TRUE line are the ones we got before. We are used to using group_by with a categorical column, which we don’t have for this problem, but in fact you can use group_by with anything that makes a categorical, including something that evaluates to TRUE or FALSE. The expression Height >= 70 evaluates to true or false or missing (there were two students who didn’t answer how tall they were). The first line of the result gives the mean parental heights for the students who were less than 70 inches all (most of them), and the last line gives the mean parental heights for the students who didn’t give their own heights.

Note that some of the parental heights were missing, even for students who gave their own heights, so I had to use the na.rm thing again (or I could have used drop_na as before, before calculating the means).

\(\blacksquare\)

Counting seabirds

Each year, bird experts associated with the Kodiak National Wildlife Refuge in Alaska count the number of seabirds of different types on the water in each of four different bays in the area. This is done by drawing (on a map) a number of straight-line “transects”, then driving a boat along each transect and counting the number and type of birds visible within a certain distance of the boat.

Variables of interest are:

• the Year of observation
• the Transect number
• Temp: the temperature
• ObservCond: visibility, from Average up to Ideal
• Bay the name of the bay
• bird: an abbreviation for the species of bird observed
• count: how many of that type of bird were observed (in that year, bay, transect).

The data are in http://ritsokiguess.site/datafiles/seabird_long.csv.

9. Read in and display some of the data.

my_url <- "http://ritsokiguess.site/datafiles/seabird_long.csv"
seabird <- read_csv(my_url)

Rows: 56895 Columns: 9
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (4): ObservCond, Bay, ObservCondFactor3, bird
dbl (5): Year, Site, Transect, Temp, count

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

seabird

There are over 56,000 observations, and the variables named (plus some others which we ignore). The pager displays a maximum of 1000 pages of 10 rows each, so you might think there are exactly 10,000 observations, but there are only this many displayed. The actual number of rows is at the top.

Extra: some more details:

• Transect reference. This talks about measuring plant cover on land, but the transects in our data are on water (so you cannot use a measuring tape!). The idea is that by always using the same transects, measurements are consistent from year to year, and by having a set of transects that cover the study area, you should be observing most or all of what you are trying to observe.
• The original paper that the data came from, with a lot more detail about how the observations were made. The aim of this study was to compare seabird numbers from one year to the next, so using the same transects every year was key. It was not important to estimate the number of seabirds in the entire wildlife refuge, but instead to see how the counts in the places you looked changed over time.

10. A data set containing the full bird names and their principal diet is in http://ritsokiguess.site/datafiles/bird_names.csv. Read in and display some of this data set.

The same again, really:

my_url <- "http://ritsokiguess.site/datafiles/bird_names.csv"
bird_names <- read_csv(my_url)

Rows: 15 Columns: 3
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): bird, bird_name, diet

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

bird_names

There are 15 different bird species observed. Some of them, like the Loon, have subspecies, but they are difficult to tell apart (at least to an observer on a boat looking through binoculars).

11. It is awkward to read the bird abbreviations in the first dataframe. Create and save a new dataframe that has the full bird names as well as the number that were observed and all the other information.

This is using the small dataframe of bird names as a lookup table (as we did with the product codes and product names in lecture). The strategy is to use the lookup table second in a left-join. Note that both dataframes have a column called bird with the bird abbreviations, and that is the only column they have in common. Hence there is no need for a join_by:

seabird %>% left_join(bird_names) -> birds_all

Joining with `by = join_by(bird)`

birds_all

Scroll across to see that you have a column containing the bird’s full name (in bird_name) and also its diet.

If you want to be clear about how the join is happening, you can specify the column to join by explicitly (note that if you don’t specify what you are joining by, left_join will tell you what it joined by, and you can make sure that this is correct):

seabird %>% left_join(bird_names, join_by("bird"))