Purrr on rdata.lu Blog | Data science with R

Skip errors in R loops by not writing loops

Thu, 21 Dec 2017 00:00:00 +0000

You probably have encountered situations similar to this one:

result = vector("list", length(some_numbers))

for(i in seq_along(some_numbers)){
  result[[i]] = some_function(some_numbers[[i]])
}

print(result)

First I initialize result, an empty list of size equal to the length of some_numbers which will contains the results of applying some_function() to each element of some_numbers. Then, using a for loop, I apply the function. This is what I get back:

NaNs producedError in sqrt(x) : non-numeric argument to mathematical function

Let’s take a look at some_numbers and some_function():

print(some_numbers)

## [[1]]
## [1] -1.9
## 
## [[2]]
## [1] 20
## 
## [[3]]
## [1] "-88"
## 
## [[4]]
## [1] -42

some_function

## function(x){
##   if(x == 0) res = 0
##   if(x < 0) res = -sqrt(-x)
##   if(x > 0) res = sqrt(x)
##   return(res)
## }

So the function simply returns the square root of x (or minus the square root of -x if x is negative), but the number in third position of the list some_numbers is actually a character. This type of mistakes can commonly happen. The result list looks like this:

print(result)

[[1]]
[1] -1.378405

[[2]]
[1] 4.472136

[[3]]
NULL

[[4]]
NULL

As you see, even though the fourth element could have been computed, the error made the whole loop stop. In such a simple example, you could correct this and then run your function. But what if the list you want to apply your function to is very long and the computation take a very, very long time? Perhaps you simply want to skip these errors and get back to them later. One way of doing that is using tryCatch():

result = vector("list", length(some_numbers))

for(i in seq_along(some_numbers)){
  result[[i]] = tryCatch(some_function(some_numbers[[i]]), 
                         error = function(e) paste("something wrong here"))
}

print(result)

## [[1]]
## [1] -1.378405
## 
## [[2]]
## [1] 4.472136
## 
## [[3]]
## [1] "something wrong here"
## 
## [[4]]
## [1] -6.480741

This works, but it’s verbose and easy to mess up. My advice here is that if you want to skip errors in loops you don’t write loops! This is quite easy with the purrr package:

library(purrr)

result = map(some_numbers, some_function)

There’s several advantages here already; no need to initialize an empty structure to hold your result, and no need to think about indices, which can sometimes get confusing. This however does not work either; there’s still the problem that we have a character inside some_numbers:

Error in sqrt(x) : non-numeric argument to mathematical function

However, purrr contains some very amazing functions for error handling, safely() and possibly(). Let’s try possibly() first:

possibly_some_function = possibly(some_function, otherwise = "something wrong here")

possibly() takes a function as argument as well as otherwise; this is where you define a return value in case something is wrong. possibly() then returns a new function that skips errors:

result = map(some_numbers, possibly_some_function)

print(result)

## [[1]]
## [1] -1.378405
## 
## [[2]]
## [1] 4.472136
## 
## [[3]]
## [1] "something wrong here"
## 
## [[4]]
## [1] -6.480741

When you use possibly() on a function, you’re politely telling R “would you kindly apply the function wherever possible, and if not, tell me where there was an issue”. What about safely()?

safely_some_function = safely(some_function)


result = map(some_numbers, safely_some_function)

str(result)

## List of 4
##  $ :List of 2
##   ..$ result: num -1.38
##   ..$ error : NULL
##  $ :List of 2
##   ..$ result: num 4.47
##   ..$ error : NULL
##  $ :List of 2
##   ..$ result: NULL
##   ..$ error :List of 2
##   .. ..$ message: chr "invalid argument to unary operator"
##   .. ..$ call   : language -x
##   .. ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"
##  $ :List of 2
##   ..$ result: num -6.48
##   ..$ error : NULL

The major difference with possibly() is that safely() returns a more complex object: it returns a list of lists. There are as many lists as there are elements in some_numbers. Let’s take a look at the first one:

print(result[[1]])

## $result
## [1] -1.378405
## 
## $error
## NULL

result[[1]] is a list with a result and an error. If there was no error, we get a value in result and NULL in error. If there was an error, this is what we see:

print(result[[3]])

## $result
## NULL
## 
## $error
## <simpleError in -x: invalid argument to unary operator>

Because lists of lists are not easy to handle, I like to use possibly(), but if you use safely() you might want to know about transpose(), which is another function from purrr:

result2 = transpose(result)

str(result2)

## List of 2
##  $ result:List of 4
##   ..$ : num -1.38
##   ..$ : num 4.47
##   ..$ : NULL
##   ..$ : num -6.48
##  $ error :List of 4
##   ..$ : NULL
##   ..$ : NULL
##   ..$ :List of 2
##   .. ..$ message: chr "invalid argument to unary operator"
##   .. ..$ call   : language -x
##   .. ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"
##   ..$ : NULL

result2 is now a list of two lists: a result list holding all the results, and an error list holding all the error message. You can get results with:

result2$result

## [[1]]
## [1] -1.378405
## 
## [[2]]
## [1] 4.472136
## 
## [[3]]
## NULL
## 
## [[4]]
## [1] -6.480741

I hope you enjoyed this blog post, and that these functions will make your life easier!

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Functional peace of mind

Tue, 14 Nov 2017 00:00:00 +0000

I think what I enjoy the most about functional programming is the peace of mind that comes with it. With functional programming, there’s a lot of stuff you don’t need to think about. You can write functions that are general enough so that they solve a variety of problems. For example, imagine for a second that R does not have the sum() function anymore. If you want to compute the sum of, say, the first 100 integers, you could write a loop that would do that for you:

numbers = 0

for (i in 1:100){
  numbers = numbers + i
}

print(numbers)

## [1] 5050

The problem with this approach, is that you cannot reuse any of the code there, even if you put it inside a function. For instance, what if you want to merge 4 datasets together? You would need something like this:

library(dplyr)
data(mtcars)

mtcars1 = mtcars %>%
  mutate(id = "1")

mtcars2 = mtcars %>%
  mutate(id = "2")

mtcars3 = mtcars %>%
  mutate(id = "3")

mtcars4 = mtcars %>%
  mutate(id = "4")

datasets = list(mtcars1, mtcars2, mtcars3, mtcars4)

temp = datasets[[1]]

for(i in 1:3){
  temp = full_join(temp, datasets[[i+1]])
}

## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")
## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")
## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")

glimpse(temp)

## Observations: 128
## Variables: 12
## $ mpg  <dbl> 21.0, 21.0, 22.8, 21.4, 18.7, 18.1, 14.3, 24.4, 22.8, 19....
## $ cyl  <dbl> 6, 6, 4, 6, 8, 6, 8, 4, 4, 6, 6, 8, 8, 8, 8, 8, 8, 4, 4, ...
## $ disp <dbl> 160.0, 160.0, 108.0, 258.0, 360.0, 225.0, 360.0, 146.7, 1...
## $ hp   <dbl> 110, 110, 93, 110, 175, 105, 245, 62, 95, 123, 123, 180, ...
## $ drat <dbl> 3.90, 3.90, 3.85, 3.08, 3.15, 2.76, 3.21, 3.69, 3.92, 3.9...
## $ wt   <dbl> 2.620, 2.875, 2.320, 3.215, 3.440, 3.460, 3.570, 3.190, 3...
## $ qsec <dbl> 16.46, 17.02, 18.61, 19.44, 17.02, 20.22, 15.84, 20.00, 2...
## $ vs   <dbl> 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, ...
## $ am   <dbl> 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, ...
## $ gear <dbl> 4, 4, 4, 3, 3, 3, 3, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 4, 4, ...
## $ carb <dbl> 4, 4, 1, 1, 2, 1, 4, 2, 2, 4, 4, 3, 3, 3, 4, 4, 4, 1, 2, ...
## $ id   <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1...

Of course, the logic is very similar as before, but you need to think carefully about the structure holding your elements (which can be numbers, datasets, characters, etc…) as well as be careful about indexing correctly… and depending on the type of objects you are working on, you might need to tweak the code further.

How would a functional programming approach make this easier? Of course, you could use purrr::reduce() to solve these problems. However, since I assumed that sum() does not exist, I will also assume that purrr::reduce() does not exist either and write my own, clumsy implementation. Here’s the code:

my_reduce = function(a_list, a_func, init = NULL, ...){

  if(is.null(init)){
    init = `[[`(a_list, 1)
    a_list = tail(a_list, -1)
  }

  car = `[[`(a_list, 1)
  cdr = tail(a_list, -1)
  init = a_func(init, car, ...)

  if(length(cdr) != 0){
    my_reduce(cdr, a_func, init, ...)
  }
  else {
    init
  }
}

This can look much more complicated than before, but the idea is quite simple; if you know about recursive functions (recursive functions are functions that call themselves). I won’t explain how the function works, because it is not the main point of the article (but if you’re curious, I encourage you to play around with it). The point is that now, I can do the following:

my_reduce(list(1,2,3,4,5), `+`)

## [1] 15

my_reduce(datasets, full_join) %>% glimpse

## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")
## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")
## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")

## Observations: 128
## Variables: 12
## $ mpg  <dbl> 21.0, 21.0, 22.8, 21.4, 18.7, 18.1, 14.3, 24.4, 22.8, 19....
## $ cyl  <dbl> 6, 6, 4, 6, 8, 6, 8, 4, 4, 6, 6, 8, 8, 8, 8, 8, 8, 4, 4, ...
## $ disp <dbl> 160.0, 160.0, 108.0, 258.0, 360.0, 225.0, 360.0, 146.7, 1...
## $ hp   <dbl> 110, 110, 93, 110, 175, 105, 245, 62, 95, 123, 123, 180, ...
## $ drat <dbl> 3.90, 3.90, 3.85, 3.08, 3.15, 2.76, 3.21, 3.69, 3.92, 3.9...
## $ wt   <dbl> 2.620, 2.875, 2.320, 3.215, 3.440, 3.460, 3.570, 3.190, 3...
## $ qsec <dbl> 16.46, 17.02, 18.61, 19.44, 17.02, 20.22, 15.84, 20.00, 2...
## $ vs   <dbl> 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, ...
## $ am   <dbl> 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, ...
## $ gear <dbl> 4, 4, 4, 3, 3, 3, 3, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 4, 4, ...
## $ carb <dbl> 4, 4, 1, 1, 2, 1, 4, 2, 2, 4, 4, 3, 3, 3, 4, 4, 4, 1, 2, ...
## $ id   <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1...

And if I need to merge another dataset, I don’t need to change anything at all. Plus, because my_reduce() is very general, I can even use it for situation I didn’t write it for in the first place:

my_reduce(list("a", "b", "c", "d", "e"), paste)

## [1] "a b c d e"

Of course, paste() is vectorized, so you could just as well do paste(1, 2, 3, 4, 5), but again, I want to insist on the fact that writing or using such functions allows you to abstract over a lot of thing. There is nothing specific to any type of object in my_reduce(), whereas the loops have to be tailored for the kind of object you’re working with. As long as the a_func argument is a binary operator that combines the elements inside a_list, it’s going to work. And I don’t need to think about indexing, about having temporary variables or thinking about the structure that will hold my results.

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Let's make ggplot2 purrr again

Mon, 09 Oct 2017 06:45:48 +0200

Update: I’ve included another way of saving a separate plot by group in this article, as pointed out by @monitus. Actually, this is the preferred solution; using dplyr::do() is deprecated, according to Hadley Wickham himself.

I’ll be honest: the title is a bit misleading. I will not use purrr that much in this blog post. Actually, I will use one single purrr function, at the very end. I use dplyr much more. However Make ggplot2 purrr sounds better than Make ggplot dplyr or whatever the verb for dplyr would be.

Also, this blog post was inspired by a stackoverflow question and in particular one of the answers. So I don’t bring anything new to the table, but I found this stackoverflow answer so useful and so underrated (only 16 upvotes as I’m writing this!) that I wanted to write something about it.

Basically the idea of this blog post is to show how to create graphs using ggplot2, but by grouping by a factor variable beforehand. To illustrate this idea, let’s use the data from the Penn World Tables 9.0. The easiest way to get this data is to install the package called pwt9 with:

install.packages("pwt9")

and then load the data with:

data("pwt9.0")

Now, let’s load the needed packages. I am also using ggthemes which makes themeing your ggplots very easy. I’ll be making Tufte-style plots.

library(ggplot2)
library(ggthemes)
library(dplyr)
library(tidyr)
library(purrr)
library(pwt9)

First let’s select a list of countries:

country_list <- c("France", "Germany", "United States of America", "Luxembourg", "Switzerland", "Greece")

small_pwt <- pwt9.0 %>%
  filter(country %in% country_list)

Let’s us also order the countries in the data frame as I have written them in country_list:

small_pwt <- small_pwt %>%
  mutate(country = factor(country, levels = country_list, ordered = TRUE))

You might be wondering why this is important. At the end of the article, we are going to save the plots to disk. If we do not re-order the countries inside the data frame as in country_list, the name of the files will not correspond to the correct plots!

Update: While this can still be interesting to know, especially if you want to order the bars of a barplot made with ggplot2, I included a suggestion by @expersso that does not require your data to be ordered!

Now when you want to plot the same variable by countries, say avh (Average annual hours worked by persons engaged), the usual way to do this is with one of facet_wrap() or facet_grid():

ggplot(data = small_pwt) + theme_tufte() +
  geom_line(aes(y = avh, x = year)) +
  facet_wrap(~country)

ggplot(data = small_pwt) + theme_tufte() +
  geom_line(aes(y = avh, x = year)) +
  facet_grid(country~.)

As you can see, for this particular example, facet_grid() is not very useful, but do notice its argument, country~., which is different from facet_wrap()’s argument. This way, I get the graphs stacked horizontally. If I had used facet_grid(~country) the graphs would be side by side and completely unreadable.

Now, let’s go to the meat of this post: what if you would like to have one single graph for each country? You’d probably think of using dplyr::group_by() to form the groups and then the graphs. This is the way to go, but you also have to use dplyr::do(). This is because as far as I understand, ggplot2 is not dplyr-aware, and using an arbitrary function with groups is only possible with dplyr::do().

Update: As explained in the intro above, I also added the solution that uses tidyr::nest():

# Ancient, deprecated way of doing this
plots <- small_pwt %>%
  group_by(country) %>%
  do(plot = ggplot(data = .) + theme_tufte() +
       geom_line(aes(y = avh, x = year)) +
       ggtitle(unique(.$country)) +
       ylab("Year") +
       xlab("Average annual hours worked by persons engaged"))

And this is the approach that uses tidyr::nest():

# Preferred approach
plots <- small_pwt %>%
  group_by(country) %>%
  nest() %>%
  mutate(plot = map2(data, country, ~ggplot(data = .x) + theme_tufte() +
       geom_line(aes(y = avh, x = year)) +
       ggtitle(.y) +
       ylab("Year") +
       xlab("Average annual hours worked by persons engaged")))

If you know dplyr at least a little bit, the above lines should be easy for you to understand. But notice how we get the title of the graphs, with ggtitle(unique(.$country)), which was actually the point of the stackoverflow question.

Update: The modern version uses tidyr::nest(). Its documentation tells us:

There are many possible ways one could choose to nest columns inside a data frame. nest() creates a list of data frames containing all the nested variables: this seems to be the most useful form in practice. Let’s take a closer look at what it does exactly:

small_pwt %>%
  group_by(country) %>%
  nest() %>%
  head()

## # A tibble: 6 x 2
##   country                  data              
##   <ord>                    <list>            
## 1 Switzerland              <tibble [65 × 46]>
## 2 Germany                  <tibble [65 × 46]>
## 3 France                   <tibble [65 × 46]>
## 4 Greece                   <tibble [65 × 46]>
## 5 Luxembourg               <tibble [65 × 46]>
## 6 United States of America <tibble [65 × 46]>

This is why I love lists in R; we get a tibble where each element of the column data is itself a tibble. We can now apply any function that we know works on lists.

What might be surprising though, is the object that is created by this code. Let’s take a look at plots:

print(plots)

## # A tibble: 6 x 3
##   country                  data               plot    
##   <ord>                    <list>             <list>  
## 1 Switzerland              <tibble [65 × 46]> <S3: gg>
## 2 Germany                  <tibble [65 × 46]> <S3: gg>
## 3 France                   <tibble [65 × 46]> <S3: gg>
## 4 Greece                   <tibble [65 × 46]> <S3: gg>
## 5 Luxembourg               <tibble [65 × 46]> <S3: gg>
## 6 United States of America <tibble [65 × 46]> <S3: gg>

As dplyr::do()’s documentation tells us, the return values get stored inside a list. And this is exactly what we get back; a list of plots! Lists are a very flexible and useful class, and you cannot spell list without purrr (at least not when you’re a neRd).

Here are the final lines that use purrr::map2() to save all these plots at once inside your working directory:

Update: I have changed the code below which does not require your data frame to be ordered according to the variable country_list.

# file_names <- paste0(country_list, ".pdf")

map2(paste0(plots$country, ".pdf"), plots$plot, ggsave)

As I said before, if you do not re-order the countries inside the data frame, the names of the files and the plots will not match. Try running all the code without re-ordering, you’ll see!

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!