Concatenating a list of data frames
It’s something that I do surprisingly often: concatenating a list of data frames into a single (possibly quite enormous) data frame. Until now my naive solution worked pretty well. However, today I needed to deal with a list of over 6 million elements. The result was hours of page thrashing before my R session finally surrendered. I suppose I should be happy that my hard disk survived.
I did a bit of research and found that there are a few solutions which are much (much!) more efficient.
Let’s create some test data: a list consisting of 100 000 elements, each of which is a small data frame.
The Naive Solution
My naive solution to the problem was to use a combination of
rbind(). It gets the job done.
Alternative Solutions #1 and #2
The plyr package presents two options.
Both of these also do the job nicely.
Alternative Solution #3
The revised package dplyr provides some alternative solutions.
A second function,
rbind_list(), takes individual elements to be concatenated as arguments (rather than a single list).
bind_rows() will concatenate data frames, matching columns by name.
Alternative Solution #4
Finally, a solution from the data.table package.
All of these alternatives produce the correct result. The solution of choice will be the fastest one (and the one causing the minimum of page thrashing!).
Thoughts on Performance
The naive solution uses the
rbind.data.frame() method which is slow because it checks that the columns in the various data frames match by name and, if they don’t, will re-arrange them accordingly.
rbindlist(), by contrast, does not perform such checks and matches columns by position.
rbindlist() is implemented in C, while
rbind.data.frame() is coded in R.
In the most recent version of data.table (1.9.3, currently available from r-forge),
rbindlist() has two new arguments. One of them, use.names, forces
rbindlist() to match column names and so works more like
rbind.data.frame(), but is coded in C so it is more efficient. Another related argument, fill, causing missing columns to be filled with NA.
Both of the plyr solutions are an improvement on the naive solution. However, the dplyr solution is better than either of them. Relative to all of the other solutions,
rbindlist() is far superior. It’s blisteringly fast. Little wonder that my naive solution bombed out with a list of 6 million data frames. Using
rbindlist(), however, it was done before I had finished my cup of coffee.
Thanks to the various folk who provided feedback, which was used to expand and improve this post.