Hanna Shevchikova introduces parallel computing in R, covering two ways to split work: by tasks and by data. Most problems use data partitioning, where the same operation runs on different chunks. The course covers the parallel package, foreach, future.apply, and reproducibility with random numbers.
Chapters
Hello and welcome to the course on Parallel Computing in R. My name is Hanna Shevchikova and I'm a senior research scientist at the University of Washington. In this course, I assume that you are familiar with the concepts covered in the Writing Efficient R Code course here on datacamp and that you already optimized your sequential code, you know how to benchmark it, and you are ready to break it into multiple pieces that can run in parallel, load balanced and in a reproducible manner. We start with showing you different ways of splitting problems into pieces and what different methods and R packages are available for running your code in parallel. In chapter two we will go into more detail of the core package called Parallel. In chapter three we will talk about two user contributed packages that make parallel programming even easier, namely Foreach and Future Apply. The course concludes by paying attention to an important and non trivial subject of using random numbers in parallel environment and reproducibility. Also in the last chapter we will put all the concepts together in form of an example. How do you split a computation problem into parallel chunks? There are two ways you can approach it. First, you can partition it by tasks where different tasks can be performed independently. Think of building a house. There are many tasks you can perform in parallel, e.g. plumbing, electrical, installing windows, etc. In the modeling world, say a demographic model, birds, deaths and migration can be modeled independently, thus in parallel providing inputs to a population model. The second form of partitioning is splitting a computation problem by data. In this case, the same task is performed on different chunks of data in the house. Example, all windows could be installed in parallel. In computing tasks, you can, for example, compute a sum of each row of a matrix. In parallel, you have the same task which is the sum applied to different data, which are the rows. In summary, if you partition a problem by task, it means different tasks are applied to the same or different data. If you partition by data, the same task is performed on different data. This way of partitioning is much more common and therefore it will be the focus of our course. Here is another example of partitioning by data. If you have a sequence of operations, say 1, 2, 3 and so on until hundred, you can split it into multiple sums, where each of the partial sums operates on a subset of the whole sequence and therefore is independent of the others. Thus, the same task, namely the sum, is evaluated on different parts of the data. If you have a large number of such independent tasks that have low or no communication needs, such applications are often called embarrassingly parallel, meaning it's embarrassing how easy it is to parallelize. Many statistical simulations belong to this category. They usually have the following structure here in a pseudocode, first a random number generator is initialized. Then a loop is constructed in which the same function here myfunc is called, usually on different data. Often in such applications, the data is generated as draws from some probability distribution, or it may be simply a subset of a bigger dataset. After collecting results from each iteration, the results are processed and, for example written to the disk, or a visualization is created. Let's look at some real examples.
