R Tutorial: Models of parallel computing

Now, when you know how to break code into independent pieces, you need to pay attention to the available hardware and the best strategy for running your application in parallel. The most important criterion is the central processing unit or cpu. In order to run your code in parallel, you need at least two on ideally more processors or cores. And those can either live on a single computer or it can be a cluster of single or multicore computers connected via a network. The second most important aspect is the memory. Take the concept of a multicore machine. Such systems have memory or RAM that all processes can access and communicate through, and is therefore called shared memory. An advantage of sharing memory is that processes can share variables and data, which often makes applications more time efficient. This is handled by shared memory software. A system with distributed memory, on the other hand, consists of multiple machines. Think of a network of workstations, each of which has its own memory. Here, a process cannot access the memory of another process. Such systems are often called message passing systems because processes communicate via sending messages to one another. The advantage of message passing software is that it can run on both systems, distributed as well as shared memory, as you will see throughout this course, which makes applications independent of the underlying hardware. This is not the case with shared memory software. Let's now turn our attention to the software side where we will look at two different programming paradigms. One is the master worker model, sometimes called the master slave model. The other is the MapReduce model. The MapReduce paradigm emerged from the need to develop generic applications that could also run on distributed data, I.e. data physically distributed on different devices. Hadoop and Spark are the main players in this area. In this course we will not cover the MapReduce paradigm. I recommend the course on scalable data processing in R if you are interested in this approach. Instead, you will learn about the master worker model, which is simple yet very powerful. Remember our embarrassingly parallel for loop pseudocode simulation? All ncalls to myfunc can be run in parallel. However, in reality N is often much larger than the number of available processors. The master worker model can deal with that problem. There is one process called master which creates a set of other processes to workers and distributes tasks among them. Workers perform their tasks and return results to the master process, which in turn performs the final processing. The master worker model is well suited for embarrassingly parallel applications. Let's look at concrete examples of such applications.