Unit 4.4.4 Parallelizing multiple loops
¶In Section 4.3, we only considered parallelizing one loop at a time. When one has a processor with many cores, and hence has to use many threads, it may become beneficial to parallelize multiple loops. The reason is that there is only so much parallelism to be had in any one of the \(m \text{,}\) \(n\text{,}\) or \(k \) sizes. At some point, computing with matrices that are small in some dimension starts affecting the ability to amortize the cost of moving data.
OpenMP allows for "nested parallelism." Thus, one possibility would be to read up on how OpenMP allows one to control how many threads to use in a parallel section, and to then use that to achieve parallelism from multiple loops. Another possibility is to create a parallel section (with #pragma omp parallel rather than #pragma omp parallel for) when one first enters the "five loops around the micro-kernel," and to then explicitly control which thread does what work at appropriate points in the nested loops.
In [25]
Tyler M. Smith, Robert van de Geijn, Mikhail Smelyanskiy, Jeff R. Hammond, and Field G. Van Zee, Anatomy of High-Performance Many-Threaded Matrix Multiplication, in the proceedings of the 28th IEEE International Parallel & Distributed Processing Symposium (IPDPS 2014), 2014.
the parallelization of various loops is discussed, as is the need to parallelize muiltiple loops when targeting "many-core" architectures like the Intel Xeon Phi (KNC) and IBM PowerPC A2 processors. The Intel Xeon Phi has 60 cores, each of which has to execute four "hyperthreads" in order to achieve the best performance. Thus, the implementation has to exploit 240 threads...