Are you studying for a (like Amdahl's law calculations)? Do you need help writing MPI or OpenMP code for a project? Are you analyzing a specific parallel algorithm ?
Michael J. Quinn’s "Parallel Computing: Theory and Practice" (1994) bridges abstract PRAM modeling with real-world MIMD architectures to address parallel algorithm design. The text emphasizes performance metrics like Amdahl’s Law and provides strategies for algorithms in scientific simulations and data processing. Access a copy of the book on Internet Archive Parallel Computing: Theory and Practice: Quinn, Michael J.
To understand why you need the PDF, consider two of Quinn’s most cited theoretical frameworks:
Nodes compute independently using the latest available data, reducing idle waiting times at the cost of complex convergence verification. Implementation and Programming Environments
States that the sequential fraction of a program ( Are you studying for a (like Amdahl's law calculations)
Assigning the agglomerated tasks to physical processors or threads to balance the computational load. 4. Performance Metrics and Analytical Models
: Processor arrays, multiprocessors, and multicomputers.
Larger problems allow parallel components to dominate execution time. Diminishing returns as processor count increases. Constant or expanding efficiency with workload scale. Quantifying Performance Metrics
Training Large Language Models (LLMs) requires splitting neural network weights across multiple GPUs (tensor parallelism and pipeline parallelism). Optimizing these pipelines requires understanding the exact interconnection network constraints and latency bottlenecks analyzed by Quinn. Michael J
The book provides a solid grounding in measuring success using metrics like speedup, efficiency, and overhead, using laws such as Amdahl's Law to explain the theoretical limits of parallel performance.
: Concurrent Read, Concurrent Write. Requires collision resolution hardware. Performance Metrics
Demonstrates the concept of recursive parallel task creation, where a master processor divides the array, hands halves off to worker threads, and merges the results.
Modern supercomputers are almost universally hybrid. They consist of clusters of distributed nodes. Inside each individual node, multiple multi-core CPUs and GPU accelerators share a local memory pool. 4. Interconnection Networks Access a copy of the book on Internet
: Most institutional libraries provide access through physical lending or internal digital networks.
: One of the most practical sections covers eight specific strategies for developing parallel algorithms, moving beyond simple "trial and error". Core Topics Covered
For distributed memory systems, MPI is the gold standard. It requires programmers to explicitly manage data movement between nodes.