Guide to Parallel Processing in Python

Pavithra Parthiban -

Parallel processing in computers is like having an efficient team working on different parts of a task simultaneously. In traditional programming, tasks are executed one after the other, like solving a puzzle piece by piece.

However, parallel processing divides the task into smaller chunks, and these chunks are handled simultaneously by multiple processors or cores.

Python provides modules that allow programs to leverage multiple processor cores efficiently. This approach significantly reduces the time it takes to complete tasks and improves the performance of CPU-bound operations.

It's similar to having multiple experts collaborating on different aspects of a complex problem, leading to enhanced efficiency and quicker results.

Lets learn more about parallel processing in this blog!

Table of Contents

  1. Introduction to Parallel Processing
  2. Concurrency and Parallelism
  3. Multiprocessing Module
  4. Parallelism with Queue
  5. The Process Class
  6. The Pool Class

Introduction to Parallel Processing

Python Parallel Processing involves executing multiple tasks simultaneously to improve program performance. In traditional programming, tasks are executed sequentially, one after another.

However, with parallel processing, tasks are divided into smaller subtasks that can be executed concurrently. It's like having several workers to complete a big job faster. In computer terms, it's about using multiple processors or cores to handle tasks concurrently.

  1. Faster Execution: Parallel processing can significantly reduce the time it takes to complete a set of tasks by executing them concurrently.
  2. Improved Performance: Parallelism allows you to make better use of multi-core processors, leading to improved performance for CPU-bound tasks.
  3. Scalability: As the number of processor cores increases, the potential for speedup in parallel processing also increases.

Before getting into parallelism, it is essential to have an understanding of concurrency and parallelism.

Concurrency and Parallelism

Concurrency and parallelism are strategies used to manage tasks in a program efficiently. Concurrency involves making progress on multiple tasks at the same time, but not necessarily simultaneously.

Parallelism, on the other hand, is about executing multiple tasks simultaneously for faster overall performance.

Here is a simple Python code example to understand both concepts:

Concurrency with Threading 

import threading
import time

def task_a():
    for _ in range(5):
        print("Task A")
        time.sleep(1)

def task_b():
    for _ in range(5):
        print("Task B")
        time.sleep(1)

# Using threads for concurrency
thread_a = threading.Thread(target=task_a)
thread_b = threading.Thread(target=task_b)

thread_a.start()
thread_b.start()

thread_a.join()
thread_b.join()

In this example, task_a and task_b are executed concurrently using threads. The time.sleep(1) simulates time-consuming operations.

Parallelism with Multiprocessing 

from multiprocessing import Process
import time

def task_a():
    for _ in range(5):
        print("Task A")
        time.sleep(1)

def task_b():
    for _ in range(5):
        print("Task B")
        time.sleep(1)

# Using processes for parallelism
process_a = Process(target=task_a)
process_b = Process(target=task_b)

process_a.start()
process_b.start()

process_a.join()
process_b.join()

Multiprocessing Module

In the context of Python, the multiprocessing module allows you to run functions in parallel, taking advantage of multiple processor cores.

Imagine you have a bunch of tasks to perform, and you want to get results faster by doing them at the same time instead of one after the other.

Here's a simple example using the multiprocessing module in Python. First let's define a function that squares a number, and then use multiple processes to square a list of numbers.

import multiprocessing

def square_number(x):
    """Simple function to square a number."""
    return x * x

if __name__ == "__main__":
    # List of numbers to square
    numbers = [1, 2, 3, 4, 5]

    # Create a pool of processes
    with multiprocessing.Pool() as pool:
        # Use the pool to apply the function to each number in parallel
        results = pool.map(square_number, numbers)

    # Print the results
    print("Original numbers:", numbers)
    print("Squared numbers:", results)

Lets decode the code for better understanding,

import multiprocessing

This imports the necessary module for parallel processing in Python.

def square_number(x):
    """Simple function to square a number."""
    return x * x

The square_number function takes a number x and returns its square.

with multiprocessing.Pool() as pool:

The Pool() class is used to create a pool of worker processes. The with statement ensures that the resources are properly released after execution.

results = pool.map(square_number, numbers)

The map method applies the square_number function to each element in the numbers list using the pool of processes. This is where parallel execution happens.

Positive Aspects of Utilizing Multiple Processors

  • The multiprocessing module allows to leverage multiple processor cores, resulting in faster execution of tasks.
  • The multiprocessing module enables better utilization of multi-core processors, leading to improved performance.
  • It can handle larger workloads efficiently.
  • It has its own memory space, which helps prevent interference between processes.
  • The module provides convenient abstractions, such as the Pool class, which simplifies the distribution of tasks among multiple processes.

The Process Class

The Process class is the  part of  multiprocessing module in Python. It is a tool that let you create and manage mini-programs. Each mini-program (or process) works on its own, doing different tasks without getting in each other's way.

So, the Process class is like a way to get your computer to do multiple things at once, making it work more efficient.

Let's say you want to count and print numbers and letters. Instead of doing it one after the other, you use the Process class to do both at the same time.

from multiprocessing import Process
import time

def print_numbers():
    for i in range(5):
        time.sleep(1)  # Simulating some work
        print(f"Number: {i}")

def print_letters():
    for char in 'ABCDE':
        time.sleep(1)  # Simulating some work
        print(f"Letter: {char}")

if __name__ == "__main__":
    process1 = Process(target=print_numbers)
    process2 = Process(target=print_letters)

    process1.start()
    process2.start()

    process1.join()
    process2.join()

    print("Both processes are done!")

The Process class helps you do these tasks at the same time, making things quicker and more efficient.

The Pool Class

The Pool class in Python, part of the multiprocessing module, is designed for parallelizing the execution of a function across multiple input values. It manages a pool of worker processes, distributing the workload efficiently among them.

Tasks are automatically distributed among the worker processes managed by the Pool, providing a higher-level abstraction for parallelism. Particularly useful when a large number of similar tasks can be parallelized and distributed efficiently among worker processes.

Suppose you have a list of numbers, and you want to perform a specific operation, such as squaring each number, but you're eager to speed things up by getting some help from your computer's multiple processors.

Imagine each number as a task, and you wish to assign each task to a team of worker processes to work on simultaneously. This is where the Pool class comes into play.

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    # Create a Pool with 3 worker processes
    with Pool(processes=3) as pool:
        # Define a list of input values
        input_values = [1, 2, 3, 4, 5]

        # Use the map function to apply the square function to each input value in parallel
        result = pool.map(square, input_values)

    # Print the result
    print(result)

This code uses the Pool class to efficiently square a list of numbers concurrently. Each worker in the pool handles one number, multiplying it by itself. The result is a list of squared values: [1, 4, 9, 16, 25].

Thus the  multiprocessing module provides a framework for creating and managing parallel processes, and these classes are two key components within that module.

Parallelism with Queue

In parallel processing, the utilization of a queue plays a crucial role in managing concurrent execution of tasks. This concept involves breaking down a larger computational workload into smaller, independent tasks or functions. These tasks are organized in a queue, establishing a structured order for their execution.

Multiple processing units, such as threads or processes, then work concurrently to dequeue tasks from the shared queue and execute them.

The queue serves as a centralized mechanism for coordinating and controlling the flow of tasks, ensuring that they are processed in a synchronized and orderly fashion.

This approach not only enhances the efficiency of task execution but also allows for dynamic load balancing and scalability, as additional tasks can be enqueued as needed.

Here's a simple Python code example illustrating this concept,

import queue
import threading

# Define tasks as functions
def task1():
    print("Task 1 executed")

def task2():
    print("Task 2 executed")

def task3():
    print("Task 3 executed")

# Create a queue for task management
task_queue = queue.Queue()

# Enqueue tasks in the desired order
task_queue.put(task1)
task_queue.put(task2)
task_queue.put(task3)

# Function to execute tasks from the queue
def execute_task():
    while not task_queue.empty():
        current_task = task_queue.get()
        current_task()

# Create and start a thread for task execution
execution_thread = threading.Thread(target=execute_task)
execution_thread.start()

# Wait for the thread to finish
execution_thread.join()

The code ensures synchronization by waiting for the execution thread to finish before proceeding, demonstrating a basic form of task parallelism using a queue in Python.

Conclusion

Parallel processing in Python is like having a team of experts working together to solve a big problem quickly. Traditional programming solves tasks one by one, like solving a puzzle piece by piece.

Parallel processing breaks tasks into smaller pieces, handled at the same time by multiple processors or cores. Python provides modules like multiprocessing to make this happen efficiently.

Imagine you have a bunch of tasks to complete. Parallel processing lets you use multiple processor cores, making tasks finish faster. It's like having several workers tackling different parts of a job at once.

The multiprocessing module has helpful tools like the Pool class, which simplifies the distribution of tasks among multiple processes. Also, using a queue helps manage tasks efficiently, improving synchronization and overall performance.

In simple terms, parallel processing in Python speeds up tasks by having a team of processors work on them simultaneously, like having many hands make light work.

Atatus: Python Performance Monitoring

Atatus is an Application Performance Management (APM) solution that collects all requests to your Python applications without requiring you to change your source code. However, the tool does more than just keep track of your application's performance.

Monitor logs from all of your Python applications and systems into a centralized and easy-to-navigate user interface, allowing you to troubleshoot faster using Python monitoring.

We give a cost-effective, scalable method to centralized Python logging, so you can obtain total insight across your complex architecture. To cut through the noise and focus on the key events that matter, you can search the logs by hostname, service, source, messages, and more. When you can correlate log events with APM slow traces and errors, troubleshooting becomes easy.

Try your 14-day free trial of Atatus.