Embarissingly parallel loop

Core functions

Core functions for template

core_functions.distribute_tasks(func_task, tasks, num_proc=1, is_distributed=False)

Distribute workload.

This function distributes the workload using the multiprocessing or mpi4py library. It simply creates a pool of processes that allow to work on the tasks using shared or distributed memory.

Notes

We need to ensure that the number of processes is never larger as the number of tasks as otherwise the MPI implementation does not terminate properly.

• MP Pool, see here for details

• MPI Pool, see here for details

Test integration

Integration tests.

This module contains the integration tests that all the individual units are combined and tested together.

test_integration.get_random_request()

Random test case.

This function sets up a random test case that differs depending on whether MPI capabilities are available or not.

test_integration.test_1()

Test a random request.

This test simply evaluates a random request. It automatically checks whether a distributed evaluation is an option.

test_integration.test_2()

Varying the number of processes.

This test evaluates the same request with different number of processes and ensures that the amount resources do not matter for the results.

test_integration.test_3()

Alternating between shared and distributed memory.

This test evaluates the same request using the multiprocessing and mpi4py library and ensures that both yield the same result.