How to run python script on multiple cores/threads (multiprocessing)

I’ve been looking for means of parallel execution in Python for quite a while. There’s a bunch of various libraries like Dask and Joblib Parallel. However using multiple threads won’t give you better performance due to the global interpreter lock called GIL.

At the moment the best solution I’ve found is multiprocessing. It uses subprocesses instead of threads thus bypassing the Python’s Global Interpreter Lock to one core. Basically it spawns child interpreters that lets leverage your hardware to a maximum potential and thence speed up your Python code execution in parallel.

Here’s an example:

import multiprocessing
import time

def calc_square(numbers, q):
    for n in numbers:
        q.put(n)
        time.sleep(0.2)

    q.put(-1)
    print('Exiting function')

if __name__ == '__main__':
    print('Now in the main code. Process name is:', __name__)
    # numbers = [2, 3, 4, 5]
    names = ['bob', 'john', 'pete', 'ian']
    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=calc_square, args=(names, q))
    p.start()

    while True:
        nq = q.get()
        print('Hello ', nq)
        if nq == -1:
            break

    print('Done')
    p.join()