Checking out the nogil interpreter.

As someone who uses Python on a daily basis, I have been following the “gilectomy” with a keen interest. This week I decided it was finally time to play around with whatever Sam Gross has been cooking in his repo located here.

After pulling and building said repo, I started out by simply copy-pasting his example. It was interesting, to say the least. The code is very simple:

import sys
from concurrent.futures import ThreadPoolExecutor

print(f"nogil={getattr(sys.flags, 'nogil', False)}")

def fib(n):
    if n < 2: return 1
    return fib(n-1) + fib(n-2)

threads = 8
if len(sys.argv) > 1:
    threads = int(sys.argv[1])

with ThreadPoolExecutor(max_workers=threads) as executor:
    for _ in range(threads):
        executor.submit(lambda: print(fib(34)))

OK, so this is a really straightforward example, but let’s see what kind of time differences I get running it with nogil (3.9) and then running it with plain-vanilla Python 3.9.

Plain Vanilla:

VS nogil:

As you can see, the speedup is crazy, which is what I expected given that the plain vanilla version only ran on a single core.

In order to give the traditional way of parallel processing in Python a fair chance, I took this a little further. In order to compare it to a ProcessPoolExecutor, I tweaked the code a little bit. The final code looked like this:

import getopt
import sys
import concurrent.futures
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

print(f"nogil={getattr(sys.flags, 'nogil', False)}")

def fib(n):
    if n < 2: return 1
    return fib(n-1) + fib(n-2)

threads = 8

executor = ThreadPoolExecutor

try:
    opts, args = getopt.getopt(sys.argv[1:], "w:p", ["workers=", "process"])

    for o, a in opts:
        if o in ('-w', '--workers'):
            threads = int(a)
        elif o in ('-p', '--process'):
            executor = ProcessPoolExecutor

    with executor(max_workers=threads) as executor:
        futures = []
        for i in range(threads):
            futures.append(executor.submit(fib, 34))
        for future in concurrent.futures.as_completed(futures):
            print(future.result())
except getopt.GetoptError as err:
    print(err)
    sys.exit(-1)

OK, so I can now switch between the two and added a getopt-based worker count flag. Let’s see what I get when I use the ProcessPoolExecutor:

And with the threads:

Still seems to be a win for the threads. Granted, we aren’t really doing any heavy lifting yet, but in the early stages, this looks like it’s worth exploring further.