corems.encapsulation.spawn_pool

Process pools that still return after pythonnet has started.

 1"""Process pools that still return after pythonnet has started."""
 2
 3from contextlib import contextmanager
 4from multiprocessing import get_context
 5from multiprocessing.pool import TERMINATE
 6
 7
 8@contextmanager
 9def spawn_pool(processes):
10    """Yield a spawn process pool and kill its workers on exit.
11
12    Parameters
13    ----------
14    processes : int
15        Number of worker processes. The caller chooses this count.
16
17    Yields
18    ------
19    multiprocessing.pool.Pool
20        Pool started with the ``spawn`` start method.
21
22    Notes
23    -----
24    Linux ``multiprocessing.Pool`` forks. A forked child deadlocks after
25    pythonnet has started CoreCLR or Mono, so the pool never finishes.
26    Spawn starts a new interpreter. Those workers ignore SIGTERM, and
27    ``Pool.terminate`` waits forever in its ``join``. The caller reads
28    results inside the ``with`` block. On the way out, including when the
29    body raises, the workers are killed and reaped.
30    """
31    pool = get_context("spawn").Pool(processes)
32    try:
33        yield pool
34    finally:
35        _kill_spawn_pool(pool)
36
37
38def _kill_spawn_pool(pool):
39    """Stop a spawn pool without ``Pool.terminate``.
40
41    ``terminate`` signals the workers with SIGTERM and then joins them.
42    A worker that has loaded CoreCLR or Mono ignores that signal, so the
43    join never returns. The results are already in the parent. Kill the
44    workers and cancel the pool finalizer, which would call ``terminate``
45    again at garbage collection.
46    """
47    pool._state = TERMINATE
48    for handler in (pool._worker_handler, pool._task_handler, pool._result_handler):
49        handler._state = TERMINATE
50    pool._change_notifier.put(None)
51    for process in list(pool._pool):
52        if process.is_alive():
53            try:
54                process.kill()
55            except ProcessLookupError:
56                pass
57        process.join()
58    pool._terminate.cancel()
@contextmanager
def spawn_pool(processes): 
 9@contextmanager
10def spawn_pool(processes):
11    """Yield a spawn process pool and kill its workers on exit.
12
13    Parameters
14    ----------
15    processes : int
16        Number of worker processes. The caller chooses this count.
17
18    Yields
19    ------
20    multiprocessing.pool.Pool
21        Pool started with the ``spawn`` start method.
22
23    Notes
24    -----
25    Linux ``multiprocessing.Pool`` forks. A forked child deadlocks after
26    pythonnet has started CoreCLR or Mono, so the pool never finishes.
27    Spawn starts a new interpreter. Those workers ignore SIGTERM, and
28    ``Pool.terminate`` waits forever in its ``join``. The caller reads
29    results inside the ``with`` block. On the way out, including when the
30    body raises, the workers are killed and reaped.
31    """
32    pool = get_context("spawn").Pool(processes)
33    try:
34        yield pool
35    finally:
36        _kill_spawn_pool(pool)

Yield a spawn process pool and kill its workers on exit.

Parameters
  • processes (int): Number of worker processes. The caller chooses this count.
Yields
  • multiprocessing.pool.Pool: Pool started with the spawn start method.
Notes

Linux multiprocessing.Pool forks. A forked child deadlocks after pythonnet has started CoreCLR or Mono, so the pool never finishes. Spawn starts a new interpreter. Those workers ignore SIGTERM, and Pool.terminate waits forever in its join. The caller reads results inside the with block. On the way out, including when the body raises, the workers are killed and reaped.