I used this script (see code at the end) to assess whether a global object is shared or copied when the parent process is forked.
Briefly, the script creates a global data object, and the child processes iterate over data. The script also monitors the memory usage to assess whether the object was copied in the child processes.
Here are the results:
data = np.ones((N,N)). Operation in the child process:
data.sum(). Result: data is shared (no copy)
data = list(range(pow(10, 8))). Operation in the child process: sum(data). Result: data is copied.
data = list(range(pow(10, 8))). Operation in the child process: for x in data: pass. Result: data is copied.
Result 1) is expected because of copy-on-write. I am a bit puzzled by the results 2) and 3). Why is data copied?
Script
source
import multiprocessing as mp
import numpy as np
import logging
import os
logger = mp.log_to_stderr(logging.WARNING)
def free_memory():
total = 0
with open('/proc/meminfo', 'r') as f:
for line in f:
line = line.strip()
if any(line.startswith(field) for field in ('MemFree', 'Buffers', 'Cached')):
field, amount, unit = line.split()
amount = int(amount)
if unit != 'kB':
raise ValueError(
'Unknown unit {u!r} in /proc/meminfo'.format(u = unit))
total += amount
return total
def worker(i):
x = data.sum() # Exercise access to data
logger.warn('Free memory: {m}'.format(m = free_memory()))
def main():
procs = [mp.Process(target = worker, args = (i, )) for i in range(4)]
for proc in procs:
proc.start()
for proc in procs:
proc.join()
logger.warn('Initial free: {m}'.format(m = free_memory()))
N = 15000
data = np.ones((N,N))
logger.warn('After allocating data: {m}'.format(m = free_memory()))
if __name__ == '__main__':
main()
Detailed results
Run 1 output
[WARNING/MainProcess] Initial free: 25.1 GB
[WARNING/MainProcess] After allocating data: 23.3 GB
[WARNING/Process-2] Free memory: 23.3 GB
[WARNING/Process-4] Free memory: 23.3 GB
[WARNING/Process-1] Free memory: 23.3 GB
[WARNING/Process-3] Free memory: 23.3 GB
Run 2 output
[WARNING/MainProcess] Initial free: 25.1 GB
[WARNING/MainProcess] After allocating data: 21.9 GB
[WARNING/Process-2] Free memory: 12.6 GB
[WARNING/Process-4] Free memory: 12.7 GB
[WARNING/Process-1] Free memory: 16.3 GB
[WARNING/Process-3] Free memory: 17.1 GB
Run 3 output
[WARNING/MainProcess] Initial free: 25.1 GB
[WARNING/MainProcess] After allocating data: 21.9 GB
[WARNING/Process-2] Free memory: 12.6 GB
[WARNING/Process-4] Free memory: 13.1 GB
[WARNING/Process-1] Free memory: 14.6 GB
[WARNING/Process-3] Free memory: 19.3 GB
Aucun commentaire:
Enregistrer un commentaire