Sometimes, we want to use numpy array in shared memory for multiprocessing with Python.
In this article, we’ll look at how to use numpy array in shared memory for multiprocessing with Python.
How to use numpy array in shared memory for multiprocessing with Python?
To use numpy array in shared memory for multiprocessing with Python, we can just hold the array in a global variable.
For instance, we write
import multiprocessing
import numpy as np
data_array = None
def job_handler(num):
return id(data_array), np.sum(data_array)
def launch_jobs(data, num_jobs=5, num_worker=4):
global data_array
data_array = data
pool = multiprocessing.Pool(num_worker)
return pool.map(job_handler, range(num_jobs))
mem_ids, sumvals = zip(*launch_jobs(np.random.rand(10)))
print(np.all(np.asarray(mem_ids) == id(data_array)))
to create the data_array global variable to hold the numpy array.
Then we assign it to a value in launch_jobs.
And then we call launch_jobs with np.random.rand(10) to set that as the value of data_array.
We start the threads with
pool = multiprocessing.Pool(num_worker)
return pool.map(job_handler, range(num_jobs))
in launch_jobs.
We use job_handler with to run code for each thread.
This works if the script is run in a POSIX compliant OS.
Conclusion
To use numpy array in shared memory for multiprocessing with Python, we can just hold the array in a global variable.
