Reporting errors when run the mixturemodel example

[Yanhui85]

Hello:
I have installed the PyDREAM package on Windows 10, when run the mixturemodel example, I get the following error:

multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 44, in mapstar
return list(map(*args))
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\core.py", line 119, in _sample_dream
raise e
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\core.py", line 105, in _sample_dream
sampled_params[iteration], log_prior , log_like = step_fxn(q0)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\Dream.py", line 406, in astep
raise e
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\Dream.py", line 254, in astep
self.last_prior, self.last_like = self.logp(q0)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\model.py", line 31, in total_logp
loglike = self.likelihood(q0)
File "D:\Python_codes\PyDREAM-master\pydream\examples\mixturemodel\mixturemodel.py", line 39, in likelihood
log_lh = np.zeros((k))
NameError: name 'np' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\Python_codes\PyDREAM-master\pydream\examples\mixturemodel\mixturemodel.py", line 64, in
sampled_params, log_ps = run_dream([params], likelihood, niterations=niterations, nchains=nchains, start=starts, start_random=False, save_history=True, history_file='mixturemodel_seed.npy', multitry=5, parallel=False)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\core.py", line 74, in run_dream
returned_vals = pool.map(_sample_dream, args)
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 644, in get
raise self._value
NameError: name 'np' is not defined

It seems that the errors are due to the multiprocessing pool.map(), maybe there are some differences of this function when it is run on Linux and Windows, but I donot known how to fix it. Would you please help me with this?

Thank you very much for your help
Best regards

[ortega2247]

It looks like it is a problem with numpy. How did you install pydream? Can you also check that you have numpy installed?

[Yanhui85]

Thanks for your reply. I installed pydream using "pip install pydream" and I am sure I have installed numpy correctly, as it can be used in other codes.
When I added a line "import numpy as np" at the start of the likelihood(params) function (Line 38 in mixturemodel.py), the error about numpy disappeared, while it reported that the "k" is not defined.

multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 44, in mapstar
return list(map(*args))
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\core.py", line 119, in _sample_dream
raise e
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\core.py", line 105, in _sample_dream
sampled_params[iteration], log_prior , log_like = step_fxn(q0)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\Dream.py", line 406, in astep
raise e
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\Dream.py", line 254, in astep
self.last_prior, self.last_like = self.logp(q0)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\model.py", line 31, in total_logp
loglike = self.likelihood(q0)
File "D:\Python_codes\PyDREAM-master\pydream\examples\mixturemodel\mixturemodel.py", line 39, in likelihood
log_lh = np.zeros((k))
NameError: name 'k' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\Python_codes\PyDREAM-master\pydream\examples\mixturemodel\mixturemodel.py", line 64, in
sampled_params, log_ps = run_dream([params], likelihood, niterations=niterations, nchains=nchains, start=starts, start_random=False, save_history=True, history_file='mixturemodel_seed.npy', multitry=5, parallel=False)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\core.py", line 74, in run_dream
returned_vals = pool.map(_sample_dream, args)
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 644, in get
raise self._value
NameError: name 'k' is not defined

[ortega2247]

Ok, I think I figured out what the problem is. It seems that the problem in Windows is that multiprocessing doesn't propagate automatically the variables defined in the parent process to the child processes. You can find more information here: https://stackoverflow.com/questions/6596617/python-multiprocess-diff-between-windows-and-linux

I think a workaround would be to explicitly pass all the required variables to the likelihood function:

def likelihood(params):
    import numpy as np
    k = 2
    d = 10
    mu1 = np.linspace(-5, -5, num=d)
    mu2 = np.linspace(5, 5, num=d)
    mu = np.array([mu1,mu2])
    log_F = np.array([-10.2880, -9.5949])
    log_lh = np.zeros((k))
    for j in range(2):
        log_lh[j] = -.5 * np.sum((params - mu[j,:])**2) + log_F[j]
    maxll = np.max(log_lh)
    post = np.array(np.exp(log_lh - maxll), dtype='float64')
    density = np.sum(post)
    post = post/float(density)
    log_L = np.log(density) + maxll
    return log_L

Unfortunately I don't have a windows machine to test this, but I think this is a workaround. Please let me know if it works

[Yanhui85]

Yes, it does work. But another errors are reported. It seems that the error is encounterd when save the results. I have searched the error type, maybe due to escape characters differences bewteen Linux and Windows? Could you help me with this? Thanks.

multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 44, in mapstar
return list(map(*args))
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\core.py", line 119, in _sample_dream
raise e
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\core.py", line 105, in _sample_dream
sampled_params[iteration], log_prior , log_like = step_fxn(q0)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\Dream.py", line 406, in astep
raise e
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\Dream.py", line 349, in astep
self.record_history(self.nseedchains, self.total_var_dimension, q_new, self.len_history)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\Dream.py", line 906, in record_history
self.save_history_to_disc(np.frombuffer(Dream_shared_vars.history.get_obj()), prefix)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\Dream.py", line 920, in save_history_to_disc
np.save(filename, history)
File "C:\ProgramData\Anaconda3\Lib\site-packages\numpy\lib\npyio.py", line 490, in save
fid = open(file, "wb")
OSError: [Errno 22] Invalid argument: '2019_08_28_11:14:45_DREAM_chain_history.npy'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\Python_codes\PyDREAM-master\pydream\examples\mixturemodel\mixturemodel.py", line 79, in
sampled_params, log_ps = run_dream([params], likelihood, niterations=niterations, nchains=nchains, start=starts, start_random=False, save_history=True, history_file='mixturemodel_seed.npy', multitry=5, parallel=False)
File "C:\ProgramData\Anaconda3\Lib\site-packages\pydream\core.py", line 74, in run_dream
returned_vals = pool.map(_sample_dream, args)
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "C:\ProgramData\Anaconda3\Lib\site-packages\multiprocess\pool.py", line 644, in get
raise self._value
OSError: [Errno 22] Invalid argument: '2019_08_28_11:14:45_DREAM_chain_history.npy'

[ortega2247]

I believe the problem is that pydream tries to save the history of the sampled points in the parameter space and assigns an automatic name with the date if a model name is not defined, the date have colons to separate hours and minutes (e.g 11:14:45) and Windows doesn't allow colons in filenames. A workaround for this would be to not save the history by running pydream like this:

sampled_params, log_ps = run_dream([params], likelihood, niterations=niterations, nchains=nchains, start=starts, start_random=False, save_history=False, history_file='mixturemodel_seed.npy', multitry=5, parallel=False)

Note that I set the save_history to false here ^

or if you need to save the sampling history you can assign a model name so that pydream doesn't use the date as a name:

sampled_params, log_ps = run_dream([params], likelihood, niterations=niterations, nchains=nchains, start=starts, start_random=False, save_history=True, model_name='mixture_model_example' history_file='mixturemodel_seed.npy', multitry=5, parallel=False)

[Yanhui85]

It works perfectly now. Thank you so much for your help!

[ortega2247]

Awesome! We haven't done a lot of testing with Windows but this also help us to improve the code, thanks!