Sparsetimefunctions fail with MPI & DEVITO_FIRST_TOUCH=1
deckerla opened this issue · comments
import devito
import numpy as np
grid = devito.Grid(shape=(25,25,25))
stf = devito.SparseTimeFunction(name="stf", npoint=1, nt=11, grid=grid, coordinates=np.array([0.5,0.5,0.5]))
u = devito.TimeFunction(name="u", grid=grid)
inj = stf.inject(field=u.forward, expr=stf)
op = devito.Operator(inj, name="inj")
op.apply()
This runs fine without MPI:
cvx@cbox-lukedecker-nocollapse:~/.julia/dev/JetPackDevito/test$ python mfe.py
Operator `inj` ran in 0.02 s
And with MPI:
cvx@cbox-lukedecker-nocollapse:~/.julia/dev/JetPackDevito/test$ DEVITO_MPI=1 mpirun -n 2 python mfe.py
Operator `inj` ran in 0.18 s
Operator `inj` ran in 0.18 s
Running this with first touch & MPI fails with the following behavior:
cvx@cbox-lukedecker-nocollapse:~/.julia/dev/JetPackDevito/test$ DEVITO_FIRST_TOUCH=1 DEVITO_MPI=1 mpirun -n 2 python
mfe.py
Traceback (most recent call last):
File "/home/cvx/.julia/dev/JetPackDevito/test/mfe.py", line 5, in <module>
Traceback (most recent call last):
File "/home/cvx/.julia/dev/JetPackDevito/test/mfe.py", line 5, in <module>
stf = devito.SparseTimeFunction(name="stf", npoint=1, nt=11, grid=grid, coordinates=np.array([0.5,0.5,0.5]))
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/basic.py", line 850, in __new__
stf = devito.SparseTimeFunction(name="stf", npoint=1, nt=11, grid=grid, coordinates=np.array([0.5,0.5,0.5]))
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/basic.py", line 850, in __new__
newobj.__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/sparse.py", line 307, in __init_finalize__
newobj.__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/sparse.py", line 307, in __init_finalize__
super(AbstractSparseTimeFunction, self).__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/sparse.py", line 463, in __init_finalize__
super(AbstractSparseTimeFunction, self).__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/sparse.py", line 463, in __init_finalize__
self._coordinates = SubFunction(
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/basic.py", line 850, in __new__
self._coordinates = SubFunction(
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/basic.py", line 850, in __new__
newobj.__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 1464, in __init_finalize__
newobj.__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 1464, in __init_finalize__
super(SubFunction, self).__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 1002, in __init_finalize__
super(SubFunction, self).__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 1002, in __init_finalize__
super(Function, self).__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 94, in __init_finalize__
super(Function, self).__init_finalize__(*args, **kwargs)
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 94, in __init_finalize__
self.data_with_halo[:] = initializer
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/finite_differences/differentiable.py", line 157, in __getattr__
self.data_with_halo[:] = initializer
File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/finite_differences/differentiable.py", line 157, in __getattr__
raise AttributeError("%r object has no attribute %r" % (self.__class__, name))
raise AttributeError("%r object has no attribute %r" % (self.__class__, name))
AttributeError: stf_coords object has no attribute 'data_with_halo'. Did you mean: '_data_with_inhalo'?
AttributeError: stf_coords object has no attribute 'data_with_halo'. Did you mean: '_data_with_inhalo'?
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[21223,1],0]
Exit code: 1
--------------------------------------------------------------------------
if DEVITO_FIRST_TOUCH actually useful? do you see benefit from using it?
Honestly I haven’t used it much. I was exploring the use of it for increasing performance. If you think it won’t do much, then don’t prioritize it.
… On May 26, 2023, at 2:30 AM, Fabio Luporini ***@***.***> wrote:
if DEVITO_FIRST_TOUCH actually useful? do you see benefit from using it?
—
Reply to this email directly, view it on GitHub <#2132 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADX3SS2LARQ7JKVMOS3V653XIBL2ZANCNFSM6AAAAAAYFRBFQM>.
You are receiving this because you authored the thread.