controlling numerical instabilities and masks
francispoulin opened this issue · comments
I have developed a 2D code to solve the vorticity equation using a pseudo-spectral approach. It works beautifully in the case when it is Fourier by Fourier, if i use an exponential filter to tame the numerical instability. I'm now working on a cheb by Fourier version, and realizing I have a lot to learn on how shenfun works and I am hoping to get some guidance.
My first thought was to essentially truncate the highest Fourier and cheb modes. I was hoping that mask might do this but I see that when I create the mask for a Function space, it gives me a 1D vector for the mask in the Fourier direction. This would only control part of the issue and I need to do something to control the small scale features in both directions. If I were to build a 2D mask function that was 1 for the first 2/3 of the domain in each direction, and multiply the transformed flux by this masked function, do you think that would do the trick?
I also looked into padding but that seems to be most useful when multiplying functions and not necessarily for filtering the resulting flux, as far as I can tell.
Any advice would be greatly apprecitaed.
Hi Francis,
I'm glad you ask:-) These things are very easy to do with shenfun. Do you mean the Fourier function mask_nyquist by the way? It creates a 1D array of ones, except for the Nyquist frequency, where it is zero. So when you multiply it by an array, then all items of that array are unchanged, except for the Nyquist frequency that is set to zero. It is a 1D array, but the default behaviour is to broadcast the array to all dimensions of the functionspace that the basis belongs to. So if you have the Fourier basis in a 3D TensorProductSpace, then the mask works exactly as expected when multiplied with a 3D array. You need to understand how Numpy broadcasting works to really understand all of shenfun. I can show how this works in a simple 2D case
from shenfun import *
SD = Basis(6, family='C')
K1 = Basis(6, family='F', dtype='d')
T = TensorProductSpace(comm, (SD, K1))
u = Function(T, val=1)
mask = K1.get_mask_nyquist()
print(mask)
print(mask.shape)
u *= mask
print(u)
Run it to get
[[1 1 1 0]]
(1, 4)
[[1.+0.j 1.+0.j 1.+0.j 0.+0.j]
[1.+0.j 1.+0.j 1.+0.j 0.+0.j]
[1.+0.j 1.+0.j 1.+0.j 0.+0.j]
[1.+0.j 1.+0.j 1.+0.j 0.+0.j]
[1.+0.j 1.+0.j 1.+0.j 0.+0.j]
[1.+0.j 1.+0.j 1.+0.j 0.+0.j]]
See how the last axis is zeroed. The array mask is of shape (1, 4), but for Numpy that means it can be broadcasted along the first axis to a (4, 4) array. In fact the array will act as this:
print(np.broadcast_to(mask, (4, 4)))
[[1 1 1 0]
[1 1 1 0]
[1 1 1 0]
[1 1 1 0]]
But Numpy is smart enough not to use any more memory for this (4, 4) array than for the 1D array:-)
The masking is not restricted to the Nyquist frequency. It's just that for the Fourier basis the operation is required so frequently that I created an explicit function for it.
Want to mask the two highest frequencies of the Chebyshev basis? You can either create a mask, like for mask_nyquist.
u[:] = 1
u[SD.sl[slice(-2, None)]] = 0
print(u)
[[1.+0.j 1.+0.j 1.+0.j 1.+0.j]
[1.+0.j 1.+0.j 1.+0.j 1.+0.j]
[1.+0.j 1.+0.j 1.+0.j 1.+0.j]
[1.+0.j 1.+0.j 1.+0.j 1.+0.j]
[0.+0.j 0.+0.j 0.+0.j 0.+0.j]
[0.+0.j 0.+0.j 0.+0.j 0.+0.j]]
See how the last two rows are zeroed. SD.sl[some slice] is a Function that creates a list of slices that is using slice(0, 4) for the axis of SD, and None for the rest. That means all the rest.
print(SD.sl[slice(0, 2)])
(slice(0, 2, None), slice(None, None, None))
and
print(K1.sl[slice(0, 2)])
(slice(None, None, None), slice(0, 2, None))
There is also another helper function si[some integer] that is very useful
print(SD.si[0])
(0, slice(None, None, None))
So this makes it easier to implement code that is working in one dimension, or in many:-) And you can create mask very easily. For example
mask = np.ones(SD.N, dtype=int)
mask[-2:] = 0
mask = SD.broadcast_to_ndims(mask)
print(mask)
[[1]
[1]
[1]
[1]
[0]
[0]]
I hope this helps:-) You could also look here.
Thank you Mikael. It is very kind of you to explain this in detail and it's great to see that shenfun can do all of this for you. It is also great that it seems to be very easy to do this. I will build a asking function based on what you've said here, after I've digested it a bit more, and I hope to give it a try again tomorrow. It should be easy enough but I want to do some experimentation to make sure that I fully understand it and it actually works for me. More soon!
Hello Mikael,
I looked at one of your older papers on SpectralDNS and saw you had a very clean way ot defining a 2D mask function. This might not be as efficient but it generated the 1's and 0's where I wanted them.
`# defining a 2D mask function
dealias = np.array((abs(K[0])<N[0]*2./3.)*(abs(K[1])<N[1]*2./3.), dtype=bool)
`
However, this seemed to give rise to some strange behavour and I could not figure out why. Therefore, I decided a different method.
For Fourier based methods I have used both exponential filters and hyperviscosity, and both work at preventing the build up of energy on the small scales and numerical instabilities from developing. It seems to me that when using a cheb basis, this is perhaps not as simple. I should probably return to Boyd's book to learn more about this but in the end, what I tried and seems to be working, so far, is to use hyperviscosity.
Below is a sample line that I added for 4th order hyperviscosity. I will need to play around with the coefficient but supsect this might be more straightforward than filtering, at least for me.
Thanks again!
`# hypervisosity
hk = div(grad(div(grad(qk))))
hr = project(hk, TF1).backward(hr)
`
Hi Francis,
The method
dealias = np.array((abs(K[0])<N[0]*2./3.)*(abs(K[1])<N[1]*2./3.), dtype=bool)
is intuitiv and nice, but if you want to use the 2/3-rule you should create the Fourier bases with keyword dealias_direct=True. This approach ensures that symmetry is preserved while setting the highest 1/3 of wavenumbers to zero before a backward transform.
The Chebyshev basis has different kind of aliasing than Fourier, but it is still there and you can use dealias_direct for Chebyshev/Legendre as well. But if you're really serious about dealiasing you should consider using the 3/2-rule. See, e.g., the Ginzburg-Landau demo.
Note that if you plan to do the hyperviscosity approach many times you should consider some optimizations. For one thing project will assemble a mass matrix every time it is called.
Hello Mikael,
Thanks for the help.
Yes, I found that hyperviscosity slowed down my code by a factor of two, and I wasn't very good at picking my coefficient so it did not even solve the numerical instabilities that developed.
I will owrk on using the dealias_direct, which looks easy enough, and learning how to use the 3/2-rule.
Just to be clear, should I be using both the dealias_direct and the padding or one of the two?
Use just one or the other, but note that padding is superior if you are interested in accuracy in the highest wavenumbers. Like for example in a turbulence simulation, where all wavenumbers may be substantial. Note that dealiasing requires that you create a space to use solely for the purpose of dealiasing. And you should only use dealiasing for nonlinear terms, like convection. All these things have been implemented for efficiency in the spectralDNS repository and you could probably learn something from the solvers there.
Hello MIkael,
Thank you again. By following the example you pointed me to, I was able to get something going fairly easily. The good news is that it is numerically stable. The bad news is that I still see ringing on the grid scale and it runs slower in parallel than in serial on a 256x256 grid. That does not seem right so clearlyI did something wrong.
I'm going to compare this with your NS2D.py code from spectralDNS, since that solves the same equations I'm working with for the moment. Unfortunately, on a first attempt I could install spectralDNS using conda but could not get the demo working. This means I can a couple of options.
One, do you have a pure shenfun version of NS2D.py that I could play with?
Two, I can post an issue on that repo to try and sort out what's going wrong. I can happily od the second regardless of the first but thought I would ask your opinon first.
Hi Francis
To install spectralDNS just clone the repo and do
python setup.py build_ext --i
It shouldn't be hard if shenfun is working. I don't have a standalone version of NS2D around, but I could easily wrap one up. BTW, which demo did you try? I think it has to be TG2D.py or Vortices2D.py.
Thanks.
I presume you mean that I can use my shenfun installation (from conda) and then clone the repo and do the above line to install it?
I tried what you had exactly and that didn't work. Basically, error because --i is not a unique prefex.
But i tried with --inplace, as is discussed on the page, and that seemed to get a lot further. It seemed to install it okay, from I can tell. Unfortunately, when I try running one of the demos from the docs and get the error copied below.
It's probably a good idea for me to write my own shenfun NS2D code so that I go through the details. The example was the TG.
`#Error message
$ mpirun -np 4 python TG.py NS
Traceback (most recent call last):
File "TG.py", line 4, in <module>
from spectralDNS import config, get_solver, solve
ModuleNotFoundError: No module named 'spectralDNS'
Traceback (most recent call last):
File "TG.py", line 4, in <module>
from spectralDNS import config, get_solver, solve
ModuleNotFoundError: No module named 'spectralDNS'
Traceback (most recent call last):
File "TG.py", line 4, in <module>
from spectralDNS import config, get_solver, solve
ModuleNotFoundError: No module named 'spectralDNS'
Traceback (most recent call last):
File "TG.py", line 4, in <module>
from spectralDNS import config, get_solver, solve
ModuleNotFoundError: No module named 'spectralDNS'
`
Hi Francis
You need to add the spectralDNS path to the PYTHONPATH for Python to be able to find it
export PYTHONPATH="path of SpectralDNS":$PYTHONPATH
Sorry for my silly mistake but i can confirm that of course that was the solution to my problem.
I am going to try and do this without using spectralDNS, so it's all contained in shenfun however.
I have code that solves the 2D vorticity equation, which is not exactly Navier-Stokes, but is the vorticity formulation for 2D. I am using the Ginzberg-Landau as a model as well. So far without padding but it should be easy to add that in after I get the non-padded version working.
Below I will copy my error and code. Can you help me figure out what's wrong?
`#error
$ python NavierStokes2D.py
Traceback (most recent call last):
File "NavierStokes2D.py", line 115, in <module>
q_hat = integrator.solve(q, q_hat, dt, t_domain)
File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/sitepackages/shenfun/utilities/integrators.py", line 322, in solve
self.update(u, u_hat, t, tstep, **self.params)
File "NavierStokes2D.py", line 97, in update
file.write(tstep, write_tstep[1])
File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/sitepackages/mpi4py_fft/io/h5py_file.py", line 118, in write
FileBase.write(self, step, fields, **kw)
File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/sitepackages/mpi4py_fft/io/file_base.py", line 65, in write
self._check_domain(group, u)
File "/home/fpoulin/software/anaconda3/envs/shenfun/lib/python3.8/sitepackages/mpi4py_fft/io/h5py_file.py", line 38, in _check_domain
self.domain = ((0, 2*np.pi),)*field.dimensions
AttributeError: 'numpy.ndarray' object has no attribute 'dimensions'
"""# Sample code
"""
Simple spectral solver for the two-dimensional vorticity equation.
In Geophysics, this model is also called the Quasi-Geostrophic (QG) Model
Prognostic equation: q_t = - (u*q_x + v*q_y)
q_hat_t = - (u*q_x + v*q_y)_hat
Diagnostic equation: q = v_x - u_y
q_hat = i*k*v_hat - i*l*u_hat
from mpi4py import MPI
import numpy as np
from shenfun import *
import matplotlib.pyplot as plt
import time
import sys
comm = MPI.COMM_WORLD
# Geometry
M = 8
N = (2**M, 2**M)
L = [20.0, 40.]
# Funcion Spaces
V1 = Basis(N[0], 'F', dtype='D', domain=(0, L[0]))
V2 = Basis(N[1], 'F', dtype='d', domain=(0, L[1]))
T = TensorProductSpace(comm, (V1, V2), **{'planner_effort': 'FFTW_MEASURE'})
TV = VectorTensorProductSpace(T)
q_hat, q = Function(T), Array(T) # potential vorticity (PV)
U_hat, U = Function(TV), Array(TV) # velocity
rhs_hat, rhs = Function(T), Array(T) # flux
gradq_hat, gradq = Function(TV), Array(TV) # gradient of PV
# Mesh
X = T.local_mesh(True)
# Wavenumbers
K = np.array(T.local_wavenumbers(True,True))
K2 = np.sum(K*K, 0, dtype=float)
K_over_K2 = K.astype(float) / np.where(K2 == 0, 1, K2).astype(float)
# Initialization
np.random.seed(1)
q_pert = 1e-4*2.*(np.random.random(q.shape).astype(q.dtype)-0.5)
q = -2.*np.tanh(X[1] - L[1]/4.)/pow(np.cosh(X[1] - L[1]/4.),2) \
+ 2.*np.tanh(X[1] - 3*L[1]/4.)/pow(np.cosh(X[1] - 3.*L[1]/4.),2) \
+ q_pert
q_hat = T.forward(q, q_hat)
U[0] = T.backward( 1j*K_over_K2[1]*q_hat, U[0])
U[1] = T.backward(-1j*K_over_K2[0]*q_hat, U[1])
U_hat = TV.forward(U, U_hat)
def Flux_QG(self, q, q_hat, rhs_hat, **params):
global T, TV, U, U_hat, gradq, gradq_hat#, qb, Ub,
# Gradient of q
gradq_hat = 1j*K*q_hat
gradq = TV.backward(gradq_hat, gradq)
# Velocity
U[0] = T.backward( 1j*K_over_K2[1]*q_hat, U[0])
U[1] = T.backward(-1j*K_over_K2[0]*q_hat, U[1])
# Flux
rhs[:] = -(U[0]*gradq[0] + U[1]*gradq[1])
rhs_hat = T.forward(rhs, rhs_hat)
return rhs_hat
# Create figure
plt.figure(figsize=(10,10))
image = plt.contourf(X[0], X[1], q, 100)
plt.draw()
plt.pause(1e-6)
count = 0
def update(self, q, q_hat, t, tstep, plot_tstep, write_tstep, file, **params):
global count
if tstep % plot_tstep == 0 and plot_tstep > 0:
q = T.backward(q_hat, q)
image.ax.clear()
image.ax.contourf(X[0], X[1], q, 100)
plt.pause(1e-6)
count += 1
#plt.savefig('QG_1Layer_{}_{}.png'.format(N[0], count))
if tstep % write_tstep[0] == 0:
q = T.backward(q_hat, q)
file.write(tstep, write_tstep[1])
if __name__ == '__main__':
file0 = HDF5File("QG_1Layer_{}.h5".format(N[0]), mode='w')
par = {'plot_tstep': 10,
'write_tstep': (10, {'q': [q]}),
'file': file0}
# Time parameters
t_domain = (0.0, 0.1)
dt = 1e-3
# Solve
time_initial = time.time()
integrator = RK4(T, N=Flux_QG, update=update, **par)
integrator.setup(dt)
q_hat = integrator.solve(q, q_hat, dt, t_domain)
time_final = time.time()
generate_xdmf("QG_1layer_{}.h5".format(N[0]))
fftw.export_wisdom('QG1L.wisdom')
# Compute energy and growth
q = T.backward(q_hat, q)
energy = comm.reduce(0.5*np.sum(U*U))
print('Total time = ', (time_final - time_initial))
`
Hi Francis,
Problem here is
q = -2.*np.tanh(X[1] - L[1]/4.)/pow(np.cosh(X[1] - L[1]/4.),2) + 2.*np.tanh(X[1] - 3*L[1]/4.)/pow(np.cosh(X[1] - 3.*L[1]/4.),2) + q_pert
This is a typical Python mistake. With this line you delete the old q, which was an array, and create a new q, which is a Numpy ndarray. Correct approach is simply
q[:] = -2.*np.tanh(X[1] - L[1]/4.)/pow(np.cosh(X[1] - L[1]/4.),2) + 2.*np.tanh(X[1] - 3*L[1]/4.)/pow(np.cosh(X[1] - 3.*L[1]/4.),2) + q_pert
Ah, yes, another rookie mistake. My apologies but thank you so much for pointing out my error and I am happy to say that it now seems to be running.