Behavior of set_fill_off() changed in v1.5.1

[Michael Klatt]

netCDF4-1.5.1 (installed via pip 19.1.1)
Python 3.6.2 (virtualenv)
macOS Sierra (10.12.6)

Based on the documentation, my understanding ofDataset.set_fill_off() is that it leaves variables uninitialized upon creation. This is useful in cases were the variable is immediately filled with real data. With earlier versions of netCDF4, this works as expected:

""" Demonstrate fill value bug in netCDF4-1.5.1+

"""
from netCDF4 import Dataset
from numpy.ma import masked_all


# Write masked data to netCDF, which will be replaced by _FillValue.

dataset = Dataset("test.nc", "w")
dataset.set_fill_off()  # comment this out and everything works as expected
dim = dataset.createDimension("dim", 10)
var = dataset.createVariable("var1", "f8", (dim.name,))
var[:] = masked_all((10,), "f8")
dataset.close()


# Read variable from netCDF. It should be masked everywhere.

dataset = Dataset("test.nc", "r")
var = dataset.variables["var1"]
assert var[:].mask.all()

As of v1.5.1, the final assertion fails. Inspection of the netCDF file using ncdump shows that var1 is set to _FillValue as expected. However, when var1 is read from the file, it is not masked anywhere. The value is _FillValue (9.9692e+36) everywhere, but the mask of the resulting MaskedArray is False.

Somehow, calling set_fill_off() when the variable is created is preventing it from being masked correctly when the dataset is read. This seems contrary to the documentation, and is a new behavior in v1.5.1.

A workaround is to not call set_fill_off().

[Jeff Whitaker]

I see the problem - a simple case of have and and instead of an or in an if statement. Fix in pull request #973

[Jeff Whitaker]

Closed by #973. Thanks for the report.