Static retrieval not working

[Jacob]

I used the example from the blogpost:

hopfield = Hopfield(
            scaling=beta,

            # do not project layer input
            state_pattern_as_static=True,
            stored_pattern_as_static=True,
            pattern_projection_as_static=True,

            # do not pre-process layer input
            normalize_stored_pattern=False,
            normalize_stored_pattern_affine=False,
            normalize_state_pattern=False,
            normalize_state_pattern_affine=False,
            normalize_pattern_projection=False,
            normalize_pattern_projection_affine=False,

            # do not post-process layer output
            disable_out_projection=True
        )

# and then to retrieve
# Y = stored, x = state
hopfield((Y, x, Y))

I used random real life images where I tried both gray-scale and rgb and resizing the image but it I couldn't retrieve the images at all, even when storing less than 10 it always returned the same image. I'm wondering if this is expected with non-simple images or if I'm missing something. I've tried altering beta but with no luck.

[Viet Tran]

Could you provide the real life images you are referring to and also the preprocessing steps you apply?
As a starting point you could have a look at a working implementation here https://github.com/ml-jku/hopfield-layers/blob/gh-pages/examples/simpsons/continuous_hopfield_pytorch.ipynb

[Jacob]

@vthuongt I used this dataset. I tried different preprocessing steps, resizing to a smaller size, setting the values between 0 and 1, -1 to 1 and original, 2d and 3d. I followed the guide from the blog which I assume is similar with simpsons.

[Viet Tran]

@FerusAndBeyond I just tried a couple of images from your linked dataset. Indeed it can happen that the wrong image seems to be retrieved. However this comes from the fact that the naive masking removes relevant information from the image and thus the masked image becomes more similar to another image in the training set (bigger inner product) than the original unmasked image. Since the pattern with the highest inner product is retrieved, that behavior is as expected.

To circumvent your problem and achieve a more stable retrieval, one can instead apply a random masking to every pixel. To not introduce any bias through the masking use a masking value of 0 when the pixel range is [-1,+1]:

def mask_image_random(images_np, value=0):
    """masks every pixel with 50% chance.  

    Args:
        images_np ([np.ndarray]): list of images, each as numpy array

    Returns:
        [np.ndarray]: list of masked images as numpy arrays
    """
    images_np_masked = []
    n_pixel = images_np[0].shape[0]
    for im_np in images_np:
        im_masked = im_np.copy()
        for i in range(n_pixel):
            for j in range(n_pixel):
                if np.random.rand() < 0.5:
                    im_masked[i][j] = value
        images_np_masked.append(im_masked)
    return images_np_masked

[bschaefl]

Closing issue. Seems to be resolved by providing an additional example as well as a corresponding explanation.