Reverse DNS bot returns same cached value for whole IPv4 subnet

[DigitalTrustCenter]

We have found that the reverse DNS expert bot seems to return wrong answers. This is due to it caching responses, but also wrongly returning those cached values for other IP addresses in the same IPv4 /24 subnet.

For example, using ReverseDnsExpertBot with default settings (cache db 7, cache ttl 86400). The IP addresses and reverse DNS values are taken from the reverse_dns unit test, and are manually verified.

# intelmqctl run ReverseDns-Expert process -s -d -m '{"source.ip": "192.0.43.7", "extra.shouldbe": "icann.org"}'
Starting ReverseDns-Expert...
ReverseDns-Expert: ReverseDnsExpertBot initialized with id ReverseDns-Expert and intelmq 3.1.0 and python 3.9.16 (main, Mar 14 2023, 03:17:01) as process 321.
ReverseDns-Expert: Bot is starting.
ReverseDns-Expert: Bot initialization completed.
 * Message from cli will be used when processing.
 * Dryrun only, no message will be really sent through.
Processing...
[
    {
        "extra.shouldbe": "icann.org",
        "source.ip": "192.0.43.7",
        "source.reverse_dns": "icann.org"
    }
]
DRYRUN: Message would be sent now to '_default'!
DRYRUN: Message would be acknowledged now!

# intelmqctl run ReverseDns-Expert process -s -d -m '{"source.ip": "192.0.43.8", "extra.shouldbe": "43-8.any.icann.org"}'
[
    {
        "extra.shouldbe": "43-8.any.icann.org",
        "source.ip": "192.0.43.8",
        "source.reverse_dns": "icann.org"
    }
]

(extra unrelated output removed)
In the second run, a different source.ip value is used, but the cached value icann.org is returned by the bot The actual correct value should be 43-8.any.icann.org.
When the cache is cleared using redis-cli -n 7 FLUSHDB, the correct result is returned:

# intelmqctl run ReverseDns-Expert process -s -d -m '{"source.ip": "192.0.43.8", "extra.shouldbe": "43-8.any.icann.org"}'
[
    {
        "extra.shouldbe": "43-8.any.icann.org",
        "source.ip": "192.0.43.8",
        "source.reverse_dns": "43-8.any.icann.org"
    }
]

This seems to be caused by the bot using the first 24 bits of the IPv4 address (as a string of 0's and 1's) as the cache key. Therefore, IP addresses in the same /24 subnet yield the same cache key. For IPv6 the full 128 bits are used and is therefore not a problem. Why this bitstring is used as cache key, instead of the IP address (as a string) is a mystery for us.

Funny enough, this would have been caught in the unit test, as in the first EXAMPLE_INPUT, a source and destination IP address in the same /24 are set. However, in the output, the destination reverse_dns is commented out.

[Sebastian]

Thank you for finding and diagnosing this bug!

This bug dates back to the beginning of this bot in 2015 (0c46999). It's very interesting that no one else yet noticed it. I guess it was indeed intentional in the beginning, although entirely undocumented.

I create PR#2395 to fix this issue by generating cache keys from the complete address.