Cache util::EndOfFileException

[esarrias]

Hi. During Bitextor testing, we encountered an issue regarding empty lines in cache output. More precisely, if the first line of the input causes the output to be empty, and then there is another occurrence of the same input line, cache crashes with util::EndOfFileException

We found out that util::Pool initializes with NULL, so *copy_to is NULL if got is an empty string.

preprocess/preprocess/cache_main.cc

Line 97 in 7f87245

 char *copy_to = (char*)string_pool.Allocate(got.size()); 

So, when the second occurrence of the first line is read, the check in Input function passes,

preprocess/preprocess/cache_main.cc

Lines 62 to 63 in 7f87245

 std::pair<std::unordered_map<uint64_t, StringPiece>::iterator, bool> res(cache.insert(entry)); 
 if (res.second) { 

,
but so does the one in Output:

preprocess/preprocess/cache_main.cc

Lines 91 to 93 in 7f87245

 while (queue.Consume(q).value) { 
   StringPiece &value = *q.value; 
   if (!value.data()) { 

value.data() returns false, even though the copy of the output was stored in the cache, and therefore the input line was not passed to the child process.

One workaround would be to allocate a byte before the loop, to make sure NULL pointer isn’t returned: char *small_init = (char*)string_pool.Allocate(1);

Alternatively, the check in the Input function could be changed to check the same thing as in the Output function, so change if (res.second) to if (!res.first->second.data()). This way, if the first line produces an empty output, it would be executed twice instead of once, but no errors.

[Kenneth Heafield]

Good point, followed your suggested fix.