Coherent mapping: wrong behavior on ID reuse

[stgatilov]

I stumbled upon something which looks like a bug in apitrace.

Here is the code catching the problem:

void repro() {
    vector<GLuint> buffers;

    for (int i = 0; i < 100; i++)
    {
        if (buffers.size() >= 30) {
            int j = rand() % buffers.size();
            GLuint id = buffers[j];
            glUnmapNamedBuffer(id);
            glDeleteBuffers(1, &id);
            buffers.erase(buffers.begin() + j);
        }
        {
            GLuint id;
            glCreateBuffers(1, &id);
            int size = 1000 + rand() % 30000;
            glNamedBufferStorage(id, size, nullptr, GL_MAP_READ_BIT | GL_MAP_WRITE_BIT | GL_MAP_PERSISTENT_BIT | GL_MAP_COHERENT_BIT);
            char *ptr = (char*)glMapNamedBufferRange(id, 0, size, GL_MAP_READ_BIT | GL_MAP_WRITE_BIT | GL_MAP_PERSISTENT_BIT | GL_MAP_COHERENT_BIT);
            memset(ptr, 0, size);
            buffers.push_back(id);
        }
    }

    for (GLuint id : buffers) {
        glUnmapNamedBuffer(id);
        glDeleteBuffers(1, &id);
    }
}

This test code is repeated every frame (full working example: bug_repro.zip).

When running the application by itself, it works fine.
But when running under apitrace recording, I get the following error instantly:

apitrace: error: VirtualProtect failed with error 0x1e7

---

The support for coherent mapping was added in #232.
On glBufferStorage, it creates a shadow buffer with protected pages and puts it into std::map by its GL name / ID:

    if ((flags & GL_MAP_COHERENT_BIT) && (flags & GL_MAP_WRITE_BIT)) {
        gltrace::Context *_ctx = gltrace::getContext();
        GLint buffer = getBufferName(target);
        auto memoryShadow = std::make_unique<GLMemoryShadow>();
        const bool success = memoryShadow->init(data, size);
        if (success) {
            _ctx->sharedRes->bufferToShadowMemory.insert(std::make_pair(buffer, std::move(memoryShadow)));
        } else {
            os::log("apitrace: error: %s: cannot create memory shadow\n", __FUNCTION__);
        }
    }

When the buffer is deleted though... nothing happens.
The shadow memory remains in the map by its ID, despite GL object being already dead.

Then later GL driver reuses the same ID for a new buffer of different size.
When we call glBufferStorage for this new buffer, the old shadow buffer entry is still in the map.
So the method std::map::insert has no effect --- it just returns {end, false} to indicate that insertion failed.
As the result, glMapBuffer tries to VirtualProtect the wrong shadow buffer of smaller size, which results in WinAPI error.

---

I believe there are two approaches to fix the problem:

1. Remove shadow buffer object from map when GL buffer is deleted.
2. Call erase by ID before inserting, turning this insert into overwrite.

[stgatilov]

Oh, this is duplicate of #830, and was recently fixed =)