Edit call for `CreatePixelShader` in Direct3D9 unclear

[Chris Djali]

If I record a trace for a D3D9 application and view it in qapitrace, CreatePixelShader calls show DXBC assembly language (and it's shown the same way with command-line apitrace). However, if I use the Edit function to swap it out for alternative DXBC assembly, (e.g. simply writing a specific colour to the output register), and replay the trace again, the original shader is still used.

I've had a look at the edited trace file created in the temp directory, and the argument to the function is instead listed as a binary blob, and dumping it shows what I believe to be the assembled shader binary from the unmodified dump (as it's got a lot more going on than the simple shader that just writes a fixed colour to the output register).

I'm unsure whether this is a bug or I'm using the feature incorrectly. If necessary, I can provide the assembled binary for my replacement shader, but it's unclear how I'd do that.

[José Fonseca]

Editing D3D9 assembly shaders is not meant to work.

Perhaps the GUI should be smarter and throw up an error and/or mark the text as uneditable.

[Chris Djali]

Other uneditable fields have a padlock symbol and don't let you edit them, so it would definitely be clearer if that were consistent.

As for accomplishing my end goal, is there any way I can replace a shader in any other way? I have HLSL for what I want to use instead, so can compile that into the final binary if necessary. If there's no officially supported way, I've got a patch somewhere that lets me write uncompressed traces, so I could always hex-edit it in, but obviously that's a hassle.

[José Fonseca]

As for accomplishing my end goal, is there any way I can replace a shader in any other way?

This is the gist of the easiest way to accomplish this as an one-off hack:

diff --git a/lib/trace/trace_parser.cpp b/lib/trace/trace_parser.cpp
index fdb1adcf..16364b2d 100644
--- a/lib/trace/trace_parser.cpp
+++ b/lib/trace/trace_parser.cpp
@@ -943,6 +943,16 @@ void Parser::scan_opaque() {
 Value *Parser::parse_repr() {
     Value *humanValue = parse_value();
     Value *machineValue = parse_value();
+    if (next_call_no - 1 == REPLACE_CREATE_PIXEL_SHADER_CALL_NO_HERE) {
+        // ignore the human readable string as it won't match the inner blob
+        delete humanValue
+        // replace the machine used blob
+        delete machineValue;
+        // Read the replacement tokens somehow, from disk, or from a header generated by HLSL
+        uint8_t buf[] = ...;
+        machineValue = new Blob(buf, size);
+        return machineValue;
+    }
     return new Repr(humanValue, machineValue);
 }
 

Note that D3D9 tokenized shaders are serialized in two forms:

1. a human readable string, a disassembly of the shader
2. the machine bytes
   The UI only modifies the former, which is why it makes no difference when replaying.

The reason this is done this way, is so anyone can look into the disassembly, even from a non-Windows OS, and to speed up dumping too.

If one wants to make this functionality more general, then we should move it out of lib/trace/trace_parser.cpp and into cli/cli_sed.cpp, and add command line options to replace the blobs. On Windows one could even call HLSL assembler/disassembler, to recreate both the machine blob and human readable string.

[Chris Djali]

Thanks. Modifying that function seemed to work, although Blob didn't have a two-argument constructor. The implementation I used is

Value *Parser::parse_repr() {
    Value *humanValue = parse_value();
    Value *machineValue = parse_value();
    if (next_call_no - 1 == REPLACE_CREATE_PIXEL_SHADER_CALL_NO_HERE)
    {
        delete humanValue;
        delete machineValue;
        std::ifstream is("REPLACE_PATH_TO_COMPILED_SHADER_HERE");
        std::vector<uint8_t> shader((std::istreambuf_iterator<char>(is)), std::istreambuf_iterator<char>());
        
        machineValue = new Blob(shader.size());
        memcpy(machineValue->toPointer(), shader.data(), shader.size());
        return machineValue;
    }
    return new Repr(humanValue, machineValue);
}

[José Fonseca]

although Blob didn't have a two-argument constructor.

My bad. I wrote from memory. Your version looks right.

[jackchentwkh]

@AnyOldName3 do you have a branch with your modified code?
I am looking for modifying the shader codes when replaying the trace recorded with a target using d3d9.
most of graphic debuggers on win11 doesn't support d3d9 any more.
apitrace seems to be a good candidate, but lacking of the ability to modify the directx shader code is really a pitty.
if your solution works, it would be a great help for me, and maybe some other people trying to debug d3d9 target.

[Chris Djali]

I just copied and pasted the above snippet into the file: https://github.com/AnyOldName3/apitrace/tree/modified-parse_repr

[jackchentwkh]

really appreciate