Linker.hx is creating duplicate variable IDs

[Ryan Cleven]

As of recently, the hxsl splitter is removing my @param variables from my fragment shader.

It seems that the ID's are colliding, I'm getting a synthetic variable that matches the same ID as my param variable.

Because they share the same ID, when the second one gets marked as 'local' mine also does, which is then removed by the splitter later and no longer exposed to the rest of the shader pipeline.

HXSL variables

		@param var visibilityInfo:Buffer<Vec4, NUM_DISTANCES>; 
		@param var visCenter:Vec4;

These get marked as local within the hxsl.Splitter.get() function because the id collides with a synthetic variable. I'm tracking this down, but I thought I'd post here in case this was already known.

added a warning:

hxsl/Splitter.hx:269: WARNING: Found collision between B and visibilityInfo with id 74 with local false
hxsl/Splitter.hx:269: WARNING: Found collision between B and visibilityInfo with id 74 with local true

[Nicolas Cannasse]

You're not supposed to have two variables with same id in source shaders 🤔 Le jeu. 26 janv. 2023 à 23:04, Ryan Cleven ***@***.***> a écrit :

…

As of recently, the hxsl splitter is removing my @param <https://github.com/param> variables from my fragment shader. It seems that the ID's are colliding, I'm getting a synthetic variable that matches the same ID as my param variable. Because they share the same ID, when the second one gets marked as 'local' mine also does, which is then removed by the splitter later and no longer exposed to the rest of the shader pipeline. HXSL variables @param var visibilityInfo:Buffer<Vec4, NUM_DISTANCES>; @param var visCenter:Vec4; These get marked as local within the hxsl.Splitter.get() function because the id collides with a synthetic variable. I'm tracking this down, but I thought I'd post here in case this was already known. — Reply to this email directly, view it on GitHub <#1124>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHZXQGEPK7MZL5R44FGWBLWULYILANCNFSM6AAAAAAUIAMIKQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Ryan Cleven]

The variable is generated within the splitter, it's not in any source files.

hl bin/pipe.hl | grep "specified" | grep "B"

hxsl/Splitter.hx:48: Variable specified in incoming ShaderData:  v_Separate_RGB_003_Red : 600 : true
hxsl/Splitter.hx:48: Variable specified in incoming ShaderData:  v_Separate_RGB_003_Green : 601 : true
hxsl/Splitter.hx:48: Variable specified in incoming ShaderData:  v_Separate_RGB_003_Blue : 602 : true
hxsl/Splitter.hx:48: Variable specified in incoming ShaderData:  visB : 697 : true
hxsl/Splitter.hx:48: Variable specified in incoming ShaderData:  occB : 699 : true

The B variable comes from somewhere inside the functions included. So the duplicate ID's are likely upstream then. Yup, the collision was introduced upstream.

,{e : TVarDecl({id : 74, kind : Local, name : B, type : TVec(2,VFloat)}
,{e : TVar({id : 74, kind : Local, name : B, type : TVec(2,VFloat)}
,{parent : null, id : 74, kind : Param, name : visibilityInfo, type : TBuffer(TVec(4,VFloat),SConst(2048)), qualifiers : null}

the linker creates new variables, with the conflict:

hxsl/Linker.hx:162: New Variable [74 ] visibilityInfo allocated using allVars length
hxsl/Linker.hx:226: New Variable, Local [74] B

[Ryan Cleven]

Ok, I don't understand why this is happening but here is the problem:

I am linking my shader with a compiled shader and a dynamic shader.

var linked = linker.link([for (s in shaders) {
			trace('linking [${s.index}] ${s.name}');
			s.inst.shader;
		}
		]);

src/sd/tools/BlenderToHXSL.hx:386: linking [0] h3d.shader.BaseMesh
src/sd/tools/BlenderToHXSL.hx:386: linking [1] DynamicShader<HxslShader>
src/sd/tools/BlenderToHXSL.hx:386: linking [2] sd.shaders.VisibilityFilter
src/sd/tools/BlenderToHXSL.hx:386: linking [3] hxsl.Shader

The last one is the output shader. Because the checker is run at compile time on my visibility shader, the id's it generates aren't compatible with the dynamic one. I'm not sure what I'm doing wrong. I'm going to keep digging.

This was something that worked until a recent Heaps update.

[Ryan Cleven]

in linker.hx,

in

function allocVar( v : TVar, p : Position, ?shaderName : String, ?path : String, ?parent : AllocatedVar ) : AllocatedVar

why does the function allocate ID's using allVars length instead of getting a new UID?

var vid = allVars.length + 1;
		var v2 : TVar = {
			id : vid,
			name : vname,
			type : v.type,
			kind : v.kind,
			qualifiers : v.qualifiers,
			parent : parent == null ? null : parent.v,
		};

This is generating a clash with the ones coming from the UIDs. Changing it to

var vid = Tools.allocVarId();

Fixes the collision, but I am uncertain of what the knock-ons might be.

[Nicolas Cannasse]

I think that the idea is that the output should always be the same so each var should have a deterministic id. But I will check that again soon. Le ven. 27 janv. 2023 à 03:44, Ryan Cleven ***@***.***> a écrit :

…

in linker.hx, in function allocVar( v : TVar, p : Position, ?shaderName : String, ?path : String, ?parent : AllocatedVar ) : AllocatedVar why does the function allocate ID's using allVars length instead of getting a new UID? var vid = allVars.length + 1; var v2 : TVar = { id : vid, name : vname, type : v.type, kind : v.kind, qualifiers : v.qualifiers, parent : parent == null ? null : parent.v, }; This is generating a clash with the ones coming from the UIDs. — Reply to this email directly, view it on GitHub <#1124 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHZXQB2DTCSTTCNNE3I3J3WUMY7RANCNFSM6AAAAAAUIAMIKQ> . You are receiving this because you commented.Message ID: ***@***.***>

[Ryan Cleven]

I've put a simpler fix in here:

#1125

I just added an initial offset to the Tool.allocVarId() one so that it avoids colliding with the allVars one in the linker.

This should keep the output always the same.

[Nicolas Cannasse]

Could you compile with -D shader_debug_dump and set hxsl.Cache.DUMP_IDS = true then post the output of the shader/xxxx.c that matches your buggy shader ? this should give us a good overview of whats going wrong here regarding conflicting ids.

[Nicolas Cannasse]

Looking further into it, I don't think that variable id matters here so I went with your original fix.

[Ryan Cleven]

k. cool.