rAzoR8 / SpvGenTwo

SpvGenTwo is a SPIR-V building and parsing library written in plain C++17 without any dependencies. No STL or other 3rd-Party library needed.

Home Page:https://twitter.com/SiNGUL4RiTY

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

SpvGenTwo

SpvGenTwo is a SPIR-V building and parsing library written in C++17, no other dependencies! No STL or other 3rd-Party library needed. The library comes with its own set of SPIR-V definitions generated from the machine readable grammar and therefore does not require any vulkan or spirv-headers includes. The generator can be found here: rustspvgen.

I built this library as a 'slim' backend for runtime material/shader-editors (like Proto) to avoid introducing enormous libraries like DXC (including LLVM and Frontend) or SPIRV-Tools to the codebase.

SpvGenTwo is still under development, many parts are still missing, but all the building blocks are there. SpvGenTwo is designed to be extensible and customizable, it is fairly easy to implement new SPIR-V instructions and extensions, use custom allocators and define own type inference rules. Note that it is still possible to generate invalid SPIR-V modules because not all inputs are checked yet. Use the SPIR-V validator spirv-val from the SDK and have the specification at hand while using this library.

I mainly focused on Shader capabilities, so the Kernel and OpenCL side is a bit under-developed. Any community contributions in that regard are very welcome!

Overview:

Examples

ConsoleLogger log;
HeapAllocator alloc; // custom user allocator

// create a new spir-v module
Module module(&alloc, &log);

// configure capabilities and extensions
module.addCapability(spv::Capability::Shader);
module.addCapability(spv::Capability::VulkanMemoryModelKHR);
module.addExtension(spv::Extension::SPV_KHR_vulkan_memory_model);
Instruction* ext = module.getExtensionInstructionImport(u8"GLSL.std.450");
module.setMemoryModel(spv::AddressingModel::Logical, spv::MemoryModel::VulkanKHR);

// global variables
Instruction* uniformVar = module.uniform<vector_t<float, 3>>(u8"u_Position");

// float add(float x, float y)
Function& funcAdd = module.addFunction<float, float, float>(u8"add", spv::FunctionControlMask::Const);
{
    BasicBlock& bb = *funcAdd; // get entry block to this function

    Instruction* x = funcAdd.getParameter(0);
    Instruction* y = funcAdd.getParameter(1);

    Instruction* z = bb.Add(x, y);
    bb.returnValue(z);
}

// void entryPoint();
{
    EntryPoint& entry = module.addEntryPoint(spv::ExecutionModel::Fragment, u8"main");
    entry.addExecutionMode(spv::ExecutionMode::OriginUpperLeft);
    BasicBlock& bb = *entry; // get entry block to this function

    Instruction* uniVec = bb->opLoad(uniformVar);
    Instruction* cross = bb.ext<GLSL>()->opCross(uniVec, uniVec); // use GLSL.std.450 extension
    Instruction* s = bb->opDot(cross, uniVec);
    entry->call(&funcAdd, s, s); // call add(s, s)
    entry->opReturn();
}

// custom spir-v binary serializer:
BinaryFileWriter writer("test.spv");
module.finalizeAndWrite(writer);

The resulting SPIR-V binary when disassembled using spirv-dis:

; SPIR-V
; Version: 1.0
; Generator: SpvGenTwo SPIR-V IR Tools(30); 0
; Bound: 20
; Schema: 0
               OpCapability Shader
               OpCapability VulkanMemoryModelKHR
               OpExtension "SPV_KHR_vulkan_memory_model"
          %1 = OpExtInstImport "GLSL.std.450"
               OpMemoryModel Logical VulkanKHR
               OpEntryPoint Fragment %main "main" %u_Position
               OpExecutionMode %main OriginUpperLeft
               OpName %u_Position "u_Position"
               OpName %add "add"
               OpName %main "main"
      %float = OpTypeFloat 32
    %v3float = OpTypeVector %float 3
%_ptr_Uniform_v3float = OpTypePointer Uniform %v3float
          %9 = OpTypeFunction %float %float %float
       %void = OpTypeVoid
          %3 = OpTypeFunction %void
 %u_Position = OpVariable %_ptr_Uniform_v3float Uniform
        %add = OpFunction %float Const %9
         %11 = OpFunctionParameter %float
         %12 = OpFunctionParameter %float
         %13 = OpLabel
         %14 = OpFAdd %float %11 %12
               OpReturnValue %14
               OpFunctionEnd
       %main = OpFunction %void None %3
         %15 = OpLabel
         %16 = OpLoad %v3float %u_Position None
         %17 = OpExtInst %v3float %1 Cross %16 %16
         %18 = OpDot %float %17 %16
         %19 = OpFunctionCall %float %add %18 %18
               OpReturn
               OpFunctionEnd

Test Project

Set CMake option SPVGENTWO_BUILD_TESTS to TRUE to build included examples:

Project Structure

SpvGenTwo is split into 5 folders:

  • lib contains the foundation to generate SPIR-V code. SpvGenTwo makes excessive use of its allocator interface, no memory is allocated from the heap. SpvGenTwo comes with its on set of container classes: List, Vector, String and HashMap. Those are not built for performance, but they shouldn't be much worse than standard implementations (okay maybe my HashMap is not as fast as unordered_map, build times are quite nice though :). Everything within this folders is pure C++17, no other dependencies (given that SPVGENTWO_REPLACE_PLACEMENTNEW and SPVGENTWO_REPLACE_TRAITS are used).
  • common contains some convenience implementations of abstract interfaces: HeapAllocator uses C malloc and free, BindaryFileWriter uses fopen, ConsoleLogger uses vprintf, ModulePrinter uses snprintf. It also has some additional classes like Callable (std::function replacement), Graph, ControlFlowGraph, Expression and ExprGraph, they follow the same design principles and might sooner or later be moved to lib if needed.
  • test contains small, self-contained code snippets that each generate a SPIR-V module to show some of the fundamental mechanics and APIs of SpvGenTwo.
  • dis is a SPIR-V disassembler tool like spirv-dis to print assembly language text.
  • refl is a SPIR-V reflection tool like SPIRV-Reflect to extract descriptor bindings and other relevant info from SPIR-V binary modules.
  • link is a SPIR-V linker like spirv-link for merging symbols of multiple modules into a new output module.
  • test contains Catch2 unit tests. When using Visual Studio, Test Adapter for Catch2 can be used with the test/catch2.runsettings config file.

Building

Use the supplied CMakeLists.txt to generate project files for your build system. SpvGenTwo allows the user to use standard library headers (<type_traits>, <new> etc) instead of my hand-made replacements (see stdreplament.h).

  • SPVGENTWO_BUILD_TESTS is set to FALSE by default. If TRUE, an executable with sources from the 'test' folder will be built.
    • Note that the SpvGenTwoTest executable project depends on SPIR-V Tools which requires python to be present for its tests (even when building without tests facepalm)
  • SPVGENTWO_BUILD_DISASSEMBLER is set to FALSE by default. If TRUE, an executable with sources from the 'dis' folder will be built.
  • SPVGENTWO_BUILD_REFLECT is set to FALSE by default. If TRUE, an executable with sources from the 'refl' folder will be built.
  • SPVGENTWO_BUILD_LINKER is set to FALSE by default. If TRUE, an executable with sources from the 'link' folder will be built.
  • SPVGENTWO_REPLACE_PLACEMENTNEW is set to TRUE by default. If FALSE, placement-new will be included from <new> header.
  • SPVGENTWO_REPLACE_TRAITS is set to TRUE by default. If FALSE, <type_traits> and <utility> header will be included under spvgentwo::stdrep namespace.
  • SPVGENTWO_LOGGING is set to TRUE by default, calls to module.log() will have not effect if FALSE.
  • SPVGENTWO_ENABLE_WARNINGS is set to TRUE by default and will enable most pedantic warnings-as-errors for all targets except the tests.
  • SPVGENTWO_ENABLE_OPERANDVALIDATION is set to TRUE by default and enables an additional validation step for every makeOp/opXZY call which is not necessary for non-development builds.

Note that I mainly develop on Windows using Clang and MSVC but I'll also try to support GCC/linux. I don't have any Apple hardware so I can't debug any issues there, but you are welcome to contribute fixes for this platform.

Tools

SpvGenTwo includes a couple of CLI tools to explore and test the libraries capabilities.

SPIR-V Disassembler

SpvGenTwoDisassembler

SpvGenTwoDisassembler source can be found at dis/source/dis.cpp, it is just a single source file demonstrating the basic parsing and IR walking capabilities of SpvGenTwo.

CLI: SpvGenTwoDisassembler [file] <option> <option> ...

Options

  • -assignids re-assigns instruction result IDs starting from 1. Some SPIR-V compilers emit IDs in a very high range, making it hard to read and trace data flow in assembly language text, assignIDs helps with that.
  • -serialize writes the parsed SPIR-V program to a serialized.spv file in the working directory (this is a debug feature).
  • -noinstrnames don't replace result IDs with OpNames
  • -noopnames don't replace operand IDs with OpNames
  • -nopreamble don't print SPIR-V preamble
  • -colors use ANSI color codes
  • -tabs " " example: use 4 spaces instead of 2 tabs

SPIR-V Reflector

SpvGenTwoReflect

SpvGenTwoReflect source can be found at refl/source/refl.cpp

CLI: SpvGenTwoReflect [file] <option> <option> ...

Options

  • -var name select variable by name (if OpVariable was annotated by OpName) for DescriptorType & Decoration printing (name has to be a UTF-8 string)
    • -var MyBuffer
  • -deco decoration select decoration to query for in the module
    • -deco print all decorations in the module
    • -deco DescriptorSet to print only DescriptorSets
    • -deco DescriptorSet -deco Binding to print only DescriptorSets & Bindings
  • -funcs list functions names in the module
  • -vars list global variables in the module (StorageClass != Function)
  • -types list types and constatns in the module
  • -id Id print SPIR-V assembly text for the instruction with result Id
    • -id 24
  • -colors use ANSI color codes

SPIR-V Linker

SpvGenTwoLinker

SPIR-V Shader library linker and patcher. See SpvGenTwoLinker for detailed description. Source can be found at link/source/link.cpp.

Documentation

Please read the documentation for more detailed information on how to use SpvGenTwo and some reasoning about my design choices.

Contributing

Please follow the "fork-and-pull" Git workflow:

  • Fork the repo on GitHub
  • Clone the project to your own machine
  • Commit changes to your own branch
  • Push your work back up to your fork
  • Submit a Pull request so that we can review your changes

Make sure to merge the latest from "upstream" before making a pull request.

Copyright and Licensing

The SpvGenTwo open source project is licensed under MIT license. Any contribution you make to this original repository shall be licensed under same license. You are still free to distribute your contributions (in your own fork) under the license you desire.

Roadmap

A list of short and long term goals for this library:

  • Implement more Instructions, at least 90% of Shader capabilities
  • Improve validation / logging / error reporting
  • Write more unit tests
  • Implement SPIRV-Tools like helper tools
  • Implement some front-end or DSL like SPEAR

Gallery

SpvGenTwo is used in:

Leave PR with your software if you want it to be added here :)

Coverage

SPIR-V IR generation progress, parsing is independent and auto generated. This table indicates whether the operation is already implemented in the Instruction class or can be generated by other facilities. This is not a complete list, if an entry is missing, assume it has not been implemented yet. You can file an Issue to request its implementation or just use Instruction::makeOp(...) directly.

Core

Instruction Implemented
OpNop
OpUndef
OpSourceContinued
OpSource
OpSourceExtension
OpName
OpMemberName
OpString
OpLine
OpExtension
OpExtInstImport
OpExtInst
OpMemoryModel
OpEntryPoint
opExecutionMode
OpCapability
OpExecutionModeId
OpTypeXXX via Module::addType()/type<T>
OpConstantXXX via Module::addConstant()/constant<T>(T t)
OpSpecConstantXXX via Module::addConstant()/constant<T>(T t)
OpSpecConstantOp
OpFunction
OpFunctionParameter
OpFunctionEnd
OpFunctionCall
OpVariable
OpImageTexelPointer
OpLoad
OpStore
OpCopyMemory
OpCopyMemorySized
OpAccessChain
OpInBoundsAccessChain
OpPtrAccessChain
OpArrayLength
OpGenericPtrMemSemantics
OpInBoundsPtrAccessChain
OpDecorate
OpMemberDecorate
OpDecorationGroup Deprecated
OpGroupDecorate Deprecated
OpGroupMemberDecorate Deprecated
OpVectorExtractDynamic
OpVectorInsertDynamic
OpVectorShuffle
OpCompositeConstruct
OpCompositeExtract
OpCompositeInsert
OpCopyObject
OpTranspose
OpSampledImage
OpImageSampleImplicitLod
OpImageSampleExplicitLod
OpImageSampleDrefImplicitLod
OpImageSampleDrefExplicitLod
OpImageSampleProjImplicitLod
OpImageSampleProjExplicitLod
OpImageSampleProjDrefImplicitLod
OpImageSampleProjDrefExplicitLod
OpImageFetch
OpImageGather
OpImageDrefGather
OpImageRead
OpImageWrite
OpImage
OpImageQueryFormat
OpImageQueryOrder
OpImageQuerySizeLod
OpImageQuerySize
OpImageQueryLod
OpImageQueryLevels
OpImageQuerySamples
OpConvertFToU
OpConvertFToS
OpConvertSToF
OpConvertUToF
OpUConvert
OpSConvert
OpFConvert
OpQuantizeToF16
OpConvertPtrToU
OpSatConvertSToU
OpSatConvertUToS
OpConvertUToPtr
OpPtrCastToGeneric
OpGenericCastToPtr
OpGenericCastToPtrExplicit
OpBitcast
OpSNegate
OpFNegate
OpIAdd
OpFAdd
OpISub
OpFSub
OpIMul
OpFMul
OpUDiv
OpSDiv
OpFDiv
OpUMod
OpSRem
OpSMod
OpFRem
OpFMod
OpVectorTimesScalar
OpMatrixTimesScalar
OpVectorTimesMatrix
OpMatrixTimesVector
OpMatrixTimesMatrix
OpOuterProduct
OpDot
OpIAddCarry
OpISubBorrow
OpUMulExtended
OpSMulExtended
OpAny
OpAll
OpIsNan
OpIsInf
OpIsFinite
OpIsNormal
OpSignBitSet
OpLessOrGreater Deprecated
OpOrdered
OpUnordered
OpLogicalEqual
OpLogicalNotEqual
OpLogicalOr
OpLogicalAnd
OpLogicalNot
OpSelect
OpIEqual
OpINotEqual
OpUGreaterThan
OpSGreaterThan
OpUGreaterThanEqual
OpSGreaterThanEqual
OpULessThan
OpSLessThan
OpULessThanEqual
OpSLessThanEqual
OpFOrdEqual
OpFUnordEqual
OpFOrdNotEqual
OpFUnordNotEqual
OpFOrdLessThan
OpFUnordLessThan
OpFOrdGreaterThan
OpFUnordGreaterThan
OpFOrdLessThanEqual
OpFUnordLessThanEqual
OpFOrdGreaterThanEqual
OpFUnordGreaterThanEqual
OpShiftRightLogical
OpShiftRightArithmetic
OpShiftLeftLogical
OpBitwiseOr
OpBitwiseXor
OpBitwiseAnd
OpNot
OpBitFieldInsert
OpBitFieldSExtract
OpBitFieldUExtract
OpBitReverse
OpBitCount
OpDPdx
OpDPdy
OpFwidth
OpDPdxFine
OpDPdyFine
OpFwidthFine
OpDPdxCoarse
OpDPdyCoarse
OpFwidthCoarse
OpEmitVertex
OpEndPrimitive
OpEmitStreamVertex
OpEndStreamPrimitive
OpControlBarrier
OpMemoryBarrier
OpAtomicLoad
OpAtomicStore
OpAtomicExchange
OpAtomicCompareExchange
OpAtomicCompareExchangeWeak
OpAtomicIIncrement
OpAtomicIDecrement
OpAtomicIAdd
OpAtomicISub
OpAtomicSMin
OpAtomicUMin
OpAtomicSMax
OpAtomicUMax
OpAtomicAnd
OpAtomicOr
OpAtomicXor
OpPhi
OpLoopMerge
OpSelectionMerge
OpLabel
OpBranch
OpBranchConditional
OpSwitch
OpKill
OpReturn
OpReturnValue
OpUnreachable
OpNoLine
OpAtomicFlagTestAndSet
OpAtomicFlagClear
OpImageSparseRead
OpSizeOf
OpTypePipeStorage via Module::addType()
OpConstantPipeStorage via Module::addConstant()
OpCreatePipeFromPipeStorage
OpGetKernelLocalSizeForSubgroupCount
OpGetKernelMaxNumSubgroups
OpTypeNamedBarrier via Module::addType()
OpNamedBarrierInitialize
OpModuleProcessed
OpExecutionModeId
OpDecorateId
OpPtrEqual
OpPtrNotEqual
... ...

GLSL.std.450

Instruction Implemented
Round
RoundEven
Trunc
FAbs
SAbs
FSign
SSign
Floor
Ceil
Fract
Radians
Degrees
Sin
Cos
Tan
Asin
Acos
Atan
Sinh
Cosh
Tanh
Asinh
Acosh
Atanh
Atan2
Pow
Exp
Log
Exp2
Log2
Sqrt
InverseSqrt
Determinant
MatrixInverse
Modf Deprecated
ModfStruct
FMin
UMin
SMin
FMax
UMax
SMax
FClamp
UClamp
SClamp
FMix
IMix Removed in v0.99, Revision 3
Step
SmoothStep
Fma
Frexp Deprecated
FrexpStruct
Ldexp
PackSnorm4x8
PackUnorm4x8
PackSnorm2x16
PackUnorm2x16
PackHalf2x16
PackDouble2x32
UnpackSnorm2x16
UnpackUnorm2x16
UnpackHalf2x16
UnpackSnorm4x8
UnpackUnorm4x8
UnpackDouble2x32
Length
Distance
Cross
Normalize
FaceForward
Reflect
Refract
FindILsb
FindSMsb
FindUMsb
InterpolateAtCentroid
InterpolateAtSample
InterpolateAtOffset
NMin
NMax
NClamp

OpenCl.std.100

Not implemented yet

About

SpvGenTwo is a SPIR-V building and parsing library written in plain C++17 without any dependencies. No STL or other 3rd-Party library needed.

https://twitter.com/SiNGUL4RiTY

License:MIT License


Languages

Language:C++ 99.5%Language:CMake 0.4%Language:C 0.0%Language:Batchfile 0.0%