Enum relation assert on windows with clang

Question

Enum relation assert on windows with clang

TBlauwe opened this issue 2 years ago · comments

Describe the bug
Adding an enum relation, e.g flecs example, assert on windows when building with clang (tested only with version 13.0) :

Here is the assert :
fatal: entity.c: 2027: assert: name_assigned == ecs_has_pair( world, result, ecs_id(EcsIdentifier), EcsName) INTERNAL_ERROR

To Reproduce
A minimal repo is available here : github repo.

Or, here is the code :

#include <flecs.h>
#include <iostream>

enum class TileStatus {
    Free,
    Occupied
};

int main(int, char* [])
{
    flecs::world ecs;

    ecs.component<TileStatus>(); //1
    auto tile = ecs.entity().add(TileStatus::Free); // 2 Also assert;
    std::cout << "Tile : " << tile.has<TileStatus>() << "\n";
}

Expected behavior
For the above code to not assert during line 1 or 2.

Additional context
Tested on master

The above code works on other configurations :

windows - msvc
linux - clang or gcc

In release mode, there seem to be no problem, the above code works. However, it seems in more complex configuration, (if we added reflection for example), the code crash (access error).

Also, here are the logs with flecs::log::set_level(3) :

info: component TileStatus created
info: id TileStatus created
info: id (TileStatus,*) created
info: id (*,TileStatus) created
info: table [Component, (Identifier,Name), (Identifier,Symbol), (OnDelete,Panic)] created with id 258
info: | table [Component, Exclusive, (Identifier,Name), (Identifier,Symbol), (OnDelete,Panic)] created with id 259
info: | table [Component, Exclusive, OneOf, (Identifier,Name), (Identifier,Symbol), (OnDelete,Panic)] created with id 260
info: | table [Component, Tag, Exclusive, OneOf, (Identifier,Name), (Identifier,Symbol), (OnDelete,Panic)] created with id 261
fatal: entity.c: 1933: assert: name_assigned == ecs_has_pair( world, result, ecs_id(EcsIdentifier), EcsName) INTERNAL_ERROR

TBlauwe · Answer 1 · Fri May 13 2022 19:19:25 GMT+0800 (China Standard Time)

Here is the cmake output :

1> Ligne de commande : "C:\WINDOWS\system32\cmd.exe" /c "%SYSTEMROOT%\System32\chcp.com 65001 >NUL && "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\CMAKE\CMake\bin\cmake.exe"  -G "Ninja"  -DCMAKE_C_COMPILER:STRING="clang-cl.exe" -DCMAKE_CXX_COMPILER:STRING="clang-cl.exe" -DCMAKE_BUILD_TYPE:STRING="Debug" -DCMAKE_INSTALL_PREFIX:PATH="D:/dev/cpp/TestBench/out/install/x64-debug"  -DCMAKE_MAKE_PROGRAM="C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\COMMUNITY\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\CMAKE\Ninja\ninja.exe" "D:\dev\cpp\TestBench" 2>&1"
1> Répertoire de travail : D:/dev/cpp/TestBench/out/build/x64-debug
1> [CMake] -- The C compiler identification is Clang 13.0.0 with MSVC-like command-line
1> [CMake] -- The CXX compiler identification is Clang 13.0.0 with MSVC-like command-line
1> [CMake] -- Detecting C compiler ABI info
1> [CMake] -- Detecting C compiler ABI info - done
1> [CMake] -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/Llvm/x64/bin/clang-cl.exe - skipped
1> [CMake] -- Detecting C compile features
1> [CMake] -- Detecting C compile features - done
1> [CMake] -- Detecting CXX compiler ABI info
1> [CMake] -- Detecting CXX compiler ABI info - done
1> [CMake] -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/Llvm/x64/bin/clang-cl.exe - skipped
1> [CMake] -- Detecting CXX compile features
1> [CMake] -- Detecting CXX compile features - done
1> [CMake] -- Looking for pthread.h
1> [CMake] -- Looking for pthread.h - not found
1> [CMake] -- Found Threads: TRUE  
1> [CMake] -- Configuring done
1> [CMake] -- Generating done
1> [CMake] -- Build files have been written to: D:/dev/cpp/TestBench/out/build/x64-debug
1> Variables CMake extraites.
1> Fichiers sources et en-têtes extraits.
1> Modèle de code extrait.
1> Extraction effectuée des configurations de chaîne d'outils.
1> Chemins include extraits.
1> Fin de la génération de CMake.

Sander Mertens · Answer 2 · Fri May 27 2022 04:30:23 GMT+0800 (China Standard Time)

Looks like this is a clang 13 issue, looking into it

Sander Mertens · Answer 3 · Fri May 27 2022 05:09:07 GMT+0800 (China Standard Time)

Yep, clang 13 changed the format of __PRETTY_FUNCTION__ for invalid enum values:
clang 13: https://godbolt.org/z/W45PseeGE
clang 12: https://godbolt.org/z/fEorG9evr

Sander Mertens · Answer 4 · Fri May 27 2022 07:08:44 GMT+0800 (China Standard Time)

Fixed! Turns out Apple clang was one minor version behind llvm clang.

TBlauwe · Answer 5 · Tue Jul 05 2022 20:58:43 GMT+0800 (China Standard Time)

Hello ! Sorry for reopening this issue. The code still asserts on Windows with clang 13 in debug mode.

In release mode, tile.has<TileStatus>(); returns 1. And compared to before, it doesn't assert anymore if we add reflection to the enum component. But, in debug mode, there is still the same assert.

Following this comment : #716 (comment)
I return the same output than clang 13 on my setup. So maybe there is another problem ?

To tried to pinpoint where the problem is, I tried to follow the behaviour of ecs.component<TileStatus>(), between msvc and clang, but my understanding is limited.

The only difference in behaviour I could find is in this piece of code in enum.hpp :

    // 1
    template <E Value, flecs::if_not_t< enum_constant_is_valid<E, Value>() > = 0>
    static void init_constant(flecs::world_t*) { std::cout << "Hello" << ; }

    // 2
    template <E Value, flecs::if_t< enum_constant_is_valid<E, Value>() > = 0>
    static void init_constant(flecs::world_t *world) {
        int v = to_int<Value>();
        const char *name = enum_constant_to_name<E, Value>();
        data.constants[v].next = data.min;
        data.min = v;
        if (!data.max) {
            data.max = v;
        }

        data.constants[v].id = ecs_cpp_enum_constant_register(
            world, data.id, data.constants[v].id, name, v);
    }

    template <E Value = FLECS_ENUM_MAX(E) >
    static void init(flecs::world_t *world) {
        init_constant<Value>(world); // here
        if (is_not_0<Value>()) {
            init<from_int<to_int<Value>() - is_not_0<Value>()>()>(world);
        }
    }

During ìnit_constant<Value>(world), msvc goes to the first function template, while clang goes to the second. It then asserts during data.constants[v].id = ecs_cpp_enum_constant_register(world, data.id, data.constants[v].id, name, v);.

You'll find attached a screenshot of local variables during this call.

Should name be equal to 128 ?

Hope it can helps !

Sander Mertens · Answer 6 · Tue Aug 23 2022 14:00:04 GMT+0800 (China Standard Time)

Circling back to this, I couldn't reproduce on clang 14/Linux but couldn't.

Could you try to run and compile this code with clang?

#include <stdio.h>

enum SparseEnum {
    Black = 1, White = 3, Grey = 5
};

template <typename E, E C>
void enum_constant() {
    printf("%s\n", __PRETTY_FUNCTION__);
}

int main()
{
    enum_constant<SparseEnum, Black>();
    enum_constant<SparseEnum, (SparseEnum)0>();

    return 0;
}

The number 128 showing up in that function as constant name suggests that there could be something wrong with the logic that detects if a valid constant is found (the pretty_function macro should return names for valid constants, and numbers for invalid constants).

TBlauwe · Answer 7 · Tue Aug 23 2022 15:53:42 GMT+0800 (China Standard Time)

I tested this code on clang 13 and 14 (on windows) and it works correctly :

void enum_constant() [E = SparseEnum, C = Black]
void enum_constant() [E = SparseEnum, C = (SparseEnum)0]

I'll investigate more !

TBlauwe · Answer 8 · Tue Aug 23 2022 16:07:09 GMT+0800 (China Standard Time)

So when calling ecs.component<TileStatus>() :

During component.hpp:151 static entity_t id_explicit :
at line 180, symbol is equal to "TileStatus"

Later on, we end up in flecs_cpp.c :

char* ecs_cpp_get_constant_name(
    char *constant_name,
    const char *func_name,
    size_t func_name_len)
{
    ecs_size_t f_len = flecs_uto(ecs_size_t, func_name_len);
    const char *start = cpp_func_rchr(func_name, f_len, ' ');
    start = cpp_func_max(start, cpp_func_rchr(func_name, f_len, ')'));
    start = cpp_func_max(start, cpp_func_rchr(func_name, f_len, ':'));
    start = cpp_func_max(start, cpp_func_rchr(func_name, f_len, ','));
    ecs_assert(start != NULL, ECS_INVALID_PARAMETER, func_name);
    start ++;
    
    ecs_size_t len = flecs_uto(ecs_size_t, 
        (f_len - (start - func_name) - flecs_uto(ecs_size_t, ECS_FUNC_NAME_BACK)));
    ecs_os_memcpy_n(constant_name, start, char, len);
    constant_name[len] = '\0';
    return constant_name;
}

and func_name is equal to :

const char *flecs::_::enum_constant_to_name() [E = TileStatus, C = (TileStatus)128]

and afterwards, 128 get stripped.

If it's more convenient for you, I can share my screen on the discord server (if you want to direct me when debugging).

My intuition was there was something wrong in this piece of code :

#if defined(__clang__)
#if __clang_major__ < 13 || (defined(__APPLE__) && __clang_minor__ < 1)
template <typename E, E C>
constexpr bool enum_constant_is_valid() {
    return !(
        (ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(bool, enum_constant_is_valid) +
            enum_type_len<E>() + 6 /* ', C = ' */] >= '0') &&
        (ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(bool, enum_constant_is_valid) +
            enum_type_len<E>() + 6 /* ', C = ' */] <= '9'));
}
#else
template <typename E, E C>
constexpr bool enum_constant_is_valid() {
    return (ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(bool, enum_constant_is_valid) +
        enum_type_len<E>() + 6 /* ', E C = ' */] != '(');
}
#endif
#elif defined(__GNUC__)
template <typename E, E C>
constexpr bool enum_constant_is_valid() {
    return (ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(constepxr bool, enum_constant_is_valid) +
        enum_type_len<E>() + 8 /* ', E C = ' */] != '(');
}
#else
/* Use different trick on MSVC, since it uses hexadecimal representation for
 * invalid enum constants. We can leverage that msvc inserts a C-style cast
 * into the name, and the location of its first character ('(') is known. */
template <typename E, E C>
constexpr bool enum_constant_is_valid() {
    return ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(bool, enum_constant_is_valid) +
        enum_type_len<E>() + 1] != '(';
}
#endif

Like maybe we do not enter the correct version on windows (see : https://stackoverflow.com/questions/28017400/why-isnt-clang-defined-when-using-llvmclang-in-visual-studio).

For example, when running this code (clang and windows):

int main()
{
#ifdef _MSC_VER
    std::cout << "_MSC_VER is defined\n"; // Is called
#endif
#ifdef __clang__
    std::cout << "__clang__ is defined\n"; // Is also called
#endif
}

Maybe the fact that both are true is creating bug somewhere else ? Like maybe for this (enum.hpp):

#ifdef ECS_TARGET_MSVC
#define ECS_SIZE_T_STR "unsigned __int64" // This is defined
#elif defined(__clang__)
#define ECS_SIZE_T_STR "size_t" // and not this
#else
#define ECS_SIZE_T_STR "constexpr size_t; size_t = long unsigned int"
#endif

TBlauwe · Answer 9 · Tue Aug 23 2022 16:32:25 GMT+0800 (China Standard Time)

Well I think I found the culprit. From last comment, I pointed that _MSC_VER is defined when using clang on windows. __clang__ is also defined.

Undefing _MSC_VER creates many compiler errors.

In this piece of code :

#ifdef ECS_TARGET_MSVC
#define ECS_SIZE_T_STR "unsigned __int64" // This is defined
#elif defined(__clang__)
#define ECS_SIZE_T_STR "size_t" // and not this
#else
#define ECS_SIZE_T_STR "constexpr size_t; size_t = long unsigned int"
#endif

MSVC is checked first, clang afterwards. Which is not the case for :

#if defined(__clang__)
#if __clang_major__ < 13 || (defined(__APPLE__) && __clang_minor__ < 1)
template <typename E, E C>
constexpr bool enum_constant_is_valid() {
    return !(
        (ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(bool, enum_constant_is_valid) +
            enum_type_len<E>() + 6 /* ', C = ' */] >= '0') &&
        (ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(bool, enum_constant_is_valid) +
            enum_type_len<E>() + 6 /* ', C = ' */] <= '9'));
}
#else
template <typename E, E C>
constexpr bool enum_constant_is_valid() {
    return (ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(bool, enum_constant_is_valid) +
        enum_type_len<E>() + 6 /* ', E C = ' */] != '(');
}
#endif
#elif defined(__GNUC__)
template <typename E, E C>
constexpr bool enum_constant_is_valid() {
    return (ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(constepxr bool, enum_constant_is_valid) +
        enum_type_len<E>() + 8 /* ', E C = ' */] != '(');
}
#else
/* Use different trick on MSVC, since it uses hexadecimal representation for
 * invalid enum constants. We can leverage that msvc inserts a C-style cast
 * into the name, and the location of its first character ('(') is known. */
template <typename E, E C>
constexpr bool enum_constant_is_valid() {
    return ECS_FUNC_NAME[ECS_FUNC_NAME_FRONT(bool, enum_constant_is_valid) +
        enum_type_len<E>() + 1] != '(';
}
#endif

So I thought that maybe by switching so that clang is checked first and msvc afterwards, it would solve the problem :

#if defined(__clang__)
#define ECS_SIZE_T_STR "size_t"
#elif defined(ECS_TARGET_MSVC)
#define ECS_SIZE_T_STR "unsigned __int64"
#else
#define ECS_SIZE_T_STR "constexpr size_t; size_t = long unsigned int"
#endif

Turns out it works now ! I haven't done an extensive check to see if there was other places where this would be a problem. I tried this :

(api_defines.h)

#if defined(_MSC_VER) && !defined(__clang__)
#define ECS_TARGET_MSVC
#endif

But there are many compiler errors.

Sander Mertens · Answer 10 · Tue Aug 23 2022 18:51:56 GMT+0800 (China Standard Time)

Thanks for looking into this! That sounds like the cause of the problem, I'll see if I can come up with a fix :)

Sander Mertens · Answer 11 · Tue Aug 23 2022 19:14:06 GMT+0800 (China Standard Time)

Can you try if this branch fixes the issue? https://github.com/SanderMertens/flecs/tree/clang_windows_fix

TBlauwe · Answer 12 · Tue Aug 23 2022 19:38:45 GMT+0800 (China Standard Time)

I've got some compiler errors :

There are ~~some typos in the commit~~ changes I needed to make, in order to build it (see below). It works great afterwards, thank you 🥳 !

In file http.c :

line 173

#ifndef ECS_TARGET_MSVC // Shouldn't this be ECS_TARGET_WINDOWS now ?
    ssize_t send_bytes = send(sock, buf, flecs_itosize(size), flags);
    return flecs_itoi32(send_bytes);
#else
    int send_bytes = send(sock, buf, size, flags);
    return flecs_itoi32(send_bytes);
#endif

and line 190

#ifndef ECS_TARGET_MSVC // idem
    ssize_t recv_bytes = recv(sock, buf, flecs_itosize(size), flags);
    ret = flecs_itoi32(recv_bytes);
#else
    int recv_bytes = recv(sock, buf, size, flags);
    ret = flecs_itoi32(recv_bytes);
#endif
    if (ret == -1) {
        ecs_dbg("recv failed: %s (sock = %d)", ecs_os_strerror(errno), sock);
    } else if (ret == 0) {
        ecs_dbg("recv: received 0 bytes (sock = %d)", sock);
    }

In file hash.c :
line 24

#ifdef ECS_TARGET_MSVC // idem ?
//FIXME
#else
#include <sys/param.h>  /* attempt to define endianness */
#endif
#ifdef ECS_TARGET_LINUX
# include <endian.h>    /* attempt to define endianness */
#endif

EDIT : now that I think about it, it may not be this simple. I thought ECS_TARGET_MSVC was renamed to ECS_TARGET_WINDOWS, but it's another define ? I haven't checked for other configurations (plaforms + compiler), but with these changes, the code compiles and run correctly on windows with clang 13+

Sander Mertens · Answer 13 · Fri Aug 26 2022 17:41:54 GMT+0800 (China Standard Time)

Yup you're right. The ECS_TARGET_WINDOWS macro should be defined when on Windows, the ECS_TARGET_MSVC macro should only be defined when compiling with the microsoft compiler.

I just checked in an update to the branch which I think follows your suggestions, let me know if it works!

TBlauwe · Answer 14 · Fri Aug 26 2022 17:48:24 GMT+0800 (China Standard Time)

Yep, it works (on clang 14 and windows) ! Thank you 🥳 !

Sander Mertens · Answer 15 · Fri Aug 26 2022 17:52:12 GMT+0800 (China Standard Time)

Awesome! I'll merge it to master then. Thanks for your help!