Gamepad / Keyboard navigation and interactions!

Question

Gamepad / Keyboard navigation and interactions!

ocornut opened this issue 8 years ago · comments

EDIT: The navigation branch has been merged to master. See #1599. Further improvements/fixes will be pushed to mainline.

Opening a new thread because the old one ( #323 ) became overwhelmingly long and unwelcoming. This is a trimmed post with info that are still relevant.

TL;DR; there is a new branch that add supports for gamepad/joystick navigation. The same system can be used for keyboard navigation to some degree, but the initial focus is on gamepad. Typically you can use that to access your tools on PS4/XBone/Wii-U/etc without a synergy/mouse/keyboard setup. Best if you can still have even a virtual mouse around (e.g. on DualShock4 touch pad).

EDIT March 2018
Link to PNG + PSD depicting the controls for DualShock 4 and Joy-Con
https://drive.google.com/open?id=1k4328OV-w20pWZfNcfpH0UoxHHQRkbQi

EDIT: Now in master!
Branch: https://github.com/ocornut/imgui/tree/navigation
(checkout branch navigation or download from web and overwrite your imgui_xxx files)

I'm calling it beta because:

This feature required changing lots of code. The branch probably has bugs.
It is rough and work in progress. The more I add and fix things the more I see new things to do. My initial schedule projection was a joke. Long tail feature.
But it is pretty useful already!
I would ideally like to merge this in master but I can only do so with more testing and feedback.
Even if you don't need gamepad navigation, using this branch without wiring the inputs would be useful testing.

What I would like from users:

See how it fits in your real-world app and what we need to fix/add.
Any bug report, questions, features request, welcome. Please be critical!

The development of this feature has been partly sponsored by Insomniac Games (thank you!).

My current mapping for DualShock4

D-Pad up/down/left/right: navigate, tweak values
Cross button: press button, hold to tweak/activate widget, enter child, etc.
Circle button: close popup, exit child, clear selection, etc.
Square button(TAP): access menu, collapsing, window options, etc.
Square button(HOLD)+Dpad: resize window
Square button(HOLD)+Analog: move window
Square button(HOLD)+L/R trigger changes window focus, ALT-TAB style
Triangle button: text input (requires user back-end reading back io.WantTextInput, possibly display an OS keyboard display).
L/R Trigger: slow down/speed up tweaking values
Analog stick: manual scroll.

Quick instructions

// Fill ImGuiIO.NavInputs[] float array every frame to feed gamepad/keyboard navigation inputs.
// 0.0f= not held. 1.0f= fully held. Pass intermediate 0.0f..1.0f values for analog triggers/sticks.
// ImGui uses a simple >0.0f for activation testing, and won't attempt to test for a dead-zone.
// Your code passing analog gamepad values is likely to want to transform your raw inputs, using a dead-zone and maybe a power curve.
enum ImGuiNavInput_
{
    ImGuiNavInput_PadActivate,      // press button, tweak value                    // e.g. Circle button
    ImGuiNavInput_PadCancel,        // close menu/popup/child, lose selection       // e.g. Cross button
    ImGuiNavInput_PadInput,         // text input                                   // e.g. Triangle button
    ImGuiNavInput_PadMenu,          // access menu, focus, move, resize             // e.g. Square button
    ImGuiNavInput_PadUp,            // move up, resize window (with PadMenu held)   // e.g. D-pad up/down/left/right, analog
    ImGuiNavInput_PadDown,          // move down
    ImGuiNavInput_PadLeft,          // move left
    ImGuiNavInput_PadRight,         // move right
    ImGuiNavInput_PadScrollUp,      // scroll up, move window (with PadMenu held)   // e.g. right stick up/down/left/right, analog
    ImGuiNavInput_PadScrollDown,    // "
    ImGuiNavInput_PadScrollLeft,    //
    ImGuiNavInput_PadScrollRight,   //
    ImGuiNavInput_PadFocusPrev,     // next window (with PadMenu held)              // e.g. L-trigger
    ImGuiNavInput_PadFocusNext,     // prev window (with PadMenu held)              // e.g. R-trigger
    ImGuiNavInput_PadTweakSlow,     // slower tweaks                                // e.g. L-trigger, analog
    ImGuiNavInput_PadTweakFast,     // faster tweaks                                // e.g. R-trigger, analog
    ImGuiNavInput_COUNT,
};

Current blurb in imgui.cpp (I know it is cropped by github but please read it)

 USING GAMEPAD/KEYBOARD NAVIGATION [BETA]

 - Gamepad/keyboard navigation support is available, currently in Beta with some issues. Your feedback and bug reports are welcome.
 - See https://github.com/ocornut/imgui/issues/323 discussion thread and ask questions there.
 - The current primary focus is to support game controllers.
 - Consider emulating a mouse cursor with DualShock4 touch pad or a spare analog stick as a mouse-emulation fallback.
 - Consider using Synergy host (on your computer) + uSynergy.c (in your console/tablet/phone app) to use PC mouse/keyboard.
 - Your inputs are passed to imgui by filling the io.NavInputs[] array. See 'enum ImGuiNavInput_' in imgui.h for a description of available inputs.
 - For gamepad use, the easiest approach is to go all-or-nothing, with a buttons combo that toggle your inputs between imgui and your game/application.
   Sharing inputs in a more advanced or granular way between imgui and your game/application may be tricky and requires further work on imgui.
   For more advanced uses, you may want to use:
     - io.NavUsable: true when a window is focused and it doesn't have the ImGuiWindowFlags_NoNavInputs flag set.
     - io.NavActive: true when the navigation cursor is visible (and usually goes false when mouse is used).
     - query focus information with IsWindowFocused(), IsAnyWindowFocused(), IsAnyItemFocused() functions.
   The reality is more complex than what those flags can express. Please discuss your issues and usage scenario in the thread above. 
   As we head toward more keyboard-oriented development this aspect will need to be improved.
 - It is recommended that you enable the 'io.NavMovesMouse' option. Enabling it instructs ImGui that it can move your move cursor to track navigated items and ease readability.
   When enabled and using directional navigation (with d-pad or arrow keys), the NewFrame() functions may alter 'io.MousePos' and set 'io.WantMoveMouse' to notify you that it did so.
   When that happens your back-end NEEDS to move the OS or underlying mouse cursor on the next frame. The examples binding in examples/ do that.
   (Important: It you set 'io.NavMovesMouse' to true but don't honor 'io.WantMoveMouse' properly, imgui will misbehave as it will think your mouse is moving back and forth.)

     // Application init
     io.NavMovesMouse = true;

     // Application main loop
     if (io.WantMoveMouse)
        MyFuncToSetMousePosition(io.MousePos.x, io.MousePos.y);
     ImGui::NewFrame();

   In a setup when you may not have easy control over the mouse cursor (e.g. uSynergy.c doesn't expose moving remote mouse cursor),
   you might want to set a boolean to ignore your other external mouse positions until they move again

THE INITIAL FOCUS IS ON USING A GAME CONTROLLER! The inputs are explicitly named using the word "pad" because the current scheme is optimized for game controllers. You can however trivially map keyboard keys on those inputs and it'll work if you use mouse+keyboard combo. Later on I will focus on keyboard and add especially named enums for keyboard (user expectation for keyboard controls are much higher/harsher than with gamepad).
Depending on your application using gamepad/keyboard might requires more fine-tuned controls of how inputs are dispatched and shared between your app/game and different parts of your imgui interfaces. We may need better options/tooling for that.
It is very probably missing or failing at something that will appear obvious to you. I've nailed many issues and big technical problems but barely started scrapping the surface of all possible usage scenarios. Please report!
The actual scoring function for navigating the graph hasn't been given a lot of love yet and is failing to behave as expected in various cases. Failure usually mean that you press a direction and don't end up exactly where you intended. Will keep improving it. Please report issues with screenshots!
Addition to the public API are rather minimal. New functions IsItemFocused(), IsAnyItemFocused(), SetItemDefaultFocus(), GetKeyPressedAmount(). I agressively made IsItemHovered() be aware of current navigation focus to maximize existing code just naturally working with Nav (typically tooltip pattern). New window flags ImGuiWindowFlags_NoNavFocus ImGuiWindowFlags_NoNavInputs. New colors ImGuiCol_NavHighlight (make it same or close to ImGuiCol_HeaderActive) ImGuiCol_NavWindowingHighlight (white and very transparent), a bunch of keys (read instructions), 1 IO setting NavMovesMouse, 3 IO outputs WantMoveMouse NavUsable NavActive.
The option io.NavMovesMouse is currently off by default. I recommend enabling it. When enabled. the mouse cursor can be moved by ImGui::NewFrame() when directional navigation is used. It does so by overwriting io.MousePos and set io.WantMoveMouse=true. It is up to your backend when that flag is set to apply the new mouse position in your OS. If you enable the option but don't honor those requests, ImGui will be very confused. (this is why I can't have it on by default).
If you are running this on VR, some suggestions: if you want to display ImGui as a static overlay (not affected by head rotation) you may want to reduce DisplaySize and avoid rendering over your entire framebuffer. You can also increase the style.DisplaySafeAreaPadding value. Popups should stay within this rectangle while you can still partly move regular window outside. It might be just better to display it within the 3D world but I haven't tried.

Following in the next message will be my test code for GLFW binding to map a DualShock 4.

TODO list

omar · Answer 1 · Mon Aug 22 2016 04:15:04 GMT+0800 (China Standard Time)

BINDING EXAMPLE CODE

Here's my ugly test binding for GLFW+DualShock4+DS4Window as rough guidance.

We shall make that less ugly but as soon as we venture into controllers it's hard to get anything portable and reliable. Will probably add a dedicated input visualization/configuration window to make it easier. Currently under "Keyboard,Mouse,etc." in the demo window there's a panel that shows all pressed inputs.

EDIT 2017/10/21: introduced ImGuiNavInput_KeyMenu which is the first explicitly keyboard-related enum. It handle the menu toggling like ImGuiNavInput_PadMenu but when held doesn't trigger the window highlight that is used for resizing/moving Windows with the pad.

Gamepad (GLFW, basing mapped on DualShock4+DS4Window)

// Setup directional navigation events/key mapping [BETA]
memset(io.NavInputs, 0, sizeof(io.NavInputs));

// Enable to allow ImGui moving mouse cursor when using keyboard/gamepad navigation
io.NavMovesMouse = true;

// Update Gamepad Inputs
const bool nav_uses_gamepad = true;
if (nav_uses_gamepad)
{
    #define MAP_BUTTON(NAV_NO, BUTTON_NO)       { if (buttons_count > BUTTON_NO && buttons[BUTTON_NO] == GLFW_PRESS) io.NavInputs[NAV_NO] = 1.0f; }
    #define MAP_ANALOG(NAV_NO, AXIS_NO, V0, V1) { float v = (axes_count > AXIS_NO) ? axes[AXIS_NO] : V0; v = (v - V0) / (V1 - V0); if (v > 1.0f) v = 1.0f; if (io.NavInputs[NAV_NO] < v) io.NavInputs[NAV_NO] = v; }
    int axes_count = 0, buttons_count = 0;
    const float* axes = glfwGetJoystickAxes(GLFW_JOYSTICK_1, &axes_count);
    const unsigned char* buttons = glfwGetJoystickButtons(GLFW_JOYSTICK_1, &buttons_count);
    MAP_BUTTON(ImGuiNavInput_PadActivate,   0);
    MAP_BUTTON(ImGuiNavInput_PadCancel,     1);
    MAP_BUTTON(ImGuiNavInput_PadMenu,       2);
    MAP_BUTTON(ImGuiNavInput_PadInput,      3);
    MAP_BUTTON(ImGuiNavInput_PadUp,         10);
    MAP_BUTTON(ImGuiNavInput_PadDown,       12);
    MAP_BUTTON(ImGuiNavInput_PadLeft,       13);
    MAP_BUTTON(ImGuiNavInput_PadRight,      11);
    MAP_BUTTON(ImGuiNavInput_PadFocusPrev,  4);
    MAP_BUTTON(ImGuiNavInput_PadFocusNext,  5);
    MAP_BUTTON(ImGuiNavInput_PadTweakSlow,  4);
    MAP_BUTTON(ImGuiNavInput_PadTweakFast,  5);
    MAP_ANALOG(ImGuiNavInput_PadScrollUp,   1,  +0.3f,  +0.9f);
    MAP_ANALOG(ImGuiNavInput_PadScrollDown, 1,  -0.3f,  -0.9f);
    MAP_ANALOG(ImGuiNavInput_PadScrollLeft, 0,  -0.3f,  -0.9f);
    MAP_ANALOG(ImGuiNavInput_PadScrollRight,0,  +0.3f,  +0.9f);
    #undef MAP_BUTTON
    #undef MAP_ANALOG
}

For keyboard (GLFW)

// Update Keyboard Inputs
const bool nav_uses_keyboard = true;
if (nav_uses_keyboard)
{
    #define MAP_KEY(NAV_NO, KEY_NO) { if (io.KeysDown[KEY_NO]) io.NavInputs[NAV_NO] = 1.0f; }
    MAP_KEY(ImGuiNavInput_PadActivate,      GLFW_KEY_SPACE);
    MAP_KEY(ImGuiNavInput_PadCancel,        GLFW_KEY_ESCAPE);
    MAP_KEY(ImGuiNavInput_KeyMenu,          GLFW_KEY_LEFT_ALT);
    MAP_KEY(ImGuiNavInput_PadInput,         GLFW_KEY_ENTER);
    MAP_KEY(ImGuiNavInput_PadUp,            GLFW_KEY_UP);
    MAP_KEY(ImGuiNavInput_PadDown,          GLFW_KEY_DOWN);
    MAP_KEY(ImGuiNavInput_PadLeft,          GLFW_KEY_LEFT);
    MAP_KEY(ImGuiNavInput_PadRight,         GLFW_KEY_RIGHT);
    MAP_KEY(ImGuiNavInput_PadTweakSlow,     GLFW_KEY_LEFT_ALT);
    MAP_KEY(ImGuiNavInput_PadTweakSlow,     GLFW_KEY_RIGHT_ALT);
    MAP_KEY(ImGuiNavInput_PadTweakFast,     GLFW_KEY_LEFT_SHIFT);
    MAP_KEY(ImGuiNavInput_PadTweakFast,     GLFW_KEY_RIGHT_SHIFT);
    #undef MAP_KEY
}

For keyboard (DX11)

    // Update Keyboard Inputs
    const bool nav_uses_keyboard = true;
    if (nav_uses_keyboard)
    {
#define MAP_KEY(NAV_NO, KEY_NO) { if (io.KeysDown[KEY_NO]) io.NavInputs[NAV_NO] = 1.0f; }
        MAP_KEY(ImGuiNavInput_PadActivate, VK_SPACE);
        MAP_KEY(ImGuiNavInput_PadCancel, VK_ESCAPE);
        MAP_KEY(ImGuiNavInput_KeyMenu, VK_MENU);
        MAP_KEY(ImGuiNavInput_PadInput, VK_RETURN);
        MAP_KEY(ImGuiNavInput_PadUp, VK_UP);
        MAP_KEY(ImGuiNavInput_PadDown, VK_DOWN);
        MAP_KEY(ImGuiNavInput_PadLeft, VK_LEFT);
        MAP_KEY(ImGuiNavInput_PadRight, VK_RIGHT);
        MAP_KEY(ImGuiNavInput_PadTweakSlow, VK_LMENU);
        MAP_KEY(ImGuiNavInput_PadTweakSlow, VK_RMENU);
        MAP_KEY(ImGuiNavInput_PadTweakFast, VK_LSHIFT);
        MAP_KEY(ImGuiNavInput_PadTweakFast, VK_RSHIFT);
#undef MAP_KEY
}

Will improve this skeleton in user examples, but I don't really expect to be able to provide working examples for all backends and controllers just because mapping gamepad buttons is an unsolved computer science mystery. I'll probably devise a set of functions where mapping can easily be setup from the outside.

Dale Kim · Answer 2 · Tue Aug 23 2016 08:46:57 GMT+0800 (China Standard Time)

Not particularly helpful feedback, but thought I'd let you know that I'm currently seeing these warnings on PS4:

imgui.cpp(2965,9): warning : add explicit braces to avoid dangling else [-Wdangling-else]
imgui.cpp(3788,13): warning : unused function 'IsKeyDownMap' [-Wunused-function]

We tend to compile with -Wall !

-Dale Kim

Enes Karaca · Answer 3 · Tue Oct 18 2016 06:24:57 GMT+0800 (China Standard Time)

Hmm. Depends on how you design the ui but this could actually make imgui useful on embedded devices where there is no mouse and only a keypad as the only input device. I will give it a try soon, possibly with a directfb backend. I have such device at my hand, which only has a numberpad, up and down buttons an enter button in the middle of them and a couple of other keys that could be used as left/right or tab keys. Right now i am drawing everything with DirectFB myself and getting input one screen at a time. (enter username on one screen press enter, enter password on another screen press enter, see a loading screen, get presented with a menu that covers whole screen etc.)

uglycoyote · Answer 4 · Wed Mar 08 2017 08:59:35 GMT+0800 (China Standard Time)

Thanks so much for the progress on getting gamepad navigation working.

We have an existing custom (non-imgui) tree-based debug menu in our game. I created an alternate implementation in ImGui and am considering deprecating the old one eventually. It mostly depends on getting the game-pad support right as we don't always have the ability to use mouse on our game builds.

The menu is essentially just a big tree containing mostly booleans, ints and floats and "triggers", so in the imgui implementation they are mostly checkboxes, sliders, and buttons.

I'm trying out the latest 2016-07-Navigation branch. At the moment I'm finding the dpad left/right behaviour particularly frustrating.

In my case the thing I'm trying to replace something that users have been using for about 10 years and the controls are ingrained in everyone's muscle memory so it would be nice to be able to configure the controls in ImGui to work as similar as possible to how the tree navigation works under that system.

This is how our existing menu system works and how I hoped ImGui would be able to configured to work:

When focused on a folder in the tree:

Press PadRight or "A" (activate) to expand a folder in the tree
Press PadLeft or "B" (cancel) to collapse the folder

When focused on a leaf node in the tree

Press PadRight or "A" to toggle a checkbox or activate a button
Press PadRight or PadLeft to adjust a slider
Press B to collapse the parent folder and set parent as the focused item

In ImGui Pressing PadRight takes the focus away from wherever you are in the tree and off to never-never land. From looking at the code I understand that it's using some kind of scoring system for determining where the focus goes, but at least in my use case it feels wrong and unpredictable. It either focuses another item higher in the tree (presumably because that item is indented more to the right) or it jumps the focus to a "Clear" button that I have on the filter box at the top of the tree. Either way it's taking the input focus away from something that you just pressed the d-pad a few dozen times to get to, so it's frustrating in that regard.

I know others on this thread and the previous one have said similar things, but the rebuttal was that the dpad navigation needs to be able to work with multiple columns and with tree nodes which have multiple things in them that you can focus on. But in my case I only have one column and each tree node only has one focusable element, (basically the simplest kind of tree) so it would be nice if it worked well in this situation.

The scoring system leads to unpredictable jumping of focus which takes the user a long time to recover from. IMHO the scoring system should only be used when there's not some simpler more direct context-specific interpretation of the arrow keys. Once you enter the "tree navigation" context it should not use the scoring system anymore but a more custom logic for interpreting directional inputs such as the rules I listed above.

omar · Answer 5 · Tue Aug 08 2017 17:31:41 GMT+0800 (China Standard Time)

NB: The branch has been renamed from 2016-07-navigation to navigation removing the date in it. I think the date made people think the branch was not keep up to date, while my intent was merely to note approximate date a branch was being worked on. Of course, for those long standing branch it doesn't make much sense. I'll be using the date when closing branches.

omar · Answer 6 · Tue Aug 29 2017 19:21:37 GMT+0800 (China Standard Time)

Update: while I don't have a date for merging this branch, I have already merged many of the non-strictly-navigation-related changes back to Master, so the diff between branches is smaller (and some of the changes will be tested a little more due to being in Master already).

Anthony Pesch · Answer 7 · Mon Sep 18 2017 08:25:59 GMT+0800 (China Standard Time)

Just gave this an initial spin - it worked right out of the box!

Out of curiosity, I skimmed #323, what was the reasoning for moving to the NavInputs array vs reusing the existing KeysDown / KeyMap setup? When initially setting everything up, I realized I was just assigning NavInputs[key] = KeysDown[KeyMap[key]]; effectively.

Edit: Just realized, it seems to make navigation from multiple inputs work, e.g. also letting the user scroll through touch events.

Anthony Pesch · Answer 8 · Tue Sep 19 2017 11:18:03 GMT+0800 (China Standard Time)

@ocornut first off, apologies for all of the issue spam, I'm on a death march to get an initial frontend UI going for https://github.com/inolen/redream.

I've been playing with this branch to try and get my UI to work as I'd like, and after hacking around a bit it works really well, but I'm not sure how to make my changes in a manageable fashion.

I've posted a few photos before, but to reiterate, the UI is just like Netflix, Steam Big Picture, etc.:

What I wanted to do here was flatten the parent / child navigation, while still enabling the child window to scroll when moving left / right starting from one of the items in the child window.

To do so, I hacked this up:
https://gist.github.com/inolen/b460ae06d05f55390c6dde4709888657

What I've done is hacked in the NavMoveResultWindow, and removed the g.NavWindow->ScrollbarX part of the conditional that enables the window to scroll.

I have no problem doing more work on this, I just need a bit of direction - I'm very motivated to get this work done.

omar · Answer 9 · Tue Sep 19 2017 16:47:28 GMT+0800 (China Standard Time)

what was the reasoning for moving to the NavInputs array vs reusing the existing KeysDown / KeyMap setup?
Edit: Just realized, it seems to make navigation from multiple inputs work, e.g. also letting the user scroll through touch events.

Also, in my experiments I found that input logic has to be specialized for gamepad type vs keyboard type, there are specifying controls that makes sense for typical gamepad (holding Square + L/R doesn't work exactly the same as Ctrl+Tabbing would work). So for now note that those enums are named after the Gamepad. The branch is also marked Beta for this reason that not everything has been 100% designed.

What I wanted to do here was flatten the parent / child navigation, while still enabling the child window to scroll when moving left / right starting from one of the items in the child window.

I think you are hitting a few barriers that we'd need to lift and this is an interesting use case. Navigating between Games and Options I'd imagine the normal Nav system would be practical, but for your Dreamcast cover browsing you are probably better off making something custom.

Similarly to your issue #1328 I think you may be using a generic-over-engineered piece of feature (the Nav system) for something you could more easy recreate yourself, may be worth going that path short term. And you are not going to get the nice custom smooth scrolling that your apps wants using the immediate focusing?

At the least I think we need to support a way for user code to hook/disable certain Nav inputs so they can be processed by the code, so you could process Left/Right yourself. And we probably need an internal flag to signify "x scrolling" allowed that's independant of ScrollbarX.

PS: I'm just blurting those thoughs but haven't actually dug into your use case or code very much. Your gist/patch is useful for reference. I appreciate it is an interesting use case that I would like to have solved, I just don't have so much time to think about it right now.

Anthony Pesch · Answer 10 · Wed Sep 20 2017 05:26:05 GMT+0800 (China Standard Time)

@ocornut thanks for the comments!

I agree wrt using the nav system, using something hand-rolled probably makes the most sense giving its limited scope right now.

Anthony Pesch · Answer 11 · Thu Sep 21 2017 20:23:28 GMT+0800 (China Standard Time)

@ocornut noodling some this morning, how would a nav hook / callback work?

Would there be an IsItemNavigated function that could be called after adding each item. with some option to disable the automatic scrolling, etc.?

omar · Answer 12 · Thu Sep 21 2017 21:09:56 GMT+0800 (China Standard Time)

I don't know yet. Part of the work is figuring out the design. It's in my list and there is a decent chance I'll be able to focus on the Navigation branch on the upcoming few months.

omar · Answer 13 · Sat Sep 30 2017 07:14:22 GMT+0800 (China Standard Time)

Hello all,

I have made various useful changes to the navigation branch recently.

Navigating menus and menu bars have been vastly improved.
It took me several attempts to get it going right, but you can now browse sibling menus of a same menu bar by pressing left-right inside any one of the child menu, that if there's no immediate match in the direction (so you can still browse left/right within a menu). The way it works is that EndMenuBar() catches the navigation scoring request and reacts if there was no match among its children. I had to rearrange some of the code to be able to store/access data that made it previously difficult to figure out those relationships.

Also fixed a handful other bugs related to browsing menus. It's not perfect now but it's much more pleasant and usable. One of the thing that is missing if that if you navigate a sibling menu it currently doesn't automatically open it. Windows does opens the sibling, but it also cause navigation issues in the case first entry of a sibling menu is itself a menu, so I'm not sure what I prefer now. Either way, it's fairly usable as is. It basically went from "frustrating" to "good enough".

Navigation scoring
In most situations I have disabled the fallback that found a connecting item in the whole plane instead of using a quadrant. This is what @uglycoyote referred with "pressing PadRight takes the focus away from wherever you are in the tree and off to never-never land.".

This fallback is still useful in some situations, especially as the way the scoring system is currently biased causes some elements to be considered as being in the wrong quadrant. Tried to find a better solution but I haven't so far. I have also experimented with other scoring ideas but nothing satisfying so far.

Tree navigation
As suggested by @uglycoyote here and in #1079, I have made some changes to navigating tree. While positioned on a tree node you can use NavLeft to close it and NavRight to open it.

As discussed above, there is a possible ambiguity here is you have elements on the right of your tree node. I haven't solved it but it occurred to me that the treenode could only hook the NavRight if it's closed, so a subsequent NavRight will allow you to navigate to its side. I am a bit unsure if that is an improvement or a regression at this point (people who have lots of stuff/button next to their tree node may not be happy), so may undo it but I thought it was worth trying. Using a 2-frames solution like the menu one may work better.

Note that this doesn't allow to close the parent tree node while positioned on a child yet, I'm experimenting with that as well but there are a few unsolved issues.

Misc
Fixed an issue when opening a popup with navigation keys in certain conditions would have the first item of the popup return true in IsItemActive().
Added internal ImGuiItemFlags_NoNav to hide some items from the navigation. Used that in the color picker so navigation skips areas that don't made sense with the gamepad. (Note that ItemFlags aren't exposed publicly).

Anthony Pesch · Answer 14 · Mon Oct 02 2017 12:21:30 GMT+0800 (China Standard Time)

@ocornut nice work!

I started experimenting with the navigation branch again as I was adding more complicated UIs (file select dialog in this case). I ended up with quite a few minor changes throughout the process to get it working for my use case that I want to clean up and post soon (most having to do with better NavFlattened / scroll support), but I wanted to go ahead and bring up the primary change to scoring I made.

At https://github.com/ocornut/imgui/blob/navigation/imgui.cpp#L2874 you purposely set g.NavScoringRectScreen.Max = g.NavScoringRectScreen.Min. However, from my understanding of the scoring, this fundamentally breaks the bounding box distance portion of the algorithm.

As an example:

In the photo, the tab to the top is being placed in the wrong quadrant (e instead of n) because of this. I had the same bug in a file dialog due to the same issue:

I've removed this change, as well as the bias at https://github.com/ocornut/imgui/blob/navigation/imgui.cpp#L2063 and so far had great results (although, I haven't tested the changes with more traditional imgui menus).

Anthony Pesch · Answer 15 · Tue Oct 03 2017 11:57:08 GMT+0800 (China Standard Time)

Another higher level question - what's the use for having PadCancel clear the active id?

I'm no UX person by any means, but from fiddling with some Android TV / smart TV apps tonight, it seems common to always have a navigable item highlighted.

Anthony Pesch · Answer 16 · Sat Oct 07 2017 12:23:21 GMT+0800 (China Standard Time)

@ocornut i went through and split up all of my various changes I made along the war path of getting my ui going. I'm hoping most of this can be upstreamed, and I'm open to doing whatever work to get the changes in a state that's acceptable to you.

Made NoNavFocus also disable unfocus support

This change lets me provide my big picture interface, where I trap the user in a single ImGui window.
inolen/redream@f63ff83

Added NoNavScroll window flag

This lets me take handle scrolling myself, adding animations, etc.
inolen/redream@cffd5e7

Remove nav scoring biases

As discussed above.
inolen/redream@3ae4122

Set NavDisableHighlight to make sure the request initiated by NavInitWindow is used

Without this, the default nav selection is never made for me.
inolen/redream@38ea46f

Don't reinit navigation when activating a window that has a valid NavLastId

This enables the previously active item to be restored properly when reactive a window.
inolen/redream@17f3743

Updated navigation scrolling code to work when NavFlattened is used

This needs to be cleaned up, but it works, and enables navigation scrolling to work when NavFlattened is used. The idea is that it scrolls until it can't anymore, and then navigation will go to the next item as usual.
inolen/redream@001c3c9

Always keep something selected

This is a total hack to get what I needed working, but as I mentioned, is there ever a time where we don't want to have an item active in a window?
inolen/redream@39cb323

omar · Answer 17 · Wed Oct 11 2017 22:23:13 GMT+0800 (China Standard Time)

@inolen Thanks for those patches. I am not sure yet how/when I can process them as most aren't trivial changes. Finishing the NavFlattened work would be useful, and it's possible that removing the PadCancel-to-clear-NavId is the right route.

Navigating those checkboxes is a good pathological test case if you want to test the navigation:

omar · Answer 18 · Sun Oct 15 2017 18:13:11 GMT+0800 (China Standard Time)

@inolen

Always keep something selected [...] as I mentioned, is there ever a time where we don't want to have an item active in a window?

The reason is that imgui applications often have to share the input with a running game, and this is one possible way to give inputs back to the game. I still haven't nailed all the details of how inputs could be shared, may need some more programmatic support for it.

Anthony Pesch · Answer 19 · Sat Oct 21 2017 02:30:10 GMT+0800 (China Standard Time)

@ocornut sorry for the slow reply. Input being shared is totally understandable, but I was more specifically questioning the idea of a window not having an active nav id. Last I looked, I believe the logic on padcancel was to unset the active nav id if it was set, and if it wasn't set, to exit the window.

I was curious the value of first unsetting the active nav id first on back vs immediately exiting the window on back.

omar · Answer 20 · Wed Oct 25 2017 23:18:02 GMT+0800 (China Standard Time)

I have a question for existing users of the Navigation branch. (@itamago? @Roflraging? @pdoane? @uglycoyote?)

Now that I have added variety of flags to IsItemHovered(), I would like to re-evaluate the default behavior of it in the Nav branch. The current behavior in the Nav branch is that when gamepad/keyboard navigation is being used, IsItemHovered() redirects to IsItemFocused().
This was designed so that in this typical use case:

if (ImGui::IsItemHovered())
   ImGui::SetTooltip("Tooltip");

IsItemHovered() would return true when keyboard navigating over an item.
I can definitively see cases where this IS desirable, other when it is NOT desirable.
So I am trying to decide what would be the more reasonable default, and interested in your feedback.

A) Make IsItemHovered() returns true when nav is being used and item is focused (as currently?)
Add a flag e.g. IsItemHovered(ImGuiHoveredFlags_NoNav) to disable that behavior.

B) Make IsItemHovered() not returns true when nav is being used and item is focused.
But when nav is being used we still tend to "disable" reaction to the mouse so it should also probably return false on the mouse-hovered item until the mouse is moved again?
Add a flag e.g. IsItemHovered(ImGuiHoveredFlags_NavFocused)

C) Would it make sense to actually control of this behavior via PushItemFlags() + add 2 override flags to IsItemHovered() ?

D) Something else?

More than just throwing a vote I'm interested in your use cases, hearing if A) felt undesirable to you, etc.

OvermindDL1 · Answer 21 · Thu Oct 26 2017 00:36:14 GMT+0800 (China Standard Time)

I'm not using that branch currently, but I'd got for B. I'd probably want to do something special for gamepad tooltips compared to mouse tooltips so I don't obscure the interface so they know how to continue to navigate properly, like I'd probably add an info pane at the bottom of a 'selected' tooltip/information compared to the mouse where it does it at the location on hover.

uglycoyote · Answer 22 · Thu Oct 26 2017 01:12:06 GMT+0800 (China Standard Time)

Is this question mainly to do with tooltips or are there places where IsItemHovered is used?

I think it should probably default to how you have it -- to show the tooltip when navigating and IsItemFocused(). But do agree that there's a possibility of the tooltip getting in the way, in which case it should be able to be disabled with the proposed ImGuiHoveredFlags_NoNav (which I guess means that they user would have no way of seeing that tooltip, ever?)

I don't have any use cases of IsItemHovered() that are particularly interesting, they are all just run-of-the-mill tooltips.

Testing in the imgui_demo.cpp examples, I did notice that the positioning of the tooltip was inconsistent when using keyboard navigation.

(Note that I had to convert these to buttons .. as written they were ImGui::Text and so there was actually no way to see the tooltip because the navigation skips over them)

Normally the position of the tooltip would be dictated by the mouse position, so it is not clear how it should work when using keyboard navigation. Neither of the positions above seemed wrong, but it was a bit worrying that it did not always show up in the same place.

omar · Answer 23 · Thu Oct 26 2017 02:23:14 GMT+0800 (China Standard Time)

I'd probably add an info pane at the bottom of a 'selected' tooltip/information compared to the mouse where it does it at the location on hover.

That sounds like the good thing to do for the application. But then if you have that setup in place you probably have high-level helpers or data structures in place to dispatch the tooltip content and so passing an additional flag to IsItemHovered() in a few calls won't be a problem. I think here we may want to judge from the point of view where IsItemHovered() is called from many spots - aka the "quick" code.

Is this question mainly to do with tooltips or are there places where IsItemHovered is used?

Both, IsItemHovered() have other uses and it'd be interesting to hear about them.

On the tooltip position,

Normally the position of the tooltip would be dictated by the mouse position, so it is not clear how it should work when using keyboard navigation.

In this situation it is using a simple formula based on the bounding box of the item.

Neither of the positions above seemed wrong, but it was a bit worrying that it did not always show up in the same place.

I agree, that's definitively a bug. I think it is related to the code that decides to move a tooltip when it it not fitting in the screen, it remembers the last cardinal direction the tooltip was moved. If your tooltip for "Or me" had to be moved because the right side of the screen was nearby there is a chance the cardinal direction wasn't reset when you got back to the first element. I'll file this as a bug to fix. Thanks!

Anthony Pesch · Answer 24 · Thu Nov 09 2017 13:11:37 GMT+0800 (China Standard Time)

I was rebasing my branch (https://github.com/inolen/imgui/commits/navigation) to latest in order to attempt to see if I could PR some of the changes I depend on.

While doing so, I saw how bad removing the code which only considered the left edge of each item broke the example menu, so tonight I attempted to refactor / simplify the navigation scoring to work well in that menu, as well as my own use cases.

In the end, I came up with inolen@793018b

The major changes are;
1.) items full bounding boxes are considered, not just their leftmost edge
2.) horizontal movements are biased to make up for this
3.) immediately adjacent menu items are handled correctly (e.g. cases where dx == 0.0f && dy == 0.0f)
4.) item ordering is favored when a tie arises

If you have a minute to give this change a spin, I'd be interested to hear how it works for you.

omar · Answer 25 · Thu Nov 09 2017 21:10:46 GMT+0800 (China Standard Time)

Thanks @inolen, I won't have time to look into it in details right away, but gave a quick test to your branch now. It generally works ok but here are some problems I noticed. Just writing them down in case you have time to look at it:

Whenever I open and close a popup, navigation position on the initial window is lost.
Navigation in those sections is quite broken:

Perhaps because of touching corners?
Here when I press Right it tends to skip over the input box and end on the [-] button.

Note that I have checked out your forked branch so it includes a few other commits you made, not just the inolen@793018b commit.

Anthony Pesch · Answer 26 · Thu Nov 09 2017 22:15:40 GMT+0800 (China Standard Time)

Ah great, 2 and 3 are good edge cases I can get to tonight.

1 is probably related to the top 2 commits on the branch which are questionable.

Anthony Pesch · Answer 27 · Fri Nov 10 2017 12:51:09 GMT+0800 (China Standard Time)

Force pushed an update to inolen@65b4c2c if you have time (thanks again for the screenshots).

The scoring issues mentioned should be resolved with this, more comments added explaining the process.

Evan Gordon · Answer 28 · Sun Nov 12 2017 17:46:46 GMT+0800 (China Standard Time)

I've been looking for ages for a good gui library with keyboard/controller support, just stumbled across your nav branch and love it! (I am using SFML so I'm kind of using a mixture of the bindings for that and these keyboard controls)

That being said with the minimal documentation provided, there are a few things I'm not sure of, and/or aren't working as I would anticipate.
Currently I'm using ImGui::SetItemDefaultFocus(); for a button to set it as being selected by default within the window. While it seems internally the 'cursor' points to that item, i seem to have to press a directional input before i can interact with an element. I'm not sure if this is a bug, or if there is there a function i can call to set this.

I'm also having trouble switching window focus, In your documentation you said to press and hold ImGuiNavInput_PadMenu and use ImGuiNavInput_PadFocusPrev or ImGuiNavInput_PadFocusNext to switch windows. Do i have to set which windows are actually linked as next or previous?

Is there any chance of a feature for just using directional input for swapping window focus. So i could have 2 windows and when i move to the farthest right i can within the structure it would then automatically move over to the next window to the left. I'm sure this would require some specific flags, or functions to define what the next window to the left would be, but I would at least love a feature like that at some point.

omar · Answer 29 · Sun Nov 12 2017 23:24:50 GMT+0800 (China Standard Time)

Hello @evan-gordon, and thanks for testing the navigation branch.

Currently I'm using ImGui::SetItemDefaultFocus(); for a button to set it as being selected by default within the window. While it seems internally the 'cursor' points to that item, i seem to have to press a directional input before i can interact with an element. I'm not sure if this is a bug, or if there is there a function i can call to set this.

Currently the Nav system automatically hide focus as soon as you move the mouse. And similarly as soon as you use Nav the mouse hovering is disabled. Maybe this is the problem you are experiencing? Could you confirm? Otherwise SetItemDefaultFocus should work as intended. Generally speaking the interaction between mouse and navigation inputs and user applications probably needs more work, and your testing/report can be useful here.

Also note this API was an early test in the Navigation and will eventually be removed and replaced by a more full featured set of focus-related functions. I actually worked for several days on that feature but had to move to more urgent things, but I'm hoping to come back to it within upcoming months. For example a typical usage pattern is that you'd maybe want to position the navigation cursor here by default, but mainly you want to mark the item as Enter or Escape key-press, regardless of where the navigation cursor is.

I'm also having trouble switching window focus, In your documentation you said to press and hold ImGuiNavInput_PadMenu and use ImGuiNavInput_PadFocusPrev or ImGuiNavInput_PadFocusNext to switch windows. Do i have to set which windows are actually linked as next or previous?

You do not have to link windows nor setup anything. PadMenu needs to be held for about 0.25 second and a window selector will appears then you can press PadFocusPrev/PadFocusNext. Note that this scheme was designed for modern game controllers in mind and doesn't makes a lot of sense with a keyboard. The equivalent keyboard mapping should be CTRL+TAB and CTRL+SHIFT+TAB but we will need to provide dedicated keyboard NavInput enum for that (it might already be possible to use it with a keyboard but it is likely going to be weird).

Is there any chance of a feature for just using directional input for swapping window focus.

When I looked at it it wasn't a good idea, because this movement scheme makes it very difficult to control window ordering. This is why Windows and Mac uses ALT-TAB style Windows selection, you get a preview and only the selected Windows is bumped to front when you release the key. If you were bumped Windows to front as you moved into them it would be an annoyance in many circumstances.

Evan Gordon · Answer 30 · Mon Nov 13 2017 13:07:39 GMT+0800 (China Standard Time)

(ADMIN Accidentally messed up this message, and restored it from the e-mail)

Currently the Nav system automatically hide focus as soon as you move the mouse. And similarly as soon as you use Nav the mouse hovering is disabled. Maybe this is the problem you are experiencing? Could you confirm? Otherwise SetItemDefaultFocus should work as intended. Generally speaking the interaction between mouse and navigation inputs and user applications probably needs more work, and your testing/report can be useful here.

I don't think the mouse moving is what I'm experiencing. I'll try to explain better using pictures.
Before i press anything:

After Pressing ImGuiNavInput_PadDown
code:
ImGui::Button("Play##1", sf::Vector2f(80.f, 30.f));//sf::vector is from sfml

After Pressing ImGuiNavInput_PadDown
code:
ImGui::Button("Play##1", sf::Vector2f(80.f, 30.f));//sf::vector is from sfml
ImGui::SetItemDefaultFocus();

Ideally I'm hoping that the Play button would be highlighted right when the menu is opened without having to press any directional input keys. I was hoping that ImGui::SetItemDefaultFocus(); would do this but that doesn't seem to be the case.

I'm trying to design my game without any mouse use at all. Are there any functions i need to call to disable scanning for mouse input from ImGui?

You do not have to link windows nor setup anything. PadMenu needs to be held for about 0.25 second and a window selector will appears then you can press PadFocusPrev/PadFocusNext. Note that this scheme was designed for modern game controllers in mind and doesn't makes a lot of sense with a keyboard. The equivalent keyboard mapping should be CTRL+TAB and CTRL+SHIFT+TAB but we will need to provide dedicated keyboard NavInput enum for that (it might already be possible to use it with a keyboard but it is likely going to be weird).

I understand that the current setup is a bit weird, which I'm fine with for the time being. However I can't seem to get this working (I am using a keyboard).. I am pressing both the keys i need to be pressing to change windows but I'm not seeing the window selector... How far out would you expect a dedicated key for this functionality to be?

When I looked at it it wasn't a good idea, because this movement scheme makes it very difficult to control window ordering. This is why Windows and Mac uses ALT-TAB style Windows selection, you get a preview and only the selected Windows is bumped to front when you release the key. If you were bumped Windows to front as you moved into them it would be an annoyance in many circumstances.

I guess i can see why that wouldn't make sense to have, thanks for your response!

omar · Answer 31 · Fri Nov 17 2017 22:47:59 GMT+0800 (China Standard Time)

@evan-gordon Something really weird happened 4 days ago when I tried to answer to you. For some reason, as repo administrator Github gives me full Edit access to every one's message (I find it rather odd, but I occasionally used it to fix code formatting). And I just noticed that 4 days ago I accidentally edited your message instead of replying, and put my replies in there. So you probably didn't receive an answer notification, and when I just looked your message held my replies and it was strange. I have now edited your message again with the contents that appears in the e-mail, and my answers are below. Sorry for the confusion.

I don't think the mouse moving is what I'm experiencing. I'll try to explain better using pictures.
Ideally I'm hoping that the Play button would be highlighted right when the menu is opened without having to press any directional input keys. I was hoping that ImGui::SetItemDefaultFocus(); would do this but that doesn't seem to be the case.

I'd suggest trying to build a minimal repro for it, because AFAIK this is working and I suspect something else is interfering with it.

I'm trying to design my game without any mouse use at all. Are there any functions i need to call to disable scanning for mouse input from ImGui?

If you never pass mouse input (so mouse coordinates don't change) it'll work.

I understand that the current setup is a bit weird, which I'm fine with for the time being. However I can't seem to get this working (I am using a keyboard).. I am pressing both the keys i need to be pressing to change windows but I'm not seeing the window selector...

The window selector appears when holding PadMenu.

If you have mapped it to ALT, can you verify that ImGui is seeing your ALT key?
The Demo window has a panel that display keyboard inputs.
Does your ALT key allows you to enter the menubar of a window?
The ALT key is a notorious problem to access under Win32 and there is a possibility that your code/backend/binding/SFML isn't providing the ALT key correctly to ImGui.

How far out would you expect a dedicated key for this functionality to be?

From a few weeks to a few months (sorry if that's vague!). It's not really just a key with the current system of using an enum to specify input semantic, but I'm starting to think it may be too confusing/hard to do with keyboard and perhaps we should keep the current system for Pad and instead for Keyboard requests explicit keys.

I map PadMenu to CTRL and PadFocusNext to TAB it works on the keyboard, it's just a little odd because PadMenu also triggers the menu.

Anthony Pesch · Answer 32 · Sat Nov 18 2017 22:55:42 GMT+0800 (China Standard Time)

@ocornut just wanted to see if you had a chance to give the updated scoring another spin.

Evan Gordon · Answer 33 · Tue Nov 21 2017 14:34:50 GMT+0800 (China Standard Time)

@ocornut hey no worries about the confusion. Thanks for your help, I've made a lot of progress on my project using your library. Besides the controls being a bit awkward i love that it does all that i need it to do out of the box! Nice work man.

omar · Answer 34 · Wed Jan 24 2018 19:32:35 GMT+0800 (China Standard Time)

I made an image file with the current controls (attached is a JPG export). The idea would be that I can provide the PSD file and people can rework it to add their own inputs, change the title, and easily share the result with their team. This is also an exercise to figure out how to easily express the control scheme.

I'm going to experiment with changing the gamepad controls scheme so that Cross toggle tweaking instead of requiring a hold. I'm worried it'll be slower to use, but perhaps more welcoming. I don't consider this final now!

omar · Answer 35 · Thu Jan 25 2018 23:34:39 GMT+0800 (China Standard Time)

I made an image file with the current controls (attached is a JPG export).

Very embarrassingly I managed to get this wrong yesterday. Attached the correct version:

Also note I've been pushing a bunch of Navigation related stuff recently. My plan is to merge the Navigation branch, flagging the feature as "Beta". I also renamed several of ImGuiNavInput enums as well, going toward having separate names for Gamepad and Keyboard, because the inputs will be specialized.

My number one uncertainty with the Nav branch at the moment is the practicality of dispatching inputs between game/application and imgui. In particular, when running a game along with a keyboard-controlled imgui interface I expect inputs dispatch to be sometimes tricky. Would be happy to hear any about real-world usage there. For the lack of them, the Navigation feature will be tagged as Beta until this is better nailed (which may lead to some future API breakage).

omar · Answer 36 · Fri Jan 26 2018 23:19:15 GMT+0800 (China Standard Time)

Some of the Navigation Branch news!

There is now a field io.NavFlags, two most importants flags are ImGuiNavFlags_EnableGamepad and ImGuiNavFlags_EnableKeyboard. Those flags are mostly here as a standard way to provide instruction to the backend as to whether inputs should be provided for those devices.
Some of the input enums have been renamed and your build will break. The updated enums are below (and should be obvious to remap). For gamepad the tendency is to move away from trying to convey semantic (e.g. Scroll) in the inputs, because the controls are too optimized/specialized to actually be remappable.
The DirectX11 backend now support Keyboard by default (if you set io.NavFlags |= ImGuiNavFlags_EnableKeyboard). The GLFW+GL3 backend now support Keyboard and Gamepad. Likewise the enable flags are not set by the backends. In the example main.cpp there will be a commented-out line to enable those flags.
io.NavMoveMouse has been turned into ImGuiNavFlags_MoveMouse. This still works and I think it is useful for console platforms but totally weird on PC (it moves the mouse cursor).
Keyboard: now supports Windows-style Ctrl+Tab (and Ctrl+Shift+Tab) if the EnableKeyboard flag is set.
Keyboard: Additionally when highlighting a windows with Ctrl+Tab you can use arrows or shift+arrow to move and resize Windows. This is a little weird and inconsistent with typical OS, but in the absence of a system menu in place of a collapsing arrow (a menu which we can add later!) this reused the reusing gamepad code and doesn't hurt.
Menu navigation has been improved several times in the past few months and is now pretty good. The main thing missing are alphabetical search.

Those are the current values of ImGuiNavInput:

enum ImGuiNavInput_
{
    // Gamepad Mapping
    ImGuiNavInput_PadActivate,      // press button, tweak value                    // e.g. Circle button
    ImGuiNavInput_PadCancel,        // close menu/popup/child, lose selection       // e.g. Cross button
    ImGuiNavInput_PadInput,         // text input                                   // e.g. Triangle button
    ImGuiNavInput_PadMenu,          // toggle menu, hold to: focus, move, resize    // e.g. Square button
    ImGuiNavInput_PadDpadLeft,      // move left, resize window (with PadMenu)      // e.g. D-pad directions
    ImGuiNavInput_PadDpadRight,     // move right
    ImGuiNavInput_PadDpadUp,        // move up                                      
    ImGuiNavInput_PadDpadDown,      // move down
    ImGuiNavInput_PadLStickLeft,    // scroll up, move window (with PadMenu)        // e.g. left stick directions (analog)
    ImGuiNavInput_PadLStickRight,   // scroll right
    ImGuiNavInput_PadLStickUp,      // scroll up
    ImGuiNavInput_PadLStickDown,    // scroll down
    ImGuiNavInput_PadFocusPrev,     // next window (with PadMenu)                   // e.g. L-trigger
    ImGuiNavInput_PadFocusNext,     // prev window (with PadMenu)                   // e.g. R-trigger
    ImGuiNavInput_PadTweakSlow,     // slower tweaks                                // e.g. L-trigger, analog
    ImGuiNavInput_PadTweakFast,     // faster tweaks                                // e.g. R-trigger, analog
    // Keyboard Mapping
    // [BETA] You can map keyboard keys on the gamepad mapping for most inputs. Will add specialized keyboard mappings as we add features.
    ImGuiNavInput_KeyMenu,          // toggle menu                                  // e.g. ALT
    ImGuiNavInput_KeyLeft,          // move left                                    // e.g. Arrow keys
    ImGuiNavInput_KeyRight,         // move right
    ImGuiNavInput_KeyUp,            // move up
    ImGuiNavInput_KeyDown,          // move down
    ImGuiNavInput_COUNT,
};

@inolen Sorry I haven't looked at your patches suggestion yet, I will eventually. Been a bit overwhelmed with amount of imgui tasks :)

Anthony Pesch · Answer 37 · Fri Jan 26 2018 23:23:39 GMT+0800 (China Standard Time)

NP, I had linked a few individual commits in here, but I've rebased the branch several times as I add fixes:
https://github.com/inolen/imgui/commits/navigation

The primary patch being the one that begins Refactored and simplified nav scoring.

omar · Answer 38 · Mon Jan 29 2018 22:57:42 GMT+0800 (China Standard Time)

@inolen: I'm currently reworking the ImGuiWindowFlags_NavFlattened behavior to work with scrolling child. It's becoming very useful and I even considered making it the default behavior (with a flag to disable it).

Mostly I am not super happy with the terminology NavFlattened, do you have suggestion for another terminology?

omar · Answer 39 · Tue Jan 30 2018 07:13:55 GMT+0800 (China Standard Time)

NavFlattened / NavTraversable flag
@inolen

a) I have committed a version of ImGuiWindowFlags_NavFlattened that work nicely with scrolling child windows:

With the old code you would sometimes get the same behavior but it would be unreliable. Imagine this (manufactured) case where I added extra spacing between widget:

Old code would tend to exit the child window prematurely.

b) My changes almost certainly created conflicts in your branch. I can still refer to the existing commits so don't worry about fixing your conflicts if you don't need to merge right away (though I would be interested in your feedback).

c) The current version has an issue when pressing ALT to toggle menu back and forth, your navigation cursor will return to the window owning the menu instead of the original child window. Will aim to fix.

d) As for the name, perhaps something along of line of NavTraverseChild ? NavTraversable ? would be a better fit?

e) If it works well, I'm still considering this might be a good option to enable by default (so the flag would become NoNavTraverseChild), will try to get feedback from known users of the navigation branch.

Even though I didn't look at your scoring function proposal yet, I must say that with the recent fixes and additions (Ctrl-tabbing, better menus), using dear imgui with a keyboard is becoming increasingly satisfying.

Stefano Cristiano · Answer 40 · Tue Jan 30 2018 17:51:34 GMT+0800 (China Standard Time)

This is beautiful @ocornut, can't wait to be able to use it in the master

omar · Answer 41 · Thu Feb 01 2018 08:00:23 GMT+0800 (China Standard Time)

This is beautiful @ocornut, can't wait to be able to use it in the master

@Pagghiu (and others) if you have a little time to try the Navigation branch - even if you don't feel you need Keyboard support right away - the feedback would be super useful.

You need to set io.NavFlags |= ImGuiNavFlags_EnableKeyboard and then just feed inputs.
You can search for if (io.NavFlags & ImGuiNavFlags_EnableKeyboard) in the DX11 and GLFW+GL3 examples:
https://github.com/ocornut/imgui/blob/navigation/examples/directx11_example/imgui_impl_dx11.cpp#L610
https://github.com/ocornut/imgui/blob/navigation/examples/opengl3_example/imgui_impl_glfw_gl3.cpp#L416

You can poll io.NavActive if you need (if will more or less return true when a window is focused unless it has NoNav flags, etc.).

With the sort of crazy animated application you have @Pagghiu I expect you will run into problems. I don't expect the Navigation branch to solve everything, but I would be interested in seeing even if in a worse case scenario we could selectively activate/deactivate navigation more contextually.

Stefano Cristiano · Answer 42 · Thu Feb 01 2018 17:12:07 GMT+0800 (China Standard Time)

I've just been merging latest imgui tip into our master yesterday, so this should make it easier to test the navigation tip in a branch and see out it works in our production softwares ! 🌎
I will try to find some time in the next week to try this out.
The problems I see with animations is mainly that the "focused" item could be sent offscreen, and I should probably monitor this situation and move it elsewhere.

omar · Answer 43 · Tue Feb 06 2018 06:34:28 GMT+0800 (China Standard Time)

I have changed the Slider and Drag widget to be tweakable by toggling NavActivate (cross, space) instead of holding it.
io.WantCaptureKeyboard is now set by default when navigation is active, window doesn't have NoNav flag and (g.IO.NavFlags & ImGuiNavFlags_EnableKeyboard).

There is a lot more we need to do with this feature, but I will try merge this branch in master tomorrow for a start. It will be marked as BETA and I will open a new topic for it.

The feature is enabled on the backend by setting io.NavFlags |= ImGuiNavFlags_EnableKeyboard or io.NavFlags |= ImGuiNavFlags_EnableGamepad. Not every backend will support it by default. I expect adoption to take a while (this is fine), I expect it to stays "Beta" for a while, but merging means we'll get more users, more testing, more feedback, and I'll get less branch merge/conflicts issues.

I will also merge #1565 which is a small but breaking change (it will break everyone's codebase - everybody will need to create ImGui::CreateContext() once!) and bump the version number to 1.59 WIP leading us to 1.60.

omar · Answer 44 · Wed Feb 07 2018 00:58:49 GMT+0800 (China Standard Time)

@inolen

you purposely set g.NavScoringRectScreen.Max = g.NavScoringRectScreen.Min. However, from my understanding of the scoring, this fundamentally breaks the bounding box distance portion of the algorithm.
In the photo, the tab to the top is being placed in the wrong quadrant (e instead of n) because of this. I had the same bug in a file dialog due to the same issue:

I'm unable to repro both of your pictures with e.g.

    // Test 1
    {
        ImGui::SetCursorPosX(ImGui::GetCursorPosX() + 90);
        ImGui::Button("GAMES", ImVec2(80, 25));
        ImGui::SameLine(0, 10);
        ImGui::Button("OPTIONS", ImVec2(100, 25));
        ImGui::SetCursorPosY(ImGui::GetCursorPosY() + 10);
        ImGui::Button("Library", ImVec2(170, 100));
        ImGui::SameLine(0, 10);
        ImGui::Button("Audio", ImVec2(170, 100));
        ImGui::Button("Video", ImVec2(170, 100));
        ImGui::SameLine(0, 10);
        ImGui::Button("Input", ImVec2(170, 100));
    }

    // Test 2
    {
        ImGui::BeginChild("List", ImVec2(0, 100), true, ImGuiWindowFlags_NavFlattened);
        for (int n = 0; n < 10; n++)
        {
            char buf[32];
            sprintf(buf, "File %d", n);
            ImGui::Selectable(buf);
        }
        ImGui::EndChild();
        ImGui::SetCursorPosX(ImGui::GetCursorPosX() + 300);
        ImGui::Button("Cancel", ImVec2(100, 30));
    }

And don't know how the result in your screenshot can be possible, since the committed code aggressively scale the horizontal distances:

    if (dby != 0.0f && dbx != 0.0f)
       dbx = (dbx/1000.0f) + ((dbx > 0.0f) ? +1.0f : -1.0f);

I'm not saying this is correct, but your examples above appear to be faulty to me, based on a local change you made? Does the top screenshots uses child window with NavFlattened? I wonder if your implement/hack of NavFlattened back then created an issue, or if you add other changes and they snowballed into those issues.

Here's your patch rebased to current Navigation branch + with stylistic tweaks.
nav_scoring_inolen_20180206.zip

omar · Answer 45 · Wed Feb 07 2018 03:13:32 GMT+0800 (China Standard Time)

I will merge the navigation branch soon

Some important changes:

To use the keyboard, you only need to set io.NavFlags |= ImGuiNavFlags_EnableKeyboard
NewFrame() will automatically fill io.NavInputs[] based on your io.KeyDown[] + io.KeyMap[] arrays.
Note that a missing enum for the space bar ImGuiKey_Space was added. NewFrame will assert if you set ImGuiNavFlags_EnableKeyboard without mapping this key in io.KeyMap[].
To use the gamepad, set io.NavFlags |= ImGuiNavFlags_EnableGamepad and fill the io.NavInputs[] array, here are the enums you want to fill:

    ImGuiNavInput_Activate,      // activate / open / toggle / tweak value       // e.g. Circle (PS4), A (Xbox), B (Switch), Space (Keyboard)
    ImGuiNavInput_Cancel,        // cancel / close / exit                        // e.g. Cross  (PS4), B (Xbox), A (Switch), Escape (Keyboard)
    ImGuiNavInput_Input,         // text input / on-screen keyboard              // e.g. Triang.(PS4), Y (Xbox), X (Switch), Return (Keyboard)
    ImGuiNavInput_Menu,          // tap: toggle menu / hold: focus, move, resize // e.g. Square (PS4), X (Xbox), Y (Switch), Alt (Keyboard)
    ImGuiNavInput_DpadLeft,      // move / tweak / resize window (w/ PadMenu)    // e.g. D-pad Left/Right/Up/Down (Gamepads), Arrow keys (Keyboard)
    ImGuiNavInput_DpadRight,     // 
    ImGuiNavInput_DpadUp,        // 
    ImGuiNavInput_DpadDown,      // 
    ImGuiNavInput_LStickLeft,    // scroll / move window (w/ PadMenu)            // e.g. Left Analog Stick Left/Right/Up/Down
    ImGuiNavInput_LStickRight,   // 
    ImGuiNavInput_LStickUp,      // 
    ImGuiNavInput_LStickDown,    // 
    ImGuiNavInput_FocusPrev,     // next window (w/ PadMenu)                     // e.g. L1 or L2 (PS4), LB or LT (Xbox), L or ZL (Switch)
    ImGuiNavInput_FocusNext,     // prev window (w/ PadMenu)                     // e.g. R1 or R2 (PS4), RB or RT (Xbox), R or ZL (Switch) 
    ImGuiNavInput_TweakSlow,     // slower tweaks                                // e.g. L1 or L2 (PS4), LB or LT (Xbox), L or ZL (Switch)
    ImGuiNavInput_TweakFast,     // faster tweaks                                // e.g. R1 or R2 (PS4), RB or RT (Xbox), R or ZL (Switch)

Clément Grégoire · Answer 46 · Sat May 05 2018 18:53:03 GMT+0800 (China Standard Time)

Nice work so far, but I have a small request.
Make PageUp/PageDown scroll !
I guess it wouldn't be too hard to implement (just need to fake a mouse scroll ?). It would be really useful for long lists/texts. I'm not sure if this has already been implemented on the client side before by some people though. Perhaps have a flag for it ?

omar · Answer 47 · Sat Aug 04 2018 00:54:02 GMT+0800 (China Standard Time)

FYI I suspect this is not very well known so I'll post a GIF, but you can CTRL+Tab between windows:

The interface was tweaked a bit a few weeks ago (for the purpose of the Docking feature), it now displays a list of windows a swell. (in 1.62 CTRL+Tab worked but without the windows list).

Alzathar · Answer 48 · Sat Aug 04 2018 05:05:29 GMT+0800 (China Standard Time)

@ocornut Thanks for this information. I tested this feature in the example_glfw_opengl3 binary on macOS 10.12.6 with ImGui 1.63 WIP (b217251) but there is a problem.

I activated the flag io.ConfigFlags: NavEnableKeyboard [beta] and I tried to use the shortcut CTRL + Tab. However nothing happened. The problem is because the key combination is not detected. When I look in the section 'Keyboard, Mouse and Navigation state', only the key '341' (CTRL) is detected, instead of the keys '258' (Tab) and '341'. The strange thing is that if I press Tab first and then CTRL, I see the list of windows and the white rectangle. If I release the Tab key, the window switch stops . But because the CTRL + Tab combination is not detected, I cannot switch to another window.

omar · Answer 49 · Sat Aug 04 2018 05:48:18 GMT+0800 (China Standard Time)

@Alzathar: I believe this could be an issue with Glfw, seeing the OSX.mm code has similar issues. I don’t use a Mac daily but if you could log the Glfw key events and find out if it behave correctly with those keys it would be nice.

omar · Answer 50 · Sat Aug 04 2018 20:55:24 GMT+0800 (China Standard Time)

@Lectem:

Nice work so far, but I have a small request. Make PageUp/PageDown scroll ! I guess it wouldn't be too hard to implement (just need to fake a mouse scroll ?).

Forgot to answer this one. PageUp/PageDown support was added in 1.62 see 6d98c03 (standard pageup/pagedown is not just faking a mouse scroll). However it's not yet supported inside InputTextMultiline(), will do later.

@Alzathar: If you could dig further into this, could you open a new issue for it later? Thanks!

Alzathar · Answer 51 · Sat Aug 04 2018 20:55:57 GMT+0800 (China Standard Time)

@ocornut I tracked the problem and this is not related to Dear ImGui. From what I read the combination CTRL + Tab is specially handled by OSX/Cocoa text input method. By patching GFLW 3.3 with the proposed code here, the key combination is correctly sent to GLFW and then Dear ImGui.

I do not see a way to fix this for the GLFW examples except if you decide to use another key combination (or if a patched version of GFLW is used).

Extra: I was not able to use the Xcode project proposed for the 'example_apple_metal' binary. Xcode 9.2 sends the error message "Incompatible project version". Thanks to the README file I created a macOS Game project and copied/replaced files. This version works without any modification in the code. The key combination CTRL + Tab is correctly recognized.

omar · Answer 52 · Sat Aug 04 2018 21:00:56 GMT+0800 (China Standard Time)

Thanks for the feedback. Could you post the second part of your message in #1929? (Also linking #1873).
Do you know what would be the Mac equivalent for CTRL+Tab to select windows within an application?

Alzathar · Answer 53 · Sat Aug 04 2018 21:29:40 GMT+0800 (China Standard Time)

By default, macOS proposed the shortcut 'command + backtick' (⌘ + `) as mentioned in this answer. However, on my laptop it does not work (or I do not understand something). Moreover, it is difficult to have the backtick character on several keyboard layouts. This post shows the key combination to have it (this is under Windows, but this is similar under macOS).

Regarding InputTextMultiline() and pageup/pagedown, I will look into it in the next days.

Alzathar · Answer 54 · Fri Oct 19 2018 11:47:46 GMT+0800 (China Standard Time)

@ocornut For your information, I submitted the PR glfw/glfw#1362 to handle correctly the combination Ctrl + Tab under macOS with GLFW.

omar · Answer 55 · Tue Sep 17 2019 01:18:15 GMT+0800 (China Standard Time)

Added support for Home/End keys which was missing until now.

(This led me to discover and fix a couple of subtle issues where some code path would layout differently when they were clipped or not, resulting in the ScrollMax value to vary slightly depending on current scroll value. The effect of that (before I fixed the bugs) being that pressing END home would sometimes put you near the maximum scroll, but off by a few pixels, and pressing END again would fix it to the new maximum scroll.)

omar · Answer 56 · Mon Feb 10 2020 18:52:33 GMT+0800 (China Standard Time)

A reminder that keyboard navigation allows you to use CTRL+Tab:

omar · Answer 57 · Fri Aug 28 2020 01:42:34 GMT+0800 (China Standard Time)

Closing this old general issues. There still a bunch of open requests/issues/todos related to navigation but the general intro thread doesn't seem necessary anymore.

Gamepad / Keyboard navigation and interactions!

EDIT: The navigation branch has been merged to master. See #1599. Further improvements/fixes will be pushed to mainline.

Push/Pop default font when drawing nav debug overlay

Reenable NavFlattened support

Expose ImGuiItemFlags_NoNavDefaultFocus

Made NoNavFocus also disable unfocus support

Added NoNavScroll window flag

Remove nav scoring biases

Set NavDisableHighlight to make sure the request initiated by NavInitWindow is used

Don't reinit navigation when activating a window that has a valid NavLastId

Updated navigation scrolling code to work when NavFlattened is used

Always keep something selected