don't rely on GetLastError() to detect errors in enable_ansi_support

[Sergey Galich]

rust-ansi-term/src/windows.rs

Lines 20 to 36 in 36b0ff3

 let std_out_handle = GetStdHandle(STD_OUT_HANDLE); 
 let error_code = GetLastError(); 
 if error_code != 0 { return Err(error_code); } 
  
 // https://docs.microsoft.com/en-us/windows/console/getconsolemode 
 let mut console_mode: u32 = 0; 
 GetConsoleMode(std_out_handle, &mut console_mode); 
 let error_code = GetLastError(); 
 if error_code != 0 { return Err(error_code); } 
  
 // VT processing not already enabled? 
 if console_mode & ENABLE_VIRTUAL_TERMINAL_PROCESSING == 0 { 
     // https://docs.microsoft.com/en-us/windows/console/setconsolemode 
     SetConsoleMode(std_out_handle, console_mode | ENABLE_VIRTUAL_TERMINAL_PROCESSING); 
     let error_code = GetLastError(); 
     if error_code != 0 { return Err(error_code); } 
 } 

GetStdHandle() can return valid handle and GetLastError() will return ERROR_NOT_SUPPORTED (50). Error must be checked only when GetStdHandle() returns INVALID_HANDLE_VALUE (-1). It happens when GUI app creates console via AllocConsole() and tries to call enable_ansi_support().

GetConsoleMode() and SetConsoleMode() return 0 (FALSE) when they fail, GetLastError() should be called after that.

[Roy Ivy III]

@ogham , any motion on this? Or the corresponding PR(#43)?

[Benjamin Sago]

Thanks for going through the Windows side of this; sorry again it took so long for me to merge.