Support for variadic version of the Format function

Question

Support for variadic version of the Format function

glenfletcher opened this issue 11 years ago · comments

The stream operator << dosen't work as expected, based on the use being equivalent to std::ostream.

fmt::Formatter<> msg = fmt::Format(format);
msg << arg1;
msg << arg2;
...
msg << argN;
result = fmt::str(msg);

throws exception, to many formatting tokens on the second line, as it immediately attempts to format the string, but not all arguments have been passed.

P.S. I need to use this method as my code is actually

template<typename... Args> inline std::string const & Format(std::string const &format,Args const & ... args) {
        fmt::Formatter<> msg = fmt::Format(format);
        RescursiveFormat(msg,args...);
        return *new std::string(fmt::str(msg)); }

Where RescursiveFormat(msg,args...); reduce to (by template meta programing, and compiler optimization)

msg << arg1;
msg << arg2;
...
msg << argN;

For however may arguments are passed.

This allows formatting to be done by
Format(<format_string>,...) i.e. like sprintf, by type-safe, and using format as the underlining formatting method.

Victor Zverovich · Answer 1 · Fri Dec 06 2013 00:56:28 GMT+0800 (China Standard Time)

To have performance comparable to that of sprintf, Format should take all its arguments at once (in one full expression). Otherwise it would have to take into account the lifetime of temporaries and either copy or preformat its arguments. In fact, in older versions of the format library this code would give a compilation error rather than exception to prevent incorrect use of the API, but there was a regression in recent versions. I have fixed the regression and it gives a compilation error again.

Formatting API has no direct equivalent in IOStreams. IOStreams use concatenation-based API which is more comparable to that of fmt::Writer:

fmt::Writer w;
w << arg1;
w << arg2;
...

That said I'd very much want to add support for variadic version of the Format function similar to the one from your example. The easiest solution is to make BasicFormatter::Arg public. Then the variadic Format function can be implemented as follows:

inline void CollectArgs(std::deque<fmt::BasicFormatter<char>::Arg> &) {}
template<typename ArgT, typename... Args>
inline void CollectArgs(
    std::deque<fmt::BasicFormatter<char>::Arg> &format_args,
    const ArgT &arg, const Args & ... args) {
  format_args.emplace_back(arg);
  CollectArgs(format_args, args...);
}
template<typename... Args>
inline std::string Format(const StringRef &format, const Args & ... args) {
  std::deque<fmt::BasicFormatter<char>::Arg> format_args;
  CollectArgs(format_args, args...);
  fmt::Writer w;
  fmt::BasicFormatter<char> f(w, format.c_str());
  for (const fmt::BasicFormatter<char>::Arg &arg : format_args)
    f << arg;
  return fmt::str(f);
}

I think there are better ways to implement this that will require minor extension of the BasicFormatter API, so I'll leave this issue open for now.

Glen Fletcher · Answer 2 · Fri Dec 06 2013 15:05:21 GMT+0800 (China Standard Time)

I've do CollectArgs as

template<typename ArgT, typename... Args>
inline void CollectArgs(
    std::deque<fmt::BasicFormatter<char>::Arg> &format_args,
    const ArgT &arg, const Args & ... args) {
  format_args.emplace_back(arg);
  if (sizeof...(Args)>0) { CollectArgs(format_args, args...); }
}

The the condition is either true or false, and the compiler optimization will eliminate it

And then Format would become

template<typename... Args>
inline std::string Format(const StringRef &format, const Args & ... args) {
  if (sizeof...(Args)>0) {
  std::deque<fmt::BasicFormatter<char>::Arg> format_args;
  CollectArgs(format_args, args...);
  fmt::Writer w;
  fmt::BasicFormatter<char> f(w, format.c_str());
  for (const fmt::BasicFormatter<char>::Arg &arg : format_args)
    f << arg;
  return fmt::str(f);
  } else { return fmt::str(format); }
}

The code is easier to understand this way I believe. The compiled output should be the same however, except for Format with no args which would be sightly more efficient. As it doesn't construct then illiterate the empty format_args, or attempt to format a string with no args, however it also eliminate any error checking if a string is passed with formatting tokens.

As for better ways to implement I would go for adding an apply formatting member to the BasicFormatter so the code would be come:

template<typename... Args>
inline std::string Format(const StringRef &format, const Args & ... args) {
  if (sizeof...(Args)>0) {
  std::deque<fmt::BasicFormatter<char>::Arg> format_args;
  CollectArgs(format_args, args...);
  fmt::Writer w;
  fmt::BasicFormatter<char> f(w, format.c_str());
  /* Here we replace the internal list, and then format the string,
      should be more efficient as we replace the list rather than
      rebuilding/copying the list
  */
  f.apply_args(format_args);
  return fmt::str(f); } else { return fmt::str(format); }
}

Victor Zverovich · Answer 3 · Fri Dec 06 2013 22:54:04 GMT+0800 (China Standard Time)

I think it is more consistent if the Format with no args checks the format string. Having the same function behave differently depending on the number of arguments may be confusing to users. Besides, other formatting functions such as printf & str.format check the format string in the case of no arguments. The efficiency doesn't matter in this case because if there are no formatting arguments, the formatting is probably no needed in the first place and the format string can be used directly.

I agree that something like apply_args or type-safe vprintf is needed in BasicFormatter or Writer.

Victor Zverovich · Answer 4 · Sat Dec 07 2013 01:14:21 GMT+0800 (China Standard Time)

OK here's the initial attempt to provide support for variadic functions in BasicFormatter: 5789930

It makes writing the variadic Format function as simple as:

template<typename... Args>
inline std::string Format(const StringRef &format, const Args & ... args) {
  fmt::Writer w;
  fmt::BasicFormatter f(w, format.c_str(), {args...});
  return fmt::str(f);
}

Glen Fletcher · Answer 5 · Sun Dec 08 2013 08:51:36 GMT+0800 (China Standard Time)

Dose the rest of the code depend upon C++11?, if not you should wrap the new code in preprocessed tags to verify that C++11 is supported, as I known VC++2010 (still in use by lots of people I believe) doesn't support C++11. And this change would break the library for them.

Victor Zverovich · Answer 6 · Tue Dec 10 2013 00:04:35 GMT+0800 (China Standard Time)

The rest of the code is C++98, so that's right, the C++11 part should be conditionally compiled.

patlecat · Answer 7 · Tue Dec 10 2013 00:19:41 GMT+0800 (China Standard Time)

VC2010 can be easily disregarded after VC2013 came out - ferchrissakes it's basically 2014 now! I would also vote for the use of C++11 wherever it makes sense and is at least supported by the latest compilers. With conditionals you can maintain C++98 compatibility as long as your heart desires :D

Glen Fletcher · Answer 8 · Sat Dec 14 2013 12:46:08 GMT+0800 (China Standard Time)

I can confirm that the variadic format now works on cygwin 1.7.25, compiled with g++ 4.7.3 using -std=gnu++11 to provide C++11 support.

using the following code:

template<typename... Args> inline std::string const Format(std::string const &format,Args const & ... args) {
        fmt::Writer w;
        fmt::BasicFormatter<char> f(w, format.c_str(), {args...});
        return fmt::str(f); }
template<typename... Args> inline std::string const Format(uint16_t const format,Args const & ... args) { return Format(GetMsg(format),args...); }
    template<typename... Args> inline std::string const Format(char const * format,Args const & ... args) { return Format(GetMsg(format),args...); }

and

} catch (std::exception &e) {
        try {
            _LogError(LANG_ID::FATAL_EXCEPTION,e.what());
            OutputBacktrace(std::cout); // Wrtie up 200 levels of backtrace to STDOUT
        } catch (std::exception &ee) {
            std::cout << "ERROR: An unhandled exception has occured" << std::endl;
            std::cout << "\t" << e.what() << std::endl;
            std::cout << "ERROR: " << ee.what() << std::endl;
        }
        return EXIT_FAILURE;
    }

and

virtual const char* what() const throw() { return I18N::Format(LANG_ID::THREAD_EXCEPTION,this->name,this->code).c_str(); }

where _LogError(Format,...) calls std::cerr<<I18N::Format(LANG_ID::ERROR)<<I18N::Format(Format,...)<<std::endl;

LANG_ID::THREAD_EXCEPTION maps to

General Thread Exception:
        {0:s} returned {1:X}

and LANG_ID::FATAL_EXCEPTION maps to

An unhandled exception has occured
    e.what() returned: {0:s}
    Program Ternimated!

and LANG_ID::ERROR maps to ERROR:

Generates the output due to an exception

ERROR: An unhandled exception has occured
    e.what() returned: General Thread Exception:
        pthread_mutex_init returned 16
    Program Ternimated!

Overall this confirms the code works with 0,1 and 2 arguments in my compilation.

Victor Zverovich · Answer 9 · Mon Dec 16 2013 07:40:34 GMT+0800 (China Standard Time)

Cool, thanks for checking it out. I think it's time to close the issue then.