Question: Help on display of ccache stats
EstebanDugueperoux2 opened this issue · comments
Hi conan community,
I would like to write a conan hook to display ccache logs stats for build whose recipe use ccache as tool_requires.
My current draft looks like:
import os
from re import match
import subprocess
def pre_build(output, conanfile, **kwargs):
assert conanfile
try:
useCcache = list(filter(lambda v: match('ccache*', v), conanfile.tool_requires))
if useCcache:
os.environ["CCACHE_STATSLOG"] = "/tmp/" + os.environ.get("CI_JOB_ID") + ".stats_log"
except:
output.info("An exception occurred")
def post_build(output, conanfile, **kwargs):
assert conanfile
try:
useCcache = list(filter(lambda v: match('ccache*', v), conanfile.tool_requires))
if useCcache:
output.info("ccache log stats:")
ccache_output = subprocess.check_output("~/.conan/data/ccache/4.6/_/_/package/36b155e63e5609b97722bf9ba648ac9c2815d17d/bin/ccache --show-log-stats", stderr=subprocess.STDOUT, shell=True)
output.info(ccache_output)
except subprocess.CalledProcessError as e:
output.warn(e.output)
except:
output.info("An exception occurred")
But when calling "output.info(ccache_output)" following is displayed:
[HOOK - ccache_show_log_stats.py] post_build(): b'Summary:\n Hits: 0 / 1388 (0.00 %)\n Direct: 0 / 1388 (0.00 %)\n Preprocessed: 0 / 1388 (0.00 %)\n Misses: 1388\n Direct: 1388\n Preprocessed: 1388\n Uncacheable: 479\nPrimary storage:\n Hits: 0 / 1388 (0.00 %)\n Misses: 1388\n\nUse the -v/--verbose option for more details.\n'
Someone do a tip to have '`n' interpreted by 'output.info()' method?
Another question, how in my post_build hook, invoke ccache from my conan cache properly?
Regards.
.info() method just expects a string, the issue is that the subprocess.check_output is returning you binary bytes. If you want a string, you need to decode() it before passing it to info(). Is this your issue?
The other question, you might try to run conanfile.run() and that will activate the same environment as the recipe does, so it will have the ccache binary in its path.
Hi @memsharded,
Thanks for your help, it works fine :).
Then I post the final snippet for the community:
import os
from re import match
def pre_build(output, conanfile, **kwargs):
assert conanfile
try:
useCcache = list(filter(lambda v: match('ccache*', v), conanfile.tool_requires))
if useCcache:
os.environ["CCACHE_STATSLOG"] = "/tmp/" + os.environ.get("CI_JOB_ID") + ".stats_log"
except:
output.info("An exception occurred")
def post_build(output, conanfile, **kwargs):
assert conanfile
try:
useCcache = list(filter(lambda v: match('ccache*', v), conanfile.tool_requires))
if useCcache:
output.info("ccache log stats:")
conanfile.run("ccache --show-log-stats")
except subprocess.CalledProcessError as e:
output.warn(e.output)
except:
output.info("An exception occurred")
Regards.
I talk too fast, "conanfile.run("ccache --show-log-stats")" give me "/bin/sh: 1: ccache: not found"
That might depend on how the consuming recipe is configured: which generators are being used? does it run self.run("ccache") in its build() method without problems?
I use CMakeToolChain and CMakeDeps.
I'm in a Gitlab CI Docker context with gcc8-ubuntu18.04 Docker image with conan upgrade to latest 1.59.0.
And I have update my recipe to have:
def build(self):
self.run("ccache --version")
self.run("which ccache")
And I have a correct result:
conanfile.py (myPackage/4.0.0): Calling build()
ccache version 4.6
Features: file-storage http-storage redis-storage
Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2022 Joel Rosdahl and other contributors
See <https://ccache.dev/credits.html> for a complete list of contributors.
This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 3 of the License, or (at your option) any later
version.
/home/conan/.conan/data/ccache/4.6/_/_/package/4e9a94dd29b7acfa5907f316bf127912ba258d11/bin/ccache
Regards.
I have no idea what could be failing and why it works in build() method and fails in the hook.
The code that runs internally is:
def run_build_method(conanfile, hook_manager, **hook_kwargs):
hook_manager.execute("pre_build", conanfile=conanfile, **hook_kwargs)
logger.debug("Call conanfile.build() with files in build folder: %s",
os.listdir(conanfile.build_folder))
with get_env_context_manager(conanfile):
conanfile.output.highlight("Calling build()")
with conanfile_exception_formatter(str(conanfile), "build"):
conanfile.build()
hook_manager.execute("post_build", conanfile=conanfile, **hook_kwargs)As you can see, there is nothing in between. Both execute inside the conanfile.build_folder too.
the "conanfile.run("ccache --show-log-stats")" should work as well as the calls in the build().
Can you please try to run ccache --version in the hook too? Just in case it is something special to --show-log-stats?
Oh, I think I have an idea, just realized while posting.
You might be using the legacy environment manangement that happens in-memory, versus the 2.0 ready one that is using files with env-vars defined. This later one is the one defining buildenv_info, [buildenv] in profiles, and managed with generators VirtualBuildEnv and VirtualRunEnv.
One of the advantages of this modern approach (has been available in 1.X for many releases already), is that it doesn't depend on the memory context or Conan execution. Anyone can easily reproduce, because environment is defined in a file that can be sourced. And this would make also the hook to work correctly, loading the environment that would allow to easily locate ccache executable.
Not sure to understand, you mean that the 2.0 way is not available in 1.x releases?
Yes, it is, it has been available in 1.X for a long time now.
But you might not be using it, that would be an explanation of why it is not working, that your environment management is still based on the legacy self.env_info instead of the new (1.X) self.buildenv_info + VirtualBuildEnv generator.