Accumulation of user statstics
rkaminsk opened this issue · comments
Currently, user statistics are accumulated, in the sense that the existing values can be updated at each step. This results in unexpected output with --stats -q0,0,0. At each step accumulated values are presented and the user statistics of the last step and in the accumulated values are identical. Should we give the user the possibility to add per step and accumulated values to better integrate with the handling of clasp's other statistics?
Does anyone actually need this feature? I'm also not sure how such a feature should look like. Do you want to have another special root key?
So far no one. I just felt that it does not quite fit with the handling of the other statistics because in the existing statistics we are using an accu key under which accumulated statistics are stored.
This could be solved with a second root, which we could put under [accu][user_defined]. The statistics callback could then simply take two root keys as arguments. A simple implementation could then only fill the accumulated statistics and, hence, no additional per-step statistics would be printed, which would cover simple use cases. In any case, I would leave the accumulation to the user.
I was mentioning this because multi-shot solving in clingo is important. So getting the per-step statistic right is probably a good idea.
I continued working on clingo's statistics branch. It turned out that with our current API it is not really necessary to use callbacks at all. So I decided to kick them out. The below patch removes the callbacks from clasp and adds support for per step and accumualted statistics. If you'ld like to keep the callbacks in your interface please feel free to do so, removing them is just a suggestion.
You might also want to remove the changeRoot logic in the statistics. I think it is not needed anymore.
What I would like to have is the extra logic to distinguish per step and accumulated statistics. I had to do two things to achieve this. Pass the final argument to the statistics visitor because it did not arrive in the ClingoView. Please check this because I did not fully understand how things are handled there. Is it still correct with this patch or is there another way to figure this out? The two statistics types are distinguished using two different keys added under the root level: "user_step" and "user_accu". Feel free to rename them. Their contents are printed by the text output if present.
diff --git a/clasp/clasp_facade.h b/clasp/clasp_facade.h
index 865bc45..8e6dd1b 100644
--- a/clasp/clasp_facade.h
+++ b/clasp/clasp_facade.h
@@ -235,7 +235,7 @@ public:
SumVec lower() const;
//@}
//! Visits this summary object.
- void accept(StatsVisitor& out) const;
+ void accept(StatsVisitor& out, bool final) const;
FacadePtr facade; //!< Facade object of this run.
double totalTime; //!< Total wall clock time.
double cpuTime; //!< Total cpu time.
@@ -330,19 +330,6 @@ public:
*/
bool read();
- typedef void(*StatsCallback)(Potassco::AbstractStatistics*, void*);
- typedef Potassco::Span<std::pair<StatsCallback, void*> > StatsCallbacks;
-
- //! Adds a callback for defining external (i.e. user-defined) statistics.
- /*!
- * \param cb Callback method to be called once before and after each solve step.
- * \param data Additional data passed back on each call to cb.
- */
- void addStatisticsCallback(StatsCallback cb, void* data);
-
- //! Gets all registered callbacks for defining external statistics.
- StatsCallbacks getStatisticsCallbacks() const;
-
//@}
/*!
diff --git a/src/clasp_facade.cpp b/src/clasp_facade.cpp
index 8188c70..0a22a43 100644
--- a/src/clasp_facade.cpp
+++ b/src/clasp_facade.cpp
@@ -571,8 +571,6 @@ double _getExhausted(const SolveResult* r) { return static_cast<double>(r->exhau
}
struct ClaspFacade::Statistics {
- typedef std::pair<StatsCallback, void*> UserSource;
- typedef PodVector<UserSource>::type CallbackVec;
Statistics(ClaspFacade& f) : self_(&f), tester_(0), level_(0), clingo_(0) {}
~Statistics() { delete clingo_; delete solvers_.multi; }
void start(uint32 level);
@@ -580,9 +578,7 @@ struct ClaspFacade::Statistics {
void end();
void addTo(StatsMap& solving, StatsMap* accu) const;
void accept(StatsVisitor& out, bool final) const;
- void addExternal(StatsCallback cb, void* data);
bool incremental() const { return self_->incremental(); }
- Potassco::Span<UserSource> external() const { return clingo_ ? clingo_->external() : Potassco::toSpan<UserSource>(); }
typedef StatsVec<SolverStats> SolverVec;
typedef SingleOwnerPtr<Asp::LpStats> LpStatsPtr;
typedef PrgDepGraph::NonHcfStats TesterStats;
@@ -598,9 +594,7 @@ struct ClaspFacade::Statistics {
public:
explicit ClingoView(const ClaspFacade& f);
void update(const Statistics& s);
- void addExternal(StatsCallback cb, void* data) { sources_.push_back(UserSource(cb, data)); }
- Potassco::Span<UserSource> external() const { return Potassco::toSpan(sources_); }
- Key_t user() const;
+ Key_t user(bool final) const;
private:
struct StepStats {
SummaryStats times;
@@ -617,12 +611,10 @@ struct ClaspFacade::Statistics {
StatsMap* keys_;
StatsMap problem_;
StatsMap solving_;
- CallbackVec sources_; // callbacks for user-defined stats
struct Summary : StatsMap { StepStats step; } summary_;
struct Accu : StatsMap { StepStats step; StatsMap solving_; };
typedef SingleOwnerPtr<Accu> AccuPtr;
AccuPtr accu_;
- Key_t user_;
}* clingo_; // new clingo stats interface
ClingoView* getClingo();
};
@@ -635,9 +627,6 @@ void ClaspFacade::Statistics::initLevel(uint32 level) {
tester_ = self_->ctx.sccGraph->nonHcfStats();
}
}
-void ClaspFacade::Statistics::addExternal(StatsCallback cb, void* data) {
- getClingo()->addExternal(cb, data);
-}
void ClaspFacade::Statistics::start(uint32 level) {
// cleanup previous state
@@ -683,8 +672,8 @@ void ClaspFacade::Statistics::accept(StatsVisitor& out, bool final) const {
const SolverVec& solver = final ? accu_ : solver_;
const uint32 nThreads = final ? (uint32)accu_.size() : self_->ctx.concurrency();
const uint32 nSolver = (uint32)solver.size();
- if (clingo_ && clingo_->user()) {
- out.visitExternalStats(clingo_->getObject(clingo_->user()));
+ if (clingo_ && clingo_->user(final)) {
+ out.visitExternalStats(clingo_->getObject(clingo_->user(final)));
}
if (nThreads > 1 && nSolver > 1 && out.visitThreads(StatsVisitor::Enter)) {
for (uint32 i = 0, end = std::min(nSolver, nThreads); i != end; ++i) {
@@ -708,7 +697,6 @@ ClaspFacade::Statistics::ClingoView* ClaspFacade::Statistics::getClingo() {
}
ClaspFacade::Statistics::ClingoView::ClingoView(const ClaspFacade& f) {
keys_ = makeRoot();
- user_ = 0;
summary_.add("call" , StatisticObject::value(&f.step_.step));
summary_.add("result" , StatisticObject::value<SolveResult, _getResult>(&f.step_.result));
summary_.add("signal" , StatisticObject::value<SolveResult, _getSignal>(&f.step_.result));
@@ -732,18 +720,12 @@ ClaspFacade::Statistics::ClingoView::ClingoView(const ClaspFacade& f) {
accu_->step.bind(*f.accu_.get());
}
}
-Potassco::AbstractStatistics::Key_t ClaspFacade::Statistics::ClingoView::user() const {
- return user_;
+Potassco::AbstractStatistics::Key_t ClaspFacade::Statistics::ClingoView::user(bool final) const {
+ Key_t key = 0;
+ find(root(), final ? "user_accu" : "user_step", &key);
+ return key;
}
void ClaspFacade::Statistics::ClingoView::update(const ClaspFacade::Statistics& stats) {
- if (!sources_.empty()) {
- user_ = add(root(), "user_defined", Potassco::Statistics_t::Map);
- Key_t oldRoot = changeRoot(user_);
- for (CallbackVec::iterator it = sources_.begin(), end = sources_.end(); it != end; ++it) {
- it->first(this, it->second);
- }
- changeRoot(oldRoot);
- }
if (stats.level_ > 0 && accu_.get() && keys_->add("accu", accu_->toStats())) {
accu_->step.addTo(*accu_);
accu_->add("solving", accu_->solving_.toStats());
@@ -957,15 +939,6 @@ bool ClaspFacade::read() {
return true;
}
-void ClaspFacade::addStatisticsCallback(StatsCallback cb, void* data) {
- POTASSCO_REQUIRE(stats_.get(), "Statistics not yet available");
- stats_->addExternal(cb, data);
-}
-
-ClaspFacade::StatsCallbacks ClaspFacade::getStatisticsCallbacks() const {
- return stats_.get() ? stats_->external() : Potassco::toSpan<std::pair<StatsCallback, void*> >();
-}
-
void ClaspFacade::prepare(EnumMode enumMode) {
POTASSCO_REQUIRE(solve_.get() && !solving());
EnumOptions& en = config_->solve;
@@ -1093,8 +1066,8 @@ SumVec ClaspFacade::Summary::lower() const {
}
return SumVec();
}
-void ClaspFacade::Summary::accept(StatsVisitor& out) const {
- if (facade->solved()) { facade->stats_->accept(out, this == facade->accu_.get()); }
+void ClaspFacade::Summary::accept(StatsVisitor& out, bool final) const {
+ if (facade->solved()) { facade->stats_->accept(out, final); }
}
}
diff --git a/src/clasp_output.cpp b/src/clasp_output.cpp
index d4461c0..68b8cc2 100644
--- a/src/clasp_output.cpp
+++ b/src/clasp_output.cpp
@@ -675,10 +675,10 @@ void JsonOutput::printSummary(const ClaspFacade::Summary& run, bool final) {
}
}
}
-void JsonOutput::printStatistics(const ClaspFacade::Summary& summary, bool) {
+void JsonOutput::printStatistics(const ClaspFacade::Summary& summary, bool final) {
if (hasWitness()) { popObject(); }
pushObject("Stats", type_object);
- summary.accept(*this);
+ summary.accept(*this, final);
popObject();
}
/////////////////////////////////////////////////////////////////////////////////////////
@@ -833,10 +833,10 @@ void TextOutput::printSummary(const ClaspFacade::Summary& run, bool final) {
}
}
}
-void TextOutput::printStatistics(const ClaspFacade::Summary& run, bool) {
+void TextOutput::printStatistics(const ClaspFacade::Summary& run, bool final) {
printBR(cat_comment);
accu_ = true;
- run.accept(*this);
+ run.accept(*this, final);
}
void TextOutput::startStep(const ClaspFacade& f) {
Output::startStep(f);
diff --git a/tests/facade_test.cpp b/tests/facade_test.cpp
index 7256db6..d580120 100644
--- a/tests/facade_test.cpp
+++ b/tests/facade_test.cpp
@@ -981,10 +981,9 @@ static void getStatsKeys(const Potassco::AbstractStatistics& stats, Potassco::Ab
}
}
-static void addExternalStats(Potassco::AbstractStatistics* us, void*) {
+static void addExternalStats(Potassco::AbstractStatistics* us, Potassco::AbstractStatistics::Key_t rootkey) {
typedef Potassco::AbstractStatistics::Key_t Key_t;
- Key_t rootkey = us->root();
Key_t general = us->add(rootkey, "deathCounter", Potassco::Statistics_t::Map);
REQUIRE(us->get(rootkey, "deathCounter") == general);
REQUIRE(us->type(general) == Potassco::Statistics_t::Map);
@@ -1182,16 +1181,16 @@ TEST_CASE("Facade statistics", "[facade]") {
REQUIRE(stats.size(arr) == 1);
}
SECTION("testClingoUserStats") {
+ typedef Potassco::AbstractStatistics::Key_t Key_t;
Clasp::Asp::LogicProgram& asp = libclasp.startAsp(config, true);
lpAdd(asp, "{x1;x2;x3}. #minimize{x1, x2}.");
- libclasp.addStatisticsCallback(addExternalStats, 0);
libclasp.prepare();
libclasp.solve();
Potassco::AbstractStatistics* stats = libclasp.getStats();
- typedef Potassco::AbstractStatistics::Key_t Key_t;
Key_t r = stats->root();
+ Key_t u = stats->add(r, "user_defined", Potassco::Statistics_t::Map);
+ addExternalStats(stats, u);
REQUIRE(stats->type(r) == Potassco::Statistics_t::Map);
- Key_t u = stats->get(r, "user_defined");
REQUIRE(stats->type(u) == Potassco::Statistics_t::Map);
Key_t user = stats->get(u, "deathCounter");
REQUIRE(stats->type(user) == Potassco::Statistics_t::Map);Hi Roland,
I like your patch but I don't understand why you had to change the signature of Summary::accept(). The final actually means accumulated and this information is already part of the Summary object accepting the visitor.
My problem was that the final argument of ClaspFacade::Statistics::accept was always false. Like I said, I don't quite understand the meaning of this parameter.
Final should be true only for the accumulated summary, i.e. ClaspFacade::accu_. The accumulated summary is returnd by ClaspFacade::summary(true) but only if a) incremental solving is active and b) more than one step was solved. When you say:
My problem was that the final argument of ClaspFacade::Statistics::accept was always false.
Can you elaborate on when (i.e. in which context) this was the case?
Furthermore, you write:
What I would like to have is the extra logic to distinguish per step and accumulated statistics. I had to do two things to achieve this. Pass the final argument to the statistics visitor because it did not arrive in the ClingoView.
What is the second thing you had to do?
That might be the reason. I solved only one step. So the assumption is that for the first step both keys are identical? In my example I was experimenting with accumulated statistics and was wondering why they do not appear. I guess this is an optimization to only have accumulated statistics for steps > 0. The downside is that this makes things more difficult to explain to the user and one always has to add both per step and accumulated statistics in the multi-shot case. We can of course document this.
I removed the questionable optimization for the missing accumulation of the first step and also got rid of the callback interface (as suggested in your patch). Summary::accept() now looks for 'user_step' or 'user_accu' depending on whether the summary belongs to the current step or the accumulation of all steps.
I close the issue now but feel free to reopen it, if I missed something important.
Thanks, that should be everything.