update man page to include diversity related parameters
mjsduncan opened this issue · comments
it was last updated in 2014.
also, https://github.com/opencog/moses/blob/master/doc/moses/README refers to a directory structure that seems to have disappeared after a round of code reorganization:
learning/moses/man -- a unix 'manual' page describing the executable.
learning/moses/documentation -- algorthm insights
learning/moses/diary -- a diary of experimental results
I should also add the technical report I wrote about diversity during the Aidyia years (stripped from the non-disclosable content). Another item to add in my Cantorian todo list.
i can copy the new stuff from the current help file into the man page. @linas do you remember where the stuff in learning/moses/documentation went so i can update that README.md file as well?
hopefully the snet service will generate more traffic to this repo and the relevant parts of the opencog wiki and i want to fix it up asap.
where the stuff in
learning/moses/documentationwent
I would have moved/copied everything, including the documentation. It's possible, that in all the shuffle, something was lost (it can be found in the deep github history) What I do remember thinking was that some of the original moses documentation was so wildly out of date, that it was just plain wrong -- referring to C++ classes that don't exist, algorithms that weren't implemented, etc. that it was unredeemable. That documentation I threw away. There was a fair amount of it. But I would have thrown that away before you ever used moses, so you shouldn't have any memory of it ...
Should I go hunt down and look through some of this old stuff?
The current repo contains this:
$ find |grep pdf
./moses/pleasure/pleasure/PleasureAlgorithm_v3-2.pdf
./doc/moses/QuickGuide.pdf
./doc/comboreduct/ComboReduct.pdf
The 2012 version of moses has less documentation:
cd opencog
git checkout 125f5254aa98e64c707dc143484c2eacfbc9362b
git log
shows moses work by Jade in 2012 while
$ find |grep pdf
./pleasure/pleasure/PleasureAlgorithm_v3-2.pdf
so PDF docs were gone by then... let me look earlier
Oh, huh... there were html docs in opencog/learning/moses/documentation that I wrote that seem to be gone. Are you thinking of the html docs?
uh, no, it seems the html documentation is still there: it's just not in the "docs" directory. Looking at the current git repo:
$ find |grep html
./moses/moses/documentation/Select_Exemplar.html
./moses/moses/documentation/Genetic_Crossover.html
and when opened in web browser, they display correctly ...
So nothing is missing, just unexpected locations maybe
I think we should move everything under doc what do you think?
Oh, well, actually the README in doc references the other places to find documentation so I suppose it is acceptable to let documentation dispersed. Just the paths should be fixed.
i'm referring to the current end of the man page which is duplicated in the opencog wiki:
TODO
Finish documenting these algo flags: --fs-scorer
-M --diversity-pressure --diversity-exponent --diversity-normalize
--diversity-dst --diversity-p-norm --diversity-dst2dp
-R discretize target var
These input flags: -G
Interesting patterns flags: -J -K -U -X
SEE ALSO
The eval-table man page.
More information is available at http://wiki.opencog.org/w/MOSES
AUTHORS
moses was written by Moshe Looks, Nil Geisweiller, and many others.
This manual page is being written by Linas Vepstas. It is INCOMPLETE.
Hah. Let me give you one of my characteristic (patent-pending) trademark answers.
I vaguely recall writing the above. There were not one, but two different efforts to promote diversity, my own, and one that Nil undertook. My own efforts were incomplete and inconclusive. I never fully understood what Nil did. I'm not sure if the above --diversity- flags control my code, or Nil's code. If its my code, then they "do something", that "should improve" speed of convergence/quality-of-result but it was unclear that it "worked well". That's the nature of unfinished work. Promoting diversity should improve results, just on vague theoretical abstract grounds. Actually making it work well in practice is harder.
The other flags -J -K -U -X were left-overs from Moshe Looks original code. They did ... something. It seemed ... interesting. Except it wasn't that interesting, didn't seem that useful, that promising. I did not explore these flags any further, because I decided (rightly or wrongly) that there was nothing promising in that direction.
Instead of documenting these, a valid alternative might be to remove them from the code. It is possible that they have already been removed from the code. It's possible that they are connected to code that doesn't actually work -- code that scrambles and garbages up results. In the "nice try, bad idea" category.
Its a lot of work to convert a "neat idea" into code that actually does something useful, and not all neat ideas can be converted. I believe that the undocumented flags are all "neat ideas" that have not reached the endpoint "useful code". Like hikers in the Himilayas, some died along the way. Maybe a few can still be rescued.
well we are attempting to use what ever is actually in the codebase to increase model diversity so if this stuff is "experimental" that should be noted in the help listing and updated in the man page:
$ moses | grep -A4 diversity
--diversity-pressure arg (=0) Set a diversity pressure on the
metapopulation. Programs behaving
similarily to others are more
penalized. That value sets the
importance of that penalty (from 0 to
--
--diversity-exponent arg (=-1) Set the exponent of the generalized
mean (or sum, if --diversity-normalize
is set to 0) aggregating the penalties
between a candidate and the set of all
candidates better than itself (taking
into account diversity). If the value
tends towards 0 it tends to the
geometric mean, towards +inf it tends
to the max function. If negative or
null is it the max function.
--
--diversity-normalize arg (=1) If set to 1 then the aggregating
function is a generalized mean.
Otherwize it is a generalized sum
(generalize mean * number of elements).
If --diversity-exponent is set to
negatively then this doesn't have any
impact as the aggregating function is
the max anyway.
--diversity-dst arg (=p_norm) Set the distance between behavioral
scores, then used to determin the
diversity penalty.3 distances are
available: p_norm, tanimoto and
angular.
--diversity-p-norm arg (=2) Set the parameter of the p_norm
distance. A value of 1.0correspond to
the Manhatan distance. A value of 2.0
corresponds to the Euclidean distance.
A value of 0.0 or less correspond to
--
--diversity-dst2dp arg (=auto) Set the type of function to convert
distance into penalty. 4 options are
available: auto, inverse, complement
and power. When auto is selected the
function is selected depending on the
--
--fs-diversity-pressure arg (=0) Multiplicative coefficient of the
diversity penalty (itself being in
[0,1]).
--fs-diversity-cap arg (=100) Place a cap on the maximum number of
feature set to consider. If zero, no
cap is used (Warning: could be very
slow). Use this to speed up diversity
computation on feature sets.
--fs-diversity-interaction arg (=-1) Maximum number of interactions to be
considered when computing the mutual
information between feature sets. This
is used in case the number of selected
features tends to be high compared to
--
--fs-diversity-jaccard arg (=1) Instead of using the expensive mutual
information between feature sets to
measure diversity, it uses a cheap
Jaccard index. In that case the feature
semantic is ignored
--fs-inc-redundant-intensity arg (=-1)
Incremental Selection parameter.
I can no longer tell. The style of writing sounds like something I'd write, but the names of the flags sound like code that Nil wrote. I think Nil gets the last word on this.
It's best if I take care of it since I wrote most of the diversity code.
As I'm going through the options I'm realizing that the --diversity-pressure option should be normalized. Currently a decent value depends on the amplitude of the score, which can range from around 1 to very large values (as shown in the diversity report) which is utterly inconvenient to the user. Instead it should be automatically scaled so that the user only need to set it from 0 to 1.
I'd be tempted to break backward compatibility since likely almost nobody has used it. Alternatively, I could add a new option like
--diversity-pressure-autoscale
and recommend that one instead of --diversity-pressure, which is pity because it has a longer name, but that way compatibility is preserved.
What do you guys think?
Or I can have a flag
--diversity-autoscale
which would rescale --diversity-pressure, so that the user can enter a value from 0 to 1 without having to worry about the scale of the fitness function.
It does mean the user will typically have to set that extra flag on top of setting the pressure, I don't like that too much, but maybe it's better than having 2 almost the same options, and again it preserves backward compatibility.
Also the diversity penalty reported in the behavioral score should be renamed uniformity penalty.
It wouldn't change any option names, diversity pressure, etc, would remain the same, but the term diversity penalty reported in the bscore is really dumb (and so is the person that came up with it).
i vote for one less flag over backward-compatibility, although maximum fine-tuning control is in keeping with the spirit of moses and it's 1000+ line help page ;)
i vote for one less flag over backward-compatibility
I'm not sure I understand, do you mean
one more flag over bc?
or
one less flag under bc?
Anyway, I thought that bc was more important, especially since moses is a legacy piece that is being superseded by asmoses, and so I went with implementing that flag.
We can always set this flag to true by default the day we crank up moses (or more likely asmoses) major version number.
Oh, sorry, I had misunderstood your sentence, you really mean one less flag. Since I've already implemented that flag I'm bending toward keeping it, but if we all agree that it should be removed, I'm OK with that too.
The man page can still be improved, but I suppose it's better left for another issue. I'm closing, feel free to reopen @mjsduncan if something has been left out.
thanks for the update, nil! i'd re-open if i could but i don't have the permissions to, since this issue title specifically refers to the man page and the man page is linked to in the opencog wiki, so your update isn't reflected in the documentation that has the most exposure.
Done as well
thanks, nil!