SinkPIT
xixihahaggg opened this issue · comments
Hi all,
I'm here for asking does anyone has experiment experience using SinkPIT? It was expected to have better performance than PIT but what I observed from my experiment is the contrary.
I really appreciate any insight of this.
Best
Thanks for the question.
I think @tachi-hi can answer your questions, as he integrated SinkPIT loss in Asteroid.
Since it is not clear enough what specific problems you are encountering, I can only answer in general terms, but basically SinkPIT is only an approximation of PIT, so if the exact PIT can be easily evaluated, then the exact PIT will probably give better results.
The following are situations in which SinkPIT are expected to be effective.
- First, SinkPIT has the advantage of providing a soft way to evaluate a weighted sum of permutations when one cannot expect to obtain the correct permutation using the exact PIT. For example, in the early stages of training.
- Another advantage of SinkPIT is that it can compute an approximate permutation even when the exact PIT is intractable, as in the case there are a large number of sources.
However, the evaluation of exact PIT was done by O(N!) brute force search in the original implementation of asteroid, but it has now been replaced by the more efficient O(N^3) Hungarian algorithm. So this advantage may be limited, compared to when the SinkPIT paper was published.
I hope this will be helpful.
Thank you for your answer! I think right now it's more clear to me.
I still have a quick question regarding the first situation: do you mean if the optimal permutation is hard to learn, SinkPIT uses a weighted sum of all permutations could relieve this problem?
Although we cannot say for sure on this point since I do not have so many experimental results, I would like to make a few comments.
In the original PIT, each element of the
If there are multiple candidate permutations, SinkPIT is likely to return an intermediate "soft permutation" (doubly stochastic matrix) as the solution, rather than settling on one of the "hard permutations". This seems to be similar to the behavior of probability weighting methods such as ProbPIT. (In my small experiments, ProbPIT often performs a little better than SinkPIT and PIT. However, the rigorous evaluation of ProbPIT is basically not scalable to the number of sources, so it may be a problem when dealing with problems with a large number of sources.)
Although we cannot say for sure on this point since I do not have so many experimental results, I would like to make a few comments.
In the original PIT, each element of the N×N permutation matrix was allowed to have only binary values (0 or 1), and the problem was to find the best one among the vertices of a polytope (called Birkhoff polytope) in N×N-dim space. SinkPIT, on the other hand, allows the entire interior of the polytope (the entire set of doubly stochastic matrices) to be the solution space. In other words, SinkPIT is a convex relaxation problem of the original PIT. (Generally speaking, by and large, convex optimization problems are often easier to solve than non-convex problems, and from this perspective, SinkPIT is expected to behave well. However, this will of course depend on other conditions of the problem, and it may not be generalizable.)
If there are multiple candidate permutations, SinkPIT is likely to return an intermediate "soft permutation" (doubly stochastic matrix) as the solution, rather than settling on one of the "hard permutations". This seems to be similar to the behavior of probability weighting methods such as ProbPIT. (In my small experiments, ProbPIT often performs a little better than SinkPIT and PIT. However, the rigorous evaluation of ProbPIT is basically not scalable to the number of sources, so it may be a problem when dealing with problems with a large number of sources.)
Thank you for this answer. This sounds reasonable, I'd like to perform some ablation experiments on this.
Again, thank you for your help.