enable blacklist calculation
vsoch opened this issue · comments
We really only need one thing here:
(1) If an experiment fails to get credit, prevent that worker from continuing (present them with a "sorry" page instead of new experiments - don't reject from the HIT).
At the end of the HIT, the worker will have incomplete experiments (as we prevented them from completing more experiments). At this point, treat them exactly as if they had just stopped completing experiments of their own volition. Their HIT should not be automatically approved, and they should instead be payed using Mturk's bonus mechanism according to the amount of time they have worked (this can be discussed later). For now, just blacklisting and not-approving is enough. The bonus payment can be handled manually.
One addition that we'll need is the ability to blacklist them when X number of HITs fails to get credit. For example, we might be okay with one or two screwed up tasks, but at 3 we want to block them. We had talked a briefly before about a worker-battery specific data object (to record things like how many times the worker repeats tasks), but is seems like a "blacklist" function called at the submission of each experiment can perform the necessary function without such an object.
the credit calculation happens at the end of the battery...
I'm aware, that's why I'm bringing this up as a reminder. We've talked about this need to blacklist people before - this was a necessary component of doing a large battery of tasks online. If we suspect someone to not be "trying in earnest" we need a way to stop them from completing more tasks, as we will pay them for whatever they have completed.
Blacklisting would require checking the data as it comes in, which isn't feasible to to during the running of the experiment because 1) parsing over all data between experiments is not suitable for runtime, it needs a separate celery task, 2) such a task gets added to a queue not in sync with the experiment running in the browser, and 3) the experiment does not return to the server at the end other than the sync function for quick sending of data, the rendering of the "next" or "redo" buttons happens before the user starts.
the only thing I can think that might work is to fire off a job at the end of each experiment to check it, and given some criteria are met, the user is flagged as blacklisted, and then this blacklist variable is checked at the onset of starting whatever the next experiment is.
and then the blacklisted variable would likely need to be json and store an entry for multiple battery ids, as a user blacklisted for one battery should still be able to do another.
we might also make a separate blacklist object to store a user and battery variable.
the only thing I can think that might work is to fire off a job at the end of each experiment to check it, and given some criteria are met, the user is flagged as blacklisted, and then this blacklist variable is checked at the onset of starting whatever the next experiment is.
and then the blacklisted variable would likely need to be json and store an entry for multiple battery ids, as a user blacklisted for one battery should still be able to do another.
That sounds good. The flag on blacklist must be a function of failed checks and some threshold conditional specified (by the user at the level of the battery).
A possible extension - instead of a blacklist variable, we may want to think of this as a worker object. Right now the worker object tracks mturk relevant features, right? We have our own worker/battery specific variables that we might want to track outside of experiments.
As some examples:
battery-specific
(1) number of credit check fails
(2) blacklist status
(3) number of repeated experiments
worker-specific
(1) which batteries are completed
That last point may be helpful if you only want workers who have completed another study of yours. But regardless of that last point, your blacklist object sounds reasonable.
That sounds good. The flag on blacklist must be a function of failed checks and some threshold conditional specified (by the user at the level of the battery).
Yes, it makes sense to add these as arguments to the battery, and as for the specifics I'll think about it.
A possible extension - instead of a blacklist variable, we may want to think of this as a worker object. Right now the worker object tracks mturk relevant features, right? We have our own worker/battery specific variables that we might want to track outside of experiments.
The blacklist does not belong with the worker object. By way of needing to store multiple parameters for worker and battery combinations, and by way of having multiple potential combinations, we need a separate Blacklist (or some other name) object. The worker object is not just for MTurk - it also keeps track of local battery participants. In a recent addition I also added number of sessions, and number of visits.
As some examples:
battery-specific
(1) number of credit check fails
(2) blacklist status
(3) number of repeated experiments
This is why the Blacklist (or other) object would have unique id as worker and battery id.
worker-specific
(1) which batteries are completed
This would be redundant information, as this can be obtained via the Results objects.
That last point may be helpful if you only want workers who have completed another study of yours. But regardless of that last point, your blacklist object sounds reasonable.