HITs submitted without all tasks completed

[Ian]

I've had a number of HITs submitted now with exactly 60 tasks completed, instead of the full 61. (I've also had some successfully completed with all 61). They don't get auto-approved (which is good), but I'm not sure what's going on.

I don't think their time is running out, but I can't check that as I only have the time when they first completed a task recorded (it may make sense to record the time when they actually accept the HIT, but that's a side issue).

I also don't think this has to do with the specific tasks. They all end with different tasks and are missing different tasks. I'll see if I can email with some of them to follow up.

[Vanessasaurus]

could you please provide a worker id for which 60 tasks were completed and the result is submit?

[Vanessasaurus]

I think I found them - two in total for the battery with status "S" to indicate submit, please confirm:

[Vanessasaurus]

confirmed that the function to retrieve worker uncompleted experiments is working, returns one more:

[Vanessasaurus]

Getting close - it appears to be that the workers in question have completed 61 experiments, but only 60 unique exp_ids, suggesting completing an experiment more than once, or having an experiment with a duplicate exp_id, still looking into it.

[Vanessasaurus]

here it is:

[Vanessasaurus]

Have any of these workers piloted before, or are they new?

[Ian]

Each worker who has had this problem (there are 3) has completed at least
some tasks after we changed how we saved data (from trial-to-trial to at
the end). They each have only done one HIT for us. I'll send you their
worker IDs over email.

On Tue, Apr 26, 2016 at 11:28 AM, Vanessa Sochat [e-mail]
wrote:

Have any of these workers piloted before, or are they new?

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#133 (comment)

Best,
Ian

[Vanessasaurus]

This seems to be an issue with adding and removing experiments in a battery that have the same exp_id. I could see something like the following happening:

• worker starts battery, completes angling risk task
• the task is uninstalled / reinstalled, generating a new Experiment object with the same exp_id associated with the battery. The worker is not considered to have completed the equivalent experiment because technically it's a different Experiment object (just pointing to the same template)
• When results are checked to determine completion, we count the total length of completed experiments, and not the total length of completed (unique) experiments.

Thus, the fix seems to be easy - to count only the unique exp_ids. I implemented it with just the length to start as I originally had the idea that we could have multiple instances of the same experiment, but this is probably unlikely.

[Ian]

Hm, yeah. So the problem is if someone completes a task, and it is updated
while they are doing the battery, they do the task a second time, and that
is recorded in the total number of tasks they are completed?

If we ever move to install specifically for a battery, that shouldn't be a
problem because experiments shouldn't be updated during a HIT. Still, if
they were, we'd probably want the old ones to not count in some way.

Unique exp_ids seems good, but I do agree with your intuition - I suppose I
can imagine someone wanting to run one task many times (adaptive n back) to
train. Unlikely, but possible.

On Tue, Apr 26, 2016 at 11:34 AM, Vanessa Sochat [e-mail]
wrote:

This seems to be an issue with adding and removing experiments in a
battery that have the same exp_id. I could see something like the following
happening:

• worker starts battery, completes angling risk task
• the task is uninstalled / reinstalled, generating a new Experiment
  object with the same exp_id associated with the battery. The worker is not
  considered to have completed the equivalent experiment because technically
  it's a different Experiment object (just pointing to the same template)
• When results are checked to determine completion, we count the total
  length of completed experiments, and not the total length of completed
  (unique) experiments.

Thus, the fix seems to be easy - to count only the unique exp_ids. I
implemented it with just the length to start as I originally had the idea
that we could have multiple instances of the same experiment, but this is
probably unlikely.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#133 (comment)

Best,
Ian