Giters

pplonski / my_ml_service

My Machine Learning Web Service

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Unable to Stop AB Testing

wergeld opened this issue · comments

commented

The tutorial states:

To stop the A/B test, please open address http://127.0.0.1:8000/api/v1/stop_ab_test/1 where 1 at the end of the address it the A/B test id. Click on POST button to finish A/B test.

When I go to this URL I get:

HTTP 405 Method Not Allowed
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "detail": "Method \"GET\" not allowed."
}

Which makes sense as we set this up as POST request endpoint. If I leave the other fields empty/default on that page that loads and click on the "POST" button I get this error on the page:

HTTP 400 Bad Request
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "status": "Error",
    "message": "float division by zero"
}

And this error in the console where I started the server:

D:\source2\repos\my_ml_service\venv\lib\site-packages\django\db\models\fields\__init__.py:1418: RuntimeWarning: DateTimeField MLRequest.created_at received a naive datetime (2020-08-16 12:18:52.567774) while time zone support is active.
  warnings.warn("DateTimeField %s received a naive datetime (%s)"
Bad Request: /api/v1/stop_ab_test/1
[16/Aug/2020 12:18:52] "POST /api/v1/stop_ab_test/1 HTTP/1.1" 400 10160

I am not sure what/where to make any updates to this.

commented

Have you sent any requests for testing? It looks like an error from:

Are you able to debug this? A/B testing might be a little tricky.

commented

Yes, I have sent requests via the ab_test.ipnyb. I have 100 results in the http://127.0.0.1:8000/api/v1/mlrequests output. Here is a snippet of the output from there:

    {
        "id": 90,
        "input_data": "{\"age\": \"32\", \"workclass\": \"Private\", \"fnlwgt\": \"127451\", \"education\": \"Some-college\", \"education-num\": \"10\", \"marital-status\": \"Married-civ-spouse\", \"occupation\": \"Prof-specialty\", \"relationship\": \"Husband\", \"race\": \"White\", \"sex\": \"Male\", \"capital-gain\": \"0\", \"capital-loss\": \"0\", \"hours-per-week\": \"38\", \"native-country\": \"United-States\"}",
        "full_response": "{'probability': 0.29, 'label': '<=50K', 'status': 'OK'}",
        "response": "<=50K",
        "feedback": ">50K",
        "created_at": "2020-08-16T15:52:27.610238Z",
        "parent_mlalgorithm": 1
    },
    {
        "id": 91,
        "input_data": "{\"age\": \"38\", \"workclass\": \"Private\", \"fnlwgt\": \"353263\", \"education\": \"Masters\", \"education-num\": \"14\", \"marital-status\": \"Never-married\", \"occupation\": \"Adm-clerical\", \"relationship\": \"Not-in-family\", \"race\": \"White\", \"sex\": \"Female\", \"capital-gain\": \"0\", \"capital-loss\": \"0\", \"hours-per-week\": \"50\", \"native-country\": \"Italy\"}",
        "full_response": "{'probability': 0.11, 'label': '<=50K', 'status': 'OK'}",
        "response": "<=50K",
        "feedback": ">50K",
        "created_at": "2020-08-16T15:52:27.654189Z",
        "parent_mlalgorithm": 2
    },

I had fixed the issue listed in #6 and I have 3 MLs in my registry. But, I setup the AB test to use the 2 different ones (RandomForest and ExtraTrees). When trying to re-request I also now get errors in the Notebook. After restarting the django server, I still get errors. All three of the ML registry items are still set to ""current_status": "ab_testing"". It seems like it is tying itself in knots.

commented

Have you stopped the A/B testing?

commented

As I said, attempting to stop the AB testing throws an error (see OP). I can stop the django server and restart it but that does not seem to make any difference as the testing is still stored some where. I have attempted to just take your git code and create an entirely new project and I still get the error that I cannot stop AB testing.

When I go to this URL I get:

HTTP 405 Method Not Allowed
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept

{
"detail": "Method "GET" not allowed."
}
Which makes sense as we set this up as POST request endpoint. If I leave the other fields empty/default on that page that loads and click on the "POST" button I get this error on the page:

HTTP 400 Bad Request
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept

{
"status": "Error",
"message": "float division by zero"
}
And this error in the console where I started the server:

D:\source2\repos\my_ml_service\venv\lib\site-packages\django\db\models\fields_init_.py:1418: RuntimeWarning: DateTimeField MLRequest.created_at received a naive datetime (2020-08-16 12:18:52.567774) while time zone support is active.
warnings.warn("DateTimeField %s received a naive datetime (%s)"
Bad Request: /api/v1/stop_ab_test/1
[16/Aug/2020 12:18:52] "POST /api/v1/stop_ab_test/1 HTTP/1.1" 400 10160

commented

You need to stop A/B testing. Please debug the code to see why it is not stopping. You can try to set some arbitrary values for accuracy to see if it is the place that causes the error. Stopping and restarting the server will not help, all status is saved in the database. You can try to start with a fresh database.

commented

I had this issue as well. The issue is that the way you filter for the requests in the class StopABTestView in views.py, it doesn't grab anything due to the upper limit specified by the date_now variable. date_now is created based on your computer's timezone, so for me being in eastern time, it was always less than the model created_at time. Here is the output from debugging:

date_now : 2020-10-27 16:27:09.636657
Model Created At: 2020-10-27 16:42:30.880037+00:00
Test Created At: 2020-10-27 20:08:59.615487+00:00

so the query below:
MLRequest.objects.filter(parent_mlalgorithm=ab_test.parent_mlalgorithm_1, created_at__gt = ab_test.created_at, created_at__lt = date_now).count()
returns nothing because created_at__lt (lt = less than) is earlier than the model created_at time.
This further leads to a division by zero error because the count is zero down below:
accuracy_1 = correct_responses_1 / float(all_responses_1)
all_responses_1 is zero.

The way to fix it is to use the same time zone as the settings.py file. Add the following imports in your views.py:

import pytz
import server.settings as application_settings

and inside the class StopABTestView make the following changes:

timezone = pytz.timezone(application_settings.TIME_ZONE)
date_now = datetime.datetime.now(tz=timezone)   #this will return on the operating system time zone if not specified

That should be it.

commented

Thank you @Ih8Coding4fun! Could you please create a PR with a fix? and maybe some tests?

commented

Absolutely! :)