AutoResponseView not working, returns same result for two fields.

Question

AutoResponseView not working, returns same result for two fields.

CleitonDeLima opened this issue 5 years ago · comments

Describe the bug
django-select2 version 7.0.4

I have two manytomany fields in a form, sometimes it happens that the 2nd field gets the same information from the 1st. When browsing the url through the browser, the results are the same.

Code Snippet
forms.py

widgets = {
    ...
    'vehicles': AddAnotherWidgetWrapper(  # first field
        VehicleSelect2MultipleWidget,
        reverse_lazy('vehicles:vehicle-create')
    ),
    'drivers': AddAnotherWidgetWrapper(  # second field
        DriverSelect2MultipleWidget,
        reverse_lazy('drivers:driver-create')
    ),
    ...
}

widgets.py

# First field
class VehicleSelect2MultipleWidget(ModelSelect2MultipleWidget):
    queryset = Vehicle.objects.filter(is_active=True).order_by('board')
    search_fields = [
        'board__unaccent__icontains',
        'label__unaccent__icontains'
    ]

# Second field
class DriverSelect2MultipleWidget(ModelSelect2MultipleWidget):
    queryset = Driver.objects.filter(is_active=True).order_by('name')
    search_fields = ['name__unaccent__icontains', 'cpf__contains']

settings.py

# config select2, js version 4.0.5
SELECT2_JS = 'vendors/select2/select2.min.js'
SELECT2_CSS = 'vendors/select2/select2.min.css'
SELECT2_I18N_PATH = 'vendors/select2/i18n'
SELECT2_CACHE_BACKEND = 'select2'

CACHES = {
    ...
    'select2': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/2',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        },
        'TIMEOUT': 60 * 60 * 24
    },
    ...
}

To Reproduce
I tried to simulate, it happened when I spent a lot of time on the form page, not sure if that's it, the reason I do not know ... To not happen this I logout and login again, the problem disappears.

Screenshots

first field

second field

Johannes Maron · Answer 1 · Thu Dec 26 2019 17:11:09 GMT+0800 (China Standard Time)

Hi @CleitonDeLima,
Thanks for reaching out. I must admit, this is the first time I hear of such behavior and if true, this must be considered a security risk. Which reminds me, I should really add a security guideline on how to disclose vulnerabilities.

Anyhow, let's dig in. Let's me start by saying thank you, for providing so many screenshots and details. That helps a lot. It seems you field_id is different for both requests. Still the response size is oddly similar for both responses.
Do you have any other full page caching in place? Any middlewares I should be aware of?

It would be also interesting if you could do:

from django.core import signing
from django_select2.cache import cache
from django_select2.conf import settings

field_id_1 = signing.loads('WHATEVER YOU SEE IN THE BROWSER')
cache_key_1 = '%s%s' % (settings.SELECT2_CACHE_PREFIX, field_id_1)

field_id_2 = signing.loads('WHATEVER YOU SEE IN THE BROWSER')
cache_key_2 = '%s%s' % (settings.SELECT2_CACHE_PREFIX, field_id_1)

print(cache_key_1, cache_key_2)

Merry Xmas,
Joe

Cleiton de Lima · Answer 2 · Fri Dec 27 2019 04:43:57 GMT+0800 (China Standard Time)

Merry Christmas @codingjoe!!

I did the tests again, follows the another example, the opposite of the other:

screenshots

field 1

field 2

json response field 1

json response field 2

python shell

from django.core import signing
from django_select2.cache import cache
from django_select2.conf import settings

field_id_1 = signing.loads('MTQwNTU1NTU3MTAxMjk2:1ikXF1:Dpj4oQMlDJ0xiAU1NsqeYJcqUps')
cache_key_1 = '%s%s' % (settings.SELECT2_CACHE_PREFIX, field_id_1)

field_id_2 = signing.loads('MTQwNTU1NTU3MTAxNDA4:1ikXF1:M-Z5wOhyg5wp-oy_cjiyYE7A0-Y')
cache_key_2 = '%s%s' % (settings.SELECT2_CACHE_PREFIX, field_id_2)

print(cache_key_1)
select2_140555557101296
print(cache_key_2)
select2_140555557101408

redis-cli

> select 2
OK

> keys *
   1) ":1:select2_140555532059480"
  ...
  94) ":1:select2_140555557101408"
  ...
 527) ":1:select2_140555557101296"
 ...
2001) ":1:select2_140555536472664"
2002) ":1:select2_140555535512016"

> MGET :1:select2_140555557101296
1) "\x80\x04\x95\xb8\t\x00\x00\x00\x00\x00\x00}\x94(\x8c\bqueryset\x94]\x94(\x8c\x16django.db.models.query\x94\x8c\bQuerySet\x94\x93\x94)\x81\x94}\x94(\x8c\x05model\x94\x8c\x14coral.drivers.models\x94\x8c\x06Driver\x94\x93\x94\x8c\x03_db\x94N\x8c\x06_hints\x94}\x94\x8c\x05query\x94\x8c\x1adjango.db.models.sql.query\x94\x8c\x05Query\x94\x93\x94)\x81\x94}\x94(h\bh\x0b\x8c\x0ealias_refcount\x94}\x94(\x8c\x0edrivers_driver\x94K\x02\x8c\x0bcore_tenant\x94K\x00u\x8c\talias_map\x94\x8c\x0bcollections\x94\x8c\x0bOrderedDict\x94\x93\x94)R\x94(h\x17\x8c#django.db.models.sql.datastructures\x94\x8c\tBaseTable\x94\x93\x94)\x81\x94}\x94(\x8c\ntable_name\x94h\x17\x8c\x0btable_alias\x94h\x17ubh\x18h\x1e\x8c\x04Join\x94\x93\x94)\x81\x94}\x94(h#h\x18\x8c\x0cparent_alias\x94h\x17h$h\x18\x8c\tjoin_type\x94\x8c\nINNER JOIN\x94\x8c\tjoin_cols\x94\x8c\ttenant_id\x94\x8c\x02id\x94\x86\x94\x85\x94\x8c\njoin_field\x94\x8c\x17django.db.models.fields\x94\x8c\x0b_load_field\x94\x93\x94\x8c\adrivers\x94h\n\x8c\x06tenant\x94\x87\x94R\x94\x8c\bnullable\x94\x89\x8c\x11filtered_relation\x94Nubu\x8c\x10external_aliases\x94\x8f\x94\x8c\ttable_map\x94}\x94(h\x17]\x94h\x17ah\x18]\x94h\x18au\x8c\x0cdefault_cols\x94\x88\x8c\x10default_ordering\x94\x88\x8c\x11standard_ordering\x94\x88\x8c\x0cused_aliases\x94\x8f\x94\x8c\x10filter_is_sticky\x94\x89\x8c\bsubquery\x94\x89\x8c\x06select\x94)\x8c\x05where\x94\x8c\x1adjango.db.models.sql.where\x94\x8c\tWhereNode\x94\x93\x94)\x81\x94}\x94(\x8c\bchildren\x94]\x94(\x8c\x1bcoral.multi_tenant.managers\x94\x8c\rCurrentTenant\x94\x93\x94)\x81\x94}\x94(\x8c\x03lhs\x94\x8c\x1cdjango.db.models.expressions\x94\x8c\x03Col\x94\x93\x94)\x81\x94}\x94(\x8c\x11_constructor_args\x94h\x17h8h4\x8c\x04core\x94\x8c\x06Tenant\x94h.\x87\x94R\x94\x87\x94}\x94\x86\x94\x8c\x0coutput_field\x94h`\x8c\x05alias\x94h\x17\x8c\x06target\x94h8\x8c\x12contains_aggregate\x94\x89ub\x8c\x03rhs\x94K\x00\x8c\x14bilateral_transforms\x94]\x94hg\x89ub\x8c\x18django.db.models.lookups\x94\x8c\x05Exact\x94\x93\x94)\x81\x94}\x94(hVhY)\x81\x94}\x94(h\\h\x17h4h5h\n\x8c\tis_active\x94\x87\x94R\x94\x86\x94}\x94\x86\x94hdhtheh\x17hfhthg\x89ubhh\x88hi]\x94hg\x89ubhJ\x8c\x0bNothingNode\x94\x93\x94)\x81\x94e\x8c\tconnector\x94\x8c\x03AND\x94\x8c\anegated\x94\x89hg\x89ub\x8c\x0bwhere_class\x94hL\x8c\bgroup_by\x94N\x8c\border_by\x94\x8c\x04name\x94\x85\x94\x8c\blow_mark\x94K\x00\x8c\thigh_mark\x94N\x8c\bdistinct\x94\x89\x8c\x0fdistinct_fields\x94)\x8c\x11select_for_update\x94\x89\x8c\x18select_for_update_nowait\x94\x89\x8c\x1dselect_for_update_skip_locked\x94\x89\x8c\x14select_for_update_of\x94)\x8c\x0eselect_related\x94\x89\x8c\tmax_depth\x94K\x05\x8c\rvalues_select\x94)\x8c\x0c_annotations\x94h\x1c)R\x94\x8c\x16annotation_select_mask\x94N\x8c\x18_annotation_select_cache\x94N\x8c\ncombinator\x94N\x8c\x0ecombinator_all\x94\x89\x8c\x10combined_queries\x94)\x8c\x06_extra\x94N\x8c\x11extra_select_mask\x94N\x8c\x13_extra_select_cache\x94N\x8c\x0cextra_tables\x94)\x8c\x0eextra_order_by\x94)\x8c\x10deferred_loading\x94(\x91\x94\x88\x86\x94\x8c\x13_filtered_relations\x94}\x94\x8c\rexplain_query\x94\x89\x8c\x0eexplain_format\x94N\x8c\x0fexplain_options\x94}\x94\x8c\r_lookup_joins\x94]\x94h\x17a\x8c\nbase_table\x94h\x17ub\x8c\r_result_cache\x94]\x94\x8c\x0e_sticky_filter\x94\x89\x8c\n_for_write\x94\x89\x8c\x19_prefetch_related_lookups\x94)\x8c\x0e_prefetch_done\x94\x89\x8c\x16_known_related_objects\x94}\x94\x8c\x0f_iterable_class\x94h\x03\x8c\rModelIterable\x94\x93\x94\x8c\a_fields\x94N\x8c\x0f_django_version\x94\x8c\x052.2.8\x94ubh\x12)\x81\x94}\x94(h\bh\x0bh\x15}\x94(h\x17K\x02h\x18K\x00uh\x19h\x1c)R\x94(h\x17h!h\x18h'uh;\x8f\x94h=}\x94(h\x17h?h\x18h@uhA\x88hB\x88hC\x88hD\x8f\x94hF\x89hG\x89hH)hIhL)\x81\x94}\x94(hO]\x94(hThneh|h}h~\x89ubh\x7fhLh\x80Nh\x81h\x83h\x84K\x00h\x85Nh\x86\x89h\x87)h\x88\x89h\x89\x89h\x8a\x89h\x8b)h\x8c\x89h\x8dK\x05h\x8e)h\x8fNh\x91Nh\x92Nh\x93Nh\x94\x89h\x95)h\x96Nh\x97Nh\x98Nh\x99)h\x9a)h\x9bh\x9dh\x9e}\x94h\xa0\x89h\xa1Nh\xa2h\xa3h\xa4h\xa5ube\x8c\x03cls\x94\x8c\x15coral.drivers.widgets\x94\x8c\x1bDriverSelect2MultipleWidget\x94\x93\x94\x8c\rsearch_fields\x94\x8c\x19name__unaccent__icontains\x94\x8c\rcpf__contains\x94\x86\x94\x8c\x0bmax_results\x94K\x19\x8c\x03url\x94\x8c\x19/select2/fields/auto.json\x94\x8c\x10dependent_fields\x94}\x94u."

> MGET :1:select2_140555557101408
1) "\x80\x04\x95\xb8\t\x00\x00\x00\x00\x00\x00}\x94(\x8c\bqueryset\x94]\x94(\x8c\x16django.db.models.query\x94\x8c\bQuerySet\x94\x93\x94)\x81\x94}\x94(\x8c\x05model\x94\x8c\x14coral.drivers.models\x94\x8c\x06Driver\x94\x93\x94\x8c\x03_db\x94N\x8c\x06_hints\x94}\x94\x8c\x05query\x94\x8c\x1adjango.db.models.sql.query\x94\x8c\x05Query\x94\x93\x94)\x81\x94}\x94(h\bh\x0b\x8c\x0ealias_refcount\x94}\x94(\x8c\x0edrivers_driver\x94K\x02\x8c\x0bcore_tenant\x94K\x00u\x8c\talias_map\x94\x8c\x0bcollections\x94\x8c\x0bOrderedDict\x94\x93\x94)R\x94(h\x17\x8c#django.db.models.sql.datastructures\x94\x8c\tBaseTable\x94\x93\x94)\x81\x94}\x94(\x8c\ntable_name\x94h\x17\x8c\x0btable_alias\x94h\x17ubh\x18h\x1e\x8c\x04Join\x94\x93\x94)\x81\x94}\x94(h#h\x18\x8c\x0cparent_alias\x94h\x17h$h\x18\x8c\tjoin_type\x94\x8c\nINNER JOIN\x94\x8c\tjoin_cols\x94\x8c\ttenant_id\x94\x8c\x02id\x94\x86\x94\x85\x94\x8c\njoin_field\x94\x8c\x17django.db.models.fields\x94\x8c\x0b_load_field\x94\x93\x94\x8c\adrivers\x94h\n\x8c\x06tenant\x94\x87\x94R\x94\x8c\bnullable\x94\x89\x8c\x11filtered_relation\x94Nubu\x8c\x10external_aliases\x94\x8f\x94\x8c\ttable_map\x94}\x94(h\x17]\x94h\x17ah\x18]\x94h\x18au\x8c\x0cdefault_cols\x94\x88\x8c\x10default_ordering\x94\x88\x8c\x11standard_ordering\x94\x88\x8c\x0cused_aliases\x94\x8f\x94\x8c\x10filter_is_sticky\x94\x89\x8c\bsubquery\x94\x89\x8c\x06select\x94)\x8c\x05where\x94\x8c\x1adjango.db.models.sql.where\x94\x8c\tWhereNode\x94\x93\x94)\x81\x94}\x94(\x8c\bchildren\x94]\x94(\x8c\x1bcoral.multi_tenant.managers\x94\x8c\rCurrentTenant\x94\x93\x94)\x81\x94}\x94(\x8c\x03lhs\x94\x8c\x1cdjango.db.models.expressions\x94\x8c\x03Col\x94\x93\x94)\x81\x94}\x94(\x8c\x11_constructor_args\x94h\x17h8h4\x8c\x04core\x94\x8c\x06Tenant\x94h.\x87\x94R\x94\x87\x94}\x94\x86\x94\x8c\x0coutput_field\x94h`\x8c\x05alias\x94h\x17\x8c\x06target\x94h8\x8c\x12contains_aggregate\x94\x89ub\x8c\x03rhs\x94K\x00\x8c\x14bilateral_transforms\x94]\x94hg\x89ub\x8c\x18django.db.models.lookups\x94\x8c\x05Exact\x94\x93\x94)\x81\x94}\x94(hVhY)\x81\x94}\x94(h\\h\x17h4h5h\n\x8c\tis_active\x94\x87\x94R\x94\x86\x94}\x94\x86\x94hdhtheh\x17hfhthg\x89ubhh\x88hi]\x94hg\x89ubhJ\x8c\x0bNothingNode\x94\x93\x94)\x81\x94e\x8c\tconnector\x94\x8c\x03AND\x94\x8c\anegated\x94\x89hg\x89ub\x8c\x0bwhere_class\x94hL\x8c\bgroup_by\x94N\x8c\border_by\x94\x8c\x04name\x94\x85\x94\x8c\blow_mark\x94K\x00\x8c\thigh_mark\x94N\x8c\bdistinct\x94\x89\x8c\x0fdistinct_fields\x94)\x8c\x11select_for_update\x94\x89\x8c\x18select_for_update_nowait\x94\x89\x8c\x1dselect_for_update_skip_locked\x94\x89\x8c\x14select_for_update_of\x94)\x8c\x0eselect_related\x94\x89\x8c\tmax_depth\x94K\x05\x8c\rvalues_select\x94)\x8c\x0c_annotations\x94h\x1c)R\x94\x8c\x16annotation_select_mask\x94N\x8c\x18_annotation_select_cache\x94N\x8c\ncombinator\x94N\x8c\x0ecombinator_all\x94\x89\x8c\x10combined_queries\x94)\x8c\x06_extra\x94N\x8c\x11extra_select_mask\x94N\x8c\x13_extra_select_cache\x94N\x8c\x0cextra_tables\x94)\x8c\x0eextra_order_by\x94)\x8c\x10deferred_loading\x94(\x91\x94\x88\x86\x94\x8c\x13_filtered_relations\x94}\x94\x8c\rexplain_query\x94\x89\x8c\x0eexplain_format\x94N\x8c\x0fexplain_options\x94}\x94\x8c\r_lookup_joins\x94]\x94h\x17a\x8c\nbase_table\x94h\x17ub\x8c\r_result_cache\x94]\x94\x8c\x0e_sticky_filter\x94\x89\x8c\n_for_write\x94\x89\x8c\x19_prefetch_related_lookups\x94)\x8c\x0e_prefetch_done\x94\x89\x8c\x16_known_related_objects\x94}\x94\x8c\x0f_iterable_class\x94h\x03\x8c\rModelIterable\x94\x93\x94\x8c\a_fields\x94N\x8c\x0f_django_version\x94\x8c\x052.2.8\x94ubh\x12)\x81\x94}\x94(h\bh\x0bh\x15}\x94(h\x17K\x02h\x18K\x00uh\x19h\x1c)R\x94(h\x17h!h\x18h'uh;\x8f\x94h=}\x94(h\x17h?h\x18h@uhA\x88hB\x88hC\x88hD\x8f\x94hF\x89hG\x89hH)hIhL)\x81\x94}\x94(hO]\x94(hThneh|h}h~\x89ubh\x7fhLh\x80Nh\x81h\x83h\x84K\x00h\x85Nh\x86\x89h\x87)h\x88\x89h\x89\x89h\x8a\x89h\x8b)h\x8c\x89h\x8dK\x05h\x8e)h\x8fNh\x91Nh\x92Nh\x93Nh\x94\x89h\x95)h\x96Nh\x97Nh\x98Nh\x99)h\x9a)h\x9bh\x9dh\x9e}\x94h\xa0\x89h\xa1Nh\xa2h\xa3h\xa4h\xa5ube\x8c\x03cls\x94\x8c\x15coral.drivers.widgets\x94\x8c\x1bDriverSelect2MultipleWidget\x94\x93\x94\x8c\rsearch_fields\x94\x8c\x19name__unaccent__icontains\x94\x8c\rcpf__contains\x94\x86\x94\x8c\x0bmax_results\x94K\x19\x8c\x03url\x94\x8c\x19/select2/fields/auto.json\x94\x8c\x10dependent_fields\x94}\x94u."

Both keys have the same content in the cache.

Do you have any other full page caching in place?

No.

Any middlewares I should be aware of?

One middleware uses cache, I believe it is not related to the problem:

class UserRestrict(MiddlewareMixin):
    def process_request(self, request):
        if request.user.is_authenticated:
            cache_timeout = settings.SESSION_COOKIE_AGE
            cache_key = 'user_pk_%s_restrict_key' % request.user.pk
            cache_value = cache.get(cache_key)

            if cache_value is not None and not request.user.is_superuser:
                if request.session.session_key != cache_value:
                    engine = import_module(settings.SESSION_ENGINE)
                    session = engine.SessionStore(session_key=cache_value)
                    session.delete()
                    cache.set(cache_key, request.session.session_key, cache_timeout)
            else:
                cache.set(cache_key, request.session.session_key, cache_timeout)

Some observations:

Redis server v=3.0.6 sha=00000000:0 malloc=jemalloc-3.6.0 bits=64 build=7785291a3d2152db
uname -a Linux coral 4.4.0-165-generic #193-Ubuntu SMP Tue Sep 17 17:42:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
I can only reproduce this in production...
I do not know the reason :/

Thanks for the attention to the problem, it really is a problem that occurs a few times, I can not reproduce locally and I do not know why.

Johannes Maron · Answer 3 · Tue Jan 14 2020 00:39:04 GMT+0800 (China Standard Time)

Thanks for the detailed information. It helps a lot, as in I have no idea why this happens. I'll need to take a good look at the code to see how there the same value can be set for two different keys. However, it would be great to know if you have a multi server/process setup. In other words, do you have multiple application servers running and do you use gunicorn?

Cleiton de Lima · Answer 4 · Tue Jan 14 2020 01:10:39 GMT+0800 (China Standard Time)

Thanks for answering,

...do you have multiple application servers running and do you use gunicorn?

Yes, i have two applications on the same server using gunicorn.
Only one application uses django-select2.

I suspect the problem is on my server, I still want to isolate my applications in containers...

Johannes Maron · Answer 5 · Tue Jan 14 2020 20:33:13 GMT+0800 (China Standard Time)

Ok, so gunicorn uses a process pool to crunch requests, not threads like waitress or an event loop. To generate the cache key, I use the object id, which is unique for each object within a process, but not across multiple processes or machines. I honestly have never run into this issue myself, but theoretically it's possible that cache keys could collide. We maybe need to switch to a random cache key. You could overwrite that bit locally and see if that resolves your issue.

Cleiton de Lima · Answer 6 · Tue Jan 14 2020 20:46:00 GMT+0800 (China Standard Time)

Got it, it makes sense.
Generating the key with uuid4 would work?

Johannes Maron · Answer 7 · Wed Jan 15 2020 18:18:45 GMT+0800 (China Standard Time)

Yes, uuid4 is random enough. We don't need a secure random ID only one that doesn't clash.