IntegrityError at /api/v2/search when searching remote handles
Ultimator14 opened this issue · comments
I'm using Tusky as client app for my own takahe server. In the takahe configuration I enabled ERROR_EMAILS.
Whenever I search for a (remote) handle the first time e.g. @takahe@jointakahe.org using the Tusky search, I get two mails informing me about a server error.
Internal Server Error: /api/v2/search
IntegrityError at /api/v2/search
doppelter Schlüsselwert verletzt Unique-Constraint »users_identity_actor_uri_key«
DETAIL: Schlüssel »(actor_uri)=(https://jointakahe.takahe.social/@takahe@jointakahe.org/)« existiert bereits.
(full message see below)
This happens independently of the remote server software. I noticed this for mastodon, takahe, kbin and also some others. This only happens if the user is unknown to the server, so the very first time I do the search. Afterwards, the error does no longer occur but only the pinned posts are shown in the Tusky app and follows are also not displayed correctly (see screenshot).
Log in error mail (The log leaks some usernames and passwords, I redacted them manually):
Internal Server Error: /api/v2/search
IntegrityError at /api/v2/search
doppelter Schlüsselwert verletzt Unique-Constraint »users_identity_actor_uri_key«
DETAIL: Schlüssel »(actor_uri)=(https://jointakahe.takahe.social/@takahe@jointakahe.org/)« existiert bereits.
Request Method: GET
Request URL: https://social.pygos.space/api/v2/search?q=%40takahe%40jointakahe.org&type=statuses&resolve=true&limit=20&offset=0&following=false
Django Version: 4.2.4
Python Executable: /usr/local/bin/python3
Python Version: 3.11.5
Python Path: ['/takahe', '/usr/local/bin', '/usr/local/lib/python311.zip', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/site-packages']
Server time: Sun, 03 Sep 2023 20:34:01 +0000
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.postgres',
'corsheaders',
'django_htmx',
'hatchway',
'core',
'activities',
'api',
'mediaproxy',
'stator',
'users']
Installed Middleware:
['core.middleware.SentryTaggingMiddleware',
'django.middleware.security.SecurityMiddleware',
'corsheaders.middleware.CorsMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django_htmx.middleware.HtmxMiddleware',
'core.middleware.HeadersMiddleware',
'core.middleware.ConfigLoadingMiddleware',
'api.middleware.ApiTokenMiddleware',
'users.middleware.DomainMiddleware']
Traceback (most recent call last):
File "/takahe/activities/services/search.py", line 37, in search_identities_handle
raise Identity.DoesNotExist()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
During handling of the above exception (), another exception occurred:
File "/takahe/users/models/identity.py", line 405, in by_username_and_domain
return cls.objects.get(
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
raise self.model.DoesNotExist(
^
During handling of the above exception (Identity matching query does not exist.), another exception occurred:
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 737, in execute
raise ex.with_traceback(None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The above exception (doppelter Schlüsselwert verletzt Unique-Constraint »users_identity_actor_uri_key«
DETAIL: Schlüssel »(actor_uri)=(https://jointakahe.takahe.social/@takahe@jointakahe.org/)« existiert bereits.) was the direct cause of the following exception:
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/api/decorators.py", line 46, in inner
return function(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/hatchway/view.py", line 302, in __call__
response = self.view(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/api/views/search.py", line 35, in search
search_result = searcher.search_all()
^^^^^^^^^^^^^^^^^^^^^
File "/takahe/activities/services/search.py", line 138, in search_all
"identities": self.search_identities_handle(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/activities/services/search.py", line 47, in search_identities_handle
identity = Identity.by_username_and_domain(
File "/takahe/users/models/identity.py", line 423, in by_username_and_domain
return cls.objects.create(
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 658, in create
obj.save(force_insert=True, using=self.db)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
self.save_base(
^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
updated = self._save_table(
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
results = self._do_insert(
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
return manager._insert(
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 737, in execute
raise ex.with_traceback(None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception Type: IntegrityError at /api/v2/search
Exception Value: doppelter Schlüsselwert verletzt Unique-Constraint »users_identity_actor_uri_key«
DETAIL: Schlüssel »(actor_uri)=(https://jointakahe.takahe.social/@takahe@jointakahe.org/)« existiert bereits.
Raised during: hatchway.view.inner
Request information:
USER: [jb@pygos.space](mailto:jb@pygos.space)
GET:
q = '@takahe@jointakahe.org'
type = 'statuses'
resolve = 'true'
limit = '20'
offset = '0'
following = 'false'
POST: No POST data
FILES: No FILES data
COOKIES: No cookie data
META:
HTTP_ACCEPT_ENCODING = 'gzip'
HTTP_AUTHORIZATION = 'Bearer <REDACTED>'
HTTP_CONNECTION = 'close'
HTTP_FORWARDED = 'for=93.205.182.221;proto=https;host="social.pygos.space"'
HTTP_HOST = 'social.pygos.space'
HTTP_USER_AGENT = 'Tusky/23.0 Android/13 OkHttp/4.11.0'
HTTP_X_FORWARDED_FOR = '93.205.182.221, 192.168.157.12'
HTTP_X_FORWARDED_HOST = 'social.pygos.space'
HTTP_X_FORWARDED_PROTO = 'https'
HTTP_X_HOST = 'social.pygos.space'
PATH_INFO = '/api/v2/search'
QUERY_STRING = 'q=%40takahe%40jointakahe.org&type=statuses&resolve=true&limit=20&offset=0&following=false'
RAW_URI = '/api/v2/search?q=%40takahe%40jointakahe.org&type=statuses&resolve=true&limit=20&offset=0&following=false'
REMOTE_ADDR = '127.0.0.1'
REMOTE_PORT = '52502'
REQUEST_METHOD = 'GET'
SCRIPT_NAME = ''
SERVER_NAME = '0.0.0.0'
SERVER_PORT = '8001'
SERVER_PROTOCOL = 'HTTP/1.1'
SERVER_SOFTWARE = 'gunicorn/20.1.0'
gunicorn.socket = <socket.socket fd=9, family=2, type=1, proto=0, laddr=('127.0.0.1', 8001), raddr=('127.0.0.1', 52502)>
wsgi.errors = <gunicorn.http.wsgi.WSGIErrorsWrapper object at 0x7f7b18e590>
wsgi.file_wrapper = <class 'gunicorn.http.wsgi.FileWrapper'>
wsgi.input = <gunicorn.http.body.Body object at 0x7f7b1d3190>
wsgi.input_terminated = True
wsgi.multiprocess = True
wsgi.multithread = False
wsgi.run_once = False
wsgi.url_scheme = 'https'
wsgi.version = '(1, 0)'
Settings:
Using settings module takahe.settings
ABSOLUTE_URL_OVERRIDES = {}
ADMINS = [('Admin', '[<REDACTED>@pygos.space](mailto:<REDACTED>@pygos.space)')]
ALLOWED_HOSTS = ['*']
APPEND_SLASH = True
AUTHENTICATION_BACKENDS = ['django.contrib.auth.backends.ModelBackend']
AUTH_PASSWORD_VALIDATORS = '********************'
AUTH_USER_MODEL = 'users.User'
AUTO_ADMIN_EMAIL = None
BASE_DIR = PosixPath('/takahe')
CACHES = {'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache', 'LOCATION': ''}}
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_KEY_PREFIX = '********************'
CACHE_MIDDLEWARE_SECONDS = 600
CORS_ALLOW_CREDENTIALS = True
CORS_ALLOW_HEADERS = "('accept', 'accept-encoding', 'authorization', 'content-type', 'dnt', 'origin', 'user-agent', 'x-csrftoken', 'x-requested-with', 'Idempotency-Key')"
CORS_EXPOSE_HEADERS = "('link',)"
CORS_ORIGIN_ALLOW_ALL = True
CORS_ORIGIN_WHITELIST = []
CORS_PREFLIGHT_MAX_AGE = 604800
CSRF_COOKIE_AGE = 31449600
CSRF_COOKIE_DOMAIN = None
CSRF_COOKIE_HTTPONLY = False
CSRF_COOKIE_MASKED = False
CSRF_COOKIE_NAME = 'csrftoken'
CSRF_COOKIE_PATH = '/'
CSRF_COOKIE_SAMESITE = 'Lax'
CSRF_COOKIE_SECURE = False
CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS = ['https://social.pygos.space/']
CSRF_USE_SESSIONS = False
DATABASES = {'default': {'NAME': 'takahe', 'USER': '<REDACTED>', 'PASSWORD': '********************', 'HOST': 'localhost', 'PORT': '', 'CONN_MAX_AGE': 600, 'OPTIONS': {'sslmode': 'disable'}, 'ENGINE': 'django.db.backends.postgresql', 'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_HEALTH_CHECKS': False, 'TIME_ZONE': None, 'TEST': {'CHARSET': None, 'COLLATION': None, 'MIGRATE': True, 'MIRROR': None, 'NAME': None}}}
DATABASE_ROUTERS = []
DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440
DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000
DATA_UPLOAD_MAX_NUMBER_FILES = 100
DATETIME_FORMAT = 'N j, Y, P'
DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M']
DATE_FORMAT = 'N j, Y'
DATE_INPUT_FORMATS = ['%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y']
DEBUG = False
DEBUG_PROPAGATE_EXCEPTIONS = False
DECIMAL_SEPARATOR = '.'
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
DEFAULT_CHARSET = 'utf-8'
DEFAULT_EXCEPTION_REPORTER = 'django.views.debug.ExceptionReporter'
DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'
DEFAULT_FROM_EMAIL = 'webmaster@localhost'
DEFAULT_INDEX_TABLESPACE = ''
DEFAULT_TABLESPACE = ''
DISALLOWED_USER_AGENTS = []
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'pygos.space'
EMAIL_HOST_PASSWORD = '********************'
EMAIL_HOST_USER = '<REDACTED>'
EMAIL_PORT = 587
EMAIL_SSL_CERTFILE = None
EMAIL_SSL_KEYFILE = '********************'
EMAIL_SUBJECT_PREFIX = '[Django] '
EMAIL_TIMEOUT = None
EMAIL_USE_LOCALTIME = False
EMAIL_USE_SSL = False
EMAIL_USE_TLS = True
FILE_UPLOAD_DIRECTORY_PERMISSIONS = None
FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440
FILE_UPLOAD_PERMISSIONS = 420
FILE_UPLOAD_TEMP_DIR = None
FIRST_DAY_OF_WEEK = 0
FIXTURE_DIRS = []
FORCE_SCRIPT_NAME = None
FORMAT_MODULE_PATH = None
FORM_RENDERER = 'django.forms.renderers.DjangoTemplates'
IGNORABLE_404_URLS = []
INSTALLED_APPS = ['django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.postgres', 'corsheaders', 'django_htmx', 'hatchway', 'core', 'activities', 'api', 'mediaproxy', 'stator', 'users']
INTERNAL_IPS = []
JSONLD_MAX_SIZE = 51200
LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ar-dz', 'Algerian Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('ckb', 'Central Kurdish (Sorani)'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('hy', 'Armenian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('ig', 'Igbo'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kab', 'Kabyle'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('ky', 'Kyrgyz'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('ms', 'Malay'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('tg', 'Tajik'), ('th', 'Thai'), ('tk', 'Turkmen'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('uz', 'Uzbek'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')]
LANGUAGES_BIDI = ['he', 'ar', 'ar-dz', 'ckb', 'fa', 'ur']
LANGUAGE_CODE = 'en-us'
LANGUAGE_COOKIE_AGE = None
LANGUAGE_COOKIE_DOMAIN = None
LANGUAGE_COOKIE_HTTPONLY = False
LANGUAGE_COOKIE_NAME = 'django_language'
LANGUAGE_COOKIE_PATH = '/'
LANGUAGE_COOKIE_SAMESITE = None
LANGUAGE_COOKIE_SECURE = False
LOCALE_PATHS = []
LOGGING = {}
LOGGING_CONFIG = 'logging.config.dictConfig'
LOGIN_REDIRECT_URL = '/'
LOGIN_URL = '/auth/login/'
LOGOUT_REDIRECT_URL = '/'
LOGOUT_URL = '/auth/logout/'
MAIN_DOMAIN = 'social.pygos.space'
MANAGERS = []
MEDIA_ROOT = '/takahe/media/'
MEDIA_URL = 'https://social.pygos.space/media/'
MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage'
MIDDLEWARE = ['core.middleware.SentryTaggingMiddleware', 'django.middleware.security.SecurityMiddleware', 'corsheaders.middleware.CorsMiddleware', 'whitenoise.middleware.WhiteNoiseMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django_htmx.middleware.HtmxMiddleware', 'core.middleware.HeadersMiddleware', 'core.middleware.ConfigLoadingMiddleware', 'api.middleware.ApiTokenMiddleware', 'users.middleware.DomainMiddleware']
MIGRATION_MODULES = {}
MONTH_DAY_FORMAT = 'F j'
NUMBER_GROUPING = 0
PASSWORD_HASHERS = '********************'
PASSWORD_RESET_TIMEOUT = '********************'
PREPEND_WWW = False
ROBOTS_TXT_DISALLOWED_USER_AGENTS = []
ROOT_URLCONF = 'takahe.urls'
SECRET_KEY = '********************'
SECRET_KEY_FALLBACKS = '********************'
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_CROSS_ORIGIN_OPENER_POLICY = 'same-origin'
SECURE_HSTS_INCLUDE_SUBDOMAINS = False
SECURE_HSTS_PRELOAD = False
SECURE_HSTS_SECONDS = 0
SECURE_PROXY_SSL_HEADER = "('HTTP_X_FORWARDED_PROTO', 'https')"
SECURE_REDIRECT_EXEMPT = []
SECURE_REFERRER_POLICY = 'same-origin'
SECURE_SSL_HOST = None
SECURE_SSL_REDIRECT = False
SERVER_EMAIL = '[takahe@pygos.space](mailto:takahe@pygos.space)'
SESSION_CACHE_ALIAS = 'default'
SESSION_COOKIE_AGE = 1209600
SESSION_COOKIE_DOMAIN = None
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_NAME = 'sessionid'
SESSION_COOKIE_PATH = '/'
SESSION_COOKIE_SAMESITE = 'Lax'
SESSION_COOKIE_SECURE = False
SESSION_ENGINE = 'django.contrib.sessions.backends.signed_cookies'
SESSION_EXPIRE_AT_BROWSER_CLOSE = False
SESSION_FILE_PATH = None
SESSION_SAVE_EVERY_REQUEST = False
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer'
SETTINGS_MODULE = 'takahe.settings'
SETUP = Settings(DATABASE_SERVER=ImplicitHostname('postgres://<REDACTED>@localhost/takahe?sslmode=disable', scheme='postgres', user='<REDACTED>', password='<REDACTED>', host='localhost', host_type='int_domain', path='/takahe', query='sslmode=disable'), ENVIRONMENT='development', DEBUG=False, DEBUG_TOOLBAR=False, LOCAL_SETTINGS=False, SECRET_KEY='<REDACTED>', STATOR_TOKEN='68f9cb85d28c1dfd842128bba90f4082e680cf2fbc400c743729fb1b5c52880462ca68cd412b071ff7662d37609547095504ddbf19156cdb9415409fe042b57fd48a0edc658ec58e73b57ecedaa79bf9671176a73ab85008eb037398b9423f0a4ae680688d7183c257d07a703eb5733667681f127b8cd4b9ca9c499cc616cea0', ALLOWED_HOSTS=['*'], CORS_HOSTS=[], CSRF_HOSTS=['https://social.pygos.space'], USE_PROXY_HEADERS=True, SENTRY_DSN=None, SENTRY_SAMPLE_RATE=1.0, SENTRY_TRACES_SAMPLE_RATE=0.01, SENTRY_CAPTURE_MESSAGES=False, SENTRY_EXPERIMENTAL_PROFILES_TRACES_SAMPLE_RATE=0.0, MAIN_DOMAIN='social.pygos.space', EMAIL_SERVER=AnyUrl('smtp://<REDACTED>@pygos.space:587?tls=true', scheme='smtp', user='<REDACTED>', password='<REDACTED>', host='pygos.space', tld='space', host_type='domain', port='587', query='tls=true'), EMAIL_FROM='takahe@pygos.space', AUTO_ADMIN_EMAIL=None, ERROR_EMAILS=['<REDACTED>@pygos.space'], ROBOTS_TXT_DISALLOWED_USER_AGENTS=[], MEDIA_URL='https://social.pygos.space/media/', MEDIA_ROOT='/takahe/media/', MEDIA_BACKEND=MediaBackendUrl('local://', scheme='local'), MEDIA_BACKEND_S3_ACL='public-read', MEDIA_MAX_IMAGE_FILESIZE_MB=10, AVATAR_MAX_IMAGE_FILESIZE_KB=1000, EMOJI_MAX_IMAGE_FILESIZE_KB=200, REMOTE_TIMEOUT=5.0, SEARCH=True, CACHES_DEFAULT=None, STATOR_CONCURRENCY=50, STATOR_CONCURRENCY_PER_MODEL=15, VAPID_PUBLIC_KEY=None, VAPID_PRIVATE_KEY=None, PGHOST=None, PGPORT=5432, PGNAME='takahe', PGUSER='postgres', PGPASSWORD=None)
SHORT_DATETIME_FORMAT = 'm/d/Y P'
SHORT_DATE_FORMAT = 'm/d/Y'
SIGNING_BACKEND = 'django.core.signing.TimestampSigner'
SILENCED_SYSTEM_CHECKS = []
STATICFILES_DIRS = [PosixPath('/takahe/static')]
STATICFILES_FINDERS = ['django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder']
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_ROOT = PosixPath('/takahe/static-collected')
STATIC_URL = '/static/'
STATOR_CONCURRENCY = 50
STATOR_CONCURRENCY_PER_MODEL = 15
STATOR_TOKEN = '********************'
STORAGES = {'staticfiles': {'BACKEND': 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'}, 'default': {'BACKEND': 'django.core.files.storage.FileSystemStorage'}}
TAKAHE_ENV_FILE = '.env'
TAKAHE_USER_AGENT = 'python-httpx/0.24.1 (Takahe/0.10.0-dev; +https://social.pygos.space/)'
TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [PosixPath('/takahe/templates')], 'APP_DIRS': True, 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', 'core.context.config_context', 'users.context.user_context']}}]
TEST_NON_SERIALIZED_APPS = []
TEST_RUNNER = 'django.test.runner.DiscoverRunner'
THOUSAND_SEPARATOR = ','
TIME_FORMAT = 'P'
TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
TIME_ZONE = 'UTC'
USE_DEPRECATED_PYTZ = False
USE_I18N = True
USE_L10N = True
USE_THOUSAND_SEPARATOR = False
USE_TZ = True
USE_X_FORWARDED_HOST = False
USE_X_FORWARDED_PORT = False
WHITENOISE_MAX_AGE = 3600
WSGI_APPLICATION = 'takahe.wsgi.application'
X_FRAME_OPTIONS = 'DENY'
YEAR_MONTH_FORMAT = 'F Y'
Screenshot from the Tusky app:
What version are you running? I suspect this was already fixed as part of #634, it's just not in a release yet.
I'm on main. The build is from yesterday
The error does not occur when I reduce gunicorn workers to 1 (GUNICORN_EXTRA_CMD_ARGS: "--reload -w 1").
But posts and follows are still not displayed. That is probably another issue.
OK - I'll see if I can investigate it soon then. It might be a few weeks. Looks like some sort of error in the new fetching code, maybe - have you tried the current stable release to see if it still breaks there?
No I haven't and unfortunately I can't downgrade due to database issues. I'll try to check this with a test instance.
Ah right, we've had migrations on main since the release (which is incidentally why I have not done a patch release yet). Let me know what you find - if this is caused by our fix for the other error breaking search, it would be good to know.
But posts and follows are still not displayed. That is probably another issue.
I did some research and at least for mastodon, only follows of the local instance are shown in the mobile app. And for posts, only future posts are downloaded to the local instance and counted.
That means if I search for a new user once, the user is known to my server from that point in time. If I check the users profile via the mobile app, it shows me 0 follows and 0 posts (except there are pinned posts, which will be backfilled). If the user posts something from now on, it appears on the profile and the post numbers goes up by one. If I follow, the follow number goes up by one. To get the actual number of posts and followers, I have to check the web page of the remote profile.
It looks like this is desired behavior.
if this is caused by our fix for the other error breaking search, it would be good to know.
The same error also occurs on 0.9.0. Error log for version 0.9.0 below.
takahe-web | Internal Server Error: /api/v2/search
takahe-web | Traceback (most recent call last):
takahe-web | File "/takahe/activities/services/search.py", line 38, in search_identities_handle
takahe-web | raise Identity.DoesNotExist()
takahe-web | users.models.identity.Identity.DoesNotExist
takahe-web |
takahe-web | During handling of the above exception, another exception occurred:
takahe-web |
takahe-web | Traceback (most recent call last):
takahe-web | File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 349, in main_wrap
takahe-web | raise exc_info[1]
takahe-web | File "/takahe/users/models/identity.py", line 362, in by_username_and_domain
takahe-web | return cls.objects.get(
takahe-web | ^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
takahe-web | return getattr(self.get_queryset(), name)(*args, **kwargs)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
takahe-web | raise self.model.DoesNotExist(
takahe-web | users.models.identity.Identity.DoesNotExist: Identity matching query does not exist.
takahe-web |
takahe-web | During handling of the above exception, another exception occurred:
takahe-web |
takahe-web | Traceback (most recent call last):
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
takahe-web | return self.cursor.execute(sql, params)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 737, in execute
takahe-web | raise ex.with_traceback(None)
takahe-web | psycopg.errors.UniqueViolation: duplicate key value violates unique constraint "users_identity_actor_uri_key"
takahe-web | DETAIL: Key (actor_uri)=(https://jointakahe.takahe.social/@takahe@jointakahe.org/) already exists.
takahe-web | The above exception was the direct cause of the following exception:
takahe-web |
takahe-web | Traceback (most recent call last):
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
takahe-web | response = get_response(request)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
takahe-web | response = wrapped_callback(request, *callback_args, **callback_kwargs)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/takahe/api/decorators.py", line 46, in inner
takahe-web | return function(request, *args, **kwargs)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/hatchway/view.py", line 302, in __call__
takahe-web | response = self.view(request, **kwargs)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/takahe/api/views/search.py", line 35, in search
takahe-web | search_result = searcher.search_all()
takahe-web | ^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/takahe/activities/services/search.py", line 139, in search_all
takahe-web | "identities": self.search_identities_handle(),
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/takahe/activities/services/search.py", line 48, in search_identities_handle
takahe-web | identity = Identity.by_username_and_domain(
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/takahe/users/models/identity.py", line 382, in by_username_and_domain
takahe-web | return cls.objects.create(
takahe-web | ^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
takahe-web | return getattr(self.get_queryset(), name)(*args, **kwargs)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 658, in create
takahe-web | obj.save(force_insert=True, using=self.db)
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
takahe-web | self.save_base(
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
takahe-web | updated = self._save_table(
takahe-web | ^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
takahe-web | results = self._do_insert(
takahe-web | ^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
takahe-web | return manager._insert(
takahe-web | ^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
takahe-web | return getattr(self.get_queryset(), name)(*args, **kwargs)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
takahe-web | return query.get_compiler(using=using).execute_sql(returning_fields)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
takahe-web | cursor.execute(sql, params)
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 102, in execute
takahe-web | return super().execute(sql, params)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
takahe-web | return self._execute_with_wrappers(
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
takahe-web | return executor(sql, params, many, context)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
takahe-web | with self.db.wrap_database_errors:
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
takahe-web | raise dj_exc_value.with_traceback(traceback) from exc_value
takahe-web | File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
takahe-web | return self.cursor.execute(sql, params)
takahe-web | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
takahe-web | File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 737, in execute
takahe-web | raise ex.with_traceback(None)
takahe-web | django.db.utils.IntegrityError: duplicate key value violates unique constraint "users_identity_actor_uri_key"
takahe-web | DETAIL: Key (actor_uri)=(https://jointakahe.takahe.social/@takahe@jointakahe.org/) already exists.
Thanks, appreciate you checking! I can't easily replicate it on my own server, but the traceback you provided should hopefully be enough to put a potential fix in place anyway.
Hmm, taking at a look at the source code, we already have something in place to stop this happening - it's meant to check if the user exists right before it makes it.
Are you able to go into the Django Admin for me and get a screenshot of what the https://jointakahe.takahe.social/@takahe@jointakahe.org/ / @takahe@jointakahe.org identity looks like, if it even exists? I suspect something about the record is corrupted but I don't know how.
Are you able to go into the Django Admin for me and get a screenshot of what the
https://jointakahe.takahe.social/@takahe@jointakahe.org//@takahe@jointakahe.orgidentity looks like, if it even exists? I suspect something about the record is corrupted but I don't know how.
Like this?
Without looking at the code I assume this is a concurrency issue because it only occurs the first time a search for a user. I guess there are two processes that check for the existence of the db entry, both get the result that it doesn't exist and both try to create it. The creation fails the second time because the first one already created it.
The issue also doesn't occur with only one worker.
Ah, I missed that it only happens on the first search for a user, sorry - you're probably right. The search endpoint should only be called once, but my guess is that the client you're using calls it twice for some reason.
OK, the commit I just landed above should fix it - I made that whole process happen in a transaction. Let me know if it doesn't and we can reopen.
Ah nvm the error still occurs. Same as before :/
I did some testing and added a few print statements to see whats wrong.
Diff
diff --git a/users/models/identity.py b/users/models/identity.py
index bebaa31..46f5b14 100644
--- a/users/models/identity.py
+++ b/users/models/identity.py
@@ -2,6 +2,9 @@ import ssl
from functools import cached_property, partial
from typing import Literal, Optional
from urllib.parse import urlparse
+import traceback
+from datetime import datetime
+from random import randint
import httpx
import urlman
@@ -396,14 +399,20 @@ class Identity(StatorModel):
domain = domain.lower()
with transaction.atomic():
+ debug_nonce = randint(0, 2**16 - 1)
+ tprint = lambda x: print(str(debug_nonce) + " " + str(datetime.now().strftime("%d.%m.%Y %H:%M:%S.%f")) + " " + x)
+ #traceback.print_stack()
+ tprint("Entering transaction atomic")
try:
if local:
+ tprint("local-before-return")
return cls.objects.get(
username__iexact=username,
domain_id=domain,
local=True,
)
else:
+ tprint("else-before-return")
return cls.objects.get(
username__iexact=username,
domain_id=domain,
@@ -412,9 +421,11 @@ class Identity(StatorModel):
if fetch and not local:
actor_uri, handle = cls.fetch_webfinger(f"{username}@{domain}")
if handle is None:
+ tprint("handle-before-return")
return None
# See if this actually does match an existing actor
try:
+ tprint("try-before-return")
return cls.objects.get(actor_uri=actor_uri)
except cls.DoesNotExist:
pass
@@ -422,13 +433,16 @@ class Identity(StatorModel):
username, domain = handle.split("@")
if not domain_instance:
domain_instance = Domain.get_remote_domain(domain)
+ tprint("domain-before-return")
return cls.objects.create(
actor_uri=actor_uri,
username=username,
domain_id=domain_instance,
local=False,
)
+ tprint("final-before-return")
return None
+ tprint("no-return hit")
@classmethod
def by_actor_uri(cls, uri, create=False, transient=False) -> "Identity":And I got the following output (random number per process, time, message), sorted by time:
244 16.09.2023 09:38:36.219160 Entering transaction atomic
244 16.09.2023 09:38:36.219348 else-before-return
61503 16.09.2023 09:38:36.227405 Entering transaction atomic
61503 16.09.2023 09:38:36.227565 else-before-return
61503 16.09.2023 09:38:36.682489 try-before-return
61503 16.09.2023 09:38:36.687559 domain-before-return
244 16.09.2023 09:38:36.704224 try-before-return
So there are two processes that run concurrently. But even with atomic transaction, the second process starts executing the code before the first one is finished.
Maybe also relevant: I see no error in the docker log, it's only sent via mail due to the TAKAHE_ERROR_EMAILS setting.
Atomic transactions don't guarantee that the code isn't run concurrently, just that the view of the database should be linear - though that's obviously not quite what's happening here.
I have some other ideas about how to fix this, so I'll try one of those in the next commit.
Did that next commit happen?