URL Parsing Error

Question

URL Parsing Error

doodlemania2 opened this issue 6 months ago · comments

Internal Server Error: /api/v2/search

JSONDecodeError at /api/v2/search
Expecting value: line 1 column 1 (char 0)

Request Method: GET
Request URL:
Django Version: 4.2.7
Python Executable: /usr/local/bin/python3
Python Version: 3.11.6
Python Path: ['/takahe', '/usr/local/bin', '/usr/local/lib/python311.zip', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/site-packages']
Server time: Mon, 15 Jan 2024 17:35:24 +0000
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.postgres',
'corsheaders',
'django_htmx',
'hatchway',
'core',
'activities',
'api',
'mediaproxy',
'stator',
'users']
Installed Middleware:
['core.middleware.SentryTaggingMiddleware',
'django.middleware.security.SecurityMiddleware',
'corsheaders.middleware.CorsMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django_htmx.middleware.HtmxMiddleware',
'core.middleware.HeadersMiddleware',
'core.middleware.ConfigLoadingMiddleware',
'api.middleware.ApiTokenMiddleware',
'users.middleware.DomainMiddleware']

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/api/decorators.py", line 46, in inner
return function(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/hatchway/view.py", line 302, in call
response = self.view(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/api/views/search.py", line 35, in search
search_result = searcher.search_all()
^^^^^^^^^^^^^^^^^^^^^
File "/takahe/activities/services/search.py", line 142, in search_all
url_result = self.search_url()
^^^^^^^^^^^^^^^^^
File "/takahe/activities/services/search.py", line 84, in search_url
document = canonicalise(response.json(), include_security=True)
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 761, in json
return jsonlib.loads(self.content, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/init.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: JSONDecodeError at /api/v2/search
Exception Value: Expecting value: line 1 column 1 (char 0)
Raised during: hatchway.view.inner
Request information:
USER:

GET:
q = 'https://www.bbc.com/sport/africa/67986623'

<Derek Note: the above q= was exploded by my link munger, but resolved to literally: https://www.bbc.com/sport/africa/67986623' <— note the ‘ at the end of the string was included in the exact string, I think there’s a parsing error in Takahe because obviously a ‘ isn’t allowed at the end of a string - anyways, it returned 404

type = 'statuses'
resolve = 'true'
limit = '1'

POST: No POST data

FILES: No FILES data

COOKIES: No cookie data

META:
HTTP_ACCEPT = '/'
HTTP_ACCEPT_ENCODING = 'gzip'
HTTP_ACCEPT_LANGUAGE = 'en-US,en;q=0.9'
HTTP_AUTHORIZATION = ''
HTTP_CDN_LOOP = 'cloudflare'
HTTP_CF_CONNECTING_IP = '172.59.194.14'
HTTP_CF_IPCOUNTRY = 'US'
HTTP_CF_RAY = '845fe000eda5eac9-DFW'
HTTP_CF_VISITOR = '{"scheme":"https"}'
HTTP_CONNECTION = 'close'
HTTP_DNT = '1'
HTTP_HOST = ''
HTTP_ORIGIN =
HTTP_PRIORITY = 'u=1, i'
HTTP_REFERER =
HTTP_SEC_CH_UA = '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"'
HTTP_SEC_CH_UA_MOBILE = '?0'
HTTP_SEC_CH_UA_PLATFORM = '"macOS"'
HTTP_SEC_FETCH_DEST = 'empty'
HTTP_SEC_FETCH_MODE = 'cors'
HTTP_SEC_FETCH_SITE = 'cross-site'
HTTP_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
HTTP_X_FORWARDED_FOR = '172.59.194.14, 172.71.170.164, 172.20.10.128'
HTTP_X_FORWARDED_PROTO = 'https'
HTTP_X_FORWARDED_SCHEME = 'https'
HTTP_X_REAL_IP = '172.71.170.164'
PATH_INFO = '/api/v2/search'
QUERY_STRING = 'q=https%3A%2F%2Fwww.bbc.com%2Fsport%2Fafrica%2F67986623&type=statuses&resolve=true&limit=1’ <— Derek note - here it is again
RAW_URI = '/api/v2/search?q=https%3A%2F%2Fwww.bbc.com%2Fsport%2Fafrica%2F67986623&type=statuses&resolve=true&limit=1' <—Derek note - and again
REMOTE_ADDR = '127.0.0.1'
REMOTE_PORT = '37070'
REQUEST_METHOD = 'GET'
SCRIPT_NAME = ''
SERVER_NAME = '0.0.0.0'
SERVER_PORT = '8001'
SERVER_PROTOCOL = 'HTTP/1.1'
SERVER_SOFTWARE = 'gunicorn/20.1.0'
gunicorn.socket = <socket.socket fd=8, family=2, type=1, proto=0, laddr=('127.0.0.1', 8001), raddr=('127.0.0.1', 37070)>
wsgi.errors = <gunicorn.http.wsgi.WSGIErrorsWrapper object at 0x7ff007ed60e0>
wsgi.file_wrapper = <class 'gunicorn.http.wsgi.FileWrapper'>
wsgi.input = <gunicorn.http.body.Body object at 0x7ff00875f450>
wsgi.input_terminated = True
wsgi.multiprocess = True
wsgi.multithread = False
wsgi.run_once = False
wsgi.url_scheme = 'https'
wsgi.version = '(1, 0)'

Settings:
Using settings module takahe.settings
ABSOLUTE_URL_OVERRIDES = {}
ADMINS = [('Admin', ')]
ALLOWED_HOSTS = ['*']
APPEND_SLASH = True
AUTHENTICATION_BACKENDS = ['django.contrib.auth.backends.ModelBackend']
AUTH_PASSWORD_VALIDATORS = ''
AUTH_USER_MODEL = 'users.User'
AUTO_ADMIN_EMAIL = '
AWS_ACCESS_KEY_ID = ''
AWS_DEFAULT_ACL = 'public-read'
AWS_QUERYSTRING_AUTH = False
AWS_S3_CUSTOM_DOMAIN = None
AWS_S3_ENDPOINT_URL = '
AWS_SECRET_ACCESS_KEY = ''
AWS_STORAGE_BUCKET_NAME = ''
BASE_DIR = PosixPath('/takahe')
CACHES = {'default': {'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache', 'LOCATION': 'cache'}}
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_KEY_PREFIX = ''
CACHE_MIDDLEWARE_SECONDS = 600
CORS_ALLOW_CREDENTIALS = True
CORS_ALLOW_HEADERS = "('accept', 'accept-encoding', 'authorization', 'content-type', 'dnt', 'origin', 'user-agent', 'x-csrftoken', 'x-requested-with', 'Idempotency-Key')"
CORS_EXPOSE_HEADERS = "('link',)"
CORS_ORIGIN_ALLOW_ALL = True
CORS_ORIGIN_WHITELIST = []
CORS_PREFLIGHT_MAX_AGE = 604800
CSRF_COOKIE_AGE = 31449600
CSRF_COOKIE_DOMAIN = None
CSRF_COOKIE_HTTPONLY = False
CSRF_COOKIE_MASKED = False
CSRF_COOKIE_NAME = 'csrftoken'
CSRF_COOKIE_PATH = '/'
CSRF_COOKIE_SAMESITE = 'Lax'
CSRF_COOKIE_SECURE = False
CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS = []
CSRF_USE_SESSIONS = False
DATABASES = {'default': {'ENGINE': 'django.db.backends.postgresql', ‘….
DATABASE_ROUTERS = []
DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440
DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000
DATA_UPLOAD_MAX_NUMBER_FILES = 100
DATETIME_FORMAT = 'N j, Y, P'
DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M']
DATE_FORMAT = 'N j, Y'
DATE_INPUT_FORMATS = ['%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y']
DEBUG = False
DEBUG_PROPAGATE_EXCEPTIONS = False
DECIMAL_SEPARATOR = '.'
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
DEFAULT_CHARSET = 'utf-8'
DEFAULT_EXCEPTION_REPORTER = 'django.views.debug.ExceptionReporter'
DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE = 'core.uploads.TakaheS3Storage'
DEFAULT_FROM_EMAIL = ''
DEFAULT_INDEX_TABLESPACE = ''
DEFAULT_TABLESPACE = ''
DISALLOWED_USER_AGENTS = []
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = ''
EMAIL_HOST_PASSWORD = '
EMAIL_HOST_USER = ''
EMAIL_PORT = 587
EMAIL_SSL_CERTFILE = None
EMAIL_SSL_KEYFILE = ''
EMAIL_SUBJECT_PREFIX = '[Django] '
EMAIL_TIMEOUT = None
EMAIL_USE_LOCALTIME = False
EMAIL_USE_SSL = False
EMAIL_USE_TLS = True
FILE_UPLOAD_DIRECTORY_PERMISSIONS = None
FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440
FILE_UPLOAD_PERMISSIONS = 420
FILE_UPLOAD_TEMP_DIR = None
FIRST_DAY_OF_WEEK = 0
FIXTURE_DIRS = []
FORCE_SCRIPT_NAME = None
FORMAT_MODULE_PATH = None
FORM_RENDERER = 'django.forms.renderers.DjangoTemplates'
IGNORABLE_404_URLS = []
INSTALLED_APPS = ['django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.postgres', 'corsheaders', 'django_htmx', 'hatchway', 'core', 'activities', 'api', 'mediaproxy', 'stator', 'users']
INTERNAL_IPS = []
JSONLD_MAX_SIZE = 51200
LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ar-dz', 'Algerian Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('ckb', 'Central Kurdish (Sorani)'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('hy', 'Armenian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('ig', 'Igbo'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kab', 'Kabyle'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('ky', 'Kyrgyz'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('ms', 'Malay'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('tg', 'Tajik'), ('th', 'Thai'), ('tk', 'Turkmen'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('uz', 'Uzbek'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')]
LANGUAGES_BIDI = ['he', 'ar', 'ar-dz', 'ckb', 'fa', 'ur']
LANGUAGE_CODE = 'en-us'
LANGUAGE_COOKIE_AGE = None
LANGUAGE_COOKIE_DOMAIN = None
LANGUAGE_COOKIE_HTTPONLY = False
LANGUAGE_COOKIE_NAME = 'django_language'
LANGUAGE_COOKIE_PATH = '/'
LANGUAGE_COOKIE_SAMESITE = None
LANGUAGE_COOKIE_SECURE = False
LOCALE_PATHS = []
LOGGING = {}
LOGGING_CONFIG = 'logging.config.dictConfig'
LOGIN_REDIRECT_URL = '/'
LOGIN_URL = '/auth/login/'
LOGOUT_REDIRECT_URL = '/'
LOGOUT_URL = '/auth/logout/'
MAIN_DOMAIN = 'ourselves.space'
MANAGERS = []
MEDIA_ROOT = '/takahe/media'
MEDIA_URL = '/media/'
MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage'
MIDDLEWARE = ['core.middleware.SentryTaggingMiddleware', 'django.middleware.security.SecurityMiddleware', 'corsheaders.middleware.CorsMiddleware', 'whitenoise.middleware.WhiteNoiseMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django_htmx.middleware.HtmxMiddleware', 'core.middleware.HeadersMiddleware', 'core.middleware.ConfigLoadingMiddleware', 'api.middleware.ApiTokenMiddleware', 'users.middleware.DomainMiddleware']
MIGRATION_MODULES = {}
MONTH_DAY_FORMAT = 'F j'
NUMBER_GROUPING = 0
PASSWORD_HASHERS = ''
PASSWORD_RESET_TIMEOUT = ''
PREPEND_WWW = False
ROBOTS_TXT_DISALLOWED_USER_AGENTS = []
ROOT_URLCONF = 'takahe.urls'
SECRET_KEY =
SECRET_KEY_FALLBACKS = '
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_CROSS_ORIGIN_OPENER_POLICY = 'same-origin'
SECURE_HSTS_INCLUDE_SUBDOMAINS = False
SECURE_HSTS_PRELOAD = False
SECURE_HSTS_SECONDS = 0
SECURE_PROXY_SSL_HEADER = "('HTTP_X_FORWARDED_PROTO', 'https')"
SECURE_REDIRECT_EXEMPT = []
SECURE_REFERRER_POLICY = 'same-origin'
SECURE_SSL_HOST = None
SECURE_SSL_REDIRECT = False
SERVER_EMAIL =
SESSION_CACHE_ALIAS = 'default'
SESSION_COOKIE_AGE = 1209600
SESSION_COOKIE_DOMAIN = None
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_NAME = 'sessionid'
SESSION_COOKIE_PATH = '/'
SESSION_COOKIE_SAMESITE = 'Lax'
SESSION_COOKIE_SECURE = False
SESSION_ENGINE = 'django.contrib.sessions.backends.signed_cookies'
SESSION_EXPIRE_AT_BROWSER_CLOSE = False
SESSION_FILE_PATH = None
SESSION_SAVE_EVERY_REQUEST = False
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer'
SETTINGS_MODULE = 'takahe.settings'
SETUP = Settings(…
SHORT_DATETIME_FORMAT = 'm/d/Y P'
SHORT_DATE_FORMAT = 'm/d/Y'
SIGNING_BACKEND = 'django.core.signing.TimestampSigner'
SILENCED_SYSTEM_CHECKS = []
STATICFILES_DIRS = [PosixPath('/takahe/static')]
STATICFILES_FINDERS = ['django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder']
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
STATIC_ROOT = PosixPath('/takahe/static-collected')
STATIC_URL = '/static/'
STATOR_CONCURRENCY = 20
STATOR_CONCURRENCY_PER_MODEL = 4
STATOR_TOKEN = ''
STORAGES = {'staticfiles': {'BACKEND': 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'}, 'default': {'BACKEND': 'core.uploads.TakaheS3Storage'}}
TAKAHE_ENV_FILE = '.env'
TAKAHE_USER_AGENT = 'python-httpx/0.25.1 (Takahe/0.10.1; +
TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [PosixPath('/takahe/templates')], 'APP_DIRS': True, 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', 'core.context.config_context', 'users.context.user_context']}}]
TEST_NON_SERIALIZED_APPS = []
TEST_RUNNER = 'django.test.runner.DiscoverRunner'
THOUSAND_SEPARATOR = ','
TIME_FORMAT = 'P'
TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
TIME_ZONE = 'UTC'
USE_DEPRECATED_PYTZ = False
USE_I18N = True
USE_L10N = True
USE_THOUSAND_SEPARATOR = False
USE_TZ = True
USE_X_FORWARDED_HOST = False
USE_X_FORWARDED_PORT = False
WHITENOISE_MAX_AGE = 3600
WSGI_APPLICATION = 'takahe.wsgi.application'
X_FRAME_OPTIONS = 'DENY'
YEAR_MONTH_FORMAT = 'F Y'

Jamie Bliss · Answer 1 · Mon Jan 29 2024 03:07:16 GMT+0800 (China Standard Time)

We spent some time triaging this in discord https://discord.com/channels/1044001375668936785/1192186468714692718/1192186468714692718

I spent a few minutes playing with Phanpy, and I am unable to reproduce this on main. I know the code in question has been touched since this release.

I did notice it didn't pick up on the new profile picture, but I didn't see any server-side problems.

Derek · Answer 2 · Mon Jan 29 2024 03:10:34 GMT+0800 (China Standard Time)

Fair enough. Should we roll new docker containers in a quick point release if there are other bug fixes that have been pulled in? Perhaps it will clear it!

Jamie Bliss · Answer 3 · Tue Feb 06 2024 10:42:21 GMT+0800 (China Standard Time)

i just made a release! So if it's cleared up, please close this. If not, please past an updated backtrace.