UnicodeEncodeError when searching for a "wrong" query
RoelofvdG opened this issue · comments
When searching for a query that has a "mistake" in it such as "Eminm" or "Rihana", an UnicodeEncodeError
is thrown in the console, regardless of it being caught in an except block. The suggestedQuery
field is also not available in the result. From looking at other posts I see similar reports, but those are from a long time ago and appear to be fixed.
Code:
try:
results = api.search("Eminm", 10)
except UnicodeEncodeError:
pass
The error message:
Traceback (most recent call last):
File "D:\Program Files (x86)\Python\Python36\lib\logging\__init__.py", line 994, in emit
stream.write(msg)
File "D:\Program Files (x86)\Python\Python36\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 32554-32556: character maps to <undefined>
Call stack:
File "E:\Gebruikers\Roelof\Dropbox\Public\Python\Google Music haxx\test.py", line 9, in <module>
results = api.search("Rihana", 10)
File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\mobileclient.py", line 1944, in search
res = self._make_call(mobileclient.Search, query, max_results)
File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\mobileclient.py", line 48, in _make_call
return super(Mobileclient, self)._make_call(protocol, *args, **kwargs)
File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\shared.py", line 90, in _make_call
return protocol.perform(self.session, self.validate, *args, **kwargs)
File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\protocol\shared.py", line 247, in perform
log.debug(cls.filter_response(parsed_response))
Message: {'kind': 'sj#searchresponse', 'suggestedQuery': 'Eminem', 'clusterDetail': [{'cluster': {'type': '5',(...)
The message goes on for longer but only lists the result. In the error message the suggestedQuery
is present, unlike in the actual result of the api call.
Interesting, it's coming from the stdlib. What version of windows are you on? I wonder what the terminal encoding is.
I am on 64 bit Windows 10 version 1803 build 17134.471. Assuming that you are on Windows 10 as well, are you able to reproduce this error?
I've got a windows 8 machine I can try with later.
Like the title suggests I only get this behaviour when the query returns a suggestedQuery
(I don't know if that is the actual problem or not) but I haven't been able to find any leads on what might be causing it.
My guess is that response has something non-ascii in it. You could check by printing the repr of that key (you may need to disable logging first to get at it).
The thing is that even though I get an error message in the console, the actual UnicodeEncodeError
is caught before I can do anything with it and any code after it gets executed as normal. When I put an except block that catches any exception the error is still thrown. The stacktrace seems to be caught and printed to the console before it reaches the program's code. Any ideas on how I can get at it?
Ah, yeah, that's because it's coming out of the logging in the stdlib: it will print encoding errors and swallow the exception so that things keep running.
How would I go about trying to determine where the error originates from?
The error is from inside the stdlib, and caused by gmusicapi attempting to log something in that response which can't be printed according to the terminal's encoding.
Something like print(repr(api.search("Eminm", 10)))
should print the response safely. I'd guess it has some u'\x...'
chars in it that would need to be encoded with something like utf-8.
This gives me the exact same result as when just doing a normal print()
.
In case I am mistaken, I included the log anyway:
Traceback (most recent call last): File "D:\Program Files (x86)\Python\Python36\lib\logging\__init__.py", line 994, in emit stream.write(msg) File "D:\Program Files (x86)\Python\Python36\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 33436-33439: character maps to <undefined> Call stack: File "E:\Gebruikers\Roelof\Dropbox\Public\Python\Google Music haxx\test.py", line 11, in <module> print(repr(api.search("Eminm", 10))) File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\mobileclient.py", line 1944, in search res = self._make_call(mobileclient.Search, query, max_results) File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\mobileclient.py", line 48, in _make_call return super(Mobileclient, self)._make_call(protocol, *args, **kwargs) File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\shared.py", line 90, in _make_call return protocol.perform(self.session, self.validate, *args, **kwargs) File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\protocol\shared.py", line 247, in perform log.debug(cls.filter_response(parsed_response)) Message: {'kind': 'sj#searchresponse', 'suggestedQuery': 'Eminem', 'clusterDetail': [{'cluster': {'type': '5', 'category': '1', 'id': 'search_genre'}, 'resultToken': 'EAU'}, {'cluster': {'type': '2', 'category': '1', 'id': 'search_artist'}, 'resultToken': 'EAI', 'entries': [{'type': '2', 'artist': {'kind': 'sj#artist', 'name': 'Eminem', 'artistArtRef': 'http://lh3.googleusercontent.com/zBsqT651ZTQxZ42mx64KO__C2Nv1J_qC4...
Maybe it has something to do with the gmusicapi not expecting the extra data generated by performing a "wrong" query? The 'kind' and 'suggestedQuery' keys are not present in the dictionary that is returned by the api call.
This oddly occurs to me on Windows even though I'm not printing to the screen. I ran a search:
mm.search('Cat Stevens Wild World')
and got the following error:
--- Logging error ---
Traceback (most recent call last):
File "c:\anaconda2\envs\py36\Lib\logging\__init__.py", line 994, in emit
stream.write(msg)
File "f:\temp\gmusic\env\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u0391' in position 30200: character maps to <undefined>
Call stack:
File "c:\anaconda2\envs\py36\Lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\anaconda2\envs\py36\Lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "f:\temp\gmusic\env\lib\site-packages\ipykernel_launcher.py", line 16, in <module>
app.launch_new_instance()
File "f:\temp\gmusic\env\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
app.start()
File "f:\temp\gmusic\env\lib\site-packages\ipykernel\kernelapp.py", line 505, in start
self.io_loop.start()
File "f:\temp\gmusic\env\lib\site-packages\tornado\platform\asyncio.py", line 148, in start
self.asyncio_loop.run_forever()
File "c:\anaconda2\envs\py36\Lib\asyncio\base_events.py", line 421, in run_forever
self._run_once()
File "c:\anaconda2\envs\py36\Lib\asyncio\base_events.py", line 1425, in _run_once
handle._run()
File "c:\anaconda2\envs\py36\Lib\asyncio\events.py", line 127, in _run
self._callback(*self._args)
File "f:\temp\gmusic\env\lib\site-packages\tornado\ioloop.py", line 690, in <lambda>
lambda f: self._run_callback(functools.partial(callback, future))
File "f:\temp\gmusic\env\lib\site-packages\tornado\ioloop.py", line 743, in _run_callback
ret = callback()
File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 781, in inner
self.run()
File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 742, in run
yielded = self.gen.send(value)
File "f:\temp\gmusic\env\lib\site-packages\ipykernel\kernelbase.py", line 357, in process_one
yield gen.maybe_future(dispatch(*args))
File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 209, in wrapper
yielded = next(result)
File "f:\temp\gmusic\env\lib\site-packages\ipykernel\kernelbase.py", line 267, in dispatch_shell
yield gen.maybe_future(handler(stream, idents, msg))
File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 209, in wrapper
yielded = next(result)
File "f:\temp\gmusic\env\lib\site-packages\ipykernel\kernelbase.py", line 534, in execute_request
user_expressions, allow_stdin,
File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 209, in wrapper
yielded = next(result)
File "f:\temp\gmusic\env\lib\site-packages\ipykernel\ipkernel.py", line 294, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "f:\temp\gmusic\env\lib\site-packages\ipykernel\zmqshell.py", line 536, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 2848, in run_cell
raw_cell, store_history, silent, shell_futures)
File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 2874, in _run_cell
return runner(coro)
File "f:\temp\gmusic\env\lib\site-packages\IPython\core\async_helpers.py", line 67, in _pseudo_sync_runner
coro.send(None)
File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 3049, in run_cell_async
interactivity=interactivity, compiler=compiler, result=result)
File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 3220, in run_ast_nodes
if (yield from self.run_code(code, result)):
File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 3296, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-68-3d98d05bb8f6>", line 1, in <module>
repr(mm.search('Cat Stevens - Wild World'))
File "f:\temp\gmusic\env\lib\site-packages\gmusicapi\clients\mobileclient.py", line 1961, in search
res = self._make_call(mobileclient.Search, query, max_results)
File "f:\temp\gmusic\env\lib\site-packages\gmusicapi\clients\mobileclient.py", line 48, in _make_call
return super(Mobileclient, self)._make_call(protocol, *args, **kwargs)
File "f:\temp\gmusic\env\lib\site-packages\gmusicapi\clients\shared.py", line 90, in _make_call
return protocol.perform(self.session, self.validate, *args, **kwargs)
File "f:\temp\gmusic\env\lib\site-packages\gmusicapi\protocol\shared.py", line 247, in perform
log.debug(cls.filter_response(parsed_response))
Message:
The "message" that follows is an API response which includes the track I searched for:
Message: {'kind': 'sj#searchresponse', 'clusterDetail': [{'cluster': {'type': '5', 'category': '1', 'id': 'search_genre'}, 'resultToken': 'EAU'}, {'cluster': {'type': '2', 'category': '1', 'id': 'search_artist'}, 'resultToken': 'EAI', 'entries': [{'type': '2', 'artist': {'kind': 'sj#artist', 'name': 'Cat Stevens', 'artistArtRef': 'http://lh3.googleusercontent.com/HDs-d4ub0wJ6mlPExcCXACNGQE5ZqAC4sNVgQVZNzo4YMqZ2Kv49cI5O_nXHZjL-e6lmJ1UAgQ', 'artistArtRefs': [{'kind': 'sj#imageRef', 'url': 'http://lh3.googleusercontent.com/HDs-d4ub0wJ6mlPExcCXACNGQE5ZqAC4sNVgQVZNzo4YMqZ2Kv49cI5O_nXHZjL-e6lmJ1UAgQ', 'aspectRatio': '2', 'autogen': False}], 'artistId': 'Avniovlu2t5mvaenz7fu4lyurby', 'artist_bio_attribution': {'kind': 'sj#attribution', 'source_title': 'Wikipedia', 'source_url': 'https://en.wikipedia.org/wiki/Cat_Stevens', 'license_title': 'Creative Commons Attribution CC-BY-SA 4.0', 'license_url': 'http://creativecommons.org/licenses/by-sa/4.0/legalcode'}}, 'cluster': [{'type': '2', 'category': '1', 'id': 'search_artist'}]}]}, {'cluster': {'type': '3', 'category': '1', 'id': 'search_album'}, 'resultToken': 'EAM', 'entries': [{'type': '3', 'album': {'kind': 'sj#album', 'name': 'Tea For The Tillerman', 'albumArtist': 'Cat Stevens', 'albumArtRef': 'http://lh3.googleusercontent.com/HkpCQAGwGohvsDhQbtV4aQ-I1gU1noheQbe2gSXev3MD7eKNhNzdvARhclfLwTBnyvCKsQvnGqI', 'albumId': 'B243eklt72x2akr555zqanwuje4', 'artist': 'Cat Stevens', 'artistId': ['Avniovlu2t5mvaenz7fu4lyurby'], 'description_attribution': {'kind': 'sj#attribution', 'source_title': 'Wikipedia', 'source_url': 'https://en.wikipedia.org/wiki/Tea_for_the_Tillerman', 'license_title': 'Creative Commons Attribution CC-BY-SA 4.0', 'license_url': 'http://creativecommons.org/licenses/by-sa/4.0/legalcode'}, 'year': 1970, 'explicitType': '2'}, 'cluster': [{'type': '3', 'category': '1', 'id': 'search_album'}]}, {'type': '3', 'album': {'kind': 'sj#album', 'name': 'Wild World: Tribute to Cat Stevens', 'albumArtist': 'Music Factory', 'albumArtRef': 'http://lh6.ggpht.com/bthn9-1JIO0AoR-ExsddMTZmWkacSjErx1clK1cM4NV9MTS6I9s7h-D_RqtimxewmKwTqr9RFw', 'albumId': 'B7tytcsvrdm46jq5x37mpigg2ga', 'artist': 'Music Factory', 'artistId': ['Afl5uxfrorf6keqvj5wvjnrcdz4'], 'year': 2014, 'explicitType': '2'}, 'cluster': [{'type': '3', 'category': '1', 'id': 'search_album'}]}, {'type': '3', 'album': {'kind': 'sj#album', 'name': 'Wild World: Tribute To Cat Stevens', 'albumArtist': 'Mystique', 'albumArtRef': 'http://lh5.ggpht.com/29C3AvPKoNr1wGyWomVrIwHJwG0WeuZFNz8cCDry_KlNv-mXtgVen4hqb_CW0ZhxzvkitG046A', 'albumId': 'Bozzyppzpljr7qmnywtjgdf6wne', 'artist': 'Mystique', 'artistId': ['Azfcsuute3rvgblikifebp2hmia'], 'year': 2011, 'explicitType': '2'}, 'cluster': [{'type': '3', 'category': '1', 'id': 'search_album'}]}]}, {'cluster': {'type': '1', 'category': '1', 'id': 'search_track'}, 'resultToken': 'EAE', 'entries': [{'type': '1', 'track': {'kind': 'sj#track', 'title': 'Wild World', 'artist': 'Cat Stevens', 'composer': 'Cat Stevens', 'album': 'Music & Highlights: Just The Best', 'albumArtist': 'Various Artists', 'year': 2012, 'trackNumber': 6, 'genre': 'Pop', 'durationMillis': '202000', 'albumArtRef': [{'kind': 'sj#imageRef', 'url': 'http://lh3.googleusercontent.com/RvOkeUZitLtotzG1nEsvap2E3Ll67hhcpmbtR0HqsjkdyEegCnR0s_LVr73hwsxVa7Y7bHvD', 'aspectRatio': '1', 'autogen': False}], 'discNumber': 1, 'estimatedSize': '8103182', 'trackType': '7', 'storeId': 'Tdl3fiome2imze5pbi75fndj7na', 'albumId': 'm3w6aafisdjmoaa5v7plfa5jji', 'artistId': ['vniovlu2t5mvaenz7fu4lyurby'], 'nid': 'dl3fiome2imze5pbi75fndj7na', 'trackAvailableForSubscription': True, 'trackAvailableForPurchase': True, 'albumAvailableForPurchase': True, 'explicitType': '2'}, 'cluster': [{'type': '1', 'category': '1', 'id': 'search_track'}]}, {'type': '1', 'track': {'kind': 'sj#track', 'title': 'Wild World', 'artist': 'Cat Stevens', 'composer': 'Cat Stevens', 'album': 'Tonight Show: One Special Night', 'albumArtist': 'Various Artists', 'year': 2012, 'trackNumber': 8, 'genre': 'Pop', 'durationMillis': '202000', 'albumArtRef': [{'kind': 'sj#imageRef', 'url': 'http://lh6.ggpht.com/p575huO2Bh4X_pwItAdTFKkS2WA1gY681ekNmxhywUy4PdS_wvKbTWUT3ZXjKuT6k-wuoSf6Dg', 'aspectRatio': '1', 'autogen': False}], 'discNumber': 1, 'estimatedSize': '8107396', 'trackType': '7', 'storeId': 'Tp53n56pa4nnhnslyukav6xhtim', 'albumId': 'udm7grgyx6zrm4k6begkng5pzq', 'artistId': ['vniovlu2t5mvaenz7fu4lyurby'], 'nid': 'p53n56pa4nnhnslyukav6xhtim', 'trackAvailableForSubscription': True, 'trackAvailableForPurchase': True, 'albumAvailableForPurchase': True, 'explicitType': '2'}, 'cluster': [{'type': '1', 'category': '1', 'id': 'search_track'}]}, {'type': '1', 'track': {'kind': 'sj#track', 'title': 'Wild World', 'artist': 'Cat Stevens', 'composer': '', 'album': 'Greatest Hits', 'albumArtist': 'Cat Stevens', 'year': 1975, 'trackNumber': 1, 'genre': 'Singer-Songwriter', 'durationMillis': '202000', 'albumArtRef': [{'kind': 'sj#imageRef', 'url': 'http://lh3.googleusercontent.com/DpSQErA2oKWlblcEdBQhXOjs98SfaUb5AP33zx31i5r9A89nrw1D5lnaTrnyTV-_JQJlpevi9Q', 'aspectRatio': '1', 'autogen': False}], 'discNumber': 1, 'estimatedSize': '8100086', 'trackType': '7', 'storeId': 'Tokn3xyzgx2vgvswz6wccqavdha', 'albumId': '5pdcjv5ai56ol6ke5gvq6llkny', 'artistId': ['vniovlu2t5mvaenz7fu4lyurby'], 'nid': 'okn3xyzgx2vgvswz6wccqavdha', 'trackAvailableForSubscription': True, 'trackAvailableForPurchase': True, 'albumAvailableForPurchase': True, 'explicitType': '2'},
It goes on for awhile with the entire response. Any idea what's happening here?
It's what I described originally.
I still can't reproduce it. It'd be helpful if you could put the full response including the bad character in a gist so I can see it in context.
It's what I described originally.
I still can't reproduce it. It'd be helpful if you could put the full response including the bad character in a gist so I can see it in context.
Here's a gist with the logging error I'm seeing: https://gist.github.com/sheilnaik/51abd72cb523dd37787f8d6d85f61505
Disabling logging with
import logging
import sys
logging.disable(sys.maxsize)
does hide the ugly error message as you mentioned earlier.
Huh, I don't see any troublesome characters in there.
I can give this another shot later with 3.6 (I couldn't reproduce on 2.7).
It was probably luck, but I did reproduce this with 3.6. The trouble seems to be misencoded strings in the names or descriptions of user-created playlists. For example, I was given:
{
'type': '4',
'category': '1',
'id': 'search_playlist'
}]
}, {
'type': '4',
'playlist': {
'kind': 'sj#playlist',
'name': '\u261403.02.2019\U0001f3d5\U0001f3d5\U0001f3d5Bowie,Tee Set,Kinks,Chuby C.,James Cotton,Beatl.,,Fats D.Rolling,James Cotton,Junior Wells,RodPeter Gr.,Tull,Pink77p\U0001f3d5\U0001f3d5\U0001f3d5',
'type': 'SHARED',
'shareToken': 'AMaBXyk3qcnsHNBcqJRJrc9T78NbwlPsOMKlau9f8bjmgjMLitKLkU8WWOmXo34oLN57g1fUraeBKIJlDrRvqVdhpE-vHZCFOg==',
'description': 'RARE 12,,POP,,Blues,MARCH 2 2019'
}
Chardet guesses {'encoding': 'Windows-1254', 'confidence': 0.37371958509666425, 'language': 'Turkish'}
, but that's not a valid encoding for it afaict. It was probably double encoded or something at some point.
I don't really think there's anything we can do to clean this up if Google is serving it this way, but we can probably change the logging encoding or allow it to drop bad characters.
That seems to work for me. Let me know if you notice the error on the dev version (you can use pip install git+https://github.com/simon-weber/gmusicapi.git@develop#egg=gmusicapi
to install it).