Giters

simon-weber / gmusicapi

An unofficial client library for Google Music.

Home Page:https://unofficial-google-music-api.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

UnicodeEncodeError when searching for a "wrong" query

RoelofvdG opened this issue · comments

commented

When searching for a query that has a "mistake" in it such as "Eminm" or "Rihana", an UnicodeEncodeError is thrown in the console, regardless of it being caught in an except block. The suggestedQuery field is also not available in the result. From looking at other posts I see similar reports, but those are from a long time ago and appear to be fixed.

Code:

try:
    results = api.search("Eminm", 10)
except UnicodeEncodeError:
    pass

The error message:

Traceback (most recent call last):
  File "D:\Program Files (x86)\Python\Python36\lib\logging\__init__.py", line 994, in emit
    stream.write(msg)
  File "D:\Program Files (x86)\Python\Python36\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 32554-32556: character maps to <undefined>
Call stack:
  File "E:\Gebruikers\Roelof\Dropbox\Public\Python\Google Music haxx\test.py", line 9, in <module>
    results = api.search("Rihana", 10)
  File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\mobileclient.py", line 1944, in search
    res = self._make_call(mobileclient.Search, query, max_results)
  File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\mobileclient.py", line 48, in _make_call
    return super(Mobileclient, self)._make_call(protocol, *args, **kwargs)
  File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\shared.py", line 90, in _make_call
    return protocol.perform(self.session, self.validate, *args, **kwargs)
  File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\protocol\shared.py", line 247, in perform
    log.debug(cls.filter_response(parsed_response))
Message: {'kind': 'sj#searchresponse', 'suggestedQuery': 'Eminem', 'clusterDetail': [{'cluster': {'type': '5',(...)

The message goes on for longer but only lists the result. In the error message the suggestedQuery is present, unlike in the actual result of the api call.

commented

Interesting, it's coming from the stdlib. What version of windows are you on? I wonder what the terminal encoding is.

commented

I am on 64 bit Windows 10 version 1803 build 17134.471. Assuming that you are on Windows 10 as well, are you able to reproduce this error?

commented

I've got a windows 8 machine I can try with later.

commented

Like the title suggests I only get this behaviour when the query returns a suggestedQuery (I don't know if that is the actual problem or not) but I haven't been able to find any leads on what might be causing it.

commented

My guess is that response has something non-ascii in it. You could check by printing the repr of that key (you may need to disable logging first to get at it).

commented

The thing is that even though I get an error message in the console, the actual UnicodeEncodeError is caught before I can do anything with it and any code after it gets executed as normal. When I put an except block that catches any exception the error is still thrown. The stacktrace seems to be caught and printed to the console before it reaches the program's code. Any ideas on how I can get at it?

commented

Ah, yeah, that's because it's coming out of the logging in the stdlib: it will print encoding errors and swallow the exception so that things keep running.

commented

How would I go about trying to determine where the error originates from?

commented

The error is from inside the stdlib, and caused by gmusicapi attempting to log something in that response which can't be printed according to the terminal's encoding.

Something like print(repr(api.search("Eminm", 10))) should print the response safely. I'd guess it has some u'\x...' chars in it that would need to be encoded with something like utf-8.

commented

This gives me the exact same result as when just doing a normal print().

In case I am mistaken, I included the log anyway:

Traceback (most recent call last): File "D:\Program Files (x86)\Python\Python36\lib\logging\__init__.py", line 994, in emit stream.write(msg) File "D:\Program Files (x86)\Python\Python36\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 33436-33439: character maps to <undefined> Call stack: File "E:\Gebruikers\Roelof\Dropbox\Public\Python\Google Music haxx\test.py", line 11, in <module> print(repr(api.search("Eminm", 10))) File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\mobileclient.py", line 1944, in search res = self._make_call(mobileclient.Search, query, max_results) File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\mobileclient.py", line 48, in _make_call return super(Mobileclient, self)._make_call(protocol, *args, **kwargs) File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\clients\shared.py", line 90, in _make_call return protocol.perform(self.session, self.validate, *args, **kwargs) File "D:\Program Files (x86)\Python\Python36\lib\site-packages\gmusicapi\protocol\shared.py", line 247, in perform log.debug(cls.filter_response(parsed_response)) Message: {'kind': 'sj#searchresponse', 'suggestedQuery': 'Eminem', 'clusterDetail': [{'cluster': {'type': '5', 'category': '1', 'id': 'search_genre'}, 'resultToken': 'EAU'}, {'cluster': {'type': '2', 'category': '1', 'id': 'search_artist'}, 'resultToken': 'EAI', 'entries': [{'type': '2', 'artist': {'kind': 'sj#artist', 'name': 'Eminem', 'artistArtRef': 'http://lh3.googleusercontent.com/zBsqT651ZTQxZ42mx64KO__C2Nv1J_qC4...

commented

Maybe it has something to do with the gmusicapi not expecting the extra data generated by performing a "wrong" query? The 'kind' and 'suggestedQuery' keys are not present in the dictionary that is returned by the api call.

commented

This oddly occurs to me on Windows even though I'm not printing to the screen. I ran a search:

mm.search('Cat Stevens Wild World')

and got the following error:

--- Logging error ---
Traceback (most recent call last):
  File "c:\anaconda2\envs\py36\Lib\logging\__init__.py", line 994, in emit
    stream.write(msg)
  File "f:\temp\gmusic\env\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u0391' in position 30200: character maps to <undefined>
Call stack:
  File "c:\anaconda2\envs\py36\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\anaconda2\envs\py36\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "f:\temp\gmusic\env\lib\site-packages\ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "f:\temp\gmusic\env\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "f:\temp\gmusic\env\lib\site-packages\ipykernel\kernelapp.py", line 505, in start
    self.io_loop.start()
  File "f:\temp\gmusic\env\lib\site-packages\tornado\platform\asyncio.py", line 148, in start
    self.asyncio_loop.run_forever()
  File "c:\anaconda2\envs\py36\Lib\asyncio\base_events.py", line 421, in run_forever
    self._run_once()
  File "c:\anaconda2\envs\py36\Lib\asyncio\base_events.py", line 1425, in _run_once
    handle._run()
  File "c:\anaconda2\envs\py36\Lib\asyncio\events.py", line 127, in _run
    self._callback(*self._args)
  File "f:\temp\gmusic\env\lib\site-packages\tornado\ioloop.py", line 690, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "f:\temp\gmusic\env\lib\site-packages\tornado\ioloop.py", line 743, in _run_callback
    ret = callback()
  File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 781, in inner
    self.run()
  File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 742, in run
    yielded = self.gen.send(value)
  File "f:\temp\gmusic\env\lib\site-packages\ipykernel\kernelbase.py", line 357, in process_one
    yield gen.maybe_future(dispatch(*args))
  File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "f:\temp\gmusic\env\lib\site-packages\ipykernel\kernelbase.py", line 267, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "f:\temp\gmusic\env\lib\site-packages\ipykernel\kernelbase.py", line 534, in execute_request
    user_expressions, allow_stdin,
  File "f:\temp\gmusic\env\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "f:\temp\gmusic\env\lib\site-packages\ipykernel\ipkernel.py", line 294, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "f:\temp\gmusic\env\lib\site-packages\ipykernel\zmqshell.py", line 536, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 2848, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 2874, in _run_cell
    return runner(coro)
  File "f:\temp\gmusic\env\lib\site-packages\IPython\core\async_helpers.py", line 67, in _pseudo_sync_runner
    coro.send(None)
  File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 3049, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 3220, in run_ast_nodes
    if (yield from self.run_code(code, result)):
  File "f:\temp\gmusic\env\lib\site-packages\IPython\core\interactiveshell.py", line 3296, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-68-3d98d05bb8f6>", line 1, in <module>
    repr(mm.search('Cat Stevens - Wild World'))
  File "f:\temp\gmusic\env\lib\site-packages\gmusicapi\clients\mobileclient.py", line 1961, in search
    res = self._make_call(mobileclient.Search, query, max_results)
  File "f:\temp\gmusic\env\lib\site-packages\gmusicapi\clients\mobileclient.py", line 48, in _make_call
    return super(Mobileclient, self)._make_call(protocol, *args, **kwargs)
  File "f:\temp\gmusic\env\lib\site-packages\gmusicapi\clients\shared.py", line 90, in _make_call
    return protocol.perform(self.session, self.validate, *args, **kwargs)
  File "f:\temp\gmusic\env\lib\site-packages\gmusicapi\protocol\shared.py", line 247, in perform
    log.debug(cls.filter_response(parsed_response))
Message:

The "message" that follows is an API response which includes the track I searched for:

Message: {'kind': 'sj#searchresponse', 'clusterDetail': [{'cluster': {'type': '5', 'category': '1', 'id': 'search_genre'}, 'resultToken': 'EAU'}, {'cluster': {'type': '2', 'category': '1', 'id': 'search_artist'}, 'resultToken': 'EAI', 'entries': [{'type': '2', 'artist': {'kind': 'sj#artist', 'name': 'Cat Stevens', 'artistArtRef': 'http://lh3.googleusercontent.com/HDs-d4ub0wJ6mlPExcCXACNGQE5ZqAC4sNVgQVZNzo4YMqZ2Kv49cI5O_nXHZjL-e6lmJ1UAgQ', 'artistArtRefs': [{'kind': 'sj#imageRef', 'url': 'http://lh3.googleusercontent.com/HDs-d4ub0wJ6mlPExcCXACNGQE5ZqAC4sNVgQVZNzo4YMqZ2Kv49cI5O_nXHZjL-e6lmJ1UAgQ', 'aspectRatio': '2', 'autogen': False}], 'artistId': 'Avniovlu2t5mvaenz7fu4lyurby', 'artist_bio_attribution': {'kind': 'sj#attribution', 'source_title': 'Wikipedia', 'source_url': 'https://en.wikipedia.org/wiki/Cat_Stevens', 'license_title': 'Creative Commons Attribution CC-BY-SA 4.0', 'license_url': 'http://creativecommons.org/licenses/by-sa/4.0/legalcode'}}, 'cluster': [{'type': '2', 'category': '1', 'id': 'search_artist'}]}]}, {'cluster': {'type': '3', 'category': '1', 'id': 'search_album'}, 'resultToken': 'EAM', 'entries': [{'type': '3', 'album': {'kind': 'sj#album', 'name': 'Tea For The Tillerman', 'albumArtist': 'Cat Stevens', 'albumArtRef': 'http://lh3.googleusercontent.com/HkpCQAGwGohvsDhQbtV4aQ-I1gU1noheQbe2gSXev3MD7eKNhNzdvARhclfLwTBnyvCKsQvnGqI', 'albumId': 'B243eklt72x2akr555zqanwuje4', 'artist': 'Cat Stevens', 'artistId': ['Avniovlu2t5mvaenz7fu4lyurby'], 'description_attribution': {'kind': 'sj#attribution', 'source_title': 'Wikipedia', 'source_url': 'https://en.wikipedia.org/wiki/Tea_for_the_Tillerman', 'license_title': 'Creative Commons Attribution CC-BY-SA 4.0', 'license_url': 'http://creativecommons.org/licenses/by-sa/4.0/legalcode'}, 'year': 1970, 'explicitType': '2'}, 'cluster': [{'type': '3', 'category': '1', 'id': 'search_album'}]}, {'type': '3', 'album': {'kind': 'sj#album', 'name': 'Wild World: Tribute to Cat Stevens', 'albumArtist': 'Music Factory', 'albumArtRef': 'http://lh6.ggpht.com/bthn9-1JIO0AoR-ExsddMTZmWkacSjErx1clK1cM4NV9MTS6I9s7h-D_RqtimxewmKwTqr9RFw', 'albumId': 'B7tytcsvrdm46jq5x37mpigg2ga', 'artist': 'Music Factory', 'artistId': ['Afl5uxfrorf6keqvj5wvjnrcdz4'], 'year': 2014, 'explicitType': '2'}, 'cluster': [{'type': '3', 'category': '1', 'id': 'search_album'}]}, {'type': '3', 'album': {'kind': 'sj#album', 'name': 'Wild World: Tribute To Cat Stevens', 'albumArtist': 'Mystique', 'albumArtRef': 'http://lh5.ggpht.com/29C3AvPKoNr1wGyWomVrIwHJwG0WeuZFNz8cCDry_KlNv-mXtgVen4hqb_CW0ZhxzvkitG046A', 'albumId': 'Bozzyppzpljr7qmnywtjgdf6wne', 'artist': 'Mystique', 'artistId': ['Azfcsuute3rvgblikifebp2hmia'], 'year': 2011, 'explicitType': '2'}, 'cluster': [{'type': '3', 'category': '1', 'id': 'search_album'}]}]}, {'cluster': {'type': '1', 'category': '1', 'id': 'search_track'}, 'resultToken': 'EAE', 'entries': [{'type': '1', 'track': {'kind': 'sj#track', 'title': 'Wild World', 'artist': 'Cat Stevens', 'composer': 'Cat Stevens', 'album': 'Music & Highlights: Just The Best', 'albumArtist': 'Various Artists', 'year': 2012, 'trackNumber': 6, 'genre': 'Pop', 'durationMillis': '202000', 'albumArtRef': [{'kind': 'sj#imageRef', 'url': 'http://lh3.googleusercontent.com/RvOkeUZitLtotzG1nEsvap2E3Ll67hhcpmbtR0HqsjkdyEegCnR0s_LVr73hwsxVa7Y7bHvD', 'aspectRatio': '1', 'autogen': False}], 'discNumber': 1, 'estimatedSize': '8103182', 'trackType': '7', 'storeId': 'Tdl3fiome2imze5pbi75fndj7na', 'albumId': 'm3w6aafisdjmoaa5v7plfa5jji', 'artistId': ['vniovlu2t5mvaenz7fu4lyurby'], 'nid': 'dl3fiome2imze5pbi75fndj7na', 'trackAvailableForSubscription': True, 'trackAvailableForPurchase': True, 'albumAvailableForPurchase': True, 'explicitType': '2'}, 'cluster': [{'type': '1', 'category': '1', 'id': 'search_track'}]}, {'type': '1', 'track': {'kind': 'sj#track', 'title': 'Wild World', 'artist': 'Cat Stevens', 'composer': 'Cat Stevens', 'album': 'Tonight Show: One Special Night', 'albumArtist': 'Various Artists', 'year': 2012, 'trackNumber': 8, 'genre': 'Pop', 'durationMillis': '202000', 'albumArtRef': [{'kind': 'sj#imageRef', 'url': 'http://lh6.ggpht.com/p575huO2Bh4X_pwItAdTFKkS2WA1gY681ekNmxhywUy4PdS_wvKbTWUT3ZXjKuT6k-wuoSf6Dg', 'aspectRatio': '1', 'autogen': False}], 'discNumber': 1, 'estimatedSize': '8107396', 'trackType': '7', 'storeId': 'Tp53n56pa4nnhnslyukav6xhtim', 'albumId': 'udm7grgyx6zrm4k6begkng5pzq', 'artistId': ['vniovlu2t5mvaenz7fu4lyurby'], 'nid': 'p53n56pa4nnhnslyukav6xhtim', 'trackAvailableForSubscription': True, 'trackAvailableForPurchase': True, 'albumAvailableForPurchase': True, 'explicitType': '2'}, 'cluster': [{'type': '1', 'category': '1', 'id': 'search_track'}]}, {'type': '1', 'track': {'kind': 'sj#track', 'title': 'Wild World', 'artist': 'Cat Stevens', 'composer': '', 'album': 'Greatest Hits', 'albumArtist': 'Cat Stevens', 'year': 1975, 'trackNumber': 1, 'genre': 'Singer-Songwriter', 'durationMillis': '202000', 'albumArtRef': [{'kind': 'sj#imageRef', 'url': 'http://lh3.googleusercontent.com/DpSQErA2oKWlblcEdBQhXOjs98SfaUb5AP33zx31i5r9A89nrw1D5lnaTrnyTV-_JQJlpevi9Q', 'aspectRatio': '1', 'autogen': False}], 'discNumber': 1, 'estimatedSize': '8100086', 'trackType': '7', 'storeId': 'Tokn3xyzgx2vgvswz6wccqavdha', 'albumId': '5pdcjv5ai56ol6ke5gvq6llkny', 'artistId': ['vniovlu2t5mvaenz7fu4lyurby'], 'nid': 'okn3xyzgx2vgvswz6wccqavdha', 'trackAvailableForSubscription': True, 'trackAvailableForPurchase': True, 'albumAvailableForPurchase': True, 'explicitType': '2'},

It goes on for awhile with the entire response. Any idea what's happening here?

commented

It's what I described originally.

I still can't reproduce it. It'd be helpful if you could put the full response including the bad character in a gist so I can see it in context.

commented

It's what I described originally.

I still can't reproduce it. It'd be helpful if you could put the full response including the bad character in a gist so I can see it in context.

Here's a gist with the logging error I'm seeing: https://gist.github.com/sheilnaik/51abd72cb523dd37787f8d6d85f61505

Disabling logging with

import logging
import sys
logging.disable(sys.maxsize)

does hide the ugly error message as you mentioned earlier.

commented

Huh, I don't see any troublesome characters in there.

I can give this another shot later with 3.6 (I couldn't reproduce on 2.7).

commented

It was probably luck, but I did reproduce this with 3.6. The trouble seems to be misencoded strings in the names or descriptions of user-created playlists. For example, I was given:

{
  'type': '4',
  'category': '1',
  'id': 'search_playlist'
}]
}, {
  'type': '4',
  'playlist': {
    'kind': 'sj#playlist',
    'name': '\u261403.02.2019\U0001f3d5\U0001f3d5\U0001f3d5Bowie,Tee Set,Kinks,Chuby C.,James Cotton,Beatl.,,Fats D.Rolling,James Cotton,Junior Wells,RodPeter Gr.,Tull,Pink77p\U0001f3d5\U0001f3d5\U0001f3d5',
    'type': 'SHARED',
    'shareToken': 'AMaBXyk3qcnsHNBcqJRJrc9T78NbwlPsOMKlau9f8bjmgjMLitKLkU8WWOmXo34oLN57g1fUraeBKIJlDrRvqVdhpE-vHZCFOg==',
    'description': 'RARE 12,,POP,,Blues,MARCH 2 2019'
  }

Chardet guesses {'encoding': 'Windows-1254', 'confidence': 0.37371958509666425, 'language': 'Turkish'}, but that's not a valid encoding for it afaict. It was probably double encoded or something at some point.

I don't really think there's anything we can do to clean this up if Google is serving it this way, but we can probably change the logging encoding or allow it to drop bad characters.

commented

That seems to work for me. Let me know if you notice the error on the dev version (you can use pip install git+https://github.com/simon-weber/gmusicapi.git@develop#egg=gmusicapi to install it).