Populate some user info from the search output to avoid unnecessary profile requests

[mconigliaro]

It looks like some user info is already available during a search (i.e. username, age, location, match_percentage, enemy_percentage) without making a request for the whole profile, but I didn't see a way to prevent the profile request. For example, in the code below...

profiles = user.search(...)
for p in profiles:

It looks like by the time p is assigned, the profile request has already been made. It would be nice if some of the profile data was populated from the search output and the rest was loaded on demand. That way, your script could keep a list of recently visited users and avoid requesting those profiles again so you don't end up looking like a stalker. Haha. Of course, #31 would help with this too.

[Ivan Malison]

There is actually some fanciness going on with the way attributes are retrieved on profile objects -- The profile request is not made when the profile object is created but when any attribute that requires the profile to be fetched is requested. This happens through the profile_tree/_profile_response cached_properties which at least prevents multiple requests from being made.

If you look back far enough in the commit history (see ce473a2 for example, you will see that okcupyd actually used to populate some of the profile info in exactly the way you are suggesting. At some point I decided to remove this in favor of the simpler approach of getting all of information from the profile page.
Your suggestion has obvious merits (fewer requests, not looking like a creep), but actually implementing what you want might be a bit more complicated than you imagine. You see, Profile objects can be created in many places, and it is important that
a) Profile objects continue to work away from search
and
b) The only parameters that are ever required for profile creation are a logged in session and a username

What needs to be done is
a) Make it so that Profile accepts keyword arguments to eagerly populate its variables
b) Use a slightly more sophisticated Processor object in the SearchFetchable function that passes these keyword arguments to the Profile constructors

Actually, as I mentioned, search used to do exactly what you wanted, so using the following code:

class MatchCardExtractor(object):

    def __init__(self, div):
        self._div = div

    @property
    def username(self):
        return xpb.div.with_class('username').get_text_(self._div).strip()

    @property
    def age(self):
        return int(xpb.span.with_class('age').get_text_(self._div))

    @property
    def location(self):
        return helpers.replace_chars(
            xpb.span.with_class('location').get_text_(self._div)
        )

    _match_percentage_xpb = xpb.div.with_classes('percentage_wrapper', 'match').span.with_classes('percentage')

    @property
    def match_percentage(self):
        try:
            return int(self._match_percentage_xpb.get_text_(self._div).strip('%'))
        except:
            return 0

    _enemy_percentage_xpb = xpb.div.with_classes('percentage_wrapper', 'enemy').span.with_classes('percentage')

    @property
    def enemy_percentage(self):
        try:
            return int(self._enemy_percentage_xpb.get_text_(self._div).strip('%'))
        except ValueError:
            return 0

    @property
    def contacted(self):
        return bool(xpb.div.with_class('fancydate').apply_(self._div))

    @property
    def as_dict(self):
        return {
            'username': self.username,
            'age': self.age,
            'location': self.location,
            'match_percentage': self.match_percentage,
            'enemy_percentage': self.enemy_percentage,
            'id': self.id,
            'rating': self.rating,
            'contacted': self.contacted
        }

b) might just end up looking like

    session = session or Session.login()
    def build_profile_from_match_card_div(match_card_div):
        match_card_extractor = MatchCardExtractor(match_card_div)
        return Profile(session, match_card_extractor.username, **match_card_extractor.as_dict)
    return util.Fetchable.fetch_marshall(
        SearchHTMLFetcher(session, **kwargs),
        util.SimpleProcessor(
            session,
            build_profile_from_match_card_div,
            xpb.div.with_classes('match_card')
        )
    )

a) is also pretty easy... basically just take **kwargs and shove that shit into self.dict in the constructor of Profile (obviously some validition and stuff would be nice).

Do you have any interest in trying to add this functionality? It actually seems pretty easy to do and I'd really appreciate a pull request.

[Ivan Malison]

Can you help me test this? New code is at https://github.com/IvanMalison/okcupyd/tree/populate_profile_from_search

[mconigliaro]

Oh, you implemented it already? I'll give it a try when I have a free moment.

[Ivan Malison]

This is fixed in d84655e and v 0.8.15