Cannot get the same result as the webpage
jmz1996 opened this issue · comments
I use interest_over_time() can not get the same result as the webpage, I notice the webpage's headers['req'] is diffrent from the requests, i change it as the webpage's, but still cannot get the same result, what should i do?
the webpage's headers['req'] is in below, Some do not seem to exist before? Is this the reason?
req: {"time":"2004-01-01 2022-11-17","resolution":"MONTH","locale":"zh-CN","comparisonItem":[{"geo":{"country":"BR"},"complexKeywordsRestriction":{"keyword":[{"type":"ENTITY","value":"/m/01hpbc"}]}},{"geo":{"country":"BR"},"complexKeywordsRestriction":{"keyword":[{"type":"ENTITY","value":"/g/11dymw9wxl"}]}}],"requestOptions":{"property":"","backend":"IZG","category":0},"userConfig":{"userType":"USER_TYPE_LEGIT_USER"}}
This problem seems to have been raised quite a few times (not just in the context of pytrends), and might be related to accessing different endpoints (Google Servers) which have a different sample of the data.
See, e.g.:
- #473
- #524
- #534
- PMassicotte/gtrendsR#428
- https://link.springer.com/chapter/10.1007/978-3-030-65965-3_25 ("In this paper, we develop an experimental setup to estimate and measure possible variation in service results for the example of Google Trends. Our work demonstrates that the inconsistencies in Google Trends Data and the resulting contradictions in analyses and predictions are systematic and particularly large when analyzing timespans of eights months or less")
- PMassicotte/gtrendsR#427
This problem seems to have been raised quite a few times (not just in the context of pytrends), and might be related to accessing different endpoints (Google Servers) which have a different sample of the data.
See, e.g.:
- Interest Over Time returns different values if run on hosted server #473
- interest_over_time() #524
- What is the API doing? #534
- Inconsistent results when scraping the same timerange, place, keywords multiple times PMassicotte/gtrendsR#428
- https://link.springer.com/chapter/10.1007/978-3-030-65965-3_25 ("In this paper, we develop an experimental setup to estimate and measure possible variation in service results for the example of Google Trends. Our work demonstrates that the inconsistencies in Google Trends Data and the resulting contradictions in analyses and predictions are systematic and particularly large when analyzing timespans of eights months or less")
- gtrendsR returning different data when compared to Google Trends website PMassicotte/gtrendsR#427
I read every information above, that is to say, there is no solution right now?
I have investigated this myself and it seems that Google Trends identifies you as a scraper user and when it does it returns wrong / fake data. I believe it's called Cloaking and is an attempt by Google to prevent automatic retrieval of data from their service.
After comparing the data, I believe that the reason is because the data returned by Google Trends is for making charts. Its y-axis always tries to scale to 100, so if a single request yields one result, requesting it along with other keywords may cause changes. One solution is to find a standard and send two keywords every time we make a request - one that we care about and another as the standard keyword. If this standard keyword always has a higher peak or its peak can be prioritized to scale to 100, then the data we want to request will be relative to this standard. By always requesting it together with the standard keyword, we can compare other data in the end.
However, establishing this standard keyword is somewhat tricky; it should be the one with highest peak value and requires repeated verification before confirmation. Moreover, if this standard is set too high, it may cause all other requested results of keywords to be zero. So caution must be taken 😂.
The current approach I am using is to employ a standard query data, along with all other query terms, for comparison. However, it is important to note that this standard should not result in all-zero queries.