FreeTubeApp / yt-comment-scraper

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Multiple different errors in the theme of "cannot read properties of undefined"

edge-loop opened this issue · comments

I'm using the npm version of this, which I think might be slightly different, but maybe the error will be solvable from here (since the issues of the fork don't seem to be available). I keep getting errors when trying to scrape the later pages of some videos (I think it's longer ones but I'm not sure of the correlation.

TypeError: Cannot read properties of undefined (reading 'continuationCommand')
    at getComments ([links n stuff]/@freetube/yt-comment-scraper/src/Youtube-Scraper.js:62:80)

and

data.forEach((node) => {
         ^

TypeError: Cannot read properties of undefined (reading 'forEach')
    at Function.parseCommentData ([links n stuff]/@freetube/yt-comment-scraper/src/htmlParser.js:5:10)
    at getCommentReplies ([links n stuff]/@freetube/yt-comment-scraper/src/Youtube-Scraper.js:80:36)
    at runMicrotasks (<anonymous>)

Basically I'm just trying to scrape all of the video's comments and replies and put them into an array:

const commentScraper = require('@freetube/yt-comment-scraper');
const id = 'foSAi-olfck'; // penguinz0's video on the rush e meme or whatever doesn't really matter

(async function()
{
	var comments = await scrape(commentScraper.getComments, { videoId: id, mustSetCookie: true });

	async function scrape(method, payload)
	{
		console.log('Loading...');

		const data = await method(payload);
		const result = data.comments;
		for(e of data.comments)
			if(e.replyToken !== null)
				result.push(...await scrape(commentScraper.getCommentReplies, { videoId: id, mustSetCookie: true, replyToken: e.replyToken }));
		if(data.continuation !== null)
			result.push(...await scrape(commentScraper.getComments, { videoId: id, mustSetCookie: true, continuation: data.continuation }));
		return result;
	}

	console.log(comments.length);
})();

I tried switching mustSetCookie: out with just setCookie: since I saw both but neither did anything
Is this a problem with my code or something else?
I'm also using Replit which is sometimes weird about things like this