Merge duplicate items

Question

Merge duplicate items

nileshtrivedi opened this issue a year ago · comments

Nilesh commented a year ago

AbdulHafeezGalib · Answer 1 · Sat Mar 04 2023 16:29:43 GMT+0800 (China Standard Time)

Hi, Can you please elaborate on this issue with an example

Nilesh · Answer 2 · Sat Mar 04 2023 16:35:08 GMT+0800 (China Standard Time)

@Abdul-Hafeez-Galib When I search for "Sapiens", 3 results are shown:

This is because 3 items exists in db/items.json for this. We allow one item to have multiple links. That's not the problem. But this is an example of there being duplicates in dataset.

We need to create a standalone NodeJS script scripts/remove_duplicates.js that updates db/items.json by merging duplicate items into one. It should have a --dry-run option because ideally, we don't want to remove items that have been linked at external sites.

AbdulHafeezGalib · Answer 3 · Sat Mar 04 2023 16:43:29 GMT+0800 (China Standard Time)

So, I should merge these duplicates by adding the multiple links for a single book.

Nilesh · Answer 4 · Sat Mar 04 2023 16:49:15 GMT+0800 (China Standard Time)

item is an abstract thing: the set of ideas given in Sapiens. Each item can have multiple concrete representations (links), for eg: book, video, summary, podcast, course etc. So, yes, you need to merge all the fields like description, links, reviews, tags into a single item and delete the duplicate items.