[Bug] There are duplicate, non-existent mods.

[balu92]

Describe the bug

Some mods have a non-existent "beginner" and "expert" version

NodeJS Version

16.5.0

npm Version

6.14.13

warframe-items Version

v1.1250.16

Access Method

github download

What happened?

Example taken from the .json files on github. Cut out some bits for easier readability.
From stats it looks like as if they were Flawed and Primed versions of the base mods, but there is only 1 Concussion rounds in the game.

    {
    "baseDrain": 2,
    "category": "Mods",
    "compatName": "PISTOL",
    "drops": [ ... ],
    "fusionLimit": 3,
    "imageName": "concussion-rounds.jpg",
    "introduced": { },
    "levelStats": [
      {
        "stats": [ "+8% <DT_IMPACT>Impact" ]
      },
       ...
      {
        "stats": [ "+32% <DT_IMPACT>Impact" ]
      }
    ],
    "name": "Concussion Rounds",
    "uniqueName": "/Lotus/Upgrades/Mods/Pistol/Beginner/WeaponImpactDamageModBeginner",
    "wikiaThumbnail": "https://static.wikia.nocookie.net/warframe/images/b/b0/ConcussionRoundsModU145.png/revision/latest?cb=20171007145338",
    "wikiaUrl": "http://warframe.fandom.com/wiki/Concussion_Rounds"
  },
  {
    "baseDrain": 2,
    "category": "Mods",
    "compatName": "PISTOL",
    "drops": [ ... ],
    "fusionLimit": 10,
    "imageName": "concussion-rounds.jpg",
    "introduced": { ... },
    "levelStats": [
      {
        "stats": [ "+10% <DT_IMPACT>Impact" ]
      },
      ...
      {
        "stats": [ "+110% <DT_IMPACT>Impact" ]
      }
    ],
    "name": "Concussion Rounds",
    "uniqueName": "/Lotus/Upgrades/Mods/Pistol/Expert/WeaponImpactDamageModExpert",
    "wikiaThumbnail": "https://static.wikia.nocookie.net/warframe/images/b/b0/ConcussionRoundsModU145.png/revision/latest?cb=20171007145338",
    "wikiaUrl": "http://warframe.fandom.com/wiki/Concussion_Rounds"
  },
  {
    "baseDrain": 2,
    "category": "Mods",
    "compatName": "PISTOL",
    "drops": [ ... ],
    "fusionLimit": 5,
    "imageName": "concussion-rounds.jpg",
    "introduced": { ... },
    "levelStats": [
      {
        "stats": [ "+10% <DT_IMPACT>Impact" ]
      },
      ...
      {
        "stats": [ "+60% <DT_IMPACT>Impact" ]
      }
    ],
    "name": "Concussion Rounds",
    "uniqueName": "/Lotus/Upgrades/Mods/Pistol/WeaponImpactDamageMod",
    "wikiaThumbnail": "https://static.wikia.nocookie.net/warframe/images/b/b0/ConcussionRoundsModU145.png/revision/latest?cb=20171007145338",
    "wikiaUrl": "http://warframe.fandom.com/wiki/Concussion_Rounds"
  },

[Matt]

sorry, i'm not gonna randomly delete stuff from the data...

[balu92]

No need to randomly delete stuff, it's likely a problem with the scraper or something as they aren't present on the: wikia.

[Matt]

You're not getting me. Unless I actively exclude, these are JSON objects directly from DE's JSON exports, not something we scrape.

[balu92]

Can't you just crossreference check the uniqueNames between the wikia page and the mobile endpoint? If it's not on the wikia, just remove it.

[Matt]

no, because we don't crosscheck the wikia page to verify DE's data. That's backwards.