Disane87 / docudigger

Website scraper for getting invoices automagically as pdf (useful for taxes or DMS)

Home Page:https://blog.disane.dev

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[WR] meine.new-energie.de/portal

Disane87 opened this issue · comments

Which website do you want to crawl?

https://meine.new-energie.de/portal/

What services do they provide?

Contracts (with recurring calculation in a defined interval)

In which industry does the company operate?

Energy (i.e. your local energy company)

If you choose other...

No response

Describe the company

It's a local energy company in germany which provides services for energy, gas, water, sewage etc.

Does the webpage have/needs an authentication?

  • Yes, the website needs an authentication

Do they provide a two factor auth?

  • Yes, the website needs an seconds factor

Would you provide us the credentials (privately)?

  • Yes, I would share my credentials

Are you willing to collaborate to get this scraper up and running?

  • Yes, I would collaborate on this actively.

What color (hex) represents the company?

#9b004b

Your recorded code

import url from 'url';
import { createRunner } from '@puppeteer/replay';

export async function run(extension) {
    const runner = await createRunner(extension);

    await runner.runBeforeAllSteps();

    await runner.runStep({
        type: 'setViewport',
        width: 1134,
        height: 1227,
        deviceScaleFactor: 1,
        isMobile: false,
        hasTouch: false,
        isLandscape: false
    });
    await runner.runStep({
        type: 'navigate',
        url: 'https://www.vodafone.de/',
        assertedEvents: [
            {
                type: 'navigation',
                url: 'https://www.vodafone.de/',
                title: 'Vodafone.de | Mobilfunk, Handys & Internet-Anbieter'
            }
        ]
    });
    await runner.runStep({
        type: 'navigate',
        url: 'https://meine.new-energie.de/portal/dashboard',
        assertedEvents: [
            {
                type: 'navigation',
                url: 'https://meine.new-energie.de/portal/dashboard',
                title: ''
            }
        ]
    });
    await runner.runStep({
        type: 'click',
        target: 'main',
        selectors: [
            [
                'aria/ZUSTIMMEN'
            ],
            [
                '#login-new-dialog-privacy-btn-confirm'
            ],
            [
                'xpath///*[@id="login-new-dialog-privacy-btn-confirm"]'
            ],
            [
                'pierce/#login-new-dialog-privacy-btn-confirm'
            ]
        ],
        offsetY: 26.5,
        offsetX: 159.25,
    });
    await runner.runStep({
        type: 'click',
        target: 'main',
        selectors: [
            [
                'aria/E-Mail'
            ],
            [
                '#input-30'
            ],
            [
                'xpath///*[@id="input-30"]'
            ],
            [
                'pierce/#input-30'
            ]
        ],
        offsetY: 5.234375,
        offsetX: 163.5,
    });
    await runner.runStep({
        type: 'click',
        target: 'main',
        selectors: [
            [
                'aria/ANMELDEN',
                'aria/[role="generic"]'
            ],
            [
                'form > div.row span'
            ],
            [
                'xpath///*[@id="router-view"]/div[4]/div/div/div[2]/div/div[1]/div/form/div[3]/div/button/span'
            ],
            [
                'pierce/form > div.row span'
            ],
            [
                'text/Anmelden'
            ]
        ],
        offsetY: 12.734375,
        offsetX: 161.5,
        assertedEvents: [
            {
                type: 'navigation',
                url: 'https://login.new.de/post/?code=234cffe92b81d0c84d8ca88f2a5b3aff22b46871f233e41a90082dc738642fac-072c06e616474cb999fa&state=bpc-%2Fdashboard&redirect_uri=https%3A%2F%2Fmeine.new-energie.de%2Fportal%2Fapi%2Foidc',
                title: ''
            }
        ]
    });
    await runner.runStep({
        type: 'click',
        target: 'main',
        selectors: [
            [
                'aria/Rechnungen'
            ],
            [
                'li:nth-of-type(3) li:nth-of-type(1) > a'
            ],
            [
                'xpath///*[@id="mainMenu"]/div[1]/ul[2]/li[3]/ul/li[1]/a'
            ],
            [
                'pierce/li:nth-of-type(3) li:nth-of-type(1) > a'
            ]
        ],
        offsetY: 10,
        offsetX: 72.6875,
    });
    await runner.runStep({
        type: 'click',
        target: 'main',
        selectors: [
            [
                '#InviteLayerRealPersonname'
            ],
            [
                'xpath///*[@id="InviteLayerRealPersonname"]'
            ],
            [
                'pierce/#InviteLayerRealPersonname'
            ]
        ],
        offsetY: 38,
        offsetX: 585.5,
    });
    await runner.runStep({
        type: 'click',
        target: 'main',
        selectors: [
            [
                'tr:nth-of-type(1) span > span'
            ],
            [
                'xpath//html/body/div[2]/div/div/div/main/div[3]/div/div/ng-transclude/div/div[1]/div/div/ng-transclude/div/div/div/ng-include/div/div/div/div[1]/div/table/tbody/tr[1]/td[3]/bpc-document-download/bpc-download/a/ng-transclude/span/span'
            ],
            [
                'pierce/tr:nth-of-type(1) span > span'
            ]
        ],
        offsetY: 2.40625,
        offsetX: 6.5,
    });

    await runner.runAfterAllSteps();
}

if (process && import.meta.url === url.pathToFileURL(process.argv[1]).href) {
    run()
}

Paste your screenshots here

image
image

Paste your html here

No response