everywall / ladder

Selfhosted alternative to 12ft.io. and 1ft.io bypass paywalls with a proxy ladder and remove CORS headers from any URL

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

unsupported protocol scheme ""

al3xandru opened this issue · comments

Using an article from New Yorker, I am noticing in the terminal many errors reported in the form:

2023/11/06 16:08:36 GET /verso/static/assets/fonts/TNYAdobeCaslonPro-SemiBoldItalic.woff2
2023/11/06 16:08:36 ERROR: Get "verso/static/assets/fonts/TNYAdobeCaslonPro-SemiBoldItalic.woff2": unsupported protocol scheme ""

I haven't looked at the code yet and wondering what the error means.

The HTML links /verso/static/assets/fonts/TNYAdobeCaslonPro-SemiBoldItalic.woff2 relative to the domain. The ladder usually tries to rewrite this to by adding /https://www.yourdomain.com to the src Tag.

/https://www.yourdomain.com/verso/static/assets/fonts/TNYAdobeCaslonPro-SemiBoldItalic.woff2

you can reproduce this error by opening http://localhost:8080/something

I should really catch this somewhere and print a more obvious error message.

Even better than a better error message, can ladder take care of resolving it correctly?

I'm also seeing this error but when using passing an encoded URL

Error:

https://ladder.example.com/https%3A%2F%2Fwww.latimes.com%2Fcalifornia%2Fstory%2F2023-11-06%2Fdisneyland-workers-salary-bump-living-wage-legal-battle

Working:

https://ladder.example.com/https://www.latimes.com/california/story/2023-11-06/disneyland-workers-salary-bump-living-wage-legal-battle

I think it would be valuable being able to process an encoded URL so that it can work with bookmark tools that only are able to pass in encoded URLs. For example, I use a tool called Raycast that I can use to make a quick shortcut to access any URL via Ladder like so:
CleanShot 2023-11-06 at 08 25 22@2x

P.S. Great tool, always thought 12ft should've been a self-hosted open source solution. Thank you.

Same issue for theatlantic.com as well, specifically /api/user_country.
2023/11/06 22:11:38 ERROR: Get "user_country": unsupported protocol scheme ""

and for several other website as well.
Medium: 2023/11/06 22:15:48 ERROR: Get "cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1?ray=8220b27e4ea53a6c": unsupported protocol scheme ""

I'm running into same situation as @axsuul, after setting up ladder as a search engine in Chrome. When I enter the URL I'd like to browse as the search query, Chrome will encode just the protocol portion, which then causes ladder to return this error.

Submitted PR that resolves this issue.

Many broken images in all sorts of sites are also fixed now.

@some-guy-23 @axsuul

Can you try again with your respective external tools with the latest version and see if it works now?


@T0BiD @al3xandru unsupported protocol scheme error should be resolved now too, can you test?

@deoxykev Can confirm it works now, thanks!

I tried to build it locally but got a panic:

panic: load embedded ruleguard rules: rules/rules.go:13: can't load fmt

goroutine 1 [running]:
github.com/go-critic/go-critic/checkers.init.22()
	/Users/alex/go/pkg/mod/github.com/go-critic/go-critic@v0.6.2/checkers/embedded_rules.go:46 +0x494
make: *** [lint] Error 2

I have tried 2 urls and I don't see that error in the logs anymore. On the other hand https://www.nytimes.com/2023/11/14/arts/music/henry-diltz.html is not loading

That page seems to be working ok for me, except with /service-worker.js paths for some reason.

Still, the page, css and images load just fine. What version are you running?

Can you try a fresh git pull and build / run a new copy? RULESET="ruleset.yaml" go run cmd/main.go

@deoxykev works for custom Chrome search engine as well, thank you! :)

@deoxykev with

Can you try a fresh git pull and build / run a new copy? RULESET="ruleset.yaml" go run cmd/main.go

First log lines:

2023/11/15 16:32:25 GET /apple-touch-icon-precomposed.png
2023/11/15 16:32:25 ERROR: Get "apple-touch-icon-precomposed.png": unsupported protocol scheme ""
2023/11/15 16:32:25 GET /apple-touch-icon.png
2023/11/15 16:32:25 ERROR: Get "apple-touch-icon.png": unsupported protocol scheme ""

Then for the NY Times link, same behavior: it's stuck loading. Maybe I could try with some other links?

@al3xandru

I can't seem to reproduce with the latest build on that nytimes article. Are you super certain it's the latest version?

Here's the logs i'm getting for https://myladder.com/https://www.nytimes.com/2023/11/14/arts/music/henry-diltz.html

2023/11/15 22:34:57 GET /https://www.nytimes.com/2023/11/14/arts/music/henry-diltz.html
2023/11/15 22:34:57 https://www.nytimes.com/2023/11/14/arts/music/henry-diltz.html
2023/11/15 22:34:57 GET /https://www.nytimes.com/vi-assets/static-assets/global-f449cfd9976ad673ef2b7ab5098b85be.css
2023/11/15 22:34:57 https://www.nytimes.com/vi-assets/static-assets/global-f449cfd9976ad673ef2b7ab5098b85be.css
2023/11/15 22:34:57 GET /https://www.nytimes.com/2023/11/14/arts/music/runtime~adslot-9ba87c41636f20977981.js.map
2023/11/15 22:34:57 https://www.nytimes.com/2023/11/14/arts/music/runtime~adslot-9ba87c41636f20977981.js.map
2023/11/15 22:34:57 GET /https://www.nytimes.com/2023/11/14/arts/music/runtime~main-69ab52f60167a16d3f4b.js.map
2023/11/15 22:34:57 https://www.nytimes.com/2023/11/14/arts/music/runtime~main-69ab52f60167a16d3f4b.js.map
2023/11/15 22:34:58 GET /https://www.nytimes.com/vi-assets/static-assets/favicon-d2483f10ef688e6f89e23806b9700298.ico
2023/11/15 22:34:58 https://www.nytimes.com/vi-assets/static-assets/favicon-d2483f10ef688e6f89e23806b9700298.ico

Some of the assets, such as the carousel and video aren't loading, but the article and beginning images are. The lazy loaded images at the bottom are broken however.

@deoxykev I did pull again and retried and the behavior is the same. Interestingly, I killed ladder and then the page showed up. I have to admit that I'm not sure what could explain this (probably something that tells the browser that the HTML cannot yet be rendered)

Update: it's a css file:

image

Was it cache then? Does it work in a fresh incognito browser?

commented

Hi! I know this issue is closed and it is supposedly fixed, but I only want to make you know that connecting to the web pixelscan.net I get the following log error:

ladder    | 2024/01/01 21:02:53 GET /https://nitter.net/reach_vb/status/1740081386218737966
ladder    | 2024/01/01 21:04:00 GET /https://nitter.net/reach_vb/status/1740081386218737966
ladder    | 2024/01/01 21:21:09 GET /https://pixelscan.net/
ladder    | 2024/01/01 21:21:10 GET /https://pixelscan.net/styles.b28bb2795d9e961d.css
ladder    | 2024/01/01 21:21:10 GET /https://pixelscan.net/assets/ua-parser.min.js
ladder    | 2024/01/01 21:21:10 GET /https://pixelscan.net/assets/fptc.min.js
ladder    | 2024/01/01 21:21:10 GET /https://pixelscan.net/runtime.94e068ef2695d55a.js
ladder    | 2024/01/01 21:21:10 GET /https://pixelscan.net/polyfills.7e16bdbffcf7884a.js
ladder    | 2024/01/01 21:21:10 GET /https://pixelscan.net/scripts.72bafbcf77b419bd.js
ladder    | 2024/01/01 21:21:10 GET /https://pixelscan.net/main.1e79559b2084ed40.js
ladder    | 2024/01/01 21:21:10 GET /https://pixelscan.net/assets/fonts/Inter-Regular.woff2
ladder    | 2024/01/01 21:21:11 GET /https://pixelscan.net/980.8c00badc42c206b3.js
ladder    | 2024/01/01 21:21:11 GET /https://pixelscan.net/common.e4ef9f69d4315943.js
ladder    | 2024/01/01 21:21:11 GET /https://pixelscan.net/859.fc31b9b9a531d311.js
ladder    | 2024/01/01 21:21:11 GET /assets/sttfpage.html
ladder    | 2024/01/01 21:21:12 GET /https://pixelscan.net/assets/icons/logo.svg
ladder    | 2024/01/01 21:21:12 GET /https://pixelscan.net/assets/icons/logo-gray.svg
ladder    | 2024/01/01 21:21:12 GET /https://pixelscan.net/assets/icons/refresh.svg
ladder    | 2024/01/01 21:21:12 GET /https://pixelscan.net/assets/icons/verified.svg
ladder    | 2024/01/01 21:21:12 GET /https://pixelscan.net/assets/fonts/Inter-SemiBold.woff2
ladder    | 2024/01/01 21:21:12 GET /https://pixelscan.net/assets/fonts/Inter-Medium.woff2
ladder    | 2024/01/01 21:21:12 GET /assets/sttfpage.js
ladder    | 2024/01/01 21:21:12 ERROR: Get "assets/sttfpage.js": unsupported protocol scheme ""

Thanks!