SavchenkoValeriy / emacs-powerthesaurus

Powerthesaurus integration for Emacs

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

HTTP 403 response from powerthesaurus.org

magne-hov opened this issue · comments

I've started receiving HTTP 403 errors for the requests that https://github.com/SavchenkoValeriy/emacs-powerthesaurus performs.

F.ex for M-x powerthesaurus-lookup-synonyms-dwim RET the RET I get the following messages:

[error] request--callback: peculiar error: 403
[error] request-default-error-callback: https://www.powerthesaurus.org/the/synonyms error

This reproduces with GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0):

(load-file "/home/mhov/.emacs.d/straight/repos/emacs-request/request.el")
(load-file "/home/mhov/.emacs.d/straight/build/s/s.el")
(load-file "/home/mhov/.emacs.d/straight/repos/emacs-powerthesaurus/powerthesaurus.el")

I'd be interested to know if others see the same thing.

When doing similar GET queries with curl I also get HTTP 403, and the following

HTML content
<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="https://github.com/cdn-cgi/styles/challenges.css" target="_blank" rel="nofollow" rel="stylesheet">
    

</head>
<body class="no-js">
    <div class="main-wrapper" role="main">
    <div class="main-content">
        <h1 class="zone-name-title h1">
            <img class="heading-favicon" src="/favicon.ico"
                 onerror="this.onerror=null;this.parentNode.removeChild(this)">
            www.powerthesaurus.org
        </h1>
        <h2 class="h2" id="challenge-running">
            Checking if the site connection is secure
        </h2>
        <noscript>
            <div id="challenge-error-title">
                <div class="h2">
                    <span class="icon-wrapper">
                        <div class="heading-icon warning-icon"></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=781b532c0ebe7330')"></div>
        <div id="challenge-body-text" class="core-msg spacer">
            www.powerthesaurus.org needs to review the security of your connection before proceeding.
        </div>
        <form id="challenge-form" action="/the/synonyms?__cf_chl_f_tk=kla4GTZ1LIFWYgVpbnZpiakOJ9Jp5NOzUaMGg0kfxH4-1672409069-0-gaNycGzNB1E" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="ZN6c0nyFS.LPI.t4B8GPGLeRNqsEDInLlMWviVVZUxw-1672409069-0-AXIcfce_rx7Lp-gMk8n4auYF6LHMOY7vpBdMiWqAZUeRnEFf1WmmGg5ac0Y9lvfrZe7lHZzLWzMHwt6aPwRR2VpCIelqkOeXMQl5PVxM4cd9oqZLUTzrIB-3KxqTeKYOk_kgZlYu1ho0rOqPdi1K7csgaGqQ3dctT789QNt8uTdAjwUVGxnhJ_Os5zWdo2Z5ZFLsBn7utKLku61i3nDnbNx3OddkRMqELUlF5uMC6RQBotiQFz0Fq8XJDyUh9y5UKKqCr980usN2QyJZaX74okfPlUn7lDkkie0baI1UTRfragvRjxqAUQcdK4A-U_TFLjt6bDRDr7M0lxOaDnE0PUcdGVYf2pP5rkYmvcArl-F_lG8hmetwG6Hpt0lKwMczZnpHZuKcgMdzUQ0c97vyrguUFv2k9h2geuAPkRfIoWif9dXNY_o1HjOvZW68WZ88SMtNn2i-yT2JmfLXihT_Taa4SCLrJ_Vm4yPQHseI3ddHNAQfzkr0meN4MnuzGP3DobXSu6OmKIBQ6D6JQrb2nufCpusL0rvMi8g-E5ZjmZnzYVRpX3ElBvjGK7BFzmKxTne5zYRMJ4K7z95HaN4Yh1kD1hidnfeySno7-oLDFjI5_DhL8qjqmPlE_wRd0UVr1VTDoN7wRWu9V53BOSPbQfu4U5KKdh7XGOAp8CJ2D3BbLvjSouAqKzEg404uVAslzIq4t-QZ4Zn3hJBcTvm3vCUy3ww456iidWOknxR0AuDG">
            <input type="hidden" name="r" value="XTsS9Ud52uDXcmnpl17K9x7LUsPClkCr1Sc_kUhFrTE-1672409069-0-AUVu0aaSU6SUlla3xhsGg36zaIzaITE6GkRt7y+MxpHCD4cxX4Fmd5zDL8gCsKHNE8u9lGs6QHJ38y7Ur9DJUjuqfViyAFYOzPoOics8JYTxN3RboXdGp+bQ/OACSclxDyYzOfAMw4169fEqM2uVVMyLL3S6eex72sWhyMTN7/TS3mxoXkGPytGME4ya4GkQ8Pynv9A7TXFQF/Qgg4uRplDIF8cUXCRG3KdAfEEUyhJLpinssr/goGSzRsB9eJZChiQMCwWRaJK2agQPOAfQF/w3oeP1diN41umxfcaPMJz2yfe+gUsQm0E/le67P5nvn98mLMMH5qpp0j1CvTLrl45r2i5RikKwHUwfQgXJvUUI4jq69uUGcCjEsB8Z3znJMY52rg6tJ4Bzph9MLA92FcBuIIaOnIoKqIPsi+GtPr7ww333uu+SORIUb3HrV0+NtfeGvKGelRgYSGLtnSrXDBG2YgcWNMHi7UTRoWj7689RnVmuHJoUm6A/DatPEjUxJC7Ymj2bdwUEnv+fOcBGVKgCOrFvhwd0NuT7b7l+uYvAPKPvtyCTZKQYXseGkFBL4K1fujWHOQLJLdccyjShaC9UTKCiHYA2URuoDRA+ijlYos5YluOXQWtWL0M8Wmy2ysJ10v3Y8rrElNcTwKtq7J8RJvDQS5kgnbgtyTTLaqDdvz4WGdHwpQD8vV5R4Uu5V2qYEgQ2gRrjaLfRVI/n/5S+L9/7fCTihkdiJNVFJhxVBGnPBn4eggJSMZ1rqdKbq38pyb0pewPM3bJqwfUKsiqN65JdTL2DYH3HhbMCmdfhI6NPBibuTxWemQQ9EOjsZUXfU9Xf8hsRblv/OgRILn8pXSEl/mSByPMMhkHUNROnK1yGbP7rcqrcv8JvnNHlA/xTsmvWQ2XeOINUqVHOfP1njTo5uSbB1bkrD/aWdbSwhl1IuVHMBHDi76DGVeElsYiTNEG1Vuo8OTMv3SpzVo/YgIdhDthr36Wo/Qt4RlmqJiM4cfriDlhkFGVJlEHmCHaZe6dPc5PmqQ4oe7+cVqhknf0gTCKOz3bAmiv+51hYoSTewrlJqhPlGbWgPuCfi2/lLH6+9QZMEqxdeu90mZmtYzE2iNeh6YzVLx840LElkHNStlhLpqcxi2SxrV6tSZmPA0lNK6apI7Gg2qanc2HYrwLKpy2LzNuVHcbaSBYWGKVkHNNbHFBees2ox/TAc8U48uXF+zsSiS0/C6qZusBVaToxir3GeZk1cMlEMnjWH4EyKmL6JYl0O92oAeD2AeXMXtCdqm/WDunrXGTztOz0EJW7NkU7264ssJnRqXF6SHGcWpe+fU1DBuOG4KspEinXu9NFpde4tTePS9dD+Dml84DZKki+ObB4ngfbbpho8dNrqQQXwzndAiJQW6MsYIURvO/N2Y0x6ueVhx+65VQ9lLOD1YJJPifB3PNWvDXr8YIgZo3RPLUshu66hhpH6kSTUB4FQ17bjVcMjVaKEWGuPHUr34avYxgUK6CxFR8M5wdSn4hLUOD6YRTcUExEkPcJesoK0cQ7kfnTyfKd+hUaaEtiOKEX6dJxVEqcwJFtUEUd/MuG6dfEaoheIdR9NYVEG9O0prz/jUiovFhFU1w4VumKX6JEJBtbPxtxM0xkvNKGKG0NgSb2N+mRUSrkeOfz/JXtzdnTAThRV43z/x87mwx220YblqGKBePChhF3OS9lACQqjm+Uoiv/8rYkHMkb5Jv61rzl1wIVcv3Cz1Aeyzgvi9UxjY9oW/T39DpcQK3PgiMN3MaE7CCy46nRxZ1+L1RmyHEj3F4+SumJ1VQ=">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cType: 'managed',
            cNounce: '9645',
            cRay: '781b532c0ebe7330',
            cHash: 'dbc0578127614d4',
            cUPMDTk: "\/the\/synonyms?__cf_chl_tk=kla4GTZ1LIFWYgVpbnZpiakOJ9Jp5NOzUaMGg0kfxH4-1672409069-0-gaNycGzNB1E",
            cFPWv: 'g',
            cTTimeMs: '1000',
            cTplV: 4,
            cTplB: 'cf',
            cRq: {
                ru: 'aHR0cHM6Ly93d3cucG93ZXJ0aGVzYXVydXMub3JnL3RoZS9zeW5vbnltcw==',
                ra: 'Y3VybC83LjY4LjA=',
                rm: 'R0VU',
                d: '6M2QaV4M49HxGSpAWrfkG2ncILwS86cUincJ5WxXFY3W3VmHTGY5gjo3vmjNIQDTf75UFbFY2PzE7DBbAsxIhmd2t4XmbLp9Uekrrz85jhfYnMmBAcAZyymw8EvjBFB3pev8ItCFeEpkqnvSycOmO6Z7ICn8W5adM6AFRB1bdM2ZEuQgxpXgN8zy54YWRJEE2Utl5KMcjC5lzn5f+bc94dcQA/7E7879kci3OSzSzm4mP2wXoNDWI1PSTfB7/o9hrQGe9nOJK1ejfRwMxYrI4TXvyQnm5wP9nlP6NmWEhUGtmlubdiaH9dD1WiyzpjJjqOM6WMmyev1rtfxKytYw+okXE8mm+smlaXV/HEridXs9zZBIKHsIvwTVBX6iC0b3gA7n1kX0u+bhdwSYyjROdtpEhB9O6fJ8QwXGnAkjw6YI9Hn/EMx8xl/Gn/eTVmwSde8lc5Y00jNTnPkVhzJ1MIMp6NVuqK/UumreFakBveO8c8Mb19UBpUR3tSG7OCX3WohpQZFAJwUDcaQYCY57Ni/YqsxAvpGuzb4BuqdEkPZT0kqNt01DyFlmhVd02ZCxuFHFi3Oj1nviZ4fqyNRlnCezKKkccmDvPwoyVcCJr9316GrJIXNmlw7cbusPqLXqCaw6rXHHGzVdkqVreJbPIDy+nWjfIEpRMeoy4dEoagWY3j1I4W599Cc9Wkq/n7CL',
                t: 'MTY3MjQwOTA2OS40NTEwMDA=',
                m: 'xvSIbLNy7b/E2qMTL9hTKXULl49a+eObXL2jj0zGFIY=',
                i1: 'KIStQyQxBeNE2Nk6z8yuJA==',
                i2: 'obUzSCn0jVU6EfkqWkDLHA==',
                zh: '6PhE8IieaoO2pX52A6WaxoaWW7PiYYaQBxbISWiTBgM=',
                uh: 'LgBfwTjckPmPFLl2OGGaoWOKkjIgTojK2wwoWSzqSQw=',
                hh: '0xYs6/ITNgpxqma3i08nU6mpOle374oFFGRHKzTmLjo=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=781b532c0ebe7330');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/managed/v1?ray=781b532c0ebe7330';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, -window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\/the\/synonyms?__cf_chl_rt_tk=kla4GTZ1LIFWYgVpbnZpiakOJ9Jp5NOzUaMGg0kfxH4-1672409069-0-gaNycGzNB1E" + window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>


    <div class="footer" role="contentinfo">
        <div class="footer-inner">
            <div class="clearfix diagnostic-wrapper">
                <div class="ray-id">Ray ID: <code>781b532c0ebe7330</code></div>
            </div>
            <div class="text-center">Performance &amp; security by <a rel="noopener noreferrer" href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=m" target="_blank">Cloudflare</a></div>
        </div>
    </div>
</body>
</html>

The following excerpt might be relevant to a potential upstream change?

Enable JavaScript and cookies to continue

That's a shame. Sometimes, you can spoof the system to think it's a browser, but the javascript block is probably a deal breaker. What a sad way to end! This was such an excellent plugin! :(

EDIT: Looking at the code, the web dev's are actively preventing this type of traffic altogether.

EDIT 2: As a writer, I really appreciate the wordnut extension. Until there might be a workaround or concession by the website dev, wordnet is pretty great.

commented

Interestingly it still works on my M1 setup but not on the intel OSX. Go figure...

Interestingly it still works on my M1 setup but not on the intel OSX.

Interesting. Does it also work from your M1 setup if your use curl directly?

commented

Same curl output on Intel OSX (bigsur), request-default-error-callback in emacs,
and M1 OSX (monterey), still working as expected in emacs.
mirror straight.el .emacs init file for both systems.

~ % curl -I https://www.powerthesaurus.org/echo

HTTP/2 403
date: Fri, 06 Jan 2023 21:48:36 GMT
content-type: text/html; charset=UTF-8
permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
referrer-policy: same-origin
x-frame-options: SAMEORIGIN
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=c%2BnStweRWPWVDzDjxF8fi1Z3Nm%2Bccxe0ILc3Z2Unhzroip4AqKWDmQxdi1P%2F%2BLHTN263iIWt1vUXlZD8wvs0BuLeVnGZJhcAz7hogJLhuTuxpUQANXEAvgaR4Y3Yd%2BWjUzzF9rK9JPw%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 7857a8a76e798e14-LHR

commented

I have updated the OSX BigSur System via Brew with curl 7.87.0 (x86_64-apple-darwin20.6.0) and checked that emacs is using that version via its shell (some /etc/paths editing required). Unfortunately same error message [error] request-default-error-callback:.
Powerthesaurus.el still works flawless on M1 setup.

commented

is there any alternative thesaurus that users can use until this is addressed?

Hey folks, thanks for reporting this!
The only system that I have is the M1 Mac (Monterey), so it works for me. I'm pretty sure that we can "dress up" the request so it works in other setups as well.
NOTE: it's so weird that this bug is system dependent.

curl has builtin support for cookies. On my system, even though we don't specify anything special for it, request.el still adds cookie flags to the curl command. So, it runs something like:

curl --silent --location --cookie ./curl-cookie-jar --cookie-jar ./curl-cookie-jar --include --compressed --header "User-Agent: Chrome/74.0.3729.169" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" --header "Accept-Language: en,en-US;q=0.5" --header "Upgrade-Insecure-Requests: 1" --url https://www.powerthesaurus.org/marvelous/synonyms

Can anyone facing this problem try this request and see if this request has the same outcome? (Maybe --cookie flags simply got lost). Then it would be an easier problem. If not, we'll need to somehow "convince" powerthesaurus that we enabled Javascript.

It looks good to me. :)

curl example

I'm not sure how that translates to your code yet, though.

It looks good to me. :)

curl example

I'm not sure how that translates to your code yet, though.

Just to double-check, this is the HTML page that you got with that curl request I posted before on a system where emacs-powerthesaurus gets error 403? Am I right here?

If this is so, can you please execute powerthesaurus-debug-connection, request the synonyms for "marvelous", and send back everything related to power thesaurus from your *Messages* buffer? Especially the message that looks like a curl request. Thanks!

Just to clarify. Yes, I ran your line above and outputted it to a local HTML file. that is what you are looking at.

I've downloaded the latest version of powerthesaurus, unfortunately, that command isn't available to me.

Just to clarify. Yes, I ran your line above and outputted it to a local HTML file. that is what you are looking at.

I've downloaded the latest version of powerthesaurus, unfortunately, that command isn't available to me.

powerthesaurus-debug-connection is not interactive, so you can't call it via M-x. You can evaluate it directly from minibuffer M-; (powerthesaurus-debug-connection), or in the *scratch* buffer by pasting (powerthesaurus-debug-connection) and using C-x C-e after the closing parenthesis.

Sorry, for the confusion, I should've been clearer.

commented

Hi, I've been having the same issue. Finding synonyms for "marvelous" after running powerthesaurus-debug-connection gives the following:

  1. HTML part alone:
    image
  2. Actual output:
debug
[debug] request--curl: --silent --location --cookie /<<MYUSER>>/.emacs.d/request/curl-cookie-jar --cookie-jar /<<MYUSER>>/.emacs.d/request/curl-cookie-jar --include --write-out \n(:num-redirects %{num_redirects} :url-effective "%{url_effective}") --compressed --header User-Agent: Chrome/74.0.3729.169 --header Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 --header Accept-Language: en,en-US;q=0.5 --header Upgrade-Insecure-Requests: 1 --url https://www.powerthesaurus.org/marvelous/synonyms
[debug] request--curl-callback: event finished

[debug] request--callback: UNPARSED
HTTP/2 403 
date: Sat, 11 Mar 2023 18:08:35 GMT
content-type: text/html; charset=UTF-8
permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
referrer-policy: same-origin
x-frame-options: SAMEORIGIN
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=yQ8AZJL0kOO1zysEn3zK4foc67ibxslvlf7h1UqlCh5Q0KDCBxQxW9nDEvFilq93HVP1roS4u20lIC43lwjKzegrmC1xIlsz%2FzPDRE0BIgCpMh3BDTrRCyYWbYeZsYjOSq30jBXQKdY%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
vary: Accept-Encoding
server: cloudflare
cf-ray: 7a65be5e3f2d3766-HEL
content-encoding: br

<<HTML for the above>>

[error] request--callback: peculiar error: 403
[debug] request--callback: executing error
[error] request-default-error-callback: https://www.powerthesaurus.org/marvelous/synonyms error

A tiny update, I reproduced the issue on a local machine and in CI: https://github.com/SavchenkoValeriy/emacs-powerthesaurus/actions/runs/4392647435
It does look like this problem manifests itself only on Intel Macs (at least judging from this issue and CI). I managed to get curl requests working correctly with GraphQL requests from https://github.com/clarencecastillo/alfred-powerthesaurus. I'll rewrite the package to use API. Hopefully it should fix the issue for some time.

Hey folks, #25 should resolve the issue (at least it did in CI). I'd really appreciate if you can update to version 0.3.0 and confirm.
Cheers!

commented

Hi, thanks for the quick reply. I have updated to 0.3.0 (via straight.el - confirmed the version via find-library).

I get the following error now when using powerthesaurus-lookup-dwim (trace from toggle-debug-on-error below):

Debugger entered--Lisp error: (error "Unknown query type ’:synonyms’")
  signal(error ("Unknown query type ’:synonyms’"))
  error("Unknown query type '%s'" ":synonyms")
  powerthesaurus--query("powerthesaurus" ":synonyms" #f(compiled-function (results) #<bytecode 0x1592866d1aa5>))
  powerthesaurus-lookup("powerthesaurus" ":synonyms" nil nil)
  powerthesaurus-lookup-dwim(nil)
  funcall-interactively(powerthesaurus-lookup-dwim nil)
  call-interactively(powerthesaurus-lookup-dwim record nil)
  command-execute(powerthesaurus-lookup-dwim record)
  execute-extended-command(nil "powerthesaurus-lookup-dwim" nil)
  funcall-interactively(execute-extended-command nil "powerthesaurus-lookup-dwim" nil)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Other functions (powerthesaurus-lookup-synonyms-dwim, powerthesaurus-lookup-definitions-dwim etc) work fine, thanks :)

Just in case : I'm on Ubuntu 20.04

@mnazaal Whoops, that's my bad. I'll fix it in a second. Update to 0.3.1 and it's going to be alright.
Closing this issue since it got resolved.

commented

Intel Mac back on track 👍 :)