toddwschneider / sec-13f-filings

Hi,
looks like there is some issue in the code that blocks access for any IP address except for the one that was originally intended.
How can it be fixed?

@603000 Hi, there is nothing in this repo that has anything to do with IP addresses

Is it possible you're hitting the SEC's rate limit? You might consider setting a User-Agent header in the SecClient#get method here:

sec-13f-filings/app/lib/sec_client.rb

Lines 193 to 197 in f002c51

    
           def get(url) 
        
             response = HTTParty.get(url) 
        
             raise RateLimited if response.code == 429 
        
             response 
        
           end

Here's the SEC documentation: https://www.sec.gov/os/accessing-edgar-data

You could try something like:

response = HTTParty.get(url, headers: {"User-Agent" => "[Your Name] [your email]@[domain].com"})

Hi,
thank you for your response.
Please, take a look at the screen. How do you think what can be the reason for such blocks?

That is a problem with text encoding, there are some characters in the SEC file that are not UTF-8. I just put in this commit which should change any invalid characters into ?: 2dad979

Let me know if you still have issues, thanks

Thank you, I'll try this.

I also encountered another problem (not sure what's the cause) while trying to open any filing from any manager - the error always looks like this:

Could you give a hint what can cause such type of errors?

I would guess that the SEC website is blocking you. Have you tried setting a user agent with your name and email address? I gave an example in my first comment on this thread

Thank you, I tried your solution but it didn't help. Looks like the problem is in something else. Maybe you have other ideas what should I check?

It's hard for me to debug without being able to reproduce the current error. I'd recommend going into the rails console and seeing exactly what HTML is being fetched for a specific 13F, e.g.

html = SecClient.new.get("https://www.sec.gov/Archives/edgar/data/1067983/000095012322006442").body

And then you'll have to poke around with the html text to see what's in there. I've seen error messages asking for the user to declare a user agent, but if you've tried that already then I'm not sure what else could be happening, so you'll probably have to do some investigating on your own

Thank you, I'll try to investigate.
But it increasingly looks like that the problem is caused by incorrect installation.
Am I right that if the installationis done according to instruction - everything should work ok?
Or maybe these issues can be caused because of VPS provider?

Yes, I've been able to download all of the various reports, both on my local machine and on an AWS-hosted server

Or maybe these issues can be caused because of VPS provider?

I don't know if the SEC website blocks certain VPS providers, but I suppose it's possible. Again I'd recommend checking the html you get back to see if it has any error messages in it

	def get(url)
	response = HTTParty.get(url)
	raise RateLimited if response.code == 429
	response
	end

Blocked IP