Failure with Linux and USAA
Dameon87 opened this issue · comments
I'm unsure what the exact cause of this may be, but I'm running into trouble using this with Linux (Ubuntu 20.10) on both python3.8 and python3.9. I can get the commands to work correctly on windows. However, every command via linux is outputting:
~/.local/bin$ ofxget stmt usaa -u 011111111 --all
Traceback (most recent call last):
File "/home/wallet/.local/bin/ofxget", line 8, in
sys.exit(main())
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/scripts/ofxget.py", line 1568, in main
REQUEST_HANDLERSargs["request"]
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/scripts/ofxget.py", line 646, in request_stmt
_merge_acctinfo(args, acctinfo)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/scripts/ofxget.py", line 617, in _merge_acctinfo
acctinfos: List[AcctInfo] = sorted(extract_acctinfos(markup), key=sortKey)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/scripts/ofxget.py", line 1380, in extract_acctinfos
parser.parse(markup)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/Parser.py", line 85, in parse
self.header, message = self._read(source)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/Parser.py", line 118, in _read
header, message = parse_header(source)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/header.py", line 294, in parse_header
header, header_end_index = OFXHeaderV1.parse(rawheader)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/header.py", line 81, in parse
raise OFXHeaderError(f"OFX header is malformed:\n{rawheader}")
ofxtools.header.OFXHeaderError: OFX header is malformed:
System Error
| USAA</title>
<style media="screen" type="text/css">
This, again, only seems to happen on linux. The same command works fine on my windows install.
Hmm, I can't reproduce this on Python 3.9 under Linux. Is this occurring on master, or an older version of ofxtools?
I've tried this on master and the most current release version. I'm quite unsure why this is happening. Only thing I can think of is it's an issue in Ubuntu. Are you able to attempt to give this a try on Ubuntu 20.10?
If you're running identical Python and ofxtools on 2 different OS'es, but on one of them you're getting different results, I'd eliminate differences in user config before turning to the OS.
The problem is that USAA isn't sending you OFX. Your Ubuntu machine is sending different requests. Try comparing the output of ofxget list usaa and ofxget stmt --dryrun usaa on both hosts to find the differences.
I'm also encountering this issue.
Docker python:3.7-alpine
reqs.pip ofxtools
Then ofxget acctinfo usaa --user <uid> throws an invalid OFX header. It looks like USAA is returning an HTML doc in response to this endpoint, so maybe it's an issue on their end?
Well shoot, now it's failing for me as well. I'll try to look into this a bit more.
@csingley bad news, USAA completely changed their OFX authentication so now only Quicken can connect
https://communities.usaa.com/t5/Finances/USAA-Creates-Quicken-Monopoly/td-p/243850
Well that's a drag. USAA seems to be following Schwab to the "OFX Secure Plus" dark side... "You'll use Quicken and like it".
The original reports on this ticket were probably formatting differences that were fixable by tweaking connection parameters. This latest change is something qualitatively different, and it ain't gonna be fixable from my end.
Haven't tested this yet, but someone claims to have got this working from the master branch:
https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045704.html
@csingley I've been able to get USAA to work again with some information from the GnuCash-devel mailing list. The following config seems to work for me with latest version from here with some addl notes.
The clientuid I extracted from the URL when authenticating via https://df3cx-services.1fsapi.com/casm/usaa/enroll. After entering USAA credentials the page is redirected to https://www.usaa.com/inet/ent_oauth_consent/authorize?0&client_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&.... It would seem there may be a link between the account and this clientuid. Also ClientUID must be in uppercase.
DTACCTUP of 19900101 is required to get good data. The default of 19901231 returned success, but only "Client up to date", no acct info.
pretty = true is also required (assuming the newline requirement).
I'm not sure if or where these changes should be incorporated, but hoping the information is useful none the less.
[USAA-NEW]
url = https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
clientuid = XXXXXXXX-XXXX-XXXX-XXXXX-XXXXXXX
org = USAA Federal Savings Bank
fid = 67811
bankid = 314074269
user = xxxxxxxxx
version = 103
unclosedelements = true
appid = QMOFX
appver = 2300
pretty = true
@gthole That was me. Was updating this post here at same time you posted. 👍
@cfazzini do you know what user agent is being used?
I ended up getting the IP block described on https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045690.html, thank fuck my assets are at Chase now...
@jackpot51 Best I can tell from the code in ofxget.py, default is a blank User-Agent, so it would appear I'm not sending one. But, could use confirmation on that. Sucks on the IP ban. Hopefully it's short term.
@cfazzini - thanks for the report! Setting CLIENTUID is not too crazy.
I'm thinking I may need to patch OFXClient to uppercase the UUID and send the truncated DTACCTUP.
Should we perhaps set up a wiki, to serve as a repository for this kind of info that's helpful to users?
Sorry just realized I wasn't too clear. It seemed to not have an issue with the full date format, the value of the date seemed to matter more. I just used the 19900101 format with the --dtacctup CLI flag and it was fine.
@cfazzini - thanks for the clarification.
From ofxtools.Client.OFXClient, by default useragent: str = "InetClntApp/3.0"
Thanks for clarification, I was looking at this in ofxget.py
parser.add_argument(
"--useragent",
dest="useragent",
help="Value to use in HTTP 'User-Agent' header (defaults to empty string)",
)
@csingley thanks for that, I believe that is the only User-Agent header that will work
Good news, the IP block was temporary. You will experience it if you provide what the new endpoint considers invalid data. Seems like it clears after about an hour
help="Value to use in HTTP 'User-Agent' header (defaults to empty string)"
Heh, that helpstring should probably be fixed; it's very unhelpful. I suppose it's technically true... argparse does indeed default to an empty string... but the OFXClient, which does all the heavy lifting, will change that to the Quicken-compatible value if you don't supply something else.
Also ClientUID must be in uppercase.
DTACCTUP of 19900101 is required to get good data. The default of 19901231 returned success, but only "Client up to date", no acct info.
pretty = true is also required (assuming the newline requirement).
I'm not sure if or where these changes should be incorporated, but hoping the information is useful none the less.
The default configs (which can be overriden by the user's ofxget.cfg or CLI args) are located in ofxtools/config/fi.cfg. I have incorporated the changes you've listed; they'e now in master.
Thank you.
So, I had been using a different Python script called PocketSense up until this point but the developer isn't very responsive or active. I'm interested in using ofxtools to work with USAA. Has anyone here had success getting it working and if so, how did you configure the ofxtools software? I was able to grab the clientID, Access ID, and PIN but I may need a little assistance using the software (on Windows).
I've been trying to replicate your success for my USAA accounts, but I am consistently getting the IP block described above. I've checked that my parameters are correct (modified with my own clientuid and credentials), waited hours, and overnight in case my previous tests triggered a temporary lock-out, but never saw a successful test. Any idea how those who got it working might be different? For example, I'm running this from a home machine, so I'm behind a NAT with a consumer-grade ISP's IP. Are you doing anything different, like running from an AWS instance like suggested in the gnucash threads?
Using latest fixes in Github, it should work fine. You need to put clientuid (retrieved from enrollment URL) in .config/ofxtools/ofxget.cfg:
[usaa]
clientuid = XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
As far as I can tell, can't enter a clientuid as a CLI paramenter. After that, I was able to retrieve acctinfo using command:
ofxget acctinfo usaa -u abcd1234
For further reference, here is a sanitized preview output (using -n parameter):
OFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:NONE
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:8CC3D3C8-EFF6-4B5C-8743-963A1F85C9F7
<OFX>
<SIGNONMSGSRQV1>
<SONRQ>
<DTCLIENT>20210210162625.502[+0:UTC]
<USERID>abcd1234
<USERPASS>anonymous00000000000000000000000
<LANGUAGE>ENG
<FI>
<ORG>USAA Federal Savings Bank
<FID>67811
</FI>
<APPID>QWIN
<APPVER>2700
<CLIENTUID>XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
</SONRQ>
</SIGNONMSGSRQV1>
<SIGNUPMSGSRQV1>
<ACCTINFOTRNRQ>
<TRNUID>776C7F66-E1B0-43C2-8EDC-59C5A657B3A1
<ACCTINFORQ>
<DTACCTUP>19900101000000.000[+0:UTC]
</ACCTINFORQ>
</ACCTINFOTRNRQ>
</SIGNUPMSGSRQV1>
</OFX>
Once that is working, can follow the ofxtools documentation for saving your accounts/configuration following the ofxtools documentation.
Hope that helps!
@csingley is it possible to pass a clientuid on the command line? It appears --clientuid generates a random one. Certainly better stored in cfg file, but might be a useful option for debugging.
I was using the Mac appid/appver (QMOFX/2300) values from your earlier post, but I just tried switching to the default QWIN/2700 values, and sadly I'm still failing to get any data from the server. Weird. Outside of my own CLIENTUID copied from my own registration and my own credentials, and the random TNDUID and timestamp, my payload is identical to yours. Included below for the benefit of others:
$ ofxget acctinfo myusaa --user myidNNN -n
OFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:NONE
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:6BFFB78F-1660-48A2-A0C1-11035696917E
<OFX>
<SIGNONMSGSRQV1>
<SONRQ>
<DTCLIENT>20210210164733.953[+0:UTC]
<USERID>myidNNN
<USERPASS>anonymous00000000000000000000000
<LANGUAGE>ENG
<FI>
<ORG>USAA Federal Savings Bank
<FID>67811
</FI>
<APPID>QWIN
<APPVER>2700
<CLIENTUID>XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
</SONRQ>
</SIGNONMSGSRQV1>
<SIGNUPMSGSRQV1>
<ACCTINFOTRNRQ>
<TRNUID>2B70DC6B-B47A-4F64-861C-2A5746E2C7D0
<ACCTINFORQ>
<DTACCTUP>19900101000000.000[+0:UTC]
</ACCTINFORQ>
</ACCTINFOTRNRQ>
</SIGNUPMSGSRQV1>
</OFX>
Dang, that all does look the same. What response are you getting? There was a comment I think on Gnu-cash about clearing the block by reregistering via the enrollment URL. When I reregister it gives me the same user,pass, and clientid. Might be worth a shot.
is it possible to pass a clientuid on the command line?
Not presently, no - the script helpstring is in fact accurate in this case. I don't think it would be difficult to tweak the argparse setup to accomplish that... modifying UuidAction near the top of the file ought to do the trick (I believe the clientuid arg is the only one that uses it).
I just tried revoking Quicken access and re-registering, and got the exact same client id, access id, and access PIN back. That doesn't give me great confidence that this app password is really revokable. Unsurprisingly, I still get no access after this re-registration process. I consistently get the exact same Incapsula firewall message reported on other boards:
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>
So there is nothing exotic about the system you're using to run ofxtools? Ordinary home broadband behind a NAT?
Also, just making sure: the username and password you are using successfully are the access ID and access PIN that came from the registration process? Not your USAA account name and password or some concatenation of PINs?
Nope nothing exotic. Standard home cable internet. ofxtools via WSL. Just access ID and PIN retrieved from registration process. Not sure what else it could be. :(
A few more breadcrumbs in case they help: I found that my acctinfo query is actually failing prior to every using my credentials. The call to _get_service_urls, which in turn calls self.request_profile and is what is resulting in the above error page, which throws an exception and terminates since it is HTML rather than the expected OFX data. This call passes dummy values for the userid and userpass.
Something to try. Redirect the output of ofxget -n to a txt file.
ofxget acctinfo myusaa --user myidNNN -n > request.txt
Edit the password to be correct in that txt file.
Then you can try curl that request.txt to USAA endpoint and it should work.
curl -isS -X POST -H "Content-Type: application/x-ofx" -A InetClntApp/3.0 --data-binary @request.txt https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
Thanks - that's a great suggestion. This makes it easy to test from another computer without having to get ofxtools working first. I just tried on another machine (that has curl but too old python to easily get ofxtools master running) and it worked! However, my home computer still fails in the same way, so the evidence points to the server having my IP or my ISP's IP range blocked for whatever reason. Irritating, but at least I know that there isn't something wrong with my account or my ofxtools setup. Thanks for your help!
Using latest fixes in Github, it should work fine. You need to put clientuid (retrieved from enrollment URL) in
.config/ofxget/ofxget.cfg:[usaa] clientuid = XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
According to the docs, that config file should be .config/ofxtools/ofxget.cfg, right? At least it started working for me when I used the correct filename.
Ah yes, sorry went off memory on that one. Edited, thanks.
I've read through this series of posts, but I still find myself unable to get this working. Any pointers that might be provided would be appreciated.
OS: Ubuntu 20.04.1
Python: 3.8.5
OfxTools: 0.9.1
I have successfully retrieved my Access ID and Access PIN from USAA. I have tried the following two configurations in ofxget.cfg (the first is a UUID from a post on the GnuCash list that someone had working, the second is the UUID from my access request to USAA).
[usaa1]
url = https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
version = 103
undisclosedelements = true
pretty = true
org = USAA Federal Savings Bank
fid = 67811
bankid = 314074269
clientuid = 39E0E763-4E1E-4918-9528-D6EBAC94EF5D
appid = QMOFX
appver = 2300
[usaa2]
url = https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
clientuid = 01F023E5-3602-4DC5-A2D8-DED36C8F6700
org = USAA Federal Savings Bank
fid = 67811
bankid = 314074269
version = 103
unclosedelements = true
appid = QMOFX
appver = 2300
pretty = true
I feel like I have to be missing something pretty basic here, but whatever it is, I can't seem to find it. When I submit ofxget acctinfo usaa2 --user <access id> with either of these two configurations (ie. usaa1 or usaa2), what I get back is:
Traceback (most recent call last):
File "/home/tracy/.local/bin/ofxget", line 8, in <module>
sys.exit(main())
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 1590, in main
REQUEST_HANDLERS[args["request"]](args)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 595, in request_acctinfo
acctinfo = _request_acctinfo(args, password)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 612, in _request_acctinfo
with client.request_accounts(
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 593, in request_accounts
RqCls2url = self._get_service_urls(
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 420, in _get_service_urls
profile = self.request_profile(
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 507, in request_profile
parser.parse(response)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Parser.py", line 85, in parse
self.header, message = self._read(source)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Parser.py", line 118, in _read
header, message = parse_header(source)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/header.py", line 294, in parse_header
header, header_end_index = OFXHeaderV1.parse(rawheader)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/header.py", line 81, in parse
raise OFXHeaderError(f"OFX header is malformed:\n{rawheader}")
ofxtools.header.OFXHeaderError: OFX header is malformed:
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>
I did attempt the test with curl that was mentioned above, and when I did that, using these commands:
ofxget acctinfo usaa2 --user <access id> -n > ~/usaa.txt
curl -isS -X POST -H "Content-Type: application/x-ofx" -A InetClntApp/3.0 --data-binary @~/usaa.txt https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
what I get back is:
HTTP/2 200
content-type: text/html
cache-control: no-cache, no-store
content-length: 212
x-iinfo: 13-79156783-0 0NNN RT(1613501217148 0) q(0 -1 -1 -1) r(0 -1) B10(4,289,0) U6
strict-transport-security: max-age=31536000
set-cookie: visid_incap_2454689=lSjXlj/bT72v90CjI/6+gCETLGAAAAAAQUIPAAAAAACvqUwlgxaNGcdtkgHfzxnu; expires=Wed, 16 Feb 2022 07:27:51 GMT; HttpOnly; path=/; Domain=.1fsapi.com; Secure; SameSite=None
set-cookie: incap_ses_1167_2454689=UoaWXVmKyASrsiG3jQQyECETLGAAAAAAa4m13ia9aMXINn8B7eK26g==; path=/; Domain=.1fsapi.com; Secure; SameSite=None
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>
which doesn't tell me much (other than that the USAA server accepted the request), but might tell you something?
I've read through this series of posts, but I still find myself unable to get this working. Any pointers that might be provided would be appreciated.
OS: Ubuntu 20.04.1
Python: 3.8.5
OfxTools: 0.9.1
Similar to mine, except I'm Python 3.8.3 on CentOS 8.
Can you confirm if you're running ofxtools 0.9.1? Or did you pip install from the ofxtools master branch? In my case, I did a pip install directly from the source checkout (so, with commits more recent than 0.9.1).
I have successfully retrieved my Access ID and Access PIN from USAA. I have tried the following two configurations in ofxget.cfg (the first is a UUID from a post on the GnuCash list that someone had working, the second is the UUID from my access request to USAA).
[usaa1] url = https://df3cx-services.1fsapi.com/casm/usaa/access.ofx version = 103 undisclosedelements = true pretty = true org = USAA Federal Savings Bank fid = 67811 bankid = 314074269 clientuid = 39E0E763-4E1E-4918-9528-D6EBAC94EF5D appid = QMOFX appver = 2300 [usaa2] url = https://df3cx-services.1fsapi.com/casm/usaa/access.ofx clientuid = 01F023E5-3602-4DC5-A2D8-DED36C8F6700 org = USAA Federal Savings Bank fid = 67811 bankid = 314074269 version = 103 unclosedelements = true appid = QMOFX appver = 2300 pretty = true
In my case, I only have this in ~/.config/ofxtools/ofxget.cfg:
[DEFAULT]
clientuid = ########-####-####-####-############
[usaa]
clientuid = ########-####-####-####-############
bankid = 314074269
user = ########
I did attempt the test with curl that was mentioned above, and when I did that, using these commands:
ofxget acctinfo usaa2 --user <access id> -n > ~/usaa.txt curl -isS -X POST -H "Content-Type: application/x-ofx" -A InetClntApp/3.0 --data-binary @~/usaa.txt https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
Does your usaa.txt file contain a <CLIENTUID> element?
In order to tell whether the curl test is working for you, you need to edit the USERPASS field to contain your real access PIN, replacing the placeholder password. Did you do that? The result should look like the samples we posted above, except for your username, password, and clientuid (and except for the fields that change on every transaction: NEWFILEUID, DTCLIENT, and TRNUID).
I was able to get curl to work, but I found that using "ofxget acctinfo" to send the same request was still consistently failing for me. With some debugging, I found that the call to _get_service_urls was indirectly causing the failure, meaning the profile request that _get_service_urls sends works, but causes the subsequent acctinfo/stmt requests to fail. I hacked the ofxtools client code to skip this call, I was able to get successful acctinfo and stmt requests. My guess is that the server expects a session cookie to be carried over between these back-to-back transactions, or something of that nature, though it is perplexing that this only happens for some of us and not others. I'll dig deeper when I get a chance and report back. If you can get the curl test to work, I suspect my hack will work for you too.
@gebailey I installed ofxtools from PyPI, using pip install ofxtools, and during the install, it showed that it was installing 0.9.1. Confirming that, I have:
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib.metadata import version
>>> version('ofxtools')
'0.9.1'
>>>
If you can tell me how to do the install from master, I'd be willing to do that (bear in mind that I know little to nothing of python or using pip other than what I have read in the last day or so).
As for the extra stuff in the ofxget.cfg file, I was trying a number of different things - info from Ofx Home, info from John Ralls' post on the GnuCash Wiki, and things I found in digging through the fin.cfg. I can pare it down without a problem, if you think that will help.
The usaa.txt file contains:
OFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:NONE
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:e0e0b5b0-72a0-4ad4-bf0d-b80785b0eb85
<OFX>
<SIGNONMSGSRQV1>
<SONRQ>
<DTCLIENT>20210216184653.576[0:GMT]
<USERID>[access id]
<USERPASS>[access pin]
<LANGUAGE>ENG
<FI>
<ORG>USAA Federal Savings Bank
<FID>67811
</FI>
<APPID>QMOFX
<APPVER>2300
<CLIENTUID>01F023E5-3602-4DC5-A2D8-DED36C8F6700
</SONRQ>
</SIGNONMSGSRQV1>
<SIGNUPMSGSRQV1>
<ACCTINFOTRNRQ>
<TRNUID>db5f7fa8-aca4-47c1-b9bd-894915cc644f
<ACCTINFORQ>
<DTACCTUP>19901231000000.000[0:GMT]
</ACCTINFORQ>
</ACCTINFOTRNRQ>
</SIGNUPMSGSRQV1>
</OFX>
@ajam000 Yes, I did change the USERPASS field to the Access PIN. I'm assuming that plain text is fine, as I didn't attempt any type of encoding on the PIN. I did get the response back that I posted, but no account numbers or other "useful" data...
Try editing your usaa.txt file as follows: (1) Change the values of NEWFILEUID and TRNUID to all capital letters (or copy and paste from my sample above - the value does not matter, but upper- vs lower-case does). (2) Change DTACCTUP field to 19900101000000.000[+0:UTC]. (3) Change APPID to QWIN and APPVER to 2700. That should cover all of the relevant differences between 0.9.1 and master branch.
Changed made, as described, with the following results....
usaa.txt:
OFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:NONE
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:E0E0B5B0-72A0-4AD4-BF0D-B80785B0EB85
<OFX>
<SIGNONMSGSRQV1>
<SONRQ>
<DTCLIENT>20210216184653.576[0:GMT]
<USERID>[access id]
<USERPASS>[access pin]
<LANGUAGE>ENG
<FI>
<ORG>USAA Federal Savings Bank
<FID>67811
</FI>
<APPID>QWIN
<APPVER>2700
<CLIENTUID>01F023E5-3602-4DC5-A2D8-DED36C8F6700
</SONRQ>
</SIGNONMSGSRQV1>
<SIGNUPMSGSRQV1>
<ACCTINFOTRNRQ>
<TRNUID>DB5F7FA8-ACA4-47C1-B9BD-894915CC644F
<ACCTINFORQ>
<DTACCTUP>19900101000000.000[+0:UTC]
</ACCTINFORQ>
</ACCTINFOTRNRQ>
</SIGNUPMSGSRQV1>
</OFX>
Returned from curl:
curl -isS -X POST -H "Content-Type: application/x-ofx" -A InetClntApp/3.0 --data-binary @~/usaa.txt https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
HTTP/2 400
date: Tue, 16 Feb 2021 21:11:13 GMT
content-type: text/html;charset=UTF-8
vary: Origin
vary: Access-Control-Request-Method
vary: Access-Control-Request-Headers
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: 0
strict-transport-security: max-age=31536000 ; includeSubDomains
x-frame-options: DENY
content-language: en-US
set-cookie: visid_incap_2454689=ro+ewOpVSZqs8J/ElLEH/vE0LGAAAAAAQUIPAAAAAACXyLgbG36FGa7epOPWP7hK; expires=Wed, 16 Feb 2022 07:27:51 GMT; HttpOnly; path=/; Domain=.1fsapi.com; Secure; SameSite=None
set-cookie: nlbi_2454689=qeslROEG+xpt0O88hXBnAwAAAAB/89r1uMWqPmSbuKy0JoL4; path=/; Domain=.1fsapi.com; Secure; SameSite=None
set-cookie: incap_ses_1167_2454689=xmi4SZoj2AJ2Y4C3jQQyEPE0LGAAAAAAXz/8sP/3W1ZusnGA0VkwFw==; path=/; Domain=.1fsapi.com; Secure; SameSite=None
x-cdn: Incapsula
x-iinfo: 3-21342823-21342824 NNNN CT(32 49 0) RT(1613509873614 0) q(0 0 0 0) r(1 1) U6
<!DOCTYPE html>
<html>
<head>
<title>
Uh Oh!
</title>
<link rel="icon" type="image/png" href="images/usaa-quicken-usaa-logo.png"/>
<style>
body {
background-color: rgba(37, 36, 43, 1) !important;
font-family: 'Source Sans Pro', sans-serif;
padding: 0 5%;
}
@keyframes fadeInAnimation {
0% {
opacity: 0;
}
100% {
opacity: 1;
}
}
@keyframes slideInFromLeft {
0% {
transform: translateX(-15%);
opacity: 0;
}
100% {
transform: translateX(0);
opacity: 1;
}
}
.head-line {
font-size: 45px;
line-height: 60px;
color: rgba(255, 255, 255, .2);
letter-spacing: -1px;
margin-bottom: 5px;
}
.secondary-text-color {
/ / color: rgba(186, 179, 227, 1) !important;
color: rgb(55 130 120) !important;
}
.content-container {
animation: slideInFromLeft ease 2s;
animation-iteration-count: 1;
animation-fill-mode: forwards;
transition: color .2s linear;
max-width: 400px;
position: relative;
top: 250.5px;
}
.subheader {
transition: color .2s linear;
font-size: 31px;
line-height: 46px;
color: #fff;
}
.primary-text-color {
color: #FFFFFF !important;
}
.clearfix {
clear: both;
zoom: 1;
}
.clearfix:before, .clearfix:after {
content: "\0020";
display: block;
height: 0;
visibility: hidden;
}
.clearfix:after {
clear: both;
}
.context {
transition: color .2s linear;
font-size: 17px;
line-height: 27px;
color: #fff;
}
.context p {
margin: 0;
}
.context p:nth-child(n+2) {
margin-top: 12px;
}
.buttons-container {
margin-top: 45px;
overflow: hidden;
}
.buttons-container a {
transition: color .2s linear, border-color .2s linear;
font-size: 14px;
text-transform: uppercase;
text-decoration: none;
color: #fff;
border: 2px solid white;
border-radius: 99px;
padding: 13px 15px 9px;
display: inline-block;
float: left;
}
.buttons-container a:hover {
background-color: rgba(255, 255, 255, .05);
}
.buttons-container a:first-child {
margin-right: 25px;
}
.border-button {
color: #FFFFFF !important;
border-color: #FFFFFF !important;
}
.button {
background-color: #FFFFFF !important;
color: #FFFFFF !important;
}
hr {
height: 1px;
background-color: rgba(255, 255, 255, .2);
border: none;
width: 250px;
margin: 35px 0;
}
</style>
</head>
<body>
<div class="content-container">
<div class="head-line secondary-text-color">
500
</div>
<div class="subheader primary-text-color">
Oops, Something went wrong.
</div>
<hr>
<div class="clearfix"></div>
<div class="context primary-text-color">
<!-- doesn't use context_content because it's ALWAYS the same thing -->
<p>
You may want to head back to the homepage.<br/>
If you think something is broken, report a problem.
</p>
</div>
<div class="buttons-container">
<a class="border-button" href="https://usaa.com" target="_blank">Go to homepage</a>
</div>
</div>
</div>
<script async type="text/javascript" src="/_Incapsula_Resource?SWJIYLWA=719d34d31c8e3a6e6fffd425f7e032f3&ns=1&cb=297782297"></script>
</body>
So, that was different than what I was getting before, but still no actual account data....
Maybe try making the timezone consistent: either both [+0:UTC] or both [0:GMT]. These are both valid representations of the UTC time zone, but it might not like that they aren't consistent.
My account access is failing again with the same Incapsula error message. Does anyone else watching this thread still have access to their account?
I changed them both to UTC, and got the same response. So I tried both as GMT. No dice.
You're consistently getting the 400 response code with the "Something went wrong" message? Or consistently getting the previous 200 response code with the Incapsula robots message? If the first, then something is wrong with your query - it could be that you accidentally altered something subtle when editing the text by hand, e.g. your editor may have removed the carriage return characters in the header. You could try capturing a fresh query of ofxget and the -n flag.
If you are getting the second error, then you are in the same boat as me - your IP is getting blocked. I was hoping that others would chime in with their own results so that we would have some idea whether they have altered their firewall configuration to block wider IP ranges, or whether results vary between us.
Sorry, I should have been more clear. I am getting the 200 error with the Incapsula message. There was a post on the GnuCash list not long ago that said an IP block wasn't permanent, that it would last a day or two before it rotated off. So I will try again this weekend and see if I can get anything using the curl method.
Yes, I'm monitoring the GnuCash thread as well for further developments. My personal experience is my IP block lasted around a week, then there was a brief window where I managed to successfully capture my statement data, and now I am blocked again, all with no changes on my side. I've not been hammering their server with requests or anything else I would expect to cause lock-outs - I've only hit their service a few times a day. The statement data imported into my accounting software fine as well, so I would be happy if it weren't for this aggressive blocking behavior.
Unfortunate to hear you are still having issues. Not that it's much help but more as another data point, I haven't had any issues since implementing the fixes.
Well, here we are about a week later, and I'm still getting the Incapsula response using the CURL string and saved file that I had from last week. (Today is the first time I've tried to do anything with this since my last post on the 18th, and I tried only the one time.) I will try it again tomorrow night, and see if that matters (since, technically, tomorrow will be the 7th day). Not sure where to go with this if I am not able to get a connection tomorrow.
You could jump ship like I did to Chase, where they still have a working OFX endpoint
I also remain blocked from home, however I set up ofxtools on an AWS instance with the exact same config, and I was able to download my account data, so it really is just blocking access from my home IP for some reason. I haven't made a decision yet whether this is an acceptable solution for me or whether just switching to a more customer-friendly bank makes more sense.
I had issues with two USAA accounts until I ensured that all *UID values were uppercase GUID's, not random strings. After that, I still get blocked downloading account data from one user, but not from the other. Downloading the list of accounts works on both users. So I think there is server-side blocking that is massively screwed up.
Well, tonight's run was rather anti-climactic. I remain blocked. So, at some point in the next week, I will probably gird my loins with excessive patience and actually call USAA and see if I can find someone there who has a clue about all this (I am not encouraged by many, many postings on their community boards) and try to get
A) unblocked
B) a successful run for account lists
C) a successful run for transactions from at least one account
Should it happen that I am successful, I will post back here with any relevant details. Should it happen that I am unsuccessful, I will at least report back that fact.
The odds are that it will be at least Tuesday or Wednesday of the coming week before I have opportunity to pursue this (given the amount of time investment required to climb through first layer tech support).
I wondering whether the people who have had success getting this to work have a paid Quicken account? I have noticed that the app access for Quicken disappears from my USAA account profile after about a week. If I re-request access, I get the exact same access ID and PIN every time, and it reappears in my USAA profile. I am wondering whether Quicken is doing something to keep the token alive that is not done by ofxtools.
No paid Quicken acct, I've continued to have no issues with USAA since getting the format correct. I download transactions intermittently, sometimes more than a week apart.
@cfazzini Could you post your working configuration (without accessid and pin, of course)? Perhaps I'll see something in yours that is not in mine (or something that is in mine that is not in yours)?
At this point, it has been about two weeks (13 days, to be exact) since I last tried it. I am probably going to wait until next week (that will put me at 19 days, which should be plenty of time for an IP block to be lifted). Then if I get nowhere, I will try to figure out AWS enough to be able to spin up an instance under the free tier to test with. Then, depending on what results I get, I will likely be trying to get in touch with USAA support...
FYI, AWS is not working for me either - it worked a few times and then I was blocked just like my home IP. I've tried IPs from multiple AWS regions, so either there is a very wide-ranging IP block or the blocking is tied to my account. I'm not sure what to conclude - I've not identified any difference in my setup other than my USAA account credentials.
If that is the case, then it has to be tied to the account. So, yet another thing to discuss with them once I get to that point...
OK, so it had been a couple of weeks since I messed with this, so I decided to try it again. Here's what I did, and the results that I got. I'm still not sure what these results mean, but perhaps someone here will.
First, I went to the main page of this repository, downloaded the code (using git clone https://github.com/csingley/ofxtools.git. Then i went into the ofxtools directory, and did pip install . which removed my existing ofxtools install and replaced it with what I had just downloaded (still shows version 0.9.1, but it does incorporate changes from the previously installed version, as will become evident in a moment).
Next, I went to USAA and re-authorized Quicken to access the account (to refresh all my data and access). Gave me the same access ID and access PIN as before, but I did copy the new UUID from the resulting URL into the ofxget.cfg file. Relevant portion of the ofxget.cfg file is:
[usaa1]
url = https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
version = 103
unclosedelements = true
pretty = true
org = USAA Federal Savings Bank
fid = 67811
bankid = 314074269
clientuid = 1A359CF9-E558-40F9-B304-AC4777F5F55A
appid = QWIN
appver = 2700
So, after that, I tried to use ofxget acctinfo usaa1 -user <accessID> to retrieve account info. Didn't work - instead, I got:
Traceback (most recent call last):
File "/home/tracy/.local/bin/ofxget", line 8, in <module>
sys.exit(main())
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 1598, in main
REQUEST_HANDLERS[args["request"]](args)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 600, in request_acctinfo
acctinfo = _request_acctinfo(args, password)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 617, in _request_acctinfo
with client.request_accounts(
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 593, in request_accounts
RqCls2url = self._get_service_urls(
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 420, in _get_service_urls
profile = self.request_profile(
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 507, in request_profile
parser.parse(response)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Parser.py", line 85, in parse
self.header, message = self._read(source)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Parser.py", line 118, in _read
header, message = parse_header(source)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/header.py", line 294, in parse_header
header, header_end_index = OFXHeaderV1.parse(rawheader)
File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/header.py", line 81, in parse
raise OFXHeaderError(f"OFX header is malformed:\n{rawheader}")
ofxtools.header.OFXHeaderError: OFX header is malformed:
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>
So I went back and did the ofxget acctinfo usaa1 --user <accessID> -n > ~/usaatest.txt command to generate the text file with the info to send via curl, which turned out as:
OFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:NONE
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:76DB7950-B0FE-4EBF-BC4B-A1310641A3D2
<OFX>
<SIGNONMSGSRQV1>
<SONRQ>
<DTCLIENT>20210317221045.836[+0:UTC]
<USERID>accessID
<USERPASS>accessPIN
<LANGUAGE>ENG
<FI>
<ORG>USAA Federal Savings Bank
<FID>67811
</FI>
<APPID>QWIN
<APPVER>2700
<CLIENTUID>1A359CF9-E558-40F9-B304-AC4777F5F55A
</SONRQ>
</SIGNONMSGSRQV1>
<SIGNUPMSGSRQV1>
<ACCTINFOTRNRQ>
<TRNUID>7F4B66FF-EE7B-471C-9342-6AA59EB6A51B
<ACCTINFORQ>
<DTACCTUP>19900101000000.000[+0:UTC]
</ACCTINFORQ>
</ACCTINFOTRNRQ>
</SIGNUPMSGSRQV1>
</OFX>
Then I tried sending that using curl using the command:
curl -isS -X POST -H "Content-Type: application/x-ofx" -A InetClntApp/3.0 --data-binary @~/usaatest.txt https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
Which netted me this:
HTTP/2 200
content-type: text/html
cache-control: no-cache, no-store
content-length: 212
x-iinfo: 7-44160127-0 0NNN RT(1616019220696 0) q(0 -1 -1 -1) r(0 -1) B10(4,289,0) U6
strict-transport-security: max-age=31536000
set-cookie: visid_incap_2454689=7ncQU8fwRnW23+kvhq6cHxR/UmAAAAAAQUIPAAAAAAApfjZcKE1eXpbwMRh34RUp; expires=Thu, 17 Mar 2022 08:21:42 GMT; HttpOnly; path=/; Domain=.1fsapi.com; Secure; SameSite=None
set-cookie: incap_ses_1167_2454689=UMitL8M5kw0O3BS1lAQyEBR/UmAAAAAAoDJkmfUKLvNnlQpx3rh08A==; path=/; Domain=.1fsapi.com; Secure; SameSite=None
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>
I did go back to USAA at one point in this process and attempt to reauthorize Quicken, and I was prompted at sign-in with a "are you human" Captcha, which I answered then it let me log in. So I'm thinking that I might have been blocked at one point, and going in and answering that Captcha unblocked me? Maybe. Anyway, this is where I'm currently at. If anyone has any additional suggestions, I'm all ears. Otherwise, I'm about to just throw my hands up at it and just forget about it again for a while (until I have another free day that I don't mind wasting on it)....
I am in the same state - I consistently am blocked with this exact same error page. I have also tried testing from multiple AWS regions in order to rule out IP blocking, and did get it to work the very first time I tried sending the request from an AWS instance, but never again. I did not change anything between the successful query and the consistent failure thereafter - there must be something tied to my account blocking access, and likely yours as well.
Good people, you are entirely welcome to discuss the rituals required to summon lesser demons of the technofinance pantheon. Let's just, you know, take it off the bug tracker, okay? I'm pushing the "Discussions" section of the Github site, which seems like a good fit for this sort of thing.