Topic: Some OFXGet Feedback

Hi, saw that OFXGet was your first Java Program.  Pretty good, but found some things.

Testing was done with Fidelity.  There are 2 institution problem threads that pertain to Fidelity.  I did not respond to those old threads as those people are likely gone, but include everything here.

The first is the invalid date. The ofx.generateInvestRequest() appears to put too early a date in the  <DTSTART> tag for transaction inclusion.  I just changed to a hardcoded:
line 116:                                        "<DTEND>20110801000000\n" +

Everything worked.  Using the end date form of the request is probably more resilient, as you could specify yesterday on the fly.  To do that:
-  Change' import java.util.Date;' to 'import java.util.*;'   This allows the Calendar class to be short hand referenced too.  Calendar is kind of a Date replacement.
-  Take the 'SimpleDateFormat format' local var in generateDate() and make it a class member, so it can be used in multiple places.
-  Easier to just list the entire function:
    private String generateInvestRequest()
    {   
                Calendar now = Calendar.getInstance();
                now.add(Calendar.DATE, -1);
        String ret =
                        "<INVSTMTMSGSRQV1>\n" +
                            "<INVSTMTTRNRQ>\n" +
                                "<TRNUID>8CCCCD65-13AF-4464-8990-5A0E108ACA3E\n" +
                                "<CLTCOOKIE>4\n" +
                                "<INVSTMTRQ>\n" +
                                    "<INVACCTFROM>\n" +
                                        "<BROKERID>" + brokerId + "\n" +
                                "<ACCTID>" + accountId + "\n" +
                            "</INVACCTFROM>\n" +
                                    "<INCTRAN>\n" +
                                        "<DTEND>" + format.format(now.getTime() ) + "\n" +
                                        "<INCLUDE>Y\n" +
                                    "</INCTRAN>\n" +
                                    "<INCOO>Y\n" +
                                    "<INCPOS>\n" +
                                        "<DTASOF>" + generateDate() + "\n" +
                                        "<INCLUDE>Y\n" +
                                    "</INCPOS>\n" +
                                    "<INCBAL>Y\n" +
                                "</INVSTMTRQ>\n" +
                            "</INVSTMTTRNRQ>\n" +
                        "</INVSTMTMSGSRQV1>\n";

        return ret;
    }


The second relates to how you are processing the return from generateAccountRequest(), when the person does not put in an account.  Fidelity accounts sometimes have non-numeric elements.  Your Regex pattern "<ACCTID>\\s*([0-9]*)\\s*<" means that if the person only has an account with a alpha in it that no account will be found, to the user id is used, which fails.

First ([0-9])* really should be ([0-9])+.  The * means 0 or more, + means 1 or more.  To include anything up to the next '<',  use "<ACCTID>([^<]+)".


Both changes I just tested on Fidelity, & it works.  I have multiple accounts, the first one listed started with an X.  Before the change I got the 2nd account, now I get the first.

2 (edited by whatIfSquared 2011-08-24 18:03:47)

Re: Some OFXGet Feedback

As read more on the OFX spec, another alternative which might be best is to not specify a <DTSTART> nor a <DTEND>.  It might be the way to go for a very generic tool like this.  I just commented  out  the <DTEND>, & ran tested once more & it works.

Re: Some OFXGet Feedback

Thank you for the excellent feedback.  You are correct that this was my first java program.  No doubt there are many things in there that would make an experienced java veteran cringe, but it seems to work more or less.

I will make your recommended changes.  I also plan to experiment with removing DTSTART and DTEND.  The trouble with OFX is that each server expects something slightly different.  I'll report on what I find.

Re: Some OFXGet Feedback

Jesse wrote:

Thank you for the excellent feedback.  You are correct that this was my first java program.  No doubt there are many things in there that would make an experienced java veteran cringe, but it seems to work more or less.

I will make your recommended changes.  I also plan to experiment with removing DTSTART and DTEND.  The trouble with OFX is that each server expects something slightly different.  I'll report on what I find.

It was not a knock.  I knew, because I read most of the forum & you said so in own of the threads.  Putting together & publishing a Java Web Start Application, that connects to multiple servers, parsing XML, & swing is a lot for a first app.

Re: Some OFXGet Feedback

No offense taken.  My only reason for saying that was to be clear that I know it isn't great java code and that I welcome any and all corrections.

I tested your suggestions and they work fine.  I will add a new version to the server in the next few days.

Re: Some OFXGet Feedback

Based on my experience - it looks like OFX responses from Fidelity only contain 3 months of transaction history, regardless of whether or not you include DTSTART/DTEND or if you specify wider date ranges. Are you seeing that as well, or am I doing something wrong?

Per the FAQ on Fidelity's site:

You can export your investment holdings, account balances, along with 90 days of transaction history for your personal accounts. Additionally, you can get transaction history up to 24 months.

Re: Some OFXGet Feedback

You are correct, My Program gets 3 months at a time as well.  Looking at possibly Fidelity documentation which you have highlighted, this is what was supposed to happen.  Jesse mentioned he is getting 3 mos from Scottrade at a time too.  I have programmed to walk back as far as possible by doing multiple statements. 

Not sure what program(s) you are using, since OFXGet did not even let you specify dates last time I ran it. The OFX specification was written with clients in mind that read going forward over time picking up new transactions from the last time a statement was pulled (see section 3.2.7).  I am doing this as well as successfully piecing together prior history from multiple calls.  If you are doing your own programming this is what you are going have to do.  Note, you cannot run multiple statements & paste them all together, and come up with a valid OFX file.

Re: Some OFXGet Feedback

A quick correction: I found that scottrade gives full transaction history.  Fidelity only gives 3 months no matter which start and end dates are chosen.  I'd like to know if there is a way to pull the full transaction history from Fidelity but I don't think it is possible.

Re: Some OFXGet Feedback

Jesse wrote:

A quick correction: I found that scottrade gives full transaction history.  Fidelity only gives 3 months no matter which start and end dates are chosen.  I'd like to know if there is a way to pull the full transaction history from Fidelity but I don't think it is possible.

Good to hear.  For either non-commercial applications like OFXGet, and / or where the data is pulled in one program for use in another, the single shot is a big deal.  Neither affect myself.

Question though what do you mean by "full"?  Is it every transaction ever done, or just over the last 24 months?  FYI, I am almost certain that U.S. Federal regulations require that customers have access to their data for 24 months, even if the account has been closed.

Secondly, why do people even want this information?  It is backward looking.  I am only going back to find the purchase price of existing holdings for future management.  Program sync with the actual account going forward is the real prize.