View on GitHub

Fech

A Ruby parser for electronic candidate, PAC and party campaign filings from the Federal Election Commission.

Download this project as a .zip file Download this project as a tar.gz file

Fech

Fech is a Ruby library for downloading and parsing electronic campaign finance filings from the Federal Election Commission. It supports filings from presidential and U.S. House candidates, F.E.C.-registered political action committees and most party committees (everything except U.S. Senate candidates and two senatorial party committees). Fech currently commits to supporting the F.E.C.'s filing formats back to version 3.0, where applicable.

Fech is tested under Ruby versions 1.8.7, 1.9.2, 1.9.3 and 2.0.0. The latest version is 1.6.0. Please consult the CHANGELOG for details on the latest release.

Note: as of January 1, 2014, only Fech version 1.5.2 or higher will properly retrieve electronic filings. Please update your applications accordingly.

Bug reports and feature requests are welcomed by submitting an issue. Please be advised that development is focused on parsing filings, which may contain errors or be improperly filed, and not on providing wrappers or helper methods for working with filings in another context. To get started, fork the repo, make your additions or changes and send a pull request.

Fech is an open source project of The New York Times.

Fech in Action

Pronunication guide

It's "fetch", with a soft "ch" sound. There are no other acceptable pronunciations.

Installation

Install Fech as a gem:

gem install fech

For use in a Rails 3 application, put the following in your Gemfile:

gem 'fech'

then issue the bundle install command.

Getting Started

Start by creating a Filing object that corresponds to an electronic filing from the FEC, using the unique numeric identifier that the F.E.C. assigns to each filing. You'll then have to download the file before parsing it:

filing = Fech::Filing.new(723604)
filing.download

Optional arguments: pass in a :quote_char argument, and Fech will use it as a quote character to parse the file (see details below). Specify the :download_dir on initialization to set where filings are stored. Otherwise, they'll go into a temp folder on your filesystem.

CSV Parsers

Not all CSV Parsers are created equal. Some are fast and naive, others are slow but handle malformed data more reliably. By default, FasterCSV is used, but you can pass in your own choice of parser with the :csv_parser option:

  filing = Fech::Filing.new(723604, :csv_parser => Fech::CsvDoctor)

Fech::CsvDoctor inherits from FasterCSV, and helps deal with certain types of malformed quotes in the data (see: Handling malformed filings). You could pass in any subclass with special handling.

Summary Data

To get summary data for the filing (various aggregate amounts, stats about the filing):

  filing.summary
  => {:coverage_from_date=>"20110101", :coverage_from_date=>"20110301", ... }

Returns a named hash of all attributes available for the filing. (Which fields are available can vary depending on the filing version number.)

Accessing specific line items

To grab every row in the filing of a certain type (all Schedule A items, for example):

  filing.rows_like(/^sa/)
  => [{:transaction_id=>"SA17.XXXX", :contribution_date>"20110101" ... } ... ]

This will return an array of hashes, one hash for each row found in the filing whose line number matches the regular expression you passed in. (SA17s and SA18s would both be returned in this example). You can also pass in strings for exact matches.

When given a block, .rows_like will yield a single hash at a time:

  filing.rows_like(/^sa/) do |contribution|
    contribution.transaction_id
    => {:transaction_id=>"SA17.XXXX", :contribution_date>"20110101" ... }
  end

Since .rows_like loads all transactions matching the corresponding line number into memory, for very large filings (for example, for presidential campaigns) you may want to use .enum_for instead:

  filing.enum_for(:rows_like, /sa/)
  => [{:transaction_id=>"SA17.XXXX", :contribution_date>"20110101" ... } ... ]

Usage

Accessing specific fields

By default, .rows_like will process every field in the matched rows (some rows have 200+ fields). You can speed up performance significantly by asking for just the subset of fields you need.

  filing.rows_like(/^sa/, :include => [:transaction_id]) do |contribution|
    contribution
    => {:transaction_id=>"SA17.XXXX"}
  end

Miscellaneous Text

Form 99 submissions contain no structured data but miscellaneous text communicated between committees and the F.E.C. The text is accessible via the summary:

  filing.summary[:text]
  => "This is in response to Requests for Additional Information received on 9 and 10 February 2012...."

Raw data

If you want to access the raw arrays of row data, pass :raw => true to .rows_like or any of its shortcuts:

  filing.contributions(:raw => true)
  => [["SA17A", "C00XXXXXX", "SA17.XXXX", nil, nil, "IND" ... ], ... ]

  filing.contributions(:raw => true) do |row|
    row => ["SA17A", "C00XXXXX", "SA17.XXXX", nil, nil, "IND" ... ]
  end

The source maps for individual row types and versions may also be accessed directly:

  Fech::Filing.map_for("sa")
  Fech::Filing.map_for(/sa/, :version => 6.1)
  => [:form_type, :filer_committee_id_number, :transaction_id ... ]

You can then bypass some of the overhead of Fech if you're building something more targeted.

Converting / Preprocessing data

For performing bulk actions on specific types of fields, you can register "translations" which will manipulate specific data under certain conditions.

An example: dates within filings are formatted as YYYYMMDD. To automatically convert all Schedule B :expenditure_date values to native Ruby Dates:

  filing.translate do |t|
    t.convert(:row => /^sb/, :field => :expenditure_date) { |v| Date.parse(v) }
  end

The block you give .convert will be given the original value of that field when it is run. After you run .convert, any time you parse a row beginning with "SB", the :expenditure_date value will be a Date object.

The :field parameter can also be a regular expression:

  filing.translate do |t|
    t.convert(:row => /^f3p/, :field => /^coverage_.*_date/) { |v| Date.parse(v) }
  end

Now, both :coverage_from_date and :coverage_through_date will be automatically cast to dates.

You can leave off any or all of the parameters (row, field, version) for more broad adoption of the translation.

Derived Fields

You may want to perform the same calculation on many rows of data (contributions minus refunds to create a net value, for example). This can be done without cluttering up your more app-specific parsing code by using a .combine translation. The translation blocks you pass .combine receive the entire row as their context, not just a single value. The :field parameter becomes what you want the new value to be named.

  filing.translate do |t|
    t.combine(:row => :f3pn, :field => :net_individual_contributions) do |row|
      contribs = row.col_a_individual_contribution_total.to_f
      refunds = row.col_a_total_contributions_refunds.to_f
      contribs - refunds
    end
  end

In this example, every parsed Schedule A row would contain an attribute not found in the original filing data - :net_individual_contributions - which contains the result of the calculation above. The values used to construct combinations will have already been run through any .convert translations you've specified.

Built-in translations

There are two sets of translations that come with Fech for some of the more common needs:

You can mix these translations into your parser when you create it:

  filing = Fech::Filing.new(723604, :translate => [:names, :dates])

Just be aware that as you add more translations, the parsing will start to take slightly longer (although having translations that aren't used will not slow it down).

Aliasing fields

You can allow any field (converted, combined, or untranslated) to have an arbitrary number of aliases. For example, you could alias the F3P line's :col_a_6_cash_on_hand_beginning_period to the more manageable :cash_beginning

  filing.translate do |t|
    t.alias :cash_beginning, :col_a_cash_on_hand_beginning_period, :f3p
  end

  filing.summary.cash_beginning ## filing.summary.col_a_cash_on_hand_beginning_period.
  => true

We found it useful to be able to access attributes using the name the fields in our database they correspond to.

Unofficial Senate Filings

Fech also supports unofficial electronic filings submitted voluntarily by Senate campaign committees. While Senate candidates are not required to file electronically, some do. To access a Senate filing, use Fech::SenateFiling:

filing = Fech::SenateFiling.new(853)

You can then work with the filing as you would any other.

Comparing filings

You may want to compare two electronic filings to each other, particularly an original filing and its amendment, since amendments are a full replacement of the original filing. Fech has a Comparison class that allows you to compare the summary or a specific schedule of itemizations between two filings. The use of Comparison requires that you have created both Filing objects and downloaded them:

  filing_1 = Fech::Filing.new(767437)
  filing_1.download
  filing_2 = Fech::Filing.new(751798)
  filing_2.download
  comparison = Fech::Comparison.new(filing_1, filing_2)
  comparison.summary

Comparison's summary method returns a hash of summary fields whose values have changed in filing_2. The schedule method accepts either a symbol or string representing the schedule. So, for Schedule A (itemized contributions): 

  comparison.schedule(:sa)

The result will be an array of Fech contribution objects that differ in any way. If an amendment updates the employer or occupation of a contributor, for example, comparison.schedule(:sa) would return that contribution. If the array is empty, no itemized records changed.

Handling malformed filings

Some filers put double quotes around each field in their filings, which can cause problems if not done correctly. For example, filing No. 735497 has the following value in the field for the name of a payee:

"Stichin" LLC"

Because the field is not properly quoted FasterCSV and CSV will not be able to parse it, and will instead raise a MalformedCSVError.

To get around this problem, you can pass in Fech::CsvDoctor as the :csv_parser when initializing the filing (see: CSV Parsers). It will then take the following steps when parsing the filing:

If you'd like to parse an entire filing using a specific CSV quote character, see: Quote character.

Quote character

By default, Fech, like FasterCSV and CSV, assumes that double quotes are used around values that contain reserved characters.

However, the parsing of some filings might fail because of the existence of the default quote character in a field. To get around this, you may pass a :quote_char argument, and Fech will use it to parse the file.

For example, filing No. 756218 has the following value in the field for the contributor's last name:

O""Leary

Attempting to parse this filing with Fech fails because of illegal quoting.

Since the FEC doesn't use a quote character in its electronic filings, we don't need a quote character at all. One way to achieve this is to use "\0" (a null ASCII character) as :quote_char:

  filing = Fech::Filing.new(756218, :quote_char => "\0")

Warnings

Filings can contain data that is incomplete or wrong: contributions might be in excess of legal limits, or data may have just been entered incorrectly. While some of these mistakes are corrected in later filing amendments, you should be aware that the data is not perfect. Fech will only return data as accurate as the source.

When filings get very large, be careful not to perform operations that attempt to transform many rows in memory.

Supported row types and versions

The following row types are currently supported from filing version 3 through 8.0: