Automating Utility Data Collection: Portals, APIs, Green Button, and the Gaps Bill Parsing Fills

Every energy analytics program starts with the same unglamorous problem: getting the data in. Before you can benchmark a portfolio, validate a tariff, or model savings, you need consumption and cost history for every meter, every account, and every fuel, in a shape a machine can read. In practice that data lives in half a dozen places and arrives in half a dozen formats. Some of it comes through clean web APIs. Some of it comes as a PDF a property manager forwarded from their inbox. Choosing how to collect it is one of the most consequential decisions an energy data team makes, because the collection method sets the ceiling on coverage, freshness, and how much manual work you carry forever.

This article maps the full utility data collection landscape: manual portal downloads, utility developer APIs, the Green Button family, third-party aggregators, screen scraping, and PDF bill parsing. The goal is to be honest about where each method wins, where it breaks, and why almost every serious program ends up running two or three of them at once.

The five ways to get utility data

There are really only a handful of mechanisms, and every vendor and in-house pipeline is some blend of them. It helps to name them plainly before comparing tradeoffs.

1. Manual portal downloads

Someone logs into each utility's customer or commercial portal, navigates to billing or usage history, and exports a CSV, XML, or PDF. This is where most programs begin because it needs no integration work. It is also where most programs stall: a mid-size portfolio can touch dozens of utilities, each with its own login, export limits, and file layout. Coverage is total in theory (if the account exists, a human can usually reach it) but the labor scales linearly with the number of accounts, and the data is only as fresh as the last time someone remembered to log in.

2. Utility developer APIs

A growing number of utilities publish their own developer programs, and large data users can sometimes negotiate direct feeds. When available, a utility API is the cleanest option: structured records, defined update cadence, and no screen to break. The catch is fragmentation. There is no single national utility API, each program has its own authentication, rate limits, and data model, and coverage is uneven. A portfolio spread across many service territories will find that only a fraction of its utilities offer a usable programmatic feed.

3. Green Button (Download and Connect My Data)

Green Button is the closest thing to a shared standard for consumer and commercial energy data in North America. It was first announced in September 2011 as a US government initiative and is now maintained by the Green Button Alliance, a nonprofit trade organization founded in 2015 that runs testing and certification. It comes in two flavors that people routinely confuse. Download My Data (DMD) lets a customer export a standardized file from their utility's site. Connect My Data (CMD) is the automated version: a third party requests ongoing access through a RESTful web service, and the utility pushes or serves the data without a human re-downloading anything.

4. Third-party aggregators

Aggregators sit between you and many utilities, presenting one API while they handle the per-utility authorization, formats, and quirks behind the scenes. They typically combine Green Button Connect, utility partnerships, and portal automation into a single integration. For teams that do not want to maintain connections to hundreds of utilities, this is often the pragmatic middle path, at the cost of a dependency and a per-account or per-call fee.

5. Screen scraping and RPA

When there is no API and no Green Button support, automation can still drive the portal: a script or robotic process automation (RPA) bot logs in as the user, clicks through, and captures the export. This extends automated coverage to utilities that offer nothing programmatic. It is also the most brittle method. Any portal redesign, new captcha, or multi-factor prompt can break the flow, and it depends on the customer sharing credentials, which many will not do.

How Green Button Connect and OAuth actually work

Because Connect My Data is the backbone of most automated feeds, it is worth understanding the mechanics. Green Button data is exchanged in the Energy Services Provider Interface (ESPI) format built on the NAESB REQ.21 standard, delivered using the Atom syndication format for metadata with OAuth 2.0 for third-party authorization. OAuth 2.0 is the same consent pattern you meet when an app asks to use your account without seeing your password, defined in the IETF specifications RFC 6749 and RFC 6750.

In Green Button terms, the utility is the Data Custodian and your platform is the Third Party. The customer authorizes the share once, and the custodian issues tokens that govern access. The Green Button Alliance token documentation describes an access_token allocated by the Data Custodian for an individual account authorization, a refresh_token issued at authorization time and used to renew an expired access_token, and a client_access_token used by third-party applications to reach bulk or multiple authorization subscriptions. The practical payoff is that once a customer consents, the feed keeps flowing without anyone logging in again, which is exactly what manual downloads cannot promise. Third parties that want to build against it can start from the Green Button documents library and developer specifications.

Comparing the methods

No single method wins on every axis. The table below is the mental model we use when deciding how to onboard a new set of accounts. Weigh coverage (can you even reach the account), latency (how fresh the data is), granularity (interval versus monthly), and maintenance burden (how often the connection breaks).

Where third-party aggregators fit

Aggregators earn their keep when your portfolio crosses many service territories and you have neither the appetite nor the staff to maintain direct integrations to each one. Providers publish developer documentation for their Green Button and OAuth flows, for example UtilityAPI's Green Button OAuth docs, and expose a single consent-and-retrieve pattern across their coverage. The tradeoffs are worth naming: you inherit the aggregator's coverage map, its pricing model, and its latency, and you are trusting a middle party with the authorization relationship. For many programs that is a fair trade. For others, especially those with a small number of very large accounts, direct utility feeds plus targeted bill parsing come out cheaper and more controllable.

Why PDF bill parsing still matters

It is tempting to assume that once APIs and Green Button mature, the PDF bill goes away. It does not, for three durable reasons.

• Coverage of last resort. Some utilities offer no API, no Green Button Connect, and a portal that resists automation. The one artifact you can always get is the bill itself, because the customer already receives it.
• Charge-level detail. Interval feeds are excellent for consumption but often thin on the cost side. The bill is the authoritative record of every line item: demand charges, riders, taxes, deposits, and true-ups that a usage feed may never expose.
• Historical backfill. When you onboard a new account, you frequently need one to three years of history that no live feed will replay. A stack of past PDFs is often the only way to reconstruct the baseline.

The nuance is that a scanned or exported bill is not structured data. Turning it into clean records is a reasoning problem, not just character recognition, because the same charge can be labeled differently across utilities and the layout changes without notice. That distinction is worth its own read in our piece on why utility bills need reasoning, not OCR.

Collected is not the same as usable

The trap teams fall into is treating collection as done once bytes arrive. A feed can deliver on time and still hand you data you cannot trust. Common issues that survive collection and only surface later:

• Estimated reads that a later actual read corrects, quietly changing a month you already reported on.
• Unit and time-zone mismatches between an interval feed and the billed total, so the numbers never reconcile.
• Overlapping or gapped billing periods when accounts change rate class or meters get swapped.
• Cost fields present in the PDF but absent from the usage API, so cost and consumption tell different stories.
• Account and meter identifiers that differ across sources, making whole-building rollups impossible without a reconciliation step.

A serious collection layer plans for reconciliation from day one, because the value of the data is only realized after it is validated and stitched into whole-building history. That quality step is the difference between a dataset you can report to a regulator or a board and one you have to caveat.

How MartinAI helps

MartinAI treats collection and validation as one workflow rather than two disconnected steps. Instead of forcing a choice between a clean API and a messy PDF, it blends the sources: where a utility offers Green Button Connect or a developer API, it takes the structured feed; where the only artifact is a bill, it reads that bill with reasoning that understands charge structures rather than guessing at pixels. The output is whole-building, meter-level history with consumption and cost aligned to the same periods and units.

The platform reconciles interval data against billed totals, flags estimated reads and period gaps, and normalizes account and meter identifiers so a portfolio rolls up cleanly across utilities and fuels. For teams, that means less time maintaining brittle connectors and chasing down why two sources disagree, and more time doing the analysis the data was collected for: benchmarking, tariff validation, compliance reporting, and cost control. The point is not the collection mechanism, it is arriving at data you can defend.

Conclusion

There is no single best way to collect utility data, and any program that pretends otherwise ends up with blind spots. Portal downloads give coverage but not scale. Utility APIs and Green Button Connect give clean automation where they exist. Aggregators trade a dependency for breadth. Screen scraping extends reach at the cost of fragility. PDF parsing fills the gaps the others leave, especially for charge detail and historical backfill. The winning strategy is to run the right blend and, critically, to validate everything into one reconciled dataset before anyone builds an analysis on top of it.