Meet the Scientists Mining Big Data to Predict the Weather
In San Diego the palm trees sway in the warm winter’s breeze, and the sun shines a reliable 40% of the year. It’s an odd location for a startup that tries to predict weather extremes. But in an office on a hill above the Pacific Ocean, a small company called EarthRisk Technologies is doing just that: crunching through massive amounts of historical weather data to determine the odds of really bad heat waves and cold snaps, all to give its customers—utilities, power producers and energy traders - the edge they need in playing dice with Mother Nature.
Started by a hedge fund meteorologist turned entrepreneur named Stephen Bennett, EarthRisk calculates the probability of market-changing bad weather up to 40 days in advance. That’s twice as long as the most farseeing conventional forecasts. On a visit to EarthRisk I’m taken by Bennett into “the Matrix,” the Web interface that shows these projections on grids and U.S. maps. EarthRisk defines extreme weather, for now, as a period of temperatures in the top or bottom 10% of the historic range that affects at least 10% of the area of a specific region of the U.S.
A Pacific Northwest utility dependent on hydropower uses EarthRisk’s TempRisk service to predict snowmelt. The U.S. operations of Iberdrola Renewables, the Spanish energy giant, taps the service to plan its natural gas purchases. “If a client knows an extreme weather event is coming they can lock in a price on natural gas or trade some power and get it cheaper,” says John “JP” Plavan Jr., EarthRisk’s chief executive.
Last fall, for instance, the consensus was that the coming winter would be cold, and natural gas prices rose accordingly. But as EarthRisk’s software crunched the historical numbers -performing an initial 82 billion calculations - its projections from November showed a warming trend in the Midwest and East, a trajectory that would accelerate as the winter wore on. (Each day the company’s software runs through 4 million calculations to update its forecast.) As EarthRisk predicted, this winter had the second-highest number of extreme heat events and the lowest number of extreme cold events since 1948. Natural gas prices plummeted.
“[EarthRisk] was instrumental in helping us ‘go against the grain’ late in fall,” says David Gold, an Iberdrola meteorology manager in Houston, via e-mail. EarthRisk says its accuracy rate for the 2011-12 winter was 76% when the probability of an extreme heat event was 40% or more.
Iberdrola was the only one of EarthRisk’s 20-odd clients willing to talk to FORBES. In the energy business they apparently prefer to keep their weather underground. Nor does EarthRisk disclose how much clients pay for the subscription service. Plavan says the startup’s annual revenue is “a couple of million” dollars.
Most federal research money for mining weather data goes to improve the accuracy of short-term forecasts to protect lives and property. “The average taxpayer doesn’t need to know the weather three weeks from now. They need to know if a tornado is about to run over their house in ten minutes,” says Bennett, 38. Research on long-term forecasts was being done mostly at research universities with giant supercomputers.
The advent of cheap and expandable cloud computing has liberated researchers anywhere to play with giant data sets. EarthRisk’s models are based on probability theories that date back to the late 1970s, when you literally had to be a rocket scientist to get access to the computing power necessary to put theory into practice.