Integrate Your Expertise with the Data

Your data science model is only as good as the data you use.  You can get better, cleaner data by using DataSignals to streamline everything from data corrections to standardizing over reporting intervals.  But what about the data not published by the ISO?  And how does your own insight and knowledge fit into data science models?  This is where feature engineering comes in!

Feature engineering is the process associated with creating additional variables in your analysis based on your expertise and knowledge of the energy markets.  A quick example - many of the ISOs don’t publish net load, but net load is often just as important if not more important than each of the variables that make it up (actual load & renewable generation).  Other good examples of important user driven features you can create include lagged variables, leading variables, statistical summaries, binning/categorical feature creation, normalization and logging. 

Here is an ERCOT specific application of feature engineering to improve our understanding of the ERCOT price adder.  We started our analysis by bringing in three series from the API: 

  • Load

  • Online capacity

  • Adder

We can run some quick statistical analyses to summarize our data and build very basic histograms to learn our data trends. 

We can see similar patterns with online capacity and load. And also noticeable is how right-skewed the adder appears. But at this point, the data isn’t actionable. We still don’t know how changes in data drivers, like load or online capacity, can impact whether or not the adder is appearing in the given market interval. To start to understand this, we can look for a simple linear correlation between two of our items - load and online capacity and then shade by the adder.

What can we learn?

  • The relationship between load and online capacity exists

  • It is a negative correlation, but maybe it’s not as strong as we may have initially suspected based on just the histogram

  • There is a lot of noise when load is lower and capacity is higher

  • The adder appears to hit at some level where load is high and the online capacity is low 

We still need to dig further into this data to understand when we will expect adders on the ERCOT LMP.  For that, let’s start by establishing a new feature or variable - specifically a boolean indicator for whether or not the adder bound.  This will allow us to set thresholds for alerts, and early binding intervals.

Now we have a strong starting point for analyzing when the adder will hit:

  • Load needs to be above 55GWs and

  • Online capacity needs to be below 15GWs.  

There are some areas where load could be lower and the adder is even more likely. In this case  we want to further target load as a category and to drill into this more. For this, let’s create one more feature, load category.  This will have five values:

  • Extreme-valley

  • Valley

  • Mid

  • Peak

  • Super-peak

Creating this as a facet or additional partition on our data also allows us to easily display other data drivers.  Let’s bring in the ERCOT HASL data now. In this example, we’ve now updated our results to display HASL as the Y-Axis.

Now we can further break down our data to understand the relationship between HASL, Load and Capacity on the adder. Based on just these charts we can start to set thresholds for multi-conditional alerts in QuickSignals. We can even take this a step further by building machine learning models off of our data. But that’s for another blog. This is the first in a series of Data Science blog posts so please subscribe to our blog to get those sent to your inbox!

If you’re a Yes Energy customer and would like to receive code samples (in R) to try this on your own click here and we’ll send you the code. Unfortunately the code will not work if you aren’t a Yes Energy customer BUT if you’re interested in learning more we’d love to chat! Fill out this form and we’ll be in touch!

Riding the ERCOT Roller Coaster: A Review of Market Signals & what to watch as summer winds down

smaller roller coaster-01.png

The last few weeks of August have been a roller coaster in ERCOT. The ISO issued two Energy Emergency Alerts (EEA1) after not issuing one for nearly five years. The week of August 12th RT prices at North Hub printed over $8,000 three days and on peak prices averaged over $1500 two days. The week of August 19th was mostly quiet, but Thursday saw a few $2000-plus prints. We looked at how to predict this market volatility using Live Power and other market data accessible in the Yes Energy product suite.

Using Live Power Data to Predict Market Volatility

Using Live Power data and the results of our previous analysis, a pattern emerged that helps us predict the timing of large price spikes. T.H. Wharton consistently ramps several hours before the largest prints. Stryker Creek and DeCordova ramp much closer to the price blowouts. DeCordova has been a particularly strong indicator. During the wild week of August 12th there were two days when prices peaked at ‘only’ $2,000. On those days DeCordova ramped to less than 10% of capacity. 

The week of August 19th ERCOT was quieter, but Stryker Creek and DeCordova remained a strong signal. Strkyer Creek was off Monday and Tuesday when the highest prints were in the $200-$250 range. DeCordova was off Wednesday and Friday. The only day they both ramped was Thursday when we saw those $2,000 prints. These plants will be ones to watch through late summer in ERCOT. 

Using Other Market Data to Signal When ERCOT Market Volatility Will Occur 

Individual pieces of market data such as Load, Wind, and Reserve Indicators including Capacity Available to Increase SCED (Cap Inc. SCED) can all be useful in alerting market participants when the ERCOT market is getting tight and prices may be about to rise, see the below chart from the week of August 12. 

*ERCOT HASL - The maximum capacity a Generation Resource may be dispatched while maintaining its scheduled ancillary services. Calculated as the High Sustained Limit (HSL) minus the Ancillary Service Schedules for Responsive & Non-spin minus Ancillary Services Responsibility for Reg-up.

However you can create even more intelligent signals using our Formula Series and Multi Conditional Alerting capabilities in Quicksignals.

Instead of keeping an eye on Load, Wind, and Cap Inc. SCED separately, we can create a custom data series of Load-Wind in Formula Series and then using our Multi-Conditional alert functionality, set up the alert (detailed below*) that will let you know every time the market is getting tight in ERCOT and North Hub prices are about to rise.

*Multi-Conditional Alert Settings:

Condition 1: Load - Wind > 62,000 MW

Condition 2: Cap Inc. SCED < 2,800 MW

ERCOT Signal Now Even Quicker in QuickSignals

ERCOT ORDC image-01.png

Summary

Yes Energy has applied TradeNOW technology to ERCOT’s ORDC Adder. This data is now flowing faster to all QuickSignals dashboards. The ORDC is usually posted before LMP so having this data coming in as fast as possible is extremely important for ERCOT BalDay traders. 

What is the ERCOT ORDC Adder?

The ERCOT Operating Reserve Demand Curve (ORDC) Adder is an ERCOT-wide price adder that the ISO adds to all LMPs to incentivize new generation to come online during times of scarcity. 

The ORDC adder is made up of two separate adders, the Online Reserve Price Adder (ORPA) and the Online Reliability Deployment Price Adder (ORDPA). The ORPA was implemented in mid-2014 to account for the value of reserves based on the probability of reserves falling below the minimum contingency level and the value of lost load. The ORDPA was implemented in June 2015 as a mechanism to ensure that reliability deployments do not distort the energy prices.The ORPA is the more active of the two adders by far. Thus, the all-in LMP in ERCOT consists of the LMP + ORPA + ORDPA. 

Previously, only the LMP + Adders data for ERCOT Hub North was available in Yes Energy with faster TradeNOW speed. Now all of the components of ERCOT Hub North LMP are available, LMP, ORPA Adder, and ORDPA Adder.

If you are a QuickSignals customer all you need to do is make sure you have the correct data in your dashboard.

Table 1. Data type definitions for ERCOT Price Adder series that are now delivered using TradeNOW technology.

Table 1. Data type definitions for ERCOT Price Adder series that are now delivered using TradeNOW technology.

Not a QuickSignals customer? Click here to learn more!



Determining which ERCOT Power Plants to Watch this Summer: The Leading Indicators of Scarcity Prices -- updated 7.2019

ercot graphic-01.png

With ERCOT once again facing tight reserve margins, traders are turning their eyes to where the generation could fall short on days with high load. Yes Energy tasked our chief economist, Dr. Scott Holladay, to dig into our data. He found some key areas where traders should be focusing their attention this summer.

Background:

In March, ERCOT released its preliminary Seasonal Assessment of Resource Adequacy (SARA) report for Summer 2019. The report stated that “In all of the scenarios studied, we identified a potential need to call an energy alert at various times this summer.” An alert allows ERCOT to call for actions to alleviate the lack of generation capacity including:

  • demand response products

  • tapping neighboring regions for generation

  • asking consumers to conserve energy

ERCOT also has two Real-time energy price adders it can use to incentivize generation when reserves are low. Real-time adders have been assessed in less than half the days over the past year. Of the days where Real-time adders were applied, only 35 had adders that exceeded an average of $1/MWh over the full day. That small bucket of days had an outsized impact on PnL. Over the last year, Hub North has exhibited an average day-ahead bias of $3.30/MWh. Filtering to days with adders above $1/MWh on average across the entire day, Hub North had a Real-time bias of $0.50/MWh. As the adders increase to $10/MWh, that average Real-time bias grows to $20 per MW/h.

Using ERCOT Security Constrained Economic Dispatch (SCED) data, we can identify the relationship between generation and ERCOT adders. SCED data is the real-time market evaluation of offers to produce a least-cost dispatch of online resources. Some plants ramp in anticipation of an energy alert. Ramps at those plants indicate big adders are on the way. Other plants may curtail output, leading to an energy alert and big adders. With detailed generation and price data, it is easy to pick out the signals that adders are coming to the market.

However, ERCOT SCED data is released on a 60 day lag, so the relationships in the data might be easy to find, but they are hard to trade on. Using Live Power’s 1 minute generation data, traders can get a signal that the plants at the top of the gen stack are ramping and adders are likely on the way.

Correlation Arrow Image one color-01.png

Research & Findings:

Dr. Holladay leveraged Yes Energy’s DataSignals API and vast database to build a dataset to analyze in Python. He queried two years of SCED data for 21 key generators in ERCOT along with the ERCOT Price Adders. With this data, he computed the relationship between changes in the generation at the ERCOT power plants and the presence and strength of price adders.

The relationships were broken down into three categories: 15 minute, 30 minute and 60 minute. These categories indicate the relationship of the time from ramp to the price adders occurring. The 15 minute measures the relationship between changes in generation over the previous 15 minutes and the change in Real-time adders over the next 15 minutes. The 60 min score measures the relationship between a plant’s ramp over the past hour and Real-time adders. The higher the score, the stronger the correlation between changes in generation and changes in Real-time adders.

Plants with a high one hour score tend to ramp before the adders come in, providing a leading indicator to traders. Plants with a high 15 minute score tend to ramp shortly before adders enter the market. Plants with scores near zero don’t have a strong relationship with Real-time adders, so their changes in generation don’t provide much information. TH Wharton has a relatively high 60 minute score, but a low 15 minute score because it ramps to full capacity well before the adders come in.

According to Dr. Holladay’s research, here are four ERCOT Plants that give the best signal on future adders:

 
UPDATED Table for Post-01.png
 

Table 1: ERCOT plants that give the best signal on future adders. The 15 min score reports the correlation between changes in plant generation in the last 15 minutes and changes in adders in the next 15 minutes. The 30 minute score is the correlation between changes in generation over the last 30 minutes and changes in adders in the next 15 minutes. DeCordova is a newer plant monitored by Live Power and has recently been added to this table as it is a strong indicator of adders happening soon. Plants like VH Braunig and Strkyer Creek with high 60 minute scores are ramping well before adders come in.


HISTORICAL EXAMPLES:

Using our Time Series Analysis module, we were able to capture some recent examples of these relationships. Here’s a look:

Chart 1: Two recent examples of the correlation between Stryker Creek and Price Adders highlighting Stryker Creek’s strong correlation across all time categories

Chart 1: Two recent examples of the correlation between Stryker Creek and Price Adders highlighting Stryker Creek’s strong correlation across all time categories

Chart 2: A recent week in May that shows the strong correlation between V H Braunig and the Real-Time Adders.

Chart 2: A recent week in May that shows the strong correlation between V H Braunig and the Real-Time Adders.

Chart 3: T H Wharton’s correlation shows up the strongest after a longer time after ramp up as these recent examples show.

Chart 3: T H Wharton’s correlation shows up the strongest after a longer time after ramp up as these recent examples show.

5 Valuable Alerts for Power Traders this Summer

Power Traders are skilled at consuming large amounts of fast-paced data followed by making fast decisions. A critical tool for catching the “right” market signals are smart alerts.  Alerts that let traders know the moment certain conditions have been met and a trading opportunity is possible.

Alerting on a single condition can be useful and might be a tool that many traders use. But being able to set up alerts that are based upon multiple conditions means that traders can better filter out “false alarms” or just basic noise. Afterall, trading strategies are rarely dependent on one single criteria but a multitude of possible market drivers.


Here are 5 alerts we’ve set up using the Alerting function in QuickSignals but you could use the same parameters to set up alerts in whichever software solution you’re using to be alerted. These alerts help Power Traders filter out the noise and be the first to know when a market signal has appeared.

  1. ERCOT is at Risk of Energy Volatility

  2. Congestion is Directly Affecting PJM West Hub Prices

  3. Forecast Errors Reach a Set Limit

  4. Load & Wind Relationship Criteria

  5. Generation Ramp Indicates Changes in PJM West Hub RT LMP

Read on to learn how to implement these valuable alerts!


Alert #1: ERCOT is at Risk of Energy Volatility

This alert will let you know when there is risk of energy volatility in ERCOT and an increased potential scarcity pricing.

When the Capacity with Energy Offer Curve Available to Increase SCED Base Points (reported in MWs) crosses below 3000 and the total Regulation up Service* that has not been deployed (reported in MWs) crosses below 100 there is an increased risk of volatility. When these conditions occur, Traders will be alerted and can then dig in to determine if making changes in their positions is necessary.

Alert Type: Multi-Conditional

  • Condition #1: ERCOT Cap w/offer IncSCED <3000

AND

  • Condition #2: ERCOT Undeployed Reg-up <100

*The Regulation Up is an Ancillary Service that provides capacity that can respond to signals from ERCOT within five seconds to respond to changes in system frequency. Such capacity is the amount available above any Base Point but below the HSL of a Generation Resource and may be called on to change output as necessary throughout the range of Capacity available to maintain proper system frequency. A Load Resource providing Reg-up must be able to increase and decrease Load as deployed within its Ancillary Service Schedule for Reg-up above the Load Resource’s LPC limit.

Alert 1.gif

Alert #2: Congestion is Directly Affecting PJM West Hub Prices

PJM Traders will want to set this alert up to stay on top of changes in West Hub LMP that are related to congestion. Using the difference between the Real-Time LMP for Western Hub and the value of the Western Hub Tick Predictor, traders can stay on top of when there is a large discrepancy between the current and predicted next tick.  When seconds matter to your decision making, this alert can ensure you look up at your Dashboard to make fast trades.

Alert Type: Spread Value

  • Spread Value #1: PJM Western Hub LMP-RT

  • Spread Value #2: PJM Western Hub Tick Predictor *

  • Condition: Spread Value percent change increases or decreases by 5%.

    *A Yes Energy calculation using 15 second dispatch rates to calculate the next 5 minute price.

Alert #3: Forecast Errors Reach a Set Limit

Wind / Load Forecast errors for ERCOT, SPPISO & MISO can be catastrophic for portfolios. This alert provides insight into how accurate (or inaccurate) forecasts are shaping up in real time. This example uses SPPISO Wind Forecast but this same alert could be applied for any other type of forecast or ISO.

Alert Type: Single Value

Formula:

(100)(A-B)/B with the following variables:

A = SPPISO Real-Time Load

B = SPPISO Forecast Load

Once this Formula Series is created, you can then set up an Alert for when this error meets the set criteria. The level of error that is worthy of an alert will depend upon the selected data series (Wind, Load, Weather etc) and the ISO. Here’s an example for Wind Error:

  • Single Value: Wind Error Alert (created using the Formula Function above)

  • Condition: Threshold < 0 (indicates wind is coming in below forecast)

*Pro Tip: There are many variations of this that could be useful, including, for example, Load Forecast Error between zones in ERCOT to help predict congestion potential (i.e. North -> Houston)

Alert #4: Load & Wind Relationship Criteria

This Alert is especially useful for SPP, MISO & ERCOT where wind can make or break a position. For this alert you will establish thresholds for both wind and load to let you know when the wind is low and load is high.

Alert Type: Multi-Conditional (ERCOT example)

  • Condition #1 - Load - Real-Time Actual > 65,000 MW

AND

  • Condition #2 - Wind Real-Time Generation < 2,000 MW

Alert #5: Generation Ramp Indicates Changes in PJM West Hub RT LMP

Generators ramping up or down can indicate a potential change in pricing. For this example we have identified a generator that, when it ramps, often indicates changes in PJM Western HUB Real-Time LMP. Setting the alert using Live Power Real-Time 1 minute generation data ensures that you know the minute a generator begins to ramp giving you a jump on your decision making.

Alert Type: Multi-Conditional

  • Condition #1 -Western HUB Real-Time LMP > 75$/MWh

AND

  • Condition #2 - Bath Co Real-Time Generation > 0 MW