The AWS RI Marketplace – a ghost town ?

In 2009 AWS launched their EC2 Reserved Instance (RI) pricing model , providing a significant discount compared to the on-demand pricing model if you are willing to commit to 1 or 3 years of usage.

Cost management on AWS is a hot topic, with hundreds of blogs on making the right choice of RI’s. A new market of AWS cost management tools emerged, with tool vendors promising massive ROI. Based on AWS billing & usage data analysis, these tools will provide RI recommendations. AWS’s own Trusted Advisor will also provide this kind of analysis, as included in their higher level support plans.

I highly recommend my SDL FredHopper colleague David Costa’s presentation on the topic, as he basically wrote the (internal) book on how to do this at scale: Cost Optimization at Scale.

While you can shift RI reservations, after you made them;

• Switch Availability Zones within the same region
• Change between EC2-VPC and EC2-Classic
• Change the instance size within the same instance type

There are several use-cases where you would want end your reservation before the reservation end date;

• Switch Instance Types.
• Buy Reserved Instances on the Marketplace for your medium-term needs.
• Relocate region.
• Bad capacity management.
• Unforeseen business or technology changes.

While there are significant savings to be had when using RI’s, one has to be aware that with AWS RI’s you would basically be stuck with the reoccurring monthly fee, almost as if the instances where CAPEX sunken hardware capital.

To solve this issue, AWS has launched a few solutions in the last 4 years:

• RI marketplace was launched in Sept 2012 as a way to sell your remaining reservation time on a specific instance to another AWS customer.
• Moving to other model in Dec 2014, away from ‘Usage’ to Up-front payment.
• Adding scheduled type RI’s in Jan 2016
• Adding convertible type RI’s in Sept 2016

This now offers the following RI options:

• Standard Reserved Instances: These provide the most significant discount (up to 75% off On-Demand) and are best suited for steady-state usage.
• Convertible Reserved Instances: These provide a discount (up to 45% off On-Demand) and the capability to change the attributes of the Reserved Instance as long as the exchange results in the creation of Reserved Instances of equal or greater value. Like Standard Reserved Instances, Convertible Reserved Instances are best suited for steady-state usage.
• Scheduled Reserved Instances: These are available to launch within the time windows you reserve. This option allows you to match your capacity reservation to a predictable recurring schedule that only requires a fraction of a day, a week, or a month.

As there are still enough use-cases left for wanting to sell your RI’s, one would expect a vibrant AWS RI marketplace.

A few months ago I was looking for a larger quantity of RI’s based on a common instance type (like M3.Large), but for business reasons the preferred reservation time was longer than 1 year but shorter than 3 years.
When starting a normal RI reservation process in the AWS console , one will be presented with the option to buy someone else his reservation if there is any applicable availability. But non where available for my instance type in the given AWS Region. After going in and out of all the Region’s, I concluded;

• There is no easy overview of available instances on the marketplace using the normal AWS console.
• Hardly any RI’s where offered in the instance types that I used. The marketplace seemed a ghost town…

4 years after the launch of the marketplace it still seems in beta, at least by judging the page header at their marketplace documentation

Luckily AWS offers excellent API’s for mining their data so I build myself a dashboard. The dashboard is accessible here at http://ri-market.janwiersma.com/ .

So current state, as a snapshot on Nov 8th 2016;

• Total instance offers: 870
• Most offered Instance type: m1.small (19%), t2.micro (8%)
• Most offered from AZ: US-East-1a (17%), US-West-1b (12%).
• Most offered OS: Linux (74%)
• Duration left on RI’s: ~2,5yrs max to ~30days min.

While during my data analysis the total instances offered spiked at ~1250 at some point, the volume still seems very low. The total amount of servers that AWS runs is a big mystery but is speculated to be between 1.5 and 5.6 million . As conservative virtualisation ratios are 6:1 , we can safely assume there are millions and millions of instances on AWS.

Why is the volume so low ?

• A. AWS customers are excellent at capacity management; they never need to sell RI’s.
• B. AWS customers are bad at capacity management; they just don’t do RI’s.
• C. AWS customers are very bad at housekeeping; they are just not selling their unused RI’s.
• D. Given the current reservation options, AWS customers only need to sell RI’s when new major instance type family is introduced. That has been a while.
• E. AWS customers need a US bank account to sell RI’s; this is a roadblock for many. By far most RI’s are offered in US Regions.

Based on my personal experience working in IT and talking to my peers, I know (A) will not be true for most companies. Given the continuing success of cost management tools and their ROI dependence on the lack of RI’s in a given company, (B) should be true for at least 50% of the AWS customers. ( C ) will be true for more than 50% of AWS customers, as we seen the same management issues during the start of virtualisation in VM-sprawl. (E) is definitely a roadblock as you either need to be a US company or have a US branch office. It allows only larger companies with some global presence to actually use the selling service. (D) is harder to judge, but we may see evidence of this when AWS announces their next instance families. This is still the most common mentioned use-case to sell RI’s, by my peers.

I will continue to track the AWS RI Market overview for the next year and share any significant findings on my blog.

Scripts and config on how I build the dashboard can be found on my Bitbucket repo.

More reading: 2014 RI-market analysis done by the Supply Frame guys can be found here.