Google Cloud cost management – how to save money on your infrastructure?

Introduction

Cost management is a very important aspect of every cloud solution. Having good expense visibility and being able to control and optimize costs is essential for each business using its cloud infrastructure to make sure it provides maximum value with the lowest costs possible.

In this article we provide information on the billing model of Google Cloud as well as the different ways of managing costs i.e. dashboards, cost breakdowns and alerts. We also add some examples on how you can optimize your GCP infrastructure costs.

Google Cloud billing

Resource overview

When you’re just getting started with the Google Cloud and the scale is still small, it’s quite easy to get a sense of the spendings, but as the number of projects and resources grow it’s very useful to use the GCP billing section to help you manage and optimize the expenses.

To understand how the costs are managed we first need to get familiar with the GCP resource structure. Everything in your Google Cloud ecosystem is under your domain and organization. Domain is the primary identity of your organization and establishes your company’s identity with Google services. An organization on the other hand is the root node of the Google Cloud hierarchy of resources, as it allows central management of your cloud services and users’ access to them.

Projects are used to group any service-level resources such as BigQuery or Compute Engine. You can use projects to represent logical projects, teams, environments or other connections that map to a business structure. Folders however are a grouping mechanism that can contain projects or other folders. You can also have labels on your resources to categorize them. It allows you to track costs at a granular-level so you can analyze your charges by a label in the billing system.

A Cloud Billing account is set up in Google Cloud and is used to define who pays for a given set of Google Cloud resources. It can be linked to one of more projects and it tracks all of the costs incurred by your Google Cloud usage. Each account has its own roles and permissions, so you can manage users for billing related functions. A Cloud Billing account is connected to a Google Payments Profile, which includes a payment instrument, defining how you pay for your charges.

Billing visualizations

The first step of setting up the cost management for the organization is the configuration of a billing account. Once we have the billing account set-up we can obtain all the spendings billed by Google in the Billing section in GCP. Apart from the simple monthly breakdown we can create a report using several filters or grouping by service or project.

Billing budgets and alerts

When you have a large cloud infrastructure, probably the last thing you want to see is a surprisingly high bill that is way over the budget set by the management. To avoid this you can set up budgets and alerts in Google Cloud to closely monitor your costs and take action when needed.

A budget allows you to track the actual Google Cloud spend against the planned spend. It lets you trigger a notification (e.g. via email) when your costs hit or are forecasted to reach a specified amount. You can specify the scope of the budget, e.g. you can set it up for an entire billing account or narrow the scope to the specific projects and GCP products. 

When setting a budget it’s also very useful to set an alert to get notified when the costs are high. When your actual (or forecasted) costs reach the specified percentage of the budget email alerts are sent to the set list of recipients. The budget section also allows sending notifications on the specified Pub/Sub topic which can help you set up advanced cost controls, e.g. by automating the cost management tasks or setting cloud functions to automatically shut down extensively used VMs.

Default functionality and different uses for budget notifications.

Source: https://cloud.google.com/billing/docs/how-to/budgets

Cost optimisation examples

Setting up cost control with quotas

In this example we’re going to show how to use quota to further control your spendings by setting a limit on the number of concurrent resources on the project. When the quota is exceeded Google Cloud blocks the access to the specific resource and the task that you’re trying to perform fails.

To review quotas for your project go to IAM & admin -> Quotas in the GCP console. Here you have access to all the quotas for GCP products and services available. 

To demonstrate how quotas work in practice we’re going to set the limit for the number of VM instances that can be created in asia-east1 region. In the Quotas page Filter select Service: Compute Engine API and Quota: VM instances. Then we can change the quota for the region, here we changed it to 2.

Now we can check the implications of the change in quota. Go to Compute Engine -> Create an instance -> select asia-east1 as region -> select a small machine e.g. e2-micro and hit Create. Repeat the step once again so you have two VMs created. Once they are provisioned we can go back to the Quotas section and check the changes. We notice that we have reached our quota limit of 2 VMs in the asia-east1 region, which means we are not allowed to create any more machines here.

If we try to create another VM in this region we’ll get the following error:

BigQuery cost optimisation

In the second example we’ll be discussing the costs in BigQuery and how we can control it.

BigQuery which is a serverless and highly scalable data warehouse offers two pricing options for running queries:

• on-demand which is based on the amount of data processed by each query you run. It’s the most flexible option, because you only pay for the queries that you run
• flat-rate for which customers purchase dedicated resources and are not charged for individual queries. This is best for customers with fixed budgets.

Note: Remember that the Google Cloud also charges you for the BigQuery storage. For more details visit the pricing page: https://cloud.google.com/bigquery/pricing  

When the on-demand option is chosen we need to make sure that we don’t overpay for our queries. For the cost optimisation we can use quotas mentioned in the example above but this time we can limit the amount of processed data e.g. you can set the limit to 2 TiB per user per day and the bill will be limited to 10$ per user for each day (pricing $5 per TB valid for Sep. 2022).

The other way of saving money on BigQuery bills is query optimisation. Google provides the best practices for data extraction such as: avoiding SELECT * statements, using preview options and using partitioned or clustered tables.

The significance of the query optimisation can be shown on crypto_bitcoin public dataset. It contains two tables: blocks and transactions with the following schema:

Let’s say we need to join those tables using the block hash column, filter the results for one week from 2021-01-01 to 2021-01-07 and extract only specific columns.

If we use the following query, which simply joins the tables and applies the filter on the timestamp column we process 1.52 TB:

SELECT *

FROM 

  `bigquery-public-data.crypto_bitcoin.transactions` transactions

JOIN

  `bigquery-public-data.crypto_bitcoin.blocks` blocks

ON

  transactions.block_hash=blocks.`hash`

  AND transactions.block_timestamp=blocks.timestamp

WHERE

  timestamp between '2021-01-01' AND '2021-01-07'

However if we notice that both of the tables are partitioned, we can add the additional filter on the timestamp month before joining the tables. We can also select only the specific columns we are interested in and altogether by running the following query we process only 1.46 GB which is more than one thousand times cheaper than before:

SELECT 

  transaction_hash,

  block_hash,

  transaction_size,

  block_size,

  block_timestamp_month,

  transactions.block_timestamp

FROM

(

  SELECT `hash` as transaction_hash,

    size as transaction_size,

    block_hash,

    block_timestamp,

    block_timestamp_month

  FROM `bigquery-public-data.crypto_bitcoin.transactions`

  WHERE block_timestamp_month = '2021-01-01'

) transactions

JOIN

(

  SELECT `hash` as block_hash,

    size as block_size,

    timestamp as block_timestamp

  FROM `bigquery-public-data.crypto_bitcoin.blocks`

  WHERE timestamp_month = '2021-01-01'

  AND timestamp between '2021-01-01' AND '2021-01-07'

) blocks

USING(block_hash)

Conclusion

Understanding the billing model in Google Cloud is essential for using its services as it allows to properly manage costs while taking advantage of all the cloud capabilities. To help with the cost management, users can visualize the spending using the billing reports and also use the budgets with alerts to receive notifications when the costs are getting high. To further control the costs you can set quotas to restrict the amount of resources that the project can use. Finally, remember to optimize your BigQuery queries to further decrease the spendings.

Do you need help with setting up your Google Cloud services with predictable costs and no unexpected bills? Contact us to learn more.