FMCG

Providing a Data Platform solution for a global leader in GCP industry to enable a holistic view of each customer in near real-time

Data Analysis

Data Migration

Data Engineering

Client

Global FMCG / CPG Company

Date

Services

Data Analysis Services, Data Migration, Data Engineering

Technologies

Google Cloud Products: DataProc, BigQuery

Challenge

The customer’s objective was to overcome data silos and construct a unified Datalake that serves as a reliable and authoritative source of data for campaigns and planning. The desired solution needed to be cost-effective, secure, and dependable, while adhering to data stewardship standards. The client had high expectations for seamless integration with multiple third-party data vendors and the implementation of scalable, fully managed ETL solutions available through the Google Cloud Platform.

‍

Our approach

To meet these objectives, we devised a comprehensive set of integration processes that guarantee efficient data ingestion, transformation, and storage. Our approach involved the creation and scheduling of more than 100 pipelines in Composer, leveraging the power of DataProc clusters, Google Cloud Storage buckets, and BigQuery. We seamlessly migrated historical data bundles using specialized Data Transfer tools. To ensure the utmost data security, we implemented Secret Manager, while also employing Logging to diligently monitor the processes.

‍

The outcome

The seamless migration to the cloud facilitated the efficient processing of data on a massive scale, surpassing a petabyte. Over the course of 6 months, we accomplished the successful migration of roughly 1.5PB of historical data, restructured numerous ingestion processes, and orchestrated streamlined pipelines on the Google Cloud Platform. Consequently, data storage and computation expenses decreased by approximately 30%. The punctual and error-free delivery of data empowers a multitude of internal applications and significantly benefits more than 50 brands operating in over 100 markets.

‍

Business Impact

The adoption of the Google Cloud Platform resulted in substantial cost savings for the business. Additionally, the implemented solution markedly enhanced data quality, availability, and security. Through the optimization of processing millions of daily events at a petabyte scale, we achieved significant cost reductions. In comparison to the previous Hadoop solution, the utilization of BigQuery enabled the processing of similar queries with a minimum of 15 times greater speed, thereby delivering more precise outcomes.