Data Migration & Performance Improvement of large data processing

Enabled a leading data processing company to process large batches of pharmacies & pharmacy supplier data quickly with reduced IT costs

Data-migration-performance-improvement

About the Client

The client is a leading data processing company in Australia. The client is connected to more than 4,500 pharmacies across Australia and wanted to improve access to information across the pharmacy supply chain. The objective was to help pharmacies identify opportunities in both their dispensary and retail through reporting and analytics, as well as support rebate payments from suppliers and patient adherence programs.

The Business Challenge

The client wanted to process multiple batches of data quickly where one data batch could contain up to a billion records. The client wanted to reduce the processing time that was at 2.2 hours for a billion records. The client also wanted to prevent any bottleneck situation due to the high volume of data.

What Aptus Data Labs Did

We migrated the client’s existing 5-node Vertica Cluster platform to Apache Spark in Hortonworks on AWS Cluster to improve the processing time and quickly adapt to new features in the future along with cost reduction.

The Impact Aptus Data Labs Made

The new analytics platform boosted the performance by 62% and reduced the data processing time. It also reduced IT costs by 400% and helped the client to handle large volumes of data smoothly.

The Business and Technology Approach

Aptus Data Labs used the following methodology for environment migration and to resolve the existing challenge. Aptus Data Labs:.

  • Deployed a 3-node HDP cluster with Apache Spark 1.3 on AWS with each node containing 30 GB memory and 80 GB solid-state drive
  • Used Spark data source API to ingest data from both database and HDFS source
  • Utilized DataFrames to store structured relational data instead of using traditional RDDs
  • Replaced the processing procedures in Vertica with User Defined Functions (UDF) in Spark
  • Deployed Spark SQL to pass DataFrames to the UDF for processing
  • Partitioned the DataFrame to process across all nodes parallel
  • Replaced Spark resource manager with Yarn resource manager to boot high availability of the cluster
  • Implemented Shell scripts to deploy and automate Spark jobs

Tools used

Apache Spark Cluster, AWS, HDP platform, Spark, & Vertica

The Outcome

The migrated analytics platform reduced the processing time from 2.2 hours for a billion records to 1 hour for 1.2 billion records that boosted the performance by 62%. The analytics platform reduced IT costs significantly using open source technologies. The platform used the yarn cluster to ensure high availability and high efficiency of the system. It also enabled the client to handle massive volumes of data smoothly without any break in the performance.

Related Case Studies

Download Case study​

Explore more of our client stories

Download Case study​