Data Migration
Data Migration from On-Premise to AWS Cloud
A Direct Banking and Payment Services company needed to migrate their on-premise data warehouse residing on Teradata to cloud-based data warehouse Redshift. The process involved migrating tables, schema mapping and conversion of historical and incremental data.
Client Challenges and Requirements
- Unavailability of any tool for migrating Source DDL to Cloud DDL with constraints
- Time-consuming manual process for migrating large numbers of tables and high volume of data
- Manual validation of migrated data resulting in inaccuracy and inefficiency of the process
- Challenges to migrate incremental data due to lack of CDC feature in Teradata
- Migration of sensitive data with Security
- No provision to handle pipeline failures and restartability
Bitwise Solution
Bitwise leveraged AWS SCT and Bitwise Cloud Data Migration framework to implement the below features:
Source to Target schema mappings and automatic schema conversion through AWS SCT.
AWS Schema Conversion Tool (SCT) and Bitwise cloud data migration framework to migrate data from source database to cloud.
Automated Data Migration - Full Data Load using AWS SCT and Incremental Load (CDC) through Bitwise cloud data migration framework
Bitwise Cloud Data Migration Framework for historical as well as CDC data validation with provision of sample and full data after migration.
Implemented tokenization process through Bitwise Cloud Data Migration Framework to handle sensitive data
Ability to monitor the data migration process to check the progress/status with user friendly dashboards and reports through CloudWatch.
Tools & Technologies We Used
AWS Schema Conversion Tool
Bitwise Cloud Data Migration Framework built using Java, Apache Spark
Containerization using Docker and Kubernetes
AWS S3, CloudWatch
Key Results
Fast processing of data migration and validation
100% accuracy of migrated data with sample and full validation
High scalability and availability with reduced manual intervention
