Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects

Reading list

Machine Learning Basics for a Newbie
6 Steps of Machine learning Lifecycle Introduction to Predictive Modeling
Introduction to Exploratory Data Analysis & Data Insights Descriptive Statistics Inferential Statistics How to Understand Population Distributions?
Reading Data Files into Python Different Variable Datatypes
Probability for Data Science Basic Concepts of Probability Axioms of Probability Conditional Probability
Central Tendencies for Continuous Variables Spread of Data KDE plots for Continuous Variable Overview of Distribution for Continuous variables Normal Distribution Skewed Distribution Skeweness and Kurtosis Distribution for Continuous Variable
Central Tendencies for Categorical Variables Understanding Discrete Distributions Performing EDA on Categorical Variables
Dealing with Missing Values Understanding Outliers Identifying Outliers in Data Outlier Detection in Python Outliers Detection Using IQR, Z-score, LOF and DBSCAN
Sample and Population Central Limit Theorem Confidence Interval and Margin of Error
Bivariate Analysis Introduction
Covariance Pearson Correlation Spearman's Correlation & Kendall's Tau Correlation versus Causation Tabular and Graphical methods for Bivariate Analysis Performing Bivariate Analysis on Continuous-Continuous Variables
Tabular and Graphical methods for Continuous-Categorical Variables Introduction to Hypothesis Testing P-value Two sample Z-test T-test T-test vs Z-test Performing Bivariate Analysis on Continuous-Catagorical variables
Chi-Squares Test Bivariate Analysis on Categorical Categorical Variables
Multivariate Analysis A Comprehensive Guide to Data Exploration The Data Science behind IPL
Supervised Learning vs Unsupervised Learning Reinforcement Learning Generative and Descriminative Models Parametric and Non Parametric model
Machine Learning Pipeline Preparing Dataset Build a Benchmark Model: Regression Build a Benchmark Model: Classification
Evaluation Metrics for Machine Learning Everyone should know Confusion Matrix Accuracy Precision and Recall AUC-ROC Log Loss R2 and Adjusted R2
Dealing with Missing Values Replacing Missing Values Imputing Missing Values in Data Working with Categorical Variables Working with Outliers Preprocessing Data for Model Building
Understanding Cost Function Understanding Gradient Descent Math Behind Gradient Descent Assumptions of Linear Regression Implement Linear Regression from Scratch Train Linear Regression in Python Implementing Linear Regression in R Diagnosing Residual Plots in Linear Regression Models Generalized Linear Models Introduction to Logistic Regression Odds Ratio Implementing Logistic Regression from Scratch Introduction to Scikit-learn in Python Train Logistic Regression in python Multiclass using Logistic Regression How to use Multinomial and Ordinal Logistic Regression in R ? Challenges with Linear Regression Introduction to Regularisation Implementing Regularisation Ridge Regression Lasso Regression
Introduction to K Nearest Neighbours Determining the Right Value of K in KNN Implement KNN from Scratch Implement KNN in Python
Bias Variance Tradeoff Introduction to Overfitting and Underfitting Visualizing Overfitting and Underfitting Selecting the Right Model What is Validation? Hold-Out Validation Understanding K Fold Cross Validation
Introduction to Feature Selection Feature Selection Algorithms Missing Value Ratio Low Variance Filter High Correlation Filter Backward Feature Elimination Forward Feature Selection Implement Feature Selection in Python Implement Feature Selection in R
Introduction to Decision Tree Purity in Decision Tree Terminologies Related to Decision Tree How to Select Best Split Point in Decision Tree? Chi-Squares Information Gain Reduction in Variance Optimizing Performance of Decision Tree Train Decision Tree using Scikit Learn Pruning of Decision Trees
Introduction to Feature Engineering Feature Transformation Feature Scaling Feature Engineering Frequency Encoding Automated Feature Engineering: Feature Tools
Introduction to Naive Bayes Conditional Probability and Bayes Theorem Introduction to Bayesian Adjustment Rating: The Incredible Concept Behind Online Ratings! Working of Naive Bayes Math behind Naive Bayes Types of Naive Bayes Implementation of Naive Bayes
Understanding how to solve Multiclass and Multilabled Classification Problem Evaluation Metrics: Multi Class Classification
Introduction to Ensemble Techniques Basic Ensemble Techniques Implementing Basic Ensemble Techniques Finding Optimal Weights of Ensemble Learner using Neural Network Why Ensemble Models Work well?
Introduction to Stacking Implementing Stacking Variants of Stacking Implementing Variants of Stacking Introduction to Blending Bootstrap Sampling Introduction to Random Sampling Hyper-parameters of Random Forest Implementing Random Forest Out-of-Bag (OOB) Score in the Random Forest IPL Team Win Prediction Project Using Machine Learning Introduction to Boosting Gradient Boosting Algorithm Math behind GBM Implementing GBM in python Regularized Greedy Forests Extreme Gradient Boosting Implementing XGBM in python Tuning Hyperparameters of XGBoost in Python Implement XGBM in R/H2O Adaptive Boosting Implementing Adaptive Boosing LightGBM Implementing LightGBM in Python Catboost Implementing Catboost in Python
Different Hyperparameter Tuning methods Implementing Different Hyperparameter Tuning methods GridsearchCV RandomizedsearchCV Bayesian Optimization for Hyperparameter Tuning Hyperopt
Understanding SVM Algorithm SVM Kernels In-depth Intuition and Practical Implementation SVM Kernel Tricks Kernels and Hyperparameters in SVM Implementing SVM from Scratch in Python and R
Introduction to Principal Component Analysis Steps to Perform Principal Compound Analysis Computation of Covariance Matrix Finding Eigenvectors and Eigenvalues Implementing PCA in python Visualizing PCA A Brief Introduction to Linear Discriminant Analysis Introduction to Factor Analysis
Introduction to Clustering Applications of Clustering Evaluation Metrics for Clustering Understanding K-Means Implementation of K-Means in Python Implementation of K-Means in R Choosing Right Value for K Profiling Market Segments using K-Means Clustering Hierarchical Clustering Implementation of Hierarchial Clustering DBSCAN Defining Similarity between clusters Build Better and Accurate Clusters with Gaussian Mixture Models
Understand Basics of Recommendation Engine with Case Study
8 Ways to Improve Accuracy of Machine Learning Models
Introduction to Dask Working with CuML
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to Machine Learning Interpretability Framework and Interpretable Models model Agnostic Methods for Interpretability Implementing Interpretable Model Understanding SHAP Out-of-Core ML Introduction to Interpretable Machine Learning Models Model Agnostic Methods for Interpretability Game Theory & Shapley Values
Introduction to AutoML Implementation of MLBox Introduction to PyCaret TPOT Auto-Sklearn EvalML
Pickle and Joblib Introduction to Model Deployment
Deploying Machine Learning Model using Streamlit Deploying ML Models in Docker Deploy Using Streamlit Deploy on Heroku Deploy Using Netlify Introduction to Amazon Sagemaker Setting up Amazon SageMaker Using SageMaker Endpoint to Generate Inference Deploy on Microsoft Azure Cloud Introduction to Flask for Model Deploying ML model using Flask
Model Deployment in Android Model Deployment in Iphone

Simplify Data Migration with AWS DMS

Ansaba
Ansaba Last Updated : 22 Dec, 2022
5 min read

This article was published as a part of the Data Science Blogathon.

 

AWS DMS
Source: https://towardsdatascience.com/using-aws-dms-to-tail-mongodb-978967aed8dd

Introduction

DMS is a service that makes it easy to migrate on-premise databases into the cloud with minimal or no downtime. It can even monitor the changes in the original database and apply them to the new database. DMS supports all databases like MySQL, PostgreSQL Oracle, SQL Server, Microsoft SQL Server, and Amazon Aurora. The advantage is that the original and new databases don’t have to match. Hence it is easy to alter one type of database into another. DMS takes care of all the tedious tasks of exporting the data from the old database and importing it into the new database in lesser time. It supports data migration tasks like database development, consolidation, and testing. While migrating, the source database will be fully operational, minimizing downtime to applications that rely on the database. This article covers an overview of DMS, how it migrates data, its process, and why one should consider DMS for migration.

What is AWS DMS?

AWS Database Migration Service (AWS DMS) helps migrate databases to AWS quickly. It allows migrations of several databases like relational, nonrelational databases, and other types of commercial and open-source databases. It continuously monitors the source and target databases besides network connectivity. If any interruption arises that halts the replication task, it is automatically rectified by DMS. It can consolidate disparate databases into a multi-petabyte data warehouse by streaming data to AWS Redshift and Amazon S3.

Why Consider AWS DMS for Data Migration?

  • Highly resilient:
    DMS can automate the backup snapshots and restore them very quickly. If the primary replication server fails, the backup replication server will take over; hence there is no service interruption.
  • Minimal downtime:
    While migrating databases through DMS, the source database will be fully operational, minimizing downtime to applications that rely on the database. So users can change databases whenever required without a need to shut down.
  • Low cost :
    DMS provides a free migration service for moving databases to Redshift, Aurora, DynamoDB, or DocumentDB. Users must pay for migrating to other databases depending on the log storing volume.
  • Comprehensive Database Support:
    DMS supports homogeneous migrations such as Oracle to Oracle and heterogeneous migrations Oracle to Aurora platforms using the SCT tool. It supports open-source and commercial databases helping in convenient database migration.
  • Failover support :
    DMS are deployed in a multi-AZ environment which helps in failover support and easy availability. In a Multi-AZ deployment, DMS automatically maintains and provisions a synchronous standby replica of the replication instance in a different Availability Zone.

AWS DMS Process- How it Works

  • Connect endpoints:
    First, connect DMS to both source and target endpoints. The primary requirement for DMS is that one user’s endpoints must be hosted on AWS service. For performing the migration, endpoint credentials should successfully connect to the databases with the essential privileges.
  • Task scheduling:
    Now schedule a task that runs on the server to migrate the data. DMS reads the data from the source database and prepares it for the target database. Full-load migration occurs when source data is transferred to the target database.
  • Sync databases:
    If any changes are made at the source database, it is cached on the replication server. Soon after the full-load migration is completed, these cached changes are applied to the target database to keep both databases in sync.

Types of Supported Databases

mongo db

Source: https://www.logicata.com/blog/aws-database-migration-service

Using AWS DMS with other AWS services. It is possible to use DMS with other AWS services, which include the following:
  • Amazon EC2 and RDS:
    Users can use Amazon EC2 or RDS DB instances as a target for data migration. EC2 is the virtual machine representing a physical server that provides secure and scalable computing capacity. RDS refers to an SQL database service allowing access to files and databases anywhere.
  • AWS Schema Conversion Tool:
    AWS Schema Conversion Tool or SCT automatically converts source schema to a compatible format for your target database. With SCT, users can migrate database schema, including stored procedures, views, functions, etc., to compatible target databases.
  • Amazon S3:
    For storage purposes or while migrating large amounts of data, Amazon S3 can be used in the intermediate step. Users can set up a Database Migration Task, which transfers specific data to the S3 bucket.
  • AWS CloudFormation :
    AWS CloudFormation can set up AWS resources for infrastructure deployment or management. Using CloudFormation templates, users can create, delete, and modify the entire stack (collection of AWS resources) as a single unit or stacks across multiple AWS accounts and regions without managing resources individually.

Components of AWS Database Migration Service

There are mainly three DMS components which include:
  • Replication instance
  • Source & Target Endpoints
  • Replication tasks

Replication instance:

The Amazon EC2 (Elastic Compute Cloud) managed instance hosts one or more replication jobs. It deals with compute-intensive workloads, heterogeneous migrations, and replications like migrating from Oracle to PostgreSQL. Memory-optimized replication instances like R4/R5 help in midsize in-memory databases, real-time big data analytics, and enterprise applications.

Source & Target Endpoints:

Endpoints link both source and target databases helping in the transport of data. All the endpoints need information like Server name, Port number, Endpoint type, Engine type, Credentials, etc. Before using endpoints for a database migration task, users should test the endpoint connection.

Replication tasks:

It is the final step before beginning a migration which transports data from the source endpoint to the destination endpoint. It denotes what data is being transferred between the target & source endpoints and when the migration has to be performed.

Use cases

Homogeneous database migration:

 

AWS DMS

Source: https://www.heptabit.com/blog/database-migration-to-aws-with-no-downtime

 

In this migration, both the source and the target database engines should match and be interoperable. Examples are Microsoft SQL Server to Amazon RDS for SQL Server, Oracle to Amazon RDS for Oracle, MySQL to Amazon RDS for MySQL, MySQL to Amazon Aurora, etc.
Heterogeneous database migration:

 

AWS DMS

 

In this migration, the database engines of the source and the target do not complement each other. Here a schema and code transformation step is needed to address the differences in database code, data types, and schema structure between the source and target databases. Examples are Oracle to PostgreSQL, Oracle to Amazon Aurora, Microsoft SQL Server to MySQL migrations, etc.

Building a migration plan:

DMS allows users to assess and keep track of the progress of their database migrations. DMS dashboard displays migration status regardless of which regions users move their databases. It can detect and resolve troubles arising out of migration.

Continuous Data Replication:

DMS helps in the replication of data continuously with high availability. Through continuous data replication, DMS helps synchronize development and testing environment locations and Disaster Recovery instance locations.

Conclusion

Ultimately, we can conclude that AWS DMS is a great tool that makes it easier to migrate database workloads to AWS along with continuous data replication. It’s easy to set up, reliable, supports a variety of databases, and performs constant network connectivity monitoring of target databases.
Key Takeaways:
  • AWS DMS helps migrate databases to AWS quickly and monitors source and target databases.
  • Deployed in a multi-AZ environment for easy availability.
  • AWS Schema Conversion Tool or AWS SCT can be used with DMS to automatically convert source schema to a compatible format for your target database.
  • Mainly three components – Replication instance, Source & Target Endpoints, and Replication tasks.
  • Support both homogeneous and heterogeneous database migrations.
  • Endpoints refer to the two data stores, i.e., the source and the target.
  • DMS helps in the replication of data continuously and synchronizes development and testing environment locations.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Ansaba
Ansaba

I'm a cloud enthusiast and have done graduation in computer science. My skills include C, C++, Php, Python, Cloud services(AWS, GCP), shell scripting, MySQL, etc. I have hands-on experience and have done several projects and certifications in areas of Python, Php, MySQL, Cloud (GCP, AWS), and C++

Free Courses

Generative AI 4.7

Generative AI - A Way of Life

Explore Generative AI for beginners: create text and images, use top AI tools, learn practical skills, and ethics.
Large Language Models 4.5

Getting Started with Large Language Models

Master Large Language Models (LLMs) with this course, offering clear guidance in NLP and model training made simple.
Prompt Engineering 4.6

Building LLM Applications using Prompt Engineering

This free course guides you on building LLM apps, mastering prompt engineering, and developing chatbots with enterprise data.
LlamaIndex 4.8

Improving Real World RAG Systems: Key Challenges & Practical Solutions

Explore practical solutions, advanced retrieval strategies, and agentic RAG systems to improve context, relevance, and accuracy in AI-driven applications.
RAG systems using LlamaIndex 4.7

Microsoft Excel: Formulas & Functions

Master MS Excel for data analysis with key formulas, functions, and LookUp tools in this comprehensive course.

Recommended Articles

Responses From Readers

Clear

Write for us

Write, captivate, and earn accolades and rewards for your work
  • Reach a Global Audience
  • Get Expert Feedback
  • Build Your Brand & Audience
  • Cash In on Your Knowledge
  • Join a Thriving Community
  • Level Up Your Data Science Game
imag

We use cookies essential for this site to function well. Please click to help us improve its usefulness with additional cookies. Learn about our use of cookies in our Privacy Policy & Cookies Policy.

Show details