ETL - Extract, Transform and Load.
Extract:
Extracting a desired subset of data from database.Transform:
Converting the extracted data into the required form using rules or lookup tables.Cleaning (e.g., "Male" to "M" and "Female" to "F" etc.)
Filtering (e.g., selecting only certain columns to load)
Splitting multiple columns into a column (Last Name and First Name into Full Name) and vice versa etc…
Load:
Writing the resulting data (either all of the subset or just the changes) to a target databaseData Validation:
1. Are all the required columns in destination are with expected constraints (NOT NULL, UNIQUE, Primary Key, Foreign key, Check, Default and Surrogate keys etc)?2. Is number of rows count in source matches with destination?
3. Is data transformed correctly according to system requirements and business rules?
4. Are there any duplicate records in destination?
5. Is source and destination data apart from the columns on which business rules are applied are exactly same?
No comments:
Post a Comment