Unit 865: Comprehensive Data Wrangling and Preprocessing

1. Introduction to Data Wrangling

• Importance of data wrangling in the data preprocessing pipeline
• Understanding messy data: types and causes
• Identifying data quality issues: missing data, noise, inconsistencies
• The role of data wrangling in preparing data for analysis and modeling

2. Data Cleaning Techniques

• Handling missing values
  • Types of missing data: MCAR, MAR, MNAR
  • Techniques: deletion, imputation (mean, median, mode, advanced methods)
• Removing duplicates and redundant data
• Detecting and handling outliers
  • Statistical methods: Z-score, IQR
  • Domain knowledge approaches
• Correcting data inconsistencies and errors
  • Standardizing data formats
  • Fixing typographical errors and anomalies

3. Data Transformation Methods

• Normalization and Standardization
  • Min-max scaling
  • Z-score standardization
• Encoding categorical variables
  • Label encoding
  • One-hot encoding
  • Target encoding
• Feature scaling and its importance for modeling
• Feature engineering
  • Creating new features from existing data
  • Polynomial features, interaction terms

4. Data Integration and Aggregation

• Combining data from multiple sources
  • Relational joins: inner, outer, left, right joins
  • Concatenation and appending datasets
• Data aggregation techniques
  • Grouping data and computing summary statistics
  • Pivot tables and multi-index aggregation
• Handling schema and format differences
• Managing data provenance and consistency

5. Handling Time Series Data

• Characteristics of time series data
• Handling timestamps and date-time formats
• Resampling techniques
  • Upsampling and downsampling
• Feature extraction from time series
  • Rolling statistics, moving averages
  • Seasonal decomposition
• Creating lagged variables and window features
• Dealing with missing time points and irregular intervals

6. Exploratory Data Analysis (EDA)

• Purpose and importance of EDA in preprocessing
• Summary statistics
  • Measures of central tendency and dispersion
• Data visualization techniques
  • Histograms, box plots, scatter plots, heatmaps
• Correlation analysis
  • Pearson, Spearman, Kendall coefficients
• Identifying patterns, trends, and anomalies
• Informing preprocessing and feature selection decisions

7. Dimensionality Reduction Techniques

• Importance of reducing feature dimensionality
• Principal Component Analysis (PCA)
  • Concept and computation
  • Interpreting principal components
• t-Distributed Stochastic Neighbor Embedding (t-SNE)
  • Visualization of high-dimensional data
• Feature selection methods
  • Filter methods (e.g., variance threshold, correlation)
  • Wrapper methods
  • Embedded methods

8. Handling Imbalanced Data

• Challenges posed by imbalanced datasets
• Techniques to address imbalance
  • Oversampling methods
  • Undersampling methods
  • Synthetic Minority Over-sampling Technique (SMOTE)
  • Class weight adjustments in modeling
• Evaluating models on imbalanced data
  • Precision, recall, F1-score, ROC-AUC