# CIFAR-10 Dataset Distillation via M-PCA-SVD with Block-wise Rotation

## Overview

This project implements a novel **gradient-free dataset distillation** pipeline for CIFAR-10 image classification. The method compresses the full 50,000-image training set into **500 distilled images (50 per class)** using a combination of Modified PCA (M-PCA), Singular Value Decomposition (SVD), and block-wise rotation. The distilled images are then used to train a ResNet50V2 classifier, evaluated on the original 10,000-image test set.

---

## Pipeline Summary

```
50,000 CIFAR-10 training images
        ↓
Step 1 — Preprocessing (normalize + standardize)
        ↓
Step 2 — Carve validation set (5,000 images)
        ↓
Step 3 — M-PCA per class (95% variance, 4,500 vectors/class)
        ↓
Step 4 — SVD + Block-wise Rotation (θ = 300°, block size 3×3)
        ↓
Step 5 — Select top 50 rows → 500 distilled images
        ↓
Step 6 — Train ResNet50V2 (frozen ImageNet backbone)
        ↓
Step 7 — Evaluate with TTA × 10 on 10,000 test images
        ↓
Step 8 — Confusion matrix + Training history plots
```

---

## Requirements

Install the required packages before running:

```bash
pip install tensorflow numpy scikit-learn matplotlib seaborn
```

Tested with:

| Package | Version |
|---|---|
| Python | 3.9+ |
| TensorFlow | 2.12+ |
| NumPy | 1.23+ |
| scikit-learn | 1.2+ |
| Matplotlib | 3.6+ |
| Seaborn | 0.12+ |

---

## Configuration

All key hyperparameters are defined at the top of the script:

```python
RANDOM_SEED          = 42       # reproducibility seed
VAL_SIZE             = 5000     # validation set size
VARIANCE_THRESHOLD   = 0.95     # PCA variance threshold (95%)
SVD_ENERGY_THRESHOLD = 0.95     # SVD energy threshold (95%)
NUM_PER_CLASS        = 50       # distilled images per class (500 total)
BLOCK_SIZE           = 3        # block size for rotation (3×3)
THETA                = np.pi * (5/3)  # rotation angle (300°)
IMG_SIZE             = 160      # image size for ResNet input (160×160)
TTA_STEPS            = 10       # test-time augmentation passes
EPOCHS               = 400      # maximum training epochs
BATCH_SIZE           = 8        # training batch size
```

---

## How to Run

```bash
python cifar10_distillation_Code.py
```

The script runs end-to-end automatically. No manual intervention is required.

---

## Method Description

### Step 1 — Preprocessing
- CIFAR-10 is loaded using the original 50,000/10,000 train/test split
- Training images are flattened to 3,072-dimensional vectors and standardized (zero mean, unit variance)

### Step 2 — Validation Split
- 5,000 images (500 per class) are carved from the training pool as a validation set
- Remaining 45,000 images form the distillation pool

### Step 3 — Modified PCA (M-PCA)
- PCA is applied per class retaining 95% of variance
- Up to 4,500 projected vectors per class are reconstructed back to pixel space
- Output: a modified matrix Δ per class

### Step 4 — SVD + Block-wise Rotation
- SVD decomposes Δ into L × Σ × R^T
- The left singular matrix L is partitioned into 3×3 blocks
- A rotation matrix ρ(θ) with θ = 300° is applied to each block
- The rotated matrix is reconstructed: Δ^m = (L^m)' × Σ^m × (R^m)^T

### Step 5 — Distilled Image Selection
- The top 50 rows of the reconstructed matrix are selected per class
- This gives 500 distilled images total (50 per class)

### Step 6 — Classification
- Distilled images are rescaled to [0,1], reshaped to 32×32×3, and upsampled to 160×160
- ResNet50V2 (pretrained on ImageNet, frozen backbone) is fine-tuned on 500 distilled images
- Head architecture: Dense(512) → BatchNorm → Dropout(0.5) → Dense(256) → BatchNorm → Dropout(0.5) → Softmax(10)
- Training uses CutOut augmentation, EarlyStopping (patience=30), and ReduceLROnPlateau

### Step 7 — Evaluation
- Test-Time Augmentation (TTA × 10) is applied on the 10,000 original test images
- Memory-efficient TTA processes images in batches of 128 to avoid MemoryError

---

## Outputs

| File | Description |
|---|---|
| `pca_variance_plot.png` | Cumulative PCA variance curve for class 0 |
| `svd_energy_plot.png` | SVD cumulative energy curves for all 10 classes |
| `best_resnet_distilled.keras` | Saved best model weights |
| `confusion_matrix_cifar10.png` | 10×10 confusion matrix heatmap |
| `training_history_cifar10.png` | Accuracy and loss curves over epochs |

---

## CIFAR-10 Classes

| Index | Class |
|---|---|
| 0 | airplane |
| 1 | automobile |
| 2 | bird |
| 3 | cat |
| 4 | deer |
| 5 | dog |
| 6 | frog |
| 7 | horse |
| 8 | ship |
| 9 | truck |

---

## Key Design Decisions

- **No gradient required** — the distillation is entirely mathematical (PCA + SVD + rotation)
- **Frozen ResNet50V2** — only the classification head is trained, preserving ImageNet features
- **Memory-efficient TTA** — avoids MemoryError by processing test images batch by batch
- **CutOut augmentation** — applied during training to reduce overfitting on 500 images
- **Label smoothing (0.1)** — applied to the cross-entropy loss for better generalization

---

## Notes

- Running the full pipeline takes approximately 2–4 hours depending on hardware
- A GPU is strongly recommended for the training and TTA steps
- The `best_resnet_distilled.keras` file must exist before TTA evaluation — it is created automatically during training
- If a MemoryError occurs during TTA, reduce the `batch_size` parameter inside `tta_predict_fast()`
