Machine Unlearning

Current Experiments

The current pipeline allows us to compute the following 5 metrics for unlearning and model performance in our MIMU method plus 3 additional baselines (SFT randomization, random masking, default).

1. Cross entropy loss of classifier under forget set (measures unlearning)
2. Cross entropy loss of classifier under retain set (measures utility degradation)
3. Probability of classifying forget class averaged over a batch (measures unlearning)
4. Score (percent correct classification) under forget set (measures unlearning)
5. Score under retain set (measures utility degradation)

Note

Raw metrics are generated as json files saved at eval/metrics_and_plots/json/<model>_<dataset>_top-{K}_kappa_{kappa}/*.json.

The main object of our experiments is the Pipeline class that abstracts away all the details:

# main.py

config = PipelineConfig(...)
pipeline = Pipeline(config)
pipeline.run()

An underneath the hood view of pipeline execution is as follows

┌─────────────────┐
│    Pipeline     │
└─────────┬───────┘
          │
          ▼
┌─────────────────────────────┐
│  run_vision_model_training  │◄──── SupportedVisionModels
└─────────┬───────────────────┘      Vision Dataset
          │                          
          │    ┌─────────────────────────────┐
          └───►│  VisionModelTrainer         │
               └─────────┬───────────────────┘
                         │
                         ▼
                    ┌─────────┐
                    │ use_ddp?│
                    └────┬────┘
                    Yes  │  No
                  ┌──────┴──────┐
                  ▼             ▼
            ┌──────────┐  ┌──────────┐
            │train_ddp │  │  train   │
            └─────┬────┘  └─────┬────┘
                  │             │
                  └─────┬───────┘
                        ▼
      ┌─────────────────────────────────┐
      │  run_gcn_graph_generation       │
      └─────────┬───────────────────────┘
                │
                │    ┌─────────────────┐
                └───►│ GraphGenerator  │
                     └─────────┬───────┘
                               │
                               ▼
                ┌─────────────────────────────┐
                │    run_gcn_training         │◄──── GCNPriorDistribution
                └─────────┬───────────────────┘      Graph Dataset
                          │
                          │    ┌─────────────────┐
                          └───►│ GCNTrainer      │
                               └─────────┬───────┘
                                         │
                                         ▼
                       ┌─────────────────────────────────┐
                       │ run_single_evaluation_round     │◄──── SFTModes
                       └─────────┬───────────────────────┘      Checkpoint Files
                                 │
                                 │    ┌─────────────────┐
                                 └───►│       Eval      │
                                      └─────────┬───────┘
                                                ▼
        ┌─────────────────────────────────────────────────────┐
        │                     eval                            │
        │  ┌─────────────────────────────────────────────┐    │
        │  │        topK_list × kappa_list               │    │
        │  └─────────────────┬───────────────────────────┘    │
        │                    │                                │
        │                    ▼                                │
        │          ┌─────────────────┐                        │
        │     ┌───►│ More combos?    │                        │
        │     │    └─────────┬───────┘                        │
        │     │         Yes  │  No                            │
        │     └──────────────┘  │                             │
        │                       ▼                             │
        └───────────────────────────────────────────────────┐ │
                                                            │ │
                                                            ▼ ▼
                                                  ┌─────────────────┐
                                                  │ Results:        │
                                                  │ List[Dict]      │
                                                  └─────────────────┘