a Audit LabCS442 · Human–Computer Interaction
In-class activity 04 · Model extraction & membership audit

One model,
two audits.

Copy a target model's predictions under a limited query budget, then audit membership leakage using confidence and loss. Compare agreement and task accuracy for surrogate models at budgets of 50/200/800, and test whether regularization reduces both leakage and utility.

Enter Student Lab → Teacher Console
70 minutes · groups of 2–3 Experiment Workflow + AI Teaching Assistant Frozen models · synthetic data
01 / EXTRACT

Copy prediction behavior

Complete the small soft-label loss task and read the provided record-alignment helper. Compare surrogate models trained on 50, 200 and 800 records.

02 / INFER

Audit membership leakage

Complete the membership-scores and confusion-count tasks. Use the provided shadow-calibration helper, then perform four audits using 400 candidates per target model.

03 / INTERPRET

Write a conclusion with limits

Distinguish agreement from task accuracy, privacy from utility, and mechanism demonstrations from deployment guarantees.

The 70-minute arc

From queries to evidence to conclusions

Each group advances independently through formal evaluations. Predict first, complete and test your code, then fetch data, run, save and evaluate in separate steps. The assistant gives hints; teachers lead discussions using code and evidence.

  1. 0–5 MINBrief and hypothesisThreat model, connection, initial prediction.
  2. 5–30 MINModel Extraction800 queries, three surrogate models.
  3. 30–50 MINMembership InferenceCalibrate, query, run four audits.
  4. 50–60 MINInterpretationCompare metrics and regularization tradeoffs.
  5. 60–70 MINSubmissionFreeze, evidence, conclusion, worksheet.
What you will practise

Use receipt metrics as evidence

  • Compare surrogate models with a fixed query order and budget.
  • Distinguish agreement from task accuracy.
  • Calibrate thresholds only on shadow data.
  • Report both privacy metrics and task utility.
What this is not

This is not a privacy audit of a real deployment

  • The data are synthetic; the models are frozen teaching assets.
  • The restricted record pool is not a full adaptive-query threat model.
  • One fixed split does not establish a general leakage rate.
  • State the limits of your conclusion instead of claiming a security guarantee.