So, You Did Some Data Science…

You have a trained model, a beautiful dashboard, or an insightful report.

Why Reproducibility and Reliability Matter

Other people will thank you.

• Collaborate more easily — your teammate can pick up where you left off
• Learn from your analysis about the process and the domain
• Feel confident in the conclusions at which the analysis arrives

Managing a Data Science Project

• Manage data — data pipeline as a DAG
• Manage code — notebooks vs. scripts
• Manage project — version control, documentation
• Manage collaboration — issues, code review

Rethink the Plumbing: Data Pipeline as a DAG

flowchart LR
    A["Survey Data<br>(raw, immutable)"] --> D["Clean &<br>Merge"]
    B["Admin Records<br>(raw, immutable)"] --> D
    C["External Data<br>(census, weather)"] --> D
    D --> E["Feature<br>Engineering"]
    E --> F["Model"]
    E --> G["Summary Stats<br>& EDA"]
    F --> H["Evaluation"]
    F --> I["Paper /<br>Report"]
    G --> I
    H --> I
    H --> J["Dashboard"]
    style A fill:#e8f4f8,stroke:#1d3557
    style B fill:#e8f4f8,stroke:#1d3557
    style C fill:#e8f4f8,stroke:#1d3557
    style I fill:#fef0ed,stroke:#e63946
    style J fill:#fef0ed,stroke:#e63946

• Each step is a node; data flows forward through edges — this is a graph, not just a pipeline
• You can trace backwards from any output to the code and data that created it
• Raw data must be treated as immutable — read and copy, never modify in place

Cookiecutter Data Science

A well-known open-source project template by DrivenData:

├── data
│   ├── external       ← Data from third party sources
│   ├── interim        ← Intermediate data that has been transformed
│   ├── processed      ← Final, canonical data sets for modeling
│   └── raw            ← The original, immutable data dump
│
├── models             ← Trained models, predictions, summaries
├── notebooks          ← Jupyter notebooks (exploration only)
├── reports            ← Generated analysis (HTML, PDF, LaTeX)
│   └── figures        ← Generated graphics and figures
│
├── src/               ← Source code for use in this project
│   ├── dataset.py     ← Scripts to download or generate data
│   ├── features.py    ← Code to create features for modeling
│   ├── modeling/      ← Code to train and predict
│   └── plots.py       ← Code to create visualizations
│
├── README.md          ← What this project does and how to run it
├── requirements.txt   ← Dependencies for reproducing the environment
└── Makefile           ← Convenience commands (make data, make train)

The Data Directory Structure

data/
├── external/       ← Third party data (census, weather, etc.)
├── interim/        ← Intermediate transformations
├── processed/      ← Clean data ready for modeling
└── raw/            ← NEVER modify these files

• Most data should not be kept in version control
  • Use .gitignore to specify files not tracked by git
  • Small and stable data files are fine (e.g., a zipcode-city lookup)
• Large or sensitive data → store externally (S3, GCS, shared drive) with a download script in your repo

Activity: Spot the Reproducibility Failures

Your colleague shares a project folder with you:

my_project/
├── data.csv
├── data_v2.csv
├── data_v2_FINAL.csv
├── data_v2_fixed.csv
├── analysis.ipynb
├── analysis_copy.ipynb
├── Untitled1.ipynb
├── model.pkl
└── figures/
    └── plot.png

In the Zoom chat: What specific issues do you see, and what would you fix first? (2 minutes)

“But wait, I added documentation!”

my_project/
├── ...                         (same mess as before)
├── README.md
├── CLAUDE.md
├── CLAUDE_FULL_ORIGINAL.md
├── CLAUDE_COMPLETE_ARCHIVE.md
├── v2_PREPROCESSING_GUIDE.md
└── CLEANUP_DOCUMENTATION.md

Notebooks Are Great for Exploration

Jupyter notebooks shine at:

• Small-scale data wrangling (manipulating a dataset into desired structure)
• Running sanity checks (summary statistics, quick plots)
• Developing models with unit tests
• Visualizing results and generating reports

But Notebooks Are Bad for Reproduction

• The flexibility of running cells in arbitrary order creates hidden dependencies
  • “In order to reproduce this figure, run cell A first, skip cell B, then run cell C, and then rerun cell A…”
• Avoid writing instructions like this — it’s fragile and error-prone
• Notebooks are hard for version control — more on this in a moment
• One more reason to be cautious: they don’t work well with AI coding assistants

If You Open a .ipynb with a Text Editor…

Jupyter notebooks are JSON files under the hood:

{
 "cells": [{
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [{
     "text": ["Hello, world!\n"]
   }],
   "source": ["print('Hello, world!')"]
 }],
 "metadata": {
  "kernelspec": {"display_name": "Python 3"}
 }
}

Refactor Code from .ipynb to .py

• Consciously extract common building blocks from notebooks into .py files
• Prevents duplication of code across multiple notebooks
• Separates concerns into logical units:
  • Data layer (ETL code) → modeling → output layer
• Make sure each notebook is always correct if “Restart & Run All”
• Use naming conventions for ordering and ownership:
  • e.g., step01-<initials>-<description>.ipynb
  • e.g., 3.1-awei-extract-audio-features.ipynb

Pseudo-Notebooks in Modern IDEs

Some IDEs (VS Code, Cursor, PyCharm) support pseudo-notebook mode:

# %% [markdown]
# # Data Loading

# %%
import pandas as pd
df = pd.read_csv("data/raw/teachers.csv")
df.head()

# %% [markdown]
# # Feature Engineering

# %%
df["experience_bucket"] = pd.cut(df["years"], bins=[0, 5, 15, 40])

Always Use Version Control

• The most widely-acknowledged norm of software engineering
• It is a good idea to initiate a project folder with a template
  • e.g., the Cookiecutter Data Science template
• It makes it natural for you to adopt other good habits:
  • Setting up an environment (reduce dependency errors, increase consistency)
  • Enabling code reviews
  • Making it easy to document decisions and design choices
• It is also how AI coding assistants work — by creating a “diff”

What to Keep Away from Version Control

• Large data files — store externally, keep download scripts in repo
• Secrets
  • OpenAI API keys, AWS credentials, SSH keys, database passwords…
  • Store them in a configuration file (.env, config.yaml) that is .gitignore’d
• Other configuration files
  • Personal IDE settings (keyboard mappings, themes)
  • Machine-dependent variables (username, path-to-project-folder)
• Use a package (e.g., python-dotenv) to load them automatically

Project Management: More Than Version Control

• Issues — track bugs, feature requests, and tasks
• Projects — organize issues into milestones or sprints
• I personally keep two documents:
  • A “Design Doc” — the plan, the why, the alternatives considered
  • A “Lab Log” — chronological notes on what I tried, what worked, what didn’t

Start with a Simple Pipeline

flowchart LR
    A["Data ETL"] --> B["Feature<br>Extraction"]
    B --> C["Preliminaries<br>(stats, plots)"]
    C --> D["Model Fitting<br>(linear is OK!)"]
    D --> E["Evaluation"]
    style A fill:#e8f4f8,stroke:#1d3557
    style E fill:#fef0ed,stroke:#e63946

Testing, Logging, Sanity Checks

• Test
  • Unit tests, integration tests — even simple assert statements
  • If you’re hesitant to try AI coding: asking AI to write tests for you is the most painless entry point — it doesn’t change your code’s logic, it increases confidence, and you start building AI into your workflow
• Log
  • Record what ran, when, with what parameters
  • Logging code is tedious but easy — a perfect task for AI assistants
  • AI can also help parse logs (recall: it can “grep” for keywords)
• Sanity check
  • Check key statistics at every pipeline stage — one of the easiest ways to catch logic errors in AI-generated code

Sanity Check: A Personal Example

We were working on a project that classifies classroom video into activity types. Transcripts are recorded in turns (speaker changes), but video labels are in seconds — so we needed to translate turn-based annotations into time-based annotations.

Missing Semester: Practical Gaps

Have you ever…

• SSH to a machine, but typing your password every single time?
  • → SSH keys: generate a key pair, copy the public key to the server — never type a password again
• Run a job on a remote machine, but afraid it gets killed when you disconnect?
  • → tmux (terminal multiplexer) or nohup: detach your session, reconnect later, job still running

Since Our February Session…

In our first guest lecture, we discussed three levels of AI in data science:

1. Level 1: AI as a coding assistant (syntax, debugging)
2. Level 2: AI as a data processor (LLM APIs at scale)
3. Level 3: AI as an analysis partner (planning, auditing)

The Rise of Coding Agents

AI coding has moved beyond chat-based assistance to autonomous agents:

• Cursor 3.0 (April 2, 2026): a new “Agents Window” — managing multiple parallel agents rather than single chats
• Claude Code: an agent that works in your terminal, across multiple files, autonomously
• OpenAI Codex: cloud-based agents that can work on your codebase while you sleep
• The common pattern: Humans steer. Agents execute.

What This Means for Researchers

Chat-first and agent-first interfaces change the workflow:

• Agents can generate code across many files at once — far more than you can review line-by-line
• They’re fast and productive, but they don’t understand your domain
  • They don’t know that your “missing” data is MNAR (Selection Bias?)
  • They don’t know which column is the treatment variable
• Long context windows ≠ deep understanding
• The researcher’s challenge: staying focused on the science while managing AI-generated engineering

Managing a Data Science Project — with AI

• Manage data — data pipeline as a DAG
• Manage code — notebooks vs. scripts
• Manage project — version control, documentation
• Manage collaboration — issues, code review

The Mental Model: Agent = Model + Harness

• The model is the AI (GPT, Claude, etc.) — powerful but non-deterministic, context-dependent, and lacking domain knowledge
• The harness is everything around the model that makes it useful:
  • Guides (feedforward): project structure, conventions, documentation — steer the agent before it acts
  • Sensors (feedback): tests, linters, version control diffs — help the agent self-correct after it acts

The Harness in Practice

Your Project (the harness)
├── README.md              ← Guide: what this project does
├── AGENTS.md              ← Guide: rules for AI assistants
├── .gitignore             ← Guide: what not to touch
├── requirements.txt       ← Guide: locked dependencies
├── src/                   ← Guide: where code lives
├── data/raw/              ← Guide: immutable data (read-only)
├── tests/                 ← Sensor: catches errors automatically
├── .github/workflows/     ← Sensor: CI runs tests on every commit
└── Makefile               ← Guide: reproducible commands

A Glimpse of the Future: Parallel Agents

The direction things are heading:

• Parallel agents on parallel branches: people already run multiple Claude Code sessions, multiple Codex tasks — Cursor 3.0 highlights this trend by building it into the interface
• Top software engineers at big tech companies report they spend more time reviewing than writing code
• This is already becoming the default workflow in some teams

The Karpathy Loop: Autonomous Research

Andrej Karpathy’s autoresearch (March 2026): an agent that runs ML experiments autonomously.

┌─────────────────────────────────────────────┐
│                                             │
│   program.md ──→ Agent proposes change      │
│                      │                      │
│               Modify train.py               │
│                      │                      │
│            Run 5-min experiment             │
│                      │                      │
│             Metric improved?                │
│              /            \                 │
│           Yes              No               │
│            │                │               │
│       git commit        git revert          │
│            │                │               │
│            └────────────────┘               │
│                      │                      │
│                  Loop forever               │
│                                             │
└─────────────────────────────────────────────┘

Autonomous Agents Beyond Code

This pattern is expanding:

• OpenClaw (2026): an open-source framework for autonomous AI agents — agents defined by markdown files (SOUL.md, AGENTS.md, TOOLS.md)
• Karpathy’s autoresearch + OpenClaw-style agents → autonomous research pipelines where agents run experiments, parse results, propose next steps
• The common architecture: a human writes the goals and constraints in markdown; agents execute within those boundaries

Discussion Break

• How has your experience with AI coding tools evolved since February?
• What has surprised you — good or bad?
• As tools become more autonomous, what worries you most about staying in control of your analysis?

The Reproducible Pipeline Checklist

• Project structure — use a template (Cookiecutter or similar)
• Raw data is immutable — never modify source data in place
• Code in scripts — extract reusable code from notebooks to .py
• Version control — git from day one, .gitignore for secrets and data
• Dependencies locked — requirements.txt or environment.yml
• Documentation — README, design doc, lab log
• Testing — even simple sanity checks count
• Reproducible commands — make data, make train, or equivalent

Try It: Homework

Step 1: If you haven’t already, install an IDE that supports AI coding:

• VS Code + GitHub Copilot (free for students)
• Cursor (VS Code fork with built-in agents)
• Claude Code (terminal-based agent)
• OpenAI Codex (via ChatGPT or the VS Code Copilot extension)

Resources

• Cookiecutter Data Science — cookiecutter-data-science.drivendata.org
• The Missing Semester (2026, incl. Agentic Coding) — missing.csail.mit.edu
• Harness engineering (OpenAI, Feb 2026) — openai.com/index/harness-engineering
• Harness engineering for coding agent users (Fowler, Apr 2026) — martinfowler.com/articles/harness-engineering.html
• autoresearch (Karpathy, Mar 2026) — github.com/karpathy/autoresearch
• Best practices for coding with agents (Cursor) — cursor.com/blog/agent-best-practices