Python Data Analysis

Every number in the report must be traceable to code that can be re-run. Profile before analysing, state assumptions, and separate what the data shows from what you infer.

Step 1 — Frame the question

Restate the question as one or more concrete, answerable queries ("median order value by month for 2025, excluding refunds"). Ask what decision the answer feeds; it changes what precision and which cuts matter. Note the grain of the data (one row = ?) before anything else.

Step 2 — Load and profile (always, even for "simple" questions)

import pandas as pd
df = pd.read_csv(path, low_memory=False)          # or pl.read_csv / pd.read_parquet / read_sql
print(df.shape); print(df.dtypes)
print(df.head(3).T)
print(df.isna().mean().sort_values(ascending=False).head(15))
print(df.describe(include='all').T)
print(df.nunique().sort_values().head(15))         # candidate keys and categoricals
dup = df.duplicated().sum(); print('duplicates', dup)

For files over ~1 GB or joins across several files, use DuckDB (duckdb.sql("select ... from 'file.parquet'")) or Polars lazy frames instead of pandas.

Record in a notes section: row count, date range, key columns, null rates over 5%, obvious outliers, duplicates, and any column whose meaning is unclear (ask).

Step 3 — Clean with explicit, logged rules

• Parse dates with an explicit format and timezone; never rely on inference silently.
• Cast numeric columns; investigate values that fail to cast rather than coercing them to NaN quietly.
• Normalise categoricals (strip, casefold, map known variants); list the mapping.
• Decide on nulls per column: drop, fill, or keep as its own category; justify.
• Remove exact duplicates only when the grain says they are errors; keep a count of what was removed.
• Filter outliers only with a stated rule (e.g., beyond 1st–99th percentile) and report both with and without.
• Keep raw data untouched; write cleaned data to a new frame or file (data/clean/).

Step 4 — Analyse

• Start with the simplest cut that answers the question: group-by, pivot, resample. Show the table.
• Compare against a baseline (previous period, overall mean, control group).
• For claims of difference or trend, add uncertainty: sample sizes, confidence intervals or a suitable test (scipy.stats); say when n is too small.
• Segment by the two or three dimensions most likely to change the conclusion (region, plan, cohort) and check whether the headline holds in each (Simpson's paradox check).
• Correlation is reported as correlation; do not use causal language without a design that supports it.

Step 5 — Visualise sparingly

One chart per finding, matplotlib (or plotly if interactivity is requested):

• Line for time series, bar for categories (sorted), histogram/box for distributions, scatter for relationships.
• Title states the finding ("Refund rate doubled after March pricing change"), axes labelled with units, y-axis starts at zero for bars, no 3D, no pie charts for more than three slices.
• Save to figures/<slug>.png at 150 dpi and reference the path.

Step 6 — Report

# Analysis: <question> — <date>
## Answer (2–4 sentences, numbers with units and period)
## Key findings
1. ... (table or chart path)
## Data notes
- Source, rows, date range, grain
- Cleaning rules applied and rows affected
- Caveats: missing data, small samples, definitions assumed
## Method
- Script: analysis/<slug>.py (re-run with `python analysis/<slug>.py`)
## Next questions

Put all code in one script or notebook that runs top to bottom from the raw file; pin the environment (requirements.txt or uv.lock).

Checklist

• Grain, row count and date range stated.
• Every cleaning step logged with row counts affected.
• Every headline number reproducible from the script.
• Uncertainty or sample size given for comparisons.
• At least one segmentation check on the headline.
• Charts titled with the finding, axes labelled.

Pitfalls

• pd.read_csv guessing dtypes differently across chunks; set dtype for ids and zip codes (leading zeros).
• Averages of averages; aggregate from the row level.
• Timezone-naive timestamps mixed with aware ones silently shift days.
• Joining on non-unique keys multiplies rows; check counts before and after every merge.
• Chained indexing (df[df.x>0]['y'] = ...) does not write back; use .loc.
• Reporting percentages without the denominator, or trends from two data points.
• Excel dates arriving as serial numbers (pd.to_datetime(x, unit='D', origin='1899-12-30')).