---
name: python-data-analysis
description: Performs reproducible exploratory data analysis on CSV, Excel, Parquet, JSON or SQL data with pandas or Polars — profiling, cleaning, aggregation, statistics and charts — and reports findings with the code that produced them. Use when asked to analyse a dataset, answer questions from data, find trends or anomalies, clean messy tables, or build a quick chart or summary.
license: MIT
compatibility: Python 3.10+ with pandas or polars, matplotlib; optional duckdb, pyarrow, openpyxl, scipy. Runs as scripts or notebooks.
metadata:
  category: data
  version: "1.0.0"
---

# Python Data Analysis

Every number in the report must be traceable to code that can be re-run. Profile before analysing, state assumptions, and separate what the data shows from what you infer.

## Step 1 — Frame the question

Restate the question as one or more concrete, answerable queries ("median order value by month for 2025, excluding refunds"). Ask what decision the answer feeds; it changes what precision and which cuts matter. Note the grain of the data (one row = ?) before anything else.

## Step 2 — Load and profile (always, even for "simple" questions)

```python
import pandas as pd
df = pd.read_csv(path, low_memory=False)          # or pl.read_csv / pd.read_parquet / read_sql
print(df.shape); print(df.dtypes)
print(df.head(3).T)
print(df.isna().mean().sort_values(ascending=False).head(15))
print(df.describe(include='all').T)
print(df.nunique().sort_values().head(15))         # candidate keys and categoricals
dup = df.duplicated().sum(); print('duplicates', dup)
```

For files over ~1 GB or joins across several files, use DuckDB (`duckdb.sql("select ... from 'file.parquet'")`) or Polars lazy frames instead of pandas.

Record in a `notes` section: row count, date range, key columns, null rates over 5%, obvious outliers, duplicates, and any column whose meaning is unclear (ask).

## Step 3 — Clean with explicit, logged rules

- Parse dates with an explicit format and timezone; never rely on inference silently.
- Cast numeric columns; investigate values that fail to cast rather than coercing them to NaN quietly.
- Normalise categoricals (strip, casefold, map known variants); list the mapping.
- Decide on nulls per column: drop, fill, or keep as its own category; justify.
- Remove exact duplicates only when the grain says they are errors; keep a count of what was removed.
- Filter outliers only with a stated rule (e.g., beyond 1st–99th percentile) and report both with and without.
- Keep raw data untouched; write cleaned data to a new frame or file (`data/clean/`).

## Step 4 — Analyse

- Start with the simplest cut that answers the question: group-by, pivot, resample. Show the table.
- Compare against a baseline (previous period, overall mean, control group).
- For claims of difference or trend, add uncertainty: sample sizes, confidence intervals or a suitable test (`scipy.stats`); say when n is too small.
- Segment by the two or three dimensions most likely to change the conclusion (region, plan, cohort) and check whether the headline holds in each (Simpson's paradox check).
- Correlation is reported as correlation; do not use causal language without a design that supports it.

## Step 5 — Visualise sparingly

One chart per finding, matplotlib (or plotly if interactivity is requested):

- Line for time series, bar for categories (sorted), histogram/box for distributions, scatter for relationships.
- Title states the finding ("Refund rate doubled after March pricing change"), axes labelled with units, y-axis starts at zero for bars, no 3D, no pie charts for more than three slices.
- Save to `figures/<slug>.png` at 150 dpi and reference the path.

## Step 6 — Report

```
# Analysis: <question> — <date>
## Answer (2–4 sentences, numbers with units and period)
## Key findings
1. ... (table or chart path)
## Data notes
- Source, rows, date range, grain
- Cleaning rules applied and rows affected
- Caveats: missing data, small samples, definitions assumed
## Method
- Script: analysis/<slug>.py (re-run with `python analysis/<slug>.py`)
## Next questions
```

Put all code in one script or notebook that runs top to bottom from the raw file; pin the environment (`requirements.txt` or `uv.lock`).

## Checklist

- [ ] Grain, row count and date range stated.
- [ ] Every cleaning step logged with row counts affected.
- [ ] Every headline number reproducible from the script.
- [ ] Uncertainty or sample size given for comparisons.
- [ ] At least one segmentation check on the headline.
- [ ] Charts titled with the finding, axes labelled.

## Pitfalls

- `pd.read_csv` guessing dtypes differently across chunks; set `dtype` for ids and zip codes (leading zeros).
- Averages of averages; aggregate from the row level.
- Timezone-naive timestamps mixed with aware ones silently shift days.
- Joining on non-unique keys multiplies rows; check counts before and after every merge.
- Chained indexing (`df[df.x>0]['y'] = ...`) does not write back; use `.loc`.
- Reporting percentages without the denominator, or trends from two data points.
- Excel dates arriving as serial numbers (`pd.to_datetime(x, unit='D', origin='1899-12-30')`).