Guide · Data quality · 8 minute read

Audit CSV data quality. Show the evidence.

A clean-looking spreadsheet can still contain blank business keys, duplicate records, wrong types, impossible ranges, and category drift. A useful audit separates what the file appears to contain from the rules it must pass — then preserves enough evidence for someone else to reproduce the decision.

Start here

Separate profiling from validation

Profiling describes the file you received: row count, columns, missing rates, distinct values, distributions, candidate keys, exact duplicate rows, and unusual numeric values. These are observations. A column that is 30% blank or contains an IQR outlier deserves review, but the profile alone cannot know whether that is acceptable for your import or report.

Validation tests explicit expectations: Customer ID is required and unique; Revenue must be numeric and non-negative; Status must be one of three approved values; Order Code must match a pattern. A row fails only when it breaks one of those rules. Keeping the two layers separate prevents a statistical surprise from being called an error and prevents a quiet but forbidden value from passing because it looked common.

Do not begin with a weighted 0–100 score. Without visible weights and business rules, “92% quality” cannot tell a reviewer which rows are unsafe or whether the remaining 8% matters. Begin with exact counts, denominators, and named checks.

→ Open the CSV data-quality audit workspace

Step 1

Inventory rows, columns and denominators

Record the file label, worksheet, row count, column count, and total cells before discussing quality. A missing-value rate is meaningless without its denominator. Column completeness is (rows − missing cells) ÷ rows; dataset completeness is (all cells − missing cells) ÷ all cells. These answer different questions, so keep both labels visible.

Confirm that headers are present and distinct and that the selected worksheet is the intended one. A parser can successfully read the wrong sheet, a footer block, or a one-column delimiter accident. Structural problems belong in a CSV validator before semantic rules run. Once the table shape is trustworthy, the quality audit can reason about columns and rows.

Step 2

Define which blanks are failures

Profile missing values for every column, then decide which columns are actually required. A blank optional Notes field is not equivalent to a blank Invoice ID. Marking every column required creates noisy audits that teams learn to ignore; marking none required lets incomplete key and metric rows flow downstream.

For each required column, count the failing rows and retain their row numbers. Treat whitespace-only cells as blank. Decide deliberately how strings such as N/A, unknown, or null should behave: they are text unless your upstream contract explicitly declares them missing sentinels. Do not silently rewrite them during an audit, because that changes the evidence you are trying to assess.

When comparing two exports, report missing-rate movement in percentage points. Moving from 5% missing to 12% missing is +7 points, not +7%. Use CSV Compare when the audit question is about that period-to-period drift.

→ Review or fill missing values after the audit

Step 3

Test exact rows and business keys separately

An exact duplicate is a full row whose values match another full row. It is useful for catching accidental copy/paste or repeated exports, but it is not the same as a duplicate business entity. Two rows can share the same Order ID while differing in Status or Revenue; an exact-row check misses that conflict.

Choose the column or composite columns that identify one record in this workflow. A business key might be Order ID, or Account ID + Month, or SKU + Warehouse. Every selected key part should be nonblank, and the complete trimmed key should be unique. Count both duplicate groups and affected rows: three rows sharing one key are one group but three records that need a decision.

Key matching should be documented. Trimming outer whitespace is usually safe; lowercasing may not be. abc and ABC can be the same customer in one system and two distinct case-sensitive codes in another. The audit should not guess.

→ Remove exact or selected-column duplicates

Step 4

Check value types, not just inferred column labels

Type inference summarizes the dominant shape of a column. If 999 Revenue cells are numeric and one contains pending, the column is still probably described as numeric — but that one value is exactly what a validation rule must catch. Test every nonblank value against the expected type rather than trusting one inferred label.

Use a small, explicit vocabulary: number, date, boolean, text, or any. Define which boolean spellings you accept and which date formats your downstream system can parse. A permissive browser date parser may understand text that a database import rejects, so use a pattern rule as well when a strict ISO shape such as YYYY-MM-DD is required.

Keep type failures distinct from missing failures. A blank optional numeric cell can be valid; not-a-number is not. Row-level evidence should name the expected type, the failing value preview, and the source row.

→ Turn trusted inferred types into a code schema

Step 5

Use explicit ranges; treat outliers as observations

A numeric minimum or maximum is a business rule: Quantity must be at least 0; Discount must be no greater than 100; Fiscal Year must fall within the accepted import window. These constraints can produce a pass or fail because the boundary is explicit and reproducible.

An outlier rule answers a different question. The common IQR fence uses Q1 − 1.5 × IQR and Q3 + 1.5 × IQR, where IQR = Q3 − Q1. Values outside the fence are far from the middle half of the observed data. They may be typos, rare but legitimate orders, fraud signals, or the most important records in the file.

Therefore, list IQR outliers under observed findings and show the calculated fences. Do not make them contract failures unless a domain owner converts the observation into a deliberate minimum or maximum. This keeps statistical shape from pretending to be business policy.

→ Inspect or flag IQR and standard-deviation outliers

Step 6

Validate categories and formats deliberately

Allowed-value rules work for controlled fields such as Status, Currency, Country Code, or Department. Compare exact trimmed strings against the explicit list and show the accepted values in the rule. Do not silently learn the list from the current file: an unexpected category would then teach the contract to accept itself.

Pattern rules are useful for order codes, account numbers, postal codes, and strict dates. Keep the expression visible and portable. For example, ^[A-Z]-\d{3}$ describes one uppercase letter, a hyphen, and three digits. Reject an invalid regular expression at import time rather than storing a contract that cannot run.

Profile case and whitespace variants separately. North, north , and NORTH often signal normalization work, but whether they are equal is a domain decision. The audit can reveal the variants and route to trimming or text-normalization tools without mutating the source.

→ Normalize outer whitespace after review

Step 7

Save the definition, never the audit result

A recurring import should reuse its expectations. Save a contract containing a version, name, business-key column names, and ordered rules. Each rule can name its column, expected type, required/unique flags, minimum, maximum, pattern, and explicit allowed values. That definition can be inspected in plain JSON, imported in another browser, and armed before the next file arrives.

Do not store the filename, rows, observed values, profile counts, findings, failed row numbers, or audit status. Those belong to one run and may contain sensitive customer, payroll, finance, or operational data. A saved definition is useful precisely because it can travel without carrying the previous dataset.

When a new file lacks a contracted column, report it as unavailable and fail visibly; do not fall back to a similarly named field or declare success on the subset that happened to run. Schema mismatch is contract evidence.

Step 8

Export aggregate proof and raw evidence separately

A management or ticket handoff usually needs aggregate evidence: status, counts, applied rule definitions, observed findings, failure totals by column/rule, and the full column profile. Standalone HTML should be script-free, network-free, and protected by a restrictive Content Security Policy; Markdown should escape HTML and table delimiters. Neither needs source rows or failing values.

The remediation owner may need the complete issue ledger: row number, column, rule, failing value, and explanation. Export that as a separate CSV with an explicit warning that it contains raw source values. This separation makes the common report safer while keeping actionable evidence available when the recipient is authorized to handle it.

Finally, record what happened next. Passing a contract proves only that the tested rules passed on that file. It does not prove semantic accuracy, causal correctness, or fitness for every future use. A reproducible audit is a decision aid and control, not a substitute for domain ownership.

→ Build and export the complete audit

Common questions
  • ·

    What is a CSV data quality audit?

    It is a reproducible review of a CSV's observed profile plus explicit validation rules for completeness, keys, types, uniqueness, ranges, patterns and allowed values, with counts and row-level evidence for failures.

  • ·

    What is the difference between data profiling and data validation?

    Profiling describes the file you received. Validation tests whether it meets rules you intentionally defined. Profile findings are observations; contract rule violations are failures.

  • ·

    Should an outlier fail a data-quality check?

    Not automatically. An IQR outlier is an observation relative to the current distribution. It should fail only when a domain owner sets an explicit minimum, maximum, or other rule that the value breaks.

  • ·

    How do I check duplicate business keys in a CSV?

    Choose the column or composite columns that identify one record, require every key part to be nonblank, group exact trimmed key values, and report both duplicate groups and every affected row.

  • ·

    What should a reusable data-quality contract store?

    Store only the contract version, name, business-key column names, and rule definitions. Exclude filenames, rows, observed values, profile results, findings, failures and audit status.

  • ·

    Can a CSV pass a contract and still contain unusual values?

    Yes. Pass means the explicit rules had zero failures. Observed findings such as concentration, case variants or IQR outliers may still deserve review and are reported separately.

  • ·

    Do audit reports need to include raw source values?

    No. Aggregate HTML and Markdown can contain counts, rules, findings and column statistics. Keep row numbers and failing values in a separate, explicitly disclosed issue export for authorized remediation.

  • ·

    Does the online profiler upload my CSV?

    No. CSV and Excel parsing, profiling, contract validation and report generation run locally in your browser; source rows and audit results are not stored.

Keep going