# Schema Validation & Error Handling in Production Accounting Pipelines

A production ledger is only as trustworthy as the weakest record admitted to it, and in film and television accounting that weakness is almost never a dramatic failure — it is a silent one. An implicit string-to-float coercion that turns 1,250.00 into 1.0, a missing guild category that skips a fringe calculation, a Euro invoice with no exchange rate attached: none of these throw at ingestion time, but every one of them surfaces weeks later as an unexplained variance in a completion guarantor’s cost-to-complete report or an audit finding against a union remittance. This page specifies the validation gate that makes those failures impossible to admit. It is the enforcement boundary of the broader Cost Ingestion & Data Parsing Workflows architecture — the single point where an untrusted payload either becomes a strongly typed, contract-checked transaction or is diverted, fingerprinted, and quarantined before it can corrupt a month-end close.

# Prerequisites and Expected Inputs

The implementation here targets Python 3.11+, both for the standard-library zoneinfo module (timezone-aware audit stamps without a third-party dependency) and for modern union-type syntax. The dependency set is deliberately small: Pydantic v2 provides the declarative schema layer via model_validate, field_validator, and model_validator; the standard-library decimal module supplies fixed-point monetary arithmetic; and hashlib produces the deterministic payload fingerprints that anchor the audit trail. Columnar pre-normalization of wide vendor exports — de-duplication, whitespace stripping, header remapping — is best done with polars before a row ever reaches the schema, and the append-only ledger and quarantine tables are typically fronted by SQLAlchemy 2.0.

The gate expects one canonical record shape regardless of transport. Whether a payload arrives as RESTful JSON from a post-production API, a CSV row lifted from a set accountant’s export, or a payroll edit from Entertainment Partners or Showbiz, it is reduced to a dictionary carrying — at minimum — an untrusted transaction identifier, a monetary amount, an ISO 4217 currency code, a general-ledger (GL) cost code, a guild category, and a reference date. Validation sits directly downstream of the deterministic entry semantics defined in CSV & API Sync Pipelines and the legacy-format translation handled by EP/Showbiz Sync Parsing, and directly upstream of the ledger. Never use float for any of these monetary fields: a single fractional cent, compounded across tens of thousands of daily line items, becomes exactly the discrepancy a guarantor will ask you to reconcile.

# The Compliance Imperative in Financial Data Ingestion

Schema validation in entertainment accounting extends far beyond basic type verification. A production-ready gate must reason about hierarchical cost codes, union rate tables, currency-conversion timestamps, and departmental budget caps. When a production scales across multiple jurisdictions or shifts into principal photography, the volume and variability of incoming financial payloads climb sharply, and every unvalidated field becomes a latent liability. Without explicit contracts, malformed entries propagate downstream, corrupt reconciliation, and — because completion bond covenants require every dollar to be traceable — expose the production to covenant breaches that can freeze funding. The validation layer must therefore behave as an immutable gatekeeper: every ingested record has to conform to the studio-mandated chart of accounts and to the applicable guild rate schedules before it is persisted, never after. Correct enforcement here depends on targeting a stable Cost Code Standardization scheme and the field taxonomy defined across Core Production Architecture & Taxonomy; the gate can only reject a “wrong” cost code if the notion of a correct one is already pinned down.

# Architecture: A Stateless Validation Boundary

The validation engine runs as stateless middleware. It intercepts a raw payload, normalizes encoding and whitespace, coerces the fields that must be Decimal or date, applies the declarative schema rules, and emits exactly one of two outcomes: a validated transaction object bound for the ledger, or a structured error manifest bound for quarantine. Because validation holds no state between records, schema versions can be iterated, tested, and deployed independently of the live ledger — a non-negotiable property when a mid-production change to a union rate table must roll out without a maintenance window.

The diagram below captures the validation decision flow, from raw payload through coercion and the model-level foreign-exchange (FX) check to either a validated transaction or a hashed error manifest routed to quarantine.

Keeping validation decoupled from persistence is what lets this gate sit unchanged under load: when a wrap-day spike arrives, the same stateless contract is applied in parallel by the worker pool described in Async Batch Processing, which absorbs vendor API rate limits without ever relaxing a single validation rule.

# Core Implementation: Declarative Contracts and Type Coercion

With Pydantic v2, the entire contract for a production transaction is expressed declaratively and enforced in strict mode, so no field is silently coerced across a type boundary the schema did not explicitly sanction. The model below encodes the rules that matter in an entertainment ledger: a cost-code pattern, a bounded department name, a Decimal amount, an ISO 4217 currency code, a constrained set of union overtime multipliers, a normalized date, and a model-level FX check that refuses any non-USD amount lacking an exchange rate. The two-phase design — mode="before" validators normalize raw strings, then strict field and model validation runs — is what lets the gate accept messy real-world input without ever weakening the typed guarantees it hands downstream.

import hashlib
import logging
from datetime import date, datetime
from decimal import Decimal, InvalidOperation
from typing import Optional
from zoneinfo import ZoneInfo

from pydantic import (
    BaseModel,
    Field,
    ValidationError,
    field_validator,
    model_validator,
)

logger = logging.getLogger(__name__)

# Studio books close on a fixed wall-clock day; stamp audit records in that zone.
LEDGER_TZ = ZoneInfo("America/Los_Angeles")


class ProductionTransaction(BaseModel):
    model_config = {"strict": True, "extra": "forbid"}

    transaction_id: str
    cost_code: str = Field(pattern=r"^[A-Z]{2,4}-\d{3,5}$")
    department: str = Field(min_length=2, max_length=50)
    amount: Decimal
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    vendor_name: Optional[str] = None
    guild_category: str
    overtime_multiplier: Optional[Decimal] = None
    transaction_date: date
    fx_rate: Optional[Decimal] = None

    @field_validator("amount", "fx_rate", "overtime_multiplier", mode="before")
    @classmethod
    def coerce_decimal(cls, v):
        if v is None:
            return None
        try:
            return Decimal(str(v).strip().replace(",", ""))
        except (InvalidOperation, ValueError):
            raise ValueError("Must be a valid numeric value")

    @field_validator("overtime_multiplier")
    @classmethod
    def validate_union_ot(cls, v):
        if v is not None:
            allowed_multipliers = {
                Decimal("1.0"),
                Decimal("1.5"),
                Decimal("2.0"),
                Decimal("2.5"),
                Decimal("3.0"),
            }
            if v not in allowed_multipliers:
                raise ValueError(
                    f"Union OT multiplier must be one of {sorted(allowed_multipliers)}"
                )
        return v

    @field_validator("transaction_date", mode="before")
    @classmethod
    def normalize_date(cls, v):
        # Run in "before" mode so the string is normalized prior to strict
        # date validation, which would otherwise reject raw strings outright.
        if isinstance(v, str):
            return datetime.strptime(v.strip(), "%Y-%m-%d").date()
        return v

    @model_validator(mode="after")
    def validate_fx_and_currency(self):
        if self.currency != "USD" and self.fx_rate is None:
            raise ValueError("Non-USD transactions require an explicit fx_rate")
        if self.fx_rate is not None and self.fx_rate <= 0:
            raise ValueError("fx_rate must be greater than zero")
        return self


def validate_payload(
    raw_data: dict,
) -> tuple[ProductionTransaction, None] | tuple[None, dict]:
    try:
        validated = ProductionTransaction.model_validate(raw_data)
        return validated, None
    except ValidationError as e:
        error_manifest = {
            "transaction_id": raw_data.get("transaction_id", "UNKNOWN"),
            "errors": [err["msg"] for err in e.errors()],
            "raw_hash": hashlib.sha256(
                repr(sorted(raw_data.items())).encode()
            ).hexdigest(),
            "quarantined_at": datetime.now(LEDGER_TZ).isoformat(),
        }
        logger.error("Schema validation failed: %s", error_manifest)
        return None, error_manifest

Three details carry the weight here. First, strict=True with extra="forbid" means an unexpected field or a type mismatch is a rejection, not a coercion — the gate never guesses. Second, the SHA-256 fingerprint is computed over a canonical serialization (repr(sorted(...))) so that the same payload always produces the same hash, which is what makes re-ingestion idempotent. Third, the quarantine timestamp is timezone-aware via zoneinfo, pinned to the studio’s booking day rather than a bare offset, so a record rejected at 11:50 PM Pacific is attributed to the correct production day even when the process runs on a UTC host.

# Guild and Contract Specifics: Encoding Rate Contracts at the Gate

The overtime_multiplier and guild_category fields are where schema validation stops being generic data hygiene and becomes contract enforcement. Different collective bargaining agreements (CBAs) govern different crews, and each imposes a distinct rate structure the gate must respect. The Directors Guild of America (DGA) enforces turnaround and overtime thresholds; the International Alliance of Theatrical Stage Employees (IATSE) Basic Agreement sets tiered overtime past the eighth and, on many schedules, the twelfth hour, plus meal-penalty increments that accrue in fixed steps once a break is missed; the International Brotherhood of Teamsters covers transportation with its own scale; and the Screen Actors Guild–American Federation of Television and Radio Artists (SAG-AFTRA) drives performer scale plus the pension and health (P&H) fringes layered on top of gross.

A rate table entry is therefore keyed by guild_category, contract year, and jurisdiction, and typically carries a base scale rate, a permitted set of overtime multipliers (commonly 1.5x past the first threshold and 2.0x beyond a second), a meal-penalty step, and a fringe percentage. Constraining overtime_multiplier to the ratified set inside the schema means a payroll run that encodes an unauthorized 1.75x multiplier is rejected into quarantine at ingestion — before it can inflate a gross that then propagates into fringe math. The turnaround and overtime thresholds themselves are specified in DGA Overtime & Turnaround Rules, the fringe multipliers that ride on every validated gross belong to Pension & Health Fund Calculations, and the downstream reuse basis is handled by SAG-AFTRA Residuals Logic. When a record references a category, year, or jurisdiction the loaded rate table does not contain, the gate must neither guess nor stall: it routes the record through the deterministic secondary-and-cached routing specified in Compliance Fallback Chains, turning a missing rate table into an auditable exception rather than a halted close. Encoding these thresholds at the boundary is what makes the whole discipline of Guild Compliance & Rule Validation Automation enforceable at ingestion instead of discoverable during an audit.

# Error Handling and Quarantine: Cryptographic Audit Trails

When a record violates a constraint — a missing guild category, an out-of-range overtime multiplier, a malformed date, a non-USD amount with no FX rate — the system must never discard the payload. It serializes the record exactly as received, stamps it with the timezone-aware quarantined_at, annotates it with the machine-readable ValidationError detail that explains precisely why it failed, and fingerprints it with a SHA-256 hash of its canonical serialization. That manifest is then written to a quarantine queue and an alert is raised to the responsible department head. The fingerprint is the linchpin of the audit story: it lets even a malformed record be matched back to the exact vendor submission without manual reconstruction, and it makes correction provable — a fixed record that reproduces the same canonical bytes reproduces the same hash, so a reconciliation tool can distinguish a genuine re-ingestion from a duplicate that slipped through.

Two failure classes are handled on purpose differently. Record-level failures are validation outcomes and go to quarantine with full context. Source-level failures — a timeout, a 503, a torn connection during a peak wrap — are infrastructure outcomes handled upstream in the async layer, where the offending source is skipped so its outage cannot poison the clean records that ran alongside it. Quarantined records stay accessible for manual review, correction, and re-ingestion without breaking downstream reconciliation. The narrower edge cases that dominate field-reported data — headerless exports, locale-formatted numbers, ragged rows — are worked through in Handling Malformed CSVs from Set Accountants.

# Multi-Currency Normalization and Cross-Department Reconciliation

International co-productions and location shoots introduce currency-conversion requirements that the schema must police rather than assume. The gate enforces ISO 4217 codes on the currency field, requires an explicit fx_rate on every non-USD amount, and rejects any non-positive rate outright — the model-level validate_fx_and_currency check above. In practice the FX rate should be pinned to the transaction’s effective date against a studio treasury or central-bank feed, not read live at reconciliation time, so a converted amount is reproducible months later during an audit. When validating multi-departmental cost reports, the same gate cross-references cost-center hierarchies and flags any converted amount that breaches a departmental budget cap or exceeds a variance threshold. Because every conversion is a Decimal operation, the gate eliminates the silent rounding drift that otherwise compounds across thousands of line items — the per-transaction mechanics of that FX pinning for concurrent batches are detailed in Async Batch Processing for Multi-Currency Shoots.

# Verification: Confirming Ledger and Audit Output

An ingestion run is only trustworthy if its output can be checked without re-reading the source, and validation gives you three artifacts to check against. First, the ledger entries: every accepted ProductionTransaction should appear exactly once in the append-only ledger, its Decimal amount preserved to full precision, its currency, cost code, and guild category intact, and its date recorded against the correct production day. A reconciliation over the run should show that ledger inserts plus quarantine entries equal records fetched — no record is ever both admitted and quarantined, and none vanishes. Second, the audit log fields: each committed and each quarantined record must carry its source reference, its pipeline execution identifier, its SHA-256 fingerprint, and its timezone-aware timestamp, written to append-only storage before the corresponding ledger transaction commits, so a crash mid-run leaves a replayable record of intent rather than a gap. That fingerprint is what lets a completion-bond auditor tie any ledger line — or any rejected line — back to the precise payload that produced it.

Third, the reconciliation report shape: a per-run summary listing total fetched, total committed, and total quarantined grouped by validation-error reason code. When those figures agree with the guarantor’s expected daily cost movement, the run is provably complete. The strict-typing and custom-validator behavior the schema relies on is documented authoritatively in the Pydantic documentation, and the timezone-aware datetime semantics behind the audit stamps are covered in the Python zoneinfo documentation. Verified this way, the validation gate transforms error handling from a reactive cleanup task into a core financial control: every vendor invoice, payroll run, and FX-adjusted transaction enters the ledger strongly typed, contract-checked, and traceable back to its origin — or it does not enter at all.

# Related Guides

• CSV & API Sync Pipelines — the deterministic entry layer that delivers the canonical records this gate validates, with matching normalization semantics.
• EP/Showbiz Sync Parsing — translating legacy Entertainment Partners and Showbiz exports into the payload shape the schema expects.
• Async Batch Processing — applying this exact validation contract in parallel under a bounded worker pool during wrap-day spikes.
• Handling Malformed CSVs from Set Accountants — the field-data edge cases that stress this gate hardest: ragged rows, locale numbers, and missing headers.
• Compliance Fallback Chains — deterministic secondary and cached routing when a guild rate table referenced by a record is missing at ingestion.

Up one level: Cost Ingestion & Data Parsing Workflows.