IngestX
Core

Ingest

The core ingestion API

The ingest() function is the primary entry point for the @izanagi-labs/ingestx browser package. It orchestrates file parsing, validation, chunk processing, and lifecycle management.

Signature

import { ingest } from "@izanagi-labs/ingestx";
import type { IngestOptions, IngestionInstance } from "@izanagi-labs/ingestx";

function ingest<TRow>(options: IngestOptions<TRow>): IngestionInstance<TRow>;

How It Works

Calling ingest() immediately kicks off the asynchronous processing pipeline. It returns an Ingestion Instance containing lifecycle methods to control the ingestion and a Promise that resolves with the final outcome.

There are three distinct layers to understand:

  1. Options: The configuration you pass into ingest().
  2. Instance: The active, observable object returned immediately.
  3. Result: The finalized data output after processing completes.

1. Options (IngestOptions)

You configure ingestion by passing an options object.

OptionTypeRequiredDescription
fileFileYesThe target spreadsheet file object.
columnsColumnConfig[]YesArray of schemas dictating how columns are bound and validated.
workerbooleanNoEnables Web Worker parsing for CSVs (default: false). Excel parsing uses workers automatically.
chunkSizenumberNoNumber of rows processed per chunk (default: 10000).
byteChunkSizenumberNoTarget byte size for file chunking (default: 10485760 / 10MB).
collectResultsbooleanNoWhether to retain all processed rows in memory (default: false). See Memory Management.
maxCollectedRowsnumberNoHard upper bound on retained rows. Halts ingestion if exceeded.
onProgressFunctionNoCallback fired as rows/bytes are processed.
onChunkProcessedFunctionNoCallback fired after each batch of rows is parsed and validated.

2. Instance (IngestionInstance)

The object returned by ingest() provides lifecycle control over the active task.

interface IngestionInstance<TRow> {
  // Current state (e.g., 'running', 'paused', 'completed')
  get status(): IngestionStatus;

  // Resolves when complete or failed
  result: Promise<IngestionResult<TRow>>;

  // Lifecycle controls
  pause(): void;
  resume(): void;
  cancel(): void;
}

Lifecycle Controls

  • pause(): Halts reading new chunks. The current active chunk will finish processing.
  • resume(): Resumes reading file chunks.
  • cancel(): Immediately halts processing and rejects the result Promise.

3. Result (IngestionResult)

The result Promise on the Ingestion Instance resolves when the file has been processed, or rejects if a fatal error occurs (like a missing required column).

interface IngestionResult<TRow> {
  status: IngestionStatus;
  data: FinalOutput<TRow>;
  error: IngestionError | HeaderMismatchError | null;
}

FinalOutput Data Structure

If collectResults: true is configured, data.validRows and data.invalidRows will be populated with the full dataset.

type FinalOutput<TRow> = {
  totalRows: number;
  validRowsCount: number;
  invalidRowsCount: number;

  // Populated only if `collectResults: true`
  validRows: TRow[];
  invalidRows: TRow[];
  
  // Detailed validation failures for specific cells
  errorsData: {
    rowWiseErrors: Array<{
      rowIndex: number;
      columnKey: string;
      receivedValue: unknown;
      errorMessage: string;
    }>;
  };
};

On this page