Core
CSV Processing
Implementation details for CSV files
IngestX employs highly optimized parsing techniques explicitly tailored to .csv files.
Incremental vs Buffered
CSV parsing is fundamentally chunk based. Instead of pulling the entire CSV document into memory at once, the engine reads and processes chunks.
- The underlying parser reads up to
byteChunkSize(default:10MB) from the file buffer. - It breaks the data into predictable batches defined by
chunkSize(default:10,000rows). - The chunk is yielded asynchronously back to the controller, allowing the main thread (React UI) to render frame updates and keep the user interface responsive.
File Reading Capabilities
- Browser: Can employ Web Workers (when
worker: trueis configured) to offload the parsing workload from the main thread. - Node: Directly hooks into
fs.createReadStream.
Memory Footprint
For large CSV datasets, use collectResults: false to avoid accumulating processed rows in memory. IngestX processes the file incrementally, so memory usage does not grow with the number of collected result rows.
Edge Cases
- Empty Files: If a CSV is entirely empty, IngestX resolves the ingestion cleanly with 0 rows processed.
- Missing Required Headers: Detected instantaneously on the first read. The remainder of the file is immediately aborted to save processing time.