Receipt ingestion: design and implementation evolution

Evidence, design changes, experiments, and open decisions for PDF and camera receipt ingestion.

Status and purpose

This is a living engineering record for receipt ingestion. It complements the original Smart Automation design, which describes the initial proposal but not the implementation changes made while testing real documents.

Status: Closed as the initial receipt-scanning investigation on 2026-07-21. The implementation, tests, evidence fixtures, and known limitations are preserved for a future follow-up if more samples justify one.

The current conclusion is narrower than "receipt scanning works":

Preserving OCR coordinates is materially better than parsing flattened text.
A store-agnostic table parser can find the two Zomato rows and the single Vijetha row in the supplied samples.
The current numeric-column lookup can reuse a Rate value as Qty when its search tolerances overlap.
Receipt quantity, package content, and inventory quantity are different concepts and are not yet modeled cleanly through staging.

No production correction for the quantity issue is part of this record. The purpose is to preserve the evidence and make the next change deliberate.

Evidence boundary

Two real camera samples have now been evaluated: one Zomato restaurant-delivery receipt and one Vijetha grocery receipt. The Zomato sample is useful for validating OCR layout reconstruction but cannot establish grocery quantity semantics. The Vijetha sample adds one genuine grocery data point, but two receipts still cannot establish cross-vendor reliability.

Both source photographs are stored in docs/fixtures/receipt-scanning/ with explicit authorization from their owner. They remain outside application resources and are not packaged into the Android application. Tests retain a small, receipt-table geometry fixture derived from ML Kit output for the Zomato sample. Controlled "copies" in the tests are transformations of that fixture; they are not independent receipts or additional OCR runs. The Vijetha result is currently an observed device result, not a checked-in OCR geometry fixture.

Claims in this document use these labels:

Observed: Seen in the supplied image or the application result reported during device testing.
Reproduced: Produced deterministically by a checked-in unit test.
Hypothesis: A likely explanation that still requires captured runtime evidence or more samples.

Design evolution

1. Flattened OCR with a generic line parser

The original camera pipeline passed Text.text to a generic parser. This discarded word coordinates and treated receipt metadata, totals, voucher lines, and other text as possible items.

Observed result: One scan produced approximately 30 review items from a two-item receipt.
Lesson: Generic text-line heuristics must not convert arbitrary OCR lines into inventory candidates. A fallback must fail closed when its result is implausible.

2. Position-aware BigBasket PDF parsing

The BigBasket PDF importer improved reliability by preserving word positions and reconstructing a visual table. It identifies columns and rows before mapping values into ParsedLineItem.

Lesson: Geometry is part of the document’s meaning. Both PDF extraction and camera OCR should preserve it.

3. Initial camera adaptation

The first camera adaptation used positioned OCR plus receipt/vendor-specific recognition. Runtime issues and sample-specific spelling accommodations were corrected during development, but vendor names and OCR misspelling variants were rejected as the long-term parsing strategy.

Lesson: Vendor identity may label a source, but structural parsing should not depend on a growing set of vendor parsers or misspelling aliases.

4. Store-agnostic positioned table parser

The current parser identifies semantic Qty, Rate, and Amount headers, anchors rows on the Amount column, reconstructs descriptions geometrically, and validates item count and subtotal when available. If Qty is missing, it may infer purchase quantity from Amount / Rate when the arithmetic validates.

Observed result: The supplied receipt now produces two item rows rather than approximately 30.
Observed result: A later live scan staged quantities 149 and 159, which are the two Rate values.

Current pipeline

Camera bitmap
  -> ML Kit text recognition
  -> positioned words and reconstructed lines
  -> positioned table parser
  -> generic fail-closed fallback
  -> template matching
  -> staged items
  -> review/configure
  -> inventory

Template behavior is independent of parser confidence: an unseen item requires configuration; a trusted matching template can reuse its saved defaults.

Current receipt-to-model mapping

Receipt concept Parser field Current downstream behavior

Receipt concept	Parser field	Current downstream behavior
Description	`name`, `brand`	Both are copied to staging.
Qty	`purchaseQuantity`	Used while parsing, but not stored as a distinct staging field.
Package content, for example `3 Pcs`	`packageQuantity`, `packageUnit`	May be multiplied by purchase quantity to produce parser `quantity`.
Derived inventory amount	`quantity`, `unit`	Copied directly to `parsedQuantity` and `parsedUnit` in staging.
Rate	`price`	Used for inference and validation, then discarded by the staging model.
Amount	`lineTotal`	Used for row anchoring and validation, then discarded by the staging model.

Description

name, brand

Both are copied to staging.

Qty

purchaseQuantity

Used while parsing, but not stored as a distinct staging field.

Package content, for example 3 Pcs

packageQuantity, packageUnit

May be multiplied by purchase quantity to produce parser quantity.

Derived inventory amount

quantity, unit

Copied directly to parsedQuantity and parsedUnit in staging.

Rate

price

Used for inference and validation, then discarded by the staging model.

Amount

lineTotal

Used for row anchoring and validation, then discarded by the staging model.

This means Puri (3 Pcs) with receipt Qty 1 currently becomes inventory quantity 3 pcs. The reported expectation for this non-grocery receipt is quantity 1: the number of purchased menu items. That expectation should not automatically decide how a grocery item such as 2 x 500 g is represented. The two cases require an explicit policy or separate fields.

Quantity defect

The parser currently searches independently around the center of each numeric header. Each search accepts a word within 7.5% of image width. The searches do not partition the row into exclusive column bands and do not prevent the same OCR word from satisfying multiple roles.

Consequently, if a printed Qty is missing and a Rate value falls inside both tolerances, the same word can become both purchaseQuantity and price. Because a quantity was apparently recognized, the safer Amount / Rate inference is skipped.

For the 960-pixel-wide geometry fixture:

Qty header center is approximately 572 px.
Rate header center is approximately 670 px.
The tolerance is 72 px in either direction.
The original first Rate center is approximately 682 px, outside the Qty tolerance, so missing Qty is inferred as 149 / 149 = 1.
In the controlled tight-column copy, the Rate values are shifted left by 50 px. They then fall inside both Qty and Rate tolerances and are reused as quantities.

The controlled reproduction proves that the current algorithm can generate 149/159. It strongly supports the explanation for the live result, but the exact live OCR boxes were not retained, so the live coordinate path remains a hypothesis until diagnostic capture is added.

Experiment record

Case Input Parser behavior Interpretation

Case	Input	Parser behavior	Interpretation
Real sample	One Zomato restaurant receipt photograph	Two rows detected. Device review later showed quantities `149/159`.	Row reconstruction improved; numeric role assignment remained unsafe.
Real grocery sample	One Vijetha receipt with `Description / Qty / Rate / Amount`; printed row `VIJETHA EGG / 1.000 / 63.00 / 63.00`	One item detected with the expected numeric values. The staged name included trailing summary text, approximately `Vijetha Egg / Items / Qty`.	Core extraction is useful on this sample. The summary line is not always recognized as a row boundary, so non-item text can leak into a multiline name.
Controlled copy A	Captured table geometry; both Qty words and package text removed; numeric columns remain separated	Reproduced `1/1` through `Amount / Rate` inference.	Missing OCR quantities are recoverable when columns do not overlap.
Controlled copy B	Same fixture as A; Rate values shifted `50 px` toward Qty	Reproduced `149/159`.	Independent radius searches permit one numeric word to serve two columns.

Real sample

One Zomato restaurant receipt photograph

Two rows detected. Device review later showed quantities 149/159.

Row reconstruction improved; numeric role assignment remained unsafe.

Real grocery sample

One Vijetha receipt with Description / Qty / Rate / Amount; printed row VIJETHA EGG / 1.000 / 63.00 / 63.00

One item detected with the expected numeric values. The staged name included trailing summary text, approximately Vijetha Egg / Items / Qty.

Core extraction is useful on this sample. The summary line is not always recognized as a row boundary, so non-item text can leak into a multiline name.

Controlled copy A

Captured table geometry; both Qty words and package text removed; numeric columns remain separated

Reproduced 1/1 through Amount / Rate inference.

Missing OCR quantities are recoverable when columns do not overlap.

Controlled copy B

Same fixture as A; Rate values shifted 50 px toward Qty

Reproduced 149/159.

Independent radius searches permit one numeric word to serve two columns.

The characterization tests are in PositionedReceiptTableParserTest. They intentionally record both the successful inference path and the known failure path. The failure-path assertion describes current behavior, not desired behavior, and must change when exclusive column assignment is implemented.

Accepted Vijetha limitation

For the current evaluation, appending Items/Qty to Vijetha Egg is an accepted imperfection because the parser still returns one item with the correct receipt quantity, Rate, and Amount. No vendor-specific cleanup rule will be added for it.

This remains worth tracking because name noise can prevent an otherwise recurring item from matching a learned template. It should be reconsidered only if repeated grocery samples show the same structural problem. A future structural solution would recognize summary rows from their position and numeric pattern rather than remove literal words from item names.

Decisions already made

Preserve word geometry from OCR instead of flattening to raw text.
Prefer structural table parsing over vendor-specific parsing.
Do not gate items using a grocery vocabulary while evaluating extraction.
Use vendor detection only as optional source metadata.
Keep the generic fallback fail-closed when item count is implausible.
Keep Configure tied to whether an item template is known, not parser confidence alone.
Prefer minimal derived geometry for deterministic unit tests. Store original receipt photographs only with explicit owner authorization and outside packaged application resources.

Open design decisions

Exclusive numeric-column assignment

The next parser design should divide the row at the midpoints between header centers. A numeric OCR word must belong to at most one of Qty, Rate, or Amount. Amount remains the row anchor. Missing Qty may be inferred only after Rate and Amount have been assigned uniquely and arithmetic validation succeeds.

Quantity semantics

The model needs to distinguish at least:

purchase quantity: How many receipt line units were bought, for example 1.
package content: What one purchased unit contains, for example 3 pcs or 500 g.
inventory quantity: The amount Homey should add to inventory.

Until more grocery receipts are tested, the scanner should not silently assume that package content always replaces purchase quantity. A safe interim policy is to stage the receipt Qty and retain package text for review.

Staging fidelity

Staging currently loses purchaseQuantity, packageQuantity, packageUnit, price, and lineTotal as structured fields. Preserving them would improve diagnostics, allow validation after parsing, and avoid forcing a semantic decision too early.

Confidence and rejection

Confidence should reflect structural invariants, including unique column assignment, arithmetic consistency, declared item count, and subtotal consistency. Low confidence should not create dozens of review items; it should reject the parse or present a clearly bounded recovery flow.

Sample and test plan

The next evaluation set should contain independent photographs, not only transformations of the current sample:

Packaged grocery items with receipt Qty 1 and package size such as 500 g.
Multiple packages such as Qty 2 of 500 g.
Loose or weighed produce where Qty itself is a weight.
Receipts with Qty, Rate, and Amount columns.
Receipts with only description and Amount.
Multi-line descriptions and wrapped numeric rows.
Discounts, vouchers, taxes, packaging charges, and negative values.
Perspective skew, shadows, folds, blur, and partial crops.
At least two receipts from each of several vendors before adding vendor-specific logic.

For each real sample, record:

Image characteristics and whether the full receipt is visible.
Raw OCR text and positioned word boxes, with personal data removed.
Detected headers and column boundaries.
Parsed purchase quantity, package content, Rate, and Amount.
Expected staged quantity and the reasoning for that expectation.
Validation results and whether the parser accepted or rejected the receipt.

Implementation references

app/src/main/kotlin/dev/sysout/homey/app/features/automation/parsing/ReceiptDocumentContent.kt
app/src/main/kotlin/dev/sysout/homey/app/features/automation/parsing/PositionedReceiptTableParser.kt
app/src/main/kotlin/dev/sysout/homey/app/features/automation/parsing/GenericReceiptParser.kt
app/src/main/kotlin/dev/sysout/homey/app/features/automation/parsing/ReceiptParser.kt
app/src/main/kotlin/dev/sysout/homey/app/features/automation/ui/ScannerViewModel.kt
app/src/test/java/dev/sysout/homey/app/features/automation/parsing/PositionedReceiptTableParserTest.kt
docs/fixtures/receipt-scanning/zomato-delivery-receipt.jpg
docs/fixtures/receipt-scanning/vijetha-grocery-receipt.jpg
docs/fixtures/receipt-scanning/README.adoc

Revision history

Date	Version	Change
2026-07-21	0.3	Added both owner-authorized source photographs as non-packaged evidence fixtures and closed the initial investigation.
2026-07-21	0.2	Added the first real grocery sample: a Vijetha receipt with correct item and numeric extraction but accepted `Items/Qty` name leakage.
2026-07-21	0.1	Recorded the flattened-text failure, position-aware evolution, two-item improvement, quantity mapping gap, `149/159` reproduction, evidence limits, and next test plan.