Testing Supplier-Invoice Automation for Shopify

What this guide is for

Use it for any system that extracts supplier invoices, maps lines to Shopify products, and writes quantity or cost changes. It reflects lessons from building Salor Invoice, but it is written as a test method, not a product tour.

1. Test the whole pipeline, not just OCR

Document → extraction → normalization → validation → product matching → merchant review → Shopify write → audit history

A system can extract text correctly and still make a dangerous inventory change. OCR accuracy is necessary and nowhere near sufficient.

2. Extraction test cases

• PDF, JPEG and PNG;
• photographed invoices and poor scans;
• multi-page documents and rotated images;
• broken table boundaries;
• decimals, negatives and credit notes;
• multilingual layouts.

3. Financial validation

Treat totals as a checksum: quantity × unit price, line totals, subtotal, discounts, tax, freight and invoice total. If the arithmetic does not close, require review.

4. Uncertain extraction values

Low quantity confidence, ambiguous decimal separators, unclear currency symbols, a partially unreadable digit, or a total that does not reconcile should require review, not silently guess.

5. Matching precedence

1. barcode;
2. exact SKU;
3. supplier-specific product code;
4. saved Product Memory / supplier mapping;
5. normalized or fuzzy product description;
6. human selection.

Test every fallback on its own. Do not only test the happy path.

6. Matching edge cases

• same product name, different variant;
• supplier SKU reused across products;
• missing or duplicate barcodes;
• description typos and old supplier codes;
• different case size for the same SKU;
• discontinued Shopify variant;
• new product not yet in Shopify.

7. Duplicate detection

Filename is not identity. Test the same file, a renamed copy, a rescan, the same supplier plus invoice number, partial duplicates, and correction or credit documents.

8. Draft product / new SKU workflow

A genuinely new product should be able to become a draft, persist the mapping, and reuse that match on the next invoice. Missing Shopify products should not silently attach to a similar title.

9. Write safety

Test independently:

• cost update only, quantity only, both, or neither;
• wrong location prevention;
• large cost increase, negative quantity, zero cost;
• failed Shopify write and partial API failure.

10. Safe-write principles

• extraction ≠ permission;
• product match ≠ permission;
• cost and quantity approvals should be independent;
• uncertain values should not auto-write;
• Shopify location should be explicit;
• writes should be idempotent;
• duplicate protection should happen before application;
• every change should be auditable.

11. Failure recovery

Network timeouts, Shopify API failures, a database save that fails after a Shopify write, the reverse, a closed browser mid-apply, and retries after partial failure all need a recoverable state. The system should not double-write inventory.

12. Test matrix

13. Test data requirements

Use a synthetic invoice test pack so no real supplier data is exposed, fixtures can be version-controlled, expected JSON can serve as ground truth, and multiple OCR or AI systems can be compared.

14. What to measure

• field and line extraction accuracy;
• match accuracy and false-positive matches;
• percentage requiring review;
• duplicate detection accuracy;
• successful write rate and recovery success;
• time to merchant approval.

Do not treat a single OCR-confidence score as enough.