KHOLO LogoDocs v1.0
Architecture

System Architecture & Data Flow

High-level architecture, message processing pipeline, and domain entities.

This document provides a comprehensive technical overview of the KHOLO platform architecture, data flow, component interactions, and multi-tenant domain models.

1. High-Level Architecture Pipeline

End-to-End Order Processing Flow
Async Background Task Queue Architecture
1. IntakeWhatsApp

Customer Messages

Incoming text messages and voice notes arrive via Meta Cloud API webhooks or counter kiosks.

2. Gateway< 50ms

Fastify & Background Task Queue

Verifies HMAC-SHA256 signature, responds 200 OK immediately, and queues job in Redis 7 broker.

3. BrainAI Engine

Transcription & Match

Internal Speech-to-Text Model transcribes audio, Gemini 2.5 Flash extracts items, and SkuMatcher canonicalizes SA trade slang.

4. DispatchSage ERP

Sales Order & Pick Slip

Clerk approves with 1 click; order syncs idempotently to Sage 200 and warehouse pick slip prints.

2. Core Components

A. Fastify REST Gateway (apps/api)

  • Runtime: Node.js $\ge$ 22.0.0 with TypeScript (ts-node in dev, compiled tsc in prod).
  • Security & Performance:
    • Raw body buffer preservation for HMAC-SHA256 Meta webhook signature validation (X-Hub-Signature-256).
    • Strict CORS origin allowlists with environment variable overrides.
    • Helmet HTTP security headers and IP rate limiting (100 req/min per IP with burst protection).
    • PIN-based authentication for warehouse clerks and Supabase JWT verification for managers/admins.

B. Background Task Queue Asynchronous Queue (Redis 7)

  • Queue Engine: Background Task Queue with Redis connection pooling.
  • Worker Isolation: Ingesting webhooks responds immediately to Meta servers in $< 50\text{ms}$ to prevent webhook timeouts or retries. All heavy processing (Internal Speech-to-Text Model audio download, LLM extraction, vector embedding, and ERP sync) executes asynchronously inside apps/api/src/queue/orderWorker.ts.

C. Multi-Tier AI Extraction Engine (OrderExtractor)

The system employs an automated tiered strategy to balance cost, latency, and accuracy:

  1. Tier 1 (Primary): Google Gemini 2.5 Flash via @google/genai SDK. Native structured JSON output and exceptional multilingual comprehension of South African slang (isiZulu, Afrikaans, Sesotho mixed with English).
  2. Tier 2 (Secondary Fallback): OpenAI gpt-4o-mini. Automatically kicks in if Gemini hits quota or rate limits.
  3. Tier 3 (Local Parser / Heuristic): Local rule-based extraction for zero-cost offline development or testing without cloud AI dependencies.
  4. Mock Mode (AI_PROVIDER=mock): Simulates 300–500ms network roundtrips with local FMCG parsing for high-throughput load tests.

D. Audio Transcription Service (TranscriptionService)

  • Handles incoming WhatsApp .ogg (Opus) voice notes up to 25MB.
  • Downloads encrypted media bytes from Meta Cloud API or accepts local uploads in simulator mode.
  • Transcribes speech via OpenAI Internal Speech-to-Text Model API (Internal Speech-to-Text Model-1) with specific prompt conditioning tailored to South African product names, numbers, and units of measure.
  • Persists raw audio in local storage (/uploads/audio/) for clerk review playback in the dashboard.

E. The SKU Matcher & Alias Learning Loop (SkuMatcher)

  • Combines:
    1. Customer-specific alias dictionary lookup (ProductAlias where customerId matches).
    2. Tenant-wide alias dictionary lookup (ProductAlias where customerId is null).
    3. South African FMCG trade canonicalizer (resolves "whites", "stoney", "zamalek", "ricoffy", "dlite").
    4. Packaging unit and pack size normalization (10kg, 12.5kg, 2L, 24x500ml).
    5. Fallback fuzzy/Levenshtein matching with penalty scoring for unit mismatches.

F. ERP Connectors

  • Sage 200 Evolution Connector: Integrates directly with the Sage Freedom Service REST SDK (/Freedom.Core/[DB]/SDK/Rest). Generates idempotent Sales Orders (ExternalOrderNo = order.id) to guarantee zero duplicate bookings.
  • Generic Webhook Connector: Dispatches HMAC-signed JSON payloads to custom third-party ERPs (SAP Business One, Microsoft Dynamics 365, SYSPRO, QuickBooks, or custom distributor backends).
  • Pick Slip Generator: Generates formatted, printable picking slips mapped to physical warehouse aisle and bay locations.

3. Data Model & Entity Relationships

The PostgreSQL database (managed via Prisma ORM) is structured around strict tenant isolation:

EntityPrimary RelationshipsCore Responsibility
TenantParent of Users, Locations, Customers, Products, OrdersRepresents the wholesale distributor; stores ERP configs and WhatsApp phone IDs.
LocationBelongs to Tenant; has Inventory, Users, OrdersPhysical warehouse branch, depot, or dispatch bay.
CustomerBelongs to Tenant; has Orders, AliasesB2B retail buyer identified by unique E.164 phone number. Stores credit status (ACTIVE/HOLD).
ProductBelongs to Tenant; has Aliases, Inventory, OrderLinesCanonical catalog item with SKU, pack size, base price, and warehouse bin location.
ProductAliasLinks Product to Tenant (and optional Customer)The Learning Core. Associates customer slang and trade shorthand with canonical SKUs.
OrderBelongs to Tenant & Customer; has OrderLinesIngested transaction with raw text, total price, status, and Sage ERP Sales Order number.
OrderLineBelongs to Order & optional ProductIndividual line item with extracted quantity, match confidence score ($0.0 - 1.0$), and notes.
AuditLogBelongs to Tenant, User, LocationImmutable compliance record capturing before/after states for all order edits and alias additions.

4. Order State Machine

Orders transition through a deterministic lifecycle from intake to warehouse dispatch:

Order Lifecycle Stages
Stage 1
RECEIVED
Ingested via Meta Webhook
Stage 2
PROCESSING
Internal Speech-to-Text Model & Gemini Matcher
Stage 3
PENDING_REVIEW
Clerk Triage & Teach
Stage 4
APPROVED
Ready for Sage Sync
Stage 5
COMPLETED
Sage SO & Pick Slip Ready

5. Technology Stack Summary

  • Backend Engine: Node.js, Fastify 5, TypeScript 5.9, Prisma ORM 6.19.
  • Database: PostgreSQL 16 with Vector Database extension.
  • Message Broker: Redis 7 + Background Task Queue 6.
  • Frontend Web Apps: Next.js 15 (App Router), React 19, Tailwind CSS 3.4, Lucide Icons.
  • AI & Voice Services: @google/genai (Gemini 2.5 Flash), OpenAI API (Internal Speech-to-Text Model & GPT-4o-mini).
  • Containerization: Docker Compose (docker-compose.yml for local, docker-compose.prod.yml for production).

Documentation Hub

Automated B2B WhatsApp order-processing and ERP integration platform for wholesale and FMCG distributors.

The Learning Loop & Alias Engine

How SkuMatcher, trade canonicalizer, and alias learning work together.

On this page

1. High-Level Architecture Pipeline2. Core ComponentsA. Fastify REST Gateway (apps/api)B. Background Task Queue Asynchronous Queue (Redis 7)C. Multi-Tier AI Extraction Engine (OrderExtractor)D. Audio Transcription Service (TranscriptionService)E. The SKU Matcher & Alias Learning Loop (SkuMatcher)F. ERP Connectors3. Data Model & Entity Relationships4. Order State Machine5. Technology Stack Summary