iDrv4 / Ceres TMS V3 Integration

How onyx forwards a connote into the Ceres TMS V3 API. Source of truth for env vars, payload mapping, idempotency, error handling, and the gating switch.

TL;DR

• What it does: Operations users click Send to iDrv4 on the Jobs drawer → the connote is POSTed to {IDRV4_BASE_URL}/consignments → on success the returned control_no is tagged into connotes.additional_data.idrv4_booking_id, the button turns into a green “Forwarded” badge, and the connote can never be double-forwarded from the UI.
• What it doesn’t do (yet): inbound webhooks (consignment.status.changed → onyx tracking). Deferred — see Out of scope below.

Architecture

Operations Jobs drawer
  └── [Send to iDrv4] button
        ↓ POST /api/connotes/<id>/forward-to-idrv4
  api/connotes_api.py::forward_to_idrv4
    ├── idempotency check (additional_data.idrv4_booking_id)
    ├── config check (IDRV4_* env vars)
    └── utils/idrv4_client.forward_connote(connote_dict)
          ├── _get_idrv4_token()           — OAuth client_credentials, 24h cache
          ├── connote_to_ceres_payload()   — onyx connote → Ceres CONSIGNMENT shape
          └── POST {IDRV4_BASE_URL}/consignments
                ↓ {success, data:[…], error?, meta}
          parse envelope → tag connote → return {idrv4_booking_id, control_no}

Configuration

Env vars (loaded by utils/config.py). Local dev sources .env; Railway prod sets them in the Variables panel.

Var	Default	Purpose
IDRV4_BASE_URL	(none)	Ceres V3 API base. https://jat.ceres.idrv.app/v3 (prod) or https://dev1.ceres.idrv.app/v3 (sandbox). Unset → endpoint returns 503 “not configured”.
IDRV4_OAUTH_TOKEN_URL	https://api.idrv.app/oauth/token	OAuth client_credentials endpoint. Override only if Ceres swaps it (sandbox uses its own host today).
IDRV4_CLIENT_ID	(none)	OAuth client id (issued per tenant).
IDRV4_CLIENT_SECRET	(none)	OAuth client secret. Railway secret.
IDRV4_SCOPES	consignment.write consignment.read	Space-separated. consignment.read is the V3 default if none requested.
IDRV4_TIMEOUT_SECS	15	HTTP timeout for both OAuth + consignment POST.

OAuth scopes available in V3: consignment.read, consignment.write, consignment.status, consignment.upload, run.read, invoice.read (others on roadmap). Forwarding bookings only needs consignment.write; reading them back (future webhook fallback poll) needs consignment.read.

OAuth flow

V3 uses Client Credentials Grant (machine-to-machine, no user creds). Tokens last 24h; utils/idrv4_client._get_idrv4_token() caches per-process keyed by (client_id, token_url, scopes) and refreshes 60s before expiry. On a 401 from the API call, the cache is evicted and the request is retried once.

OAuth client modes (set by your Ceres account manager):

• Scoped: the OAuth client is pinned to a single customer. The payload’s customer block is ignored. This is the mode onyx assumes today.
• Admin-scope: the OAuth client can act on any customer in the tenant. Each POST must include a customer block per consignment. Not yet supported — adding it is straightforward (extend connote_to_ceres_payload to include a customer block from customers table when IDRV4_CLIENT_MODE=admin) but would also need disambiguation handling for AMBIGUOUS_CUSTOMER / INSUFFICIENT_CUSTOMER_DATA per-item errors.

Payload mapping

utils/idrv4_client.connote_to_ceres_payload(connote_dict) is a pure function — Connote.to_dict(include_addresses=True, include_packages=True) in, Ceres CONSIGNMENT shape out. Exposed for unit testing.

Ceres field	onyx source
CONSIGNMENT	connote.connote_number (must be unique)
STREET_NO1 / STREET1	parse pickup.address_line1 via regex ^\s*(\d+[A-Za-z]?(?:/\d+)?)\s+(.*)$; if no leading number, STREET_NO1=None, STREET1=address_line1
SUBURB1 / STATE1 / POST_CODE1 / COMPANY1	pickup address row
DATETIME1	pickup_date + pickup_time_from (default 09:00), formatted d-m-Y H:i
ADD_IN1	5 (Tailgate) when is_tailgate_pickup, else null
STREET_NO2 / STREET2 / SUBURB2 / STATE2 / POST_CODE2 / COMPANY2	delivery address row (same shape)
DATETIME2	delivery_date + delivery_time_from (default 17:00) — must be after DATETIME1
ADD_IN2	5 (Tailgate) when is_tailgate_delivery
PRICE	base_rate
FUELLEVY	fuel_levy_amount
SERVICEFEE	0 (no equivalent on connote today)
INSTRUCTIONS	special_instructions
DG_TYPE / DG_CLASS	not mapped today (onyx carries is_dangerous_goods boolean only)
FREIGHT[]	one entry per connote_packages row: {DESCRIPTION, ITEMS=quantity, LENGTH=length_cm, WIDTH=width_cm, HEIGHT=height_cm, WEIGHT=weight_kg}. If no packages, a single default row is synthesised from totals so Ceres’ “at least one freight” rule is satisfied.

ADD_IN values

1=Residential 2=Hand Unload 3=Trailer Split 4=Extra Crew 5=Tailgate 6=Tail Lift. onyx maps only Tailgate today.

Response envelope (V3)

Success — 201 Created (single-success) or 207 Multi-Status (batch with some failed items):

{
  "success": true,
  "data": [
    {
      "consignment":   "CN20260528ALKK",
      "error":         false,
      "message":       "Consignment successfully booked.",
      "connote_id":    123,
      "control_no":    "CN000001",
      "customer":      { "id": 482, "created": false }
    }
  ],
  "meta": { "request_id": "...", "timestamp": "..." }
}

Per-item error (still inside data[], alongside successful items):

{
  "consignment":     "CN20260528ALKK",
  "error":           true,
  "error_code":      "AMBIGUOUS_CUSTOMER" | "INSUFFICIENT_CUSTOMER_DATA" | null,
  "message":         "...",
  "validation_errors": [...],
  "candidates":      [...]   // for AMBIGUOUS_CUSTOMER
  "missing":         [...]   // for INSUFFICIENT_CUSTOMER_DATA
}

Envelope-level error (401 / 422 / 5xx):

{
  "success": false,
  "error": {
    "code":    "VALIDATION_ERROR",
    "message": "The given data is invalid.",
    "details": [ { "field": "consignments.0.STREET1", "message": "The STREET1 field is required." } ]
  },
  "meta": { "request_id": "...", "timestamp": "..." }
}

The client (forward_connote) normalises both shapes into:

# success
{'ok': True, 'control_no': 'CN000001', 'consignment_ref': 'CN20260528ALKK', 'customer': {...}, 'status_code': 201, 'raw': {...}}

# error
{'ok': False, 'error': 'message', 'error_code': '...', 'validation_errors': [...], 'candidates': [...], 'missing': [...], 'status_code': 4xx, 'raw': {...}}

Idempotency

Once a connote is successfully forwarded, four keys are written into connotes.additional_data:

Key	Value
idrv4_booking_id	control_no returned by Ceres (e.g. CN000001)
idrv4_consignment_ref	what we sent as CONSIGNMENT (i.e. the onyx connote_number)
idrv4_forwarded_at	ISO timestamp of the forward
idrv4_status	'sent'

On subsequent UI clicks the backend short-circuits — it doesn’t re-call Ceres, it returns the existing booking id with already_forwarded: true. The drawer also doesn’t show the Send to iDrv4 button when idrv4_booking_id is set; it shows a green “Forwarded to iDrv4 (CN000xxx)” badge instead.

Forcing a re-send

If Ceres ever needs a re-send (their side purged the record, or the consignment ref needs to change), clear the tags with:

UPDATE connotes
SET additional_data = additional_data
    - 'idrv4_booking_id'
    - 'idrv4_consignment_ref'
    - 'idrv4_forwarded_at'
    - 'idrv4_status'
WHERE id = ?;

The button reappears. Note Ceres requires unique CONSIGNMENT values — if their record still exists you’ll need to suffix the connote number too.

Rate limiting

V3 standard tier: 120 req/min. Sandbox: 30 req/min. Premium tier: 600 req/min. Responses include X-RateLimit-Limit / Remaining / Reset headers. onyx doesn’t currently surface these; if we start bulk-forwarding we should read them and back off.

Gating switch

The button itself lives in templates/portals/operations/jobs_list.html (the forwardToIdrv4(id) function). To gate it again (e.g. during an outage), replace the body of forwardToIdrv4 with:

window.forwardToIdrv4 = function(id) {
    showToast('iDrv4 offline. Please try again later.');
};

The backend route and payload mapper stay live — engineers can still test against a working sandbox by curl-ing POST /api/connotes/<id>/forward-to-idrv4 directly.

Out of scope (deferred follow-ups)

• Inbound webhooks — Ceres consignment.status.changed → onyx. Would land at a new route POST /api/idrv4/webhook and append to connotes.additional_data.tracking_events. Plan: register endpoint via Ceres Developer Portal, verify HMAC signature against IDRV4_WEBHOOK_KEY, update connotes.status + tracking on each event.
• Admin-scope OAuth client support — see OAuth flow above. Need: IDRV4_CLIENT_MODE=admin, payload customer block from customers, per-item error handling for AMBIGUOUS_CUSTOMER (prompt ops to pick) and INSUFFICIENT_CUSTOMER_DATA (prompt ops to fill).
• Re-sync on edit — if a forwarded connote is edited, iDrv4’s record drifts. V3 doesn’t expose PUT /v3/consignments/{id} today (PUT /v3/consignments/{id}/status exists but is disabled — returns 403 NOT_ALLOWED).
• Bulk-forward + rate-limit aware backoff — single-connote forwards are fine; bulk operations would need to honour X-RateLimit-Remaining and back off on 429.

Quick reference

File	Role
utils/idrv4_client.py	OAuth token cache, payload mapper, forward_connote() HTTP wrapper
api/connotes_api.py (route /<id>/forward-to-idrv4)	Idempotency + DB tagging + envelope normalisation
templates/portals/operations/jobs_list.html (function forwardToIdrv4)	Drawer button + toast handling
utils/config.py	idrv4_* env-var properties
production_env_template.txt	Env-var template for Railway

Migration markers

• PR #29 — initial integration (live route + payload + button)
• PR #30 — gated the button with a “coming soon” toast (V3 not deployed yet)
• PR #31 — toast text tweak to “iDrv4 offline. Please try again later.”
• PR #34 — promoted the gated state to staging (button visible but no-op)
• PR ? (this PR) — un-gated against the now-live V3 endpoints; added V3 per-item error code handling (AMBIGUOUS_CUSTOMER, INSUFFICIENT_CUSTOMER_DATA); added this doc.