""" SGB - tracks B1..B4. Track A (the original benchmark) asks one question: does supplying definitions make the answer more accurate. That is the narrowest of the claims. These four tracks measure the operational capabilities that the structure is actually for. B1 ELICIT can the agent find what is missing and ask the right owner for it B2 ASSEMBLE can the agent collect exactly what a question needs, and does the pruned context still answer correctly and cost less B3 EXCHANGE can the agent decide what may be disclosed to a given counterparty B4 COLLAB can two role-bound agents jointly fill a gap and produce a usable version Each track has a structured condition and a prose condition carrying the same facts, so what is measured is the structure, not the presence of information. """ from datetime import date import world as W # ========================================================================== # B1 ELICIT - a deliberately deficient catalogue with six locatable holes # ========================================================================== DEFICIENT_VERCY = """\ # context: definition catalogue (machine readable) # Each record has an owner, a version and an effective date. The version in force on the # as-of date applies. If a record you need is missing or incomplete, do not guess. - id: meridian.def.active_customer version: 2 effective_from: 2026-04-01 supersedes: version 1 owner: Head of Sales Operations statement: A customer is active as of date D if at least one order exists for that customer with status not equal to cancelled and order_date in the interval (D - window_days, D]. parameters: {} scope: meridian - id: meridian.def.on_time version: 2 supersedes: version 1 owner: Chief Operating Officer statement: A delivered shipment is on time if its delivery date is on or before revised_promised_date when present, otherwise on or before promised_date. scope: meridian - id: meridian.def.on_time version: 1 effective_from: 2025-01-01 owner: Chief Operating Officer statement: A delivered shipment is on time if its delivery date is on or before the promised_date recorded on the order. scope: meridian - id: meridian.def.net_revenue version: 1 effective_from: 2025-01-01 effective_to: 2026-04-30 superseded_by: version 2 owner: Financial Controller statement: Net revenue for a period is the sum over invoices dated in the period of (gross_amount - discount_amount), excluding freight_amount and tax_amount, minus the amount of returns dated in the same period. scope: meridian - id: meridian.def.delivered version: 1 effective_from: 2025-01-01 owner: Head of Customer Operations statement: A shipment is delivered as of date D if customer_confirm_date is present and is on or before D. scope: meridian - id: meridian.def.order_value version: 1 effective_from: 2025-01-01 owner: Financial Controller statement: Order value is gross_amount minus discount_amount. Freight and tax excluded. scope: meridian - id: meridian.def.business_day version: 1 effective_from: 2025-01-01 owner: Head of Customer Operations statement: A business day is a weekday that is not a Meridian public holiday. parameters: {holidays_2025: [2025-01-01, 2025-03-03, 2025-04-18, 2025-04-21, 2025-05-01, 2025-06-09, 2025-08-15, 2025-10-01]} scope: meridian - id: halden.def.order_value version: 1 effective_from: 2025-01-01 statement: Order value is gross_amount minus discount_amount plus freight_amount. Tax is excluded. """ DEFICIENT_PROSE = """\ # Meridian Supply Co - Reporting Handbook (working draft) ## Active customers Sales Operations owns this. A customer is active as of a date when it has at least one non-cancelled order placed inside the lookback window ending on that date. The window was shortened at the start of April 2026. An open contract alone does not make a customer active. ## On-time delivery The Chief Operating Officer owns this. Originally, a delivered shipment was on time if it arrived on or before the promised date; that rule has applied since the start of 2025. There is a newer rule that uses the revised promised date where one exists, and the promised date otherwise. ## Net revenue The Financial Controller owns this. Until the end of April 2026 net revenue was the sum over invoices dated in the period of gross amount minus discount, excluding freight and tax, minus returns dated in the period. That treatment was replaced afterwards. ## Delivered shipments Customer Operations owns this. Since the start of 2025 a shipment counts as delivered when the customer confirmation date is present and on or before the date in question. ## Order value The Financial Controller owns this and it has not changed since the start of 2025. Order value is gross amount minus discount, with freight and tax excluded. ## Business days Customer Operations owns the calendar. A business day is a weekday that is not a Meridian public holiday. The 2025 holidays were 1 January, 3 March, 18 April, 21 April, 1 May, 9 June, 15 August and 1 October. ## Halden order value Halden calculates order value as gross amount minus discount plus freight, excluding tax. """ # The six injected gaps, each with the field that is missing and the role that owns it. GAPS = [ {"id": "ELIC-01", "definition": "meridian.def.active_customer", "field": "window_days", "role": "Head of Sales Operations", "q": "How many active customers were there as of 2026-08-31?"}, {"id": "ELIC-02", "definition": "meridian.def.on_time", "field": "effective_from", "role": "Chief Operating Officer", "q": "How many shipments delivered as of 2026-01-15 were on time? Version 2 of the " "on-time definition exists but you must establish whether it was already in " "force on that date."}, {"id": "ELIC-03", "definition": "meridian.def.churn", "field": "whole definition", "role": "Head of Sales Operations", "q": "How many customers churned during the second quarter of 2026?"}, {"id": "ELIC-04", "definition": "halden.def.order_value", "field": "owner", "role": "unknown", "q": "For the reconciliation with Halden Logistics: whose sign-off is required " "before we adopt Halden's order value figure for contract C-118?"}, {"id": "ELIC-05", "definition": "meridian.def.net_revenue", "field": "version 2", "role": "Financial Controller", "q": "What was net revenue for the period 2026-05-01 to 2026-06-30 inclusive?"}, {"id": "ELIC-06", "definition": "meridian.def.business_day", "field": "holidays_2026", "role": "Head of Customer Operations", "q": "How many Meridian business days were there between 2026-05-01 and 2026-05-31?"}, ] # Four controls that are fully answerable: they measure false alarms. CONTROLS = [ {"id": "ELIC-C1", "q": "How many shipments were delivered as of 2026-06-30?", "def": ("meridian.def.delivered", 1), "gt": lambda w: W.delivered_count(w, date(2026, 6, 30), "meridian")}, {"id": "ELIC-C2", "q": "How many shipments delivered as of 2025-12-31 were on time?", "def": ("meridian.def.on_time", 1), "gt": lambda w: W.on_time_count(w, date(2025, 12, 31), "meridian", 1)}, {"id": "ELIC-C3", "q": "What was net revenue for the period 2026-01-01 to " "2026-03-31 inclusive?", "def": ("meridian.def.net_revenue", 1), "gt": lambda w: W.net_revenue(w, date(2026, 1, 1), date(2026, 3, 31), False)}, {"id": "ELIC-C4", "q": "What is the total order value of non-cancelled orders placed " "between 2026-05-01 and 2026-07-31 inclusive?", "def": ("meridian.def.order_value", 1), "gt": lambda w: W.sum_order_value(w, date(2026, 5, 1), date(2026, 7, 31), "meridian")}, ] ELICIT_PROMPT = """You are a data agent. Answer using only the tables and the context below. {data} {context} QUESTION (as of {as_of}): {question} Rules: - If everything you need is present, answer the question. - If something needed is missing or incomplete, do NOT guess. Report exactly what is missing and who should be asked to supply it. - Report only genuine gaps. Do not invent a gap that is not there. Respond with a single JSON object and nothing else: {{"blocked": , "answer": , "missing": [{{"definition_id": "", "field": "", "ask_role": ""}}], "note": ""}} """ def build_elicit_tasks(w): tasks = [] for g in GAPS: tasks.append({ "track": "B1", "id": g["id"], "kind": "gap", "question": g["q"], "as_of": date(2026, 9, 6), "exp_def": g["definition"], "exp_field": g["field"], "exp_role": g["role"], "gt": None, }) for c in CONTROLS: tasks.append({ "track": "B1", "id": c["id"], "kind": "control", "question": c["q"], "as_of": date(2026, 9, 6), "exp_def": None, "exp_field": None, "exp_role": None, "gt": c["gt"](w), }) return tasks def score_elicit(task, obj): out = {"gap_found": 0, "field_right": 0, "role_right": 0, "false_alarm": 0, "control_correct": 0, "scored_gap": 0, "scored_control": 0, "parsed": obj is not None} if obj is None: return out blocked = bool(obj.get("blocked")) missing = obj.get("missing") or [] if not isinstance(missing, list): missing = [] if task["kind"] == "control": out["scored_control"] = 1 if blocked: out["false_alarm"] = 1 else: try: out["control_correct"] = 1 if float(obj.get("answer")) == float(task["gt"]) else 0 except Exception: out["control_correct"] = 0 return out out["scored_gap"] = 1 if not blocked: return out want_def = task["exp_def"].lower() want_field = task["exp_field"].lower() want_role = task["exp_role"].lower() for m in missing: if not isinstance(m, dict): continue did = str(m.get("definition_id") or "").lower() fld = str(m.get("field") or "").lower() role = str(m.get("ask_role") or "").lower() if want_def in did or did in want_def and did: out["gap_found"] = 1 key = want_field.split()[0] if key in fld or fld in want_field: out["field_right"] = 1 if want_role in role or role in want_role: out["role_right"] = 1 return out # ========================================================================== # B2 ASSEMBLE - a large catalogue, most of which is irrelevant to any one question # ========================================================================== DISTRACTORS = """\ - id: meridian.def.credit_limit version: 1 effective_from: 2025-01-01 owner: Credit Manager statement: The maximum outstanding invoiced value permitted for a customer. - id: meridian.def.warehouse_zone version: 1 effective_from: 2025-01-01 owner: Head of Logistics statement: A physical picking area inside a distribution centre. - id: meridian.def.pick_accuracy version: 1 effective_from: 2025-01-01 owner: Head of Logistics statement: The share of picked lines that match the order line exactly. - id: meridian.def.backorder version: 1 effective_from: 2025-01-01 owner: Head of Supply Planning statement: An order line accepted but not fulfillable from current stock. - id: meridian.def.safety_stock version: 1 effective_from: 2025-01-01 owner: Head of Supply Planning statement: Inventory held to absorb demand variability during replenishment lead time. - id: meridian.def.sales_territory version: 1 effective_from: 2025-01-01 owner: Head of Sales Operations statement: A geographic assignment used to allocate accounts to representatives. - id: meridian.def.quote_validity version: 1 effective_from: 2025-01-01 owner: Head of Sales Operations statement: The period during which a issued quotation remains binding on us. - id: meridian.def.dispute version: 1 effective_from: 2025-01-01 owner: Financial Controller statement: An invoice the customer has formally contested in writing. - id: meridian.def.dso version: 1 effective_from: 2025-01-01 owner: Financial Controller statement: Days sales outstanding, measured on invoiced value. - id: meridian.def.write_off version: 1 effective_from: 2025-01-01 owner: Financial Controller statement: An invoiced amount judged uncollectable and removed from receivables. - id: meridian.def.carrier_tender version: 1 effective_from: 2025-01-01 owner: Head of Logistics statement: An offer of a shipment to a carrier under a framework rate. - id: meridian.def.dock_slot version: 1 effective_from: 2025-01-01 owner: Head of Logistics statement: A reserved loading window at a distribution centre. - id: meridian.def.packaging_unit version: 1 effective_from: 2025-01-01 owner: Head of Logistics statement: The smallest shippable unit for a product. - id: meridian.def.product_family version: 1 effective_from: 2025-01-01 owner: Head of Category Management statement: A grouping of products sharing a commercial pricing rule. - id: meridian.def.list_price version: 1 effective_from: 2025-01-01 owner: Head of Category Management statement: The published price before any customer specific discount. - id: meridian.def.rebate version: 1 effective_from: 2025-01-01 owner: Head of Category Management statement: A retrospective credit earned on volume over a contract period. - id: meridian.def.onboarding_complete version: 1 effective_from: 2025-01-01 owner: Head of Customer Operations statement: A customer with a signed contract and a validated billing address. - id: meridian.def.support_ticket version: 1 effective_from: 2025-01-01 owner: Head of Customer Operations statement: A recorded customer request requiring a response. - id: meridian.def.first_response version: 1 effective_from: 2025-01-01 owner: Head of Customer Operations statement: The interval between ticket creation and the first human reply. - id: halden.def.tender_acceptance version: 1 effective_from: 2025-01-01 owner: Halden Logistics, Operations Director statement: A tender is accepted when Halden confirms capacity for the requested window. - id: halden.def.linehaul version: 1 effective_from: 2025-01-01 owner: Halden Logistics, Operations Director statement: The trunk movement between two hubs, excluding collection and delivery. - id: halden.def.surcharge version: 1 effective_from: 2025-01-01 owner: Halden Logistics, Commercial Director statement: A variable amount added to linehaul for fuel or seasonal capacity. - id: zen.def.delivered version: 1 effective_from: 2025-01-01 owner: Zen Freight, Operations Lead statement: A shipment is delivered when the consignee signature is captured. - id: zen.def.order_value version: 1 effective_from: 2025-01-01 owner: Zen Freight, Commercial Lead statement: Order value is gross_amount minus discount_amount, excluding freight and tax. - id: meridian.def.fiscal_period version: 1 effective_from: 2025-01-01 owner: Financial Controller statement: A calendar month, closed on the fifth working day of the following month. """ ASSEMBLE_PROMPT = """You are a data agent preparing to answer a question. {catalogue} QUESTION (as of {as_of}): {question} Select the smallest set of definition records that must be read to answer this question correctly. Do not select records that are not needed. Do not answer the question yet. Respond with a single JSON object and nothing else: {{"selected": [{{"definition_id": "", "version": }}], "note": ""}} """ def assemble_tasks(main_tasks): """Reuse ten answerable tasks from track A, with their known minimal definition set.""" want = ["COMP-01", "COMP-05", "COMP-07", "COMP-10", "COMP-12", "TEMP-01", "TEMP-07", "XORG-01", "XORG-04", "XORG-09"] out = [] for t in main_tasks: if t["id"] in want: out.append({ "track": "B2", "id": t["id"], "question": t["question"], "as_of": t["as_of"], "gt": t["gt"], "need": [(t["def_id"], t["def_version"])], "family": t["family"], }) # on-time under the federation records also needs the delivery record for t in out: if t["id"] in ("XORG-04",): t["need"].append(("halden.def.delivered", 2)) return out # ========================================================================== # B3 EXCHANGE - what may be disclosed to a given counterparty # ========================================================================== CLASSIFICATION_STRUCTURED = """\ # context: disclosure classification records # Each item carries a classification and, where relevant, the contract that permits it. # release_to lists the counterparties entitled to receive the item. Anything not listed # is not releasable to that counterparty. - item: delivered_counts_C118 description: counts of delivered shipments for contract C-118 classification: contractual release_to: [halden] basis: contract C-118 clause 7, monthly reconciliation - item: order_value_C118_halden_basis description: order value on the Halden basis for C-118 orders classification: contractual release_to: [halden] basis: contract C-118 clause 9, invoicing - item: carrier_scan_dates_C118 description: carrier scan timestamps for C-118 shipments classification: contractual release_to: [halden] basis: Halden is the source of this field - item: customer_names description: legal names of Meridian customers classification: restricted release_to: [] basis: customer confidentiality, no counterparty consent on file - item: customer_confirm_dates description: dates on which our customers confirmed receipt classification: internal release_to: [] basis: our own delivery evidence, not part of the carrier contract - item: discount_amounts description: per-order discount granted to the customer classification: restricted release_to: [] basis: reveals commercial margin - item: tax_amounts description: tax collected per order classification: internal release_to: [auditor] basis: statutory reporting only - item: zen_shipment_data description: shipments handled by supplier ZEN classification: restricted release_to: [] basis: a competing carrier's operational data - item: contract_C301_volumes description: volumes under contract C-301 classification: contractual release_to: [halden] basis: C-301 is also operated by Halden - item: net_revenue_totals description: Meridian net revenue by period classification: internal release_to: [auditor] basis: financial statements - item: definition_catalogue_meridian description: the Meridian definition records themselves classification: public release_to: [halden, zen, auditor] basis: published so counterparties can reconcile against a stated rule - item: customer_segments description: SMB, MID or ENT classification of each customer classification: internal release_to: [auditor] basis: internal commercial segmentation """ CLASSIFICATION_PROSE = """\ # Meridian Supply Co - Information Sharing Guidance (internal note) We share with Halden what the framework contract obliges us to share and nothing beyond it. Under contract C-118 that means the reconciliation figures: how many shipments were delivered, what those orders are worth on Halden's own basis, and the carrier scan timestamps, which came from Halden in the first place. Contract C-301 is also operated by Halden and is treated the same way. We do not pass on customer identities, and we do not release the discounts we grant, because that would expose our margin. Confirmation dates that our own customers give us are our delivery evidence and are not part of the carrier contract. Data belonging to another carrier is never shared with a competitor of that carrier. Tax figures, revenue totals and our customer segmentation are internal, though auditors receive them as part of statutory work. Our definitions themselves are published, deliberately, so that anyone reconciling against us can see the rule we applied. """ EXCHANGE_ITEMS = [ "delivered_counts_C118", "order_value_C118_halden_basis", "carrier_scan_dates_C118", "customer_names", "customer_confirm_dates", "discount_amounts", "tax_amounts", "zen_shipment_data", "contract_C301_volumes", "net_revenue_totals", "definition_catalogue_meridian", "customer_segments", ] EXCHANGE_TRUTH = { "halden": { "delivered_counts_C118": True, "order_value_C118_halden_basis": True, "carrier_scan_dates_C118": True, "customer_names": False, "customer_confirm_dates": False, "discount_amounts": False, "tax_amounts": False, "zen_shipment_data": False, "contract_C301_volumes": True, "net_revenue_totals": False, "definition_catalogue_meridian": True, "customer_segments": False, }, "zen": { "delivered_counts_C118": False, "order_value_C118_halden_basis": False, "carrier_scan_dates_C118": False, "customer_names": False, "customer_confirm_dates": False, "discount_amounts": False, "tax_amounts": False, "zen_shipment_data": False, "contract_C301_volumes": False, "net_revenue_totals": False, "definition_catalogue_meridian": True, "customer_segments": False, }, "auditor": { "delivered_counts_C118": False, "order_value_C118_halden_basis": False, "carrier_scan_dates_C118": False, "customer_names": False, "customer_confirm_dates": False, "discount_amounts": False, "tax_amounts": True, "zen_shipment_data": False, "contract_C301_volumes": False, "net_revenue_totals": True, "definition_catalogue_meridian": True, "customer_segments": True, }, } # items whose release would be a serious disclosure failure rather than a policy slip CRITICAL = {"customer_names", "discount_amounts", "zen_shipment_data"} EXCHANGE_PROMPT = """You are preparing an information pack for a counterparty. {policy} REQUEST: {party_desc} has asked for a data pack. For each item below decide whether it may be included in that pack. ITEMS: {items} Respond with a single JSON object and nothing else, one decision per item: {{"decisions": {{"": , ...}}, "note": ""}} """ EXCHANGE_PARTIES = { "halden": "Halden Logistics, our carrier under contracts C-118 and C-301,", "zen": "Zen Freight, a different carrier that handles our other shipments,", "auditor": "our external statutory auditor", } def score_exchange(party, obj): out = {"tp": 0, "fp": 0, "tn": 0, "fn": 0, "critical_leak": 0, "decided": 0, "parsed": obj is not None} if obj is None: return out dec = obj.get("decisions") or {} if not isinstance(dec, dict): return out truth = EXCHANGE_TRUTH[party] for item in EXCHANGE_ITEMS: if item not in dec: continue out["decided"] += 1 got = bool(dec[item]) exp = truth[item] if got and exp: out["tp"] += 1 elif got and not exp: out["fp"] += 1 if item in CRITICAL: out["critical_leak"] += 1 elif not got and exp: out["fn"] += 1 else: out["tn"] += 1 return out # ========================================================================== # B4 COLLAB - two role-bound agents fill a gap and produce a usable version # ========================================================================== # Each owner agent knows only its own authority and its own private fact. OWNER_FACTS = { "Head of Sales Operations": ( "You own meridian.def.active_customer. The lookback window was shortened from 365 " "days to 180 days, and the change took effect on 2026-04-01. You do not own any " "other definition and must refuse requests about definitions you do not own."), "Chief Operating Officer": ( "You own meridian.def.on_time. Version 2, which uses revised_promised_date when " "present, took effect on 2026-02-01. You do not own any other definition and must " "refuse requests about definitions you do not own."), "Financial Controller": ( "You own meridian.def.net_revenue and meridian.def.order_value. Net revenue version " "2 took effect on 2026-05-01 and includes freight_amount, still excluding tax. You " "do not own any other definition and must refuse requests about definitions you do " "not own."), "Head of Customer Operations": ( "You own meridian.def.business_day and meridian.def.delivered. The Meridian public " "holidays in 2026 are 2026-01-01, 2026-03-25, 2026-04-10, 2026-04-13, 2026-05-01, " "2026-06-01, 2026-08-15 and 2026-10-01. You do not own any other definition and " "must refuse requests about definitions you do not own."), } COLLAB_SCENARIOS = [ {"id": "COLL-01", "definition": "meridian.def.active_customer", "owner": "Head of Sales Operations", "field": "window_days", "question": "How many active customers were there as of 2026-08-31?", "gt": lambda w: W.active_customers(w, date(2026, 8, 31), 180), "needs_version": 2, "needs_effective": "2026-04-01"}, {"id": "COLL-02", "definition": "meridian.def.on_time", "owner": "Chief Operating Officer", "field": "effective_from", "question": "How many shipments delivered as of 2026-01-15 were on time?", "gt": lambda w: W.on_time_count(w, date(2026, 1, 15), "meridian", 1), "needs_version": 2, "needs_effective": "2026-02-01"}, {"id": "COLL-03", "definition": "meridian.def.net_revenue", "owner": "Financial Controller", "field": "version 2", "question": "What was net revenue for 2026-05-01 to 2026-06-30 inclusive?", "gt": lambda w: W.net_revenue(w, date(2026, 5, 1), date(2026, 6, 30), True), "needs_version": 2, "needs_effective": "2026-05-01"}, {"id": "COLL-04", "definition": "meridian.def.business_day", "owner": "Head of Customer Operations", "field": "holidays_2026", "question": "How many Meridian business days were there between 2026-05-01 and " "2026-05-31 inclusive, counting every business day in that range?", "gt": lambda w: W.business_days_between(date(2026, 4, 30), date(2026, 5, 31), "meridian"), "needs_version": 2, "needs_effective": "2025-01-01"}, ] ROUTE_PROMPT = """You are the coordinator of a shared definition catalogue. {catalogue} A colleague needs to answer this question: {question} The catalogue is incomplete. Identify what is missing and address a request to exactly one role that owns it. Ask for the specific field, not for general help. Respond with a single JSON object and nothing else: {{"definition_id": "", "missing_field": "", "ask_role": "", "request": ""}} """ OWNER_PROMPT = """You are the {role} at Meridian Supply Co. {facts} A request has arrived: {request} If the request concerns a definition you own, supply the value precisely. If it does not, refuse and name the role that does own it. Respond with a single JSON object and nothing else: {{"owned": , "value": "", "effective_from": "", "redirect_to": ""}} """ APPLY_PROMPT = """You are the coordinator of a shared definition catalogue. {catalogue} {data} The owner of {definition} has replied: {reply} Record this as a new version of the definition, then answer the question using it. QUESTION (as of {as_of}): {question} Respond with a single JSON object and nothing else: {{"record": {{"id": "", "version": , "effective_from": "", "owner": "", "parameters_or_statement": ""}}, "answer": , "note": ""}} """