AI and Business Operations — where the work actually moves

1. Attribute-specific thresholds on a human-labelled validation sample; Instacart reports 95% accuracy for one simple “organic” attribute example.
2. LLM evaluation calibrated against human-labelled samples, with low-confidence and sampled outputs reviewed by humans.
3. Instacart reports similar quality at 70% lower model cost for simple attributes; difficult attributes experienced a 60% accuracy drop with the less powerful model.
4. Instacart reports a 10% recall increase over text-only models for the sheet_count experiment.

Gate: schema and business-rule validation, source traceability, regression detection, versioned rollback, and incident-specific stop conditions.
Evidence status: externally reported production architecture; not an inAi autonomy measurement.

1. These controls address specific Local Law 144 audit and notice requirements where the law applies; they do not eliminate wider discrimination, privacy, accessibility, employment, or data-protection obligations.
2. Where Local Law 144 applies, preserve the independent bias-audit record, public summary, and required candidate or employee notices. Treat those requirements as a minimum, not a complete fairness or legal-risk framework. Ongoing monitoring should also examine job relevance, data quality, subgroup outcomes, human override, accessibility, contestability, and the consequences of false positives and false negatives.

Monitoring fields

• the monitored quantity;
• the denominator;
• the population and time window;
• minimum sample sizes;
• intersectional groups;
• what happens when small samples make the chart unstable;
• the difference between a legal audit, ongoing fairness monitoring, and a statistical process-control alert.

Google Cloud’s Gelato customer story reports that automated engineering-support ticket assignment increased correct routing from 60% to 90% and saved 120 hours of weekly labor. The source is a provider-hosted customer case study; local deployment should separately measure confidence, exception handling, misrouting cost, and time to correction.

1. DoorDash reports a 99% reduction in potentially severe compliance issues; track the remaining severe-event rate, denominator, observation period, and escalation outcome.
2. A shallow online check handles straightforward cases; flagged responses receive LLM-based evaluation, while broader offline quality monitoring is calibrated against random human-reviewed samples.
3. LLM-evaluator requirement
   Treat an LLM evaluator as a calibrated measurement instrument, not as ground truth. Validate it against a human-labelled set, report agreement by defect class and language, test order and prompt sensitivity, repeat a sample across runs, and route uncertain or high-impact cases to human review. Severe-policy gates should not depend on an uncalibrated reference-free judge alone.

Klarna’s February 2024 launch metrics illustrate the scale and speed possible in bounded customer-service workflows. Later 2025 reporting indicates that the company rebalanced toward human availability, service quality, and growth. The combined case supports a reversible configuration model: automation levels should respond to customer outcomes and organizational strategy, not only containment and cost.

Outcome metrics

• repeat-contact rate;
• complaint rate;
• customer effort;
• successful human escalation;
• vulnerable-user outcome;
• abandonment;
• reversal or correction rate;
• long-tail intent coverage;
• employee knowledge and quality feedback;
• downstream revenue or retention where causally interpretable.

• model inference;
• retrieval or data access;
• guardrails and evaluation;
• human review;
• rework;
• integration and orchestration;
• monitoring and observability;
• incident investigation and recovery;
• vendor and infrastructure overhead;
• legal/compliance operations where genuinely attributable;
• change management and worker training;
• downstream error cost.

• a defined lot or time window;
• a stable defect taxonomy;
• random or representative sampling;
• clear handling of duplicate/correlated outputs;
• stratification where risk differs by language, category, retailer, candidate group, intent, or complexity;
• explicit producer and consumer risk;
• a procedure for rejected lots;
• a rule for increasing sampling after incidents or changes.

Sample proportionally to traffic for overall quality estimation, but oversample low-volume, high-impact, newly changed, multilingual, or historically weak segments. Report weighted aggregate results and segment-level results separately.

• who receives the alert;
• who can stop the workflow;
• who approves restoration;
• what evidence is needed;
• how affected users or partners are informed where appropriate.

1. Use domain terminology controls and row-level source or evidence links where they improve reviewability. For EU workloads, verify the exact provider and feature configuration for data location, retention, logging, grounding, caching, and subprocessors; do not rely on an endpoint label alone.
2. Measure downstream catalog consequences—rejection, correction, return, search failure, compliance escalation, and customer-impacting defect—not only extraction accuracy.

1. • Where an AEDT law or high-risk AI regime applies, preserve the required independent audit, public summary, notice, and role-specific records. Treat those controls as a minimum rather than a complete fairness or employment-law framework.

• Monitor selection or scoring outcomes, false-positive and false-negative consequences, job relevance, data quality, subgroup and intersectional effects, human override, accessibility, and contestability. Statistical stability does not by itself establish fairness.
• Preserve enough information to reconstruct the decision-support process: tool and version, intended use, inputs and features where lawful, output, human reviewer action, override or appeal, and the reason for the final decision. Avoid logging sensitive data merely because it could be useful later.

1. Employment-related AI should support accountable human decision-making unless a lawful, validated, and appropriately governed use justifies another configuration. The page should not suggest that a bias dashboard alone makes automated shortlisting safe.

1. Use inexpensive deterministic or similarity checks for straightforward cases where they are valid, then apply an LLM evaluator, retry, or human escalation to flagged cases. Calibrate the evaluator against random human-reviewed samples and monitor severe defects separately.
2. Track containment together with repeat contact, successful resolution, complaint rate, human-handoff success, customer effort, abandonment, and long-tail failure. A high automation rate is not sufficient evidence of a better service.

15.1 Output quality

• precision, recall, accuracy, or exact match where appropriate;
• groundedness or source consistency;
• severe-defect rate;
• calibration;
• coverage and abstention;
• multilingual and segment-level performance;
• consistency across model or prompt versions.

15.2 Operational value

• accepted outputs per hour or fixed budget;
• cost per accepted output;
• review and rework time;
• escalation burden;
• latency to accepted result;
• incident and recovery cost;
• integration and maintenance burden.

15.3 Downstream outcomes

• catalog correction and rejection;
• customer return or complaint;
• repeat support contact;
• time to actual resolution;
• human-handoff success;
• candidate false-positive/false-negative consequences;
• employee or user outcome;
• revenue, retention, or risk only where causal interpretation is credible.

15.4 Human and organizational effects

• expertise retention and learning;
• reviewer fatigue;
• deskilling or overreliance;
• override and appeal behavior;
• distribution of gains across experienced and less-experienced workers;
• job redesign and role clarity;
• employee acceptance and work quality;
• ability to recover when the automated system is unavailable.

15.5 Governance and recovery

• audit-log completeness;
• incident detection time;
• time to containment;
• rollback success;
• restoration approval;
• affected population;
• notification and redress where relevant;
• provider/model/configuration inventory;
• date of last revalidation.

• accepted output;
• operating mode;
• acceptance gate;
• severe defect;
• LLM evaluator;
• acceptance sampling;
• p-chart/u-chart;
• control limit;
• policy/specification limit;
• rollback;
• intended purpose;
• provider/deployer.

Current field evidence does not support a single answer to whether AI should assist, supervise, or run work. Results vary by task, worker, expertise, workflow design, and how well the task fits the system’s capabilities. Some studies find substantial short-term productivity gains, especially for less-experienced workers or tasks inside the capability frontier. Other results show quality losses when users rely on AI outside that frontier. Research on teams also suggests that the design of human–AI collaboration and expertise integration matters, not only individual speed.

For business operations, the practical conclusion is to evaluate the complete work system: task decomposition, quality, human knowledge, escalation, incentives, downstream effects, and reversibility. Faster output is useful only when the organization can still detect error, preserve expertise, and achieve the intended outcome.

Current page

• inAi, “AI and Business Operations — where the work actually moves”

Applied cases

• Instacart Engineering, “Scaling Catalog Attribute Extraction with Multi-modal LLMs”, 1 Aug 2025.
• Uber Engineering, “From Restaurants to Retail: Scaling Uber Eats for Everything”, 6 Aug 2025.
• DoorDash Engineering, “Path to high-quality LLM-based Dasher support automation”, 17 Sep 2024.
• Google Cloud, Gelato customer case study.
• Klarna, “AI assistant handles two-thirds of customer service chats in its first month”, 27 Feb 2024.
• OpenAI, Klarna case page.
• Reuters, “Europe’s AI poster child Klarna taps the brakes on chatbots”, 10 Sep 2025.

Employment and legal sources

• NYC Department of Consumer and Worker Protection, Automated Employment Decision Tools.
• NYC DCWP, AEDT Frequently Asked Questions.
• NYC Council, Local Law 144 legislative record.
• European Commission, Guidelines for providers and deployers of high-risk systems, current page reviewed Jul 2026.
• European Commission, Navigating the AI Act.
• European Commission, Draft high-risk classification guidelines, May 2026.

Data location and retention

• Google Cloud, Agent Platform and zero data retention.
• Google Cloud, Abuse monitoring.
• Google Cloud, Data residency.
• Google Cloud, Service Specific Terms.
• Google Cloud, Platform Terms of Service.

Statistical methods and AI risk management

• NIST/SEMATECH, e-Handbook of Statistical Methods.
• NIST/SEMATECH, What is Acceptance Sampling?.
• NIST/SEMATECH, Lot Acceptance Sampling Plans.
• NIST/SEMATECH, Choosing a Sampling Plan with a Given OC Curve.
• NIST/SEMATECH, Attribute Control Charts.
• NIST/SEMATECH, WECO Rules.
• NIST, AI Risk Management Framework.
• NIST, Generative AI Profile.

Organizational and productivity research

• Brynjolfsson, Li, and Raymond, “Generative AI at Work”, Quarterly Journal of Economics, 2025.
• Dell’Acqua et al., “Navigating the Jagged Technological Frontier”, Organization Science, 2026.
• Dell’Acqua et al., “The Cybernetic Teammate: A Field Experiment on Generative AI and Teamwork”, Organization Science, 2026.
• OECD, “The Effects of Generative AI on Productivity, Innovation and Entrepreneurship”, 2025.
• Raisch et al., “Roles of Artificial Intelligence in Collaboration with Humans”, Management Science, 2025.

LLM evaluation research

• Shi et al., “A Systematic Study of Position Bias in LLM-as-a-Judge”, 2025.
• Panickssery et al., “LLM Evaluators Recognize and Favor Their Own Generations”, NeurIPS 2024.
• Sheng et al., “Analyzing Uncertainty of LLM-as-a-Judge”, EMNLP 2025.
• Li et al., “Opportunities and Challenges of LLM-as-a-judge”, EMNLP 2025.

---

SOURCE Instacart Engineering — “Scaling Catalog Attribute Extraction with Multi-modal LLMs” Published: 1 Aug 2025 Evidence type: company engineering report Relevant scope: catalog attribute extraction Reported result: 95% accuracy for one simple “organic” attribute; 10% recall increase for multimodal sheet_count; 70% lower model cost for simple attributes; 60% accuracy drop on difficult attributes with weaker model Limits: not an independent cross-company benchmark; results are attribute- and configuration-specific [Read source]

1. Instacart PARSE.
2. Uber INCA.
3. DoorDash support automation.
4. Gelato customer case.
5. Klarna launch case plus later update.
6. NYC Local Law 144 official materials.
7. EU AI Act official materials.
8. NIST acceptance sampling and control charts.
9. NIST AI RMF.

This can be a compact references drawer rather than a large visible section.

---