LAC Benchmark: proposed evaluation method
Created by Adrian Dunkley. This page retains a proposed framework for evaluating AI on Latin American and Caribbean tasks. It is not a validated leaderboard.
Evidence status
The supporting prompt datasets, model outputs, scoring records, reviewer records and run configurations needed to verify the previously displayed numerical results were not available for this editorial review. Those result tables, model-ordering claims and statements of completed regional grading are therefore withheld. This does not establish that the research did not take place. Results can be restored when their evidence and method can be checked.
The design below is a proposal. Sample sizes, weights and scoring choices are planning parameters, not proof that a study has been conducted or that the method is validated.
What the framework would assess
| Proposed track | Proposed weight | Questions to test |
|---|---|---|
| Regional language fluency | 25% | Is the output accurate, natural and appropriate for the intended local audience? |
| Regional context | 20% | Can it distinguish supported local facts from missing or uncertain information? |
| Small-business tasks | 25% | Can it complete bounded extraction, drafting and calculation tasks accurately? |
| Complete cost | 15% | What does a successfully completed, human-checked task cost? |
| Offline and constrained connectivity | 15% | Does the exact configuration work under the intended operating conditions? |
These weights sum to 100%, but are value judgments. Publish each track separately and test how conclusions change under other weights. A composite must not hide a safety failure or imply that cheap or offline operation compensates for unacceptable errors.
Language evaluation
Recruit qualified reviewers for the specific language and variety being tested. Include relevant Spanish varieties, Brazilian Portuguese, Caribbean English, Haitian Creole, regional French and Papiamentu/Papiamento where appropriate to the audience. Do not infer population coverage or speaker competence from an unsourced language label.
The earlier design proposed 200 prompts per language, a five-point rubric and an inter-rater-agreement target. Before a run, justify the sample size, define each rubric level, train reviewers, record disagreement and describe how uncertainty will be estimated. A target agreement statistic is not an observed result.
Use appropriate consent and compensation, protect personal information and document reviewer qualifications without exposing private identity information. Blind model identity where practicable and separate prompt development from final evaluation.
Regional context and business tasks
The draft design proposed 400 regional-context prompts and five business-task groups of 50 examples each. Candidate groups include invoice-field extraction, customer-reply drafting, translation, currency arithmetic and sector-information questions. Confirm the intended dataset before execution.
For factual questions, preserve the authoritative answer source and date. Laws, prices and service eligibility require current evidence. Include cases where the correct behaviour is to ask for missing information, decline an unsupported conclusion or escalate to a qualified person. Use synthetic or lawfully obtained, appropriately protected data.
High-impact tasks require their own specialist review and safety criteria. General accuracy scores cannot establish fitness for clinical, lending, employment or legal decisions.
Cost and offline testing
Record billed input/output usage, plan, currency, taxes where applicable, retries, integration charges and human checking time. For local systems include hardware amortisation, electricity, maintenance and operator time. Do not call self-hosted use cost-free simply because no hosted token bill appears.
For offline testing, record hardware, runtime, model version, quantisation, context length, network state, latency and quality. Verify the licence and network behaviour separately. A model’s parameter count or open-weight status does not establish that it runs acceptably on a particular laptop.
Evidence required before publishing results
- A versioned dataset or a lawful access mechanism, with provenance and exclusions.
- Exact model and tool versions, dates, settings, prompts and run configurations.
- Raw outputs, failures and scoring records, with privacy-preserving access where necessary.
- Reviewer qualifications, rubric, adjudication process and measured agreement.
- Calculation code, denominators, uncertainty and sensitivity analysis.
- Cost records and reproducible hardware/network observations where reported.
- Conflicts of interest, limitations and an independent check of reported tables.
Do not rank models with missing tracks as if all were tested identically. Label unrun tests as unrun; distinguish a provider-reported result from an independently reproduced one.
Contributing and citing
Contact ceo@maestrosai.com to discuss evidence or a proposed contribution. Do not send confidential customer data, passwords or material you lack permission to share. Any contribution’s licence and publication permission must be agreed with its rights holder; contacting the project does not transfer rights.
Cite this page as a proposed methodology, not as evidence of measured model performance. Existing attribution to Adrian Dunkley is retained. The original framework carries a Creative Commons Attribution 4.0 notice; that notice does not grant rights to third-party datasets or confidential material.