OVERVIEW
General capability does not automatically become professional competence. Financial work requires a model to interpret customer context, calculate accurately, recognize missing information, weigh interacting trade-offs, use tools correctly, and communicate uncertainty without losing the decision at the center of the task.
01 · Research question
Can expert work make a general model reliably better at a defined financial workload?
We worked with a U.S.-based financial vertical AI company whose identity is withheld at its request. The engagement used Kimi K2.6 as the open-weight base model and focused on the layer around the model: the system that converts expert judgment and real workflows into training and evaluation signals.
The goal was deliberately narrow. We were not trying to make the base model universally superior to frontier proprietary systems. We wanted to make it materially better at the financial reasoning and multi-step workflows the application actually performs.
Demonstrations transfer reasoning patterns
Expert examples should improve how the model structures decisions in context.
Preferences transfer professional judgment
Pairwise choices should distinguish merely plausible answers from useful ones.
Environments improve execution
Realistic workflows should improve tool use, state management, and task completion.
Signals are complementary
The combined pipeline should improve more dimensions than supervised tuning alone.
02 · The training system
Expert judgment became a training signal, not just a source of labels.
Tasks were built around realistic customer states: income, expenses, cash, investment holdings, retirement accounts, liabilities, tax circumstances, documents, objectives, and constraints. Experts identified the actual decision, material facts, missing information, calculations, alternatives, trade-offs, and the conclusion supported by the evidence.
- 01Observe real work
- 02Generate expert tasks
- 03Train and roll out
- 04Verify outcomes
- 05Target failures
Demonstrations
8,400 reviewed examples captured contextual reasoning, assumptions, calculations, and decision structure.
Preferences
6,700 pairwise choices taught distinctions in relevance, actionability, risk awareness, and uncertainty.
Agent trajectories
Controlled environments recorded tools, documents, calculations, intermediate state, and final outcomes.
Verification
Deterministic checks, structured rubrics, and expert adjudication matched the verifier to the task.
03 · Experimental design
Four configurations isolated the value of each training stage.
A fixed 1,200-task held-out set represented investment, tax, debt, retirement, insurance, cash-flow, and cross-domain decisions. No evaluation example was used directly in training. A separate 300-workflow agent evaluation tested end-to-end execution, while approximately 240 outputs received independent expert review.
Base
Kimi K2.6 with no financial-specific post-training.
SFT
Supervised fine-tuning on expert demonstrations.
SFT + preference
Supervised tuning followed by preference optimization.
Full pipeline
SFT, preferences, environment trajectories, and outcome-based optimization.
- Financial correctness30%
- Contextual reasoning25%
- Decision quality20%
- Completeness15%
- Communication10%
04 · Overall performance
The full pipeline improved every scored dimension and cut critical errors.
Supervised fine-tuning produced the first large improvement. Preference optimization added alignment with professional judgment. Environment trajectories and outcome-based training produced the strongest gains in decision quality and multi-step work.
| Evaluation dimension | Base | SFT | + Preference | Full Alpheva |
|---|---|---|---|---|
| Aggregate expert score | 67.4% | 75.8% | 79.6% | 81.9% |
| Financial correctness | 74.8% | 80.6% | 84.3% | 88.1% |
| Contextual reasoning | 61.2% | 72.1% | 77.4% | 82.7% |
| Completeness | 64.7% | 72.9% | 77.1% | 80.4% |
| Decision quality | 60.9% | 70.4% | 75.9% | 79.8% |
| Expert preference | 54.8% | 68.3% | 73.9% | 78.6% |
| Critical-error rate | 8.7% | 5.4% | 4.0% | 3.1% |
All values are percentages. Higher is better, except critical-error rate.
05 · Agentic workflows
The largest shift was coherence across steps, not isolated tool-call accuracy.
The 300-workflow agent evaluation required models to inspect information, call tools, perform calculations, maintain intermediate state, and verify a defined outcome. The post-trained model was more likely to use intermediate results to guide later actions, recognize insufficient information, and confirm the final state.
Scores from the full paper’s agent evaluation. Gains are measured in percentage points.
06 · Ablation analysis
Different training signals corrected different classes of failure.
Removing one signal at a time showed that the stages were not interchangeable. The result is a practical map from observed failure to the kind of data or feedback most likely to address it.
Context became generic
Broad knowledge remained, but the model lost domain-specific patterns for structuring financial decisions.
Judgment alignment fell
Correctness held up better than the ability to prioritize, explain trade-offs, and calibrate uncertainty.
Multi-step execution degraded
Static answers remained comparatively stable while tool use and long-horizon completion declined.
Plausibility displaced outcomes
Language graders could reward strong-looking responses that failed numerical or tool-based checks.
07 · Limits and conclusion
Benchmark gains are evidence of capability, not a substitute for production proof.
Expert data is expensive, professional judgment is not always deterministic, graders can encode their own failures, and financial distributions change with regulation, markets, products, and customer behavior. Specialization can also create trade-offs on unrelated tasks. Production monitoring and recurring expert calibration remain part of the system.
Preserve legitimate disagreement
Ambiguous financial decisions should not be forced into a false single-answer reward.
Calibrate the graders
Automated evaluation must be checked against the behaviors domain experts actually value.
Monitor distribution shift
Tax rules, products, regulations, markets, and customer circumstances continue to change.
Measure the deployed workflow
Held-out evaluation supports a claim; it does not prove reliability in every customer interaction.
Conclusion
The competitive asset is the improvement loop itself.
A vertical AI company does not need to build a foundation model to own differentiated model capability. It can own the workflows that expose failures, the experts who diagnose them, the environments that reproduce them, the evaluators that measure them, and the post-training pipeline that converts those signals into better behavior. For financial AI, the objective is not simply a model that knows finance. It is a model that can perform financial work reliably.
THE COMPLETE RESEARCH
Read the full technical paper.
Domain Adaptation and Post-Training of an Open-Weight Model for Financial Reasoning
23 sections · 6 appendices · Original document