by Jasper Kyle Catapang
Hey there! Jasper here. I’m an AI Engineer in the AI Cowork Agent team here at Money Forward. I also do research on AI ethics, interpretability, and safety. I just got back from ACL 2026, the 64th Annual Meeting of the Association for Computational Linguistics, held in San Diego from July 2–7. I’d like to share both our contributions and my takeaways from the largest NLP venue in the world.
We contributed to three papers: two I’ll go into detail on here, and a third — a thesis proposal on detecting harmful content in code-switched media, presented at the Student Research Workshop (SRW) — that’s interesting work but less directly connected to what we do at Money Forward (SRW paper).
The two main papers are about very different problems on the surface. But sitting in sessions all week, I kept noticing the same thread running through a lot of the work — not just ours, but the conference broadly: how do you catch an AI system being wrong when it’s not going to tell you?
📄 The Papers
When RAGs retrieve conflicting sources of information
“When Image and Text Disagree: Cross-Modal Evidence Conflict in Multimodal Retrieval-Augmented Generation” — MAGMaR workshop, top 10% of submissions.
As you might have seen at the top of this post, that’s me at the MAGMaR poster session — presenting this very paper.
Multimodal RAG systems retrieve both images and text to answer a question. The problem I looked at: what happens when those two sources contradict each other?
The failure mode is specific and hard to catch. Models don’t flag the conflict — they quietly pick a side, or produce something that appears to reconcile both sources but actually matches neither, with no signal that anything went wrong.
The chart below is from the paper and makes this concrete. Even in the image-correct condition — cases where the image actually has the right answer — most models still lean heavily toward the text source (TXT, in orange). Qwen3-VL-4B is the closest to balanced, but even there, text dominates. The gray (NEI/ABS) represents abstentions or unanswerable labels.
In financial document processing, this matters a lot. An invoice image showing one amount and a text record showing another is exactly the kind of mismatch that should trigger a review, not a confident answer.
When your metric is measuring the wrong thing
“Toward a Metric Typology for Language Model Evaluation” — GEM workshop.
A position paper arguing that NLP has accumulated a lot of evaluation metrics without a good shared framework for when to use which. Teams end up measuring something that correlates with what they care about, rather than the thing itself — and then wonder why improvements in evaluation don’t translate to improvements in production.
The core proposal is a four-primitive decomposition: any metric can be expressed as a combination of how outputs are represented (φ), how they’re compared (D), how signals are aggregated (A), and what context (C) the measurement happens in. The formula and the four primitives are in the figure below.
“93% accuracy” is almost meaningless until you know accuracy against what, under what conditions, and whether accuracy is even the right property to be measuring for your specific task. The typology makes those choices explicit rather than implicit.
A model that picks the wrong side of a conflict confidently is hard to catch. A team that picks the wrong metric confidently is also hard to catch — and arguably harder, because the feedback loop is slower.
🔭 The Conference Was Thinking About This Too
Philip Resnik opened with a keynote arguing that the commercial explosion of LLMs has created a “success catastrophe” — so much investment in shipping applications that the foundational linguistic research underneath them is being neglected. Hard to dismiss, honestly — the reliability problems that actually show up in production tend to trace back exactly to the foundational gaps he was describing. I’m glad to be in Money Forward where we take both into account seriously.
The awards reflected the same instinct: the most significant cluster went to work on Reinforcement Learning with Verifiable Rewards, training models to verify their own reasoning steps against ground truth rather than just producing plausible output. New adversarial evaluation tools like REST stress-test models by running them through multiple problems simultaneously to find where they actually break, not where they score well. Less “look at these benchmark numbers,” more “where does this actually break in the wild.”
🏆 A Good Coda
The CL High-Impact Paper Award went to Universal Dependencies, by Marie-Catherine de Marneffe, Christopher Manning, Joakim Nivre, and Daniel Zeman — a shared grammar annotation framework that works across 100+ languages, and the backbone that makes cross-lingual NLP possible at all. The photo below is from the award ceremony.
It’s load-bearing work for basically all of cross-lingual NLP — the kind of thing nobody mentions because it’s just always there. In our context, the reason LLMs can handle both Japanese and English as fluently as they do — the thing we rely on every day in our LLM agents without thinking about it — traces back to this framework giving models a shared syntactic structure to learn from across languages.
Good timing given Resnik’s keynote — it’s a concrete example of what the slow, careful work actually produces.
🧳 What I’m Bringing Back
Both papers ended up connecting to things I’ve been thinking about on the team.
The conflict detection finding changes how I think about the agent’s retrieval layer. Agents regularly pull context from multiple sources — documents, conversation history, structured data. When those sources disagree, the current behavior is to let the model sort it out. That’s exactly the failure mode I studied. A lightweight conflict-flagging step before the model responds would make disagreements visible instead of silently resolved.
The metric typology is something I want to apply to how we evaluate agents overall. Right now we tend to reach for a small set of familiar metrics and apply them across the board. But different parts of the pipeline have genuinely different properties worth measuring — response structure, factual grounding, task completion — and conflating them into a single score makes it hard to know what’s actually improving. The typology gives us a framework to be more deliberate about this.
Big thanks to the AIPD division, Matsumoto-san, and Horisawa-san for the support and encouragement — it genuinely helps to have people behind you when you’re trying to do product work and research at the same time. And to Money Forward for making it possible to show up to these conversations at all.