From LSA to LLMs: twenty years of teaching machines meaning

A chatbot that measured angles

In 2004, for my Computer Engineering thesis at the University of Palermo, I built LSAbot. The chatbots of the day matched keywords against hand-written rules, and they broke the moment a user chose a different word. I wanted one that would answer a question it had never seen phrased that way.

The tool was Latent Semantic Analysis, a method from information retrieval. You count how often each word appears in each candidate answer, factorize that matrix with a singular value decomposition, and keep only the strongest dimensions, usually a few hundred. Words that keep company with the same other words end up close together, so “price” and “cost” land near each other even though no rule says they should. A user’s question is projected into the same space, and the answer whose vector makes the smallest angle with it is returned.

That was the whole trick: meaning as a direction, similarity as an angle. The project was titled “LSAbot: the chat-bot that understands and learns”, and it could answer questions phrased in words nobody had matched in advance, because it did not depend on hand-written rules or exact keywords. In 2005 it won the Targa Under 30 at the first Pier Giorgio Perotto Innovation Award, the prize named after the engineer behind the Olivetti Programma 101. It is the reason I say I have been working on natural-language AI since 2003.

What LSA got right

The idea that a word is defined by the company it keeps is older than any of this software. What LSA showed, and what LSAbot showed me, is that the idea is computable: you can build the geometry from text alone and it works.

That geometry never went away. In 2013, word2vec learned word vectors from prediction instead of counting, and the vectors were better. In 2017 the transformer replaced the fixed vector with one recomputed for every token in context, and in 2018 BERT made those contextual vectors the standard starting point for almost every language task. The models behind today’s assistants still turn text into vectors at every layer, and they still compare them with dot products. The scale changed by many orders of magnitude. The instinct did not.

Meaning is geometry. That part I got in 2004, and it has not needed revising.

What it got wrong

Almost everything else. LSA ignores word order, so “the bank approved the loan” and “the loan approved the bank” are the same vector. It is linear, so it cannot represent a word that means two things. It could only choose among answers that already existed.

The knowledge base was the real limit. Every answer LSAbot could give had to exist before the conversation started. A modern model generates the answer, and it learned its geometry from more text than any person will read. The inspectability went the other way, though. LSAbot’s space was small enough to examine, and the factorization behind it could be reasoned about. A language model has billions of parameters and behaviour that has to be measured rather than derived.

The detour that was not a detour

After LSAbot I did not go deeper into language. I went to the other side of AI: a PhD in automated reasoning at the University of Genoa with Enrico Giunchiglia, and a stay at the Cork Constraint Computation Centre with Barry O’Sullivan. SAT solving, optimization under preferences, and the automatic generation of high-quality test cases and plans.

That is the formal side of the field, where an answer comes with a proof rather than a probability. It felt like leaving language behind. It turned out to be the training that makes my language work different now.

In between I learned to ship. At ETT I coordinated the team that built interactive museum systems, and we wrote them up at the IEEE Digital Heritage congress in 2013. At App2Check I went back to language with far better tools: our sentiment and aspect systems won all three tasks at ATE_ABSITA 2020 and placed second in irony detection at SENTIPOLC 2016 by 0.0068, and EVA, a chatbot that built its own knowledge base from unstructured text, became a patented method. That was the 2004 problem again, approached differently: the knowledge base LSAbot had to be given, EVA extracted from the text itself.

The formal training came from the PhD. The machine-learning side I kept adding through courses, starting with Andrew Ng’s Machine Learning at Stanford.

Why the formal side still matters

Language models learned the geometry better than anything I could have built. What they did not learn is a guarantee. My 2026 work is about putting the guarantees back from the outside, with the tools from the detour.

One result makes the point. Take a sensitive question a model declines to answer. Re-encode it deterministically in first-order logic, with no language model in the transform, and ask the same model to reason over the formula. Across fourteen frontier models from four providers, commitment rose from 19.8% of direct questions to 48.1% under the symbolic encoding. The paper’s explanation is that safety training was learned mostly on natural-language surface patterns and does not generalize to formal contexts, while the reasoning does. It is a reproducible finding, which is rare for red-teaming, and it comes straight from treating a model as a system to be tested formally rather than a black box to be prompted.

The other half of the same idea is optimization. When a recommendation has to respect hard constraints as well as preferences, a language model can propose, but it cannot promise. A solver can. The method behind my US Patent 12,717,799 B2 computes the provably optimal result under both, and the product built on it pairs the two: the model for understanding, the solver for the guarantee.

Meaning is geometry. Trust is proof. It took me twenty years to want both in the same system.

If I could send one note back to the 2004 version of me, hunched over a term-document matrix, it would be this: the angle was the right idea, keep it. Then go and learn how to prove things, because one day the models will be so good at meaning that proof will be the only thing left to add.

The receipts

Every claim above links to a public record.