ICM solves sourcing · Clief Notes

ICM solves sourcing

9 months ago I had to architect a multi-agent system that had a major requirement: all facts the agent says must be cited from a 300+ domain (10M+ relevant webpages) allowlist given by the client.

Needless to say, making sure the LLM didn't do one of these three things was really hard:

A. Present a fact from its training data with no sources

B. Present a fact from its training data with a source (but not related to the source)

C. Present a fact from a source but not related to the query

D. Invent instead of saying it doesn't know

So I had to make this multi-agent system very complex, to be able to check and validate each claim.

ICM + any modern coding agent fixes this issue mostly. These harnesses make the models smart enough to backtrack and validate each claim, without making the architecture very complex.

On my initial tests of accurate sourcing with ICM, I can definitely say that sourcing stops being a major bottleneck. Some optimization had to be made to the stages and rules for the agent to comply to your sourcing rules, but it's nothing compared to the over-engineering I had to do months ago.

4 comments

ICM solves sourcing