Post Summary
Convergence’s analysis of 16,000+ Form ADV filings in 2020 identified at least one error or omission at approximately 97% of them.
The prevalence of filings with issues. It is not the percentage of individual data points that were wrong, and it is not an assessment of their materiality.
Citing a filing shows where information came from, not that it is accurate. An AI system can reproduce an incorrect figure, and an agent can carry the error into a sequence of decisions and actions.
Verify the source, validate the record, confirm the context, and check the output.
It says it has developed thousands of business rules that interact with data before it reaches clients, and it views data curation, validation and domain expertise as the essential foundation.
How reliable are regulatory filings such as Form ADV?
Consider Form ADV, a foundational source of investment adviser information, and one of many used by agents. Convergence’s analysis of 16,000+ ADV filings in 2020 identified “at least one error or omission at approximately 97% of them.” That measures the prevalence of filing issues, not the percentage of individual data points that were wrong or an assessment of their materiality.
Public information can be valuable, including regulatory filings. But availability alone does not establish accuracy, completeness or relevance. Proprietary data should meet the same standard.
Why does filing accuracy matter for LLMs and agents?
For LLMs and agents, the lesson is clear: citing a regulatory filing establishes where the information came from. Its accuracy still needs to be tested. An AI system can faithfully reproduce an incorrect asset figure, incomplete disclosure or outdated relationships, and carry that error into its recommendations and actions. An LLM can produce a confident but incorrect answer. An agent can carry that error into a sequence of decisions and actions, multiplying its impact.
This is why source verification, data validation, reconciliation and domain expertise matter. Reliable AI requires scrutiny of both the evidence and the conclusions drawn from it.
What four checks does Convergence apply to data?
Convergence has developed thousands of business rules to interact with data before making it available to clients:
- Verify the source. Can the information be traced to credible, original evidence?
- Validate the record. Has the provider checked the data for errors, conflicting values and missing information? Do they publish a Confidence Level in the data you use?
- Confirm the context. Are the entity, definitions and reporting periods consistently captured, measured and correct?
- Check the output. Does the evidence support the conclusion, and are consequential actions to improve confidence taken and subject to appropriate review?
What does Convergence say builds confidence in data?
At Convergence, we view data curation, validation and domain expertise as the “essential foundation” needed to create “confidence” in the data we provide clients. Quality data strengthens Confidence Levels in your AI. Don’t fool yourself. Data is a serious business and the use of AI in your business amplifies the risk to your investors and your firm.
Improve the quality of the data behind your decisions
Contact Convergence to learn more about how to improve the quality and lower the cost of the data you need to support key growth, profitability and risk management decisions in your business.
Contact Convergence About Data Quality
About the data
Source: Convergence’s analysis of 16,144 Form ADV filings for the 2020 filing year, of which 15,623 had at least one error or omission (approximately 97%). The figure measures the prevalence of filings with issues. It is not the percentage of individual data points that were wrong, and it is not an assessment of materiality.
Key Points
What did Convergence’s Form ADV analysis find?
- Scope. 16,000+ Form ADV filings in 2020.
- Finding. At least one error or omission at approximately 97% of them.
- What it measures. The prevalence of filing issues.
- What it does not. The percentage of individual data points that were wrong, or their materiality.
- Role of Form ADV. A foundational source of investment adviser information, and one of many used by agents.
Why does accuracy matter more with LLMs and agents?
- Citation is not accuracy. Citing a filing establishes where information came from. Its accuracy still needs testing.
- Reproduction. An AI system can faithfully reproduce an incorrect asset figure, incomplete disclosure or outdated relationships.
- LLMs. Can produce a confident but incorrect answer.
- Agents. Can carry an error into a sequence of decisions and actions, multiplying its impact.
- Standard. Public and proprietary data should meet the same standard.
What are the four checks?
- Verify the source. Can the information be traced to credible, original evidence?
- Validate the record. Checked for errors, conflicting values and missing information, with a published Confidence Level?
- Confirm the context. Are the entity, definitions and reporting periods captured, measured and correct?
- Check the output. Does the evidence support the conclusion, with consequential actions subject to review?
What does Convergence say it does?
- Business rules. Thousands, applied to data before it reaches clients.
- Foundation. Data curation, validation and domain expertise.
- Confidence. Quality data strengthens Confidence Levels in your AI.
- Risk. The use of AI amplifies the risk to investors and the firm.
- Next step. Contact Convergence about the quality and cost of data for growth, profitability and risk management decisions.