By Jordan Hanley, Senior Solutions Engineer
Background
Here at Hyperscience we talk every day to customers that are hoping to extract highly correct data and business intelligence from handwritten and machine print documents. In a world before Generative AI (we know, it seems long ago) the businesses that we engaged with to automate a business process considered the human cost of completing that process along with the sometimes less tangible costs of incorrect data entering their downstream enterprise systems.
Today, as businesses start to consider the impact that generative AI can have on their day-to-day processes, the requirement for trusted data to ground fine tuned foundation models or the data that fuels retrieval augmented generation (RAG), requirements for data correctness have a renewed focus.
“Through 2026, 30% of generative AI projects will be abandoned after proof of concept due to poor data quality, inadequate risk controls, escalating costs or unclear business value.”
Gartner (2024) – A Journey Guide to Delivering AI Success Through ‘AI-Ready’ Data
With this correctness requirement in mind we also need to consider and balance automation of data ingestion pipelines and business processes. In our conversations with customers, we are often asked to consider document straight through processing (STP).
What is document STP?
Firstly let’s set the definition. In the context of this blog, document STP refers to a process where a document that has been ingested, processed into machine-readable data and output to a downstream system with, all without any human interaction.
Does this approach work?
Often during our conversations, businesses will approach Hyperscience with an idea of the STP level they are looking to achieve. This figure may have come from an incumbent IDP tool or may have come as part of market research. Largely, businesses are looking to achieve 70-90% automation and tools within our industry are commonly achieving these levels. But what does this mean? Let’s use an example:
Business A
Automating a business process that uses 1 document as an input and takes 300 seconds for a human to complete the entire document extraction. The business process is run 1 million times per month.
Document STP: 90%
Work generated: 30,000,000 seconds of human work time equivalent to around 347 days.
The outcome of using this measure leads to businesses being able to fairly accurately predict the amount of resource required to process and manage the exception queue. What is not known is of the 90% that were not sent to a human, how much is correct? Typical solutions do not provide the ability to do this. This can lead to bad decisions being made during further processing or incorrect information making its way to the business or its customers.
During a recent vendor comparison, Hyperscience worked with a customer to measure the correctness of straight through processed documents from a competitor’s solution and found that only 11% of the documents data were over 98% correct.
How can this be done differently?
To take a different approach we must look at the 2 problems in isolation:
- Automation: How do we ensure that the maximum amount of work is being done by the machine?
- Correctness: How do we ensure that we can trust all of the data flowing into our enterprise systems and GenAI?
Hyperscience solves for problem number 1 by introducing machine learning models, grounded on enterprise samples to mimic human behavior when extracting data from a document, breaking the process down into 3 key steps:
- Classification: inferring which type of document is being processed
- Identification: inferring where the interesting information is located in the document
- Transcription: inferring the value of the machine print or handwritten text on the page
Along with this novel approach, Hyperscience introduces a human in the loop (HiTL) interface to catch any instance where a model is not confident that it can do the specific task to the level of correctness that has been set as an input to the process.
“Hey, you said that the only real way to know if data is correct is to check!” we can hear you calling. Hyperscience is here to help with that too, our parallel quality assurance process allows for humans to check and validate a sample set, normally 1-2%, of both the human in the loop as well as the machine’s tasks (because, let’s be honest… humans get it wrong sometimes also). This means that not only can you report on the effectiveness of the business process but also report on the correctness of the data pipeline feeding your Gen AI systems.
Let’s move back to our example from before;
Business B
Automating a business process with Hyperscience, running 1 million times per month, that uses 1 document as an input and takes on average 10 seconds for a human to complete a single task that the machine has flagged for HiTL in order to reach 98.5% correctness (typical human level correctness).
Document STP: 0%
Work generated: 10,000,000 seconds of work equivalent to around 115 days
Importantly, business B can also report on the level of trust that can be placed in the data that is being output from this process and fed into downstream systems as well as generative AI.
The human effort in reviewing these small tasks is significantly reduced due to the size of review that is required and when matured, businesses can easily predict the amount of human resource required. Most importantly, due to the quality assurance process businesses are able to place trust and report on the correctness of the data that is flowing to downstream decisioning or generative AI solutions.
Conclusion
In the face of the changing requirements and consumption of the data that is generated from business processes, it is most important to consider the trust that is placed on this data that is grounding fine tuned foundation models and fuels RAG systems. STP cannot now, and some would argue, could never have delivered on the promise of business value that is required both from an economics as well as a risk perspective.
About the Author
As a member of the Hypersciences international solutions team, Jordan has wide industry experience as both a technical user and provider of automation and customer experience AI powered software. Reach out at [email protected]
