Regardless of the healthcare setting, sooner or later, teams considering prompt-based extraction methods and RAG in healthcare face the decision to build or buy. While one assumes considerable spending on R&D, the other involves selecting the right solution that fits the bill. The purpose of this article is to assist practitioners in making a decision in light of important issues associated with both internal development and buying an off-the-shelf product.
What Does Prompt-Based Extraction and RAG Involve?
Prompt based extraction is a process where natural language models (NLMs) take a prompt indicating information to be extracted from a document. For instance, the prompt may be “Please extract the patient name, the diagnosis and the date of the last visit”. In turn, RAG entails an NLM querying a document library to find relevant resources to answer the query. For instance, RAG requires a set of credible or relevant documents (such as payer policies) that will answer the question, thereby offering evidence that backs up the answer provided. Together, extraction and RAG allow the user to go through a document, extract useful information from it, and verify the information by comparing it to credible documents. On the surface, it looks like one tool, but it is actually a combination of different elements as a chain.
What Does Building Involve?
In the case of building an in-house solution, the user takes on the responsibility of ensuring that all components work together seamlessly. Thus, developing such a system requires access to:
- A document scanning tool (for scanned or faxed documents);
- Language models (LMs) with prompts;
- Document search and library of trusted documents;
- Controls to flag inaccurate responses;
- A review board to edit incorrect responses;
- Security and privacy controls;
- Personnel to maintain the system.
The components are not complicated per se, but the successful integration and maintenance of the entire system may be challenging.
When Is Building an In-house Solution More Advantageous?
An in-house solution may yield better results in circumstances such as:
- Unique requirements that cannot be addressed by commercial solutions;
- The availability of LM, search, and healthcare expertise in the organization;
- The need for full control over information placement and security;
- High-volume processing (building may become more cost-effective over time).
If these factors apply to the healthcare setting in question, prompt-based extraction and RAG may well be worth the investment.
What Does Buying Involve?
In essence, the buyer only needs to provide documents and trusted information sources. The vendor, for their part, is responsible for scanners, LM, search, interfaces, and security. A competent vendor will have extensive experience working with challenging documents, particularly in the healthcare setting. Thus, they will have the knowledge and know-how gained through years of working with messy files. For the buyer, such a proposition may be beneficial in terms of time and effort.
When Is Buying More Advantageous?
The buy option is more advantageous than building when:
- Rapid deployment is desired (within months, as opposed to years);
- The healthcare team is small and/or does not have the necessary expertise;
- The organization focuses on the business value of extraction rather than the technical detail;
- High-level security and compliance are priorities;
- Ongoing model maintenance and updates are not a concern.
The disadvantage of buying is the lack of control and customization within the bounds of a single platform. Nevertheless, the vendor will offer sufficient flexibility to meet the organization’s requirements.
Unique Considerations for the Healthcare Field
Healthcare has specific considerations for both options. First of all, patient data is confidential, which means that the organization must know exactly where the data is and who has access to it. Secondly, errors in extraction may result in billing or treatment denials, which means that every extracted element must be traceable to the source document, and human verification is necessary. Thus, these two factors apply to both build and buy options and should be considered during selection and implementation.
Key Questions to Ask When Talking to a Vendor
At a minimum, vendors should be able to answer the following questions:
- Where are our data stored, and who has access to it?
- Can we see the source of any extracted element in the original document?
- Can we edit or create our own prompt?
- Can we add our own documents or policies to the library?
- Can we take our data with us if we cancel the subscription?
- How does the price scale with volume?
If these questions elicit satisfactory answers, the vendor is worth further consideration.
The Reality Check Before Finalizing the Decision
Ultimately, the decision to build or buy is based on several key questions:
- How soon does the solution need to be deployed?
- Does the organization have the personnel to build and maintain the system for years to come?
- How unique are the requirements as compared to the competition?
- What are the security and privacy requirements?
- How much is the company willing to spend over the next three to five years on the solution?
If the system needs to be developed quickly, the team needs to make sure that the in-house RAG and prompt-based extraction solution is viable from a financial standpoint. If the company has the expertise, unique requirements, and volume to justify the costs, building can be a viable option. In other cases, a commercial solution may be a more advantageous option. In many cases, a combination of the two approaches may be the most beneficial.
Conclusion
Build solutions offer more control but require the commitment of the organisation in terms of time, cost, and expertise. At the same time, buying has its advantages in terms of speed, performance, and support, but such a decision requires trust in another organisation. Either way, a proof of value with a small set of documents and a small team of reviewers can offer valuable insights and should serve as the foundation for the final decision.
Frequently Asked Questions
How does HIPAA compliance work?
In-house builds involve the management of BAAs, access logging, and vector database security. Vendor platforms provide you with pre-audited, HIPAA-compliant infrastructure and standardized BAAs.
How do solutions guard against AI hallucinations?
In-house solutions need custom engineering of confidence scoring loops. In vendor solutions, there are built-in grounding systems with clickable citations linked to source documents.
Is it possible to move from procurement to building later?
Yes. Starting off with a platform helps quicken the process and define your requirements. Then, you can move on to developing custom solutions when the volume increases, ensuring data export capability.
What happens with scanned or handwritten documents?
Dealing with poor-quality faxes or clinical handwriting involves sophisticated OCR technology. Vendors provide you with fine-tuned models; in-house solutions require considerable computer vision engineering.
