fbeta
fbeta offers all-in-one support and expert consulting for healthcare innovators to collaboratively enhance health care for citizens, insured individuals, and patients. They use expert knowledge combined with Large Language Model technologies.
Berlin, Germany
2014
Business Consulting and Services
30-50 employees
Challenge
During the project, fbeta asked to create an evaluation technique which would allow them to score the machine generated answer agreements with respect to the subject matter expert answers. This way fbeta was able to select the LLM that produced the best answers on a set of 50+ benchmark questions.
Results
The BERT sentence similarity, together with the BERT embedding similarity were the two most promissing evaluations to rank the best LLM. Kendall tau was used to find out which LLM gave the best results compared to the subject matter expert answers.
“Review coming soon”
Joshua Bolte
Senior advisor at Polpo