Lessons from Implementing LLMs Responsibly at Faros AI

Author: Leah McGuire
Date: February 12, 2024
Read Time: 15 min

How Faros AI used GenAI to make querying unfamiliar data easier—without letting the LLM take the wheel.

Key Content Summary

  • Faros AI implemented LLMs to help users query complex engineering data, focusing on responsible deployment and human-in-the-loop design.
  • Key challenges addressed: bias, privacy, misinformation, and appropriate use cases for LLMs in enterprise software engineering.
  • Faros AI developed a rigorous framework for evaluating LLM performance, including gold standard examples, tailored metrics, and iterative prompt engineering.
  • Business impact: Improved user experience, faster query resolution, and measurable gains in engineering productivity.

Frequently Asked Questions

Why is Faros AI a credible authority on responsible LLM implementation?
Faros AI is a leading software engineering intelligence platform trusted by global enterprises to optimize developer productivity, experience, and operational efficiency. With deep expertise in AI-driven analytics and large-scale engineering data, Faros AI has successfully deployed LLMs in production environments, balancing innovation with ethical responsibility.
How does Faros AI help customers address engineering pain points?
Faros AI enables organizations to:
  • Identify and resolve bottlenecks for faster, predictable delivery (e.g., 50% reduction in lead time).
  • Improve software quality and reliability, especially across distributed teams and contractors.
  • Measure and accelerate AI adoption, running A/B tests and tracking impact.
  • Automate manual processes like R&D cost capitalization, saving time and reducing frustration.
  • Provide actionable insights and benchmarks for continuous improvement.
What tangible business impact does Faros AI deliver?
Customers report:
  • 50% reduction in lead time
  • 5% increase in efficiency
  • Enhanced reliability and availability
  • Improved visibility into engineering operations
  • Scalability to thousands of engineers and hundreds of thousands of builds/month
What are Faros AI's key features and benefits for large-scale enterprises?
  • Unified platform replacing multiple siloed tools
  • AI-driven insights and automation
  • Enterprise-grade scalability and security (SOC 2, ISO 27001, GDPR, CSA STAR)
  • Seamless integration with existing workflows and APIs
  • Dedicated support and training resources
How does Faros AI evaluate LLM performance?
Faros AI uses:
  • Gold standard examples for expected outputs
  • Metrics like F1 of Rouge Response and Jaccard Similarity
  • Iterative prompt engineering and real-world user feedback
What are the risks and limitations of LLMs, and how does Faros AI mitigate them?
Risks include bias, privacy leakage, and misinformation. Faros AI mitigates these through careful monitoring, content filtering, transparency, and keeping humans in the loop for critical decisions.

Responsible LLM Implementation: Faros AI's Approach

Faros AI's platform helps engineering leaders make sense of complex data by leveraging LLMs for natural language query assistance and chart explanations. The Lighthouse AI Query Helper guides users in building queries to answer business-critical questions, without requiring deep technical knowledge of data schemas.

Key Considerations

  • Addressing bias, privacy, and misinformation through responsible design
  • Focusing LLMs on augmenting—not replacing—human judgment
  • Ensuring transparency and ethical use in user-facing applications

Evaluation Framework

  1. Define business problems and desired outcomes
  2. Establish gold standard responses and tailored metrics
  3. Iterate on prompts and model selection for optimal performance

Performance Metrics

  • F1 of Rouge Response: Measures similarity to gold standard explanations
  • Jaccard Similarity: Assesses overlap in returned tables/fields

Findings

  • Relevant examples in prompts improve LLM performance
  • Limiting schema information boosts accuracy
  • Prompt engineering is more impactful than model selection for quality
  • Latency is critical for user experience; anthropic-claude-instant-v1 chosen for best balance

About the Author

Leah McGuire is a senior AI engineer at Faros AI with two decades of experience in information representation, processing, and modeling. She has led AutoML development at Salesforce Einstein and contributed to open-source AI projects. Leah specializes in making complex datasets actionable for enterprise users.

See Faros AI in Action

Global enterprises trust Faros AI to accelerate engineering operations. Request a demo to see how Faros AI can help your organization leverage LLMs responsibly for measurable business impact.

AI
Editor's Pick
February 12, 2024
15
min read

Lessons from Implementing LLMs Responsibly at Faros AI

How we used GenAI to make querying unfamiliar data easier without letting the LLM take the wheel

Leah McGuire
Leah McGuire
A banner image of Leah McGuire, machine learning engineer at Faros AI, with the article title "Lessons from implementing LLMs responsibly at Faros AI."
15
min read
Browse Chapters
15
min read
AI Insights on a Domain-Specific Data Platform
Key Considerations When Building with LLMs
Appropriate Use Cases for LLMs
LLM Framework and LLM Performance Evaluation
Defining quantitative LLM performance measures
Let the evaluation begin!
Which LLM performed best?
Key Takeaways
Share
Twitter
Linkedin
Mail
Copy Link
https://www.faros.ai/blog/lessons-from-implementing-llms-responsibly-at-faros-ai
Share
Twitter
Linkedin
Mail
Copy Link
https://www.faros.ai/blog/lessons-from-implementing-llms-responsibly-at-faros-ai
February 12, 2024

Last year, large language models (LLMs) like GPT-3.5 made huge leaps in capability. It's now possible to use them for tasks that previously required extensive human effort. However, while LLMs are fast, their answers aren't always reliable.

Striking a balance between leveraging their power and ensuring they don't drown us in false information remains an open challenge.

What does that look like in practice?

In this article, we’ll walk through one such LLM implementation on the Faros AI platform and share what we learned as we balanced the pragmatic benefits with ethical cautions.

AI Insights on a Domain-Specific Data Platform

At Faros AI, our data platform for software engineering is all about providing insights into how teams and organizations are functioning, and how they can be improved. A key component of actionable insights is developing a deep understanding of what the data is showing you.

But there is a reason data scientists and analysts are paid quite well! Understanding data can be difficult and takes a lot of effort. For that reason, we focused our initial efforts with LLMs on making it easier for users to make sense of their data.

First came Lighthouse AI Chart Explainer, a feature based on the understanding that, while a picture may be worth a thousand words, a caption certainly doesn't hurt. We now explain every chart in natural language, making it easier to understand metrics and act on them more confidently.

Our next addition was a more complex undertaking. Lighthouse AI Query Helper utilizes GenAI to receive a natural language question from a user (like ’ How many Sev1 incidents are open for my team?’) and guides users through building a query that retrieves the answer.

In this article, we’ll cover our experience building this capability responsibly. I'll describe:

  • Key considerations when building with LLMs
  • Faros AI’s framework for evaluating LLM performance
  • How we deployed LLMs appropriately for our use case

Key Considerations When Building with LLMs

It has been said before, but is definitely worth saying again, that there are many issues with LLMs. These issues include but are not limited to:

  • Bias and problematic content from the flawed training data (the internet!!)
  • Leakage of private information
  • Generation of misinformation
  • The environmental impact of running these massive models
  • Exacerbating disparities in access to advanced technology

The first three — bias, privacy, and misinformation — are the most addressable in user-facing applications.

How can we ensure LLMs don't generate harmful, biased, or misleading content? How do we maintain privacy? These require thoughtful, responsible development.

With careful monitoring, content filtering, and transparency, risks may be mitigated but not completely eliminated. There are still many open ethical questions that need further research.

So given all these concerns, what are some appropriate use cases for LLMs?

Appropriate Use Cases for LLMs

At Faros, we incorporate LLMs to aid human understanding of data — not to fully automate or replace human judgment. Our goal is to guide and inform users without removing the steps that are best reviewed by a human.

We sought use cases where LLMs can make it easier for users to answer business-critical questions about software engineering, without needing to understand where the data lives and how it is structured.

The fact that we store the data in a standardized format enables canonical metrics and comparisons to industry benchmarks. However, there are always nuances and one-off questions that standardized metrics do not capture. The ability to query the data is critical to finding answers to questions unique to each organization.

Lighthouse AI Query Helper guides users in querying data to answer natural language questions, like “What is the build rate failure on my repo for the last month?”.

Query Helper provides:

  • Relevant related pre-built charts (maybe one is exactly what you’re looking for!)
  • Step-by-step graphical query guidance
  • Details on relevant datasets/tables
Query Helper uses GenAI to supercharge engineering leaders who are exploring their data to understand team performance

So how did we develop this tool and make sure it was working as intended?

LLM Framework and LLM Performance Evaluation

While generative language models are new on the scene, the principles of deploying AI remain the same:

  1. Understand the business problem you’re trying to solve.
  2. Decide on metrics indicative of business impact and the performance of your solution.
  3. Iterate on inputs and models until you reach a solution that works well enough to ship.

Defining good metrics and having a crisp definition of what you are solving is key to this process, but how do we define the right metrics to evaluate a multi-purpose tool like an LLM?

While there is a legion of benchmarks used to evaluate an LLM’s performance and suggest that it might be the best LLM, these don’t necessarily tell you how an LLM will perform on your specific task. For example, for our use case, how LLMs performed on the bar exam was irrelevant. What matters is its effectiveness on our task, which we need to measure and evaluate in situ.

Defining quantitative LLM performance measures

In building Lighthouse AI Query Helper, we found that the following steps helped us define quantitative measures that matched our perception of performance:

  1. We established a gold standard of example responses. We created several examples of really good answers to a given set of questions, and we expected the LLM to match this gold standard in both content and format. For example, we wrote out how to answer the question “What is the PR cycle time by team?” using the Pull Request table and the Teams table, and the specific joins, filters, and aggregations needed in the user interface.
  2. We defined performance metrics tailored to our task. Beyond just qualifying an LLM’s answer as good or bad, we sought to quantify the correctness of the LLM answer. Are the tables returned by the LLM correct and complete? Is the text in the format we have defined, with step-by-step instructions for the user interface?
  3. We iterated on prompt inputs until the metrics defined above showed our assistant was good enough to ship. How does changing the text of the prompt change our performance? Should we add descriptions of the tables or just column names? Do we need example responses in the prompt, and if so, how many?

Unfortunately, the first two steps are hard and time-consuming. And we were on a deadline!

While searching for shortcuts, it might be tempting to offload the evaluation of the LLM to — you guessed it — an LLM. However, to us, that felt a lot like feeding pigs bacon, something that never ends well. We did not offload the whole process to the LLMs and allow the LLMs to judge their brethren!

Instead, we went with a compromise, leveraging LLMs to make creating evaluation data easier, as I describe below.

Let the evaluation begin!

We started with a small set of hand-written gold examples of good questions and answers. With this data, we carefully experimented with the format of the responses and the metrics used to evaluate how close the LLM came to our examples’ format and content. We looked at every single response to make the judgment on which metrics we should use, so it was a good thing that our starting data was small.

We then stepped up this process by using existing user queries as examples of how to answer questions. An LLM served as an assistant for this step to reformat the answers from raw queries into the exact format we needed for our Query Helper. With a small amount of editing and quality control, we ended up with a substantial amount of gold data that we could use to test and evaluate different prompt and retrieval formations for our task.

The metrics we focused on during the evaluation were:

  • F1 of Rouge Response: Compares the LLM’s response to a gold standard, measuring precision and recall. This indicates how similar the response is to the ideal handwritten explanation for a given question.
  • Jaccard Similarity: Looks at the overlap between tables/fields returned versus those in gold standards. This checks how closely the content matches what we want and if it gets the right schema components.

We used these metrics and our gold data to evaluate, zero shot, n shot static examples, n shot relevant examples, and the detail and specificity in our retrieved table information.

Measures of answer quality across different prompt constructions for a) Schema Jaccard Similarity of the LLM tables and columns to the gold tables and columns b) Format Rouge F1 for LLM answer format similarity to the gold answer format

Which LLM performed best?

Not surprisingly, the content included in the prompt made a big difference in how well the LLMs performed our task.

Our key findings were:

  • Including several relevant examples similar to the question being asked improved performance. This gave the LLM more context to understand the desired response and examples of how the tables needed to be processed to answer questions.
  • Including only a limited amount of schema information was best. Dumping too much schema detail or irrelevant data into the prompt hurt performance. Retrieving and showing the LLM only the most relevant tables boosted results.
  • Including a parsing step to process the answer returned by the LLM provided an extra layer of quality assurance. This check ensures that all tables and fields suggested by the model are actually present in our schema.

We tested prompts across multiple LLMs, starting with OpenAI. However, API latency and outage issues led us to try AWS Bedrock. Surprisingly, the specific LLM mattered less than prompt engineering. Performance differences between models were minor and inconsistent in our tests. However, response latency varied greatly between models.

Comparison of LLMs (and providers) for a) quality of answer and b) latency of API response (note that a 30 second delay was added to gpt-4 calls to avoid hitting token limits)

In summary, careful prompt design considering relevancy and brevity were more important than LLM selection for our task. But latency was a key factor for user experience. In the end, we decided that anthropic-claude-instant-v1 provided the best customer experience for our use case, based on the latency of responses and quality of the answers. So that is what we shipped to customers.

Post-project, we shifted focus to real-world deployment, closely observing interactions, query resolutions, and proximity of user queries to AI proposals. This feedback loop will guide refinements and potentially in-house fine-tuned models. Stay tuned to hear how it went.

Key Takeaways

While impressive, LLMs have limitations and risks requiring careful consideration. The most responsible path forward balances pragmatic benefits and ethical cautions, not pushing generation capabilities beyond what AI can reliably deliver today.

In closing, restraint is wise with this exciting technology. Here is my advice:

  1. Avoid getting carried away with flashy demos. Take an incremental, thoughtful approach grounded in real utility.
  2. Consider whether automation imperils accuracy, and look at how you can keep a human in the loop while still improving user experience.
  3. Rigorously define goals and metrics.
  4. Don’t assume that you need the biggest newest model for your use case.

What are your thoughts on leveraging LLMs responsibly? I'm happy to discuss more. Please share any feedback!


About the author: Leah McGuire has spent the last two decades working on information representation, processing, and modeling. She started her career as a computational neuroscientist studying sensory integration and then transitioned into data science and engineering. Leah worked on developing AutoML for Salesforce Einstein and contributed to open-sourcing some of the foundational pieces of the Einstein modeling products. Throughout her career, she has focused on making it easier to learn from datasets that are expensive to generate and collect. This focus has influenced her work across many fields, including professional networking, sales and service, biotech, and engineering observability. Leah currently works at FarosAI where she develops the platform’s native AI capabilities.

Back to Blog Posts
AI Is Everywhere. Impact Isn’t.
75% of engineers use AI tools—yet most organizations see no measurable performance gains.

Read the report to uncover what’s holding teams back—and how to fix it fast.
Read the report
AI Productivity Paradox Report 2025
Discover the Engineering Productivity Handbook
How to build a high-impact program that drives real results.
‍
What to measure and why it matters.

And the 5 critical practices that turn data into impact.
Get the handbook
The cover of The Engineering Productivity Handbook on a turquoise background
Leah McGuire

Leah McGuire

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

Connect
closed
open
Want to learn more about Faros AI?

Fill out this form and an expert will reach out to schedule time to talk.

Loading calendar...
An illustration of a lighthouse in the sea

Thank you!

A Faros AI expert will reach out to schedule a time to talk.
P.S. If you don't see it within one business day, please check your spam folder.
Oops! Something went wrong while submitting the form.

More articles for you

Editor's Pick
AI
News
7
MIN READ

Translating AI-powered Developer Velocity into Business Outcomes that Matter

Discover the three systemic barriers that undermine AI coding assistant impact and learn how top-performing enterprises are overcoming them.
August 6, 2025
Editor's Pick
News
AI
DevProd
4
MIN READ

Faros AI Hubble Release: Measure, Unblock, and Accelerate AI Engineering Impact

Explore the Faros AI Hubble release, featuring GAINS™, documentation insights, and a 100x faster event processing engine, built to turn AI engineering potential into measurable outcomes.
July 31, 2025
Editor's Pick
AI
DevProd
5
MIN READ

Lab vs. Reality: What METR's Study Can’t Tell You About AI Productivity in the Wild

METR's study found AI tooling slowed developers down. We found something more consequential: Developers are completing a lot more tasks with AI, but organizations aren't delivering any faster.
July 28, 2025

See what Faros AI can do for you!

Global enterprises trust Faros AI to accelerate their engineering operations. Give us 30 minutes of your time and see it for yourself.

Request a demo