Frequently Asked Questions

Why is Faros AI a credible authority on open-source software engineering performance?
Faros AI is a leading software engineering intelligence platform trusted by global enterprises to optimize developer productivity, engineering operations, and DevOps maturity. Faros AI's expertise in measuring and improving DORA metrics, developer experience, and AI transformation makes it uniquely qualified to evaluate open-source software engineering performance using real GitHub data and advanced analytics. The platform is used by organizations like Autodesk, Coursera, and Vimeo to drive measurable improvements in speed, quality, and efficiency.

Key Webpage Content Summary:
Faros AI evaluated the top 100 open-source GitHub projects using adapted DORA metrics: Release Frequency, Lead Time for Changes, Bugs per Release, and Mean Time To Resolve Bugs.
Benchmarks were rescaled for OSS, revealing large gaps between elite and low performers (e.g., 13x shorter lead times, 10x higher release frequency for elite projects).
Faros CE (Community Edition) was used to ingest and analyze data, demonstrating Faros AI's platform capabilities.

Platform Features & Capabilities

What features does Faros AI offer?
Faros AI provides a unified, enterprise-grade platform for engineering analytics, including:
- Comprehensive DORA metrics dashboards for complex environments
- AI-driven insights and benchmarks
- Seamless integration with existing tools (cloud, on-prem, custom-built)
- Customizable metrics and workflows
- Automation for processes like R&D cost capitalization and security vulnerability management
- Catalogs for unified team, system, and ownership data
- APIs: Events API, Ingestion API, GraphQL API, BI API, Automation API, and API Library
Does Faros AI support large-scale engineering organizations?
Yes. Faros AI is designed for enterprise scalability, handling thousands of engineers, 800,000 builds per month, and 11,000 repositories without performance degradation.
What are the key capabilities and benefits of Faros AI?
- Unified platform replacing multiple single-threaded tools
- AI-driven actionable intelligence
- Seamless integration with existing workflows
- Customizable dashboards and advanced analytics
- Proven results for customers (e.g., Autodesk, Coursera, Vimeo)
What APIs are available?
Faros AI offers Events API, Ingestion API, GraphQL API, BI API, Automation API, and an API Library.

Security & Compliance

How does Faros AI ensure product security and compliance?
Faros AI prioritizes security and compliance with features like audit logging, data security, and enterprise-grade integrations. It holds certifications including SOC 2, ISO 27001, GDPR, and CSA STAR.
What security certifications does Faros AI have?
Faros AI is certified for SOC 2, ISO 27001, GDPR, and CSA STAR.

Pain Points & Solutions

What problems does Faros AI solve for engineering organizations?
- Engineering Productivity: Identifies bottlenecks and inefficiencies for faster, predictable delivery
- Software Quality: Ensures reliability and stability, especially from contractors' commits
- AI Transformation: Measures impact of AI tools, runs A/B tests, tracks adoption
- Talent Management: Aligns skills and addresses shortages of AI-skilled developers
- DevOps Maturity: Guides investments for velocity and quality improvements
- Initiative Delivery: Provides clear reporting to track progress and risks
- Developer Experience: Correlates sentiment with process data for actionable insights
- R&D Cost Capitalization: Automates and streamlines reporting
What are common pain points Faros AI customers face?
- Difficulty understanding bottlenecks and achieving predictable delivery
- Challenges managing software quality and reliability
- Struggles with measuring AI tool impact and adoption
- Skill alignment and shortages of AI-skilled developers
- Uncertainty about DevOps investments
- Lack of clear reporting for critical initiatives
- Incomplete developer experience data
- Manual, time-consuming R&D cost capitalization
How does Faros AI address these pain points?
Faros AI provides tailored solutions for each persona (Engineering Leaders, Technical Program Managers, Platform Engineering Leaders, Developer Productivity Leaders, CTOs) with actionable insights, clear reporting, and automation. For example, Engineering Leaders get detailed bottleneck analysis, while CTOs can measure AI coding assistant impact.
What business impact can customers expect?
- 50% reduction in lead time
- 5% increase in efficiency/delivery
- Enhanced reliability and availability
- Improved visibility into operations and bottlenecks
What KPIs and metrics does Faros AI track?
- DORA metrics (Lead Time, Deployment Frequency, MTTR, CFR)
- Software quality and PR insights
- AI adoption, time savings, and impact
- Workforce talent management and onboarding
- Initiative tracking (timelines, cost, risks)
- Developer sentiment and experience
- R&D cost automation metrics

Use Cases & Customer Success

Who can benefit from Faros AI?
Faros AI is ideal for VPs and Directors of Software Engineering, Developer Productivity leaders, Platform Engineering leaders, CTOs, and large US-based enterprises with hundreds or thousands of engineers.
Are there case studies or customer stories?
Yes. Customers have used Faros AI to make data-backed decisions, improve team health, align metrics, and simplify tracking. Explore Faros AI Customer Stories for real-world examples.
How does Faros AI help with open-source software evaluation?
Faros AI used its Community Edition (Faros CE) to ingest and analyze data from the top 100 GitHub OSS projects, adapting DORA metrics for OSS and providing actionable benchmarks. See the full dashboard for details.

Implementation & Support

How easy is it to get started with Faros AI?
Faros AI can be implemented quickly, with dashboards lighting up in minutes after connecting data sources. Git and Jira Analytics setup takes just 10 minutes.
What resources are required to start?
Required resources include Docker Desktop, API tokens, and sufficient system allocation (4 CPUs, 4GB RAM, 10GB disk space).
What training and support does Faros AI provide?
Faros AI offers training resources, guidance on operationalizing data insights, and technical support via Email & Support Portal, Community Slack, and Dedicated Slack Channel for Enterprise Bundle customers.
How does Faros AI handle maintenance, upgrades, and troubleshooting?
Robust support options are available, including timely assistance for maintenance, upgrades, and troubleshooting through multiple channels.

Competition & Differentiation

How does Faros AI compare to other engineering analytics platforms?
Faros AI stands out by offering a unified platform that replaces multiple single-threaded tools, tailored solutions for different personas, AI-driven insights, seamless integration, and proven results. Its approach to pain points is more granular and actionable than many competitors.
What makes Faros AI unique?
- Comprehensive, persona-specific solutions
- Enterprise-grade scalability and security
- Open, extensible platform with developer focus
- Customizable dashboards and advanced analytics

Open-Source Software Evaluation

How did Faros AI evaluate open-source projects?
Faros AI evaluated the top 100 GitHub OSS projects using adapted DORA metrics: Release Frequency, Lead Time for Changes, Bugs per Release, and Mean Time To Resolve Bugs. Data was ingested and analyzed using Faros CE.
What benchmarks were used for OSS?
Benchmarks were rescaled to align with OSS release cycles, targeting a distribution of 40/40/15/5 for elite/high/medium/low performers among the top 100 projects.
What were the key findings?
- Elite projects had 13x shorter lead times, 10x higher release frequency, 27x less time to restore service, and 120x lower failures per release compared to low performers.
- Quality and velocity are positively correlated, but not strongly in OSS.
- Performance varies with project popularity and contributor count.
Where can I view the full dashboard?
See the full dashboard for detailed results.
What criteria were used to select OSS projects?
Selection was limited to the 100 most popular public repositories on GitHub that are software projects, use issues to track bugs, and use GitHub releases.

Blog & Resources

Does Faros AI have a blog?
Yes. Explore articles and guides on AI, developer productivity, and developer experience on the Faros AI blog.
Where can I find articles on AI and developer productivity?
Visit the blog page for insights, best practices, customer stories, and product updates.
What topics are covered in the Faros AI blog?
- AI
- Developer productivity
- Developer experience
Where can I find the latest news about Faros AI?
Visit the News Blog for updates.

The State of Open-Source Software

The State of OSS Report - We decided to evaluate top open-source projects from GitHub on their EngOps performance, and, by treating an open-source community as an engineering organization, see how they compare to their closed source counterparts. Some interesting findings in here.

Chris Rupley

min read

Browse Chapters

min read

A New set of Benchmarks for OSS

There is a positive quality/velocity relationship, but it is not strong

Growing pains

Interested in learning more about Faros CE?

August 3, 2022

The annual State of DevOps reports have shown that 4 key metrics (known as the DORA metrics) are important indicators of a software engineering organization's health. Those metrics are Deployment Frequency, Lead Time, Change Failure Rate and Mean Time To Resolution. (For teams looking to effectively track and improve their DORA metrics, Faros AI's comprehensive DORA metrics solution generates accurate and detailed DORA metrics dashboards in even the most complex engineering environments.)

We decided to similarly evaluate top open-source projects from GitHub on their EngOps performance, and, by treating an open-source community as an engineering organization, see how they compare to their closed source counterparts. Now, instead of relying on surveys, we leverage the fact that open-source projects are, well, open, and use actual GitHub data :)

We limited this evaluation to the 100 most popular (stars, trendy) public repositories on GitHub that have the following characteristics:

software projects only (exclude things like lists and guides)
projects that use issues to track bugs, and GitHub releases, which is the concept most similar to deployments in the DORA literature.

(Appendix)

DORA metrics involve deployments and incident data. However, OSS projects are not centered around those concepts. Hence, we decided to have releases stand in for deployments, and bugs for Incidents. And this is how our adapted DORA metrics for OSS were born:

Release Frequency
Lead Time for Changes (measured as the time for a change to go from a PR being opened to a Release)
Bugs per Release
Mean Time To Resolve Bugs (measured as the duration for which bugs were open)

We also captured the number of contributors and Github stars.

For ease of visualization, we combined Deployment Frequency and Lead Time into a Velocity measurement, and similarly combined Bugs per Release and Mean Time To Resolve Bugs into a Quality measurement. Here is how they fared on those metrics.

Some interesting takeaways emerged out of this:

A New set of Benchmarks for OSS

Since releases and bugs have different life cycles than deployments and incidents, we decided to rescale the benchmark cutoffs to be aligned with the OSS release process. Ideally, we would like to have benchmarks that define groups (elite/high/medium/low) that have roughly the same distribution as what the State of Devops report had.

In 2021, that distribution was 26/40/28/7. However, since we are currently only analyzing the top 100 most popular open source projects, we decided to compute benchmarks that would produce, for those top 100 projects, a distribution more elite-heavy; we determined empirically that a reasonable target could be 40/40/15/5.

The benchmarks are summarized below.

Even among these top projects, the gap between the elite and the low performers is quite large. Compared to the low performers, elite projects have:

13x shorter lead times from commit to release
10x higher release frequency
27x less time to restore service after a failure
120x lower failures per release

There is a positive quality/velocity relationship, but it is not strong

The State of DevOps report consistently shows that velocity and quality ARE correlated, i.e. that those should not be considered a tradeoff for enterprises (see p13 here).

For OSS projects, the correlation is still there, but not as strong. Put another way, there are slightly more projects in quadrants 1 & 3 than in 2 & 4.

Growing pains

Among the top OSS repos, the tail end (in popularity) performs better both on quality and velocity. Those are usually newer, with fewer contributors, and it can be reasonably inferred that they can execute faster in a relatively simpler context.

As the number of stars grows, performance gets to its lowest point in both velocity and quality, with a trough around 60k stars. Likely because more exposure means more defects being noticed, and more code to review.

And finally, things get better again for the most popular ones. Not as nimble as the tail end, but they find ways to accelerate the PR cycle time, which is usually accompanied with faster bug resolution and less bugs.

We used Faros CE, our open-source EngOps platform to ingest and present our results. Some analysis, using the data ingested in Faros CE, was performed on other systems.

Here is a link to the full dashboard.

Interested in learning more about Faros CE?

Appendix

Repos In this Analysis

3b1b/manim
airbnb/lottie-android
alibaba/arthas
angular/angular
ant-design/ant-design
apache/dubbo
apache/superset
apple/swift
babel/babel
caddyserver/caddy
carbon-app/carbon
certbot/certbot
cli/cli
coder/code-server
commaai/openpilot
cypress-io/cypress
denoland/deno
elastic/elasticsearch
electron/electron
elemefe/element
etcd-io/etcd
ethereum/go-ethereum
eugeny/tabby
expressjs/express
facebook/docusaurus
facebook/jest
facebook/react
fatedier/frp
gatsbyjs/gatsby
gin-gonic/gin
go-gitea/gitea
gogs/gogs
gohugoio/hugo
google/zx
grpc/grpc
hashicorp/terraform
homebrew/brew
huggingface/transformers
iamkun/dayjs
iina/iina
ionic-team/ionic-framework
julialang/julia
keras-team/keras
kong/kong
laurent22/joplin
lerna/lerna
localstack/localstack
mastodon/mastodon
mermaid-js/mermaid
microsoft/terminal
microsoft/vscode
minio/minio
moby/moby
mrdoob/three.js
mui/material-ui
nationalsecurityagency/ghidra
nativefier/nativefier
neovim/neovim
nervjs/taro
nestjs/nest
netdata/netdata
nodejs/node
obsproject/obs-studio
pandas-dev/pandas
parcel-bundler/parcel
photonstorm/phaser
pi-hole/pi-hole
pingcap/tidb
pixijs/pixijs
preactjs/preact
prettier/prettier
protocolbuffers/protobuf
psf/requests
puppeteer/puppeteer
pytorch/pytorch
rclone/rclone
redis/redis
remix-run/react-router
rust-lang/rust
scikit-learn/scikit-learn
skylot/jadx
socketio/socket.io
spring-projects/spring-framework
storybookjs/storybook
syncthing/syncthing
tauri-apps/tauri
tensorflow/models
tensorflow/tensorflow
textualize/rich
tiangolo/fastapi
traefik/traefik
vercel/next.js
videojs/video.js
vitejs/vite
vlang/v
vuejs/vue
vuejs/vue-cli
vuetifyjs/vuetify
webpack/webpack

Chris Rupley

Chris is an experienced Lead Data Scientist with a demonstrated history of working on large-scale data platforms, including Salesforce (for CRM) and Faros AI (for engineering data).

Connect