Intro

54% of companies don’t currently track the performance of AI / LLM systems.

This has been an interesting insight from The Engineering Performance Report 2025 done by LeadDev, which I closely looked into.

This tells me that companies are keen on adopting AI, but they neglect the actual performance of their systems.

Of course, in some cases, you don’t need a very detailed tracking of performance, but if you are building user-facing apps, that’s definitely very useful, as performance is a significant factor for users.

Some additional interesting insights from the report:

• 67% think performance is a high or top priority for their job

• 69% use the number of incidents as a performance metric

• 20% of organizations track AI model performance

• 69% say it’s easy to secure budget for AI initiatives

• 31% are using OpenTelemetry

In today’s article, I am going over the report and sharing my thoughts on the data.

This is an article for paid subscribers, and here is the full index:

- The survey was conducted between 19 September and 10 October 2025 and is based on 400 respondents
1. Measuring Performance
1.1 Which of the following best matches your organization’s uptime requirement?
1.2 Which of the following metrics does your team use to measure the performance of your systems?
1.3 Of the following, which 3 options would be most effective at improving the performance and reliability of your organization’s software and systems?
1.4 How easy it is to currently secure new budget for a certain initiative?
1.5 What are the top three factors that are holding back the performance and reliability of your organization’s software and systems?
🔒 2. The AI Impact on Performance
🔒 2.1 What is the status of your organization’s use of generative AI tools, agents, or models to improve software performance?
🔒 2.2 How has the introduction of generative AI tools, agents, or models helped improve the performance of your software or systems?
🔒 2.3 How are you currently monitoring the performance of AI/LLM-driven systems?
🔒 2.4 Where do you see the most potential for AI and LLMs to impact software performance?
🔒 3. Observability Tools and Practices
🔒 3.1 Which observability capabilities are you currently deploying in your organization?
🔒 3.2 Of the capabilities below, which 3 have the potential to add the most value for your day-to-day work?
🔒 3.3 Which observability tools does your organization currently use?
🔒 Last words

If you like these kinds of articles, where I go over a report that is highly relevant and share my insights on it, you’ll also like these articles:

How Engineering Teams Set Goals and Measure Performance Gregor Ojstersek · Oct 22 Read full story

The Impact of AI on Engineering Teams Gregor Ojstersek · Sep 10 Read full story

The State of Software Development in 2025 Gregor Ojstersek · Aug 3 Read full story

The State of Engineering Leadership in 2025 Gregor Ojstersek · Jun 15 Read full story

Let’s start!

The survey was conducted between 19 September and 10 October 2025 and is based on 400 respondents

Here are the full demographics:

1. Measuring Performance

Before we jump into the impact of AI on the performance of systems, it’s important that we go through how important overall performance is for engineers and companies.

1.1 Which of the following best matches your organization’s uptime requirement?

These are interesting insights → a lot of the people mentioned 99% uptime. As that means that the uptime/downtime is not as critical for the users. However, 7 hours is still quite low (even if it sounds like a lot).

This means that one night of outage is going to break the SLA (Service Level Agreement). So, you still need at least some level of observability, at least an on-call rotation to get paged if the app is down.

And then 7% mentioned 99.999% → which only allows the 26 seconds of downtime per month. That means that you need to automate the downtime happening, and just a simple on-call rotation with getting paged doesn’t work anymore.

1.2 Which of the following metrics does your team use to measure the performance of your systems?

We mentioned this in one of the previous articles as well → the number of incidents is a good metric, and also the severity. You normally want to combine those with doing incident retrospectives.

That’s important as you want to learn as much as you can from every incident, especially from those that are more critical.

And then, of course, meeting SLAs is a very important metric as that’s what you are promising to your users.

What I found interesting is the Customer experience being quite high with 45%, I am curious how companies measure that in regards to the performance of their systems.

1.3 Of the following, which 3 options would be most effective at improving the performance and reliability of your organization’s software and systems?

Tech debt is what I’m hearing the most from engineers and engineering leaders as well, so that being in the #1 spot doesn’t surprise me.

And yes, automated testing is also definitely what I’m hearing a lot, but some companies do it a lot more than others. I also know teams that use TDD regularly.

So, I didn’t expect that to be that high, especially now with AI, you can generate test cases for automated tests quite fast.

I’d move “Improve collaboration and communication” to be one of the most important. As Conway’s law is very true, and it states:

“Organizations which design systems (in the broad sense) are constrained to produce designs which are copies of the communication structures of these organizations.”

So, with that in mind. Bad collaboration and communication equals bad architecture and system stability.

That is the root cause of a lot of unstable systems, in my opinion.

1.4 How easy is it to currently secure new budget for a certain initiative?

No surprise at all here :) Everything related to AI is getting the budget approved very quickly. I am hearing that a lot as well, and it aligns quite well with many company leaders experiencing FOMO regarding AI.

But yeah, it’s also a bit alarming, as just blindly investing in AI features and tools may actually provide a counter-effect.

I’ve seen that in some cases where companies invested a huge amount of money in AI features and a whole AI transformation, but then actually ruined the overall experience for the users, because they enhanced something the users actually didn’t want.

1.5 What are the top 3 factors that are holding back the performance and reliability of your organization’s software and systems?

The interesting answers here are “Need to demonstrate engineering value to leadership”, “Shift to distributed systems”, and “Investment in AI/ML capabilities”.

To do a commentary on the first one, it’s a pity to see that doing something just for the sake of demonstrating value to the leadership → that often means time and resources wasted in the long run.

And then, the good old shifting to microservices (distributed systems), I’ve been there a few times where we did a full transformation, and yes, it’s quite a process, and I recommend companies and teams to stay with a monolith as long as possible.

I’ve seen a solid working monolith being able to support big traffic, and also many engineers working on it as well.

Also 8% mentioned that investment in AI/ML is actually holding back the performance and reliability, and it makes sense. When you invest in one thing, you focus less on the other.

Now, let’s move to how AI is impacting the performance.

2. The AI Impact on Performance...