AI has brought about an interesting new tradeoff to workplaces. On the one hand, it's opened up faster, more comprehensive ways to do tasks and fill in skill and knowledge gaps. On the other, it's opened up a plethora of new security risks, especially as the convenience means that a lot of people end up sharing sensitive data with their favorite AI assistant.
One of the biggest problems is simply being able to identify and measure AI tool use in your organization. Are users using the right tools and using them correctly? Are tokens being over (or under) used and throwing off the budget? Are certain users relying on tools much more than they should be? Without being able to monitor and measure AI use, you can't easily answer these questions.
In this article, we'll cover some of the important reasons why you should care about measuring AI usage and some solutions to monitor how it's being used by users throughout your organization.
Why should you care about measuring AI use?
There are many reasons to measure AI use, but three important ones that come up for most organizations is to stop sensitive data being leaked to AI, avoid overspending and blowing out the AI budget, and keeping the human touch where it still matters.
Stop sensitive data being leaked to AI
From a security standpoint, unmonitored AI tools are extremely risky. If you have no idea what users are putting into AI and what they're getting back, sensitive data can leak. It's even worse if the tools being sent this kind of data are free, personal, or unsanctioned accounts.
Because it's so easy to have AI do the work (and because a lot of people are unaware of what AI tools do with data and how they form responses), a lot of users knowingly or naively give AI first-party data. Not only can this put the organization in breach of many compliance standards, but it's also giving that sensitive data directly to the tool, which in some cases can be accessed by a human on the other end.
Monitoring AI tool use can help you see widespread this issue is in your organization so you can decide the best next steps on how to deal with it. It might mean talking to a few users, writing new policies, or even banning all but approved tools.
We talk about the risks of sensitive data with AI in more detail in our article Managing sensitive data in AI.
Blowing out the budget
AI is getting better, more advanced, and more accurate. But those improvements come with a price, and it's rising. With all the compute required to run and improve models, AI assistants were never going to stay free forever. And of course, these newer, better models that work faster with fewer hallucinations cost more to use than the simpler ones. They're also far more likely to exceed allocated budget.
Measuring AI use can help you get ahead of the issue before it becomes a problem. Maybe that deep research task a user has run daily doesn't need to run every day, or doesn't need that level of reasoning. Maybe the time saved from having agents do a few simple tasks cost more than simply doing them manually. By knowing how AI is being used, you can know where to make adjustments before the budget runs out.
And if the budget does run out, users who rely on AI might find it too hard to go back to the old ways, so turn to other tools, like free consumer versions that don't have the security safeguards that businesses need.
Keeping the human touch
The simplicity of having AI do the busywork greatly simplified repetitive tasks that were a bit too complicated to automate through basic scripts. Agents can even do a whole series of these tasks on their own, and can be a massive increase to efficiency. But the problem that some businesses are finding is that it's also being used for other tasks that they think should require that human touch and oversight.
Take for example, writing an important message to a client. A cautionary phrase seen in forums with regards to AI-generated content is "If you didn't care enough to write it, then why should I care enough to read it?" with some users claiming they flat-out ignore messages if they're filled with enough common AI writing tells. Some organizations are taking this to heart, with a push for purely human-generated content because of the authenticity and the learning and practice that goes along with going through the full creative process instead of skipping it. And in the case of the example, some important clients will be thankful for that extra effort too.
How to measure AI use across the organization
Measuring how AI is being used isn't as simple as checking the analytics of your sanctioned AI tools. While this can give some kind of an idea, it's much more challenging to extrapolate what the AI tools are being used for and if team members are using them safely and appropriately.
Probably the most straightforward way to measure AI use is to simply survey users by asking what tools they're using and how they're using them. You might get some surprisingly insightful answers from this, like that some users are supplementing weak points while others are saving hours per week by implementing AI agents to handle repetitive tasks.
Of course the obvious gap here is that the users have to be forthcoming about their use. And not only that, they need educated and aware about how they're using the tools. For instance, asking someone if they've "used sensitive data with AI" might get a resounding NO, but if no one's ever told them that a client's email address could be considered sensitive data in some contexts, they'd be answering truthfully... But also wrong.
Surveying teams about AI use isn't necessarily a bad idea. And if you only want a general idea of how AI is being used, it could be all you need. But it isn't comprehensive on it's own.
To fully grasp and measure AI use, you almost certainly will need to use a more technical solution. There are several different tools that can help you measure use, and the right tool for your use case will depend on your reasons for measuring use, your budget, and how complex of a solution you can handle.
Some common tools for measuring AI use are AI Discovery tools, AI engineering measurement tools, AI spend management tools, and tools for identifying sensitive data use.
AI Discovery / Shadow AI tools
Before measuring how much AI is being used by teams, you need to know about the tools themselves. And simply finding out what tools are being used isn't always straightforward. This is where AI Discovery tools come in. They'll typically sit on the network and monitor traffic and inform you when there is traffic going to known AI addresses. Traffic to somewhere like chatgpt.com likely being used for AI queries, and if your organization doesn't have a business account there, it can be useful to know if users are still visiting the site.
Depending on how thorough these tools are, they can go much further, such as reading prompts and responses sent to known AI tools. Of course the more intricate and advanced solutions tend to come with a significantly higher cost (and also more complex deployment) so they aren't always feasible for smaller teams.
AI Engineering measurement tools
This type of software is designed around AI use in engineering. While AI can undoubtedly speed up coding and allow more code to be generated, more code isn't always better. Tools In this category measure AI usage in an engineering context and can help give insights into the type of work that is happening, how time is being spent, and how performance has been impacted. Higher tier plans of the tool itself can give some insights here too, like Claude Code enterprise, where admins can have access to sessions throughout the organization.
AI spend management tools
As AI models improve, the cost to use the most advanced models with the most capabilities also tends to go up. For instance, ChatGPT 6 Astra costs 100x more than the Luna model. Without measuring token use, costs can skyrocket to unsustainable amounts, especially with the frontier models.
Dedicated AI spend tools are typically designed for large organizations working with a number of AI tools at once. They help give a consolidated view around use across these different tools, show how AI is being used, and give projected spend based on this data. These tools can also help drill down to deeper usage levels, such as seeing if an individual is consuming a lot of tokens or chooses to spend on higher models. The tools help you get a full view of how sanctioned tools are being used, the cost, and sometimes even insights into where to make changes to cut costs.
Sensitive data use with AI
While AI has helped boost productivity for many users, it's also opened a massive new area of risk. The more accurate the information you give to AI, the better the results tend to be, which has led to many users sending sensitive data to these AI tools.
Tools that measure how sensitive data is being used with AI can let you know how sensitive data is being handled in your organization. These tools often monitor prompts that go to AI and flag and alert admins about data that could be sensitive, such as PII, credit card numbers, and confidential information.
Some go a step further and can stop the data from being sent completely, or they can replace it with a cleaned, redacted version. With a more thorough solution, this can essentially allow tools that are unsanctioned from being used in a safer manner, and some security professionals choose this controlled method of use over outright banning tools.
This is one of the main advantages of Oveka.
Try Oveka to monitor, measure, and put lightweight guardrails around AI
Measuring AI tool use in your organization is important, and there are tools to help you measure what aspects are important to you.
Oveka focuses on two core measurement areas: AI discovery and sensitive data use with AI. By combining these two areas into a single tool, you can easily find which AI tools are being used throughout the organization, how they're being used, and if sensitive data is being sent to them.
From there, you have full control over what to do next. Want to block all unsanctioned tools completely? You can do that in a few clicks. Prefer to only stop sensitive data from going to specific tools? You define what that means for your teams and what data should be restricted.
Setup is fast and simple as well. Oveka is ready to go in minutes and can be deployed through multiple methods, but is completely in the cloud with no infrastructure. To see if Oveka is right for you, you can start a free trial now or setup a demo call with our team.