Rethinking AI Utilization: Beyond Vanity Metrics to Real Business Value

01I have found almost without fail, the very next sentence out of their mouth is, well, you know it's not right all the time and it flu hallucinates. Right. That's a that is a really that's a really common concern for everyone. I think that's true. It's at the bottom of every page of AI that you ever look at. You know, right at the bottom of Claude or Chad GPT, it says, Hey, you're using AI. This might be wrong. Right. So they're they're not running away from it. That's It's it's not an incorrect like like observation, but you know, me, like, you know, cause I don't know. Cause you know, there there are times where maybe I'm you know, I like to like like pick, right? So when I hear that, right, ⁓ you know, it it's wrong often, it it hallucinates. I'm like, ⁓ excellent. Give me an example. And crickets. Every time. I feel like that that opening line for every one of them is this changes every and we're back. Thank you for coming to Denver to do this in person. I know. I like it.

02And we've already had a I'd I'd say we had a pretty productive ⁓ first day. We went around town a bit, had some client meetings and ⁓ you know, talked about a lot of the things that we're probably going to touch on today. ⁓ with regard to, you know, what is AI look like out in the wild right now? I'm in person, in Denver. A little client meeting. Right. Yeah. So right. I think for our folks that are watching, right? It's welcome to Functional Executives Live, IRL even. ⁓ ⁓ yeah. I said that the other day to my son, and he thought it was the cringiest thing ever that I even knew that word. But that that's probably been around longer than he's been alive. I think so too. But he just doesn't think I know anything. He's thirteen. I I would think so. Yeah. That's ⁓ mine too. And yeah, there you go. I get I get told similar. So ⁓ yeah, so here we are in Denver. ⁓ we're hanging out, we're doing a little bit of like, you know, kind of corporate planning for ourselves and and working through some th some things.

03And ⁓ we're talking today on functional executives about ⁓ a lot of the th topics we we we touch on, but we're talking about utilization today. Yeah. And and what does utilization mean? You know, so if you use Chat GPT or ⁓ Cloud or one of these other tools, I use it for eighty percent of my work or I use it I use it for everything. I write all my emails that way. I'm I'm using it constantly. What does that what does that mean? Well, I mean, maybe you are. Maybe you're using it a lot, but right, we kind of have talked about like it's a little bit of a vanity metric. So Hundred percent of vanity metric. It's a lot like impressions in advertising, at least in my opinion. It's I mean they're directionally they tell you whether or not there's momentum or something's happening. But what I really care about, especially in my seat as a CFO, is like, okay, but what fell out of the funnel? Right. You have two million impressions, why do we only have ten leads? Right. And like I mean it's it's a metric of like well I use it a lot. Do you use it well?

04And I think that's what we're gonna touch on today is the difference between whether you use it a lot or whether you use it well. Yeah, I think one of the things I've talked about is utilization, you know, doesn't necessarily equal value. Right. Value creation. Yeah. And utilization we talked about last time is a false metric that Amazon, other companies right now are paying a price tag for because they set that as the bar. Utilization. Yeah. But I mean, so to be fair, okay. I think in some ways we all kind of did that, all of us, because well, at least, you know, in corporate environment, we did that because we needed to see that we were getting traction. It was like the first step towards adoption. The same with advertising. You have to get your face out there a ton. Right. It might not be the most effective advertising, but You can see that something is happening by the fact that you have a whole lot of impressions. Right. So there's activity. I think that's what usage really says is like, do I have activity? Yes, I have activity. Okay.

05And so that is a good bar as far as like it is, you know, right, journey of a thousand step of a thousand miles or whatever. First step. First step, you're using it. Right. Right. And and nobody expects you to be a master on day one, right? Like karate kid, you gotta wax on, wax off, paint the fence. Yeah. A hundred percent. I mean, and that is what you're that's really what you're doing is you're just learning when you're when you're focused on usage, I think it's really just, okay, am I familiar with the tool? Do I open it every day? Right. I mean, a lot of people open Excel every day and use the tool. I know I use Excel a lot as an example, but it is a good one because there's so much depth to that software. You know, you you start at What is this thing? And then you go into, okay, I understand it. Now I can write some formulas and do some things. And then you can do some automation. And then you can really kind of make it a full blown right system all within the side the software. And I think AI is a little bit synonymous. Yes.

06No, like you no one can even expect you to begin the journey of of doing well at it until you you you have some familiarity, you know where the buttons are. And you open it every day. I was actually just doing a project ⁓ that project that I was showing you. Yeah. And and one of the things that kept happening was is I had to go into another system and turn this thing on before I could ever use the system. And I kept I I didn't even want to like go any further on the project until I could fix that point where okay, all I have to do is click the icon and if when I click the icon, it pops open and it's ready to go. I didn't I knew I wouldn't use the system if I didn't open it every day. And that was blocking me from opening it every day. So whether it's mental block or, you know, an actual user interface block. Sure. It's like you can't have a block. So have we removed all the roadblocks just for you to click the little open AI or Claude icon? Okay, we can see you clicked it. We can see you did something in it. Now we're on the path. We've taken the first step.

07Okay. So you take that first step, you use it every day. You open it up every day. You've started this journey where you have utilization. And now we ask this qu I I I asked this question out at events or I'm, you know, I'm out having drinks with folks. I'm you know, I'm like, hey, this is what I'm working on anyway. Hey, you you know, you, you know, random press you know, friend, acquaintance that I'm hanging out with. Do you use AI? Yeah. Well, what do you do? And and normally I ⁓ you know, I either get somebody who is a tinkerer and will start d go down a rabbit hole of like, you know, of of you know, experimental things that they're doing, or I get a yeah, I use it from time to time. I've used it to ⁓ research something. Right, right. And that is that is pretty much how people are using it. At that level, I think, maybe some of the research, but at that level, I think you got like a really expensive, you know, kind of spell checker. Right. It's Google Plus. Yeah. I I think that's where you are there.

08So just using it doesn't really get all the value that you could get out of it. Google Plus. And by far the the folks that I talk to that fall into that camp of like, well, I kinda use it from time to time, I have found almost without fail, the very next sentence out of their mouth is, Well, you know it's not right all the time and it hallucinates. Right. That's a that is a really that's a really common concern for everyone. I think that's true. It's at the bottom of every page of AI that you ever look at. You know, right at the bottom of Claude or Chad GPT, it says, Hey, you're using AI. This might be wrong. So they're they're not running away from it. That's It's it's not an incorrect like like observation, but you know, me, like, you know, 'cause I don't know, 'cause you know there there are times where maybe I'm you know, I like to like like pick, right? So when I hear that, right, ⁓ you it it's wrong often, it it it hallucinates. I'm like, ⁓ excellent. Give me an example and crickets. Every time. Yeah.

09Yeah, it's b ⁓ they've read an article. They've heard from other people it's wrong all the time. Yeah. Maybe there's fear about losing jobs and, you know, not having the time to learn. I I don't think it's as much fear as I think it's just people have time constraints. Yeah. But that's my take. But do think that people have anchored, right? And once you know, we talk about like vr you know in sales and other things that, you know, once you can get somebody anchored to a price or to something like that, you know, you you know, that's the that that's the pivot foot. Yeah. And they've got people anchored right now to ⁓ this stuff never is right and it produces you know I they scared everybody with all the attorney stories at the beginning. Right. So utilization doesn't necessarily equate to an output that's worth anything. Right. That's a hundred percent sure. So the other thing that you hear about all the time on I don't know, YouTube or even in You know, the press, most of the press is, well, what kind of tool are you using?

10Are you using the latest and greatest? Right. Right. As we're recording this right now, right? Like I think yesterday ⁓ you know ⁓ Anthropic was finally able to release stable five and more guardrails. And and this was a s ⁓ last ⁓ you know the at the time that we're recording this, you know, this was, you know, kind of a locked away s thing for several weeks now. And right. And and so With a lot more guardrails. Working with the government back and forth. The question is for a lot of folks, well, ⁓ were were you waiting? Were you waiting for that to rel 'cause you didn't want to embark on the journey with yesterday's tool? and and I think we the the argument that we're presenting to everybody today is that ⁓ the underlying model that that you're using, the ⁓ the prompt that you're you're writing that we've talked about, right? And that we've talked about giving the tool context and managing the context window, the memory of the the the tool well, right?

11And and then we've talked about a little bit about OpenClaw and Hermes and you know and just you know whether you use codecs like me or ⁓ cowork co work like you do. Those are all call those are all basically harnesses. Right. And that so it was kinda prompts, gotta be good at those. Context gotta be really good at doing that. And use ⁓ C Ps and tools to connect to manage your context. And now it's ⁓ here's a harness. Now we're gonna now we're gonna get the AI really going for long, long runs. Right. The harnesses, right? time. And it's gonna work on something and finish, right? Like and it's gonna it's gonna just grind away at a problem. And so So let's say you have you're the best prompter, you've figured out all the tips and tricks, or you're really good at managing context, or you understand harnesses and you're using cowork or or codex. Okay. So now what? You're using it a lot. Yeah. I've it's it's really you could use cowork, codex. What have you built?

12What value have you created with the Are you we have lived in a world where right we we we've worked hard to solve problems. People bring us problems, we solve the problem. And ⁓ and the and the joke often when you have like a vendor come to your door and they want to sell you something is that, you know, they've got, you know, it's a hammer in search of a nail, right? Right, right. And and ⁓ and you know, nobody nobody's asking you whether you wanna, you know. hang up a picture or you know, like you got a whole bunch of bolts that you need to ratchet, right? But I got this great hammer. I felt I've felt that way for a long time about products, these product management or these work OS. Sure. I felt that way from the beginning. It's like, well, this company has no processes. Your hammer is to kind of codify process and make it more streamlined. But the company doesn't have any processes or controls to begin with. Right. Where should they start? Probably not with a hammer. They don't even have a nail to hit in yet. Right.

13You know, they're we're working on something totally different. So, you know, I think, you know, which which i to folks who might be listening to us, like there might be the fact that like this is reductive. Like you we are yes, like we we are saying that you need to work on real business problems, right? Solve a problem using these tools. And the what we're saying, yeah, I don't know, I don't know if it really is the quiet part or saying it out loud, but is the fact that like no one cares or should care right now about The tech stack that you're using, whether it's anthropic, whether it's Google, it'll matters some, but what are you actually trying to solve? Yeah. Doing with it. I right. You know, I feel like I feel like a lot of people use it to do a lot of their routine work, but then they're losing, they're they're skipping over the most important part, which is like sitting with it for a while. Okay, so now I got done really fast.

14I still I still see a lot of people pushing that work out as their own and and kind of taking credit for it and not sitting with it. Yeah. not understanding it, you know, if you're a quick learner, you're gonna understand quickly. You're gonna be able to, you know, talk to whatever it is. If it's something that's deep in your you've done for 25 years, you're gonna be able to speak to whatever that that topic is pretty naturally. But if it's something kind of new, so let's say you're doing an analysis on a marketing campaign. Right. And you've got to interpret the data, it's pretty easy to interpret the data right now. You run it through XYZ L L ⁓ analysis comes out and then you send it. You skipped, I feel like you skipped the most important step in there. And I I see that a lot. Yeah, the judgment, the discernment. Absolutely. Right, the judgment. Right.

15That like w was that metric that was being i is is impressions and overall number of impressions and how many, you know, leads were qualified and became qualified leads and then got and and closed as business. ⁓ is that what we I mean, ultimately we care, but do we care that it was people aged forty five to fifty that lived in these particular states from this particular socioeconomic group and like Like th that's where our marketing dollars need to go or Well, I mean and and if you get all that, and that's rich data. So I don't want to like you know, kill just the d that's rich data. But okay, first of all, let's assume in this scenario that you checked it and it's accurate. Because we just talked about a a minute ago. Yeah. Hey, is it was the AI wrong? Did it just come up with citations or did it, you know, pick something from twenty fifteen? You know? Right. And if it's from twenty fifteen, it's out. It shouldn't have made it. It wouldn't have made your criteria if you were doing a Google search and looking it up. Right.

16So ⁓ yeah, and we should talk about tra how these models are trained and like when the training data stops at some point. Yeah. ⁓ I don't want to go on that sidetrack right now, but but it okay, so if it gives you all that data, that's great. Right. But did you take the time to say, how does that apply to my business? Like what are the insights that I can pull out of this? And even if you're not gonna write those down yourselves, did you ask the next question in AI, hey, thank you for the data. I asked you for the data, you produced it. Now I need insights. I need to know in my convenience store chain, how does this data apply? I don't it forty-five to fifty years old doesn't mean anything unless you know that. Forty five to fifty year olds make one more stop on average to the gas station than most other people. And then what reason do they do that? Right.

17And that whole last thirty seconds of thinking is independent of the model, the harness, the prompt, the you know, like the like that that's the that's the judgment piece that I think is getting lost in the noise and the conversation about, you know, do I have the latest and greatest toy? What model, which is good, which one is the is is government approved, which one is safe, which one has security, which one ⁓ you know, is is ⁓ ⁓ you know, has the most market share right now. ⁓ So so if so we're not it's not just hey, use it a lot. Right. And I think you kinda came to that realization and you can you can talk to that. So That's a good one. Like like like like like this is me like maybe looking out at the viewers or that might may might watch it or saying to you, like, I'm sorry. Like I I I think I've been operating, frankly, on a little bit of a like like like a bad assumption. And and and and I think I've really realized it maybe over the last couple of weeks. ⁓ What was that assumption?

18My assumption was AI and AI tools are new and you should use them. And if you can just get a few more at bats in, if you can just get a few more reps in the gym, like that muscle is gonna get built and it's going to enlighten you to what is possible with AI and with Claude and with Codex and with Gemini. Yeah, so did was there a moment so you had a realization. Was there a moment you had the realization and then like where are you at now in the way that you think about it? So I'm I so I as I talked with folks and I started having more conversations with ⁓ clients and and prospects about the work that we're doing, and I would ask this question, right? How are you using it? And they're like, ⁓ I, you know, I have an you know, and I and I would get some you know some really good answers from people of like, I have an account, you know, I'm you know, I I I signed up a few months ago, paying twenty dollars a month. I'm like, good for you. You know, that then or you know, we're we're using cloud teams in the office. And and we're using it.

19And then I'm like, show me how you use it. And they're like, ⁓ well, you know, I ⁓ I I had it ⁓ write this email or I copied this email into it and I asked it to like make it better. You're right. And or a job description. Or code. Yeah. You know? ⁓ but then, you know, right? I mean, like I would push, right? I'm like, awesome, good ⁓ like good, you're using it, right? I got I would get excited and then I'm like, ⁓ you know, what how did you ⁓ give it Description. ⁓ you know, ⁓ the that start giving it taste and and and judgment. Did you upload your brand guidelines or did you give it ⁓ you know some type of sample document or ⁓ and and ⁓ and and I would get really quizzical looks like why would I do that? Like it gives really good answers. Or, you know, I I or they're like, well it gives me an answer and then I kind of work from it. You know, it's a good, you know, brainstorming thing. And and so and I and I'm like, ⁓ but but but you're using it often. They're like, ⁓ yeah, all the time. It's really, it's really cool. And I'm like, okay, that's exciting.

20But I think you're missing the real value add of using it. And I thought, because I thought the next part of the conversation, right, the evolution of I use it every day turns into, well, what do you do every day that you don't need to be doing anymore? Right. What is the thing that is a process within your company that you Have decided can be ⁓ replaced or not ⁓ we we we've we've we've jumped around that a little bit. It's not about replacing people, but the processes can be like replaced by overnight processes, daily processes, simple simple things that you just don't have to do every day. Now you can go do something different and you can spend more time ⁓ working with people rather than doing administrative tasks. Yeah. Or spending time even, I mean, like, you know, sometimes the person that you can work with is is you. Yeah. You can spend more time in deep thought, really, you know, challenging. I think the the the best thing is is when it's a sparring partner and you can if you can get your mind into challenging it, you know, be a little bit more skeptical.

21I might have a leg up. I think legal people probably, ⁓ you know, lawyers probably have a little bit of a leg up because they're trained skeptics. They are and they have a lot to lose. And so, you know, we don't want to be like kind of heavy handed with like everybody should use it and and you know you're dumb if you don't. It like they and I I know a lot of attorneys, Westlaw, one of the tools that a lot of attorneys use, has built a lot of AI powered features within their tool set, and attorneys are using that. Yeah. Right. And and so now they've got tools that people have built that are curated for them and and ⁓ d handle some of the risk. Yeah. ⁓ I've got some friends that are journalists. They're also not skeptical, but they're a little reticent, right? Because, you know, being called out on a a bad fact, right? Something that you reported as fact is is is is i that's, you know, that's career ending. Yeah. Absolutely. I mean, for attorneys, I mean it's it's at the it's career ending at worst and it's like significant reputational damage at best.

22Even if it's ⁓ even if the blast radius is small, like there's a few people that are gonna be like, mm, and that's out there in the ether now. So you you you said, you know, like did you like figure this out one day? I'm like over the last like these conversations have led me to be like, I think I've had I think I was just giving too much it's not credit, but I I I think I was trusting or hoping too much that exposure was enough. And ⁓ and so we talked about this the ⁓ the other night, you and I, and and and the the the thing is is like I, you know, your son, my son. I could hand them both an algebra book and I can put it in their room and I could leave that book in their room for the next year. Doesn't make them awesome at algebra. No. Right. Osmosis or or diffusion of the of the of the material. They can't just read it. No. What do they so they have to apply it. Correct. And they have to try and sometimes fail. I I tell my kids all the time, you know, if you're making mistakes, that means you're headed in the right direction. Right.

23I've told them that their entire life, you know. It is critical that you just try be safe about it, do it. You know, that's why I think there's so much wisdom in the folks that say try this on personal tasks before you Go into business trying it. But every developer out there ha has ha has to do Hello World before they, you know, attack a module for a Fortune five hundred company. It starts someplace and you gotta and you gotta build it over time. And my first like y back early in my career visual basic program, your first cash flow model, how proud are you of that? ⁓ yeah. Today? Yeah. Right? I mean like ⁓ You ⁓ the first one? No, I mean anything I did the first time in my life, I'm pretty embarrassed at. And and you know so shouldn't be. I shouldn't I shouldn't be embarrassed. ⁓ that was me learning. Right. But nobody nobody thinks that way in real life and that's why I think it's so hard. The older you get, the worse that gets too. No we should I learned Absolutely. But you know, you're building competency, right?

24That is why our kids do homework, right? To get it to to practice it, right? To get those reps in, but those are real reps that lead to a braid that then they can see whether they learned it or not. And so I think the the realization piece that applies to the AI piece is it's not just reps, but it is a little bit of curriculum.