You've gotta read what this thing gives you. You know Yeah. The LLM gives you a ton of information, perfect, but you still have to wade through it and make sure that it's okay. I mean, I've probably got I won't tell any of them because I'm afraid to like give away who did it, but I've got at least three good personal stories about folks using the output, sending it on.

Well you mean.

And whoops, that's not what I meant. We should have reviewed that more carefully.

Feel like the that opening line for every one of them is is changes every

I went to ask my second brain a question this week and you know, while I was preparing for this and some other things and it it didn't work. The information like I went immediately and you know just kind of checked everything and nothing was gone. the connection to my open brain to Claude or, you know, Chad GPT, whatever you're using, that all works still. But it didn't give me it didn't give me anything. I first I had like the wrong credentials, so tokens, all that stuff. And then so I fixed that. And then it's been about a week now and I'm still fixing things.

Okay, so it's delivering really great value to you right now.

Parts parts of it are, but it's a lot more complicated. I mean I think the the biggest thing is is how you know, I'm seeing a little bit of a microcosm of what companies I think are gonna see if they start when they start building their own things.

Well, it's a good intro to the fact that a couple of months ago, three months ago, maybe, two months ago, we talked about building an open brain. We actually had a lot of feedback from folks on it, and it made for good conversation for me, I know in town, for you in town to talk to folks about like here's a pro Although it is a little nerdy and a little, you know, required some setup, it it was a practical use of This emergent technology. So today we're going to talk a little bit about a field report of what happened when Jim and I started building an open brain. And what have we had to duct tape together along the way? And now, kind of, you know, what are we seeing? Good, bad, ugly.

Yeah, I mean we gave everybody the pitch deck, which is always glossy and really nice. And ⁓ we should probably start there. Let's talk about what the marketing material said about open brain.

Right? We said well I mean th the problem was you have lots of information, you can't necessarily access it all i and you're having to give little pieces of context to your L L ⁓ every time.

⁓ yeah, a hundred percent. The issue was it's like how do I switch from Claude to Chat GPT to a local model, whatever it is, and still have it know everything about me, the work I'm doing, and be able to give me meaningful suggestions. That's the whole goal of an open brain. And it's a lot. It's actually not just one project, you know?

There was a data autonomy com ⁓ piece to that. I mean, it was it was solving a lot of problems. It was it was trying to address my data is my data. I should be able to use any tool I want. I should be able to access all my data. I should be able to synthesize it and come up with something net new. And I think that was the brochure. Like, do all those things.

Yeah, a hundred percent. I mean, how I would guess the best question to ask you is you built a product, you call it Cyrno, which I th I still think is a really clever you know, wording Cyreno balcony, and you've got all sorts of Cyrino related it's a theme, right? Of all these little products that you've put together. So again, you call it the brochure, I said pitch deck, but marketing materials were really flashy. How does it how are you seeing it like in action as you use it day to day, like what what are you seeing? What's the what's real life?

It's really useful. I mean, and I think the what what we talked about to people a while back was, you know, I built we all started looking at how do we use AI and how do we use these model harnesses and w you know how do how do they give us value and and you know, I started with like, can it manage my task list? Can I build something that updates my to-do list? That turned into can I give it context? Can I give it more information? And and then I like, ⁓ I'm just gonna give it everything, but you know, what do I need to build? And but I built this thing and it has been useful. And I say useful.

So let me ask you this if they took it away from you, like you're mad, or you're just kind of like, ⁓ that stinks.

No, I'd be upset. It's generated revenue for me.

How do you mean?

So over the course of the last six months, I've had opportunities to put things in front of, you know, clients and leads. And some of those things that I've put in front of them, I used this product to pull my history, you know, things that I've built in the past and and content that I've, you know, curated over the years, like like consulting framework stuff. And I used it. to get in front of my clients faster, put in put you know, put a pitch in front of them, get a proposal in front of them, close the work. And yeah, I've collected payment for services rendered with things that I built using this that got me there faster.

Yeah, so I guess, you know, so what are what do you exactly have at this point? So what is it doing for you? So like take one of those, hey, it made me revenue and see if you can kind of like make it a little more real.

All right. Well, ⁓ I have what an ex I'm trying to think like what's a good example here of of how did it make it more real? Being able to say I have a client who wants me to do a supply chain analysis for them, like a and how they're viewing their supply chain meant, you know, putting together a little bit of a pitch that says, well, this is how you know supply chain works and the and these are the different, you know. specialties in your field and and here's how I would, you know, attack your problem and and help you guys, you know, come to us, you know, f figure out a way to to to have better insight into your into your supply chain. I've done that work before. I've had I have material, but but it's you know for ⁓ it it's general stuff, right? It's stuff that Gartner or Forester had or or ⁓ I need to ha synthesize that. Normally I'd sit down with a, you know, a report and I'd, you know, take some best practices and I'd kind of, you know, pull something together on my own, but I've been teaching the system, like here's all all the different pitches I've done before, here's some here's here's my slide decks, here's things that I've put together. You do it and start with the first draft, which is normally the longest, hardest part.

Yeah, no, I agree. I think for me, if I had to bring it to life a little bit, the the one piece that I use the most is probably email functionality. So my open brain will suggest about I think I limited it to five, like what are the five most impactful emails that I could send today? So it looks at everything and then what it does is it actually sends me a summary of those in it's like CRM heartbeat is what I call it. So every morning I in my inbox I have this list of things that I might want to do. it's suggesting them. And then I also have drafts written of those sitting in my drafts folder in Outlook. So if I want, I can just click send. Now typically I don't just click send. I still don't think this thing understands my voice completely. I continue to tweak that. But that I Sincerely believe that's made me revenue because I'm not losing threads. That's what I always say. I've been using that terminology for a long time, but I hate loose ends. I hate losing a thread. I think it's done a great job of reminding me of details that I otherwise would have forgotten. Which really helps you keep up with clients and make them see that you're listening. Cause I really am listening. I will act on all those details, but only if I remember them. So it also makes me wonder how many times I've forgotten details, you know, the c and little action items across the course of my career.

Well, I think that like that's a great one, right? I mean, it's not just helping or creating content or creating something. It's it's allowing us to work at the top of our game, right? And and and not kind of get burdened with this, you know, this mental load of like, did I do all the things that I said I would do? You've got somebody who's kind of sweeping up behind you. right. Mine goes through my notes and my transcripts at night and so that I've got determinist I've got scripts that run ⁓ on my Mac and it actually synthesizes and puts a morning briefing together, right? And that's my starting point. How did I do that before? I yeah, I had my to-do list, right? I'd look in my calendar, I'd move the things that were I didn't finish over to the next day and I you know, I'd look in my notebook, but that's where I would miss things, right? I'd miss the ⁓ a week and a half ago, I talked to this, you know, I I had lunch with somebody and and and I mentioned something. and and that I was gonna follow up and and I missed it or I was gonna follow up in a week and a half. And all of that gets put into one place for me in the morning, right, as a starting point.

a starting point though. So that's how the whole thing started for both of us, I think, was, hey, can you look across everything and just tell me like what do I need to do today? Just quick email this morning. And then it ballooned.

Right. And as it ballooned, I think the thing that I think both of us as as we kind of started this conversation today, it worked all the way up until it didn't work.

A hundred percent. I mean, we thought, okay, or at least I thought, I don't want speak for you, but I thought, well, if I just dump all of this into a database like all these people online are telling me, and that's that's kind of what we're trying to be is the antithesis of what everybody else is saying, all the hype, you know, it's this easy, use this tool, and you're gonna be 10x what you were yesterday. There's a lot of maintenance involved, and you keep adding layer upon layer on layer and You think, ⁓ okay, well these L LMs are way smarter. They think a lot quicker well, I shouldn't say smarter, but they think a lot quicker. No, they're smarter. And they think a lot quicker than me too. They think a lot quicker than me. So I just said, Hey, if I dump it all here, go look at it. Figure it out. You're smart. I I don't think in reality that's the way this whole deal works.

Yeah. I think there was a this perception that you look at the open brain type project, or you just look at like one of these, like Claude or Cowork or Codex, and you're and you're just like, that big AI thing, you know, is is is one thing and it'll just work. And, you know, it's it's it's three raccoons in a trench coat just stacked on each other. Like they're just like it's like there's a Yes, there's an but there's data and then there's a there's like design and thinking and and but but it's it's kinda hidden from you until you start building stuff and then you're kind of building this you know, you keep feeding the raccoons and they're inside that trench coat and you try to pretend like they're not there. If you do pretend like they're not there and you just use the tool, th this is I think what happened to both of us recently and it's kind of strange that we kind of both hit kind of a a point where both of our tools cause We built two very different tools in a in a lot of ways. I mean, like we approached it a little bit differently and and we approached it differently than the folks who are building these tools using things like an obsidian vault or Notion to power their open brain. Yeah.

Or markdown or mark. I mean like a series of markdown files was another way to do it, all on your computer. You don't need to build a database. And then I and you we built full on SQL Postgres SQL databases and we're continuing to maintain those.

Well, and like I think we made educated choices when we started with that. Like I think we we looked at the landscape at like what do what do I get if I do it one way or the other way. We thought this way made sense for us. I think it still makes sense.

You know, to be fair, I wanna I do wanna yeah, I think I mean I think a lot more about architecture now than I ever have in my career. Cause so a lot of times it was just like, Okay, well, how do we get the job done? And I think that's how I started with this project. So I did think about all these other methods. I mean, like we were all of this was happening all at the same time, these marked the you know, the architectural decision that we've been discussing. I think I partially landed on this though because of my comfort level with SQL and SQL databases, data warehousing, and also I like to build things. And this felt like more of a fun project than just creating a bunch of flat files that I stored in a folder. I mean, that's the honest truth. Like I wanted to build. I have all these new tools. I've been saying this a lot lately. Is like they gave us all these non-deterministic tools, all these tools, and they're really messy. They don't give you the same answer twice. That's what I mean by non-deterministic. And they allowed us all of a sudden to build all these deterministic tools that we've always had. And so, but we could tailor them and customize them exactly the way we wanted. So that is really why I you know, I had a comfort with this, I thought it was cool, and I knew that I could build with this new LLM thing. That's really kind of ultimately how I made the decision. I wish I could I was telling you it was architectural. I think it's better now. I think I think a lot more about that now, but at the time that's my thought process.

Well, I mean, but that is a like like it was maybe not the the the technical architectural process, but there was a like I'm gonna solve this problem because I have this mountain of of competencies and I have, you know, th this area of comfort and and capability. I'm gonna ⁓ mix all that together, build something that works for me. And you did, like like ⁓ you it wasn't, you know, like just throwing darts ⁓ you know, and just trying to you know, just picking something just based on, you know, what it hit. I mean like you're there was some there was more than just a little bit of thought that went into it.

⁓ yeah, a hundred percent. I mean there's a lot of thought. I mean it it wasn't it wasn't just I threw something together.

Yeah. Right. And I would say that you did do architecture. You built around what you knew at the time. And I did too. And some of this is emergent. Some of it, you know, we could have spent maybe more time doing, you know, kind of load testing on these systems. But then I would say that we went with the approach of it's gonna give me value. I think it's gonna give me value. Let me go keep building it. We did. It did I bring us real, real world value. And now we're hitting some some some bumps where like there's there's some daily maintenance stuff that, you know, we talked about we said at the beginning, like, you know, well where where where has this either created

Yes.

new and additional work or where has this not worked out for us. It's just create some of the things are that yeah, you have to if you build a system, you have to update API keys or

Right. And that and that's like the first thing that I think a lot of people will run into, which is something that IT pros like you have been dealing with for so so long. As a matter of fact, I think you go to conferences where basically that's all everybody at the IT table talks about is ⁓ we had to rotate four keys today and everybody got mad and yeah. So I mean like that was the first really unsexy thing. that that I think fails. And so how do you tie that to I mean and I think as we go through each of these, so that's like the first stumbling block. Or not stumbling block. That's the first piece of maintenance that you'll probably run into. Most people run into is you either accidentally expose a key in an LLM and they're like, hey, you need to go change it, rotate it now. And I always do because I'm like security first. Plus it's my information. I don't want it out there. So there's definitely that. And then as we talk through these, I think we should try to tie it back to like if you were at a you know $500 million, $1 billion company and you're doing these things, because we're trying to draw a line between what we're doing at a small scale and how this would work at a larger business. So the first thing is you build this thing and you've got rotation problems and that or key rotation problems. Passwords, security stuff. So that will definitely play out in a larger company and and be much more critical than it is on our little brains.

Yeah. But I think the the next big thing is is you know that there's a line between does it work and does it work effectively?

⁓ yeah. Yeah. You're talking about the search. Like does the when you ask it a question, does it give you an answer that you actually can use that is useful and adds value to your day?

Right. And and I think that that that is like a little choose your own adventure thing too, because it either lands at it got you the wrong thing, or ⁓ like like it got you an answer that feels right but is wrong, or it can't answer your question. It can't find, you know, like it it just fails at at finding and and bringing back and retrieving the things that you ⁓ that you want.

Yeah, so scale that up. So we both had these problems, right? We've we're using different methods and I don't really want to get too deep into the methods that we're using because I that's yeah super nerdy. But like so we have these different ways of having the LLM and go search our data. Great. We have those. If they aren't giving us a good answer, you know, that's a problem. So scale it up to a mid market company and you've got this and it's not giving you the best answer. How do you find it? What problems does that cause at, you know, in real life when you've got a board that's going to ask you a ton of questions about it?

Well, I mean, like a piece of it is like right, like that's one of the raccoons in the in the in the trench coat, right? I mean it it it that ⁓

Probably the second one. No, actually it might be the it might be the third raccoon. By the way, it's a great visual. I love you for the way that you paint a picture sometimes 'cause I really do see these raccoons.

It's like the like Alvin and the chipmunks, except they're raccoons. They're just

Trash panda three. So it's the trash panda at the top, pretty much, giving you the answer.

He's given you the answer based on the data. The data, you know, is not inherently good or bad. You know, data is just data, right? But is it noisy data, right? Is it is it actually what you wanted to find? Does it think the spreadsheet that says ⁓ you know, product SKUs nineteen ninety-seven is this year's? Like is is it is it relevant? Should it include it in the in the r results set because it you know, there's thirty-two copies of it in your enterprise from back from nineteen ninety seven. And now the the search and the results are skewed.

So I think this is actually important. I always do this, but it really is critical. You've gotta read what this thing gives you. You really do. I mean, like people keep talking about losing jobs and and all this other stuff. It's just a new way of working and it gives you a ton of information. LLM's it. The LLM gives you a ton of information. Perfect. But you still have to wade through it and make sure that it's okay. I mean, I've probably got I won't tell any of them because I'm afraid they'll like give away who did it. But I've got at least three good personal stories about folks using the output, sending it on, and whoops. That's not what I meant. We should have reviewed that more carefully.

And that's a I mean, like the egg on your face piece when when someone does that. I mean we've we've all made a mistake. Like we've all misinterpreted something in the course of our career where, you know, when questioned, somebody looked at it and then we had to go back and, you know, ⁓ you know, my my assumption was wrong. I I did something wrong. Here, you know, I think what we're seeing is people being just a little too cavalier with it, right? And and and and r like foregoing that that you know, human review before sending it out. ⁓ but I think like it ⁓ I mean it'll get worse if folks aren't reviewing. And then second, like the search and retrieval systems, they need to get better. I I've talked to several people here in town that, you know, they're they're in industries that are highly regulated and therefore, you know, end users can't really just go download, you know, Claude on their desktops and use it rightfully so. Those companies are trying to invest in. They're trying to use like their Microsoft Shop and they're giving, you know, they're they're investing into Copilot. I've worked for a lot of companies where I go in as a as an interim. I go in for a short short amount of time and and I've got to quickly get up to speed on, you know, kind of where they're at. And ⁓ often I go somewhere and they're like, ⁓ you know, we've been here for 20 years. We have SharePoint. We've, you know, all the stuff's in there. And then you go digging, right? And and it's just you know old, unmaintained files from fifteen years ago, ten years ago, you know, people uploaded files and it's right like you know, cash forecast, final, final, final, V two final.

Yeah, you so it's funny you if it's funny you mention that because I think that is one of the human things that we keep talking about. So when everybody says, hey, you have to have your data ready, right? So if I talk to any IT professional or finance professional, and a lot of CEOs have listened to the the you know, their team and and so they're saying the same thing, we can't use it until our data's ready. This is a I think this is like you just gave the one of the best examples of what that means. It's not we're not talking about some big data warehousing process, you know, where it costs you four hundred thousand over six months, or, you know, if it's a larger company, four million over six months. We're talking about do we have like we don't need the nineteen ninety-seven data anymore, but we still have it on the server. How do we tell the LLM to ignore that? And there are so many other little nuances like that. That's what we mean by how do you get the data ready?

And that's where, right, the this kind of like blind faith, right, in the in the L L ⁓ like it'll figure it out, right? Like it w it it it'll know that that's old and it'll ignore it and and ⁓ true.

It will not. You have to be very explicit. This is the deterministic non-deterministic. You need a piece of code that tells the LLM or that pulls the relevant information for the LLM based on really hard and fast rules. Perhaps that could be one way of doing it.

That's what I well and the thing is is that those rules either need to be built into the products or the users need to like what we don't want to do is create a world where everybody has to now be very, very, very specific in every s time they interact with the system. Like that's that like that that's maybe worse than dealing with just sifting through, you know, a hundred directories on a SharePoint site and just trying to find whatever it is that somebody had asked for. But There's something here where we start teaching the systems that when I say I want this, this is what I mean. Right. And that's what I'm having to do for my system is rework it so that my like my retrieval system started getting dumb because it started having a lot of content to go back and look through. It had a lot of LLM conversations, it had like meeting. you know, teams meeting transcripts, it had stuff like that, that it the more data it had, the more it worked hard to go index all of it. And then it would try to c you know and and try to come back with an answer. It wasn't doing a smart job of saying, ⁓ you know, he's asking this question about what did he talk to Jim about? Maybe I should limit my search to only transcripts, not anything that could be construed as a con. conversation and I should limit to the last thirty days and without me having to explicitly say it.

Right. So I would say so that's like the third issue that we've that we've really run into is when we when the volume gets big, you know, it it starts to fall over just a little not fall over, but it starts to just give you muddier information or If it's not worth if you don't have the right model selected, which is what everybody's going through right now, if you don't have the right model selected, and that's I think this is one of the reasons why LLMs are why the continued improvement, that's why you see Fable and Astra and all these new products, right? is because they're dealing with the same issue that we're talking about right now. It's like, how do you give better answers as the volume of data continues to grow and grow and grow and grow? Because some of the shorter or the quicker models are designed to go find something really fast. And that's kind of the second problem we talked about, you know, like what is the search? But they're designed to go find the problem really or answer really fast and then just show it to you. What we're talking about is is when there's tons and tons more data, not just one email inbox that you're looking through, when there's a lot of data, you need something to really think. It doesn't need to like go through one or two things until it finds the best. It needs to go through everything and then kind of go back and say, All right, well, now that I know everything, what does number one look like in the context of all the things? And it has to do all that really quickly. And I think that's what these new models are really doing.

They're doing it and they're doing it well. And I think the like as we become better users, meaning we we know what good looks like better each time. Because right, there's there's there's I think a big part of like what does good look like? You don't know until you've seen bad. Right. And then you and then now you have an opinion about what good is. And so now I know. You know, because th at first I'm like, ⁓ this is wonderful, this is great, this is doing exactly what I wanted it to do. And then it stopped doing that. And now it's bad. And now it's well, why? ⁓ right. That the and now it's a question of like, is it my data? Is it the model? Is it, you know, that I'm that that I'm I'm not giving building tools that allow it to do smart retrieval? Am I tagging things properly? So I mean it keeps coming back to data. This is so useful though for you, for me, for other, you know, you know, knowledge workers and executives that I mean, I think we've invested a lot of time into it. I think we're telling people to invest time into it because it's worth the trouble.

⁓ it's entirely worth the trouble. Or at least I believe that to be true. It is really a new way of working. So we basically talked about three problems. One is is there's like the routine security, I don't know, we'll just call it infrastructure maintenance. And then there's this whole idea of how do I how do I get it to search? Even if it's a small amount of data, how do I get it to search and give me a decent answer over that regular amount of data and that's some trial and error. And then you're going to run into that again. So the third problem is is as the volume of data increases, now you have to start making some model decisions and some

So, you're

Want yeah, how long do I want this thing to think ⁓ it to to really refine things?

Well, and to bound your search, right? I mean like the like that's the piece where like do you try to do do you start with a a a a universe of a million things and search the million, or is there a way to start even narrower by just some things like you know, simple things, right? Like date or provenance of where it came from, to say, ⁓ I'm only gonna search a ⁓ a known universe of fifty thousand things and then I'm gonna you know, like the LLM will grind really hard on that, but then produce ⁓ a better answer than one that's generized over a million data points. It's generici you know, it's it's synthesizing something from fifty thousand or from five thousand.