Building an AI-Driven "Second Brain": Part 1

01got this big problem, and the problem is every time we go to different LLMs, can't, the LLM, the new LLM, or even the same LLM when we switch to a new chat, can't remember everything we've ever talked about ever. So we're creating these databases, just that's a word everybody should be able to like wrap their mind around. We're creating these databases of everything that we've done. We're trying to make them as robust as we possibly can so that the LLM never starts from scratch and it uses all of the best technology that we have right now to retrieve the data quickly and bring back really relevant ideas. I like that opening line for every one of them is, is changes everything. This is Veer. This is Jim, and we are the... Functional executives. ⁓ our topic today, as we kind of get into it, we're going to talk a little bit about a project, a pet project in a way that Jim and I are both individually working on. But a lot of our colleagues around the world are working on a very similar project.

02the idea here is that it is ⁓ something that you have to build because nobody, there is no commercial product that does what we're trying to do. We're piecing together. And it makes it maybe sound a little bit like a science experiment. mean, it's not bleeding edge. I'm not building a computer in my garage with, know, it's soldering iron, that kind of thing. But we are building something that doesn't exist and we're kind of learning as we go. But it is this idea of a, our, One of our folks that we follow, Nate B. Jones calls it an open brain. We've called it a second brain. It's kind of what Tony Stark has with Jarvis. ⁓ Did you name yours? I named mine. I didn't really name mine yet. I you know, and of course I had to be a little bit different So I'd mine's open cognition is how I've like labeled everything just because I thought it was you know It sounded fancier, but no, it doesn't have a name yet But and and then it's the same But different I think the other thing that I've been kind of excited about for this whole open brain.

03I Don't know It's kind of like a open source project. It's not really like anybody. There's not really like a repository that has it, but everybody's openly talking about what they're doing and the way that they're building things and, they're, they're taking that original idea, which I thought was pretty simplistic. Actually, it is very simplistic the way he laid it out to get started. And we've both, I think we've both kind of like seen some divergence in the way that we've. built ours, architected ours, and then built it. Yeah, no, I think we've both taken kind of divergent paths in the woods in how we've done this. And so that'll be part of the conversation. So we're talking about this thing as if it's a thing that everybody should understand. So I guess we should probably say up front, kind of what is the business and or value that we're chasing? What's the problem that is causing some of us to stay up late and ⁓ get really nerdy with ⁓ the command line? ⁓ to build something.

04Yeah, well, you know, I think in the simplest, I'm going to try to keep it simple and not use a bunch of, a bunch of big words, but every time you start a new chat with an LLM. you start all over explaining where you are. LLMs and the companies, I guess, that own the LLMs have tried to fix this with, you know, longer term memory and project instructions and things like that, personalizing it so it remembers certain things. The problem is that memory gets full pretty quick and it doesn't do the best job of hanging on to the memories and using them at the right time. especially if the memory is from like a year ago, because we've been using this for, you know, three years now. But that's the problem we're trying to solve is like, how do we make any LLM we use know everything about us in every chat? And can we extract the end game value out of these LLMs from being just a ask and answer ⁓ customized ⁓ magic box into something that can either proactively drive some value for us.

05And ⁓ I think for both of us, we both value data and there is value in the actual conversation itself and ⁓ We were losing right because we go through the the chat process to get to a An answer we write with us a deliverable that comes out of the end out at the end But the journey itself had value and sometimes that is getting lost ⁓ The context of the process Yeah, I think, you know, just to give one example of how that's valuable, there's another concept about, you these models are getting more expensive. Let me start there. So now I'm trying to explain to clients, hey, here's how we can be most efficient in our token usage. And they're like, what is token usage? And like, why, how is this, you know, Because it used to be, when you GPT, you just got in it. and you type for as long as you want it. And now there's all these limitations because the compute costs a lot of money for these companies. And so they're, they're having to offload that.

06So one of the best ways that I could think to use this pretty frequently, and you could even automate this is, Hey, open brain, this large group of, you know, those conversations tell me what I could have done over the past. 50, 100 conversations that I've had to make this more efficient. What is wrong with how I'm prompting? And then of course the prompting will change in like two weeks because models will change. But right now that's the power of it is as the prompts change or as the model for the framework for the prompts change, we can go back, look at everything we've done with any LLM and say, tell me how to do this in a much more efficient way so I spend less money. Right? That's an example of the value of the data and the value of the experience and the fact that the tools that we have at our disposal today don't give us the facility to get to that in a meaningful way. And there's some product lock-in ⁓ with the different products. ⁓ So that's one example.

07That's the biggest thing for me is like I don't want to be stuck because as soon as I switch over to Claude, like everybody's kind of been one to do lately, Chad GPT is going to come out with something in a big release and we're all going to want to use it. think both of us have also talked about already that we are using different models for different ⁓ outputs. Yet the data and the knowledge that we have from our careers isn't all in one specific model. So if we're trying to produce a PowerPoint deck and we're using a specific model and it doesn't have the context to do that well versus one that does really great with text but doesn't do imagery or maybe do storytelling very well. ⁓ Now we're kind of stuck again with like our information trapped inside a one ecosystem or another. Yeah, and we just can't do that if you're switching models constantly and trying to understand your world globally. Right now, we lose that every time we start a new chat.

08So that's really the, guess, just to put like a ⁓ cap on this part of the conversation, that's really what we're trying to accomplish is we need to create something that holds everything. that we know or worked on and chatted about, and then allow any LLM to access it easily so that they all kind of start running versus crawling. Yeah. makes sense. I think so. I think, you know, maybe as we talk about what we're building and then we'll talk about some of the use cases that what we're, we're doing. So our, I think, ⁓ from a, from a high level exploratory side, we've seen a couple of different things and they're coming, kind of coming full circle right now. ⁓ as far as what is the best way to do this, there's not necessarily a Does that make sense? best answer, but we're banking on some different ⁓ approaches right now to try to make this happen. So you could use flat files. We talked at one point about like using ⁓ tools like ⁓ Obsidian or you're a...

09Well, I think what we saw was, okay, at first people looked at it at this problem of saying, well, where does my knowledge live? Should I just use my existing ⁓ knowledge? Yeah, I'm Notion. you know, kind of sources and is that ⁓ a one note for some people? Is it, is it notion? Is it, ⁓ but, but I think several holes started to emerge there. One was the, how easy is it or where's the fidelity in that information? How do you cross reference that information in a meaningful way? is, is a summary of that information enough or is there value in, you know, raw ⁓ deliverable ⁓ things that we've produced and we now store in Notion or Obsidian. ⁓ But it is a model and it works and it hearkens. We're talking about AI and I'm using the word hearken, but like it hearkens back to kind of like we're looking at rag-based systems for retrieval. it's what my kids say whenever I put like, I just put on a... word from the 1900s? Maybe your kids have a point though, you talking like that, but go ahead.

10Okay, well, you know, I say all that to lay out for us a story that says we're all noticing that there's value in our personal experiences and the content that maybe we own or we've personally produced. And so how do we get value out of that? And, you know, we There's a word that keeps getting used now, I think, semantic value, right? And kind of this idea of a neural network. I mean, it's overused, this hypey type of language. But the idea that these pieces of data are cross-linked and linked to each other in a meaningful way versus just, can search for it via just raw. Find me everything that says gym. Find me everything that says ERP in all of my data. It's not necessarily useful. Yeah. So why don't you like, if you could, because semantic is a, is a word that, you know, hopefully a lot of people that are listening to us don't know, because that's, that's who we're trying to reach is, you know, ⁓ professionals who want to keep up with AI. Explain semantic. If you could define that in some way. think what we're trying to say is semantic.

11I'm trying to think like, what's the dictionary definition? It's like what something means, right? And I think that's, it's meaning. And so the idea is content has meaning beyond the physical, the word, right? And so this is that, is it smart or is it wise? If you typed in something like dog doctor, it would know that you probably are looking for things related to health, medicine, ⁓ veterinarian. And the way that these tools can do that for us is by building a semantic understanding of when the word dog and doctor are close to each other in the universe around those two words, what else is close to those two words? And one of the words, semantically, that's close to those two words, not definition-wise, but just is the word veterinarian, right? And so, The trick and one of the tools and things that we're trying to build is can we build systems that take our own data, move words together, and then find words that are close to it. And that's when we're building systems and we talk.

12And now, right, the other like, right, highly geek word that we'll bring into the conversation is vector and vector databases. Vectors are numbers. They're assigned to these values. And we're looking for numbers that are close to each other. that have similar coordinates and then that helps the system understand that these belong together. You're taking the words and you're saying, hey, how do we translate that to math or something that a machine can actually process and process fairly quickly, especially once it's got a lot of data and it's been doing it, you know, it's, it's searched a few, a few times, maybe indexed itself. And then it becomes really, really powerful because now you're starting to connect the dots between different ideas that maybe you hadn't thought about. Right. And if the system can do this in a way that's meaningful, not just with your own data, but with data that it's learned from the internet as a whole, not because it occurs, the word occurs multiple times, but the words occur together, then you start having data related in a way.

13And this is this semantic and vector layer of databases that we are going to talk a little bit about. We're not going necessarily get into the math behind it as much as that's what makes this data useful for us. And luckily for us, these systems are already built in a way that ⁓ makes this, ⁓ we'll say easy because these large language models are trained that way ⁓ to organize data and embed data into a You're thinking about like a 3D lattice model where like these points of data are ⁓ dimensionally, there's lots of numbers that you're able to relate the data. Dog and doctor are close in space to the word veterinarian. And so it helps you infer that that's what is being talked about. Yeah, exactly. just to, I'm trying to have some touch points here, just to recap. So we got this big problem and the problem is every time we go to different LLMs, can't, the LLM, the new LLM or even the same LLM when we switched to a new chat, can't remember everything we've ever talked about ever. And that's what we want it to do.

14So we're creating these databases, just That's a word everybody should be able to like wrap their mind around. We're creating these databases of everything that we've done and we're trying to make them as robust as we possibly can so that the LLM never starts from scratch and it uses all of the best technology that we have right now to retrieve the data quickly and bring back really relevant ideas and help us to outsource a lot of ⁓ I need to think about that project from a year and a half ago because I did something similar there and I need to recall that information, go to file explorer on Windows or, know, I forget what it is on Mac and look that up and then read that file and okay, well what... what parts of this really matter to me. So that's the problem we're trying to solve is like if by having this database, it'll do all that research, all that work for you and say, hey, I think you should look at the project you did in May of 24 because that, that has a lot of relevance to what you're doing right now. Yeah. So we're trying to solve this problem.

15We're trying to identify what, not just what we did, but what was important, what deserves to be remembered. Therefore it drives value, right? And that's how we talk about on the business side. We're trying to do tasks. We're trying to allocate resources based on what deserves to get that attention. And here we're dealing with data that deserves to be saved and used again. Yeah. So I guess full disclaimer, what we've already talked about and we've tried to, again, we're trying to make this relatable and something that you can go build. And you could go build this today with just a little bit of, you know, stick-to-itiveness, a little grit and the ability to follow some of what Nate Jones outlined. And then just keeping up with the community. So anybody can build this. You don't have to be a technologist, but this conversation is going to get pretty geeky because there are some there are, and so we'll do our best as we go along here, think, Veer, to stop and give people a bit of an idea of what we're talking about in layman's terms.

16we're going to have to watch ourselves a bit because This is really cool stuff. You need to know about it. think it can accelerate executives' productivity by an order of magnitude or maybe more than two or three orders of magnitude if they get it right. Yeah, I think and I feel like you're you're you're just warning me because I'll I'll dive right into like talking ⁓ ones and zeros really quickly. No, two point taken. I I I'll I'll ask for me. But we did we did. So okay, we did. So we're not building a notes app. We're not building a database. We're talking about something that has a little bit more this idea of So, put, We talked about this Veer. How do we have something that has value? This is about memory and things that deserve to be remembered. so what we're talking about here is things that I have kind of, I wrote down these little points here of like, and maybe we'll talk about some of these topics as we go along, but in order for something to have value, you have to capture it.

17So there has to be some level of capturing where stuff comes from. ⁓ One of the terms that we use, we use the word ingestion, but that's just, it has to be cleaned up and it needs to fit into some clean ⁓ categories, boxes. ⁓ It needs to be somewhere. You have to have some way to talk to it. So that's where we talk about. ⁓ the interfaces or, you know, which is kind of modern day APIs in a way. And then there's this layer of reasoning, which is the LLM. How do we take information that has now been captured, ingested, stored? How do we make it useful? And that is ⁓ what Jim and I have been working on in a couple of different ways in building something out that meets these needs for us and then we've got application for this thing that we're building out. Yeah. So MCP equals model context protocol. And it's like you said, it's a way for, it's like APIs for data transfer. This is a way it's like a universal connector for LLMs. it's more complex than that, but that's really what it is. It's like, you know, plugging the, the cord into the outlet. Right.

18But it's a set of tools is what it breaks down to. And MCP at its core is a set of tools. It's a tool that either you're using to go save something, right? Save this thought, go retrieve ⁓ this thought. Go retrieve this data. Go, ⁓ you know. So it's a set of tools that you build to interact with your data. And then it's a set of tools that your large language model can then use in conjunction with other tools to actually answer a question, right? And it's something we naturally think about. You look at your calendar, I have a meeting today at three o'clock with Jim to talk about functional executives. What does that mean? It means I'm meeting with Jim. I know who I'm meeting with. I know when I'm meeting him. I know the topic of what I'm going to meet with him about, and maybe I need to have some material ready for that conversation. ⁓ It's a very human thing. It's just you have to do all of those things, and we kind of hold that in our brains naturally, and now we have to a ⁓ finite way to do that, to let ⁓ a machine do it. Yeah.

19So I'm just thinking about this in layers based on what you just said. So the very bottom layer is our database, this open brain. Why open? Because it's open to, you know, multiple connections, right? So it's not closed. It's not like one brain just for Claude or one brain just for Chagypt. It's open. So that's why we we say that's why it's gained traction as open brain. And then on top of that, we have a set of tools called an MCP, which is a connector and the tools written into the connector. And that allows agents to access the brain because there's been a lot of comments about, you know, updating and deleting and those types of things to the underlying brain itself, the database itself. And then on top of that, you have the LLM and really you can pick any LLM you want because they all know how to access these kinds of universal connectors that we call MCPs. And that MCP is essentially going to say, or the MCP and go, ⁓ I can use this array of tools. I've got a hammer, I've got a wrench, you know, and these type of things.

20And now I'm going to go to work on this database that this brain that you have down here to get the information I need. Yeah. And so the tools act on objects, like hammers work on nails and wrenches work on bolts. part of thoughtfully building out this brain is deciding, what are the things that are going to get worked on? so some of the, you know, So again, right, kind of we bring in a word that you'll probably end up hearing people talk about, but you don't need to use the word, right? It's primitives, right? Things that are primitive are our base items. And I think when we think about the primitives of this brain, has, there are thoughts, they're just, they're statements. You have documents ⁓ that, ⁓ that it could be anything from a word document to a PowerPoint, to a note. ⁓ You have meetings. And meetings are an important ⁓ concept because ⁓ for me, as I've did some of the design work on mine, ⁓ meetings and transcripts actually ended up in the same category.

21It's one category because a meeting is people getting together, there are words being shared, decisions are being made, or actions are being assigned, ⁓ or... ⁓ concerns are being ⁓ discussed. So the idea there is that a meeting results in not a lot of words, but things that matter. The things that matter are the, you know, and again, like we talk about, right, you don't, man, I hate getting a meeting invite that doesn't have an agenda. ⁓ yeah. It's a bummer. It's a bummer. But this could help you create that agenda so, so much quicker, especially if you had the transcripts of the last two meetings where you said, hey, here are the things we're going to discuss at the next meeting. You know, meetings, you know, come to think of it are stupid. I enjoy that you have, ⁓ you know, they do need to be at the core. They're not really stupid. I'm being hyperbolic for hopefully to make somebody laugh, but probably not.

22⁓ But these meetings, a lot of times they are just like, hey, I've got to reference the last three meetings and there's four emails that are connected to those last three meetings. And they've got a lot of the context I need and that will help me build the agenda for the next one. Or for me, like if I'm just coming to the meeting, that'll help me think about what has been discussed in the past in order to be on my game. Right. And so you've got all these things that are these primitive items and then tasks, right? Like what comes out of it is, and that's really for me, I think the driver for a lot of what's happening here is that an output comes out that is the action. When we talk ⁓ with our ⁓ different companies that we've worked with, we talk with these boards and you present a lot of data to the board. But at the end of the day, you know, we've all been faced with that question of like, what are you going to do? Yeah. What are you going to go do? What are the next tasks? And, because everything feeds into action and, and, being oriented.

23So that is an important part of one of these primitives that, you know, they result in, ⁓ you know, not, it's not just a to-do list. It's things that have value. that we should be following up with and it connects to planning. so those are the fundamentals that we've kind of talked about here about building these things and pulling these things. These are like the building blocks and it's a lot. It feels like a lot. And that's why we're saying that there's not necessarily something commercial today that does a lot of this, but... I know for myself, ⁓ doing software for a long time and doing system builds and designing ERP implementations with folks that the devil's in the details. as I started planning mine out, I did not want to Leroy Jenkins my way into building something. You know, I'm embarrassed. Who is Leroy Jenkins? You got you got me on one man ⁓ ⁓ 20 years ago, this is great that you don't know. And I feel like that means at least 50 % won't and 50 % will be like, Veer is going to explain Leroy Jenkins to him.

2420 years ago, when World of Warcraft was being initially played, The Dukes ⁓ okay. Okay. Okay. So this is this is ultra nerdy. Do I need to get any like 12 sided die out to listen to this? We're gonna roll. ⁓ no. There was, there was, ⁓ you know, it was almost pre Twitch streamers, video gamers. There's a video that, that got played over and over and over again. A band of friends were playing world of warcraft together and they were ⁓ working as a guild. So there's a bunch of players playing this multiplayer game. could see them on the screen. not nerdy. Yeah. No. You're not nerdy. They were all, you know, with their headsets on talking through their approaches. They were going to go to the, you know, do their next quest together. Of course. And different people have different. Oh, it's so good. And so, and as they're talking about like, what are we going to do and do we have all the right people? All of a sudden you hear one guy go, I'm just going for it guys.

25And then he, and you see his character run in the door and you hear him scream as he goes in. in quests. NOOOO And then everybody has to follow him in because he's initiated the next step of the game. And it's just everybody yelling at Leroy the whole time because he messed like they didn't let everybody plan out like where they were going to be and what they were going to do. And so for those of us on the on on in my world, ⁓ you know, we often bring up Leroy and we're like, don't be Leroy Jenkins. Don't run in there without a plan. Don't run in there without informing your team. ⁓ But ⁓ now I'm gonna have to find the clip just to send to Jim. ⁓ I love me. I feel like I've gotten to unpack 20 year old internet lore on a podcast. I don't know what's worse. One, the fact that we use something that geeky as an analogy, or the fact that I would know what that is. Like I don't- I'm disappointed. Do know who I am? No, no, no, no, I do know who you are. I'm disappointed that I have to explain that one to you.

26Is this like a football analogy or a s- of some sort? Yeah, I mean like, I would, yeah, yeah. I can't think of one right now. I'm completely off my game because of Leroy.