[0:00] Last [music] year I was constrained by [0:02] token. Now I fixed that by joining [0:05] OpenAI. [0:07] Then I was [music] constrained by CPU [0:12] and now I feel my constraint is actually [0:15] attention. [0:16] You want [music] to optimize how often [0:19] do you need to be in the loop with your [0:21] agent. [0:25] Here [music] [0:33] we go. [0:33] Okay, everyone, [0:35] don't worry. This one this one's going [0:36] to be a little bit quicker. [0:38] So, I'm Peter, recently moved here. [0:42] Um [0:43] and you know when you do let's when you [0:46] do aching engineering [0:49] I feel last year up until maybe February [0:54] I was constrained by token. Now I fixed [0:57] that by moving by joining OpenAI. [1:00] [laughter] [1:01] Then I was constrained by CPU. [1:07] Um I fixed that with something crapbox I [1:10] show you later. [1:12] And now I feel my my constraint is [1:15] actually attention right. So so [1:18] all the things that you heard about like [1:22] creating loops I mean for me this was [1:24] like mostly a [ __ ] post and it kind of [1:26] exploded. But what you really want to [1:28] optimize when you work uh secure windows [1:33] what you really want to optimize when [1:34] you work is you want to optimize how [1:38] often [1:40] do you need to be in the loop with your [1:41] agent. So right you need to like we want [1:43] to expand all the things that your agent [1:46] can do from the prom to like [1:47] verification. So this is really where [1:50] you get more value out of it. [1:53] Also, by the way, I I forgot the ending [1:56] of the Pokemon thing. So, apparently [1:58] there's 46 endings that have exactly two [2:00] Pokemon. Uh I don't know which one it [2:04] was, but yeah, [2:05] it's like well code fixes everything. Um [2:11] so, I'll I'll show you like three little [2:13] things today that I think are really [2:15] cool and help our workflow. So number [2:18] one, you know, like I I built this thing [2:20] little thing called open claw and [2:24] one of the big issues we had is we're [2:27] getting a lot of PRs, like a lot of PRs. [2:32] And the problem is [2:34] some people literally make a PR by [2:36] typing into the coding agent fix and [2:39] then some number and it'll fix it and [2:42] make a PR, right? like zero effort and [2:45] then you have a PR that's like this long [2:47] like 4,000 lines and you don't really [2:49] know like how much time did that person [2:51] actually is even worth my time. Uh so [2:54] one thing I experimented and actually [2:55] works out pretty well is [2:58] I'm going to I made a small skill and [3:01] the skill does the following like after [3:03] after you create a PR be it with codex [3:06] or we're not discriminating here cloud [3:09] um it'll ask you hey do you want to [3:11] upload the transcript like a a sanitized [3:14] version of the transcript because [3:16] and and then we tell you if you do like [3:19] the chances that someone looks at your [3:22] PR are much higher. [3:23] So now actually we're getting a lot of [3:25] PRs look at this that like include [3:31] how people worked on this [3:35] and I find it not just me me and also my [3:38] agent find it incredibly useful because [3:40] now I actually have a more data point. [3:41] It's like did you spend one minute [3:43] prompting this? You spend one hour [3:44] prompting this? And like the longer your [3:46] transcript is, the higher is my [3:48] confidence that you actually care about [3:50] what you fix or you actually understand [3:52] or at least try to understand what you [3:54] fix. Um, [3:57] and [4:00] this took me way too long because like I [4:01] was thinking too complicated. Oh, do I [4:03] need hooks? How do I build this? But [4:05] actually it's just a skill. [4:07] So I wrote this little agent transcript [4:10] skill that just basically explains the [4:12] agent how to find the transcript. I made [4:15] it so it works with every coding agent [4:16] because ultimately it's just JSON. I [4:18] load the JSON. It'll like try its best [4:20] to sanitize it. I mean [4:24] ultimately trust the same agent. So it [4:25] probably not be perfect, but you [4:26] probably don't like don't write your [4:29] your most private secrets into the [4:31] coding session. Uh so usually we'll be [4:34] fine [4:36] and and that's it. And because I'm the [4:38] benevolent detector for life, I just [4:40] made this the a default skill. So when [4:42] anyone who who checks out the repository [4:45] automatically [4:47] gets the skill and the agent knows how [4:50] to load the skill and the and the [4:51] description says load the skill when you [4:53] make a PR or an issue. Um and bam like [4:57] our quality of PRs and issues for some [5:00] people who do it went massively up. Big [5:01] win easy to add. [5:04] Now the second thing I want to show you [5:07] is [5:09] auto review. [5:11] Um, this my favorite skill of all time. [5:14] Also, it made everything way slower. [5:16] It's kind of like a continuation of what [5:18] George started. You know, whenever you [5:21] you create code, it the agent will [5:25] probably do its best and like get a [5:27] version that's good and then you would [5:29] do a PR and then like some other agent [5:31] will spin up and do a review, [5:34] but that's slow and that's also like [5:39] not really very practical. Um, [5:44] and again it took me way too long [5:45] because I was thinking too complicated. [5:47] But really this skill again is just a [5:49] default skill and [5:52] the agent gets told load this when you [5:54] do a PR or a commit [5:57] and then it will just invoke the CLI and [6:00] by default it just look what's what you [6:02] have in your system. Maybe maybe Codex [6:04] maybe cloud maybe pi maybe is anyone [6:07] using open code? But it supports open [6:09] code too. Um, [6:13] and I'll just invoke invoke the skill [6:15] and then invoke the CLI. It's like here [6:17] review this. It'll get feedback. The [6:20] beauty is [6:22] feedback goes to the coding session [6:24] where you work because that one has a [6:27] lot more context because what I [6:28] sometimes saw is somebody would create a [6:31] PR I would load it into my codex code do [6:35] a review find three things I fix those [6:37] things but I actually haven't fully [6:39] understood what the PR is about and all [6:41] the constraints and I actually break [6:42] things because [6:46] like from a naive standpoint yeah this [6:47] might be issues but like if you deeply [6:49] understand it. Maybe that's actually a [6:51] design decision. Right? So the beauty of [6:53] this skill is it invokes review. It gets [6:57] the review back to the original session [6:59] and then the session can decide yeah [7:01] this is right but actually no. And the [7:05] beauty of this again is if you do that [7:06] then I addit instructions that it will [7:09] actually change the PR description to [7:11] include the decisions. [7:14] So then the next person who actually [7:15] looks at my PR doesn't break it again [7:17] because the knife review would find [7:19] something that you actually don't want. [7:21] Uh very easy to add again because I'm [7:23] the dictator. I just made it a default. [7:26] I think I made like both entropic and [7:28] OPI millions with just this because it's [7:30] like the massive token burner. [7:33] Um [7:34] but you know like they have to like get [7:35] their money back at some point. [7:37] [laughter] [7:39] And uh so the third thing, the thing [7:42] that I love the most is [7:45] you know if you like if you have this [7:49] long running sessions you're going to [7:51] you don't want to stare at the code. You [7:52] do another session another session [7:54] another session and you're like you have [7:55] like 10 coding sessions at some point [7:57] you're it feels like your your your [7:59] MacBook starts to become like will take [8:02] off because the CPUs are spinning as [8:04] crazy and everything become really slow. [8:06] Um, [8:11] that was seriously slowing me down. Like [8:13] at some point I had, you see, like I [8:15] have multiple machines here. This is [8:17] like my computer. This is my Mac in the [8:19] cloud. This is my Mac Studio. And then [8:20] it was like had multiple Macs where I [8:22] was like working. But that's also kind [8:24] of annoying and it can't possibly be the [8:26] perfect solution. That felt like a crux. [8:29] So [8:31] because you keep the crustation seams [8:32] double crap box. [8:35] Um and I actually got this idea because [8:37] I used uh blacksmith had this thing [8:40] where they created test boxes but they [8:43] had like daily daily out so I got [8:45] annoyed and I wanted a fallback and then [8:48] once I had this like the affordances [8:50] kept coming and what it does really is [8:52] it gives your agent a little box where [8:55] it can arync [8:57] uh your changes of the repo like it can [8:59] actually sync it starts with the the [9:01] same state that you already have in your [9:02] GitHub actions. So by default, if you [9:06] have CI, which I assume you do, it'll [9:08] just work and it'll arc the changes and [9:11] then run the tests in that box. So now [9:14] no more CPU spinning because all [9:16] everything that's expensive, it just [9:18] does in that box. And by now it feels [9:21] like it's his own Pokemon because I [9:23] support like 30 providers. Um, including [9:27] really weird things if you work at [9:28] company with high security like OpenI, [9:32] uh, or even parallels. So you will find [9:35] one provider that you like [9:38] but then like once I had this I was like [9:40] I was getting ideas and I was like in [9:43] initially it was just Linux and I'm like [9:45] ah I have this spark but it's like a [9:47] launch debug. It's only for Mac OS. And [9:50] this these bugs are like especially [9:51] nasty because if I test it on my system, [9:54] my system probably is already in some [9:57] way configured and I have like something [9:59] so my agent will probably break [10:01] something or maybe maybe passes because [10:04] I changed something in the past. What [10:06] you really want to do is to increase [10:08] your confidence that what you actually [10:09] prompt is something that works. You want [10:12] to start with a fresh machine. [10:15] So I just added Windows and Linux [10:16] support. And now my agent has this new [10:20] primitive where if I tell them [10:24] oops [10:25] if I tell him hey install openclaw for [10:29] me on what OS is this? This is Linux [10:34] and [10:36] Mac OS [10:38] and Windows. It can just do that. [10:44] And then I I gave my agent a little bit [10:46] of eyes and ears. Uh not ears, just [10:48] eyes. Um so it can it can get [10:52] screenshots. It can click into it. So [10:54] have like cheapsman computer use. So it [10:58] can actually set up your system just as [11:00] you want. And a agents by now are really [11:02] good because we train them on it. So you [11:05] can just tell them, hey, set up a system [11:06] where you like open open a web browser, [11:09] you preconfigure this and this and it'll [11:12] it'll work and like you can verify [11:13] because you can see the screens and it [11:15] might take a while but then at some [11:16] point you have this thing where you can [11:19] exactly on a fresh system verify [11:22] uh your setup and it's it's much nicer [11:25] because if you if you would use [11:27] classical computer use at some point you [11:29] would fight with your mouse cursor or [11:31] like the focus would change or like you [11:33] know it's or like even browser use it [11:34] would get annoying and because that one [11:36] uses a fresh box and then you just like [11:38] VNC into it um you don't have any of [11:42] that pain. So like let's see if I can [11:45] work this like here I have like one [11:46] thing where [11:48] I was like this looks too boring. Make [11:51] it super gay and rainbows. [11:55] [laughter] [11:57] I should have done this before. It'll [11:58] probably take too long. Um [12:02] or even break. Let's see. Let's see if [12:04] it manages. The the other the other [12:06] thing that that is cool on that is um [12:10] those links are sharable. So if I create [12:13] an environment where like hey I'm going [12:15] to testing this new feature. [12:17] I can just send this to a friend. I mean [12:20] we're using like GitHub or as as the odd [12:22] layer, but you can like use any odd [12:23] layer and then they can click around and [12:25] try it and it's instant, right? [12:29] Um, so like I have like right now it [12:31] says you control, but I can take this [12:34] link. Go on my other computer. [12:42] Oh yeah, I also added dark and white [12:44] mode on Linux. I have no clue how it [12:45] works, but I just told Codics to do it. [12:47] And now changes everything to light and [12:49] dark mode because it's important when [12:50] you work all day. Um, and then like [12:54] somebody else in your or could take [12:55] control and try the new thing. [13:00] And it doesn't really matter if it's [13:01] slow or fast like it [13:04] it'll get the job done. Like you can [13:05] imagine hypothetically spoken that I [13:09] work at some company where we have a [13:11] feedback channel and like people say [13:15] wouldn't this feature be cool and then [13:17] like hypothetical spoken we have like a [13:20] a bot in that where the bot would say [13:22] yeah let me implement this and then it [13:24] would reply with a link to to a VNC [13:27] session where someone who doesn't know [13:29] how to code can try out the new feature [13:32] and then realize it's a very dumb idea [13:34] uh or [laughter] [13:37] but ultimately that's what you want, [13:38] right? you want to you want to extend [13:40] the loop so so [13:42] any input can actually be verified and [13:44] like and then some people realize oh [13:46] yeah that actually was a dumb idea or [13:48] some people are like yeah no do it like [13:49] this and then it was a great idea and [13:52] then I only need to look through like [13:54] maybe a third of the feedback things to [13:56] figure out what's actually interesting [13:58] because it already has been verified uh [14:00] and again just by people working in [14:03] Slack using [14:05] clause [14:08] Uh, so where's my rainbow? Let's see. I [14:12] guess it's still working. [14:15] I should have I should have just went [14:17] with like frock green. [14:20] But yeah, those are pretty much it. [14:21] Those are the three things that [14:24] changed my coding over the last few [14:25] months the most. We have [14:29] transcripts, which also is really great [14:31] at companies. We have auto review. [14:35] Um sometimes it can a little go a little [14:37] bit overboard. So you want to make sure [14:39] that in your agents [clears throat] [14:41] indeed you define some some invariance [14:43] like no these files will not change [14:47] after the app started or like we can [14:48] assume things you know like coding [14:50] agents always like write for the worst [14:52] case. And number three [14:56] crap box. Thanks for my TED talk. [14:59] [applause] [15:06] Yeah. [15:07] The the first time we met, you told me [15:08] about your first aha moment with open [15:11] call where you sent it a voice note and [15:13] it didn't have like it found an API key [15:15] on your computer and sent you back voice [15:17] or something along those lines. [15:19] Scary. [15:20] And that was probably one of your like [15:22] aha moment. Have you had anything [15:24] recently that had just blown your mind [15:26] where something really surprised you? [15:30] Yeah, but I can't tell you. [laughter] [15:34] Was it five? [15:36] Uh, [15:40] fumble five. No. [laughter] [15:42] Oh, we're getting the rainbow. Look. [15:44] Look. [15:45] Yeah, that's matching for Pride. [15:50] So, are you using loops? [15:53] Um, [15:54] I mean, honestly, like I think loops is [15:56] just a very fancy word for workflow. So [15:59] if you create an issue now in one of my [16:00] open source repositories, an agent will [16:02] spin up and like verify if this is a [16:06] great idea or a very dumb idea based on [16:08] vision.MD where I just kind of define [16:11] like where I think the project should [16:12] go. If the agents think it's a great [16:14] idea, it might still be a dumb idea, but [16:16] it will it will like create a PR. [16:18] Another agent will spin up. Is that [16:20] already a loop? It's like basically it's [16:22] loops and loops. Agents already loops, [16:24] but another agent will spin up like [16:25] we'll will review the PR. will like [16:27] repair the PR, make sure everything is [16:28] great. So when I then actually have my [16:31] little time and look at the look at the [16:33] open source, I can see this this this [16:34] and everything's ready to be merge. I [16:36] don't have to to look 20 minutes at like [16:40] a stream of coding agents doing their [16:43] thing. So yeah, [16:47] when you're designing these feedback [16:48] loops, how much uh human in the loop do [16:50] you do now versus when you first [16:52] started? [16:55] I mean, in the beginning, I was like [16:57] kicking and screaming because the model [16:59] wouldn't do what I want. When I started [17:00] like a year ago, I know the the times [17:03] where I was screaming at Claude. Uh, [17:07] and now [17:11] now it's more syncing up front, [17:14] but then it might take one hour, 10 [17:17] hours to implement. And then it's [17:19] basically a get a look at for example [17:23] What you can very easily implement when [17:25] you have crapbox, you can create prompts [17:27] that make videos. So there it's just one [17:30] feature because videos don't really work [17:32] on GitHub. We'll make a GIF GIF, I don't [17:34] know. Uh and then you basically have the [17:36] before and after state. And from the [17:40] prompt, I have my visual verification. [17:42] It's like, yeah, I see the the bug here. [17:43] I see the fixed here. And I can press [17:45] merge. So yeah, much less. [17:50] At the beginning, you talked about [17:51] attention being a problem. Now, have you [17:53] solved that? [17:58] I can't kill them all, you know. [18:00] [laughter] [18:01] No. Uh, [18:04] well, to a degree. To a degree. Like, I [18:07] I my agents need much less babysitting [18:09] because you just give them more tools to [18:12] do the work until at some point, but you [18:14] still need the thinking of like, is this [18:17] actually something we want? Nobody can [18:19] take away. [18:24] Do you caught anyone faking a transcript [18:26] to make themselves sound smarter? [18:30] I'm honestly honestly like you know we [18:33] had this review bot a grab dial for a [18:35] while that gives the the PR rating [18:38] rating from one to five and we found [18:41] like more than one instance where it [18:43] would give it a two and then people just [18:45] manually edited the description to give [18:46] it a five and say this is a great PR [18:48] merge. So I would not I would not be [18:50] surprised. [18:52] What bothers you the most about your [18:54] current workflow right now? [19:05] that it still requires so much syncing. [19:09] Same [19:09] like like we produce much more code and [19:13] we build much more complex system which [19:15] just means that [19:18] it requires a lot more thinking because [19:20] like Asians are just not really good at [19:22] understanding how does this one thing [19:24] fit into the big picture. they focus on [19:26] the one thing you you know they're like [19:28] this like horses and you kind of push [19:30] them to like look around [19:33] but if you don't keep the system like [19:35] how does everything fit together does it [19:37] actually feel right yeah agents can't do [19:41] that yet [19:44] all right [19:45] sweet thank you [19:46] thank [applause] you