every once in a while a few times in your life maybe you'll encounter something that changes the way you think forever and on that list for me is something i've talked about on the show many times the book thinking fast and slow in fact it was so impactful for me that i decided that i would have a standing offer that anybody who wants to read thinking fast and slow and will commit to reading it i would buy them the book and i'm struggling with whether i'm going to keep this standing offer not because uh you know i've changed my mind about the experience that i had when i read it but rather because i've been wrong about a lot of things that the book says as it turns out this book that i have been holding up the underpinning of this book is a long list of scientific studies and many of these scientific studies uh were performed by kahneman himself and many were not on that long list of studies there is a significant portion of studies that are uh have been kind of uh listed or implicated in the replication crisis and if you don't know what this is in the mid-2010s the community of psychologists academic psychologists were interested in the uh the seemingly elegant success and the amount of success that psychology broadly was having in its studies and so people started to think about the fact that started to invest time, energy, money, resources, grants, etc. into replicating these studies. There's two kinds of, you know, kind of two ways to think about replication. One is reproducing and the second is replication. Reproducing a study is taking the data that was collected and kind of walking through the reasoning and the analysis that was done on the data. So the data is trusted. But, you know, the reproducing of the kind of outcomes using that base data is carried out again. And, of course, the analysis would be to evaluate whether the reproduced analysis lines up with the original analysis. So there may be variations or perhaps there's a huge variation or maybe you find that it supports. The original. Replication, on the other hand, is recreating the study conditions kind of from the ground up. And as it turns out, a lot of the studies that were in Kahneman's book fall prey to this replication crisis. Now, this isn't incredibly surprising, believe it or not. You may be asking, well, what's wrong with this? Why did Kahneman choose all these bad studies? Well, as it turns out, the base rate, which this is a term that I learned from Kahneman, actually, the base rate of studies that fall into this list that I would consider, you know, questionable, is, is above 50%. In fact, as it turns out, if you were to randomly pick a study from a long list of studies on psychology, even peer-reviewed and, you know, well-respected studies, it is more likely that you would pick one that has a replication problem than not. And so Kahneman didn't necessarily, you know, make an individual error here. If you were to randomly sample from his studies versus the broader academic community studies, you would get about the same rate of error. And so, if, like me, you held Kahneman up on a pedestal, it's tempting, it's tempting to think that this is a huge fall from grace. That, this is commentary on whether Kahneman was, you know, a good researcher or if he was a grifter. To be clear, that is definitely not what I'm saying in this episode. And in fact, I want to convince you today that this is not the end. In fact, this is very normal, it's not that we should accept the replication crisis. Instead, I want to talk about a bigger pattern, a bigger pattern of learning and adaptation that as humans, as software engineers in our careers, day after day, we have to learn this, this skill. We have to learn this broader pattern of progress and understand how to participate in it. because I still believe that Kahneman's book, first of all, and it does have a handful of very good studies. If you're looking for a quick heuristic, again, the thing that I learned from Kahneman, if you're looking for a quick guide, there are a handful of links that talk about this. I can probably drop a couple of these in the show notes. As a general rule, the discussions that Kahneman had on System 1, those are weaker than the discussions on System 2. If you've read Kahneman's book, that's just kind of for you, all right? But I don't want to focus only on Danny Kahneman and on the replication crisis. I want to talk about the broader pattern here. The broader pattern of what do we do when we recognize that we've been wrong? And more importantly, how does that happen in the first place? How can we be so wrong in the first place? We'll talk about that right after we talk about today's sponsor, SERP API. SERP API is the web search API for your needs. If you're building an application that needs real-time search data from the web, whether that's an AI agent, an SEO tool, a price tracker, or anything else that needs to know what's happening right now, SERP API is the web search API that handles that for you. You make an API call, you get that clean JSON, they deal with all of the in-between, the proxies, the captures, the parsing, the scraping, things you don't want to think about. And trust me, I've tried doing some of that stuff. It is not worth your time. You should just rely on SERP API for this. They support dozens of search engines and platforms. They're fast. They've been doing this long enough. Companies like NVIDIA and Adobe and Shopify all rely on SERP API. There's a free tier to get started. So you can build and test your application before you commit to anything. And by the way, if you are building with AI, SERP has an official, SERP API has an official MCP. So make getting up and running a simple task that you hand off to an agent. If your app needs to search the web in real time, check out SERPAPI.com. That's S-E-R-P-A-P-I.com. Thanks again, SERP API for sponsoring today's episode of Developer Tea. So the broader pattern that we see, in the replication crisis, this is something that we should be aware of going forward in our careers because it's hitting us hard right now. I want you to imagine or think about some of the principles that you learned early in your career. Principles, broadly speaking, stay mostly the same. But that's not true forever. Principles like learning how to code will always serve you in your career. This may be true. It may still be true. But the way that it's true has changed. The things that we once believed that made sense to us that clicked, that could explain our current situation, right? The things that could explain our current situation elegantly enough that we believe it, we may encounter information in the future that makes us realize that our understanding was incomplete. Right? So the idea is, if we have an incomplete understanding, then were we wrong? Does it mean that we are fundamentally incorrect? That we didn't get it right? There's a book called The Beginning of Infinity. And it's written by a physicist, David Deutsch. I think that's how you say his name. Hopefully I'm getting that right. And one of the fundamental arguments in the book is that all progress is based on the fact that we are fundamentally incorrect. And that all progress depends on improving explanations. So this is kind of a philosophical claim. All progress that humans undertake, all of our advancement, all of our learning, starts with taking an existing explanation, maybe one doesn't exist, and creating a better one. This is the fundamental move I want to talk about today. Our explanations get progressively better. Our understanding, for example, of physics. This is some of what David talks about in the book. Our understanding of physics may have been partially right with Newtonian physics, but it wasn't the whole picture. And so Einstein comes along and provides another explanation, a better explanation, something that could explain parts that Newtonian physics couldn't explain. And so as we encounter things that we're wrong about, as we encounter replication crises, as we encounter the changing of our jobs, The things that we used to be right about deserve a better explanation. The beliefs we used to hold about what made a good engineer, now we need to rethink because those beliefs are less correct now. And by the way, when we talk about beliefs here, we're not talking about permanent truth. Our beliefs are just a calibration, a recognition of our reality. And so when I'm talking about beliefs here, what I'm really talking about is in a given circumstance, in our current circumstance, what makes for a good engineer. That is not an eternal truth. What is true now may not have been true last year. And so our explanations are evolving, but they're evolving also in response to our situation, in response to the current reality, in response to what's happening in our lives, what's happening in the industry. All of these things evolve together. And so our beliefs, therefore, if everything is changing around us, our beliefs may change as well. Additionally, even if everything was static around us, even if, you know, not, nothing was progressing, if we were stalled out, if there was very little change in the industry, right, if nothing was changing, then it's still possible that we could produce better and better explanations of reality. We could produce better and better, you know, descriptions of the right way to think about something. And so, this muscle, this mental memory is less about not trusting things. That is not the message that I want you to take away from this. Not trusting research or, you know, throwing away all science because p-hacking is possible. That's not the message I want you to take away. Instead, what I hope you take away from this is that our beliefs should remain somewhat fluid. Even our understanding of principles and values, the things that we set in our minds as immutable, even those immutable things, we should hold a little loosely because when a better explanation comes along, if you hold on to a belief because of dogma, if you hold on to a belief because it's what you've always believed because it's comfortable, because someone else you trust holds onto that, that belief, if all of those things are held tightly, and you're struggling to replace them with better explanations, then you're going to fall behind in some way. That might be in your career. It may be in your personal development. You may not even recognize the effect of that. It may have no material effect on you at all. But, this muscle memory, you could, if you were to develop the ability to think about and replace your existing belief with a better one, your existing explanation with a better one. This is, as David Deutsch explains, this is the path to progress. And progress here doesn't have to be the broad, societal progress. It's the path to the future, right? If you can update your own beliefs about the world, about your job, about your situation, about, you know, your career, if you have the ability to improve your understanding of reality, you're going to have better outcomes. Because you'll be able to calibrate to that reality better. Right? And so, the fundamental learning here is, focus on recognizing, and, when something explains more thoroughly, the reality that you're seeing. A simple example of this, because it is on everybody's minds, it's on everybody's minds, is the belief that, so let's propose kind of two explanations, two explanations of reality. One explanation of reality, is that, we, we have, adopted AI too quickly. Right? We being the industry. And, as a result of that, you know, we're going to have, you know, massive security breaches, and it's only going to produce problems. That's one belief. One explanation. Or, or, really it's a prediction. Right? So, therefore, as engineers, or as managers, we should be more wary. We should be more afraid. We should, be more reticent, to adopt AI. Another explanation, another explanation of reality, is that, AI tooling moves quickly. And, that while we will experience, a lot of negative effect, that there will be security breaches, there's also a big upside. And, that the adoption, is not pure hype, there is actually value there. Right? So, which of these explanations, would you rather adopt? Right? Which one feels more correct? And, you can go and validate these things. And, this works, both at that kind of broader, philosophical, more, future prediction, and trying to simulate the future, in your mind, to imagine which one feels more correct. Right? It kind of works there, but it also works, at this much smaller level. When you're trying to hypothesize the reason for, someone's behavior, at work. Or, when you're trying to understand, you know, a bug in your system. And, you create an explanation for that bug. And, you find that, the explanation doesn't quite come out, to cover all the cases. Right? We're looking at a bug, that is showing up, only on Mondays. Maybe there's, there's a, an issue, with a cron, that runs on Mondays. Okay. Maybe that's the case, but, the explanation doesn't, quite explain why, it doesn't happen, every Monday. So, now, we start to see, okay, we had this idea, that maybe there's a problem with the cron, but sometimes the cron runs just fine. Maybe, there's a better explanation. Right? So, you can see how this piece is together. The critical factor here is, the tendency that we have, is to hold on. My tendency may be, to reject, for example, the replication crisis information. So, that I can continue to, hold up, thinking fast and slow, as, right? That's a protective tendency that we have to avoid changing, to avoid the discontinuity of disagreeing with my former self. That's really what's going on here. Deep inside, I know there are things that I have said, even on this podcast. There are things that I've supported vehemently with my friends, family, co-workers, that are based on information that now I'm not even sure is true. I'm not even sure it's true. There is a part of my ego, there's a part of my self-identity or self-perception that recognizes that I made an error and I didn't even realize it, that I made a mistake. And that the mistake was not an egregious, obvious wrong, but a subtle one. That I got, you know, I wouldn't say tricked because nobody had malintent here, but that I believe something that I probably shouldn't have believed or that I could have gone and like figured out, maybe I could have figured out the replication crisis myself. I'm not really sure what my hypothesis is about that. Um, you know, why I would feel this protective mechanism over this, but this is the problem that we have with holding onto our beliefs, right? And it is a skill to be able to deconstruct the belief that you have to break it down into its constituent parts, to inspect them, to recognize that there's parts of this that don't totally make sense, that are not matching up with reality, that are questionable, that could be explained better by something else. That is the skill that we have to develop as engineers and as people living in a constantly changing world. So I hope you will consider, even still, consider reading Thinking Fast and Slow with this context in mind now, and you may be able to come to the book with new questions, new questions to ask about that replication. Right? Right? Right? What parts, if it wasn't this, then perhaps it's a shade of this. Maybe there's another, uh, um, you know, parallel explanation that fixes this, the problems with this one, right? But more importantly, that you can approach problems like the changing best practices in the industry, the changing targets, the moving goalposts, this is such a common, uh, thing. we see as a problem is the idea that we're moving goalposts. The truth is the goalposts are moving on their own. In most cases, the goalposts are moving on their own and we are searching for them and that's really a lot of our jobs as engineers is to find where the goalposts are and try to predict where they're going. Thank you so much for listening to this episode of Developer Tea. Hopefully this discussion on change is hitting well because I think a lot of us are trying to reckon with having to update our own beliefs, having to update our own perception of reality, our own understanding of our jobs, and there's not a lot of guidance for how to do that. Hopefully this will give you a sense that the change is actually a path to improvement, not necessarily just accepting that you are wrong, but that this is actually the work that we do is updating our beliefs over time. Thank you so much for listening. Thank you again to you. Thank you to SERP API for sponsoring today's episode. If you need a live search API, SERP API has you covered. Go and check it out at serpapi.com. That's S-E-R-P-A-P-I.com. Thank you again for listening. If you are listening on a podcasting app, please go and subscribe on whatever podcasting app you're currently using. You may be watching on YouTube. If you haven't yet subscribed on YouTube, go ahead and do that now. You can do it in both places. So that way, you're sitting at your computer, you watch a YouTube video. Here I am. And we've been investing in the YouTube platform a little bit more. And then of course, you know, if you're riding on the train or whatever, podcast is there for you to listen to as well. Thanks so much for listening. And until next time, enjoy your tea.