Practical Application of Information Theory
Published 3/27/2024
In this episode we discuss a practical application of information theory. Information theory is similar to the classic theory of communication, but distinct in important ways; these two theories together are a useful combo for creating a strategy for clarification.
๐ Today's Episode is Brought To you by: Unblocked
Unblocked provides helpful and accurate answers developers need to get jobs done. It tailors answers by augmenting your source code with existing team knowledge in GitHub, Slack, Confluence and more. Spend less time digging for context and more time building great software. Check it out at getunblocked.com.
๐ฎ Ask a Question
If you enjoyed this episode and would like me to discuss a question that you have on the show, drop it over at: developertea.com.
๐ฎ Join the Discord
If you want to be a part of a supportive community of engineers (non-engineers welcome!) working to improve their lives and careers, join us on the Developer Tea Discord community by visiting https://developertea.com/discord today!
๐งก Leave a Review
If you're enjoying the show and want to support the content head over to iTunes and leave a review! It helps other developers discover the show and keep us focused on what matters to you.
Transcript (Generated by OpenAI Whisper)
we've recently discussed the concept of cost of delay and hopefully those conversations were useful for you and we're going to keep on discussing this uh this idea of different kinds of value represented by what you don't yet have different kinds of unrealized value and measuring unrealized value this is in many ways one of the most important things that we can do in our startups in our lives we are making decisions about how to capture value that we care about and not just value that we care about but value that is uncertain we're trying to predict some benefit of an action and today we're going to talk about this in terms of information theory in the mid-1900s claude e shannon uh wrote a a paper on this subject and we're not going to go through the paper uh directly but we can basically frame information and information theory in a way that is analogous or at least it's similar uh to the cost of delay theory and a reminder cost of delay theory is what is some unrealized benefit or some unrealized cost as of present moment that would be realized and what is some unrealized benefit or some unrealized cost as of present moment either through action by prevention of that cost or delay by realization of that cost then remembering that cost of delay does not differentiate between profit and loss by the way so this idea of information theory maps very similarly and this is related in some ways to our discussion on communication of course communication and information are tightly related conceptions but it's also related to the idea of information theory and the idea of information theory maps very similarly and this is related to the idea of information theory but the more formalized version of information theory looks at things from a slight slightly different angle more of like an economic angle and more specifically the theory of information as laid out by shannon kind of looks like this you have some holder of information some emitter if you want to call it that you have it a a receiver of of that information when it is once it is emitted and then the receiver has some level of uncertainty now depending on what kind of information you're talking about is it a closed system where there is some uh kind of bound for the information that can be known uh then the uncertainty level can approach zero in that situation we'll discuss a little bit more about the specifics and how this kind of mechanically works after we talk about today's sponsor but the information that the emitter provides and that the receiver receives can reduce the uncertainty it can reduce the uncertainty we're going to talk a little bit about this theory in more depth right after we talk about today's sponsor in this episode we're talking about the value of information and the value of information specifically reducing uncertainty that is the core uh kind of way to describe the value of of uh information and it's likely that you've experienced uh a lack of information recently and in order to get it you've probably spent hours digging through slack channels through your prs maybe reading the code directly or jira tickets uh maybe your wikis or you're trying to answer the questions there we'll talk in a little bit about misinformation what about your wikis that have gone stale how about the last time you bounced out of the ide to answer a colleague's question they're looking for the same kinds of information uh that you might have or that they might have locked in their heads maintaining a shared understanding of the code base gets harder and harder as an engineering team grows more services uh more more repos to look across longer history of changes and why did we make that change and why did we make that change and why did we make that change and why did we make that change in the first place becomes time consuming when you're onboarding new team members or working on refactoring existing projects and unblocked solves these time sinks and provides helpful and accurate answers to questions about your code base in seconds the answers are specific to your team and application because it complements source code with relevant discussions from github slack jira notion confluence and more like an extended team member who never sleeps unblocked is aware of every decision and every decision that you make and every decision that you make and every decision that you make and every decision that you make and every decision that you make and every discussion for every part of your code base and with unblocked teams will ship faster by spending less time digging for here's the key word information and dealing with interruptions check out unblocked and get unblocked.com so the basic theory of information we've already discussed the uh the holder of the information that's going to emit it in some way to a receiver uh by the way the theory of communication still lays over nicely on on this um the the sender and the receiver of a message in this case we've kind of looked at this from a slightly different angle because a message may or may not contain information right this is kind of the differentiation uh between this these two theories one of the critical differentiations is that the unblocked teams will send a message to a receiver and the the uh the message may or may not contain information meaning that if you were to look at the receiver end and measure the amount of uncertainty before and after the absorption of a given message it could be zero all right it could be a delta of zero rather but if you have information to convey and the message uh contains that information then the sender of the message uh the sender gets their message across the receiver receives that message and they uh they actually receive the information in the message then the uncertainty will go down the uncertainty will reduce by some amount now the amount by which it reduces is very important to understand uh in bounded systems the amount uh that a given message can reduce the uncertainty uh is kind of a utility of that particular message it is the bandwidth of a given message assuming that all of the content of that message was actually information all right so let's uh kind of back out of the uh combination of these two theories or models uh explaining different kinds of information transfer and instead we want to talk about kind of the mechanics of this and some important caveats associated so we've talked a little bit about uh information that reduces uncertainty we've talked about kind of no information uh these are those attempts to to convey some kind of information and the result being no reduction in uncertainty but there's a third category and hopefully you can see where this is going the idea of misinformation now i don't want this to be a loaded term because misinformation as i'm using it in my book is a term that's used a lot in my book and i'm using it in my book right now is not necessarily intentional or malicious it is instead misleading but not necessarily on purpose misinformation may come in the form of someone trying to explain what they think is wrong let's say that you are trying to trace down a bug in your software maybe another engineer has looked into it they have a hunch and they share that hunch with you and they may even share it in terms of what they think is wrong and they may even share it in terms of what they think is right but they believe rather than calling it a hunch they may say i think it's coming from you know this particular part of the code base now while this in well there is some information possibly here for example you now know what this person has been looking into right there's some uncertainty reduction there there may also be some uncertainty addition that occurs in other words it's not uh cleanly one direction or the other right you may have increased uncertainty in one area uh you may have increased uncertainty in one area uh you may have increased uncertainty in one area uh you may have increased uncertainty in one area uh you may have increased uncertainty in one area and no change in uncertainty in a third area so if you were to look at any piece of information that you receive and try to identify the areas of uncertainty that this information speaks to right um interestingly there are many areas that almost every interaction uh we encounter that information is providing us some kind of change in our levels of uncertainty let's say you have a a meeting with a report and the report uh does not want to share what they're thinking or feeling with you right as a manager you might believe that this only goes one way right that that kind of messaging that they are providing you is only increasing your uncertainty about what's going on but you've also received a decrease in another kind of uncertainty for example before the meeting you could pretty unequivocally say that you knew less uh about this person's state of mind than you do after this doesn't necessarily mean that you have an exact understanding of what's going on but the signals that this person gave had a positive amount of information available for you to learn from so why is this all useful to talk about on this show why are we uh discussing this on developer team certainly if you have a cs degree you may have had some kind of uh information theory class and it is it is definitely a foundational concept in computer science but that's not necessarily why we're talking about it on this show uh other than to kind of take advantage of the fact that many of you probably already know something about this theory to begin with instead because so much of what we do on the show is about decision making and that goes back to the idea of decision making and decision making is a very important part of the show and so when youension your evolution you may have taken the evolution of evolution evolution perspective, and purpose, all three of those things will help you enable decision-making in your careers. Information theory is kind of the fundamental material that you will use to make decisions. And so as a broad and general rule, using this theory as a basis of understanding, some of the most valuable ways you can spend your time is to decide what areas of uncertainty are important for you to focus on and how you may become more certain about those areas. This is very simple. Your gathering of information, measuring information, going and figuring out who else may have this information for you, whatever it is that you're using to reduce that uncertainty, deciding what areas of uncertainty you care about in your career, in your personal life, and then determining what ways you want to reduce that uncertainty. This is the, once again, the fundamental material that you'll use to enable decision-making. If our goal on this show is to help provide clarity, perspective, and purpose, clarity especially is, in many ways, the opposite of uncertainty. Clarity is based on the ability to find information or reduce uncertainty. Unnecessarily, this theory is broad in its application, but it can be used in very narrow circumstances as well. So I'd encourage you, as a quick kind of five-minute investment, a five-minute investment in yourself at the end of this episode. We've been doing some really good work on this. So I'd encourage you to do that. We've been doing some really good work on this. So I'd encourage you to do that. We've been doing some really good work on this. So I'd encourage you to do that. Take some longer episodes. I'm going to cut this one short to give you your five minutes. If you've gotten used to the 20-minute episodes, we're going to go a little bit shorter today. Take the five minutes, okay? Take the five minutes and identify three areas of uncertainty that you think are worth investing in. Three areas of uncertainty that you think are worth investing in. Pick one of them and then write the top three quick ideas, top of mind, ideas. These don't have to be absolutely correct, all right? And I'll tell you why in a second. But write the top three ideas that you have for reducing uncertainty in that very important area of your life, all right? And here's why they don't have to be perfectly correct. There's two reasons. The first reason, if there is sufficient uncertainty, in other words, if there's a lot of uncertainty especially, then anything that you're going to be able to do is going to be you choose to reduce that uncertainty is going to have some value. Think about it like this. If there is a lot of uncertainty and that is paired with a high value on becoming more certain, then any step you take is probably going to be more valuable than the cost of the step itself. So even if it's a suboptimal version of information, it's probably going to leave you in a better off place. So pick three things that you think are going to reduce uncertainty the most. And finally, choose one thing, one simple thing that you can do today or this week that will take you in the right direction. This seems simple and it is. It's oversimplified in this podcast because we only have 15, 20 minutes to talk about this stuff. But if you can chase this one idea down, of reducing uncertainty in important areas of your life, if you can chase this idea down, you're very likely going to have a positive result from this. All right. I want to hear more about what happens as a result of this. You may realize, hey, you know what? I've been totally neglecting this part of my career growth. I've been totally neglecting seeking feedback on my resume. That's a very common one. I have a lot of uncertainty. How good is my resume? That's probably an area that if you're listening to this podcast, you don't have a clear line of sight on your resume quality. That's an area of uncertainty that would have a high yield of reducing the uncertainty. So what are some steps, right? So think about this in this manner and make it very practical for your situation. Gathering information will reduce uncertainty. Reducing uncertainty can have some tangible benefit, probably right away, immediately in your career. Thank you so much for listening to this episode. Thank you again to today's sponsor, Unblocked. Unblocked provides development teams helpful and accurate answers to questions about their code base. It tailors answers by complementing source code with relevant discussions from GitHub, Slack, Jira, and more. And with Unblocked, teams can ship faster by spending less time digging for information and dealing with interruptions. Check it out at getunblocked.com. That's getunblocked.com. If you find out that this episode has been valuable to you, please subscribe to our YouTube channel and hit the bell icon. We'll see you next time. Thank you so much for listening. And until next time, enjoy your tea.