Hey, everyone, and welcome to Developer Tea. My name is Jonathan Cottrell, and in today's episode, I continue my interview with Sam Lambert. Sam is the Director of Systems at GitHub. In today's episode, we start out by talking about Hubot. If you missed out on the first part of the interview, make sure you go back and listen to it by going to spec.fm. Of course, every episode from the past of Developer Tea can be found at spec.fm. Make sure you go to spec.fm. That's where you find the show notes for the episodes, and there are other podcasts that you should check out there as well. So spec.fm, that's where you'll find the previous episode, the first part of the interview with Sam. So I've asked you guys to do this for a few interviews in a row now. I'm going to ask you to go onto Twitter and thank Sam Lambert for coming on. The podcast. His Twitter name is ISamLambert. That's I-S-A-M-L-A-M-B-E-R-T. Of course, that will be in the show notes as well. Just go and thank Sam for being on the show while you are listening today. He will appreciate the thank you, and I will as well. I'm very excited to welcome a brand new sponsor today. For today's episode, the sponsor is Rollbar. With Rollbar, you can put errors. In their place. In your application, you will detect, diagnose, and defeat errors. You get the context, the insights, and the control you need to find and fix bugs faster. We will talk more about Rollbar and what Rollbar provides to developer T listeners later on in today's episode. But as always, first, we're going to jump straight into the second part of the interview with Sam Lambert. Hubot. You've called Hubot the hardest working GitHubber in the past. And I'd love for you to just give kind of the elevator pitch for why Hubot deserves the biggest raise from anybody in the company. Hubot's commit history. Hubot does a lot of work. So, for people who don't know, Hubot is our chatbot. And also, a central focal point of our culture. So, you can do anything in chat via Hubot. But you haven't finished shipping something until there's that way of controlling it via Hubot. So, the concept of chatops is fascinating to me. And it's so interesting because it brings the place you work, aka chatroom, to life and brings everyone in there with you. So, it's like having all your teammates in your terminal window. So, you're controlling infrastructure. You're controlling services and processes. And people are watching you and seeing it happen. And that just removes, like, so much communication overhead. If you call up a graph into chat and you start looking at something. I've seen it. It's probably happened today already. Someone has pulled a graph into chat and discussed it with someone else. And you just see it happening in line. And you can just go in and track that conversation happening. And not only are you kind of involved, you're watching people do their work. And that's where it happens. And it's just fascinating. I just... There's something truly amazing about it when you see it done right. It's just... It's wonderful. Yeah. So, Hubot allows for people to essentially issue commands in a chat window. They, you know, mention Hubot or whatever. And you guys... What chat system... You guys use an internal chat system? Or what do you use? We're using Slack for chat at GitHub. We previously had a sort of a half-done system that we built ourselves. You know, we... We were running our own client, but now we've moved over into Slack with Hubot there. Well, and the reality is you can use pretty much any chat system with Hubot. And I guess it's worth mentioning that right now, developers, you can go and look at Hubot. It's obviously... It's open source, right? You can go and, you know, clone Hubot into your company and use it for the same things that GitHub... Or same types of things that GitHub uses it for. Yeah, absolutely. And there's loads of open source scripts and plugins. You know, there's lots of companies that are contributing. And there's lots of companies that are contributing to the Hubot ecosystem. We do as well. And we build a lot of stuff internally. It's really easy just to get going and start adding things. And companies like Heroku have added ways of being able to deploy from Hubot. You know, it's fantastic. The range of things you can do is huge. And we just have fun things to very serious things. You know, you can command our entire fleet of servers from Hubot while everyone watches and collaborates. And we have a lot of great things with you doing things. You can deploy servers from Hubot. So I was doing it recently. You essentially tell Hubot you want X number of hosts of this class. And they will come back within, you know, 10, 15 minutes. They'll be ready and available all from your chat window. It's the one common interface I think that most teams around the company use is Hubot. You're right. I think chat ops is a big part of our future. It's like assisted intelligence, right? I guess so. A side question that I have for you. Do you have any plans of kind of implementing machine learning into Hubot into the future? Or is that just too much? I don't know. I wouldn't say we had plans. It could be where someone in the community takes it. But I wouldn't say we're planning on necessarily adding machine learning to Hubot. Right. But now it would be interesting. I mean, if there's anyone listening that wants to try, you know, Hubot's open source. We accept pull requests. You know, it'd be interesting to see what you... You could do that. But no. There's no specific like problem to solve in that with machine learning. I don't think so. There's, you know, chat can solve problems in terms of how you can surface information, but not necessarily specifically to Hubot. So chat brings up an interesting problem that I think a lot of people are facing right now in agencies. And that is, you know, trying to maintain focus while also staying connected. So, for example, at Whiteboard where I work, we have... Chat windows that will fill up enough that for me to catch up, it takes more time than for me to just ask somebody in that moment, you know, update me on what's going on. So I'm interested to know, how do you see, you know, focus alongside working with a chat program? In other words, you know, do you stay connected pretty much constantly? Or are there times where people are kind of disconnected from chat and working, you know, in code for a little while and then they come back? Or how does that work? Absolutely. Absolutely. Again, no dictated workflow. Everyone works differently, but sometimes I'll close the chat window and just ignore it. And that's the great thing. You know, it's very hard to close down your office when you want to zone out. You know, there's still people around you. People might tap you on the shoulder and talk to you. But when it's all happening in chat, you can close the chat window and just go and do whatever makes sense to you in that focused environment. You can turn notifications off in chat. You know, it's like, it's really flexible because it's like this pipe of information that you can choose to subscribe to at the right time. And, you know, again, it's about social definition, whether you'd codify these. I don't know, but it becomes around your culture. Do you do everything in chat or do you take it to a point where the next conclusion is we open an issue and discuss it there? You know, it's very good for quick fire discussions. That's not quite a topic yet. And then you lead that into an issue or, you know, the best answer is roughly talking about things in chat and then start hacking on it, build up a loose prototype and talk about that from there. You know, it's talking about work is work in the abstract. You can apply it to code and send a pull request. And then that's where it starts to begin. And it's fantastic to see, actually, you know, I see this all the time. I see team members and engineers. They just, they spike out a concept. And that's the beginning. And then my tip for anyone who uses pull requests is open that pull request as early as possible. Quite often, I'll make a near empty commit. I'll lay out some file structure, commit, send a pull request. I'll talk about my intent for this pull request. This is where, by the time I get to the stage of, of closing or merging this pull request, I want to get, I want to go through this cycle. And it's about communicating and bringing those people with you in that journey. So it's fantastic when you see someone who's incredibly experienced in their field, join the company, be shown how to do a good pull request. And you watch, this is my intent. And I will iterate towards that goal while communicating that intent all the way through. And it's a fantastic way to work. Again, it's this, it's, you know, it's rich with information. And it tells the story of how you built. And so, yeah, if I was to give a tip to any of the listeners, it's absolutely open pull requests early. They shouldn't be, I don't think they should be the last thing you do to get a change made. Cause then you haven't collaborated. You've, you've worked on your own, toiled away. And then you've, you're presenting this final thing. Well, the potential of what it could be is so much more when you're discussing it all the way through. And the tools are there within a pull request to be able to do that. Yeah. It's, it's a very powerful process for sure. And I think, so issues can be, multiple issues can be closed. And a single pull request, right? So let's say you have three or four issues that have cropped up in the past week or something, and you want to be able to close all those. It's, I would say, and you could correct me how you guys do things here, but opening a pull request and then mentioning those issues is better than trying to go and work on each of the issues in their, on their own. Right. Yeah. Because everything is networked together. And so you could just mention an issue and it'll show up in both sides, both feeds basically. Yeah, absolutely. And the pull request acts. Yeah. It's like this central trunk essentially. And then, so imagine you were rolling out a new class of host, for example, and you're doing a pull request and you hit an issue. A problem has come up. You can either CC the team that could help you with that into the pull request, or you can open issue on the repositories they look at. These, we organize around repositories as well. So, but the cross, the rich cross-referencing is amazing. So when I joined GitHub, GitHub had been a company for four years, just over four years. And I was able to go and look at pull requests. I was able to go and look at pull requests that are years old. But because the people that were working in the company at that time understood this rich tapestry of information you can weave and build by communicating clearly through issues and pull requests. I would say, see a setting in a file, for example, I'd want the history of how that was set that way. First DBA at the company, people before then were very skilled, but not necessarily experts in MySQL, for example. I would be like, okay, this is, this variable is set for a reason. What is that reason? We look at the git blame. You go back to the pull request that introduced it. You look at the discussion. People have pasted in graphs from the incident that caused people to set that setting. You then look at the issues that were opened before then. You just get this huge piece of information all from just the history. So you should, if you treat pull requests and issues correctly, you build up this really amazing body of information. Why spend your time documenting playbooks and wikis and things when you can just, the process of writing code is, is amazing. It's a living discussion. In fact, one of our, one of our major projects that we're working at now started as a pull request that was never merged. It was two, two engineers hacking on a concept, throwing stuff around and then closing the pull request and extracting out the, the real learnings from that pull request and, and then really getting going on the project. The final goal isn't always merging the pull request. It's enabling the debate around the change that you're trying to make. Yeah. Yeah. And that actually brings up kind of an interesting point. I think it's a really interesting point. I think it's a really interesting point. Yeah. It's an interesting point that I'd love to love to shift over to, which is features at GitHub. Before we talk about features at GitHub, today's episode is sponsored by Rollbar. With Rollbar, you can put errors in their place. Rollbar allows you to detect, diagnose and defeat errors. You will have the context, the insights and the control you need to find and fix bugs faster. You know, dealing with errors sucks. There are so many different types of errors. There are so many different types of errors that you can deal with. And a lot of times they are difficult to find, especially if you're trying to dig through the logs. And of course, that is not a fun situation to be in, especially when you're in the high intensity situation of trying to fix a server that is experiencing a bunch of errors under load. Rollbar works with all major languages and frameworks, and you can start tracking production errors and deployments in eight minutes or less. And you can start tracking production errors and deployments in eight minutes or less. So you can integrate your Rollbar into existing development workflows. So you can send the alerts to Slack or HipChat, or you can create issues automatically in GitHub, for example. How appropriate we're talking about GitHub today anyway, or pretty much any other service that you already use. Now, customers that use Rollbar include Heroku, Twilio, Kayak, Instacart, Zendesk and Twitch. Now, customers that use Rollbar include Heroku, Twilio, Kayak, Instacart, Zendesk and Twitch. Now, Rollbar is offering DeveloperT a free 90-day bootstrap plan. That's 300,000 errors that you get to track totally for free simply by going to rollbar.com slash developerT. Of course, that link will be found in the show notes at spec.fm. Thank you so much to Rollbar for sponsoring DeveloperT. So from birth to shipping, what is that? What is that? What does that flow look like? Where do the features come from? Who suggests them? Who makes those? Who decides eventually what's going to happen? Or what's going to actually be put into development? I think it's different for nearly every feature. Some features become really obvious at a certain point. You hear customers talking about these features. Or we use the product ourselves. It's the best dogfooding experience you can imagine when you're using the thing to produce. To build the thing. The tagline on the GitHub repository is, you're looking at it. And it's true. You are. We're using our own product to build our product. And that's this amazing experience. We've got these wonderful developers that have access to change the thing that they care about and they use to do their day job. So features come out of just that being a thing. That being possible. Other features come from listening to people that maybe work in a different area of software. Or within a different type of company. And they share with us how they believe GitHub could be made better for them. And we think about that. We look at data. We look at research. We have research teams that will go and do studies on how GitHub is being used. And we will learn. Because you can only empathize with your users to a certain degree. And then it becomes really, you have to look for that. You have to mine the information and the usage patterns that tell you how things can be made better. And you have to quantify that. So features come out of those processes. Or they just come out of a random late night discussion in chat where, Oh, you know, this would be cool. Oh, I can start hacking on that. I can build that up. And then suddenly other people get involved and it rolls and it rolls and it becomes ready to go. And so it's different every time. Again, not prescribed. Too much process around that would inhibit different ways of doing things. We always explore new ways. And they work for certain areas that we're working in. And the next project may just, it might not be applicable. That's really interesting. Because it sounds like a relatively organic process where things can come from pretty much anywhere. Yeah, and we have a wonderful product team that tend to the future of the product. They think about how the product should grow and be cared for. And there's something, you can't underestimate that either. There's some things that just don't necessarily always happen organically. You have to really proactively think about your product and care for it. And our product team do that. Yeah. So what's amazing about this to me is that GitHub has over 100 engineers at this point. And having that many minds and that many different types of experience and different contexts, that's a powerful machine, right? If you only have one gatekeeper that decides, well, this feature is good enough to put some energy behind. Well, that's not really a useful, nearly as useful as taking advantage of that many mindsets, that many creative people. So I love the idea of allowing for that organic growth of features and marrying the process-oriented stuff where you're going out and interviewing your customers, really, to determine what are some features that we never could have come up with ourselves, that our customers are pushing us towards. But also using the product internally and saying, these features make sense for us. They're probably going to make sense for other companies as well. Absolutely. With that said, what are some of the big problems that GitHub is facing today? Well, we're scaling a workload that's relatively new. The use of Git is growing daily, and especially on GitHub. During this podcast and conversation, the number of bytes we're storing for Git has grown, and it's always growing. And that brings new problems. You have to revisit your approaches, and you have to think about the workload that you're scaling. And it's a fairly new one. There's better understood workloads across the industry, like a web apps database workload. It's fairly understood. But it changes as the use of Git grows. It becomes a larger and larger scaling problem for us. So it's keeping ahead of that, and building systems that are intelligent and resilient to the different workloads that will come up, and fail in predictable ways. So there's a continual growing challenge. And we have the typical challenges that a growing company has. And I guess we have this problem that you could imagine the future of any feature on GitHub. It's so open-ended almost. And it's about keeping the core essence of why GitHub is important and relevant to our users, and not diluting that too heavily. You can get lost in just a mire of features, and it becomes confusing and hard to use. Part of the reason I love using GitHub is it's simple, and it's fairly opinionated about a workflow. And I believe it's the right workflow, and it's a workflow that I've seen work very effectively. And so that's a challenge as well, is how does it evolve to include everyone's workflow without just being a cludge of features that represent necessarily every flow. Sure. Yeah. Well, it is a powerful and validated many times over by large and small projects alike. And so I think that's a challenge. I think you're on to something about the workflow there. Yeah, I think it's a good workflow. Again, no workflow should be prescribed to anybody in different industries and different types of people who use a different workflow. And that's the great thing about it. We have an API. We have various different ways that you can just plumb in how you want to do things. And you see really novel and interesting ways that GitHub is being used. It's fascinating. It's open-ended enough to be flexible, but still simple and elegant in the way that things are done. Sure. Yeah. So, Sam, this has been a great discussion. I think we could go on for hours, I think, and really unpack all these topics. But this show is short. It's intentionally short. We don't have tea here today. It's a shame. I feel like that was definitely a shame. I could have brought some tea from home. I have my own stash. But I do want to ask you two more quick questions, questions that I always ask every guest that comes on the show. The first question is if you could talk to every developer, give them 30 seconds of advice, what would that advice be? I'd say don't believe in magic. There's things that will be presented to you throughout your industry, throughout the industry, throughout your career, that will appear like magic. And the truth is none of it is magic. You really need to understand every trade-off that you're making. Your path towards being an experienced engineer is really, you get quicker at understanding trade-offs that you're making. The thing I love about working with really experienced engineers, and this is the thing that's unique to truly experienced engineers, is they understand the trade-off that they're making. They're constant. You will always be making a trade-off. You know, there's, there's, there gets to a stage where it's, it's physics at the end of the day. And not acknowledging the trade-off just diminishes your ability to truly understand the systems that you're, you're working on. Go deeper than what's presented to you. Understand the trade-offs that software is making for you. In the database world, the databases that promise they will do everything for you. They'll do automated scaling and failover and high availability and data consistency. And all those things all at once. It's not possible. There's something that gives. And the point is you understand the trade-off you're making and you understand the failure case and you build around it. You build systems that fail predictably, not that don't fail. That's a, you're never going to achieve the systems that don't fail. You know, power supplies go out. You know, it just happens. Don't hold onto that as something you can reasonably achieve. But instead, try and understand the way it's definitely going to fail and make sure you're doing it. And make sure you understand that failure case. It's great advice. Don't believe in magic. The second question I like to ask every developer, and this is really a question that I like to ask pretty much anybody I come in contact with. But what do you wish more people would talk to you about? What topic do you wish you could talk with people more about? I think we could have more production, productive conversations as an industry if we talked about the type of innovation that we should and shouldn't be doing. Like, I feel like people, I feel like people feel tasked with innovating every area of their stack. I don't actually believe that's the right thing to do. You know, we have important things to build that are unique to what we're doing. So, if you choose to spend your time innovating or inventing your own database, you're going to waste your time because it's done. A lot of these problems have been solved. Unless you're at a huge, huge scale, what's good for most companies around your scale is good for you. And so, don't be unusual. Build off of boring technologies. There's technologies that are in use at huge scale that are fine. If it's fine at that scale, it's fine for you. Just understand that you don't need to be completely unique and different in every case. Make your product unique. Make the things you offer your users unique. Don't necessarily have to have a weird and interesting and odd stack. I think we have more productive discussions if we just said, this is a solved problem. Or this is a problem that maybe we just don't even need to care about. And then continue and do what's actually interesting and what we can do with technology. We can sit and write our own databases day in, day out if we want. Is that really the best use of the time of our engineers when we can actually use technology to do interesting and new things in the world? So, yeah. People should ask me about the things I don't care about. You know, the boring stuff. Yeah, that's great. Well, perfect. Thank you so much for joining me on the show. And thank you to GitHub for creating a product that all of us use pretty much. Yeah, thanks for coming. Thanks for everyone that uses GitHub. And to the open source community and the communities that have surrounded us as a company. It's wonderful to see the way technology is changing and growing. I'm just excited to be at the company that's in the middle of all of that. Awesome. Thank you, Sam. Thank you. Thanks for listening to today's episode of Developer Tea. Of course, if you missed out on the first part of my interview with Sam Lambert, you can go and check it out at spec.fm. Thank you again to GitHub for inviting me to talk to you today. And I'm excited to come and visit the GitHub office. And thank you for listening to today's episode. This podcast would not exist without you, the listeners. So thank you so much for being here and listening to me through your headphones or through your car speakers or however you're listening to this podcast. By the way, if you're enjoying Developer Tea, make sure you leave a review for Developer Tea in iTunes. There is a link directly to Developer Tea's page in iTunes in the show notes. So you can find those in your podcasting app. Of course, you can find them at spec.fm. And if you leave a review, this is the best way to help other developers find Developer Tea. And it's the best way that you can help the show continue doing what we do. So thank you so much for listening and for leaving a review. And thank you to today's sponsor, Rollbar. With Rollbar, you can detect, diagnose, and defeat errors in your application. Go and check out rollbar.com. Thank you so much for listening to today's episode. Until next time, enjoy your tea.