Hey everyone and welcome to Developer Tea. My name is Jonathan Cutrell and in today's episode I interview Sam Lambert. Sam is the director of Systems at GitHub. If you haven't used GitHub in the past, you probably will in the future. At the very least you've probably heard it. You've certainly heard it if you've listened to every episode of Developer Tea. So I'm really excited to talk to Sam about scaling such a large operation. Today's episode is sponsored by Hired. If you are a designer or developer and you're looking for a job, check out Hired.com. Companies like GitHub have actually used Hired to find employees to work in their company. So go and check it out Hired.com. We'll talk more about what Hired has to offer Developer Tea listeners later on in today's episode. But first I want to jump straight into this interview which I did live at GitHub. Thank you very much to GitHub for bringing me into your office and allowing me to check out all the awesome stuff that you have gone on there. I really appreciated the time there and I appreciated getting a chance to meet Sam in person and do this on site. So thank you again to GitHub for bringing me into your office. And once again, let's go ahead and jump into the first part of this interview with Sam Lambert. Sam welcome to the show. Thank you. I'm excited to have you. I'm ready to talk about GitHub systems. Sam, you are, well, why don't you tell the audience of Developer Tea what your position is at GitHub and what you do for GitHub on a day-to-day basis. So I'm the director of systems here at GitHub working with and for the system's team, the system team primarily responsible for the scaling of Git, scaling our data stores such as MySQL and basically doing general systems engineering around our stack. So the skill levels range from kind of see engineering to then running sort of MySQL and databases at scale. So we'll spend our time doing various different things and helping out in various different areas of the company. But like a large part of our daily work is around scaling Git and keeping Git available to all of our users. So we run a large amount of infrastructure to support storing as many Git repositories as we do. And the kind of the scaling and the proxying and the working, basically making Git available across our entire fleet of servers is the responsibility of the systems team. Do you have maybe an estimate that you can give listeners of how many repositories at this moment there are? Is there a good estimate? So we're above 30 million projects on GitHub.com right now and that number is growing significantly day on day. So there's a huge amount of repositories, a large amount of users. The use cases for this workload as well are very varied. We run anything from CI farms, hundreds and hundreds of hosts connecting the single repo, downloading the code running CI against them to individual users using GitHub every day for their normal workflow. So among those above 30 million repositories we have different workloads and different huge open source projects to private small repositories that people are running. That's great. I know I've used GitHub for, well, I've been a member for at least eight years now. So since I was in college actually, I've been using GitHub. I was introduced to it by a company that I did an internship at. And this was when I had first learned a little bit about front end development and I was pushing things through FTP and that kind of stuff. I've done such horrible things as well. Sure. And funny thing is, there are people who are listening to this show and this shows how large of a field this is and how varied it is. There's people who are listening to the show right now who surprisingly are still using that method because they haven't really been shown the better way. And when I say a better way, I mean, anything that saves versions pretty much. But for those who are listening, I think it's worthwhile to mention that there is a, we may need to disambiguate between Git and GitHub. So Sam, you can probably actually do that more eloquently than I can. Yeah, absolutely. So Git itself is an independent open source project. GitHub contributes to Git. We have core Git maintainers working for us and we've pushed a lot of changes into Git. But Git as it stands as an open source project, we host Git, we allow, we build on top of Git to allow you to collaborate using our software and our site. But fundamentally Git itself is a standalone version control system. And then we use that and get hub to fill form the basis of what we do. Well, I guess it is definitely the most popular repository, SCM, right? Yeah. This first control. And possibly partially to do with the fact that GitHub exists, right? GitHub pretty much requires that you use Git. Is that true? Do you guys don't allow for? We do actually quite interestingly. You can transparently use SVN through on any of the any repository on GitHub. You can access it via SVN. You can also import from other version control systems into GitHub. Very interesting. Yeah. But the default, the thing we are promoting is essentially the use of Git. Whether or not you choose to use SVN or when people do, in fact, an engineering, one of our engineering blog posts came out today on how we build that, how we maintain that SVN bridge. So that people who do use SVN can still get the benefits of GitHub. And Git actually wraps SVN. Isn't it? Isn't there? There's not a SVN wrapper and Git. I was under the impression that there was a way to do SVN work directly and Git. But maybe not. I think that could be projects that would allow you to interface, but I'm not sure about what your referencing necessarily. See, I'm one of the developers who was introduced into source control through the lens of Git. So I don't even have experience with these other ones. There's definitely scripts for trends. Like I went to a company that moved from SVN to Git and we used this new responsive scripting you can to support between the two. But primarily Git is completely standalone. So I want to kind of back up a little bit and ask you about how you got to where you are now because an interest in systems, you didn't just wake up one day and say, well, I guess today I'm going to start building systems. So how did you arrive as a systems engineer? So I kind of fell into it really actually. I was doing development years ago at a small web company that was doing e-commerce sites. And there was no one really maintaining the servers looking after things, the databases. And I kind of just, I really just want to work on the things that are most impactful for the place that I'm at. What that looks like, it doesn't matter to me really. As long as I'm making impact, I'm happy. And that was the place I could make the most impact was really looking into the operations of things, the back end. And I sort of grew into doing that. And then I moved to a company that was an early stage project, it was like a prototype project, which just exploded. It was a telemetry platform that when we started, it was for electric vehicles and they would send back thousands of piece of information a second per vehicle into this into this into a queue. We processed it into like lots of databases. And it was like data analysis large-scale systems and that went from something tiny to something huge. And that's where it really changed for me. It's like we had this need and this necessity to understand systems at a deeper level. And that's where the interest really started to grow to me. For me, it was just really, really enjoyable, especially lighting databases. And that's how I came to GitHub. So GitHub didn't have a DBA. They were looking for one and I applied, got the job here, started doing database work here. And then my interest started to wander a little bit into more elements around, how can we make positive changes to group, how can we work on things like making the site more resilient, more available, start introducing some processes around how to do that in different ways of thinking. And it sort of moved in from that into kind of a leadership role within the company. And so now that's how I'm doing what I'm doing. So telemetry, just for anyone who doesn't understand what the term telemetry means, I think that's probably one of those terms that could be confusing. So yeah, I can't recall the dictionary definition, but really it's collecting data on a remote device and sending it back to a central location. So you have like telepharm. Yeah, there's telemetry everywhere, really. I remember at one point we were talking to people that would do telemetry on floodgates so that the floodgates could report back. So the state is of what they were doing. Like telemetry can be absolutely anywhere. Cars run a lot of telemetry now and essentially they phone home and gather data and bring it back. So like the original internet of things then, I guess, kind of at an industrial scale, I guess. So in our case, it was electric vehicles, they were commercial electric vehicles, they were huge. And it was a very much experiment to get these out among drivers and understand how they were being used. And there was a lot of mean, at that point there was a lot of misconceptions about electric vehicles and how they worked and we were able to use data to actually to solve some of the problems drivers were seeing. And sure, it was fascinating actually to see that type of data centrally. Today's episode is sponsored by Hired. On Hired Software Engineers and Designers can get five or more interview requests in a given week. And each offer has salary and equity up front. Now Hired has full time and contract opportunities. So it's not just for people who are looking for a long term job, but also those who are looking for the jobs that will fill in the cracks between other contract opportunities. Users can view these interview requests and accept them or reject them before you ever talk to any companies. So you as the developer, you never have to have any awkward conversations. Hired works with over 3,000 companies from startups to large public companies from 13 major tech hubs in North America and Europe. And perhaps the best part for you is that it's totally free for users. Now if that didn't already sell you, then perhaps this will hired normally provides a $1,000 thank you bonus just for using their service once you accept a job through hired. But if you use a special link that you can find in the show notes at spec.fm, then that bonus doubles to $2,000 just for using the special link in the show notes. That special link is hire.com slash Developer Tea. If you want to go straight to it, but of course once again it will be in the show notes at spec.fm. And thank you again to hire for sponsoring Developer Tea. Go and check it out hire.com slash Developer Tea. But I'd like to know a little bit and you've already shared with me about the remote working culture just a little bit. But I'd love for you to kind of talk about what a typical day a GitHub looks like for a young developer. Somebody let's say I was to get hired at GitHub tomorrow. What would my job look like? So I think it depends on what area of the company you're working in. And also there is no real typical day. A lot of the work people do is very much self-guided. We have the company is largely distributed over 50% of the company aren't based in San Francisco they're based around the world and people working in different time zones and just different situations really. You know there's members of the we have certain members of the team that are nearly nocturnal they work in just what in the hours that makes sense to them. When I was I started my first two years at the company was from I spent my time in England working from there and I would just work even I could just it was great I would work at times that was sort of would fit around my life as well and you know enjoyed working in the evenings. So really there is no typical answer and that's what makes GitHub really interesting and fascinating to me. People they work hard people dedicate themselves to this company but it doesn't necessarily represent itself as showing up to a desk between nine and five. There's activity on weekends people might feel a flurry of creativity and they wake up and they just get going and start doing things and start start building. So there's no real prescribed way of working. Yes there's no typical we have a chat room that we have chat rooms that we're continually chatting in that's where that seems to be that's where the company happens really. Even though we have members of the systems team that are based in San Francisco we tend to do our work asynchronously through chat through poor requests through issues because if we were to sit in a meeting it would be about us and not about the people aren't there and those people have valuable contributions to make context to provide and so doing it within chat and in an asynchronous manner benefits all of us. There's this almost this contract that sort of social contract that if you send a poor request you're not expecting for you to be productive you're not expecting someone else just to be the other side ready to just respond to you right away you have to manage your time and what you do in a way that allows contributions to come from people all over the world who may or may not be working that day may not work in the same similar schedule to you it's just kind of the way we do it and it's actually once you get used to it it's a fantastic and empowering way to work that's also extremely productive. Yeah you said quite a few things there that we could spend you know all day talking about I think there's a lot of a lot of approach to working environments that that kind of the idea for example you mentioned that you may have a flurry of creativity over the weekend that's something I've talked about on the show before the idea that we have kind of seasonal working habits right so you can go through a week where you're not very productive at all and then you can have five hours of super productivity and then you go through another week where it's just pretty normal or you can go through a whole season where you're either only kind of productive or you know the idea being that you can't really expect an exact amount of productivity out of somebody you know in a given period of time it fluctuates people fluctuate and so it's interesting to hear that especially when you have team members that are remote you know how I guess what I'm interested in hearing from you now is how do you ensure that you know people aren't taking advantage of the system and then I would assume that this is something that you kind of take care of in the hiring process right but the flexibility is such an important piece of the remote culture you know how how do you ensure that as a culture everyone kind of agrees that there's a certain level of productivity that we are you know shooting for like what is the goal I guess is what I'm saying yeah so we I don't think we guarantee it to a level above any other company guarantees it I think I don't think sitting at a desk for eight hours a day is a guarantee of anything you can game any system if you're not interested if you're not engaged if you don't care you're not passionate it's fun I've went on when I've worked nine to five jobs in the past that I wasn't passionate about I did my work I showed up I had unproductive days I'm productive days just like everyone else does I think we do it the same way everyone does and we we have a strong peer feedback culture we give each other feedback we understand the standard that we have for contributions and you it's very difficult to quantify anyone's value absolutely and it the difficulty doesn't get there's no more difficulty to do that in our culture than I think in in in any work culture really it's about is that person making positive change towards the goals of the company does the persons manager does you know the persons peers believe that that's happening and is it as you're being seen then that's what we need to worry about I think in that sense that's so I pay attention people who work at smaller companies GitHub is a very large company in relation to the average you know web development firm for example and the quote that I kind of pulled out of that is the nine to five is not a guarantee of anything the eight hour workday is not a guarantee of anything it feels like well it really works as kind of a security blanket right we create this system this process where everybody follows it and that makes it measurable yeah or predictable in some way but the reality is the output isn't necessarily guaranteed and really what you're hiring is a person not a person's time yeah absolutely you want once that person is understood as someone that you want to work with is then about optimizing for their creativity and then and not by not dictating a single workflow and GitHub we optimize for a range of people I'm not saying this is the best workflow for a legal team I'm not saying it's the best workflow for a HR team I'm saying for the engineering side of the company this works well and we don't dictate that everyone works the same way there's just different workflows as somebody works and leadership my workflow is highly synchronous I spend a lot of time talking to people and I feel I benefit from being located near the HQ but you know it's possible without that you know it doesn't there's no use dictating a workflow to anybody and this seems to be the one I believe is the right workflow for engineering teams my views on that may change as the company shapes and grows I'm not I don't have a dogmatic approach towards any workflow I'm happy and I'm open to it as things evolve and I've watched the workflow evolve to get a evolve at GitHub over the years and I it's exciting to see how it built like it's exciting to see the things we've discarded and the things that we you know we're holding dear to ourselves as things we won't want to leave our culture but nothing's rigid forever and that's the important thing that's exciting thing so it has to be a different company month on month and that's wonderful we're growing towards the goal and and doing good things and let's continue the exciting thank you so much for listening to the first part of my interview with Sam Lambert the director of systems at GitHub if you don't want to miss out on the second part of the interview which will come out in just two days from now make sure you go ahead and subscribe to Developer Tea and whatever podcasting app you use you can do that in iTunes you can do it very soon you'll be able to do it in Spotify hopefully some of you listening now you are listening through Spotify so so check and see if Developer Tea is in your Spotify feed it's called shows in Spotify instead of instead of podcasts but any podcasting app you use pretty much you can subscribe to Developer Teaso make sure you subscribe if you don't want to miss out on future episodes thank you once again to hired for sponsoring Developer Teaif you are looking for a job in your designer or you are a developer I'm regardless if you want a contract job or a full-time opportunity hired is a fantastic place to start and you don't have anything to lose because you can get on there for free and view interview requests and on top of that if you end up getting a job through hired they give you a $2,000 bonus of course that is the special bonus for Developer Tealisteners you're going to want to go to hired.com slash Developer Tea and as long as you sign up through that link that bonus that is normally $8,000 doubles to $2,000 so again thank you to hired for sponsoring today's episode and thank you for listening to Developer Teaas always you can reach out to me at developertea@gmail.com or you can reach me on twitter at at Developer Tea and until next time enjoy your T