« All Episodes

Part 1: Ben Lesh on Reactive Programming at Netflix, Massive Data, and the Threadsafe Future

Published 4/15/2015

Ben Lesh is a senior UI engineer at Netflix, where he works on projects that use reactive programming in JavaScript to manage massive amounts of data. Ben talks with me about a variety of topics, focusing on reactive programming and data stream processing. Ben also gives his advice to you, whether you are a young developer or well into your career. Make sure you Tweet at Ben at https://twitter.com/benlesh and thank him for coming on the show!


Transcript (Generated by OpenAI Whisper)
Hey everyone and welcome to Developer Tea. My name is Jonathan Cutrell and today I have the pleasure of interviewing Ben Lesh. Ben is a senior UI engineer at Netflix. At Netflix, Ben is actually working on a lot of reactive programming, specifically he's working on front-end, JavaScript, reactive programming. And that's exactly what we talk about in this interview. It's a two-part interview. Make sure you check out the second part. And make sure you follow Ben on Twitter. His Twitter handle is at Ben Lesh, B-E-N-L-E-S-H. And then tweet at him and say thanks for being on at Developer Tea. If you send him a bunch of tweets, I will appreciate it and so will he. Now let's get to the interview with Ben Lesh. Ben welcome to the show. Thank you so much for being here. Thank you for inviting me. I'm excited to talk to you because I watched your presentation at NGConf. I guess more accurately, I listened to it. And you and I were talking before the show started and I was impressed by, first of all, your stage presence as another person who does speaking for people. That's not an easy thing to do. You're right. You're right. It wasn't easy for me at the time. It's just a matter of, I guess, preparing ahead of time and practicing and taking a deep breath every now and then when you're looking out at that many faces. Sure. And what's interesting to me, though, is that of all types of people, you work at a company people might would have heard of by now. It's a little startup that does these videos. Can you tell me about what you do? Sure. I'm a senior UI engineer at Netflix on the Edge Tools and Insights team. So what I work on there is a real-time streaming dashboard that's all browser-based, targets evergreen browsers. And it's a real-time streaming dashboard of all of the events that happen in Netflix Cloud and on their devices. So you can imagine everybody's movie starts or all the various services that Netflix runs and when they spin up or get torn down or all of the latency vectors and that sort of thing. So there's a lot of data that we cram into a real-time screen. And that's what I work on. Yeah, I bet there's a lot of data. I can only imagine how many different types of services that you're having to integrate with on a daily basis. So right now, what I do is we hit a backend server. That backend server goes through a collection of servers called Mantis as a collection of services. And what they do is they actually operate in tiers to step all of those events down and aggregate them into something that we can consume. And it just flies it out to us over a web socket. Interesting. So that kind of helps unify all of the data coming in. I'm assuming. Right, right, right. Yeah, it's not something that I would want to do by connecting to multiple services directly from the browser. It's a, you do some of that. We do some of that. But most of it goes through a single backend service because it would just be way too much to try to coordinate all that in a single thread and keep my code base small. I bet I can only imagine. So when you say a coordinator all that in single thread, you're referring to the JavaScript environment, right? Right. That's right. So and you're most of what you're doing is front end JavaScript work essentially. All of it. Yeah. Okay. And are you guys, so I know that you spoke at NGConf, but you mentioned that you worked with Imbar. Do you have any particular tools that you kind of stay within or do you keep yourself pretty open in terms of what tool to use on a given thing? Well, I'm very open. So when I was hired in Netflix, I was actually hired for my Angular expertise. And prior to me starting there, they had contracted Eric Brin, who's an Ember core team member and gotten the project started in Ember. And they were like, hey, are you willing to work in Ember? Of course. I'm always, I'm always willing to try new things. I'm always willing to learn new things. Honestly, the best part of my job. So absolutely. I'm always open to new tools. But for this particular project, we've got a lot of momentum in Ember. So I don't see it switching gears any time soon. Sure. That makes sense. Yeah. And it seems like you were mentioning in your in GConf, the culture at Netflix is a good culture. I don't want to blanket statement that too big because it's going to sound kind of buzz wordy, but explain to me kind of, OK, so when I think of good culture, I think of enabling learning, for example, and enabling exploration and being able to solve problems in new and creative ways. What do you think about Netflix, the culture at Netflix? Can you just speak a little bit to kind of how you guys work internally? Well, I mean, enabling is probably the best way to put it. Maybe they have this, it's a culture of trust. So they trust that I'm there to make good decisions and that I know what I'm doing. So really, the supervisors I've had at Netflix have been very much a part of helping me, guide me through the decision making process. But most importantly, kind of giving me what I need. And then they kind of get out of your way, they let you do your job and they let you make the decisions you feel you need to make. And you feel you need to use some newer technology or if you have a demonstrable, good, new or good way to try to do something, they're very open to it. But it's all about me being responsible for my own output and my own results. Sure, yeah, which I believe is really essential to being able to work in an environment where you're able to learn, right? Because we all know as Developer That things move fast and you mentioned one of the funny things you said in G-Com was that if you have a slow JavaScript app, you wait a year and it's two times faster because things progress so quickly, right? And that may not be true anymore because your specific point then was about CPU cores and the fact that they aren't going to be getting much faster than they currently are. But the point was that things are constantly changing and you can expect them to change. You can plan on things to change. And for you to stay up with that, you have to be able to learn, you have to be able to explore. And I think it's totally, it's not even an option anymore to have a culture where learning is enabled. It's like a fundamental thing. You have to be able to have that culture or at least enable Developer To learn in their off time, whether that means giving them extra off time or free hours to explore things or whatever. I think that's completely essential, especially now, especially for front end. Right. Yeah, I would say now one thing I would say though is I would put the, I would put the onus on the individual for, you know, promoting their own learning. It's good to have a company that allows you the freedom to do what you feel you need to do at work to get things done. However, I was just having this discussion with a friend of mine the other day about how you know, he was saying, oh, my, my particular job is, is boring. And I told him, I said, you know, I can't, I don't think I've ever had a development job where I didn't have some downtime. And you know, I've never been at a development job where there also wasn't some need, some hack that you've thought of that you could, you could put together to make life a little better for the company or for yourself at the company. And, you know, there is an opportunity to try something new. So, you know, a job is really about as boring as, as you make it. Now there are some places where you can't get away with that sort of thing, I'm sure, but I think it's very important that people push themselves to do things that are hard. A lot of the things that I've learned about Angular or about Amber, in fact, the way I was discovered by Netflix was because of this stack overflow, which I think we're all familiar with. When I first started answering questions on Angular, on stack overflow, it was to learn Angular because I didn't know anyone who knew it. There wasn't a lot of good documentation. So I would find questions that I did not know the answer to and I would learn the answer and answer the question. So that was like a goal I had to try to learn Angular. And actually, it was my stack overflow account that grabbed somebody's attention in Netflix. Wow. Whenever I was recruited. So, you know, it can pay dividends in other ways, I guess, too. But, you know, I always say to people, if you're not learning something new every day, you're probably just forgetting something you used to know. Yeah. Yeah. Yeah. It's very, very important to push yourself to learn new things. The least in my opinion. That's really awesome. That story about you using stack overflow to learn. We talk about learning about learning on this show, the kind of meta exploring how different people learn. And that, I think, is a really good one to hold on to for anybody who is trying to learn something, go and find out what other people are asking about it. If you don't have a project to work on, especially, like we've said on the show, Ben, you can maybe speak to this a little bit more. I've said multiple times that the best way to learn is to actually work on something, work on a project, even if it's a pet project, put a real problem in front of your face. But what you're saying is you were putting real problems in front of your face just not your own. Right. Yeah. I mean, other people are going to find problems, you know, at n times faster than you, where n is the number of people more than you, right? So. Sure. And there's a lot of problems out there waiting for someone to solve them, I mean, realistically. So I wouldn't do it for the stack overflow points or whatever your scoring system is. But it's definitely worth going in and I mean, even just checking out problems that have already been solved. Like if you're reading through them, you know, sort by votes, go through and find something that looks kind of interesting and check it out. I mean, there's some, there's some pretty cool stuff out there. Yeah. I actually learned quite a bit of Ruby, while Rails stuff. There was one guy who was looking to, it was something to do with a database query. I can't remember exactly what it was, but I did the same thing. I actually decided, you know what? As part of my learning exercise for the day, I'm going to sit down and figure out a really performant way of handling this query. And I wrote the thing out and it actually ended up becoming like the top answer for that question, which is valuable in a lot of different ways. It's, but most importantly for me, I learned that day, you know, and so regardless of that little number taking up next to my name, I actually was able to learn something that I could use in the future about databases. Sure. Sure. Speaking of databases, nice little segue there. You guys, you mentioned a really incredible statistic. I'm sure you mentioned you, you can't, this is kind of pounded in your head every day. But Netflix handles how much of the bandwidth in North America? You said like one and a third. It's more than a third. So it's 35% I think is the last number I saw. That is an incredible amount of data. Yeah. Yeah. So that's, but that's the funny thing is that, you know, so that's just the bandwidth we're using. Yeah, I just, I had, we had an internal meeting yesterday and somebody said something and I'm paraphrasing here, but it was something like Netflix is a metrics engine that happens to stream videos. Because we have a lot of, a lot of metrics, a lot of data on just about anything you can imagine. Like any, anytime I think, hey, do we have data like this, like, you know, geo located who started watching House of Cards, right? Like you can get that. There's, they, they have a very, very, very wide variety of metrics that they're collecting to nearly insane degrees, I think. Yeah. I would imagine that there are a lot of insights that can be, I'm going back to my master's degree. I did a little bit of work with recommendation engines in my master's degree studies. And that data is probably very rich. They can, I can imagine they could predict just about every show that I would watch at this point. Right. Right. And you're also looking at, you know, data from tens of millions of users across, you know, millions and millions of devices and all of the people rating movies and checking movies out and watching movies and, and people like you. And then I'm your friends list and so on and so forth. So I think that's just fascinating. I haven't had a chance to work with anybody from the departments that come up with those, with those suggestions. But yeah, they've got a lot of data they can work with. Yeah. Yeah. So I'm going to show my business ignorance here. But there are things like taking multiple variables into account to determine, you know, what is the most likely next step for somebody? So like you can imagine that there are probably, let's say, 5,000 people who have almost exactly the same tastes as you. And if 4,990 of them have watched one show that you haven't watched, then it's very likely you're going to watch or you're going to like that one or you're going to watch it eventually. That kind of prediction, I would imagine at Netflix is like par for court, par for the course. Uh, possibly I'll have to be honest, my knowledge of how they, how they come up with their recommendations is is about as good as any of your listeners. I don't, I don't work directly with with that, uh, yeah, that particular piece. So, but you're on the opposite side. So you're working with internal teams and how many people are using or well, I guess how, who is using what you're building? Is that all of the internal teams or is it like just data analytics folks or how is, how is it being used? It's currently it's being used primarily by the people in the, the edge platform teams. So those are the people that work on all of the cloud services and reliability and that sort of thing for the Netflix cloud. It also gets used by the customer service so that the top tier customer service will be able to use some of the tools that I work on that to be able to go in and say, you know, hey, what's happening with this particular user's account? I can look and I can see that they've started streaming here and it went up to 480p and up to 420p and that up to 1080 and so on and so forth and, and get like different error events and that sort of thing off of people's accounts to help them out when they call in with problems. That's really interesting. You know what's interesting about this is that there is a whole section of Netflix that your job just doesn't even have to like you don't even have to be aware of those pieces which makes it like you're working on something entirely different and a totally different and compartmentalized piece of the company but still empowering like a completely separate part of the company which I guess is is is normal but I think a lot of people at tech companies they think that they have to have like let's say for instance, I wanted to go and work at Netflix. I might would think that I have to have you know a lot of knowledge about, I don't know, television shows or something which just probably isn't the case you know. Yeah, that's I mean not unless you were you know part of a team that was acquiring television content maybe then it would be important right? Right, right. What they generally what I've seen at Netflix is they hire people that are experts at a given field and then you know you have these these people that individually are you know an expert in front end or an expert in back end or an expert in you know maybe DevOps or or something like that and and they each kind of we're not silo I mean we're all part of a team but we're each kind of working on our own piece of an overall system and with with all of the freedom and responsibility that give everybody everybody there I mean people do dive into other aspects of that but the really great thing is everybody that works there is truly an expert I'm always impressed every time I talk to anybody about anything about what they're working on at Netflix or the other thing is it's particularly impressive is Netflix hack day so that's they do that once a quarter I think and everyone comes up with their own little Netflix related and hack projects that they do. I started to mind Jeff Bush came up with like a globe that visualized stream starts for people like real time so you could see around the globe like geolocated like who's watching what and as it was happening and of course it's flickering and it's amazing. Very cool. Another team made a Nintendo and NES cartridge that played Netflix I guess I know I haven't seen this but my my what I was told this was actually going on during NG comp what I was told is it actually played like an 8-bit you know blocky version of the video too. That's fantastic. Yeah I mean there's there's some really really smart people that work there and everybody that works there is just tremendous. So very very great place to be to meet smart and interesting people. So listeners you might be detecting that work that we are kind of obviously being positive about Netflix I just want to make a disclaimer Netflix is in no way giving Developer Tea money right now. This is entirely a net just a conversation between me and Ben but I know that Netflix is hiring and I I'm relatively certain that there are people listening to this show who Netflix could be the next place that you work. Ben can you tell me what they need to do if they're interested in that kind of thing? Yeah you would go to jobs.netflix.com and I would recommend looking at the the slide deck there. That what it talks about Netflix culture and I have to say every time I say culture and I know I said this is NG comp to or I hear someone say oh our culture is great I just think that's that's bold that's just you know it's like a it's like a company tag line right but it really it really isn't bold I've been there for a year now and it's I mean it's it's lived up to whatever hype they they gave it before I started so it's it's been it's been really great it's been really great. Awesome so go check out jobs.netflix.com if you're interested in working with Ben. Thank you so much for listening to this episode of Developer Tea with Ben Lesh make sure you check out the second part of the interview with Ben we will continue talking about reactive programming we'll also talk a little bit about imposter syndrome and Ben's unique approach to handling imposter syndrome. Thanks so much for listening to Developer Tea without you the show would be useless and meaningless and so I really appreciate your time make sure you follow me on twitter at Developer Teaou can reach out to me there and if you would like to support the show go to DeveloperTea.com front slash donate any kind of donation is a huge help and I see every single person who donates to this show it's it is a huge help to get those financial donations thank you so much for listening and until next time enjoy your tea.