« All Episodes

Part 2: Ben Lesh on Reactive Programming at Netflix, Massive Data, and the Threadsafe Future

Published 4/17/2015

Ben Lesh is a senior UI engineer at Netflix, where he works on projects that use reactive programming in JavaScript to manage massive amounts of data. Ben talks with me about a variety of topics, focusing on reactive programming and data stream processing. Ben also gives his advice to you, whether you are a young developer or well into your career. Make sure you Tweet at Ben at https://twitter.com/benlesh and thank him for coming on the show!


Today's episode is sponsored by DigitalOcean. Go to https://digitalocean.com to get started, and use the promo code "DeveloperTea" on the billing page after you create your account to get a $10 credit!

Transcript (Generated by OpenAI Whisper)
Hey everyone and welcome to Developer Tea my name is Jonathan Cutrell and today is the second part of my interview with Ben Lesh. In this part of the interview we talk about reactive programming a little bit more and we also talk quite a bit about imposter syndrome and then Ben gives you some really good advice for new and experienced programmers alike. Thank you so much for listening to Developer Teawhile you were listening to this episode if you didn't do it last time make sure you open up whatever podcast application that you use. Go ahead and open that up and then subscribe to the show and what that does first of all is it guarantees that you never miss another episode of Developer Tea. It gives you a quick alert whenever a new episode comes out which is four times a week by the way we typically do every Monday, Wednesday, Friday and then either Tuesday or Thursday will have an inspiration episode of Developer Tea. So if you subscribe you don't miss any of those and also you don't have to think about it it takes some time out of your day of having to remember having that kind of cognitive overload of going and saying I need to go and check out the newest episode of Developer Teaor just entirely forgetting it altogether and then falling behind on that episode and all that so go ahead and subscribe that is a huge way that you can take a little bit of load off of yourself and help out the show and help out the show by just by just listening to it. That's an amazing thing about the show is that you just listening to the show helps me out. It's really awesome. Thank you so much for listening to Developer Tea. Now let's get to the second part of the interview with Ben Les. So Ben you are at a senior role at Netflix correct. That's right. That's right. And you were were you hired directly into that role? Yes, I was so actually most people at work at Netflix are senior level. So Netflix doesn't really hire junior level developers now I'll say that with a disclaimer maybe they started to now but generally what they do is they look for people who are already experts in their field. Like you mentioned in previous in the previous episode right right. So that's that's really interesting because I've I've listened to a lot of what you well let's let's save that part of this conversation for later. I first I want to talk to you a little bit about something that I think is is a hot topic. Well it is a hot topic if you go and like just look at hacker news I'm sure if you search for the word functional on that page you get like 30 30 instances on can you tell me just give me like the the tweet version of what functional programming is. And then we'll talk a little bit about some of the things that functional programming can address in I know that also well just let's just start there. Okay well the tweet version I don't not count the characters but I mean the real tweet version to be functional programming is programming a functions right but now functional programming is generally where you're trying to program using the scope or the closure of your functions and. And in the mutable state to pass data from from one step to the next to in your in your function so the real thing about functional program the reason it's so important is functional programming lend itself very well to concurrency and while we don't have real concurrency and JavaScript right now we are definitely going to have it in the future there's there's simply no way around it. You know as I had stated in my my other talk the. Chores are not getting any faster we're just getting more of them so. Now that we're building these larger and larger and larger browser applications with JavaScript JavaScript in order to scale these applications we're going to need to be able to utilize those other cores which means concurrency. It's a good idea to learn functional programming now and get used to the ideas behind it to prepare for that future because in paired a programming makes life very hard in a concurrent world. Sure yeah so let me go back on my previous statement it's impossible to talk about functional programming and a tweet length conversation because it's just a totally different paradigm it's quite literally if you were to search programming paradigms there are entire courses that teach you how to think in these different paradigms one being imperative and then the other one being functional those are the two big ones and then there's different versions of those of those paradigms. The interesting thing to me is that functional programming is not functional programming is not technically new right we've had a list for years and years but the reason why it's so important or why functional is seen as kind of the future trademark is because of this concurrency issue we only have you know eight to 16 cores on our computers depending on it. If you're a hyper threading who knows exactly how that works but I'm sure somebody listening to the show knows how hyper threading works I actually don't have no idea how hyper threading works anyway so we we only have you know limited number of cores so our JavaScript is is really fast for I mean for now right it's fast enough to do what we've been needing to do but the future as we said on the last show everything is changing and it's just a little bit more interesting. Everything is changing and it's changing very fast and functional programming addresses this concern because it helps you handle state and that that is really the way that I see kind of the major benefit if I had to package it in a very small amount of words I would say that functional is the reason why it works so well for this is because you handle state much better when you don't mute it. You don't mutate your data or you don't mutate what's in memory with multiple multiple different things coming around and changing whatever is in memory if it's if it's functional then you don't have that data isn't changing it's just you're creating new copies of that data and passing them around to those different processes right yeah theoretically if you're doing it properly you'll be using what's called a mutable state where you're just making a new copy for each function that's that's not going to be a new copy of that data. It's a new function that's that's doing something with that data the I mean for I guess for your listeners that don't quite understand you know what which you mean by mutating state that would be. So long a lines of let's say you have two threads or two processes doing something the same time and there there's one array let's say that that they're both attempting to add something to you're going to have to block one of those threads and let the other thread do its work in order to keep them from both trying to update that array at the same time that would be mutating the rate at the same time so that case the array would be mutable data. And that's that's not that's not ideal so. Generally it's better to allow us that just that some of that still happens with functional programming it's just that it's being generally it's being handled by a framework or it's being handled natively. And not by the author at the time and that's usually where you get into some sticky business so yeah it's functional programming is is very very important going forward yeah so I work with Ruby quite often and we see the the term thread safe which is thrown around as if it means concurrent and it's not the same thing. Thread safe just means that you're not going to see two things changing that stuff in memory at the same time you'll have you know mutex or something that locks up that other thread and keeps it from you know keeps basically keeps the threads from doing things that they shouldn't be doing and it's not an easy to problem to solve even with all those tools I watched a Ruby top as episode I don't know if you if you work with Ruby ever been a magic probably not. So there's a or. Avdagram is a member of the Ruby community and he created this site called Ruby top is and it has these videos that he spends like five or six minutes you know explaining a concept and he goes through a thread safe exercise essentially and it man it if you've never worked with that it is kind of a brain vendor. The first time you see it because we're used to thinking that way we're used to kind of relying on you know at most our parallel code working in parallel and then our when I say parallel i'm sorry I mean our synchronous code working synchronously and then at most like asynchronous in event oriented things where it comes back from whatever call it made and you can respond to it in a call back right but this is. Totally different from that because it doesn't even know about the other thread necessarily right so yeah that's I spent most of my career actually as a C sharp developer so in C sharp we did have threads we had semaphores and locking in all these these various mechanisms to deal with that. The one thing that people kind of skip over when it comes to like a functional style of doing things and versus you know the more imperative way in dealing with semaphores and mutex is in things of that nature to you is is you only have a finite number of threads on a machine so you have to. You have to watch you don't want to have blocked threads you don't want to have a lot of block threads where you have a lot of things sharing something that it's mutating so you know nine threads are locked waiting for one of the threads to do its job to this this object and then and then continue on that's why that's why things like note in the event loop. Work so well because it's it's all about you know non blocking IO and and that sort of thing but it does get more difficult when you're talking about taking that same event loop and you know getting concurrent or getting parallel i. I would not want to be the implementers when they when they have to finally tackle that issue with with JavaScript but it's like promises but not completely. Right yeah that'll be interesting I would imagine that a lot of a lot of discussion around you know how is it different from asynchronous is is bound to happen right. Go ahead. For sure that's the biggest the biggest thing about I what I suspect is going to happen is that they'll have things like say your maps and your filters and your four reaches as long as you're dealing with immutable data when you're when you're inside of those functions that those sorts of things could be parallelized. I'm just I'm just taking a stab but i'm going to guess with JavaScript they're probably going to lean on ergonomics more than more than that you know having the raw powerful things like mutex is in semifors to deal with. I mean it's all it's all fucking texture right now anyways because it's this as far as I know it's not happening right now but it it has to be discussed it has got something absolutely going to happen so sure and it'll probably happen in open source first I would imagine probably some someone's going to fork IO or fork node and then do something that will enable at this is over my head this is where my listeners this is where my programming knowledge comes to a halt I don't know how. I would not know how to go about implementing you know multi threading with JavaScript I would be scared to even if I did know how I would be scared to because you have people with pitchforks and. You did this wrong yeah I wouldn't want to be a part of that yeah so either way functional programming is is an important thing to learn because of this future and it also cleans up some other some other issues that you can run into with imperative programming where you're maintaining a lot of state and. Having to do a lot of mental math about when what is happening those those things get a little hard have you have you messed around with react at all. Now with reacts okay i've played with it but i can't say that i've written an app with it okay yeah so i've done you know i could probably say i played with that i haven't really done anything super big with it or anything like that but. That was the first moment where I had the epiphany of wow this is why just from you know from the development standpoint from being a happy developer at least employing some of these concepts of immutable data and you know flux if you want to use the Facebook term but the one way data flow. Why that can be a very powerful and empowering practice like it's not just about concurrency although it is that is a huge part of it but it's also about like this helps you control state. Right right it right now the pragmatic reason to deal with react or or even with reactive programming which isn't necessarily react per se but like libraries like bacon or most or arcs yes is is. For that reason of trying to control state more importantly trying to control side effect to when it comes to like reactive streams libraries in particular like arts jazz. So that's we use arcs yes heavily at netflix for the app that i'm on and and rx java is very heavily used in in my department arcs yes is very heavily used. I believe for the customer facing you I so that we're we're big proponents of reactive programming and Netflix sure and there's a lot of great reasons for that that I mean above and beyond the reasons I just gave you for functional programming. You know reactive programming with functions not functional reactive programming which is something different. So tell me what you mean because i'm sure there's quite a few people listening who this is the first time they're hearing about reactive tell me what you mean by a side effect because that's also talked about quite a bit in functional programming right so the basic thing of the side effect is is kind of the final result of of. You know whatever your whatever your program is doing so you would have some series of inputs and. With that series of inputs you're going to do something to them like say maybe input might be a button click or you know an Ajax call coming back and you're going to massage that into something that you're going to display in this screen. Or maybe trigger another event the active displaying something on the screen or triggering the other event that final result is the side effect and the active transfer transforming whatever inputs you have into something that you want to send to those outputs. That that that that's where you don't want to cause any side effects right sure. And example around this is some of the pain that people see with two way binding so two a binding and say something like angular or amber one of the one of the the weird things about it is you you're still you're taking bits of data and steps say with computed properties and number for example where you say. Properties computed from these other two properties which are computed from these other two properties and so on and so forth so when you update a property it goes to this chain and transforms all the data but the difference between that and say reactive stream processing is that it each step. Those properties are broadcasting to everybody hey i've changed so if anybody else needs me for anything you can you can do something with my new value and they say nothing about the intention of why it's changed or we know what it was changed in the process of doing what or why. And so you you can incur side effects right so it's it's whereas when you're doing a reactive stream programming what you do is you you have a stream an observable stream of events coming and over time and you combine them emerge them however you need to and then you map them and maybe you call some other events and you like another Ajax call you get the result back and it's still part of your stream. And at the very end you have a do or a for each where you're saying here this is where I want to express to the outside world what i've done and you know right to the DOM or. You know make a call over a service finally this is this is the landing place of my transform data. Very interesting yeah it's it's really good because it allows you to control the flow of your application very easily and very expressively. Whereas you know I think we all work on very complicated applications where you kind of you just get lost in the code and it's not very easy to see the path that something's taking through through your application. Yeah yeah and so that's that's implemented on you say you keep on mentioning streams i'm assuming that what that is is like a like a socket or something like that. So what I'm specifically talking about generally is is observe are observables which are they have this concept in rx jess or bacon jess or most jess but. There they are streams data and observable is a stream of values so any collection of values emitted over any amount of time so there could be instantly it could be once every so often what they are you're familiar with the iterator i'm assuming. So that's so an iterator is any collection of values and you technically get them over time but you get them over time every time you say next give me the next one so you're telling it to give you the next one and if it has something it gives it to you. Yeah an observable is the same thing but instead of you having to tell it to give it to you it just pushes it at you with an event. So you can take any collection of things over any amount of time and you can start listening to it and here however many you need to hear or receive however many you need to receive and then stop listening to it so the observable also embodies the setup and tear down of that underlying data stream could be a socket or an event listener or something like that. So in other words it doesn't matter right it doesn't matter so the the one thing about reactive programming is you could have some widget that expects some input and I can give you this observable that says here's your inputs and you can just use it now you have no idea where that came from I could be getting that that observable stream from voice to text I could be getting it from joystick input I could be getting it just from an array. It doesn't really matter all all you know is you you've got this the stream of values right so you've been coupled from whatever my implementation is that is giving you those values. Wow that's super powerful yeah it is it's the neatest thing about it too is that because it's a collection you can do all of the same fun low dash type things that you could do with an array right yeah so or even an interrater so you can merge them you can zip them you can. You know do all of these really map produce whatever right very very expressive very expressive way to write your code yeah that's so that's in you said that's what what library implements that right now Rxjs is the library that I use. It's it's it's pretty robust it's got a lot of it's got a bit more a few more features I think then some of its contemporaries out there right now but the other ones are really solid to sure most jazz is a is a promise based implementation the same thing. And then bacon jazz it's just another tech on reactive streams programming they're all really powerful I can't say enough things about Rxjs though I really like it is this a concept that you think will eventually become like a native JavaScript thing or is it more opinionated than that it's it's funny you should ask that this is exactly what this is going to be so there's a there's a gentleman that I work with named jafar Hussein who you might have seen it and you come talking about Falcore. He has an async generator proposal for yes seven it's got some traction I think it's in my opinion it's going to be implemented at least it should be implemented and it's very similar to Rxjs right now there's actually an effort around Rxjs that I'm a part of to create Rxjs three as yes six modules and have it adhere to the same. The same sort of interface that that jafar had come up with for his async generator proposal so the idea will be once it's native that if you create an async generator so that's just a generator function that has the async keyword in front of it will return to you and observable that will be no different than the server will have just been discussing. Oh wow so yeah it's it's a very very very powerful way to develop applications it takes some there is a middle jump I mean there is a middle jump that you have to make to start doing real reactive programming. As opposed to the imperative programming that we're used to but it's it's definitely worth learning in my opinion because it's it's a way to embody embody all of these asynchronous problems that you see in the web whether it's animations or Ajax or you know Web sockets or mouse input you name it it applies with observables and reactive stream processing so yeah. That's incredible and so like I've seen so when I started learning react which just again to clarify is not specifically reactive programming but when I started learning react it was interesting to me that it basically and this is true for angular and for amber these are meant to be like the full application is very hard to adopt any one of these things for like a widget on your site you you have to kind of adopt these for a much larger scale of thinking to really get the benefits out of them from what I've from what I've researched now I'm sure that it is possible to just consider it like okay well you know the widget is like a little screen and you could ship a reactive widget and that's the only thing that uses reactive programming but I think that kind of misses the point of these which is to shift the entire mindset of a given project over to one of these methodologies. Yeah that's that's true now I will say that I have seen react used in angular projects and react used in amber project. It's I don't I don't know if it's a best practice to do that or not but I have seen that happen now though I don't think you'd see it the other way around because of the way that react in particular structured but yeah it's the the mental shift with reactive programming is I mean here's a here's a good example so this is the same example that I brought up on the screen at ng comp which is you know let's say that you have two two variables a and b and when one of them changes via some event you don't know what event could be causing but I presume that you know what event causes it but either way when one of them changes via some event you need to produce a new variable where you add them together and you create variable C and then you do something to see you display it or whatever so every time a or b changes you have to add them together stick them and see and then do something to see so now whatever you're doing with C is actually tightly coupled to a and b but it has to have knowledge of a and b. So reactive programming the reactive way to do this would be to create a stream of a's and a stream of b's so you would take the events that are changing a and you would create a stream of a and you can use it however you wanted and you do the same thing with with b whatever events changing b you create a stream of b values and then you can do a combined latest on them which just says hey give me the last value I got from either one of these streams. I want to add them together and emit a value for C so every time a pumps out of value it takes the last value I got from b adds it to a and gives you a value and the C stream so now you have the stream of C values and you can you can you can subscribe to that and you can do whatever it is you were going to do with C display to the screen or whatever but the interesting thing is that code is now completely decoupled from your knowledge of what you're doing. So you know that knowledge of where that stream of C's came from right yeah it has no idea that it came from a or b so you could substitute in anything else you could merge in another stream of alternate C values if you if you so chose and it would not it would not break your code sure so that's that's what it's sort of a high level idea behind reactive programming. So it gives an idea of what a stream is fundamentally which is a changing value over time like you said any collection of values really over time right right awesome I'm going to take a quick break for a sponsor and then we're going to talk about a different topic altogether with Ben we're going to talk a little bit about imposter syndrome and an interesting thing that Ben does when he talks and even when he talked to me when I asked him to be on the screen. So I'm going to talk about that when I asked him to be on the show. Today's episode of Developer Tea is sponsored by digital ocean digital ocean is simple cloud hosting built for Developer They're dedicated offering the most intuitive and easy way to spin up a cloud server and in just 55 seconds you can deploy a solid state drive cloud server plan start at only $5 per month for 512 megabytes of RAM a 20 gigabyte solid state drive once CPU and a full terabyte of transfer. In addition to offering simple and affordable SSD cloud hosting digital ocean is dedicated to building out a strong community and supports open source software they offer a vast collection of hosting tutorials and invite Developer To submit articles and they pay $50 per published piece deploy your SSD cloud server with digital ocean today by going to digital ocean. Now digital ocean has been kind enough to provide Developer Tealisteners a discount of $10 when you use the code Developer Teaso go to digital ocean dot com and use the code Developer Teato get $10 off today and you'll get up and running with your own SSD cloud server and just 55 seconds that digital ocean dot com. Are you familiar with the concept of I'm assuming you are but if not in posture syndrome yeah I'm posture syndrome my my understanding is that it's the feeling that you don't quite belong in the shoes are standing in so you maybe don't you don't belong in this room of super smart people or maybe you don't know everything that you thought that you knew or that sort of thing it's a confidence problem. Right and so I've experienced this and I think a lot of a lot of other developers have as well and I think some of it is driven by this perspective of you know the developer that I should be which is which is just a fake like perception that I have in my mind of what I kind of aspire to be in the the difference between the developer that I am currently and that developer. And I we often drive this based on you know the really smart Developer That we see online somehow sharing their knowledge in one way or another whether that's through an open source project or like been speaking at in G comp but been said something at in G comp that was really interesting to me and it's it speaks to this idea of imposter syndrome in the opening of his talk. Ben you said that you were happy to be wrong that you were you were more happy to be wrong because that meant that it was an opportunity for learning can you talk a little bit about about what that means to you about what being wrong actually is because again let me remind everyone Ben is working at a perhaps one of the most important tech companies in our world today they they process it. A ton of data and he's he's leading well he's in a senior position at that company and he's openly saying that he very often is wrong can you explain the value of being wrong sometimes sure well the value of the value of being wrong is is plainly put it gives you an opportunity to learn why you're wrong right so. You know you walk away from being wrong a smarter person i'm always i'm always always happy when I post something on like a blog or I post on the internet and somebody comes and corrects me you know sometimes it happens in a very argumentative adversarial way but you know that that doesn't bother me much i'm i'm only the only thing with bother me is if somebody told me I was wrong and didn't tell me why. I think well great now I now I have to go figure out why I'm wrong in my own and that's that's a lot less fun than you just telling me why I'm wrong but it's worth verifying all claims anyways yeah it's it's it's something that i have learned over the course of my career so one thing a lot of people don't know about me is I am entirely self taught when it comes to programming so in early days of my career there is quite frequent that i would run into things that you know almost every developer knows some term and i have no idea what it is. And i will i will say that back then. It did made me feel stupid but then i realized well but now i know right so i felt stupid for a second but it's not a big deal because now i know and i'll never feel stupid about that again right right. So that that was that was kind of the origins of of my thoughts around this and it's the other thing is if you're if you're afraid of being wrong which i think everybody is. But if you're actively afraid of being wrong or being corrected publicly then you're going to be afraid to speak up publicly and it's it's going to really it's going to really hinder you. I have a few advantages over you know when you say in imposter syndrome there's there's a whole swath of marginalized people in tech right now that i have to deal with this this particular issue in ways that i can't imagine every day. Because you're going to work you literally don't look like anybody else where you work right there's there's i can't i can't imagine what that's like but. It is important you know when you when you encounter fears at least the fear of being wrong or the fear of saying something and sounding stupid. So that that you realize that the worst that can happen is that you're corrected and if if you if you take that correction well generally speaking all that really happens from that is you walk away having learned something. So and what I mean by taking it well is you know not being offended by you know someone correcting you that you were technically wrong on something. Yeah and I think the emotional side of things is is one that we often ignore or try to push down and and deal with it in the wrong kinds of ways like for instance let it affect the way that we're feeling in the future about that same thing like very often if you if you feel like you were wrong about something and somebody decides to correct you. Even if their tone is slightly off or condescending it's easy to say oh wow that is who I am that's my identity of whatever that person just pointed out of me that i'm weak at. And now i'm just going to spiral downward and it's because of some objective part of who I am rather than wait a minute i'm just feeling the emotion of rejection or i'm just feeling the emotion that comes along with like a shame. A moment of shame where you did something wrong and I think we can mitigate a lot of that if we go ahead and know number one the best of the best like people like been i'm going to go ahead and call you one of the best of the best because I can. They still make mistakes and happily make mistakes and you're going to make mistakes forever like there's no there's not going to be a day probably unless maybe you have a fluke day but there's not going to be a day in your life where you don't make some mistake in some aspect whether that's on a relationship level or if it's on an actual just you're writing bugs in your code there are mistakes that are going to litter your life that's just the. That's just the truth of life and so what Ben is saying and what i'm also agreeing with Ben and saying is make the best of those mistakes and and turn them around to be good things and learn from them. Right, absolutely, absolutely, the other thing to understand to about you know the idea of the developer that you think you want to be this this this mythical person you like to become is you know i work with a lot of extraordinarily intelligent people and i have to say that from time to time i feel i'm intimidated by some of the book that i work with because they are so smart and they're so brilliant. But the truth is that all of these people and we've actually sat around and discussed this recently in the office every every single one of them. So we'll for the most part admit that on a daily basis you know they find themselves utterly confused by something they're doing or you know just kind of it's it's this this this thing with developers were really what you do as a developer is you you break code like 99% of the time and the 1% that it works you committed right. That's that's the job essentially and and everybody's everybody's got to learn something new all of these people that I work with would not be as smart as they are if they weren't willing to risk being wrong or you know risk admitting that they didn't know something because they have to go out and learn it. So yeah it's any any developer that they will not admit that they don't know what they're doing at least one time a day is probably an arse assist they're probably just more to it because they they have a to i have an opinion of themselves. Just like just like any developer who says that there's nothing wrong with a given framework right you mentioned that in gconf I thought that was so pointed. So you've given so much so much advice in this interview but I'm going to ask you for a little bit more this this kind of ends out ends out the interview and I usually do this with guests so. It's kind of interesting to see how much advice can come out of these conversations and and i'm really thankful that you've been on the show if you had 30 seconds to sit down with a beginner developer or a experience developer well into the career and give them advice from your experience what would you say to them. Well I think I would I think I would tell them to just just approach software development with with a passion I mean don't don't do it because it's a paycheck to it because you actually enjoy it if you're working on something at work that is particularly boring then work on something on a side that's not or or figure out a way to make it not boring but most importantly it's learn something new every day if you find something you don't know or something. That's confusing to you that's perfect that's what you're looking for you want to go find the thing that's confusing to you and then make it not confusing like learn learn why it's confusing you and learn about it and then you'll walk away a better person for it so that that would be my advice is always always learn new things. It's great advice thank you so much for being on the show then. Thank you thanks for having me. And thank you for listening to Developer Tea. I really appreciate your time Ben has been awesome I hope that you enjoyed this interview as much as I enjoyed interviewing Ben absolutely fantastic. If you haven't subscribed make sure you subscribe and also if you would like to support the show I know that not everybody can do this there just a few people who are able to support the show financially but if you can go to Developer Tea. If you're on the front slash donate that is a huge help. However if you can't support us financially if you leave a review that actually is a huge help as well just go to iTunes and leave a quick review for Developer Tea. That is a huge help. Thank you for listening and until next time enjoy your tea.