What is it like being the lead developer on a large open source project or a lead advisor at least on a large open source project? Oh, we've picked kind of the biggest one for today's episode. We're interviewing the lead for security of WordPress, Aaron Campbell. My name is Jonathan Cutrell. We're listening to Developer Tea. My goal on this show is to provide you with the information and the inspiration that you need, the coaching that you need as a developer to become the great developer that I know that you want to be. Today we're talking about security. We're talking about WordPress. We're talking about open source. Tons of fantastic information coming from Aaron. We did this interview live at Squares, not live in front of an audience, but we did it in person at Squares Conference in Great Vine, Texas just a few weeks ago. If you haven't checked out Squares, I recommend that you check it out when it's coming up next year around the same time in April of 2018, most likely. Let's get straight into this interview with Aaron Campbell. Summit Squares with Aaron Campbell. Aaron was kind enough to stay after one of the panels, I'm sorry, not a panel workshop with me here, and do a quick interview. I'm really excited to talk. We have dinner coming up after this, so it was very nice of him to stop and hang out with me. Aaron, you are the reading this directly from your GitHub right in front of you, but the core security team lead at WordPress. But WordPress isn't like a place. How is this possible? It's not a company. How is this possible? Tell us a little bit about what you do there. Yeah, I guess that that's an interesting thing to hold the position in an organization that doesn't have a company behind it. But I'm actually funded by GoDaddy, so I'm employed by GoDaddy, but work full time as the WordPress core security team lead. So they donate my time fully to WordPress to help pushing ahead that project and security is the area that I do that in. Very interesting. What is a typical day as a WordPress security expert, the lead of this team? What does a typical day look like for you? You know, that varies a lot, but a lot of what I do is processing incoming security reports that we get either through our security at WordPress.org, email address, or more recently through our hacker one profile that we just launched, triaging them, finding out what actually needs attention, and then getting the right person's attention to that. Because we have a whole team. There's about 50 of us on the security team. But everybody has their own specialties, and so I triage and push stuff off to the right people. This is what I find so interesting about your position. For developers, we're talking about the security of WordPress, which is notoriously in the past has been attacked by people. We're talking about a company that a lot of people for better or for worse have had poor experiences within the past. So you're in this really contentious position of the donated time open source developer for GoDaddy, working on security for WordPress. I find that such an interesting position. So I really want to dive into, so forward that day to day, are you actually working with the core code, or are you mostly trying to come up with ways, are you solving actual, big direct problems, or are you directing teams? How's that actually working? So since I took over the security team, I've actually spent less time in the WordPress code, making changes and whatnot. But I still spend quite a bit of time around it every time that the reports come in. You have to go through and test them, you can't just assume that something is valid or that it's not, you have to go through and try to reproduce based on their steps. A lot of times that involves getting into the code and looking through and seeing where the various security steps are taken to make sure that there wasn't actually some way around that that you hadn't thought of before those kinds of things. But mostly it's interacting with people and sort of running the team and sometimes running the security releases as well. Very interesting. There's a couple of things that I want to ask you more specifically about. One is the hacker one profile. We'll talk about that in a second. The other thing that I want to ask you, and maybe we can leave this to the more towards the end, but what are some of the most common mistakes that WordPress developer, or someone who is relatively uneducated about web development, they're getting into web development, they start building a WordPress site. As a web developer, I know a lot of people have attacked WordPress, but usually in my experience at least it's been something that is totally unrelated to core and it's usually like a plug-in or maybe the default settings on a given server stack that weren't set up properly or whatever it is is out of date software. I'd love to talk about that in a minute, but first tell me a little bit about hacker one. Yeah, so hacker ones in an interesting tool that we've really just finished moving to. It's a place for security researchers. A lot of people get a little confused around the term hacker, which in this case it's definitely talking about people that are penetration testing, people that are still reporting responsibly rather than just making their findings public or exploiting them. But it's a system that lots of companies and projects are starting to use that lets those security researchers report the vulnerabilities to you in a safe and secure way, lets you interact with them throughout the process, talking back and forth, even sending them code to test to make sure that it blocks all of the things that they had found, those kinds of things. And it also allows us to have a bounty program, which lets us pay not massive amounts, but pay rewards to people for disclosing these things responsibly to us because the security of WordPress is really important to us specifically, but to lots and lots of people. So we want to make sure that we have a system in place to sort of reward that good behavior. Sure. Yeah, and I'm looking at it now. I can't believe I had no percept. This is so outside of my normal day to day work, but hacker one is actually, it is a platform, but you've created a profile on hacker one. Right. So now at hackerone.com slash WordPress, you can report security vulnerabilities about all of our properties, which include WordPress and BudiPress and BB Press and WPCLI. It also includes the actual WordPress.org website, the word camp websites, all of our properties. Very cool. And it looks like people have already been rewarded and there's been some people who are resolving reports as long as a couple months ago. Yeah. But this is just recently kind of becoming a public thing, right? We ran it as a private profile for a little while, inviting people who were already reporting issues to us, inviting them to that, testing it, getting it set up, working through the kinks, making sure that as we switched to it, that we were prepared to handle all of the reports through there, which was, it was really good that we did that because when we did go live with it, we got a... Very large volume of reports very quickly. And so having already gotten ourselves used to the process there and stuff was really important so that we didn't end up drowning right after, right after going public with it. Yeah. I interviewed Andrew Norcross about some of the work that he's done on WordPress and with the agency work that he's done in the past. We discussed in pretty long detail some of these same questions as a developer coming from a long WordPress background, I have maybe over the top tendency to defend it. But I'd love to know this stigma is unfounded, generally speaking. There have been problems, but a lot of that, at least from what I understand, a lot of it can be attributed to the fact that WordPress is quite honestly just very popular. And so you're going to have a lot more attention on WordPress and a lot more of a reason for it to be hacked, right? If it's a much smaller thing than it's not going to be hacked as often. At this point, WordPress powers a little over 27% of the internet, which means that we have a big target on our back. If you're trying to compromise sites, especially by volume, which is a very common thing now, scripted attacks that attempt to compromise many, many sites, even if they're small, just to put up some ads and make a couple bucks off each one. WordPress is definitely the thing you want to target to do that because more than one in every four websites that you find is going to be running WordPress. And many of those are running out of date versions and those kinds of things. And so, yeah, we definitely have more of a target on our back than pretty much any other CMS. And so we've been very targeted, but at the same time, I think that we run a really good team around security and we've really done a lot all along to keep WordPress itself secure. Something that you mentioned was, could it be an issue outside of core plugins, themes, those kinds of things? And that's extremely common. Unfortunately, no one uses WordPress just by itself. The wordpress that they use is some combination of WordPress and a theme and plugins likely by third party developers or companies. But when something happens, when there's some sort of security issue, to them, it's just WordPress. And we were recognizing that and we're actually as a security team trying to push out a little bit more and our plan is over the next year or two to start bringing some of the bigger, more popular plugins in under our umbrella so that our security team can basically help take care of the security of some of those plugins and try to slowly expand that out so that even though those aren't ours, we're still doing everything that we can to keep our users secure. Sure. Yeah, it's an ecosystem, right? When you have a platform like WordPress that is so highly dependent or not dependent necessarily, but so highly codependent with these other plugins, the security of, for example, advanced custom fields has a massive impact on the security of WordPress or at least, practically speaking, the sites that are running WordPress, you're going to have a larger impact on security for the overall number. Exactly. And there are a lot of big plugins like that as you look at Jetpack and some of these that are just, they have a huge percentage of users that are using them and we want to make sure that all of those users are secure as well. And it's an interesting take to try to push out from just securing this one chunk of software that we really have kind of total control over to trying to help secure other software that we have a little bit less control over, but it's not completely unprecedented. We've worked with a lot of plugin authors in the past. We're just trying to take a more active role in that. And potentially, we may even start adding some of those to our hacker one as something that you can report directly to us. Sure. Yeah. And because if you think about it, there are, if you're a WordPress developer listening to this, to this episode right now, you can probably think of two or three plugins that like almost before you get started, you go ahead and throw those in there, right? I know that the multiple thumbnails, very, very simple example. There's nothing really, there's not like a huge business behind multiple thumbnail plugins, right? Necessarily, but that plugin is so commonly, or so commonly used. And it's difficult because a small team or even a single person can develop an extremely useful plugin that many, many people want to use, but they may not have the security expertise or a team big enough to have a security expert on it. And we have a team of 50 volunteers that all have expertise in various security areas. And I feel like if we can use that team to help those developers, ultimately, our goal of keeping our users secure is better served that way. We'll be right back to talk more about WordPress and about client updating and security and multiple systems working together in just a few minutes. But first, I want to talk about today's fantastic sponsor. Thank you to Dolby. Dolby is today's sponsor. Today's users, they want better audio. 90% of digital device users rank sound quality as important across the digital entertainment ecosystem. This means not only when they're sitting in their living rooms, but also when they have their headphones on and they're playing a game or they're watching a video on their mobile device as well. Improving your app or services audio doesn't mean you need better audio assets necessarily. You probably need a better audio codec to make sure your users hear everything. Now, here's the amazing thing. This doesn't just go for the living room. It doesn't just go for native apps. It also goes for your browser. And Dolby is supporting your browser today. Asset encoding is easily accomplished with the tools you already use, for example Adobe Audition or you can also use Dolby developers free online encoding utility if you're not used to, if you're not the one who's editing the audio, for example, the new iPhone 7, for example, supports these new codecs that Dolby is developing. So if you don't want your users to miss out on this incredible new audio technology, that Dolby is providing, go and check it out, spec.fm slash Dolby. This is Dolby for the browser. It's normal website browsing. The things that you are uploading to your website, the video and the audio that you're uploading to your, even your WordPress sites can take advantage of these new codecs. You can go and check it out, spec.fm slash Dolby. Thank you again to Dolby for sponsoring today's episode of Developer Tea. We work with tons of clients who use WordPress and so many problems that they face in retainer phase after we've developed the initial project. So many of those problems are because they updated something or they didn't update something. And keeping those things in sync and saying, hey, locking this version down is really important. Or always updating this particular thing is really important. Always keeping up to date to the latest version of Apache, but not major version upgrades. For someone who doesn't understand some of that stuff, that can be a little bit daunting. And it's really interesting to know that security is the other side of the coin for quality. We have actual functionality on the one hand, but on the other hand, if your site is insecure and your content gets replaced with a bunch of ads that you probably don't want on your site, then it's equally bad. So it is extremely important for WordPress Developer To be cognizant of this kind of stuff. As a WordPress developer, thank you to the team and also to GoDaddy for sponsoring WordPress. GoDaddy is not sponsoring this podcast, by the way. It can be very clear with that, but they are certainly sponsoring the effort for security on WordPress. Yeah, I mean, GoDaddy's been doing a fantastic job of really pushing forward some of these open source projects like WordPress that so many people are using and their sponsorship of WordPress has been fantastic. And obviously, I'm thankful for it as well. I've been working with WordPress, being involved in core for 10 years now, I guess. But that was always volunteer time in addition to the work that had to pay the bills. And so GoDaddy kind of gave me this opportunity to take what I was already doing, what was already important to me and expand that out and do it full time. I feel like it's been able, I've been able to help make a pretty big difference because of that. That's great. It's fantastic. So I'm going to switch gears a little bit and we're going to do a list like every good article on Buzzfeed has. If you could list off maybe three or four things that as a developer specifically, I can do or that I very likely and the mistakes I would make as it relates to WordPress security, what are those top three or four things that you see? So I like to try to do kind of two different lists here. There's a little bit that's very specific to developers, which I know is the majority of the people that are listening, which that's getting comfortable with the WordPress-specific sanitization and escaping functions, making sure to use those at the proper times, making sure that you're escaping data just before it's displayed, making sure that you're sanitizing before it's stored and making sure that you're checking capabilities everywhere that you ought to be not just assuming that a user has the capabilities that they have. So those are three big, like basic things that a developer should always look for. But the thing that I've found is that even as developers, and I've been doing web development for 16 years now, and even worse, I've been doing web development for 16 years now. Even working in security all the time, we often fall prey to the same really basic security issues that every regular user falls to, which is that we get password. We get lazy with passwords. We have so many of them. If you have any passwords that are not in your password manager, you're doing it wrong, right? You can't have long, random, unique passwords without using a password manager. Are you turning on two factor everywhere and adding that to your WordPress sites? It's really easy to add, and it makes such a massive difference. And are you keeping everything up to date, even as developers, we often, like you mentioned earlier, we think that we want to lock this to a specific version because it's what we've tested and what works. And that's great as long as you are making sure to test and deploy updates quickly. But otherwise, there's a reason that WordPress automatically updates for security releases because it makes a huge difference. We are constantly putting in roadblocks to stop the bad guys from getting in your sites. And if you're not making use of that, you're really tying our hands as a security team. Sure. Yeah. Because if you're using three versions ago, WordPress, that's like the beta release or something like that. Exactly. You're probably going to get into some trouble. Exactly. I'm going to add to that specific point and say, understanding the difference between a major release or a feature release and a security release. And the WordPress versioning system isn't quite the same as everyone else's. That last digit, so 4.7.4, that second four. Anytime that number is changing, go ahead and update the other two numbers, the first two, those are our major version numbers. Those will have new features as well as security. The potential deprecation. Exactly. We're really focused on backwards compatibility, so deprecations are really rare in WordPress, but that is where they could potentially happen. Sure. Yeah. And it is pretty incredible if you update WordPress on the average case, you hardly ever are seeing serious problematic deprecations at least. I have plugins that were written for 10 major versions of WordPress ago and haven't needed any changes or tweaks because everything just continues to work. Yeah, which is pretty incredible. And some of those plugins that everybody uses, they haven't been touched in years because so much of that underlying API for the developer, the private API is still basically what it was back then or at least it's backfilled. So you have exit and that's really what it is. It's not necessarily that those things haven't changed, but we've been very careful to make sure that we have that sort of compatibility layer so that even if it has changed and you've got, you know, you can make use of cool new stuff, the things that you were using before still work. Yeah. Yeah. So I have a couple more questions to ask you and then we'll wrap this thing up. The questions that I like to ask every guest that actually comes on the show. The first one is if you could have people talk to you about one thing more often, what do you wish more people would ask you about? You know, it doesn't have to be development related by the way. No, but it is actually. And I think that so I'm in this security position, but it's because it's something that I'm passionate about. And I think that I guess give me a minute to sort of set that stage. The web is no longer something that's fun or a novelty, right? It started out that way. But now it is a basis that we build large parts of our lives and our livelihoods on. It is something that has become a basic sort of utility like running water and electricity, right? So the future of that I think is extremely important to our future. They're very much tied together and therefore the security of that that makes sure that future exists safely is important. So I really wish that more people were asking me about the simple things, those things that I was talking about just a little bit ago, you know, better passwords and two-factor authentication, keeping things up to date. I wish people were asking me about what those simple things that they could do that are big wins, security wise, but are simple and easy to do. I sort of try to throw those out to people all the time anyway, but I wish that people were more interested and asked about it more often. Well, it's the gravity of the situation that people typically don't understand because it is. It's, you know, especially for our generation we grew up without internet and then it came along and it feels very much like almost like a toy, right? Even still, there are parts of it that feel not serious to me. And stop and think about the fact that there will never be another generation that does that. The internet is now available from birth on to everyone. And it's very similar to teaching your children how to lock the door, right? Teaching them about a smart password. They could have even more detrimental circumstances than if somebody were to come and rob their house if they get their identity stolen on the island or something like that. Very important. So a piece of homework then maybe would be to go and do a quick audit, even of your five most important passwords. But certainly the ones that, if you're reusing passwords on multiple sites, that's a very common mistake that people make. They forget about a password or especially for example your email, somebody gets access to your email then there are still a lot of services that they can go and reset your password on and they'll get the email because now they can read your email and then they'll be able to click and reset them. So this is why things like two factor authentication are so important. Exactly. But a very simple piece of homework would be to go and invest in whatever password manager. Do you have a specific one you like? I like last pass. A lot of people that are heavy into the sort of Apple ecosystem like one password. I definitely don't care near as much about which one you use or that you use one. Yeah. And we use one password because we have, it's like there's some team account functionality and then you can share the vaults and that kind of stuff. And last pass has that too. Yep. But there you go. So it doesn't really matter. The point is longer than eight characters or longer than 12 characters or however long you want to make it, certainly longer than eight is kind of like the minimum standard. Yeah. And once you start using a password manager, length won't matter anymore which is what's so amazing. And when I tell people that almost all my passwords except where it's not allowed, basically all my passwords are 50 random characters. People like their mind gets blown but the truth is it doesn't matter. Once you start using a password manager, length is no longer an issue and so you might as well make it long. Yeah. It's very similar to the concept of the touch ID, right? Because if you were to actually look at the generated data behind that touch ID, it would be so long that it would be impossible to replicate. Right. Same concept. Very cool. So if you meet Aaron at a WordPress camp or something like that, am I saying word camp? If you meet Aaron at something like that, then ask him about more about security and about the simple things you can do to make your stuff more secure, your site and also your personal accounts, your Facebook, whatever your services you use. Absolutely. The second question I like to ask every guest that comes on the show is if you had 30 seconds of advice, which I feel like maybe we've covered this, but if you had 30 seconds of advice to give to every developer, what would you say? So that one's a little bit tougher. I mean, I guess that the biggest thing is no matter what cool, neat thing you're developing, consider the security of it every step of the way. It's so easy to be so focused on developing this cool thing that you deal with security at the end. Like you finish it and then you try to make it secure. But the best way to do it is to think about it every step along the way and it will not only end up with a more secure product, but it will be easier to really integrate the security as you go. Yeah, it's very important. I agree with this very similar to the way that we used to think about mobile, right? You do your whole site and you go back and make it mobile friendly, right? So let's do security first and then mobile first or mobile second, I guess. Sure. Or just do it all, you know, the full the agile methodology way, which is do the whole stack all the way through to a sufficient quality. If you're in, if your end goal is to have something that is secure and good on mobile and good on desktop, you should really be thinking about all of those the whole way through. Sure. And just build something smaller that meets all of those goals to begin with. Yeah. We say it on the show all the time, build smaller things and compose the smaller things together. And eventually you realize the value of small allows you to focus on the quality of every piece of it rather than taking on all that debt. It is technical debt. Yeah. If you have to go back and re implement security, go back and re implement mobile, go back and re implement, you know, whatever things that you want to, you're going to find that you have far more refactoring and far more work to do than you ever imagined that you would have to do in those spaces. So it is absolutely essential to be thinking about this things from day one. Absolutely. Aaron, thank you so much for the time on the show. Yeah. Thanks for having me. And I will definitely see you around the conference. All right. Thank you so much for listening to today's episode of Developer Tea. Thank you. Huge thank you to Aaron Campbell for coming onto the show and sharing his experience and his knowledge with us. Thank you for listening. Thank you to Dolby for sponsoring today's episode of Developer Tea. Remember good audio assets are not only for the living room or for the theater, they're also for your applications and for the browser. Go and check out what Dolby is offering and spec'd out of him slash Dolby. Thank you for listening. I hope you will subscribe so you don't miss out on future episodes. We released three episodes a week. So it's easy to miss if you're not subscribed. Go ahead and subscribe. And whenever podcasting app you use and until next time, enjoy your tea.