Natalia Tepluhina (GitLab)
As today’s guest, Natalia Tepluhina walks us through her professional history and describes her experiences at GitLab. After getting to know Natalia a little better, we talk about the challenges she has faced as a staff engineer. We then get into what makes GitLab a documentation-first company. Natalia goes on to explain the inherent traits of being a documentation-first company, like practicing open-sourced work and documenting every step of your knowledge journey. As our conversation with Natalia evolves, she describes her day-to-day responsibilities as a staff engineer and touches on the processes she implemented for her team to achieve success. From here, we take a deep dive into GitLab’s organizational structure and discover the differences between managers and technical leaders, finding out why Natalia prefers the former. Later in the show, we talk to Natalia about what goes into being a technical leader and hear more about her experiences collaborating with other divisions internally and as a GitLab consultant. To draw the episode to a close, we ask Natalia what it takes to surpass the staff level and for her to share her most influential resources. Find out what these are and much, much more by joining us today!
Links
Listen#
Transcript#
Note: This transcript was generated using automated transcription and may contain errors.
David: Welcome to the Staff Eng podcast where we interview software engineers who have progressed beyond the career level into staff levels and beyond. We’re interested in the areas of work that set staff plus level engineers apart from other individual contributors. Things like setting technical direction, mentorship and sponsorship, providing engineering perspective to the org, etc. My name is David Noel Romas and I’m joined by my co host, Alex Kessinger. Alex, please tell us a bit about today’s guest.
Alex: Yeah, this is a great interview with Natalia Teplohina. She works at GitLab as a staff engineer on the front end. Fun fact. She was also the first front end staff engineer at GitLab. In addition to her work at GitLab, she’s also a core committer on the Vue JS project. This interview is a lot of fun to do and I think you all enjoy listening to it, so let’s get into it.
David: Well, Natalia, thank you again so much for taking the time. Maybe we could start by having you introduce yourself. If you could tell us where you work and sort of what your role is and what has led you to that point.
Natalia: I work currently as a staff frontend engineer at GitLab and as a side activity I’m also a core member of UGS Framework.
David: How did you get to staff? Were you leveled into the role at GitLab or did you start that way?
Natalia: I was hired as a senior and after a year and a few months in being a Senior Engineer at GitLab, I was promoted to the staff and for GitLab it was first staff engineer in front end because previously we had a very nicely developed career stages for backend engineers through senior staff, distinguished and fellow. But our front end engineers were stopped at the senior and there was no possibility to be promoted to staff. I was kind of paving the way for front end engineers.
Alex: That’s interesting. Was there a specific project or something that you felt was like your staff level project?
Natalia: I think so, because why didn’t we have staff engineers before? Because there was a strong belief that front end side is not complicated enough to have staff engineers. Why would you promote someone to staff if it’s something simple, if it’s something basic to work on? And my main work after I joined was about introducing GraphQL and Apollo to our front end site. I was actually building the first front end application fully relying on GraphQL API and Apollo client on the front end and we discovered it to be complicated enough. So I was responsible also to share this knowledge, write the documentation and level the whole front end team to be knowledgeable about Apollo client and state management with it. So I think it was evaluated as a staff level effort.
David: What were some of the challenges around you mentioned like leveling the team up around sort of the Apollo stuff and GraphQL in general. What’s the process around that for you?
Natalia: Why it’s challenging? First of all, because Apollo client itself is not only a GraphQL client, it’s not something that we can say is Axios for the rest API. Apollo also offers you caching and caching the data on the front end side can replace any state manager you’re using on your front end, be it Redux for React or Vuex because we’re using Vue js. And aside of this, when you use Apollo client you kind of change the architecture of your components as well because you are querying data in components and this is very complicated for people who got used to I just dispatch an action and state manager is doing this for myself. It’s denied to have a query in the component and teaching them to this paradigm of Apollo cache in general. Also it was complicated because it relies on observables and people who never worked with observables previously were kind of surprised of how it works. How much magic is baked inside this? And GitLab is documentation first company. What do we mean by this? If you own a piece of knowledge, you are responsible for documenting it and making it available for the whole team. We have front end development guidelines, backend development guidelines, they are open source so anyone can check them and see what we’re doing and how. And part of being a senior or staff engineer is is also documenting everything you’re doing. So you discovered how to test Apollo on the front end with Vue js Because this is non trivial task as well, you need to write the whole documentation detailed and explain things to your team as well.
David: Was there a process for advocating for the use of Apollo client or any other sort of technologies that you primarily sort of, that you introduced to GitLab?
Natalia: This is a great question by the way, because when I just joined we started with GraphQL on the backend so there was an API but front end part was almost non touched in this case I think there was one query done somewhere deep in the component and that was it. And when I joined I created an issue because I wanted to know what path are we selecting going forward? Will we use some client just as a client? So just making queries and mutations with GraphQL or do we want also to leverage State management, because these are two completely different strategies. And I built a proof of concept like a small project. I explained possible options there and it was a great achievement because since this moment we decided to have front end RFCs, which is request for commands. Before it was just like create an issue and discuss. But after this we created a separate repository for RFCs. So any new thing you want to bring is discussed within the RFC. I kind of paved the way for RFCs as well for GitLab. And yes, there was a huge discussion about this. How should we go? What should we use? We analyzed our own state managers and how much we are relying on the API responses in them because of course there is some local state and there is a state that we are storing locally on the front end, but which in fact is just caching data we are fetching from the backend. So kind of a mirror on the front end. And we found out that we have like around 90% of our data just mirroring backend. They’re actually not even local. This is just mirror. And in this case using Apollo was good idea because this work with the cache is done almost automatically for you without the boilerplate. So this was a process of discussion.
Alex: That’s interesting. Did you work with a team in order to discover sort of how this was working? And like, was there a process that you followed to sort of like organize your team in order to sort of achieve even just like the description or the documentation first and then the project?
Natalia: Yes, definitely. Because I didn’t want this to be just brought by one person and okay, I decided on the architecture. You should be all happy about Apollo client. And I was asking other people who were working with GraphQL and Apollo previously, before GitLab to make sure that we are going the right direction. And after this it was an iterative process. So we implement one feature, we document it. I bring this to the front end call because we also have front end calls like hey everyone, please look here. If you have any questions, let me know. If you have any concerns about what we’re doing here, maybe this is not the best architecture for us, please let me know again. And after this, when other teams started implementing the same strategy, they were first of all copy pasting the code we had. I think this is a normal process for every single company and this is how bad practices are spreading. The most people just copy paste them from your code and reading the documentation. And this is the moment when documentation starts to improve because I’m writing it down how I understand it, but there are People that read documentation say, hey, it’s not really clear from the documentation how I do this thing. Can you please explain me? And this is the moment when you start feeling a bottleneck because you are one person and if it’s like one developer calling you and asking questions, second, third, it’s still just asynchronous calls. And this is the moment when you start improving documentation. Okay, if it’s not clear, we should just make it more clear in our documentation. We should just maybe give better explanation there. And this is the moment when team starts to contribute as well. Because the main idea about being staff engineer is scale yourself at GitLab. It’s not something that there is you, you cannot be replaced. You just some unique diamond in the team. If you are scale yourself, make other people diamonds as well.
David: So first of all, you mentioned that when you started you were the first front end staff engineer at GitLab. Has that changed?
Natalia: Yes, this changed. There was a second staff engineer because there was a person who was on track in parallel with me, arrived here a bit later. He was working with GitLab definitely longer than myself. I’ve been senior for a year and something. He had been working with GitLab I think for five years at this moment and also discovered lots of things on our front end. He was building our front end in these like five people that were starting on the frontend side at GitLab and he’s also working with Apollo. As myself, you mentioned a little bit.
David: About sort of the expectations for the staff level at GitLab. How would you describe sort of the day to day role of both sort of your responsibilities and also if you think it’s different, sort of the general responsibilities of a staff engineer at GitLab.
Natalia: So first of all you kind of inherit all the responsibilities from senior. So this is building a code with little guidance, building complicated features, collaborating with all the different areas like pm, ux, backend. Definitely. But stuff also has a bigger perspective in general. So you’re not building a feature like a small feature that PM just brought and created a task for this. When you’re building such features, you are also looking for the architecture and staffs at GitLab are responsible for the front end architecture. This is a big pain for our company. Unfortunately as I mentioned, we didn’t have staffs and frontend was considered being simple. This eventually led to the things when we were building features on top of features on top of features without real planning behind them. So we have lots of legacy code right now and staff engineers are responsible for cleaning this up and planning this cleanup for the team. It’s not that much on engineering managers and I will mention this bit later, it’s more on tech leadership, which are staffs. So you should identify the places that are bad. And by bad I mean when you develop a feature and 90% of your time is not developing is not just trying to fit this feature into existing code. And when you discover such area, it’s your responsibility to plan how we should make it better and what is the workload and how this workload is to be planned for the team. You cannot assign tasks. Again, this is for engineering manager, but you should describe this area and bring it up to the manager that okay, we have this kind of tech debt and this should be fixed, of course with product as a priority, but still. And speaking about engineering managers, I better describe a bit our organizational structure because right now we are speaking about staff and this is something like undefined. Okay, we have staff engineers and something else. So at GitLab, when you an engineer, be it back end or front end, the track is single until you’re a senior. So we go middle engineers and senior engineers. There are no Juniors at GitLab. There are sometimes interns, but we start usually with middle engineers. And after senior this track splits on two. If you have an interest of being a manager, you go through engineering manager, senior engineering manager, director, and so on. Some people do not have any interest in managing other people, which is completely fine, it’s just how it’s designed to be. But they have interest in tech leadership. So in this case we go through staff engineers. Right now we have a small reorganization, so there should be principal engineers after staff. Currently there is no such stage, but it’s in review, I think. So it will be announced maybe next week and then after this we have distinguished and fellow. If you are a staff engineer, this still means that you have to communicate with people as well as engineering managers do and have some kind of leadership as well as engineering manager does as well.
Alex: Would you describe the way that you operate as a staff engineer as similar to the other staff engineers that you have or is there like a common expectation for staff engineers at GitLab?
Natalia: There is a common expectation, definitely. But I would say that it’s very different for front end and back end staff engineers. Even though written responsibilities are the same, like advocating for innovations, trying new architectures, improving the code, learning something new and teaching other people to this knowledge, and so on. I would say that our backend engineers are mostly focused on their team work or stage. So they have an area of responsibility and they are digging into this one. And for front end you are more widespread. So for example, if we speak about GraphQL and Apollo, this wasn’t for my stage or my team, this was for the whole front end. This was a thing that everyone needs to adopt essentially as we adopt GraphQL on the backend and front end. So I would say that me and other frontend engineer, staff, front end, engineer are working rather on the level of front end than on the level of our stages when backend is more focused.
David: You mentioned sort of the split that happens when you move past the senior engineer role. You either go into technical leadership or you go into management. Did you ever consider going into management?
Natalia: Not really. I just don’t like the management position and I think I’m just not ready sacrificing the pleasure of writing the code. Because at GitLab, when you are a manager you are to communicate with people and this is another kind of pleasure. I can’t say that something is better or something is worse for everyone. But for myself, I really enjoy writing the code. I really enjoy discovering new things and refactoring and bringing this knowledge to my team to just explain things. Being a manager, I know that they sacrifice writing the code and I don’t want to.
David: Yeah, I think that makes sense. What advice would you have for someone who is making the transition from senior engineer to staff level, especially someone at GitLab?
Natalia: It’s really nice question. First of all, I would say that be ready for definitely way more communication that you had on the senior level. And be ready to answer questions, be ready to explain things. Be ready to explain things multiple times, even to the same person, just one by one. And don’t be irritated over this because this is the thing. And also there is something I recently complained about and it was really funny because there are people that come to you for guidance or for validating a solution of sanity checks. But there will be really hard for you because you don’t have anyone to make this sanity check. Sometimes you can do this with your colleague, which is a single staff engineer. And you’re like, you too are like, okay, I think this is fine. I hope we are not breaking something essential here. So this is like walking on a swamp. You are not sure that your next step is not a fail. And failing is inevitable because you are discovering new things for your team as well. You have this knowledge after this to bring to them so they are safer and they can ask you. But be ready for this for like to experience something that there is no advice on. Sometimes it’s things that you cannot ask anyone in the Internet. So a quick example, we have a Vue JS as our main front end framework and we have Apollo client as our GraphQL client. Unfortunately for us, this is not a mainstream stack. If you work with Apollo mainstream stack as React, Apollo and Typescript for this stack, you can find anything in the intranet. You can find courses, articles, guidelines, stack overflow answers. If you’re not sure about something, you can ask Apollo team and they will answer because they work with React and they develop something with React as well. But if you work with Vue, you’re about to write things yourself. So we needed to test our components for example, and our components didn’t have any kind of way to mock a client, which is a library for React. But for Vue it’s non existent. And still to this day if you Google for test view components with a poll. I’m just finding my own articles on dev too. This is a bit frustrating when you’re an engineer because you really want to feel safer. So yeah, if you’re going for this one, you should be prepared.
Alex: That’s really interesting. I just wanted to clarify something I think I might have missed. When you’re talking about sort of like being in a swamp, you were saying that like, because you’re a staff engineer, you might be encountering something for the first time. And so if you become a staff engineer, you should be ready to sort of misstep and that will just be inevitable. And so I’m curious, you know, when you’re doing this work with less of a safety net, how do you have that sort of like self compassion that you’re going to make mistakes or how do you bring psychological safety to those kind of situations?
Natalia: So first of all, here will be a praise to my employer, which is non intentional, but this is the context because a few years ago probably people that work in the industry remember that GitLab had a massive, let’s say fail. It was a database drop, it was a database that was deleted and unfortunately there was no backup for I think five hours of data. I wasn’t working at a company at this moment, but I was reading about this and watching this stream and it was really nice to see that there was no blame from the company to the person who actually did this mistake. And when I joined the company, when you joined the GitLab, first of all, you have to have five coffee calls with random people across GitLab you just select them and call. I called the person who actually dropped the database. I really wanted to hear, like, how it was. He said that there was no pressure from the company. No pressure at all. Because we don’t punish people for their mistakes. Wow. Because every single mistake is a thing to learn something. And we learned a lot. And this guy, he became a security specialist after this because he made this mistake. So there was a lot to learn. How to not repeat this mistake and how to be more cautious about this and what comes company should do. So this brings a lot of safety to staff engineers as well, because when you’re building a proof of concept, you already that this might not work. There is always a possibility that this is the wrong way and management is ready for this as well, which is really nice. So it’s like, usually it’s said with a manager, okay, we just give it a try for a week. If it works, it works. If it doesn’t, well, okay, we learned something. We just throw it away and we try some other things. So this really helps in psychological sense.
David: I think it touches on such an interesting aspect of what staff engineers do. Because when I first got to staff, one of the things that I kind of that scared me about it relative to my role before was that, you know, I was really sort of expected to find my own things to work on. And there wasn’t really the same sort of roadmap carved in front of me for me to follow. That’s really intimidating. It kind of lines up with the loneliness that you described, where there’s less sort of sounding boards available around you. Even in an organization with lots of staff engineers, there’s still, you know, other people are focused on other things and you’re expected to carve out your own area, and it’s hard. Do you find yourself doing anything to sort of help other people along that journey? Maybe people that are. Maybe people that are still at the senior level, but that are moving up another way of asking the question, maybe reframing it away from sort of the loneliness. Although I think that’s. That’s still a factor. But sponsorship and mentorship in general, to what extent is that? Is that something that is relevant to your work?
Natalia: So first of all, do I help other seniors? Definitely, yes. Because there are always people who aim to go to staff position and they will ask you questions about what do I lack, what do I need to focus on to be there. Usually it’s really nice to help them to find their spark, let’s say spark, because I was recently Watching the soul and I really like the term. So to find the spark to work on and to bring to the team where they shine. This is also helping with some communication strategies because at GitLab, if you can have any major blocker on your way to stuff, this will be communication. There should be no conflicts. Or I mean by conflicts I mean something really hard. You shouldn’t be conflicting, you shouldn’t be pulling ranks, you should discuss with people really nicely being on their level. Because the main fear for myself for example, was and still is like when you get starstruck, when you feel, okay, I’m a staff engineer, so I’m kind of a queen here and I will just be above you all. This is non acceptable at all. And this brings us to the mentorship because and sponsorship in general. So for me it was something that I really enjoyed being staff because you are required to mentor people, you are required to bring them to your level, to a level of knowledge in particular. And I join it because in my work on Vue js, I’m writing documentation. So I’m kind of mentoring people on VUE in my open source contributions. So the same happens with GitLab. And while we are speaking about Spark, why I’m trying to help people find this because I see that people that enjoy some area, let’s say of the code, are great with mentoring and advocating for it. Sometimes of course we are probably too much into it. So when I was building like very first things with Apollo client, there were people that were trying it, using it in a wrong way, which happens. But blaming the technology which is like, oh, this thing is just not working. This library is so bad. No, it’s not bad. You’re using it wrong. But this is something that makes you want to teach people like, okay, let me explain how to use it in the right way. And when you’re mentoring other seniors to go to the staff level, it’s communication strategies and mostly finding the area they really like.
Alex: That’s really great to hear. Sort of helping people level up through working with them. Earlier you alluded to the fact that you work a lot with sort of other business partners. Well, I guess in GitLab it is a business, but I know it’s also an open source project. So I don’t know how you refer to your sort of cross functional partnerships. What role do you feel like you have as a staff engineer when you collaborate with another area of the organization with product? I think you mentioned, what do you see your role as a staff engineer is in working with product or other cross functional partners.
Natalia: It’s a great question because we are required to be cross functional in general. And with product engineer sometimes you can see that you need to squeeze some refactoring or re engineering or maybe even changing the whole structure of the feature PM already planned. This is really hard because you need to explain what technical difficulties you have here and what do you need to change and how it could or should be implemented. And also when you’re working with pm, sometimes it’s given flesh and bones to the ideas product venture. Have an example to this very abstract description is my product manager was about okay, let’s make a page widget. So let’s abstract separate features to widgets that can be added to the page, removed to the page, like very basic thing. But with our architecture it’s not basic, it’s very complicated because we have let’s say far from perfect front end structure and building widget in terms of this was really complicated because you don’t even know what widget is like. Okay, I want to add widgets. What is widget, is it application, is it component, how it should communicate, what contracts should it have, how it should collaborate with the backend. And here we’re bringing backend to the party. It’s like okay, how do you want to play? And shaping the API because again GraphQL is shaped mostly by our frontend. This is backend is giving you data you want to see in the schema. And this is also working with the UX because we have our own design system. We have great UX designers and explaining to them what attack difficulties and what can we see here? How widgets should be changed and so on and so on is also part of the game. But also in terms of cross functional, GitLab is not that monolithic. We have three main sections which are DevSeco and Ops. And these teams rarely collaborate so they work on their own areas. But this is not for staffs. So staffs are also included as let’s say internal consultants. So you need to build something complicated. You don’t have a staff engineer in your team. There are only two front end staffs for the whole company, not even one for the section. And we’re both actually in the same section. Other teams kind of go to your manager and say hey, we need the help from your staff engineer. And this is help that needs like lots of working hours. Because if it’s like small mentoring, if it’s just asking a few questions, this doesn’t need to be scheduled, right? But if they need you as a consultant, sometimes you join their team and help them build architecture, help them involve the practices you already learned and consult them on what is better for them currently. And we had a kind of opposite process as well. When there is someone joining your team as an intern to learn something from you as staff engineer, like building architecture, for example.
David: That consulting model sounds really interesting. Can you say a bit more about the process around that? So you mentioned that a manager will come to your manager and and ask to sort of borrow you. At what stage in the project does that happen and sort of how do you influence the design and to what extent? Like basically what’s your time commitment look like to that project while you’re, while you’re sort of loaned out to it, et cetera.
Natalia: It’s different, it happened on different stages already. Sometimes it’s something that they were already building, but they are stuck currently because there is problems with scaling the project. There are problems with like adding new feature. So this is kind of like a legacy thing and you are asked to go there to make it better, kind of clean up the whole dirt. I know the term is not really nice, but that’s what you do. And sometimes you are involved really early when there is an idea to build new complicated feature and people do not know how to make it better. With some new technologies like Polo, like we want to have really extensive local state, can we use this or that? Sometimes it’s somewhere in the middle. So a quick example was an application that already had a VUEX state management. So it’s like flux, like state management for Vue js and they also had a part with Apollo and there was a problem just to synchronize two states properly because two single sources, not even single two sources of truth already. So how to make it happen? And this is why I mentioned observable, because people had no idea they can subscribe to the Apollo cache and this was very new to them in general. And building a proper one way connection was a problem. So I was asked to be there around and to help them. As for time commitment, it really depends because when I was promoted to staff there was an agreement with my manager that there is like 70% of my time working on our deliverables on our own product and like 30% of time for, let’s say staff responsibilities in general. Even if you’re not involved into other team, it’s like again building documentation, improving it, thinking about migration strategies. Because libraries always update and there is a major update of two main libraries. We Use Apollo and view Both the version 3 kind of version of Doom. It will be really hard to migrate and this is part of responsibilities as well. But when sometimes there is a high priority for other team to build something new and they really, really need stuff. So if it’s within 30%, the process is not even formal, they just let you know and let know your manager, okay, she’s working with us, is it fine? Okay. Okay. But if it’s like some really big time commitment, they should be agreed with the whole team, like with a manager with pm, sometimes director level as well. And in this case it might be like even 100% of time of the staff engineer dedicated to this team.
David: Something that I was curious to circle back to as well. With that consulting model, you said that sometimes there’s like 100% sort of commitment to the new role. In that case, would you be participating in the design aspect of the project as well?
Natalia: In technical design, definitely yes. In this case, you would be starting with a very basic architecture and mostly based on our GraphQL API. So what we can do, because it’s still limited. So you’re bringing the consultant on this stage and you’re also explaining really high level architecture to the team.
David: Right. Going back to another thing you mentioned as well, you’ve talked a lot about sort of the introduction of the Apollo client and you’ve also talked about sort of some migrations from different state management systems. In general, what kind of strategies do you use for driving a big migration across sort of potentially multiple systems and multiple teams?
Natalia: First of all, what we do is we start with removing the parts that are deprecated in a new version and adjusting them to be working for both versions. So this already happened for Vue 3, so we are preparing our code base, at least removing the things that are deprecated in a new version and same for Apollo. So before we even start the migration, we are identifying the parts that we should fix and we can fix in the scope of the current version. So I can say for Apollo client, for example, there are cache updates and they need to be immutable and it’s not a requirement for version 2. It’s recommended to have them immutable in version 2.6, which is the latest of 2, but you can still go with mutable. So what we did, we haven’t migrated to version three right now, but we already made all the cache updates immutable. And how this happens. First of all, again, you create an rfc. Everything starts with an rfc. You explain why this should be Done. And how to do this? You build a prototype, you show this to the team, you build a documentation about this and after this you add a linter. Because any single explanation cannot beat a linter in terms of being effective. Don’t mutate the state, we’ll int against this. This is the first part of the migration. Next steps are usually you plan what should be done first. So in terms of our Vue 3 migration, for example, we have a component library, GitLab UI, which is unfortunately a wrapper around Bootstrap View. And unfortunately, because Bootstrap view is not migrated to version three, so this is a major blocker, you cannot start a migration. You can only clean up your database and prepare everything to be working as much as possible on both versions. So you need to change as less code as possible. So this is first thing and bringing the migration. It’s actually good that GitLab is not monolithic, that we have some standalone app like customers portal, because this is a playground, because you can start migration on these smaller applications to see what else we discovered during the migration and what should be brought to the main code base without breaking it. Also, for such things as like major migration change, we introduce feature flags usually. So if something goes really, really wrong, we just like scroll down. Okay. Feature flag off. Sorry, this wasn’t dropped. It works so something like this.
Alex: So it sounds like RFC is like the main tool of technical alignment. Does that extend to team level alignment too? Like, do you use RFCs on a team to sort of align on what you’re going to do?
Natalia: Technically, not really. This never happened on a team, from my memory. It actually only happens for the front end. I recommend the backend team to adopt this strategy as well. So this is not a company initiative. It was initiative of the team to make us aligned and to agree on things. Maybe for the backend it’s less effective or maybe even less necessary due to the nature of the front end. Because basically on the backend we have Ruby and Ruby on Rails, so they have more or less monolithic again stack, let’s say. So there is no that huge JavaScript platform that includes billions of tools and even for us and also architectures. Again, Ruby has lots of magic baked in, but if we compare with, let’s say Vue js, Vue JS is way more flexible. This is good and bad. It’s good because it allows you to build whatever you want. It’s bad because it allows you to build whatever you want. So you can build anything and you can shoot yourself on the leg without knowing it. So maybe due to this flexibility, we really needed to agree on some things that might sound basic for backend engineers or even for other JavaScript engineers. For example, things like do not use JavaScript classes on view data because for us it doesn’t work and there were so many things about it. But there are also major discussions that do not have a correct answer. For example TypeScript, there was a long discussion about should we use types on our frontend or not. While for TypeScript fans it might sound like very obvious, like why not use types? It’s type safety. In fact, when we analyze the effort we need to put into it and the profit we would have from it with our test coverage, which is pretty impressive, around 70% for the front end side. For the backend, I think it’s like closer to 100. There is not that much benefit from TypeScript for us, but without the discussion in the RFC it wouldn’t be that visible.
David: How do RFCs work tactically? Is it a Google Doc? What platform do you write them in?
Natalia: It’s a great question because GitLab is all about dog fooding. Eat your own dog’s food. So all the discussion happens in issues, but it’s not something specific to development. At GitLab, everything happens on issues. If you are human resource specialist, we call them people ops, not human resource. If you’re sales special, it doesn’t matter. You work with issues and merge requests always. That helps. Testing our own product. So yeah, RFC has issues.
David: So then is the RFC like a markdown document that you submit as a merge request or is it actually like an issue where you write up sort of the thing and then people comment in a thread underneath it?
Natalia: It’s an issue. First of all, it’s always an issue. And there are two major things you need to do for the rfc. First of all you need to prove that this is two way door decision because if it’s not, it’s way much harder to convince people to move to it. You should be able to go back, you should be able to roll this down. And second one that we required after like first few months of doing RFCs is implementation. You want to bring something, build a proof of concept, build a small project because not everyone is grasping concepts just from the text. Build me a project so I could run it, play around it and see what is good and what is bad about it. So these are two hard requirements. And after this there is just discussion in the issue. And when everyone is like stopped commenting, we have two weeks and if there are no hard objections from everyone. So if there is kind of consensus on the issue, it’s closed and we assign the DRI direct to the responsible individual. Usually this is an author, so if you have any kind of initiative, be ready to implement it because it’s yours. Otherwise it just doesn’t make sense. Just bring an idea like, I want a typescript now. My dear team, enjoy implementing it yourself. No, it doesn’t work this way.
David: You mentioned something really interesting about two way doors. That resonates with me. I think obviously trapdoor decisions are worthy of a whole lot more scrutiny. However, it’s really hard to avoid trapdoor decisions sometimes. I’m curious how you think about that. Like from my perspective, adopting TypeScript, once you get to full adoption, that’s pretty close to a trapdoor decision. I mean, obviously you can undo it, but the work to undo it is significant. So how would you go about composing an ARC RFC that describes it as a as a two way door rather than a trapdoor?
Natalia: When you have a no 2 way door. Okay. When you have a trapper one, you should have really good arguments that are measurable. Don’t say, okay, TypeScript is cool, we will have type safety. No, type safety doesn’t work. What it will give us in terms of like less unit tests. How much unit tests is it definitely like this, even in just in code streams. How much more code will we need to write in terms of our framework? Evaluate our framework. Because one thing, if it’s angular with built in TypeScript support and actually written in TypeScript with Vue JS, it’s more complicated. Then you need to obviously describe any kind of caveats. Because if you propose a decision that is not two way door, analyze caveats and describe them as detailed as possible so we will be able to evaluate. Okay, we are going this way. This is dangerous one. We want to know dangers before we start. If you don’t do this, usually this is a team doing this and this is usually against the tower decision. This was basically one of the things why TypeScript was not adopted because yeah, too much work, not that much added value, measurable added value for the team. And the fact that going back would be really hard and would block us for using libraries that do not have typings because this still happens for Vue js, not every single library has typings. And yeah, this is really complicated because sometimes we have these decisions and we have them adopted. For example, Apollo client as a state manager is also not the way door because going back to Vuex would require as much work as writing it from scratch. But it was a discussion about benefits and the benefits were actually measured in terms of even the right written boilerplate. And this is measurable because you can compare the boilerplate for two examples and see, okay, this is more preferable for us.
Alex: When you’re making the argument for a specific RFC, or if you’re trying to ask an organization, especially with something like Apollo or GraphQL, which feels like it has a very big impact, do you try and align with some sort of larger organizational goal or are you making almost like a purely technical argument?
Natalia: No, no, of course this always goes with organizational goal. Maybe Apollo itself on the front end is smaller side, so it can be decided technically. In our case it wasn’t. It was also discussed with lots of PMs and I can even tell you why, because we were going for real time updates as well and this was a major product goal. Definitely not tech team goal. Our product were about, okay, we want issues in merger quest to be updated in real time, which sounds fine in 2020 I guess. And when we were working together with backend team, we discovered that using GraphQL subscriptions in this particular case with our API would be way more performant than just hitting websockets and then just sometimes because our websockets were without payload. So WebSocket has a signal you are refreshing the query. And our REST API queries if you check like GitLab jsons that are returned from REST API, it’s huge. It’s just a super huge JSON object. So when discussed this all so product goal, real time updates, technical implementation. The best one is GraphQL subscriptions. GraphQL subscriptions better handled with Apollo client. So this leads to like, okay, this is one more point to this one. As for GraphQL API itself, it was discussed I think on a much higher level and way before I joined the company. And it was an interesting moment when we already had an API, but we were not using it on our own front end for a year and a half, I think I have no idea why that happened, so don’t ask me.
David: Switching gears a little bit. One of the things that you mentioned earlier was that GitLab is introducing the principle as rung on the career ladder. I’m curious how you think about sort of progressing past the staff level and what you think is required to get there.
Natalia: Great question, because I was in the working group for defining an individual contributor path and this is still a tough question for me because we decided to have principal level to be merged between backend and front end, which is complex because you are going deeply into one area of course doing some work that relies on backend. So I’m writing Ruby code obviously and our backend engineers sometimes write JavaScript code as well. But this is not on the level of senior engineer by no way. A full stack introducing principal role is still challenging for us because this will be even higher level than staff engineer and you would need to understand not only front end challenges, but also the challenges of the code base in general to making decisions there. I have to say that at this moment the responsibilities of principals look rather badly defined to me. That’s why we’re not merging this merge request and we are still trying to find out how it should be and how we should work with management. But this is different management level as well. Currently staff engineers report to managers and principal will be reporting to pan level hire, which is senior manager and working with them. And there is also a decision to limit principal engineers to one per team for both backend and front end, which is again very questionable for me because there are some teams like security teams that are working on cutting edge of the technology and maybe there was more sense to have more steps and more principles there. And there are teams that do not require principle at all, which is completely fine. Again. So yeah, it’s a good question, but I don’t have a good answer for this.
David: Interesting. One of the things you mentioned there that piqued my interest. You mentioned that we’re not going to merge the merge request quite yet. Obviously you’re again dogfooding GitLab. Can you say more about how organizational changes fit into a git repository?
Natalia: Basically, when you started the company, one of the first requirements for yourself is merge something to the handbook. Because we have a handbook which is an impressive document of thousands and thousands. No, hundreds. Hundreds of thousand.
David: I’ve seen it. It is pretty crazy.
Natalia: Yes, it is. So you’re requested to make a change there because handbook is continuously evolving and as I said, everything is a merge request. Our guidelines, actually documentation and you met merge requests too. Our organizational structure is again placed on GitLab pages and you just make a change there with a merge request. Our team is changing with a merge request. If you check GitLab team, it’s like you might merge request. So yeah, even the word merge request. It’s so funny because when you join GitLab, most of people use GitHub for their open source activities or their own repositories and you’re constantly fixed. You’re like, I made a pull request. We call it merge request here. It’s definitely funny. And you get used to this, like, do not merge a merge request.
David: I confess that I’ve bitten my tongue a couple of times over the course of this interview between the difference on those two.
Natalia: I know, right? But for me, and this is not because I’m working for GitLab, that are merge request is kind of more intuitive than pull requests. Okay, I have changes. I request to merge them to this repository. While pull request is like pull them. What do you mean?
David: What are some resources that you’ve learned from in your sort of journey as a staff engineer and getting there? Books, blogs, people, et cetera.
Natalia: Oh, this is really complicated because I think my main resource, and it is funny, was my engineering manager at the moment and the whole promotion was because it was his initiative to promote me and to start working on the promotion document. And my main resource was actually the promotion document because it outlines the areas you need to have and what you’re lacking in this case. So for me, for example, I needed to prove that I am able to work with performance and to improve performance, which is non trivial if you think about front end, because what you can improve in terms of front end performance, rendering times. This is bound to Framework lcp. Okay, you have largest contentful pain. This is something you can work with, but again, limited. So it was rather like finding out what I still need to learn and learning with specific resources. So for performance it was just basically JavaScript performance and GS performance articles mostly. Yeah, not really helpful, but that totally.
Alex: Makes sense, you know, in general, are there any other blogs or books or people that you see as role models or just like sort of interesting thinkers?
Natalia: Yeah, I definitely have a role model. And this is Sarah Dresner, who is vice president of. I might be wrong with her position, honestly. So she’s Vice President of Developer Relations, I think, at netlify and we are friends because we are both working for Vue JS Framework as well. Not working, we’re just contributing to Vue JS Framework. But it’s like another work and I definitely learned a lot in terms of communication and resolving conflicts and being understandable. How to communicate with management, for example, or how to communicate with your teammates, how to resolve conflicts in the team. That all I learned from her.
David: On that note, I’ve been meaning to ask you a little bit about your contributions to Vue JS as well. How long have you been doing that and how do you see that work sort of influencing your work at GitLab or sort of your outlook on software engineering in general.
Natalia: So I’ve been contributing for more than two, almost three years. Of course, not in the core team for all this time, but I started with working with Vue JS and noticed a few things that I didn’t understand from the documentation or they were wrong in the code, in my opinion, and started like small contributions around to the framework, to the documentation. And also at this moment, I started communicating with the team because when you do some subsequent contributions, they reach out and ask you and you’re just, okay, let’s fix this, let’s fix that. And then eventually you gain a core team title because you’re already doing lots of workaround and everyone’s like, oh yeah, she’s core team. No, I’m not. I just lack a formal title. Okay, you have it. This is very informal at vgs. If you do something for the framework, you will be in the core team. And how this influence actually a lot because this is beneficial for both areas. In terms of GitLab, I can bring the good practices from the documentation and from the framework and also the latest news as well. So it’s like, okay, in version three, we are deprecating this even before the documentation is published. Because I’m not even an insider. I write the documentation so I can share this with my team. This is not a secret, it will be published. But also GitLab is a huge code base in Vue and I believe at the moment it’s the biggest open source project written with Vue js. And this helps us discover bugs and bring them to the team. That helps us develop good practices and bring them to the world. I know that people are copy pasting our unit tests with Vue a lot and also this helps to shape the product. So, for example, testing library Vue test utils that we have, they were planning a few breaking changes, but when I said, okay, could you please look at our code base in terms of tests? And you are going to break more than 5,000 tests with this change for the company. And this kind of gives library authors the idea of how their product is used in practice. Because when you’re writing a framework, sometimes you have no idea how people can use it and they can use it in all the different ways. So this gives an idea to Vue JS and Vue JS ecosystem authors how the product is used and what to do and what not.
David: That makes a lot of sense. One last question. We’re almost out of time, but sort of the perennial question for staff engineers. How much time do you still spend coding?
Natalia: A lot, actually. Last week I think all my working time was coding because I’m in the middle of future factoring, currently unifying the same stack, and it was a full week of coding. I’m really happy about this because previous months were mostly mentoring, which is nice as well, but less coding, less fun.
David: Awesome. Well, Natalia, thank you so much for your time. It was really, really a pleasure.
Alex: Yeah, thank you. It was nice to meet you.
Natalia: Thank you. It was nice to meet you as well.
David: Take care. That’s it. Thanks so much for listening to staff Eng. If you enjoyed today’s show, please consider adding a review on itunes, Spotify, or your podcaster of choice. It helps others find the show. It is a really useful signal to us that folks are finding value of this so that we keep doing it.
Alex: You can find the notes from today’s episode at our website, podcast.staffenge.com the website also has our contact info. Please don’t be shy.