Tips on Take Home Projects/Case Studies

It’s been awhile since I’ve written one of these (and hopefully soon my cadence will be more than 1x a year when things slow down a bit) but this one felt important given the market.

Stemming from the practices of places like McKinsey and Google, lots of folks these days feel a take home project (case study, presentation, technical challenge) is the best way to get a read on a candidate. I want to start by being very clear that there is some major truth to this and when I was at Flatiron, we did it for many of the leadership roles. That’s because every TA leader has seen the research that unequivocally supports the fact that “work test” or “work samples” are a very strong predictor of expected performance. However, what people don’t necessarily realize is that not all take home projects are created equal and sometimes, they can make you lose great candidates, or worse, give irrelevant or contradictory signals on performance.

So, here are some considerations to keep in mind to make sure you’re getting the info you need while providing an amazing candidate experience:

Candidate Level: This is, without question, the most important consideration when thinking about doing a take home project as part of your interview process. This is because the more senior a candidate is, the more they will feel that their vast experience and plentiful list of references speak louder to their qualifications than a work test. There is a lot of truth to this too, so you should be respectful of this perspective and not dismiss their reluctance to do these projects as arrogance. That said, it’s also understandable that what you need is unique and different to other companies they’ve worked at. These things aren’t mutually exclusive…if you set up the project appropriately:

Relevance to the Role: The first question to start with is whether the take home project actually ties closely to a problem you would expect this person to solve in the role AND whether it tests critical competencies needed to solve that problem that can’t be well tested through the other portions of the interview process. Here are some examples:

Good:

Testing an engineer for their coding skills by giving them a coding challenge that mirrors the type of technical challenges they will face.
Testing a Sales Leader for their ability to sell by having them pitch/present to you on their current company.
Having an strategic/operations leader present to you a transformational case study or change mgmt initiative they led and how they led it in their previous role to test their framework thinking, stakeholder management, and understanding of first, second and third order consequences.
Having a Product leader walk you through a product they launched.
Having a People Leader talk you through a current organizational challenge at your company, while asking questions and gathering context

Bad:

Testing a CTO for their coding skills using the same code test you give your ICs. Believing a leader should be able to complete any project their direct reports can is a huge hiring fallacy. Leaders specialize in leading others in their work, not by doing it for them. Instead, test them on how they would coach a person to solve the challenge or better, how they have trained and developed their orgs to solve problems like this at scale, and how they know if they’re doing it well or not, how they’ve changed course when they identified issues. This isn’t to say that leaders shouldn’t have content knowledge — they should, and we believe earlier in their career they should have had a superpower in some important category under their umbrella (e.g. a Finance leader who came up in FP&A, but maybe not Accounting, an HR leader who came up in Talent Development, but not Total Rewards) . This experience will help them earn the respect of the team. However, their day to day now will not be doing that work so it doesn’t make sense to double down on it in the interview process. If that hands-on work is a major portion of their day to day, you should rethink the level of the role.
Asking an operations leader to do a do a generic operational challenge, especially one that has no relevance to your business or involves a level of modeling/analysis that will *not* be present in their day to day. Sometimes people do this to test “core” skills, but really what it tests is “b-school” skills. If it’s not part of the job day to day, we believe it introduces unnecessary bias and doesn’t actually give any signal on a person’s ability to do the job (unless their real job will be opening a widget store ;-).
Testing a Sales Leader by asking them to pitch your current company. This applies to any time you ask a candidate to solve a problem you currently have ahead of time/at home vs. one they have expertise on already. Chances are, if they’re doing it at home, they won’t have the context you do, and won’t solve it as well as you would. Asking them to do this will only show their research/guessing skills/powerpoint skills, not their actual skills in the role once they are fully ramped, supported and knowledgeable. So if you do this, it will be hard to separate if they did it well or poorly in comparison to your own deeper knowledge on the topic. Two exceptions to this rule: 1) It IS reasonable to give real technical challenges that aren’t company specific. For example, asking a Benefits/Rewards leader to present how they roll out a 401k program across different states and employee types. Super technical, might be a real challenge you have, but also something they should know that isn’t impacted by company context. 2) It IS ok to provide a real problem you have ahead of time, give them time to *think* about it, and then talk it through their thoughts/questions in real time so you get to an answer together. This will show their thinking and problem solving ability, but with the context needed and require low prep.

Stage in the Process: While it is 100% appropriate to give a coding exam as a first step in the process for an IC engineer, giving a work test to a C-level candidate as the first step is not appropriate. It comes off as disrespectful of their time and presumptuous about their interest — most senior folks are passively looking and hoping for a first conversation (with the CEO) that will reel them in, not test their basic skills. So the guidance here is simple – the more senior the role, the later in the process this should occur. For VP/C-level, this means the final step.

Time requirement: Always, but especially in this insane market, the best candidates have lots of options, are entertaining lots of companies and also juggling a current job and responsibilities in their personal lives. Asking them to invest several hours of their time for free is not a great strategy for closing someone you’re excited about. While you may think your company is super special and if they want it, they’ll do it, the reality is that the market is hot and there are a lot of special opportunities out there. Worse, by optimizing for this “interest”, you might be actually optimizing for candidates who have fewer options and are willing to do this sort of project as a result, which may be because they are more junior or less qualified than candidates who opt out. Finally, there’s lots of research that points to the fact that women play a larger role in personal/family responsibilities than men, so they may have even less time for something like this and asking for a big time investment could therefore diminish the gender diversity of your talent pool. So, we strongly suggest any take home project not be not more than 1-2 hours of time (and really 1-2 hours, not 1-2 hours that is really 5 if done well). This fits well with requests of presentations of previous work they have done or presenting framework thinking around new problems or demonstrating core technical skills. If it requires more than 1-2 hours, you’re probably asking them to actually solve a problem for you rather than demonstrate their *ability* to solve it and that, combined with the time asked, starts to feel like free consulting and a disregard for their years of experience. This is when we see great folks drop out. Lastly, the time requested should correspond to the point in the process — if it’s a first step or post screen test for a junior/mid-level person, even 2 hours is too long.

Expectation setting: If you are going to have a project, we can’t emphasize highly enough how important it is to tell candidates early (ideally along with the layout of the whole process) so there are no last minute surprises of more work when they are expecting an offer. We do this for our clients, but if you’re not working with a search partner and have no internal recruiter, you should do it yourself. Also, before the project, we highly recommend a 15 min call from the hiring manager directly to set the stage on the project, answer questions, explain why you want them to do it and what you’re testing so they are clear on how it ties to the role and they are set up for success in what they deliver. This also shows you investing your time in them, so it feels more like a two way street.

I hope this is helpful and if you do decide to work with us at some point, this is exactly the type of thing we can brainstorm on and help craft so that we can ensure we’re getting a deep, accurate assessment while also providing an amazing candidate experience.