By Josselin Feist, Principal Security Engineer

Finding talent is hard, especially in the blockchain security industry. The space is new, so you won’t find engineers with decades of experience with smart contracts. Training is difficult, as the technology evolves constantly, and online content quickly becomes outdated. There are also a lot of misconceptions about blockchain technology that make security engineers hesitant to enter the space. As a result, the pool of people who are able to both master blockchain technology and grasp the mindset of a security engineer is fairly small.

We have now been working on blockchain projects for more than half a decade, and we have always struggled to find qualified applicants. Last year, to alleviate this problem, we created an intensive apprenticeship program to give apprentices the equivalent of two years’ experience in only three months. The program has been a huge success, and we have offered full-time positions to all of our apprentices!

Read on for more information about the program and the apprentices we’ve hired so far, as well as pointers for future applicants.

The apprenticeship program

The main goal of the program is to train our apprentices to become highly technical security engineers. We set high standards for our employees, and we want to enable our apprentices to quickly meet our expectations. There are two key aspects of the program:

Mentorship

Every apprentice has a mentor from the blockchain team (someone of at least the senior level). Each mentor has one apprentice at a time, which ensures that the mentor can provide personalized feedback and support. The mentor is responsible for making sure that the apprentice understands our processes and techniques and is challenged technically. For example, the mentor might task the apprentice with reading a section of the Yellow Paper and answering related questions; the apprentice could also be asked to study a new attack happening in the DeFi ecosystem (and to master the underlying technique). We have also developed a set of in-house challenges and exercises to help our apprentices grow.

Mentorship is a key part of our apprenticeship program and makes the training process fast and efficient.

Audit shadowing

Our apprentices work full time and participate in our audits, though their hours are not billed to our audit clients. By shadowing audits, apprentices learn how we approach a codebase, practice using our tools, write reports, and have a chance to interact with the team and clients.

This is a hands-on experience for our apprentices, and we want to give them as much exposure as possible to different approaches and code review strategies. To do that, we have our apprentices switch auditing teams: they may work with their mentors, but they could also work with anyone else in our Assurance Practice.

Who we are looking for

While we’ve seen a lot of different kinds of applicants, from recently graduated engineers to more experienced professionals, this opportunity is intended for exceptional entry- to mid-level professionals with experience in blockchain development or auditing. Over the past year, we’ve had eight apprentices:

  • Four of them had about one year of blockchain experience.
  • Two had previous cybersecurity experience.
  • Two had completed the Secureum bootcamp.
  • One had graduated one year before starting the apprenticeship.
  • Coincidentally, three of them had founded a startup in the past.

We’ve found two kinds of applicants to be the best fit:

Blockchain experts / security enthusiasts

These are exceptional blockchain engineers / researchers without a professional security background. People who fall into this category already have in-depth knowledge of Solidity and the EVM but have never done an audit in a professional setting. We help them strengthen their understanding of how to conduct an audit and train them to think outside of the box and to use our tools.

For example, take Jaime Iglesias. When Jaime joined our apprenticeship program, he had been working in the blockchain space for a couple of years and already had expertise in smart contracts. (He was one of the winners of the 2020 Underhanded Solidity Contest.) During his apprenticeship, Jaime learned how to conduct a professional audit and how to approach a codebase from an attacker’s point of view. He also learned how to write and structure reports and how to effectively manage and work with clients.

Security experts / blockchain enthusiasts

These are experienced security researchers with a background in traditional InfoSec. They know how to perform an audit and have been learning about blockchain technology in their free time, but there may be some gaps in their understanding of edge cases.

For example, Anish Naik was an offensive security analyst before becoming an apprentice. He knew how to think like an attacker and to participate in an audit, but he was working on blockchain projects only in his free time. During his apprenticeship, Anish had the opportunity to work full time on blockchain projects and to perfect his understanding of Solidity and the EVM. He also learned various auditing strategies from our team members and gained exposure to the latest tools, threat intelligence, and development practices.

How to get accepted into the program

We recommend that candidates do the following:

  • Strengthen your understanding of real-world vulnerabilities and auditing.
    • Review the material offered by Secureum, which will be useful as you start your blockchain security journey. Watch Secureum’s YouTube videos to gain an understanding of the most common vulnerabilities and to test your knowledge through quizzes.
    • Read our audit reports to get a better picture of real-world vulnerabilities, including less common bugs. Pay special attention to the descriptions of vulnerabilities and the structure of those descriptions. Reading our reports will help you to write better reports yourself.
  • Increase your knowledge of advanced topics, including the use of tools.
    • Read our blog posts. In particular, master the concept of contract upgradeability and learn about how we used Echidna to fuzz a library and how we fuzzed the Solidity compiler. Our blog posts detail technical challenges and pitfalls of blockchain security and will help you gain in-depth technical expertise.
    • Complete the exercises in the “Program Analysis” section of building-secure-contracts. Our building-secure-contracts repository contains guidance on how to efficiently use our program analysis tools (specifically Slither, Echidna, and Manticore). We use these tools in our professional audits, and they significantly enhance our auditing capabilities. Mastering them is key to becoming an expert auditor.
  • Put your knowledge to the test.

We receive a lot of applications, but you can stand out from the pool of applicants by demonstrating your knowledge publicly, through blog posts or tool contributions.

For example, before applying, Simone Monica made direct contributions to Slither (PR850: “Add support of ERC1155 for slither-check-erc tool”). Troy Sargent created a tool based on Slither to solve an Ethernaut challenge (as he explains in his blog post “Slithering Through the Dark Forest”). He ended up expanding on this work after joining the company and has since built slither-read-storage, a general tool for reading on-chain variables. (See his recent blog post for more information.)

By contributing to our tools, Simone and Troy demonstrated their technical expertise and ability to make contributions to the community.

Frequently asked questions

  • Is the apprenticeship program remote?
    Yes. Trail of Bits is a remote-first company; most members of the blockchain team are in either the Eastern time zone or Europe. We can hire apprentices in time zones from Pacific time to Indian standard time. The one requirement is that their hours overlap with the morning of the Eastern time workday.
  • What happens if an apprentice is not ready for a full-time position after three months?
    We find that on average, we need three months to train someone. However, if an apprentice is ready for a full-time role early, we can hire the apprentice right away (as we’ve already done multiple times). If someone is not ready after three months but would likely be ready after a bit more training, we can extend the apprenticeship. Our goal is to help apprentices successfully join our team, and we will invest the resources necessary to reach that goal.
  • What tech will I work on?
    At Trail of Bits, we work on many different aspects of blockchain technology, including smart contracts, consensus mechanisms, and virtual machine architecture. However, the apprenticeship focuses only on smart contracts; this gives us the time we need to help our apprentices become highly technical experts and meet our expectations. Once the apprenticeship is done, our new employees will have the opportunity to gain exposure to other components.
  • Do apprentices work only with the Ethereum chain?
    No, we are also looking for candidates with backgrounds in chains including Algorand, Cairo, Cosmos, Solana, and Substrate. Candidates who have experience with these chains may receive dual training (in Ethereum and an additional chain).
  • How many candidates do you accept?
    We usually welcome a new apprentice every month.

Join our team

Our apprenticeship program has been a successful experiment for us, and we’ve gotten positive feedback from our former apprentices (all of whom we’ve hired). Here’s what a few of our apprentices had to say about the program.

Anish Naik, who was an offensive security analyst and developer prior to joining us:

The apprenticeship was an incredible opportunity for me to enter the blockchain security space and learn from some of the best auditors. You get to work on a research-oriented and collaborative team, increase your knowledge of a variety of tools and technologies, and make a positive impact in the industry!

Justin Jacob, who graduated in 2021 and was working in blockchain analytics before starting the apprenticeship:

The apprenticeship is one of the best learning opportunities I have had in my career. Spending the day working with some of the smartest professionals in the space was extremely helpful and drastically improved my skills as an auditor. Furthermore, since being hired full time, I’ve loved the opportunities I have had to do more research about up-and-coming blockchain technology, learn new skills and techniques, and improve my overall understanding of the industry. The flexibility of the company allows me to dive into anything I find interesting, which I really appreciate. This has been such a positive growth opportunity, and I would highly encourage anyone interested in the program to apply.

Robert Schneider, who joined us after demonstrating his skills through the Secureum bootcamp:

In the apprenticeship program, you’re not just an observer, watching the process unfold—you’re a full-fledged member of the team! In my first audit, I researched issues, contributed to bug reports, and interfaced with the client—all while learning the trade from some of the best smart contract auditors in the industry.

The next round of the program starts in October, so be sure to apply for an apprenticeship if you are interested in joining our team!