Lessons Learned from 8 months in Web3 and My Top 5 OpSec Insights
A practical OpSec roadmap for defending Web3 teams from phishing, account takeovers, and everyday security failures.
Author:
Seth HallemA few weeks ago, one of our employees at Certora fell victim to a minor hack that was quickly addressed, and that incident prompted me to reflect on my first eight months in Web3 and what I’ve learned about the differences between security in the DeFi space versus the wider tech industry.
On the one hand, it is incredibly exciting for a cybersecurity expert like myself to join an industry that is literally on the bleeding edge of applied cryptography. It’s inspiring to see the level of sophistication in our industry and how advanced cryptography is being used to guarantee safety in a trustless system. At the same time, many of the best in our industry have fallen prey to the simplest attacks: phishing efforts that exploit vulnerabilities in communication tools like Telegram, Google Workspace, X. In the best case, falling prey to phishing is embarrassing, and in the worst case it’s very costly.
To avoid these failures, there are several essential OpSec lessons that are well known and widely followed outside of the Web3 world that we would be wise to learn from.
Like many companies in Web3, Certora is on a long OpSec journey. When I joined the company in June, multi-factor authentication (MFA) was the only security measure enforced across the company. As many hacks in the industry have illustrated, MFA is necessary but it is by no means sufficient on its own.
From that starting point, which I suspect we share with many companies in the space, we have been progressively improving our OpSec. Our experience only emphasizes the urgency of our security roadmap. We see a constant, daily flood of phishing attempts directed at our company through all available communication channels, and both vigilance and a strong OpSec discipline are essential. In the hope of guiding others to take immediate action, I thought it might help to walk through my OpSec Top 5. We are still working through these steps while we help more and more of our clients do the same. I would recommend this same roadmap to anyone starting from a similar position.
As noted above, MFA is necessary but not sufficient as an OpSec strategy. That said, if you have not yet implemented MFA you are making a grave mistake. Changing course had better be the first thing on your agenda as soon as you stop reading this article.
Not all MFA is created equally. I recommend the following:
Once you have MFA in place, you are ready to move on to the next step. However, before you declare your MFA journey a success, make sure you haven’t forgotten any of your communication tools along the way. In this industry we often use a combination of X, Signal, and Telegram, and each of them can and should be protected with an additional authentication factor.
As noted above, we use SentinelOne at Certora. One of our clients uses Sophos, and we both purchased and configured that Sophos system on behalf of our client. Others use Cynet, Crowdstrike, Palo Alto, or other alternatives on the market. While the MITRE attack evaluations are the gold-standard of EDR testing, it is far more important to have something in place and configured correctly than it is to have the best possible EDR according to MITRE.
One common gap in EDR implementations is that EDRs require special permissions on Mac computers. MacOS requires that a security extension is granted permission by an admin user in order to effectively protect against malware. Disk scanning requires further permissions, which must again be granted explicitly by the admin user. For those EDRs that protect against malicious browsing, there is a network extension that must be granted appropriate permissions as well. The installation process cannot force these permissions changes - they must be done manually by each user - and this can be quite a challenge with users who are less technical or may not be comfortable in the depths of the MacOS Settings app. However, without these permissions, your EDR is essentially useless on Mac devices.
It is critically important that across all platforms you use (Windows, Mac, Linux) your EDR is properly configured, has all required permissions, and you use the server-side interface to verify that all endpoints have the requisite green checkmark (or similar) indicating that protection is in effect. Don’t learn this lesson the hard way - make the time to work patiently with each user to get EDR setup and configured in the right way, then double check your work on the server side.
Adding an enterprise password manager is not a nice-to-have. It is an absolute must. Why?
First, the best practice in password security is to have your users remember a single, complex password and to never use that one password anywhere other than as the “master” password for their password manager. It is reasonable to ask users to remember one password that is never stored online (an offline backup can be in a safe deposit box or similar). But it is not reasonable to ask users to remember more than one such password.
If you choose a well known enterprise password manager with a strong security track record, you can be confident that they follow industry leading practices for protecting your password. Other sites may not be as careful. Use your good, memorized password as your master password, and do not reuse this password anywhere else. Ever. Also, do not store it in Google’s password manager in Chrome (or any browser). This password lives in your head and in an offline, locked, safe location.
Second, password managers insulate your users from poor password handling by 3rd party sites and they isolate the impact of a phishing attack. Users cannot remember an infinite number of complex passwords. The inevitable result is passwords that are re-used, either identically or close to it. If one such password is stolen, a smart attacker has what they need to break into every 3rd party site you use if you have reused that password. MFA helps here, but as noted above phishing attacks are getting better at coercing users into MFA mistakes. Secure, randomized passwords managed by a password manager ensure that a single successful phish has a single, localized impact.
Third, password managers allow for controlled, permissioned password sharing. Password sharing is NEVER a good idea for “hot” accounts, meaning the accounts that you use for daily work. However, many 3rd party SaaS applications have a root account. The root account should always be a “cold” account, but it still has credentials that have to be recorded and shared with a select group of company admins. Password managers are great for this purpose: they offer convenient, secure sharing capabilities. At Certora, we adopted 1Password. Some of our clients use Bitwarden. We have experimented with NordPass. All have their tradeoffs, but if you ask me to make the call, I think that 1Password is the most comprehensive solution on the market.
Once you have your password manager in place, take the extra step of resetting all your passwords to a secure, randomized, and managed password.
IT departments the world over have long known that you should never work out of an admin account. Email, web browsing, and pretty much anything else that you do during daily work does not require an account with permissions to install/uninstall software or to make system-wide configuration changes. When you work out of an admin account, you are one click away from a major disaster. If you work out of a regular user account, you at least have a second chance to reconsider when a password prompt appears after you click where you shouldn’t have.
Unfortunately, most users still do work out of admin accounts when using personally purchased and configured devices. It is easier to set up your system that way, but it is a terrible, awful, and dangerous practice. It is essential that all users create a standard user account with a unique password and migrate to that account for daily work ASAP. Allow yourself and your employees to make a few mistakes without triggering the next disaster.
Once you have made this change, educate all your employees that an admin password prompt is a red flag in almost all circumstances. Unless explicitly guided otherwise during software installation or maintenance, stop at the password prompt, cancel, and ask for a second opinion before proceeding.
An increasingly common and dangerous attack vector is to use OAuth/Open ID to allow a malicious 3rd party application to access a company account. Most services (e.g., X) only allow OAuth connections to be initiated from a root account. The issue, though, is that users often work out of a root account. If you have an X window open in your browser logged in via the root account to your company X account, a malicious 3rd party application is now one errant click away from an X account takeover. The same applies to many other 3rd party services. Never work out of root accounts. Instead, follow the principle of least privilege.
When you set up a new service (X, Cloudflare, AWS, etc.), create the account using a root account that you will keep cold. Nobody ever logs into this cold account unless absolutely necessary to perform a function that cannot be performed any other way. Otherwise, the root account credentials (and a TOTP for MFA) are stored in the password manager and shared with a limited group of company-wide admins who are similarly educated that this account is a cold account.
Individual accounts are created with the privileges that each user requires to accomplish their daily work. X has delegate access. Cloudflare allows for an unlimited number of user accounts whose permissions can be set at the domain level. AWS has IAM Identity Center for exactly this purpose. With few exceptions, almost every business critical application that you use can be configured in this way. If you have not done so, you are asking for trouble.
While you are on your top 5 journey, and perhaps even after you have completed all the steps, things can still go wrong. Phishing attacks are increasingly sophisticated, and they can fool both your users and your EDR. At some point in time, something will go wrong. Here is what you do:
This article could probably go on for another 10 pages, but perhaps the first and most important lesson that you should learn about operational security is that the inertia of doing nothing is your most formidable foe. Start with the top 5. If things go wrong, respond immediately and effectively as I have outlined above. Attackers are always looking for the path of least resistance. Do your best to make sure that path is not you.
If you hit the top 5 are you done? Not really. There is more to go in our roadmap at Certora, and I would recommend the same for any company: