Questions We Should Be Asking About EOS Block Producers

Running block production nodes is no different than operating any other sort of datacenter infrastructure. As a datacenter veteran, I’ve got a long list of concerns regarding the trustworthiness of EOS block producers and block producer candidates.

For context, here’s a summary of things I’ve done and which disciplines of engineering/tech I’ve spent my time in over the last 10 years:

• Datacenter operations — SysAdmin, Networking, Network & Systems Security, Incident Response and Forensics, Project & Deployment Management
• Systems Engineer || Systems Architect — I’ve built, secured, automated, scaled, and operated large systems of software and infrastructure (and teams) for a variety of companies…both in the cloud, and on bare-metal. This includes 24/7 on-call duties as the Infrastructure/Ops lead, 365 days a year.
• Security, Compliance, and Governance — I’ve successfully run multiple attestation cycles for HIPAA, PCI Level 1, and SOC compliance for colocated datacenters as well as cloud-based software companies/service providers, essentially serving in the functional capacity of a CISO without the title.
• Software Engineering — Over the years, I’ve helped write, deploy, and architect various applications and tools in Ruby, JavaScript, Clojure, Java, Scala, some C++ (a long time ago), and some Python. Some have been client-side, some have been back-end, some have been for desktop and mobile. This includes blockchain-y things.
• BlockChain — As an engineer, I’m not a blockchain newb. I’ve spent significant time with architecture, code, and implementation for EOS (and Ethereum) at nuts and bolts levels.
• DevOps — All those things I listed out up there ^^ fold nicely into the cross-functional amalgam that comprises the responsibilities delegated to a DevOps Engineer. I’ve been Lead DevOps/Director of DevOps in multiple prior roles. It’s fun.
• Technical Leadership & Entrepreneurship in BlockChain — I serve as CTO and co-founder for two projects that we’re currently incubating and developing, and am advising a few others. Both of my projects are “on the blockchain.” One is a tokenized implementation, and the other (surprisingly) is not. I’ve been learning what it’s like to spin up a technology startup from scratch, and I spend most of my free time thinking about architecting blockchains, governance, tokenomics, and the like.

Now that the preening, posturing, egotistical stuff is out of the way, I hope I’ve at least established myself as a credible enough individual that you’ll care about the questions being raised here.

“BlockChain” is *NOT* a Magical Security Elixir

Let’s skip the contextual, theoretical, and philosophical stuff that we’d have to discuss. This article assumes that you’re sufficiently knowledgable in the subject space. If you’re not, I’ll try to keep it as simple as possible.

There exists a prevalent attitude in this new and growing technology space that taking any old web technology/service/data and “putting it on the blockchain” somehow, magically, makes it more secure or accountable.

Why?

…because of tokenization, decentralization, and tokenomics, bro. It’s secure because it’s decentralized.

Taken in these reductionist terms, this notion is abjectly false.

Just because you were able to compile and run an EOS testnet node on your MacBook, on an EC2, or in a Docker Container, it doesn’t mean that operating EOS nodes as an elected block producer is the same thing. You can’t just set up a bunch of machines in a datacenter somewhere, flip the EOS switch on, and walk away.

I know Vitalik has argued that the 21 elected block producer architecture might constitute a plutocracy, but I don’t want to get into that discussion in this post. Let’s just assume that we’re on the “block-producers-don’t-represent-a-central-trust” side of the EOS architecture and governance debate.

Under this assumption, block producers are — I think — ethically obligated to handle themselves and maintain block producer infrastructure in EXACTLY the same way that Stripe, BrainTree, Kaiser Permanente, Equifax, and other audited/compliance-constrained entities are expected to operate their software and infrastructures. In fact, I’m betting confidently that we’ll eventually see (in the next 5–10 years) some new, hybrid compliance framework(s) emerge that apply specifically to people operating blockchain infrastructures where sensitive information is being transacted over the network.

Whether or not a given network is operated by a decentralized body of users or block producers, it is not unreasonable to hold the operators to some standard.

Now, you might be thinking, “…holy sh*t dude, that’s kind of harsh. Who are you to say that, anyway?”

EOS Nodes will be running on servers, or in containers, or on Virtual Machines. Since latency reduction and performance are important factors, it’s safe to assume you’ll get the best performance out of baremetal implementations (at least for now), and I’d bet that we’ll see BP’s slide over to containerized approaches in the coming months. These servers all come with tons of attack vectors and exposure surfaces — this is an implicit ground-truth.

Why? Because the EOS node software will need to run within an operating system, wheresoever it resides.

When contemplated through this contextual frame, block producers are really just another example of boring organizations that operate public web services for others, at scale. Taken in its most reduced terms, it’s the same as running a bunch of MySQL servers, and letting people access the MySQL service through the infrastructure you’re hosting.

So, thinking along these lines, it’s perfectly reasonable to assume that some group of people could be running a cluster of MySQL servers, and that those servers can be hacked — not by any fault or inherent security vulnerability of MySQL’s — but due to poor security controls, bad administration, outdated OS-level packages with vulnerabilities, etc. on the part of the systems and people hosting that MySQL infrastructure.

You could argue that even if an intruder gained access to the system, that MySQL can be protected from an intruder. This is true…sorta. One can escalate their privileges on a system if they’re able to get in. Worse yet, an intruder may have gained root privileges outright.

If you’ve got root (or successfully escalate your privileges to root) on a box running a service like MySQL, there’s nothing that can save that service. At minimum, an attacker can simply wreak havoc on the service itself by downing the host. That, however, would be the least interesting thing an attacker could do.

If the group managing a block producer infrastructure are not absolutely SURGICAL about the ways in which they handle credential and KEY materials, they are — no matter what — going to get hacked. It’s not a matter of “if”, but “when.’

If you need examples of “unimaginable” security disasters, you don’t have to reach too far back in time. Think back to HeartBleed that affected SSL, the discovery that OpenSSH had been vulnerable in the worst of ways (for years), the massive Equifax hack, the MyEtherWallet DNS hijack we saw just a few weeks ago, the handful of social engineering hacks we’ve seen in the last couple of years that managed to bypass even Apple’s multi-layered and stringent security safeguards….and the list goes on.

So, what should we be asking block producers and block producer candidates?

• How are your administrative credentials secured?
• Does there exist sufficient hardware redundancy at each of the individual tiers of the infrastructure?
• What type of bandwidth, uplink, and power SLAs are provided by the datacenter/hosting providers where the hardware is running/colocated?
• Is the block production infrastructure geo-distributed? Is there any infrastructure redundancy at the level of having hardware running at more than one site?
• Do you know where every single copy of your key and credential materials reside, and are they vaulted safely?
• Have you built and tested your network ACL’s to make sure no one can get in to the systems where the nodes are running?
• Are the systems in which keys and credentials stored (even in vaults) secured and maintained hygienically? For example: Someone on the team pasted an important key into their terminal. Is their BASH history erased on some regular interval, or better yet, set up in order not to save history to disk?
• Have you enabled and implemented sane firewall rules on the network gear and on individual instances to prevent having some local service like SSH exploited? Let’s not forget about the OpenSSH debacle of 2016 that technically affected millions of servers going back many, many years.
• Is your network gear locked down tightly enough to ensure that no one can get into your switching/routing infrastructure and poison traffic packets or mess with traffic routing?
• Is there any application or use of DNS (internal or public-facing) at any of your network layers that may introduce the possibility for a massive traffic redirection attack? What architectural considerations are in place in order to mitigate or detect tampering/attacks on DNS?
• Which operating system has been selected for running your EOS nodes?
• Which version of that operating system is in use?
• How regularly does it receive updates? Why?
• Does there exist (at minimum) a dual-control scheme for the handling, storage, and management of your security credentials for operating the servers, the network gear, and the EOS block producer nodes?
• Is deployment and administration of infrastructure taking place by hand, or in some automated fashion? Is your automation using sufficiently secure strategies for deployment and configuration — that is to say, are you passing keys, passwords, or other administrative credentials in plain text?
• How are you handling patch management and vulnerability tracking for the systems and infrastructure you’re running your EOS block producers on?
• How quickly are 0-day vulnerabilities for things like OpenSSH (for example) being patched on your systems?
• How quickly and how regularly is the firmware for your network gear in your datacenter or headquarters being updated?
• Has the organization that is administering the block producer infrastructure undergone any type of penetration testing or external audit?
• Has the organization that is administering and operating the block producer infrastructure conducted ANY type of risk assessment, in order to know where any potential attack vectors might exist? All security-sensitive organizations (should) conduct risk assessments so that they can identify where any potential trouble might occur. Not EVERY attack surface is as likely to be exploited as others. Companies do this so that they can clearly identify where risks lie, and then assign weight to these risks, and based on weight/likelihood of attack, either choose to accept or decline those risks. Declining the risk means architecting security around the risk. If accepting the risk, it’s at least good to have a reference of potential attack surfaces you might want to investigate when you’re inevitably compromised.
• If you’ve answered “yes, mostly, kind of, i don’t know” or “it’s none of your business” to any of these questions — then the final question to ask is: Have you had these controls and measures tested and validated by an external, 3rd party? If not, would you be willing to?

…so, now what?

I urge you all to stop and think about these things, and to ask these questions to the block producers you vote for, elect, and keep elected.

It’s easy to get carried away in the excitement and allure of the blockchain frenzy we’re all in. If we want to replace traditional systems of information with blockchain-backed systems of value, then these new systems of value — made possible by blockchain technology — need to stand up to the same questions and scrutiny that those traditional systems have. Until such time that we can answer/solve these questions, we can’t factually assert the security superiority of a blockchain like EOS.

Moreover, if we hope to play nicely with “traditional” sectors, investors, markets, governments, companies, and technologies…we’re going to have to speak their traditional language, and answer their traditional questions. Eventually, we’ll all be speaking the same language, and the pointless questions that don’t apply to what we build will no longer be relevant, or asked.

While it is DEFINITELY true that blockchain, by its innate architecture, obviates many of the security concerns that could plague traditional systems, it would be foolish to think that blockchains architected like EOS obviate them all.

Why?

Because, they don’t. An EOS node exists within a system of systems that all commingle their vulnerabilities into one systemic fabric. If the system where the node runs is vulnerable (or the system of systems in which that system runs), then there exists the distinct possibility that the node itself can somehow be exploited. Given the decentralized nature of the network, we operate with the hope and confidence that a single misbehaving node in the network will be identified and ignored as consensus is being achieved and maintained between all involved parties…but you can never be too safe. We’re talking about building financial systems, health-record systems, and deed/title registration systems on these blockchains.

IF we’re hoping for any such type of mass-adoption blockchain “revolution,” then we should be taking it as seriously as we can, and applying as much of our battle-tested and proven security methodologies as we can.

If you’re a block producer candidate, I ask you to stop and consider the implications of the questions I’m raising here. If you’re worth your salt as an engineer, you know that arguing “well, the data’s encrypted and hashed on the chain, and the chain is immutable, so it’s ultimately mostly safe…” isn’t really a sufficiently compelling argument to make.

If you’re voting for block producers right now, I ask you to take the time to read through any and all material you can on the block producers you’re electing to see if their team possesses relevant background, and if the architecture they propose or lay out indicates a sufficient level of security and best-practice considerations.

If you’ve got questions, need advice, would like consultation on security, infrastructure, or architecture…or if you just wanna chat — drop me a line:

email: me@rmnr.net

linkedIn: https://www.linkedin.com/in/armenr/

telegram: @Rmenr