Do yourself a FAVAR: security mindset

post by lemonhope (lcmgcd) · 2022-06-18T02:08:47.415Z · LW · GW · 2 comments

Contents

  More examples
    See also
None
2 comments

(We're always in search of a concise and actionable way to communicate security mindset. Here's my attempt.)

Just as a sprocket you don't use never fails and code you don't write has no bugs, an input constraint you don't require is never violated.

Want a secure system? Then DO yourself a FAVAR:

• Describe Objectives: How is the system supposed to function? What constraints does its operation and output have?
• Find Assumptions: What is the system assuming about its inputs, its environment, or itself?
• Validate Assumptions: Are the assumptions we're making always correct? Even if they all are, then ↓
• Remove the ones you can: Of course any incorrect assumptions have to be removed, but it is important to remove as many assumptions as possible. Helicopters land even if the engine dies.

Let's take this transferMoney procedure in a token program or an online bank as a case study:

transferMoney(fromId, fromSig, amount, destId):
	if amount < 0: return
	if not validate(fromId, fromSig): return
	curBal := db.get(fromId)
	if curBal < amount: return
	db.set(fromId, curBal - amount)
	db.set(destId, db.get(destId) + amount)

• Our objective is for total balance to be conserved, and for transactions to only be initiated by the sender.
• Finding assumptions is the hard part. This code actually assumes at least this much:
  • Addition and subtraction do not overflow the numbers
  • (Signature validation is trusted)
  • Nothing else updates the db during transfer's execution
  • The second-to-last line never throws an error
• Validating: We scour the codebase. It seems, hopefully, the numbers come from a trusted source, the function caller locks the records, the database is very reliable, etc. But we notice that if we're wrong about any of these, even once, then a user could probably create or destroy unlimited money. Simply double-clicking the transfer button might actually send 2x the money but only remove 1x!!
  • Note, we could add more checks, try/catches, do more static analysis, etc. Instead, we'll try this:
• Remove assumptions: Instead of being so careful in everything about the function's environment, we can eliminate most of these assumption by writing the code differently!

transferMoneyBetter(fromId, fromSig, amount, destId):
	if amount < 0: return
	if not validate(fromId, fromSig): return
	with db.transaction as tx:
		curBal := tx.get(fromId)
		if (curBal < amount): return
		db.set(fromId, safeSub(curBal, amount))
		db.set(destId, safeAdd(db.get(destId), amount))

Now the system can tolerate untrusted input, a database connection that fails, simultaneous calls to transfer, etc. We reduced complexity instead of increasing it.

When we're trying to make safe AI, it is so so much higher impact to remove an assumption than to add a check or balance.

The hardest parts are finding and removing assumptions, so the full message is "DO yourself a FAVAR, FR!"

More examples

• You can try to make absolutely certain that the back door always has a security guard posted, or you can remove it.
• You can try to make absolutely sure that your password file is protected, or you can just use hashes instead.
• Whitelisting instead of blacklisting is (typically, broadly) about picking things you want from a known set instead of  trying to think of everything you might not want from an unknown set.
• A system that is literally, physically unable to do X requires less careful analysis than one which is somehow otherwise prevented or trained to avoid  X.

See also

• Godzilla Strategies [LW · GW]
• Security Mindset and Ordinary Paranoia

2 comments

Comments sorted by top scores.