Software Security: Execution Semantics and Buffer Overflows

Topics

  • Basic Execution Model
    • Compiler workflow
    • Register/memory move and addressing modes
    • Control flow
    • Program Memory Organization
    • Functions

Goals

  • Simulate the execution of x64 assembly

Execution Semantics

  • How does high-level code become machine code?
  • What is the basic processor execution model?
  • What are the different register addressing modes?
  • How can you distinguish AT&T vs Intel syntax?
    • AT&T: Look for percent signs, dollar signs, and parentheses
    • Intel: Look for square braces and fewer annotation symbols
  • Memory addressing
    • Endianness
    • How can you read and interpret control flow in assembly?
    • Memory layout
  • What problems do calling conventions solve?
    • Just a convention!
    • Details of System V AMD64 ABI
  • Need to be able to read assembly and simulate execution