Talk notes: Ignite Talks (#DevOps Day)

DevOps Day, Santa Clara, CA
June 25, 2010

I'm here to participate on a panel called "DevOps Outside of Web Operations."

Ignite Talks

5 minutes, all slides advance every 30 seconds

• Adam Rosien (@arosian)
  • "undeployed code == wasted warehouse space"
  • deploy canaries: self test
  • auto rollback, exponential deploys
  • commit messages, deploy services
  • zookeeper, json, collectd, nagios, hudson, ant, rpm, yum, jcollectd, rrrdtool, type-systems, jmx, rabbitmq, esper
  • "splunk lite": all exceptions in the last 30 minutes
  • we're hiring: jobs@kaching.com
• Alex Honor, DTO Solutions: deployment tool chains: http:/google.com/group/devlops-toolchain
  • dev: specify packages they need in operations
  • operations: perform release and deployment
  • dev need: app code, configs, third party packages
  • release managers: issue tracking status, QA approval, change control sched, promote the artifacts
  • ops need: assess scheduling conflict, decide how to batch updates, deploys packages
  • managers need: responsibilities and boundaries, enforce process through auth, audit and trace changes
  • need: self-service: each role has service service
  • keep it simple: use freely available tools, reflect roles and process, easy to understand
  • tool: the "meta" package: coupled set of packages
  • tool: yum repository: central storage and index, resolve and install package dependencies
  • tool: source is in SCM: developers have commit access
  • tool: CI job; devs modify/run job when desired
  • tool: runbook jobs: promote by release manager and deploy for ops admin
  • process: specify package needs
  • audit compliance
• The Cloud is a Trendy Mainframe: Erica from Bitnami: Erica Brescia
  • 1956: $23K/month for 4.17MB of storage
  • Now: $0.10/month for 1GB
  • Backup: then: tape
  • Now: Amazon EBS
  • I/O: 1.44 MB vs. ship storage device to Amazon.com
  • CPU
  • Distribution: acoustic modem vs. instant
  • Hardware: messy heat/power vs. AWS, Joyent, etc.
  • Price: Commodore PET for $2.7K @ 1MHz vs. 1 GHz w/1.7GB @ 9.5 cents/hr
  • App deployment: BitNami creates prepackaged builds
• Really fast moving software, Clint Byrum, Canonical Ubuntu Server team: http://fewbar.com: clint@ubuntu.com
  • API Contracts: they used to be stable, dependencies were few and loosely coupled
  • how do we keep our sanity? test coverage: continuous integration, automatic dependency resolution
  • Reality check: if you don't have the right culture, you don't have it
  • the good: new versions are typcially more stable and faster
  • the bad: stuff breaks, repeated integration cost
  • case study: libmemcached: 0.31 v 0.40 releases
  • MongoDB: Ubuntu included a version, rapidly judged "don't use"
  • So why do we bother packaging? for predictability, so you know what's on the OS: like Southwest Airlines: one type of jet
  • What can authors do: "stop breaking your APIs! You're killing us!"
  • What can distributors do? Cry.
  • What is better way? Something sysadmins already do
  • Not "no more handwaving"
  • Leverage core compentency
    • Python people don't want distro python
  • Rather than build
    • PPA or "personal package archive"
    • launchpad service to easily build packages and then deploy them easily
  • Author participation: get involvement early
  • Derivatives made easy?
• Metrics Simplified: Mark Lin, mlin@admob.com
  • hard
  • bottlenecks: ops need to do all of this; tough on ops people
  • graphite, mabbitmq, graphite local proxy
  • path to graph
    • now developers implement metrics before ops even asks (cool)
  • graph = post event forensics
  • rocksteady, metric as event
  • revelation
  • beyond simple metric
  • what we learned
    • make metric sending simple
    • nice UI to make sense of data
    • real time processing of metric rock