Utilizing Amazon S3 and EC2 in Rails

by Jonathan Weiss

Starting Point one machine ruby / rails mysql

Worst Case Popularity

problems vs users
    exponential

difficult path
    migrate from one server to multiple servers... if not made from the very beginning

PROBLEMS

Backup
    high availability
    redundancy
    very big data sets

File system
    important files must be accessed from many servers
    NFS/Samba not practical

Spontaneous Traffic
    slashdoted!
    digged! 

Load Fluctuation
    load variability across different time slots
    when not being used, you're still paying for those resources

SOLUTION: Amazon Web Services?

S3 - Simple Storage Service
    redundant data store
EC2 - Elastic Computer Cloud 
    virtual server per hour

S3
as much as you like max 5Gb per object organized in buckets web-services API for uploads

Buckets
    unique over all S3
    contains many key-valued-metadata tuple
    cannot contain other buckets
    key can contain "/"
    access control

S3 with AWS::S3 ruby package / class for interacting and manipulating S3 objects

S3 object manipulation

upload  
    S3Object.store(...)

download
    obj = S3Object.find (..)

EC2 based on XEN virtualization on demand virtual servers controlled with web service API 1,7GHz 1,75 RAM 160 Gb local file storage (non-persistant!!!)

use your favorite linux distr. (linux kernel 2.6.16)
    amazon machine images (AMI) are stored on s3
ACLs for hosts/ports access control

EC2 Tools

list available images
    ec2-describe-images

start a new instance
    ec2-run-instances ami-xxxxx -k $RSA_KEY

login with ssh
    ssh root@domU-xx-xx-xx-xx-xx.usma2.compute.amazonaws.com

shutdown instance
    ec2-terminate-instance i-xxxxxxx

EC2 with Rails

amazon-ec2 gem

?? How does this solve my problems ??

S3-Backup s3sync.rb brackup jungle disk s3 fuse s3dav duplicity s3Browser Firefox S3 Organizer Extension

    s3sync.rb (the most used)
        modeled from rsync
        backup
            s3sync.rb -v -r -ssl <localdir> <remotedir>
        restore 
            s3sync.rb -v -r -ssl </remotedir><remotedir> <localdir>

S3 Asset Host client static requests are sent to S3 client dynamic requests go to your main host
S3 Authenticated User Data user request something webservice request to s3 hosts replies with a redirect user request the redirect link

    try-it:
        attachment_fu rails plugin

On-demand computing with EC2 load variability along the days/weeks

time-based with cron
        0 18 * * * start_additional_instances.sh
        0 18 * * * stop_additional_instances.sh

Load based with monit 
        tail /etc/monitrc

            check system example.com
            if cp usage (user) > 70% for X cycles then
                exec start_additional_instances.sh
            else-if cp usage (user) < 70% fro X cycles then
                exec stop_additional_instances.sh
            end

client Apache load app server ec2 app server ec2 DB balancer app server ec2 app server ec2 Slave DB app server ec2 app server ec2 S3 monitor

EC2 for extra capacity

    apache load balancer                    app server ec2
    app server                  vpn         app server ec2
    DB                                      app server ec2

    own data-center                         amazon data-center


problems? latency...

Load Balancer / Proxy

    client

    apache mod_proxy balancer

        server      server      server      server
    port8000    port8000    port8000    port8000
    mongrel     mongrel     mongrel     mongrel


swiftiply
    proxy backbend clients
    not much security

        client
        apache

        clusterport 80
        swiftiply proxy
        backendport 8000

        client      client      client      client
        swiftiplied swiftiplied swiftiplied swiftiplied
        mongrel     mongrel     mongrel     mongrel


        install
            gem install swiftiply -y
        start
            swiftiply -c swiftiply.yml
                clustr_address: 0.0.0.0
                cluster_port: 80
                daemonize: yes
                ...

swiftiply Mongrel

    gem plugin that patches mongrel
    transforms mongrel in swiftiply client
    experimental </localdir></localdir></remotedir></localdir>