Archive for the ‘ec2’ Category

Guys from EC2 announced micro instances – it costs 2 (two) cents per hour for linux and now it’s will costs less than traditional dedicated hosting with root access – monthly payment for EC2 micro instance will be about 15 USD, and price for root/linux on dedicated hosting will be about 30 USD/month. It’s really good news – you can have 100 boxes cluster just for two usd per hour! Bad thing is that micro instances don’t have their own disk space – EBS only, looks like this best ever use case for this type of instances will be highly-distributed computational grid with all data stored in RAM. And don’t forget that EBS will costs you some money – $0.10 per allocated GB per month Amazon EBS also charges $0.10 per 1 million I/O requests you make to your volume . Fredrick Poller’s already check out micro instances performance by sysbench : Amazon EC2 Micro instance, how fast is it?.

Short how-to about configuring Ubuntu to work with EC2 API ( for Ubuntu 10.04 users – Sun JDK was moved to partner repository so please check out Where’s sun JDK on Ubuntu 10.04 ? before follow described steps ).

Continue reading ‘Configuring Ubuntu ( 9.10 ) to work with Amazon EC2’ »

Update in Amazon Web Services:

  • 2 high cpu instance types : 64 bits – Double Extra Large с 34.2 GB RAM, and 13 ECU (4 virtual cores *3.25 EC2 compute Unit=ECU), 64-bit platform
    and Quadruple Extra Large – 68.4 GB of RAM/ 26 ECU (8 virtual cores* 3.25 ECU) : New EC2 High-Memory Instances
  • Instance prices changes ( us-east is still cheaper thatn eu-west ) : Amazon EC2 – Now an Even Better Value
  • New service for relational DB ( provisioning, scaling and other nice things ) : Introducing Amazon RDS – The Amazon Relational Database Service
  • Security stuff : Vulnerability identified in Amazon’s cloud computing
  • Amazon EC2 – Ubuntu at google groups
  • 5 years ago Amazon announced Amazon Simple Queue Service – top points of AWS for last 5 years
  • * Cloud Computing makes your IT excessively dependent on the Internet
    * Cloud Computing will attract clients mainly from Western markets
    * Cloud Computing makes you dependent on the goodwill of your ISP
    Continue reading ‘10 reasons why cloud computing is bad idea’ »

    Here’s some steps which I frequently use to configure Ubuntu to work as EC2 manager
    Continue reading ‘Quick steps to configure Ubuntu for with EC2 API’ »

    Sun Grid Engine’s top engineer Richard Hierlmeier wrote article ( and some bash scripts which implements it – btw why you not to put it them onto your cvs? ) about using SDM in compute cloud ( here’s EC2 as example, I suppose that GoGrid can be used also without too many changes ) – Using SDM Cloud Adapter to Manage Solaris Zones.

    Sun released new version of Sun Grid Engine – 6.2 Update 3. That’s new:

    upd. Also there’s new Sun Studio 12 Update 1 is available too.

    Recently I found another strange thing in EC2 – if I use “shutdown now” on ec2 instance for AWS this node will be still alive and in marked as “running” in ec2-describe-instances host list. For those who want to have simple way to shutdown ec2 instances from script I recommend to use this code ( I use it in our Convergence – Data-Aware routing on EC2 project ) :

    ec2-describe-instances | grep `hostname` | awk "{ print \"ec2-terminate-instances \" \$2; }" >> /root/tools/stop_it.sh

    You may just run stop_it.sh or schedule it via at

    at -f /root/tools/stop_it.sh "now + $TIMEOUT minutes"

    just not forget to replace $TIMEOUT with something usefull 🙂

    As base AMI i used ami-7db75014 – it’s OpenSolaris supported by Sun, common informartion about installing and using OpenSolaris in EC2 also available in Sun’s Amazon EC2 Getting started guide – in this post I will focus almost in SGE using in Amazon EC2. As SGE distributive i use all-in-one tar package – i choosed “All supported platform” in Grid Engige download page – it takes about 350 Mb, but I don’t worry about platform architecture – if sun support it – it will be in this package. This ge62u2_1.tar.gz contains bunch of other tar.gz’s ( and even hedeby’s core package ) and can be unpacked by :

    root@ec2-server:~/tools/archive# gzip -dc ge62u2_1.tar.gz | tar xvpf –

    So I just go inside ge6.2u2_1 and unpack them all using something like this

    for myfile in *.tar.gz
    do
    gzip -dc $myfile | tar xvpf –
    done

    One important thing – hedeby-1.0u2-core.tar.gz contains old versions of some files from ge-6.2u2_1-common.tar.gz – there’s conflicts in files common/util/arch  and common/util/arch_variables – here’s diff for them – may be sometimes it can be usefull, but for my configuration it causes very strange errors when I try to install executor host :

    value == NULL for attribute “mailer” in configuration list of “ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com”
    ./inst_sge[261]: Translate: not found [No such file or directory]
    ./inst_sge[263]: Translate: not found [No such file or directory]
    ./inst_sge[264]: Translate: not found [No such file or directory]

    When I replace this files from ge-6.2u2_1-common.tar.gz installation works as expected. Next point it’s DNS configuration – SGE is very picky to DNS and it will cause some problems in running SGE Amazon EC2 instances with SGE, this stuff can be fixed using host_aliases file in SGE, or other way it’s to use /etc/hosts file for it – some kind of this technique used in Hedeby-SGE on Amazon EC2 demo, for example if we have master this name and 2 executor hosts I put this lines into /etc/hosts :

    #internal_ip external_full_name external_short_name internal_full_name internal_short_name
    10.yyy.xyz.zzz ec2-RRR-TTT-ZZZ-YYY.compute-1.amazonaws.com ec2-RRR-TTT-ZZZ-YYY domU-mm-ww-PPP-WWW-FFF-GGG.compute-1.internal domU-mm-ww-PPP-WWW-FFF-GGG
    10.yyy.qwe.ttt ec2-aaa-bbb-ccc-ddd.compute-1.amazonaws.com ec2-aaa-bbb-ccc-ddd domU-mm-ww-JJJ-HHH-DDD-SSS.compute-1.internal domU-mm-ww-JJJ-HHH-DDD-SSS
    10.yyy.pre.ppp ec2-yyy-rrr-eee-qqq.compute-1.amazonaws.com ec2-yyy-rrr-eee-qqq domU-mm-ww-UUU-III-OOO-PPP.compute-1.internal domU-mm-ww-UUU-III-OOO-PPP

    Also I use hostname ec2-RRR-TTT-ZZZ-YYY ( external_short_name )  to set instance hostname – this names I use as hostnames when I configure SGE.

    There’s very good erlang library with email services – erlmail. Unfortunately there’s no support to authorization in it – you can easy add this feature to library by this patch ( erlmail with authorization )
    Here’s an example of using erlmail to send e-mail:

    send_mail(UserEmail, Data) ->
        {ok, Pid} = smtpc:connect(“smtp.server.com”, ?SMTP_PORT),
        smtpc:ehlo(Pid, ?SMTP_SERVER),
        smtpc:auth(Pid, ?MAIL_LOGIN, ?MAIL_PASSWORD),
        smtpc:mail(Pid, ?MAIL_FROM),
        smtpc:rcpt(Pid, UserEmail),
        smtpc:data(Pid, Data),
        smtpc:quit(Pid).

    send_mail(UserEmail, MessageSubject, MessageBody) ->
        Data = “From:” ++ ?MAIL_FROM ++ “\r\n” ++ “To:” ++ UserEmail ++ “\r\n” ++ “Subject:” ++
    MessageSubject ++ “\r\n\r\n” ++ MessageBody,
        send_mail(UserEmail, Data ).

    test() ->
        send_mail(“user@bokov.net”, “this is subject”, “this is message body!”).

    Amazon announced that Windows Server ( 32 and 64 bit ) will be available as host OS for Amazon EC2 instances. Running instances can be accessed via Remote Desktop. Currently this feature available for private beta ( Windows Server and SQL Server ) only – and public release sheduled before the end of 2008. You may provide your contancs to Amazon to be notified when this product will be available.

    Guys from Amazon posts in Amazon Web Services Blog interesting document – AWS Security White Paper.
    Main points –

    • there’s no backup for data ( EBS, S3, anything), but all data redundantly stored in multiple physical locations
    • for EC2 they have four security levels – host OS ( access : only AWS administrators ),  guest OS ( access : only customers, AWS admins can’t log onto guest OS ),  Firewall ( indirectly configured by customers, AWS admins also have access ) , Amazon EC2 API ( access : only customers ).
    • all guest OS running by hypervisor XEN – so instances have no direct access to hardware resources, and can’t read-write any data which owned another instances – including network packets, disk devices and memory.
    • network traffic ( for different instances ) can not be sniffed, external DDoS secured by firewal, port scanning inside network prohibited by Acceptable Use Policy ( customer will be blocked for this, I suppose ), instances can’t use IP spoofing ( because of hypervisor Xen :-). Anyway, Amazon reccomend to use SSL for network connections.
    • Security in S3 and SimpleDB based in ACL ( access control list ) – only data owner may edit access permission.
    • All data in S3 and SimpleDB stored ‘as is’ without any encryption. If you want to be more secure – you should store encrypted data – “Encrypting before sending to SimpleDB guarantees that no party, including AWS, has access to sensitive customer data”.  Once you delete data from S3 or SimpleDB all external links to them will be unaccessible – “area is then made available only for write operations and the data is overwritten by newly stored data”.  S3 and SimpleDB nodes use SSL for network connection, so there’s no chances for Man-In-The-Middle attack.