Alexey Bokov's weblog » ec2

Amazon EC2 announced micro instances – 2 cents/hour

Alexey Bokov — Fri, 10 Sep 2010 11:41:59 +0000

Guys from EC2 announced micro instances – it costs 2 (two) cents per hour for linux and now it’s will costs less than traditional dedicated hosting with root access – monthly payment for EC2 micro instance will be about 15 USD, and price for root/linux on dedicated hosting will be about 30 USD/month. It’s really good news – you can have 100 boxes cluster just for two usd per hour! Bad thing is that micro instances don’t have their own disk space – EBS only, looks like this best ever use case for this type of instances will be highly-distributed computational grid with all data stored in RAM. And don’t forget that EBS will costs you some money – $0.10 per allocated GB per month Amazon EBS also charges $0.10 per 1 million I/O requests you make to your volume . Fredrick Poller’s already check out micro instances performance by sysbench : Amazon EC2 Micro instance, how fast is it?.

Interesting links ( June/July 2010 )

Alexey Bokov — Fri, 11 Jun 2010 09:09:27 +0000

Configuring Ubuntu ( 9.10 ) to work with Amazon EC2

Alexey Bokov — Wed, 11 Nov 2009 09:45:56 +0000

Short how-to about configuring Ubuntu to work with EC2 API ( for Ubuntu 10.04 users – Sun JDK was moved to partner repository so please check out Where’s sun JDK on Ubuntu 10.04 ? before follow described steps ).

1) Remove openjdk, gij
sudo apt-get remove openjdk-*;
sudo apt-get remove gij-*;

2) Install good one java powered by Sun
sudo apt-get install sun-java6-jdk

3) Install EC2 API
cd /home/abokov/

http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip

unzip ec2-api-tools.zip

4) Configure environment : add this lines onto the end of ~/.bashrc
#ec2 deal with stuff
export JAVA_HOME=/usr/lib/jvm/java-6-sun
# setting up Amazon EC2
export EC2_HOME=/home/abokov/ec2-api-tools-1.3-30349
export PATH=$PATH:$EC2_HOME/bin:
export EC2_PRIVATE_KEY=/home/abokov/keys/pk-XXX.pem
export EC2_CERT=/home/abokov/keys/cert-XXX.pem

export EC2_ACCESS_KEY_ID=***
export EC2_SECRET_ACCESS_KEY=***

You may check out how it works using ec2-describe-instances.

Amazon Web Services links – October/November 2009

Alexey Bokov — Wed, 28 Oct 2009 11:11:26 +0000

Update in Amazon Web Services:

2 high cpu instance types : 64 bits – Double Extra Large с 34.2 GB RAM, and 13 ECU (4 virtual cores *3.25 EC2 compute Unit=ECU), 64-bit platform
and Quadruple Extra Large – 68.4 GB of RAM/ 26 ECU (8 virtual cores* 3.25 ECU) : New EC2 High-Memory Instances

Instance prices changes ( us-east is still cheaper thatn eu-west ) : Amazon EC2 – Now an Even Better Value

New service for relational DB ( provisioning, scaling and other nice things ) : Introducing Amazon RDS – The Amazon Relational Database Service

Security stuff : Vulnerability identified in Amazon’s cloud computing

Amazon EC2 – Ubuntu at google groups

5 years ago Amazon announced Amazon Simple Queue Service – top points of AWS for last 5 years

10 reasons why cloud computing is bad idea

Alexey Bokov — Wed, 12 Aug 2009 10:07:03 +0000

* Cloud Computing makes your IT excessively dependent on the Internet
* Cloud Computing will attract clients mainly from Western markets
* Cloud Computing makes you dependent on the goodwill of your ISP

* Cloud Computing can expose you to the unethical practices of your ISP
* Cloud Computing is against the spirit of Personal Computing
* Cloud Computing makes your Cloud Data subject to American law
* Cloud Computing can expose your Confidential Data to “Corrupt Elements”
* Cloud Computing is sounding more like a Lobby than a Trend
* Cloud Computing may be of little consequence for the Average Small Business
* Cloud Computing may not contribute to your national economy

Found it here : Ten Reasons Why Cloud Computing is a Bad Idea by Hamad Subani.

Quick steps to configure Ubuntu for with EC2 API

Alexey Bokov — Fri, 17 Jul 2009 11:28:00 +0000

Here’s some steps which I frequently use to configure Ubuntu to work as EC2 manager
1) Remove all non sun’s java – openjdk, gij

apt-get uninstall openjdk-*

apt-get uninstall gij-*

2) And install java from sun
apt-get install sun-java6-jdk

3) Install EC2 API
cd /home/abokov/

http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip

unzip ec2-api-tools.zip

4) Configure environment : I just add lines below to my ~/.bashrc
#ec2 deal with stuff
export JAVA_HOME=/usr/lib/jvm/java-6-sun
# setting up Amazon EC2
export EC2_HOME=/home/abokov/ec2-api-tools-1.3-30349
export PATH=$PATH:$EC2_HOME/bin:
export EC2_PRIVATE_KEY=/home/abokov/keys/pk-XXX.pem
export EC2_CERT=/home/abokov/keys/cert-XXX.pem

export EC2_ACCESS_KEY_ID=***
export EC2_SECRET_ACCESS_KEY=***

It’s should be ready – you may check it out by run in terminal :

ec2-describe-instances.

Using SDM to managing OpenSolaris zones

Alexey Bokov — Tue, 07 Jul 2009 08:03:44 +0000

Sun Grid Engine’s top engineer Richard Hierlmeier wrote article ( and some bash scripts which implements it – btw why you not to put it them onto your cvs? ) about using SDM in compute cloud ( here’s EC2 as example, I suppose that GoGrid can be used also without too many changes ) – Using SDM Cloud Adapter to Manage Solaris Zones.

New version of Sun Grid Engine

Alexey Bokov — Tue, 23 Jun 2009 09:52:32 +0000

Sun released new version of Sun Grid Engine – 6.2 Update 3. That’s new:

Amazon EC2 adapter – SDM Cloud Service Adapter now is avaialable to scale SGE cluster in EC2 – they only need to have OpenVPN installed on instances, and special configuration for IP addresses ( SGE is still very picky about DNS ) – more about restriction is here and here’s bash scripts which used for EC2 deployment. By the way I see that OpenSolaris is still only one OS which available for SGE on EC2, if you’re looking for some SGE/EC2 solutions you may check out Convergence proejct which deal with SGE/GemFire cluster installation on EC2 – see Installing SGE on EC2, or Hedeby installation on EC2.
Now only one JVM can be runned on master or managed host – in previous version SGE ran 3 JVM on every host – one for cs_vm ( configuration service ), executor_vm ( executor component ) and rp_vm ( resource provider ). In SGE terminology it called SDM simple install.
Now SGE have Exclusive Scheduling – this helps to guarantee predictable performance and to avoid interference when a job is not using all of the slots that are available on a host
Sun declared that now SGE have Microsoft Vista Support ( don’t think that there’s too much SGE installations on Vista ) and also some usual marking speech about Power Saving features

upd. Also there’s new Sun Studio 12 Update 1 is available too.

Shut-me-down, EC2

Alexey Bokov — Thu, 28 May 2009 17:59:18 +0000

Recently I found another strange thing in EC2 – if I use “shutdown now” on ec2 instance for AWS this node will be still alive and in marked as “running” in ec2-describe-instances host list. For those who want to have simple way to shutdown ec2 instances from script I recommend to use this code ( I use it in our Convergence – Data-Aware routing on EC2 project ) :

ec2-describe-instances | grep `hostname` | awk "{ print \"ec2-terminate-instances \" \$2; }" >> /root/tools/stop_it.sh

You may just run stop_it.sh or schedule it via at

at -f /root/tools/stop_it.sh "now + $TIMEOUT minutes"

just not forget to replace $TIMEOUT with something usefull

Installing Sun Grid Engine on Amazon EC2

Alexey Bokov — Tue, 07 Apr 2009 17:54:45 +0000

As base AMI i used ami-7db75014 – it’s OpenSolaris supported by Sun, common informartion about installing and using OpenSolaris in EC2 also available in Sun’s Amazon EC2 Getting started guide – in this post I will focus almost in SGE using in Amazon EC2. As SGE distributive i use all-in-one tar package – i choosed “All supported platform” in Grid Engige download page – it takes about 350 Mb, but I don’t worry about platform architecture – if sun support it – it will be in this package. This ge62u2_1.tar.gz contains bunch of other tar.gz’s ( and even hedeby’s core package ) and can be unpacked by :

root@ec2-server:~/tools/archive# gzip -dc ge62u2_1.tar.gz | tar xvpf -

So I just go inside ge6.2u2_1 and unpack them all using something like this

for myfile in *.tar.gz

gzip -dc $myfile | tar xvpf -

done

One important thing – hedeby-1.0u2-core.tar.gz contains old versions of some files from ge-6.2u2_1-common.tar.gz – there’s conflicts in files common/util/arch and common/util/arch_variables – here’s diff for them – may be sometimes it can be usefull, but for my configuration it causes very strange errors when I try to install executor host :

value == NULL for attribute “mailer” in configuration list of “ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com”

./inst_sge[261]: Translate: not found [No such file or directory]

./inst_sge[263]: Translate: not found [No such file or directory]

./inst_sge[264]: Translate: not found [No such file or directory]

When I replace this files from ge-6.2u2_1-common.tar.gz installation works as expected. Next point it’s DNS configuration – SGE is very picky to DNS and it will cause some problems in running SGE Amazon EC2 instances with SGE, this stuff can be fixed using host_aliases file in SGE, or other way it’s to use /etc/hosts file for it – some kind of this technique used in Hedeby-SGE on Amazon EC2 demo, for example if we have master this name and 2 executor hosts I put this lines into /etc/hosts :

#internal_ip external_full_name external_short_name internal_full_name internal_short_name
10.yyy.xyz.zzz ec2-RRR-TTT-ZZZ-YYY.compute-1.amazonaws.com ec2-RRR-TTT-ZZZ-YYY domU-mm-ww-PPP-WWW-FFF-GGG.compute-1.internal domU-mm-ww-PPP-WWW-FFF-GGG
10.yyy.qwe.ttt ec2-aaa-bbb-ccc-ddd.compute-1.amazonaws.com ec2-aaa-bbb-ccc-ddd domU-mm-ww-JJJ-HHH-DDD-SSS.compute-1.internal domU-mm-ww-JJJ-HHH-DDD-SSS
10.yyy.pre.ppp ec2-yyy-rrr-eee-qqq.compute-1.amazonaws.com ec2-yyy-rrr-eee-qqq domU-mm-ww-UUU-III-OOO-PPP.compute-1.internal domU-mm-ww-UUU-III-OOO-PPP

Also I use hostname ec2-RRR-TTT-ZZZ-YYY ( external_short_name ) to set instance hostname – this names I use as hostnames when I configure SGE.