Alexey Bokov’s weblog

We have git repo at github (  and would like to move onto some our corporate box inside our network, for example on internal.bokov.net. This repository is shared for group of developers and all who have ssh access onto internal.bokov.net shall have right to commits onto git repository. So first of all prepare place for git repo at new hosting :

ssh user@internal.bokov.net
cd var
mkdir git_repo
chmod 777 git_repo
cd git_repo
mkdir my_project
cd my_project
git init --bare --shared=all
cd objects
chmod g+ws *


Next clone existing repo and change its settings to point onto internal.bokov.net

git clone --bare git@github.com:bokov/github_project.git
git remote add -t master -m master origin ssh://internal.bokov.net/var/git_repo/my_project/
git push origin master


to start work with new repo just use

git clone ssh://internal.bokov.net/var/git_repo/my_project/


have fun.

[root@my-box ~]# useradd -g users -u 560 tuser
[root@my-box ~]# passwd tuser

Short how-to about configuring Ubuntu to work with EC2 API

Continue reading ‘Configuring Ubuntu ( 9.10 ) to work with Amazon EC2’ »

Update in Amazon Web Services:

Distributed, non-relational database systems – DNRDBMS – see

• The Future Is Big Data in the Cloud
• NoSQL – the new wave against RDBMS

If anyone interested in it – here’s new update for Sun Grid Engine 6.2 – update 4. It almost about bug fixing and man’s changes – list of changes is here. Sources’ tag for CVS is V62u4_TAG (make sense for Grid Engine, ARCo, SGE Inspect ), by the way as I know Hedeby is still 1.0u3.

• Recursively add files and foldes into CVS

find . -type d -print | grep -v CVS | xargs -n1 cvs add

• Delete “.svn” files from current folder

find . -type f  -exec echo \"{}\" \;  | grep ".svn" | xargs rm

Grid Dynamics nominated “2009 Top 150 Cloud Computing Players” by Cloud Computing Journal. Look at letter G – here we are, around Google. .

Public cloud providers : hybrid cloud vs instance types, API logging vs stability – comparing goGrid and Amazon EC2 .GoGrid is officially out of beta, so now it’s time to compare GoGrid with the public cloud market leader, Amazon EC2. Both vendors have relatively stable performance metrics and they both implement the basic functionality of a cloud provider:

• Programmatic control via REST API
• Scalable data storage solution: Elastic Block Storage in Amazon AWS and CloudStorage in GoGrid
• Ability to assign external IP addresses
• Ability to grant root/administrator credentials
• Many options to configure the cloud cluster
• Windows and Linux support
• Web console for monitoring and management

So far, as cloud providers GoGrid and Amazon EC2 look very similar, but practical experience of using them in real-life projects helps us identify some unique features that can be important for specific applications – for highlighed features and bad points for both providers look onto GoGrid is out of beta on Grid Designer’s Blog

NYC big ideas :

• Realtime data on subway / bus arrivals / locations for use in mobile apps,
• Free WiFi locations in all 5 boroughs
• Mobile app to replace metro cards and kiosks

Sun Grid Engine’s top engineer Richard Hierlmeier wrote article ( and some bash scripts which implements it – btw why you not to put it them onto your cvs? ) about using SDM in compute cloud ( here’s EC2 as example, I suppose that GoGrid can be used also without too many changes ) – Using SDM Cloud Adapter to Manage Solaris Zones.

My way – run sqlplus, spool in using html markup and convert this html onto text by using sed replacements. It works fast enought, and better than parse resulted text by spool without markup. Bash script which impelement it is under below Continue reading ‘How to dump oracle table into tab separated text file ( csv )’ »

strange enough, but in some Ubuntu 9.04 ( 64 bit ) apache2 may tell

instead callin apache2 from /usr/bin use /etc/init.d/ :

Sun released new version of Sun Grid Engine – 6.2 Update 3. That’s new:

• Amazon EC2 adapter – SDM Cloud Service Adapter now is avaialable to scale SGE cluster in EC2 – they only need to have OpenVPN installed on instances, and special configuration for IP addresses ( SGE is still very picky about DNS ) – more about restriction is here and here’s bash scripts which used for EC2 deployment. By the way I see that OpenSolaris is still only one OS which available for SGE on EC2, if you’re looking for some SGE/EC2 solutions you may check out Convergence proejct which deal with SGE/GemFire cluster installation on EC2 – see Installing SGE on EC2, or Hedeby installation on EC2.
• Now only one JVM can be runned on master or managed host – in previous version SGE ran 3 JVM on every host – one for cs_vm ( configuration service ), executor_vm ( executor component ) and rp_vm ( resource provider ). In SGE terminology it called SDM simple install.
• Now SGE have Exclusive Scheduling – this helps to guarantee predictable performance and to avoid interference when a job is not using all of the slots that are available on a host
• Sun declared that now SGE have Microsoft Vista Support ( don’t think that there’s too much SGE installations on Vista ) and also some usual marking speech about  Power Saving features

upd. Also there’s new Sun Studio 12 Update 1 is available too.

The central idea we were working on was this idea of de-localized information — information for which I didn’t care what computer it was stored on. It didn’t depend on any particular computer. I didn’t know the identities of other computers in the ensemble that I was working on. I just knew myself and the cybersphere, or sometimes we called it the tuplesphere, or just a bunch of information floating around. We used the analogy — we talked about helium balloons. We used a million ways to try and explain this idea – hn Markoff and Clay Shirky talk to David Gelernter – Lord of the Cloud

Great stuff unmaintainable code – ROFL Just some quotes from it :

• Refactoring is a most emotionally-satisfying activity.  It is second only to sex. Your own inchoate intention suddenly shines through with blinding clarity. You must restrain the impulse.
  
• Too Much Of A Good Thing(TM): Go wild with encapsulation and OO. For example:
  myPanel.add( getMyButton() );
  private JButton getMyButton()
  {
  return myButton;
  }
  That one probably did not even seem funny. Don’t worry. It will some day.

• The XML fad has created a bonanza of opportunities for obfuscation. The basic technique is to pick a random hunk of code, then invent an obscure way of representing its logic in XML. Then replace the piece of code with an XML properties file and an XML parser. Make sure the XML representation you choose is so limited that almost anything other than the original logic cannot be expressed in it. Of course you never document the XML language extension or the parser. Nobody questions the simplicity of XML. Using this technique, you should easily be able to balloon 10 lines of simple Java code up to 100 lines of perfectly opaque XML.

• Nest as deeply as you can. Good coders can get up to 10 levels of ( ) on a single line and 20 { } in a single method. C++ coders have the additional powerful option of preprocessor nesting totally independent of the nest structure of the underlying code. You earn extra Brownie points whenever the beginning and end of a block appear on separate pages in a printed listing. Wherever possible, convert nested ifs into nested [? : ] ternaries. If they span several lines, so much the better.
  

Recently, I crawl throught amazon.com and found that they have some bugs in their services – in some categories they have wrong stuff counter and also guys have really bad html code for their product pages. Continue reading ‘Amazon.com have about a 10% of waste web-traffic’ »

Recently I found another strange thing in EC2 – if I use “shutdown now” on ec2 instance for AWS this node will be still alive and in marked as “running” in ec2-describe-instances host list. For those who want to have simple way to shutdown ec2 instances from script I recommend to use this code ( I use it in our Convergence – Data-Aware routing on EC2 project ) :

ec2-describe-instances | grep `hostname` | awk "{ print \"ec2-terminate-instances \" \$2; }" >> /root/tools/stop_it.sh

You may just run stop_it.sh or schedule it via at

at -f /root/tools/stop_it.sh "now + $TIMEOUT minutes"

just not forget to replace $TIMEOUT with something usefull

As base AMI i used ami-7db75014 – it’s OpenSolaris supported by Sun, common informartion about installing and using OpenSolaris in EC2 also available in Sun’s Amazon EC2 Getting started guide – in this post I will focus almost in SGE using in Amazon EC2. As SGE distributive i use all-in-one tar package – i choosed “All supported platform” in Grid Engige download page – it takes about 350 Mb, but I don’t worry about platform architecture – if sun support it – it will be in this package. This ge62u2_1.tar.gz contains bunch of other tar.gz’s ( and even hedeby’s core package ) and can be unpacked by :

So I just go inside ge6.2u2_1 and unpack them all using something like this

One important thing – hedeby-1.0u2-core.tar.gz contains old versions of some files from ge-6.2u2_1-common.tar.gz – there’s conflicts in files common/util/arch  and common/util/arch_variables – here’s diff for them – may be sometimes it can be usefull, but for my configuration it causes very strange errors when I try to install executor host :

When I replace this files from ge-6.2u2_1-common.tar.gz installation works as expected. Next point it’s DNS configuration – SGE is very picky to DNS and it will cause some problems in running SGE Amazon EC2 instances with SGE, this stuff can be fixed using host_aliases file in SGE, or other way it’s to use /etc/hosts file for it – some kind of this technique used in Hedeby-SGE on Amazon EC2 demo, for example if we have master this name and 2 executor hosts I put this lines into /etc/hosts :

#internal_ip external_full_name external_short_name internal_full_name internal_short_name
10.yyy.xyz.zzz ec2-RRR-TTT-ZZZ-YYY.compute-1.amazonaws.com ec2-RRR-TTT-ZZZ-YYY domU-mm-ww-PPP-WWW-FFF-GGG.compute-1.internal domU-mm-ww-PPP-WWW-FFF-GGG
10.yyy.qwe.ttt ec2-aaa-bbb-ccc-ddd.compute-1.amazonaws.com ec2-aaa-bbb-ccc-ddd domU-mm-ww-JJJ-HHH-DDD-SSS.compute-1.internal domU-mm-ww-JJJ-HHH-DDD-SSS
10.yyy.pre.ppp ec2-yyy-rrr-eee-qqq.compute-1.amazonaws.com ec2-yyy-rrr-eee-qqq domU-mm-ww-UUU-III-OOO-PPP.compute-1.internal domU-mm-ww-UUU-III-OOO-PPP

Also I use hostname ec2-RRR-TTT-ZZZ-YYY ( external_short_name )  to set instance hostname – this names I use as hostnames when I configure SGE.