Posts tagged ‘ubuntu’

SWAP partitions in Azure are quite interesting thing – SWAP space can be created on the local resource disk with the Linux Agent by enable swap in /etc/waagent.conf. It will automatically use the resource disk (which comes with every VM) to create the swap ( i.e. there’s no need to create a disk for it and also it means that please do not use swap on OS or data disk ). Good links which surely may help here – it’s not a rocket science, just changing Azure Linux Agent configuration file.

There’s a recommendation regarding where should be SWAP placed – we highly recommend to use resource disk for that ( i.e. /mnt/resources ) and not using system drive for that purpose. There’s several reason why it should be like that and all of them are about performance ( so yes, technically you can do it, but consequences of that configuration will be bad and non-so-predictable performance ):

  • OS drive is a storage account backed entity and it saved during restarting/moving VMs. So it’s not good idea to slow down that account by additional workloads.
  • OS drive is optimized almost for fast boot and other additional workloads will make performance of that resource not so good ( or very bad even ).
  • OS drive is from technical side is a remote storage, so latency might be higher compared to local resources.
  • Resource disk is located locally and not survived during host changed, but it’s much faster and this is a place designed especially for swap-like workloads.

Also we support a way inject a script or other metadata into a Microsoft Azure virtual machine at provision time – it’s custom data and cloud-init :

 

Examples of configuration :

Short how-to:

root@215003:~# mysql -u root -p
Enter password:
mysql> CREATE DATABASE s_db;
mysql> GRANT ALL PRIVILEGES ON s_db.* to 's_user'@'localhost' IDENTIFIED BY '12345678';
mysql> FLUSH PRIVILEGES;
mysql>exit
mysql -u s_user -p s_db < database.sql
<
code>

Guys from EC2 announced micro instances – it costs 2 (two) cents per hour for linux and now it’s will costs less than traditional dedicated hosting with root access – monthly payment for EC2 micro instance will be about 15 USD, and price for root/linux on dedicated hosting will be about 30 USD/month. It’s really good news – you can have 100 boxes cluster just for two usd per hour! Bad thing is that micro instances don’t have their own disk space – EBS only, looks like this best ever use case for this type of instances will be highly-distributed computational grid with all data stored in RAM. And don’t forget that EBS will costs you some money – $0.10 per allocated GB per month Amazon EBS also charges $0.10 per 1 million I/O requests you make to your volume . Fredrick Poller’s already check out micro instances performance by sysbench : Amazon EC2 Micro instance, how fast is it?.

By unknown reasons Sun JDK was moved to partner repository, so to use sun jdk you need to do this steps

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
sudo apt-get update
sudo apt-get upgrade

and then enjoy :

sudo apt-cache search jdk | grep sun
sun-java6-source - Sun Java(TM) Development Kit (JDK) 6 source files
sun-java6-jre - Sun Java(TM) Runtime Environment (JRE) 6 (architecture independent files)
sun-java6-jdk - Sun Java(TM) Development Kit (JDK) 6
sun-java6-javadb - Java(TM) DB, Sun Microsystems' distribution of Apache Derby
sun-java6-demo - Sun Java(TM) Development Kit (JDK) 6 demos and examples
sun-java6-bin - Sun Java(TM) Runtime Environment (JRE) 6 (architecture dependent files)

You can use sed :

sed -e 's/\([^\d]*\)/\L\1/' in.txt > out.txt

or perl inliner :

perl -ne 'utf8::decode $_; $_ = lc $_; utf8::encode $_; print' in.txt > out.txt
.
Both guys works fine for unicode file too.

Using telnet in bash scripts to automate some stuff ? It’s really easy, here’s an example to tell “hello world” on telnet server :

req="hello world"
server="my_server 1234"
val=`( echo open ${server}
sleep 3
echo "${req}"
sleep 1 ) | telnet`
echo $val

Okay, we have text file with list of urls and want to have firefox’s screenshots from this pages and also we need to have this screenshots in some normalized resolution ( like all images should be in 300×400 – thumbnails ). First of all you need to install Command line print Firefox add-on. Then create some simple script which will run firefox with needed url, print screenshot and close ( in my case via kill – may be it’s too brutal ) firefox in cycle. It may look like this ( url_list.txt – file with urls – each url on its own line :-)), after running this script you will have many *.png files which is screenshots for ulrs – 0.png – for first url in urls_list.txt, 1.png for second and so on.

#!/bin/bash
id=0
while read line
do
firefox -print $line -printmode png -printdelay 10 -printfile ${id}.png
ps ax | grep firefox | awk '{ print $1 }' | xargs kill -9 ;
id=$[$id+1]
done < urls_list.txt

And now then we have screenshots ( all this guys are in different resolution in common ) then we need to normalize them – to create thumbnails for all images in 300×400 resolution – convert helps!

for f in *.png;
do
convert -thumbnail 300x400! ${f} thumb_${f}
done

And we have many thumb_*.pn with 300×400 resolution all. A little note – using resolution without ! sign will work in another way – resize will be processed proportionally with using resize only for one dimension ( bigger one ).

Here’s some stuff about ( yeah, I know – ‘geeks, stop migrating from svn and git and vise versa – just developer code and don’t spent time on this!‘ ) svn and git comparison.

What do we have : git and svn installed on a same server, empty both svn and git repositories – I will add same data and provide same changes in both repositories,  and also, in another folder, I will checkout how update changes works – so in this tests I’m going to figure out how it works for ‘commiters’ and for ‘those who update changes’. My tests deal with  typical operations with version control systems  – checkout time for adding, pushing and pulling various types of data and checkout how update for them will work – test’s detailed information is below table. For git I use git+ssh protocol, for svn I use svn protocol. I don’t provide any special configuration nor for git nor for svn – all settings is about to be default. Also I would like to note that this tests is partially incomplete – I don’t checkout how delete, branching, merging and some other functions works – so in future I will try to add these points onto this test. Main result of this test – git is faster than svn about a 2-3 times in typical configuration. There’s only one really bad points – git works bad on pretty big files ( > 300 mb size), for some big size ( ~1gb ) it git even can exit with error about memory leak – there’s some special git-related project – like git-bigfiles – this stuff should help you to work with git and big files together.

svn git svn/git
Test 1 : adding boost 1_43 add  ( 1 sec )+ commit ( 600 sec ) = 601 sec add ( 3 sec ) + commit ( 86 sec ) + push ( 612 sec ) = 701 sec 0.85
Test 2: checkout repository after Test 1 svn co = 109 sec clone = 18 sec 6.05
Test 3 : small changes in sources – add first line with comment to some files – 346 files changed svn commit = 5 sec commit(2 sec ) + push ( 2 sec ) =4 sec ~1
Test 4: update after this Test 3 svn update = 15 sec git pull ( 7 sec) ~2
Test 5: bigger change  – add first line with comment to 5363 files svn commit = 103 sec commit ( 6 sec ) + push ( 18 sec ) = 24 sec 4.29
Test 6: update after Test 5 svn update = 28 sec git pull = 12 sec 2.3
Test 7: real-life project add ( 2.6 gb of data ) svn add(21) + svn commit(1h 2m 25s) = 3 766 sec add (38 ) + commit ( 1m23 ) + push ( 18m28 ) = 1 m 51 + 18m 28sec= 20 m 19 sec = 20 m 19 sec = 1 219 sec 3.1
Test 8: update after Test 7 svn update ( 11m 58 sec ) = 718 sec git pill = 5m 27 sec = 327 sec ~2.2
Test 9: source change (change namespace visibility) svn commit ( 1m 28 sec ) = 98 sec commit (5 sec ) + push ( 3 sec) = 8 sec 12.25
Test 10 : update after Test 9 svn up  = 22 sec pull ( 9sec) 2.4
Test 11: full checkout project svn co 12m 45sec = 765 sec pull ( 4 min 3 sec ) = 243 sec 3.14
Test 12: big file test : apache log ( 50 mb ) svn add(3) + commit (1m 27s ) = 1m 30 sec = 90 sec add (3) + commit(6) + push (1m 18 sec) = 1m 17 sec = 77 sec 1.16
Test 13 update after test 12 up = 26 sec pull  = 17 sec 1.52
Test 14 : big files test: c++ sources in one file ( 70 mb ) add(3) + commit(1m 15 sec) = 1m 18 sec = 78 sec add (2) + commit (2) + push ( 56 sec ) = 1 m = 60 sec 1.3
Test 15 : update after Test 14 up = 28 sec pull = 14 sec ~2
Test 16: 100 MB xml with encoded data add(3) + commit (8m 37 sec)= 8m 40 sec = 520 sec add (9 sec) + commit (1) + push ( 24m 34 sec) = 24 m 44 sec = 1484 0.35
Test 17 : update after Test 16 up = 52 sec pull = 47 sec 1.10
Test 18 : 300 MB xml with encoded data add ( 1 ) + commit( 9 m 26 sec )= 9m 27 sec = 567 sec add (3) + commit(8) +push (17m  12 sec) = 17 m 33 sec = 1053 sec 0.53
Test 19:  update after test 18 1m 39 sec pull= 1m 29 sec 1.1
Test 20 : 1200 MB xml with encoded data 1h 24m 34s failed ?
Test 21 : update after Test 20 5m 47 sec failed ?
~2.6

So – In common we have that git is about 2.6 faster than svn.

I understand that this is may be very non-precious tests and also I don’t check how delete or branching works, but anyway I checkout my own most used operations during this test. In common we can see that git is more than 2 times faster than svn – everything looks fine with git, except one really bad thing  – work with big ( more than 100 mb size ) files – for this type of file svn is really works faster, and for too big ( more than 1 GB files ) git even crashed during ‘git push’ command with ‘fatal: Out of memory, malloc failed‘ message.


Test1
boost 1_43 C++ library :  29135 files and 31609 objects including folders, 286 Mb summary size

Test 3
I just add first line to some cpp files :

find ./ -name "*a???.cpp" -exec sed -i 1i"//test comment `date` : {}" {} \;

Test 5
Change all *.cpp files in the same way :

find ./ -name "*.cpp" -exec sed -i 1i"//big test comment2 `date` : {}" {} \;

Test 7:  Add  real-life repository :  many sources in cpp, perl, erlang, makefiles, bash scripts, MS word documents, images, pdf’s, compiled binaries, resource files, etc,  in sum it contains 4605 files, size is about 2.6G

Test 9:
Refactoring simulation : make to use “std::” specifier for all cerr and cout. Not sure it’s really safe in real-life-development, but it’s okay for test purpose.

find ./ -name "*.h"   -exec sed -i "s/  cerr/std::cerr/g" {} \;
find ./ -name "*.cpp" -exec sed -i "s/ cerr/std::cerr/g" {} \;
find ./ -name "*.cpp" -exec sed -i "s/ cout/std::coutr/g" {} \;

Test 11:  checkout whole project

Test 12: adding 50 mb text file – apache logs

Test 14: adding 72 mb file – many c++ sources concantenated in one big file

Test 16: small xml : 100 mb – xml file sample from real life

Test 18: medium xml : 300 mb – same ( but bigger ) real-life xml fil

Test 20: large xml : 1200 mb – sample of pretty big real-life xml

git push failed with error message of memory leak ( as I can see ) : “fatal: Out of memory, malloc failed”

I collect some usefull information about different source text repositories version control systems : git, svn, cvs, mercury links – take a look on this if you’re interested in different VCS system using.

Using json in bash :

  • jsawk : work with an array of JSON objects read from stdin, filter them using JavaScript to produce a results array that is printed to stdout. For example get’s json ( from Yahoo image search by ‘madonna’ query ) and add some info ( madonna’s real name ) to this
  • wget http://bokov.net/json_example.json
    cat json_example.json | ./jsawk "this.ResultSet.RealName=\"Louise Ciccone\"" > updated_info.json

  • you can use some bash/awk/sed stuff to parse json. For example – list all image urls from json
  • wget http://bokov.net/json_example.json
    cat json_example.json | sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | grep '"ClickUrl":' | sed 's/:/ /1' | awk -F" " '{ print $2 }'

  • or use library from json.org for language which you like 🙂

via v.kruchkov

We have git repo at github (  and would like to move onto some our corporate box inside our network, for example on internal.bokov.net. This repository is shared for group of developers and all who have ssh access onto internal.bokov.net shall have right to commits onto git repository. So first of all prepare place for git repo at new hosting :

ssh user@internal.bokov.net
cd var
mkdir git_repo
chmod 777 git_repo
cd git_repo
mkdir my_project
cd my_project
git init --bare --shared=all
cd objects
chmod g+ws *

Next clone existing repo and change its settings to point onto internal.bokov.net

git clone --bare git@github.com:bokov/github_project.git
git remote add -t master -m master origin ssh://internal.bokov.net/var/git_repo/my_project/
git push origin master

to start work with new repo just use

git clone ssh://internal.bokov.net/var/git_repo/my_project/

have fun.

  • Ubuntu doesn’t asking for login after boot up ( may happen after unexpected shutdown ) :
    press Ctrl-Alt-F1 to login in terminal mode and use fsck to fix it
  • Add user short memo :
    [root@my-box ~]# useradd -g users -u 560 tuser
    [root@my-box ~]# passwd tuser

    ps. sudo will be required in not-root case 🙂
  • Proper access rights for .ssh files :
    chmod 700 ~/.ssh
    chmod 600 ~/.ssh/id_rsa
    chmod 644 ~/.ssh/id_rsa.pub
    chmod 644 ~/.ssh/authorized_keys
    chmod 644 ~/.ssh/known_hosts
  • List all files in folder sorted by size :
    find . -type f -exec du -k "{}" \; | sort -n

When I tried to restart mysql
sudo /etc/init.d/mysql restart
it failed with
* Stopping MySQL database server mysqld [ OK ]
* Starting MySQL database server mysqld [fail]

Continue reading ‘Quick fix problem with mySQL starting on Ubuntu 9.10’ »

Short how-to about configuring Ubuntu to work with EC2 API ( for Ubuntu 10.04 users – Sun JDK was moved to partner repository so please check out Where’s sun JDK on Ubuntu 10.04 ? before follow described steps ).

Continue reading ‘Configuring Ubuntu ( 9.10 ) to work with Amazon EC2’ »

Update in Amazon Web Services:

  • 2 high cpu instance types : 64 bits – Double Extra Large с 34.2 GB RAM, and 13 ECU (4 virtual cores *3.25 EC2 compute Unit=ECU), 64-bit platform
    and Quadruple Extra Large – 68.4 GB of RAM/ 26 ECU (8 virtual cores* 3.25 ECU) : New EC2 High-Memory Instances
  • Instance prices changes ( us-east is still cheaper thatn eu-west ) : Amazon EC2 – Now an Even Better Value
  • New service for relational DB ( provisioning, scaling and other nice things ) : Introducing Amazon RDS – The Amazon Relational Database Service
  • Security stuff : Vulnerability identified in Amazon’s cloud computing
  • Amazon EC2 – Ubuntu at google groups
  • 5 years ago Amazon announced Amazon Simple Queue Service – top points of AWS for last 5 years
    • Recursively add files and foldes into CVS
    find . -type d -print | grep -v CVS | xargs -n1 cvs add
    • Delete “.svn” files from current folder
    find . -type f  -exec echo \"{}\" \;  | grep ".svn" | xargs rm

    Some Sony Vaio owners ( my own is VGN-FZ31ER ) may have MAC is in deep sleep! problems – WiFI doesn’t work under Ubuntu because WLAN driver goes into forever sleep ( some bug in power safe logic I suppose ). This bug can be fixed by adding “noacpi” parametrs to Ubuntu boot command line.

    On one of my apache I use suexec to run some tasks – some weeks before I configure all it and all stuff works well, but some days later when I try to run my cgi-perl script I found I have “500 Internal error”. I change nothing in apache config or in my scripts – I just install fresh updates for ubuntu. In apache’s error log I’ve got :

    [Wed Jul 29 08:15:28 2009] [error] (13)Permission denied: exec of ‘/usr/lib/apache2/suexec’ failed
    [Wed Jul 29 08:15:28 2009] [error] [client my_ip] Premature end of script headers: script.pl

    It’s strange, but anyway it doesn’t works. I spent some time trying to fix it ( some guys recommend to rebuilt suexec – but it’s not a simplest way to fix this problem ) – in my case it was fixed by changing owner of my suexec. I check out current properties :
    ls -la /usr/lib/apache2/suexec
    -rwsr-xr– 1 root www-data 18872 2009-07-10 18:40 /usr/lib/apache2/suexec
    and change them :

    chown root:root /usr/lib/apache2/suexec
    chmod 4755 /usr/lib/apache2/suexec

    And don’t forget to restart :

    /etc/init.d/apache2 restart

    ps. by the way suexec’s id’s are :

    #id www-user
    uid=1000(www-user) gid=109(www-group) groups=109(www-group)

    Here’s some steps which I frequently use to configure Ubuntu to work as EC2 manager
    Continue reading ‘Quick steps to configure Ubuntu for with EC2 API’ »

    For those who those who install skype under Ubuntu and get “Error: Wrong architecture ‘i386’” after download and run skype-debian package from skype.com – check out this usefull bash stuff ( just copy-paste it in terminal window ):
    sudo apt-get install ia32-libs lib32asound2 libasound2-plugins; wget -N http://bokov.net/packages/ubuntu/getlibs-all.deb; wget -O skype-install.deb http://www.skype.com/go/getskype-linux-ubuntu-amd64; sudo dpkg -i skype-install.deb; sudo dpkg -i getlibs-all.deb; sudo getlibs -p libqtcore4 libqtgui4 bluez-alsa