Alexey Bokov's weblog

Recently I updated to Ubuntu 18.04 Bionic Beaver and tried to install latest docker-ce on that laptop – I expected more-less smooth process, but find some interesting blockers during installation process. So let me share my experience – hope that it will save others time. In default Ubuntu repository I find docker v.15 and docker-ce v.17, but it’s obviously not my choice when I can install latest one 18.05-rc1 🙂 But actually even if you install docker from default repository you’ll struggle with same issues – so you may skip installation part and go ‘make it work’ part instead.

Install from default Ubuntu repository

In my case I can easily install docker 1.17 with
sudo apt-get install docker.io
or docker 1.5 with that
sudo apt-get install docker
Actually I highly recommend to use apt-cache show command to figure out which version you’re trying to install before install it
sudo apt-cache show docker.io

Install docker 18.05 rc1 from docker repository

Installation should go through that process – well described on linuxconfig : How to Install Docker On Ubuntu 18.04 Bionic Beaver:
sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt update
sudo apt-cache show docker-ce
Last line is needed just to check that you really get docker-ce 18.05 added from docker repository. Then just install and check it
sudo apt install docker-ce
sudo docker run hello-world
If everything works fine for this moment, then you’re good – enjoy docker and have fun with containers. But in my case and I believe it’s also case for many others who running VPN client on your Ubuntu, there’s errors on starting docker daemon after installation stage. In my case I got errors:
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
docker.service - Docker Application Container Engine
Process: 21542 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
Main PID: 21542 (code=exited, status=1/FAILURE)
dpkg: error processing package docker-ce (--configure):
installed docker-ce package post-installation script subprocess returned error exit status 1
Actually if you take a look onto systemctl status docker.service
abokov@abokov-ubuntu:/var/lib/docker$ systemctl status docker.service
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2018-06-01 13:29:07 MSK; 2s ago
Process: 31921 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
Main PID: 31921 (code=exited, status=1/FAILURE)
Jun 01 13:29:07 abokov-ubuntu systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Jun 01 13:29:07 abokov-ubuntu systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Jun 01 13:29:07 abokov-ubuntu systemd[1]: Stopped Docker Application Container Engine.
Jun 01 13:29:07 abokov-ubuntu systemd[1]: docker.service: Start request repeated too quickly.
Jun 01 13:29:07 abokov-ubuntu systemd[1]: docker.service: Failed with result ‘exit-code’.

and journalctl -u docker.service says something like this:
Jun 01 12:44:53 abokov-ubuntu dockerd[21542]: time="2018-06-01T12:44:53.884718719+03:00" level=info msg="Loading containers: start."
Jun 01 12:44:54 abokov-ubuntu dockerd[21542]: Error starting daemon: Error initializing network controller: list bridge addresses failed: no available network
Jun 01 12:44:54 abokov-ubuntu systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jun 01 12:44:54 abokov-ubuntu systemd[1]: docker.service: Failed with result 'exit-code'.
Jun 01 12:44:54 abokov-ubuntu systemd[1]: Failed to start Docker Application Container Engine.
Jun 01 12:44:54 abokov-ubuntu systemd[1]: docker.service: Service hold-off time over, scheduling restart.

Make it work

Root cause of this problem is that I’m running VPN client on my laptop on the same time and Docker daemon is unable to get network access to find suitable network range to use for container’s network. Actually fix can be shutdown VPN client ( not my case) or manually add network interface bridge and assign a network range for this bridge.
ip link add name docker0 type bridge
ip addr add dev docker0 172.17.0.1/16
One thing that you will lost all these configuration after restart, so you need to choice VPN or add this onto startup/run every time manually.

This week I spent good time BigChainDB folks in Berlin on Microsoft/BigChainDB hackathon – so we hacking, designing and coding together. Besides other outcome ( which should be not only barely working code in python) there’s some thought which came up into my mind and I want to share it there.

So we have devices – it can be any IoT like devices as rapsberry Pi or Riddle&Code cryptodevices which support AES, Diffie Hellman and secure way to keep keys inside of cryptomodule, or even smart cars like Kia or VolksWagen who collects car telemetry, or even musical instruments or not even device in common meaning like luxury clothes or paintings . And we would like to to get data from device or prove device authenticity and we’re going to do it in secure way. For example, we would like to get information from smart car about its telemetry/mileage and this information should be shared to new potential car owner or insurance company as important insights about smart car status.

• Here we go to first important topic in that area – its ‘how to identify device in some kind of secure way?’. Basically from that point of view there’s two types of devices (or not devices – do you remember paintings and musical instruments mentioned before)  – first are provide some secure features like ‘I can keep private/public key’ and  second one are about ‘i don’t have any keys at all’. In first case we’re very good – there’s device which have it’s own unique keypair and this keypair can be efficiently used anywhere to identify device. In second case ( 99.95% or even more devices ) we have only properties which should be treated as publicly exposed information, for example:
  – serial number ( VIN ), mostly never changed and mostly unique, but not always – for example for VIN there’s a cases where VIN can be changed ( for example you change engine on your car, then you have new engine with new VIN, but car itself is the same )
  – consumer properties like color, size, weight and so on. Some of that properties are immutable ( like weight or size ) in case if device is in proper state, some like color can be changed easily (after washing for example ) without any affect to main functionality of our device.
  – manufacturer properties: year, factory id, name of device, person who made an assembly and so on. These properties are immutable because device got them as part of its existence/historical data. Yes, serial number is also some kind of manufacturer properties and the reason why I differentiate it from other manufacturer properties because serial number is going to treated like unique identifier, at least it has much more changes to be treated in that way than any other properties.

For these non-secure devices we do have two options kindly provided by blockchain technology – option one it’s to use One Authority center to

note: post is under development 🙂

Azure Service Fabric implements great concepts of microservices ( sorry for saying that so simplified ), there’s a lot of demos and docs which demonstrate nice pictures and moving boxes, but I surely see that most important and key basement features of that service are tough to find, especially for those who just started with Service Fabric.

• Service Fabric cluster capacity planning considerations – great doc which describes underlying layer of Service Fabric which is Azure Scale Set with different node types

I believe many of you aware about great Eric Brewer theorem which says that any net­worked shared-data system can have only two of three desirable properties: Consistency or high Availability or tolerance to network Partitions (i.e. this property means that network may loose any packet/message ) – this theorem usually called CAP-theorem. This theorem is quite important (or even very fundamental) for many distributed workloads like computational and data grids and “2 of 3” principle is basis for many architectural decisions in cloud world. Below I combine couple of great links around that topic which surely may shred some light on that principle as well as give you more insights how that basic rule is changed nowadays . And yes, by the way CAP theorem  is key thing for most of NoSQL or any other data distributed solutions – below you might good reading about that topic, including some new views on that topic.

• Towards Robust Distributed Systems – PODC keynote presentation slides, by Eric Brewer
• Perspectives on the CAP Theorem by Seth Gilbert and Nancy Lynch
• Great FAQ about CAP theorem
• You Do It Too: Forfeiting Network Partition Tolerance in Distributed Systems by Nicolas Liochon
• @eric_brewer on Twitter

SWAP partitions in Azure are quite interesting thing – SWAP space can be created on the local resource disk with the Linux Agent by enable swap in /etc/waagent.conf. It will automatically use the resource disk (which comes with every VM) to create the swap ( i.e. there’s no need to create a disk for it and also it means that please do not use swap on OS or data disk ). Good links which surely may help here – it’s not a rocket science, just changing Azure Linux Agent configuration file.

• Add swap space to Azure Ubuntu IaaS VM
• How to create a swap for Azure Ubuntu VM?

There’s a recommendation regarding where should be SWAP placed – we highly recommend to use resource disk for that ( i.e. /mnt/resources ) and not using system drive for that purpose. There’s several reason why it should be like that and all of them are about performance ( so yes, technically you can do it, but consequences of that configuration will be bad and non-so-predictable performance ):

• OS drive is a storage account backed entity and it saved during restarting/moving VMs. So it’s not good idea to slow down that account by additional workloads.
• OS drive is optimized almost for fast boot and other additional workloads will make performance of that resource not so good ( or very bad even ).
• OS drive is from technical side is a remote storage, so latency might be higher compared to local resources.
• Resource disk is located locally and not survived during host changed, but it’s much faster and this is a place designed especially for swap-like workloads.

Also we support a way inject a script or other metadata into a Microsoft Azure virtual machine at provision time – it’s custom data and cloud-init :

• Custom Data and Cloud-Init on Microsoft Azure
• Injecting custom data into an Azure virtual machine

Examples of configuration :

• SUSE Linux Enterprise Server Configuration For Windows Azure
• Creating secure Linux VMs in Azure with SSH key pairs
• Non-endorsed Linux distribution configuration in Azure
• Create an Azure Image from an Existing Virtual Machine
• VM Image – how to use

As you know we recently add DNS services into preview mode and you can subscribe to this service using Request-AzureProvideFeature like that

Request-AzureProvideFeature -ProviderNamespace Microsoft.Network -FeatureName azurednspreview

So good question here is about how get list of all possible available options for FeatureName ?

Continue reading ‘List of all available preview services in Azure’ »

I will describe how this schema works for single VM with running Linux/Jelastic. So first of all let check that do we have for that VM by using Get-AzureDeployment command and then let go deep that happened inside that file

Continue reading ‘All about Azure Service Configuration Schema .cscfg’ »


Get-AzureVM | Where-Object {$_.InstanceSize -ne 'Basic_A0'} | Set-AzureVMSize "Basic_A0" | Update-AzureVM


Q: How to get Azure VM external IP address ( if you not use PublicIP service ) ?

A:  If you’re not using ReservedIP service field PublicIP will be empty ( as soon as there’s no reserved IP ) for command Get-AzureVM. I using Swtich-AzureMode just in case to be sure that we’re in right Azure mode in powershell right now.

PS C:\> Switch-AzureMode -Name AzureServiceManagement
PS C:\> Get-AzureVM


PS C:\> Get-AzureVM -ServiceName "abokov-jelastic" | Select-object  Name, DNSName, *IP*


Name                    : abokov-jelastic
DNSName                 : http://abokov-jelastic.cloudapp.net/
IpAddress               : 10.0.0.4
PublicIPAddress         :
PublicIPName            :
PublicIPDomainNameLabel :
PublicIPFqdns           : {}
OperationDescription    : Get-AzureVM

You may see that this VM have DNS name, internal IP, but public IP is empty, nevertheless that there’s is an external IP address which is linked to that VM. To get external IP you may use command Get-AzureEndPoint which returns all information about all endpoints for that cloud service – in my case I have 7 open ports and to make this more readable I select only few fields from output.

PS C:\> Get-AzureVM -ServiceName "abokov-jelastic" | Get-AzureEndPoint | Select-Object Name, Port , *ip*

Name Port Vip VirtualIPName
—- —- — ————-
dns-TCP 53 23.99.218.74
dns-UDP 53 23.99.218.74
http 80 23.99.218.74
http4848 4848 23.99.218.74
https 443 23.99.218.74
https4949 4949 23.99.218.74
ssh 22 23.99.218.74

First of all you need to have Microsoft Azure PowerShell SDK installed, after this step is done you may type
           Add-AzureAccount
and this will authenticate you via web browser in Azure account ( you need to have working Azure account obviously ). Alternatively you may use PublishSettings :

1. Open that link to Donwload Azure Publish Settings Profile: https://manage.windowsazure.com/publishsettings , then save you credentials.
2. Open Microsoft Azure Powershell and run Import-AzurePublishSettingsFile :


PS C:\> Import-AzurePublishSettingsFile C:\tools\azure\my_settings.publishsettings
Id : c09f1074-6e2e-45be-aeee-935d6642424d
Name : Windows Azure MSDN - Visual Studio Ultimate
Environment : AzureCloud
Account : 24636FAB2C68B96D43777561A3CD728092912C03
Properties : {[SupportedModes, AzureServiceManagement]}

Then you need to keep in mind that Azure PowerShell (from 0.8.0 ) contains many modules, here we will talk about three main modules ( complete list of eveything which is inside SDK is here – see Azure SDK CmdLets reference ):

• AzureProfile : operates with profiles commands like Add-AzureAccount, Get-AzureSubscription, and Switch-AzureMode. Complete list of commands in AzureProfile module is availalbe here – see Azure Profile Cmdlets. Commands from AzureProfile are always available – not like two next modules which are not designed to be used in the same PowerShell session.

• Azure ( or AzureServiceManamagement ): contains commands which operated by core ( or basic :-)) cloud functionality like virtual machines, web sites, storage accounts and  so on. Complete list of commands inside this module is here – see Azure Service CmdLets. You can switch to that module using Switch-AzureMode -Name AzureServiceManagement

• AzureResourceManager : contains commands which operates logical units of Azure. To switch into AzureResourceManager mode you may use command  Switch-AzureMode -Name AzureResourceManager  Below is a list of most important modules inside AzureResourceManager :
  • Azure Batch: contains commands related to Microsoft Azure Batch services ( Get-AzureBatchAccount, New-AzureBatchTask, Remove-AzureBatchPool and others )

   

  • Azure Data Factory (ADF ): commands managing data factories in Azure – for more information about this service you may refer to What is Azure Data Factory and FAQ  and Azure Data Factory (ADF) Documentation

   

  • Azure Compute and Storage: commands managing VMs and storage accounts ( New-AzureVM , Start-AzureVM , Add-AzureVMDataDisk  and others ).

   

  • Azure Key Vault : commands which helps safeguard cryptographic keys and secrets used by cloud applications and services you can encrypt keys and secrets ( authentication keys, storage account keys, data encryption keys, .PFX files, and passwords) by using Host Security Modules ( HSM ). For more details please see What is Azure Key Vault

   

  • Azure Networking : commands related to load balancer, NAT, IP-address management and virtual networks ( Set-AzureNetworkInterface , New-AzurePublicIpAddress , Get-AzureLoadBalancerProbeConfig and others ).

   

  • Azure Resource : commands related to resource group templates, Azure tags – using this techniques you may operate groups of different resources by using one template or tag. For more information about Azure Resource groups please refer to Azure Resource Manager Overview

   

  • Azure SQL Database : SQL Azure helpers, contains commands like Get-AzureSqlDatabase , New-AzureSqlDatabase , Set-AzureSqlElasticPool  and others.

   

  • DNS : commands to manage your DNS records and resolving ( New-AzureDnsZone , Get-AzureDnsRecordSet , Add-AzureDnsRecordConfig and others ). More detailed information can be found here – Microsoft Azure DNS documentation

   

  • Traffic Manager : cmdlets which allow you to manage TrafficManager, including adding on-premise end-points which is not yet available in web portal – detailed example about how to add on-premise end-points is here – Azure Traffic Manager external endpoints & PowerShell support ( New-AzureTrafficManagerProfile, Get-AzureTrafficManagerProfile and other commands ). Here you may find documentation about Traffic Manager and also I recommend this blog post by Jonathan Tuliani from Azure Product Group :  Azure Traffic Manager External Endpoints and Weighted Round Robin via PowerShell

   

 Note: sometimes ( it happened with me as well ) default installation of Azure SDK cmdlets do not include some modules ( in my case it happened with DNS and Azure SQL Database ), then you have choice how to install them – first of all you always may use Azure powershell sources on github to get full and latest one version of cmdlets or you also may check files inside “C:\Program Files (x86)\Microsoft SDKs\Azure” folder and try to import modules from there like this ( it worked for me ):

 
 import-module "C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ResourceManager\AzureResourceManager\AzureResourceManager.psd1"
 

Using PowerShell scripts may give you a lot of power and here’s some nice things which I found very useful for myself. For example we have Azure SDK command ‘Get-AzureLocation’ which give you a complete list of cloud resource types and their locations. For me output of that command looks a little bit complex ( adding ‘more‘ makes it a little bit better ) :

A lot of information, isn’t it ?  So what I really want is just list where my VMs can be deployed. To make this list I’m going to filter this output only with “*compute*” mask and I will do it only for first column. Before I will do it I need to figure out name of first column ( if it’s printed under ‘Name’ column it not always means that objects returned by Get-AzureLocation are under exact that name ), so I will check that properties of object do we have for that output with Get-Member:

PS C:\temp> Get-AzureLocation| Get-Member
TypeName: Microsoft.Azure.Commands.Resources.Models.PSResourceProviderLocationInfo
Name MemberType Definition
---- ---------- ----------
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
Locations Property System.Collections.Generic.List[string] Locations {get;set;}
LocationsString Property string LocationsString {get;set;}
Name Property string Name {get;set;}


Okay so here we have three properties : Locations, LocationsString and Name. So let’s print everything which is about my VMs ( can be filtered with ‘*compute*’ ) using Where-Object : Get-AzureLocation | Where-Object Name -like "*compute*

Okay, so now it’s much better. Just in case if that command doesn’t work try to switch into AzureResourceManager mode ( it might happen that you’re now in AzureServiceManagement mode which works in a little bit different way ) :

Switch-AzureMode -Name AzureResourceManager


We can improve that by printing only ‘LocationsString’ property by Select-Object :
Get-AzureLocation | Where-Object Name -like "*compute*" | Select-Object LocationsString

Just check it out Visual studio 2015 preview

Recently we made an update in CDN network, so right now it’s 29 nodes all around the world ( more than before ) and there’s some changes in locations. Full list of Azure CDN nodes is here Azure Content Delivery Network (CDN) Node Locationsand also you may take a look on them here – CDN node locations on bing map

For some people ( who doing Azure-related presentations and slides a lot like me 🙂 ) this may be useful – now we have 17 datacenters ( including 2 in Australia and 2 Gov datacenters in US ), so here’s updated map :

• Windows Azure CDN locations : 24 locations for February 2012 – also see list of Windows Azure CDN node locations
• Windows Azure CDN Helpers : simple library of MVC helper methods that will allow you to quickly integrate the Windows Azure CDN into your web application to speed up your site
• Using the Windows Azure CDN for Your Web Application
• Working with the ASP.NET OutputCache module for CDN Friendly HTTP Headers, vary:* Headers, compressed content with the CDN, use of IIS Virtual Application, provide your own OutputCache module implementation – Hosted Service as a Windows Azure CDN Origin Tips . Great post by Shawn Cicoria.
• Serving GZip compressed Content from the Azure CDN
• Windows Azure Content Delivery Network – Hands-On Lab

update : Also take a look on Forbes analitycs : Identifying The Tech Leaders In LTE Wireless Patents

“They’ll ask abt ur skillz and propr kmand of ze English lng” 🙂 Continue reading ‘Job interview via facebook – they’ll ask abt ur skillz and propr kmand of ze English lng’ »