Prepare Mac OSX to Test Chef recipes for AWS OpsWorks

This post is a supplement to Cookbooks 101 in AWS OpsWorks documentation to prepare a Mac OSX to test Chef recipes for OpsWorks using the following tools:

Installing Vagrant, VirtualBox and Chef Development Kit

Download the latest Vagrant, VirtualBox and Che Development Kit from the above links and install them. They can be installed by double clicking the downloaded dmg package of each tool. An installation window will appear similar to the screenshots below. Double click each package and follow the on-screen instructions to complete the installation.

chefdk vagrant
virtualbox

Installing Vagrant Plugins

From Terminal, execute the following commands to install plugins to integrate with Berfshelf and Omnibus respectively:

sudo vagrant plugin install vagrant-berkshelf

sudo vagrant plugin install vagrant-omnibus

Vagrant Berkshelf automatically downloads and installs cookbooks into Vagrant Virtual Machines.

Vagrant Omibus automatically installs the desired version of Chef via the platform-specific Omnibus packages into Vagrant Virtual Machines.

It may prompt you to install Command Line Developer Tools if you don’t have the tools. Proceed to install the tools and re-run vagrant plugin install commands to install vagrant-berkshelf and vagrant-omibus plugins.

developer_tools

Adjusting DHCP Option for VirtualBox

VirtualBox comes with a default DHCP server that set to the following:

$ VBBoxManage list dhcpservers
NetworkName:    HostInterfaceNetworking-vboxnet0
IP:             192.168.56.100
NetworkMask:    255.255.255.0
lowerIPAddress: 192.168.56.101
upperIPAddress: 192.168.56.254
Enabled:        Yes

However, vboxnet0 is set to a different IP adddress space when it is created:

$ VBoxManage list hostonlyifs
Name:            vboxnet0
GUID:            786f6276-656e-4074-8000-0a0027000000
DHCP:            Disabled
IPAddress:       172.28.128.1
NetworkMask:     255.255.255.0
IPV6Address:     
IPV6NetworkMaskPrefixLength: 0
HardwareAddress: 0a:00:27:00:00:00
MediumType:      Ethernet
Status:          Up
VBoxNetworkName: HostInterfaceNetworking-vboxnet0

In order to reconcile the conflict, execute the following command from Terminal to remove the default DHCP server:

VBoxManage dhcpserver remove --netname HostInterfaceNetworking-vboxnet0

Adjusting VCS files for Berkshelf

If you plan to use git, you may encounter a permission denied error while running vagrant provision:

 stderr: /opt/chefdk/embedded/lib/ruby/2.1.0/fileutils.rb:1402:in `initialize': Permission denied @ rb_sysopen - /home/schen/.berkshelf/vagrant-berkshelf/shelves/berkshelf20150223-20325-2b14d0l-default/test_vm/.git/objects/01/6f2ce56a1eff7451b477a2b52a16ed288117da (Errno::EACCES)
    from /opt/chefdk/embedded/lib/ruby/2.1.0/fileutils.rb:1402:in `open'

If it happens, add ‘**/.git’ to the EXCLUDED_VCS_FILES_WHEN_VENDORING in the /opt/chefdk/embedded/apps/berkshelf/lib/berkshelf/berksfile.rb to instruct Berkshelf to ignore git’s metadata which is stored in the .git folder.

Installing AWS Command Line Interface

If you are planning to interact with Amazon Web Services , you will need to install AWS Command Line Interface (AWC CLI). It can be installed using pip. If your Mac OSX does not have pip, download get-pip.py from https://pip.pypa.io/en/latest/installing.html and execute the following command from Terminal to install pip:

 python get-pip.py

Once pip is installed, execute the following command from Terminal to install AWS Command Line Interface:

pip install awscli

Configuring AWS Command Line Interface

Execute the following command from Terminal, it will prompt you to enter your AWS profile information:

aws configure

If you have multiple profiles, edit ~/.aws/credentials directly. Refer to AWS Official Documentation for more details.

Posted in Uncategorized | Tagged , , , , , , , | Leave a comment

Report Number of Objects and Bucket Size for S3

Use the AWS Command Line Interface (AWS CLI) to report number for objects and bucket size for S3 in Amazon Linux.

It will print out three columns separated by tabs. The first column is the bucket name. The second column is the number of objects in the bucket. The third column is the total size of the bucket in GB. Below is the sample output.

bucket-outputs

Posted in Uncategorized | Tagged , , , , | Leave a comment

IAM Policy for Self-managing Credentials and MFA Device

This is an IAM policy to allow users to manage their own MFA device and credentials including access keys on the AWS console.  The CloudFormation script below creates the policy and assign it to an existing group.

Refer to http://docs.aws.amazon.com/IAM/latest/UserGuide/Credentials-Permissions-examples.html for reference.

Posted in Uncategorized | Tagged , , , , , | Leave a comment

Setting up EC2 Operator Instance with CloudFormation

This post is to extend my post about Auto Start and Stop Your EC2 Instance. I put together a CloudFormation template to automate the process to set up the EC2 Operator instance. You can find the CloudFormation template in my github repository.

I am going to go through each section of the CloudFormation template. It consists of four sections:

  1. Parameters
  2. Mappings
  3. Resources
  4. Outputs

Parameters

The Parameters section defines input variables that will be used for creating resources.

  • InstanceType – Instance type for the EC2 Operator Instance
  • KeyName – Key pair to ssh the EC2 Operator Instance’s console
  • SSHLocation – The IP address range that can be used to SSH to the EC2 instance

Mappings

The Mappings sections define what AMI should be used while launching the EC2 Operator Instance. It is essential a mapping table between regions and AMIs. Each region uses a different AMI. They are just a standard Amazon Linux AMI. If these AMIs are no longer available, please replace them with the latest Amazon Linux AMIs.

Resources

The Resources section consists of the resources that will be created:

  1. SecurityGroup – A security group for the EC2 Operator Instance
  2. OperatorInstance – The EC2 Operator Instance
  3. WaitHandle – Wait handle to pair with Wait Condition
  4. WaitCondition – Set how long to wait for the EC2 Operator Instance to set up
  5. OperatorInstanceProfile – Instance Profile for the EC2 Operator Instance
  6. OperatorRole – IAM Role to specify the actions the EC2 Operator Instance can perform

Security Group

The SecurityGroup resource defines the firewall rules for the EC2 Operator Instance:

  • TCP 22 – SSH access

If you are not planning to SSH to the instance, you may want to remove this rule. The source of this rule is taken from the SSHLocation variable you specify in the Parameters section.

Operator Instance

It installs python-pip package and gcc package, which are specified in the Metadata section. The following activities are included in the UserData section:

  • Interpret metadata
  • Install croniter PHP library
  • Download the PHP script template
  • Add crontab to run the PHP script
  • Signal the WaitCondition the EC2 Operator Instance is ready

Wait Handle and Wait Condition

WaitHandle and WaitCondition are always used together. CloudFormation is set to wait 300 seconds for the EC2 Operator Instance to signal ready.

Operator Instance Profile and Operator Role

Instance Profile and IAM Role are always used together. The EC2 Operator Instance needs the following permissions to manipulate instances:

  • ec2:DescribeInstances
  • ec2:StartInstances
  • ec2:StopInstances

Outputs

The Outputs section outputs a the instance ID of the newly created EC2 Operator Instance.

 

 

 

 

 

Posted in Uncategorized | Tagged , , , , , | Leave a comment

Connecting Multiple Windows Azure Virtual Networks with AWS

This post is inspired by the Connecting Windows Azure to Amazon post that Michael Washam wrote. In his post, he showed how to connect a Windows Azure Virtual Network (VNET) to a Virtual Private Cloud (VPC) hosted in Amazon Web Services (AWS) with a site-to-site VPN and OpenSwan.  I will extend his post to show you how to connect multiple Windows Azure VNETs together. I will use the following architecture diagram as an example for illustration.

cloud hub

The address space for the VPC in AWS is 10.10.0.0/16, and there are four VNETs in Windows Azure with the following address spaces.

  • Windows Azure VNET 16: 172.16.0.0/16
  • Windows Azure VNET 20: 172.20.0.0/16
  • Windows Azure VNET 24: 172.24.0.0/16
  • Windows Azure VNET 28: 172.28.0.0/16

The VNETs do not have to be in the same Windows Azure account, and they don’t have to be in the same region.  The address space has to be unique among VNETs and VPC. You need to create one local network per VNET. You can create a Local Network when you create a VNET, but I recommend to create Local Networks in advance. Then you can just pick a Local Network from the drop down list when you create a VNET.

local networks

There are four VNETs so four Local Networks are required. The Address Space of each Local Network is basically all other address spaces minus the address space of the VNET. I called the Local Network as vnet16-local, vnet20-local, vnet24-local, vnet28-local for VNET 16, VNET 20, VNET 24, and VNET 28 respectively.

  • vnet16-local: 10.10.0.0/16, 172.20.0.0/16, 172.24.0.0/16, 172.28.0.0/16
  • vnet20-local: 10.10.0.0/16, 172.20.0.0/16, 172.24.0.0/16, 172.28.0.0/16
  • vnet24-local: 10.10.0.0/16, 172.20.0.0/16, 172.24.0.0/16, 172.28.0.0/16
  • vnet28-local: 10.10.0.0/16, 172.20.0.0/16, 172.24.0.0/16, 172.28.0.0/16

Local Networks in Windows Azure are equivalent to Route Tables in AWS. The VPN Gateway Address is the Elastic IP of the OpenSwan Linux instance in AWS.

In the OpenSwan Linux instance, you will need to create four connections. I recommend to use one configuration file per connection under /etc/ipsec.d folder. The files have to have .conf as an extension.  I called them aws-to-vnet16.conf, aws-to-vnet20.conf, aws-to-vnet24.conf, and aws-to-vnet28.conf respectively. The format of the configuration files are slightly different than the one Michael showed.

  • [CONNECTION NAME] – The name of the IPSec tunnel connection. I would just use the name of the configuration file without the extension, such as aws-to-vnet16.
  • [LOCAL NETWORK ADDRESS SPACE] – The Local Network address spaces. They are defined in the Local Network in Windows Azure. For example: 10.10.0.0/16, 172.20.0.0/16, 172.24.0.0/16, 172.28.0.0/16
  • [AZURE VNET GATEWAY] – The Gateway IP Address of  the Windows Azure VNET
  • [AZURE VNET ADDRESS SPACE] -The Windows Azure VNET address space. It is defined in the Virtual Network in Windows Azure. For example: 172.16.0.0/16

You will need to update the /etc/ipsec.secrets file to include one entry per VNET with the following format.

  • [AZURE VNET GATEWAY] – The Gateway IP Address of  the Windows Azure VNET
  • [PRE-SHARED KEY] – The pre-shared key of the VNET Gateway. You can retrieve it from MANAGE KEY in the VNET Dashboard.

manage key

This is the sample /etc/ipsec.secrets.

Don’t forget to update the Security Group in AWS to allow UDP 500 and 4500 from Azure VNET Gateways to the OpenSwan Linux instance.

security rules

Restart the ipsec service in the OpenSwan instance to establish IPsec tunnels between VPC and Windows Azure VNETs.

If you need other instances in your VPC to connect to Windows Azure VNETs, you will also need to add route entries to the Route Table in AWS. The Destination is the Windows Azure VNET Address Space, and the Target is the Elastic Network Interface or the Instance of the OpenSwan Linux instance.

route table

You should be able to communicate among VNETs through AWS. Please notice the OpenSwan Linux instance is a single point of failure in the architecture. You may want to consider to create a second elastic network interface for the OpenSwan Linux instance and  set up a standby OpenSwan Linux instance for fail over. When the primary OpenSwan Linux instance is down, you can switch the second elastic network interface to the standby OpenSwan Linux instance to take over the primary OpenSwan Linux instance.

Posted in Uncategorized | Tagged , , , , , , , | 4 Comments

0 KB DATA IN/OUT in Site-to-Site VPN with Cisco ASA 8.4

I was working with a customer to set up a site-to-site VPN between Windows Azure and a corporate network. On the Windows Azure Virtual Network Dashboard, it showed the VPN tunnel was connected but data in and out were 0 KB even after a long time. Firewalls were open to allow the Windows Azure gateway in the corporate network.  What went wrong?

0_in_out

The router on the corporate network was Cisco ASA 5500 Series device with ASA OS version 8.4. A VPN configuration script was downloaded from the Virtual Network Dashboard in Windows Azure but the script was for OS version 8.3.

Obviously, the  script did not work well for OS version 8.4.  It ended up two changes were required for the following sections to resolve the issue.

  • Internet Key Exchange (IKE) configuration
  • Tunnel configuration

Internet Key Exchange (IKE) configuration

In this section, replace isakmp with ikev1 on the second line before policy 10.

old_ikenew_ike

Tunnel configuration

In this section, add ikev1 in front of the keyword pre-shared-key.

old_tunnelnew_tunnel

After re-running the modified script in the Cisco VPN device, the IN/OUT KB started to increase. VMs were able to communicate between the two networks via PING. Everything seemed to work fine.

in_out

Posted in Uncategorized | Tagged , , , , , , , | 3 Comments

Auto Start and Stop Your EC2 Instances

Have you ever forgotten to shut down your test instances that you just used for few days and got a surprise when the monthly AWS bill came? Do you want to stop your development instances automatically at night and weekends when you don’t use them? If your answers are yes, you will want to continue reading this post.

It is fairly easy to set up a schedule to start and stop your EC2 instances automatically to control your EC2 usage. There are quite a few 3rd party providers offering this service. If you search for “EC2 schedule” or “auto start stop EC2″ in google or AWS forums, you will be able to see what is available.  I list few of them in alphabetical order for your reference in case you are interested:

Of course, there is a monthly subscription fee associated with these 3rd party services, and you also need to give out your access key and secret key to the 3rd party service provider. These service providers offer more feature than just auto start and stop EC2 instances. It is probably worth using them if you are planning to use other features as well. Otherwise, it may be hard to justify when you can do it on your own without revealing your credentials to a 3rd party provider.

In fact, AWS has provided all the APIs you need to manipulate EC2 instances. That’s how these 3rd party service providers are able to manage AWS resources. It is indeed pretty straight forward.  All you really have to do is to put together a simple Python script to leverage these APIs with Boto, which are well documented and supported.

In this post, I will show you how to use a micro instance with an IAM role to start and stop your EC2 instances automatically within a region. Here are the 3 simple steps to get started:

  1. Create an IAM role
  2. Provision a micro EC2 instance
  3. Tag your instances

STEP 1: Create an IAM Role

On the IAM Management Console, go to Roles and click Create New Roles to open a wizard to start the process to create a new role. In the screenshots below, I called the IAM role as ec2-operator. You can call it whatever you want as long as you can identify it.

roles

Type a self-explanatory name to the Role Name field. It is used for identifying your IAM role. You will need to use the name later when you create a micro EC2 instance.

role-name

On the next screen, select AWS Service Roles and then Amazon EC2.

create-role

On the next screen, select Custom Policy.

custom-policy

Type an arbitrary name to the Policy Name field.

policy-document

Paste the following JSON block to the Policy Document field to indicate the IAM role is allowed to perform describe instances, start instances and stop instances action.

Review the role information and click Create Role to finish. The new role should be created immediately.

create-role

STEP 2: Provision a micro EC2 instance

I recommend to use a micro EC2 instance. I think it is sufficient to handle what we need here to set up a cron job to run a Python script to process auto-start and auto-stop requests, but feel free to use a different instance type.

On the EC2 Management Console, click Launch Instance to start the process to provision a micro EC2 instance. Let’s call this instance as an EC2 operator.

lanuch-instance

I recommend to select the latest Amazon Linux AMI, which has AWS tools, python and boto pre-installed.

linux-ami

You can pretty much take the default value for each step except on Step 3: Configure Instance Details and Step 6: Configure Security Group.

On the Configure Instance Details screen, you need to make sure you select the IAM role you created and assign it to the instance. In my case, I selected ec2-operator as the IAM role. You will not be able to change the IAM role for the instance once it is launched.

instance-details

You can cut and past the following shell script  into the User Data field. It will install few Python libraries and modules, create a Python script, and set up a cron job to execute it every 5 minutes. The shell script is set to operate instances for all regions. In other words, the script will go through each region to start and stop instances.

On the Configure Security Group screen, make sure you are not open the instance to the world (0.0.0.0/0) for the security group. In fact, you can get rid of the security rule. You don’t really need to SSH to the instance.

STEP 3: Tag your instances

The last step is go to EC2 Dashboard Console to tag your instances that you want to start and stop automatically at certain time. The EC2 operator instance looks for the auto:start and auto:stop tags to determine when to start and stop instances. The name of the tags are case-sensitive. The value of the tags should be in a cron format. For instance, 0 14 * * * is 2 pm everyday. The time is based on the OS time of the EC2 operator instance. If you have not changed the default time zone of the EC2 operator instance, it should be in UTC. It will ignore any instances that do not have the tags set or the format is invalid. The auto:start tag holds the start schedule, and the auto:stop tag holds the stop schedule. Do not tag the EC2 operator instance, otherwise it will get stopped by itself.

auto-tags

The script may not start or stop instances at the requested time exactly. For the start operation, it will start the instance between 30 minutes prior to the start time and the requested time. For the stop operation, it will stop the instance between the requested time and 30 minutes after the requested time. The EC2 operator instance is set to check every 5 minutes. It will process any auto-start and auto-stop requests few times if they have not started or stopped during the 30-minute window in case the operation does not go well.

I have been using the script for a while, and it has been performing pretty well. I am able to get all my development instances stopped during non-business hours.  I am even able to check the operations with CloudTrail to audit if the instances were indeed started and stopped within few minutes of the requested time.

Posted in Uncategorized | Tagged , , , , | 33 Comments