After painfully going through the first Load Balancer I found it necessary to convert the steps below into a one liner.

List all Load Balancers in the Subscription
$ az network lb list

List all frontend IP Objects in the Load Balancer
$ az network lb frontend-ip list --lb-name kubernetes --resource-group <your resource group>

Show the AzureRM Resource ID for the IP object.
$ az network lb frontend-ip show --name <object name> --lb-name kubernetes --resource-group <resource group name> --output json --query publicIpAddress.id

Show the actual IP Address
$ az network public-ip show --id <azurerm resource id> --query ipAddress

So you can see in the list above it takes 4 separate commands to get to the IP Address. You can shortcut that list if you know the specific objects in the detailed output. So now below you will see a shorter version of the 4 steps above. The output of the command below will be the Azure Resource ID and the IP Address. I find the Resource ID plentiful for the environments I maintain as the Resource Group name tells me Region, Product and Environment each resource lives in.1 (Example ue-xyz-dev-<extra details>. Where "xyz" is the product abbreviation). You may need to alter this to be more detailed for your needs.

$ for id in $(az network lb list --output json|jq -r '.[].frontendIpConfigurations[].publicIpAddress.id');do print "\n${id}"; az network public-ip show --output tsv --id ${id} --query 'ipAddress'; done

Hopefully this will save you some time looking up resources in your Azure environment(s).

You will need to have installed and configured on your system:

• Python
• mysql-client
• gpg [https://www.gnupg.org/download/]

The code:

#!/usr/bin/python3

import os
import sys
from os import walk

gpgKey = '%%passphrase%%'
thisTable = '%%table_name%%'

filesPath = '/data/incoming/'
files = []
for ( dirpath, dirnames, filenames ) in walk( filesPath ):
    files.extend( filenames )
    break

for file in files:

    fileWithPath = filesPath + file

    # drop table if exist
    tbl = file.replace( '.sql.gz.gpg', '' )
    cmd = "mysql " + thisTable + " -e 'DROP TABLE `" + tbl + "`'"
    os.system( cmd )

    # extract encrypted compressed file into temp dir
    cmd = "gpg --decrypt --passphrase \'" + gpgKey + "\' --no-use-agent --cipher-algo AES256 " + fileWithPath + " | gunzip > /data/tmp/working.sql"
    os.system( cmd )

    # import data into MySQL database
    cmd = "mysql " + thisTable + " < /data/tmp/working.sql"
    os.system( cmd )

    # remove temp file
    os.remove( '/data/tmp/working.sql' )

    # remove uploaded file
    os.remove( fileWithPath )

This document expects that you have some basic understanding of Linux and
command line experience.

First you need to clone this repo. With the following command you can place this
repo in a directory of your choice. Note the trailing period which means to
clone into the current directory. Removing the trailing period will clone this
into its own directory within the one you have changed to in the cd  command.

cd to/your/directory/of/choice
git clone git@github.com:k7faq/ansible-container.git .

EXAMPLES
Both of these examples accomplish the same task.

mkdir /user/home/myuser/containers/; cd $_
git clone git@github.com:k7faq/ansible-container.git .

OR

mkdir /user/home/myuser/containers/
cd /user/home/myuser/containers/
git clone git@github.com:k7faq/ansible-container.git .

Then you need to build the container using the following command FROM WITHIN the
directory you cloned your repo into. You need to build the container. A shell
script is provided:

./build

You now have a container built named ansible_2.7.

During the build process above an alias was created on your behalf. Provided
your Linux distro uses a .bash_profile  in your home directory to store custom
settings you should be good to go using the following commands:

If all has succeded then you should be greeted with success reports like this:

PLAY RECAP *****************************************************************

localhost                  : ok=1    changed=1    unreachable=0    failed=0   

How do I use this?
This container configuration allows the user to specify the command they wish to
execute against the installed and configuration application(s). Such as Ansible
offers several executable commands, including ansible  and ansible-playbook,
etc. Following are two examples of how to use this container:

ansi ansible-playbook configure_env.yml -vvvv


ansi ansible --version

Why does this container not have an ENTRYPONT or CMD?
Ansible offers more than just one command (functional feature). Not specifying
either of these features allows me to specify which command I want to execute
without needing multiple containers. Yes, there may be other options, this is
just that I have chosen.

Some notes of interest
If you take time to look at the contents of the run command you will find that
your .ssh/  directory in your user home directory is passed to the container.
Why? Simply because of how Ansible works using ssh keys to connect to resources.
In my case I only use a username and password during the initial startup of a
VM. This initial step creates the user id and establishes the appropriate key(s)
for future connections. Thus each subsequent connection will utilize the key(s)
in my user home directory -OR- from the location specified.

HELP! All has failed
If the above failed, please look in your ~/.bash_profile  file to verify an
alias of ansi  was created for you. If you find one then please log out and log
back into your terminal window and try again. If still no success then try one
of the following:

Create a custom shell script * This will need to be in your $PATH

• use echo $PATH  to verify your options
• typically /usr/local/bin/  is a great place for these

Create a sym(bolic) link named ansi

echo `sudo ln -sf ${PWD}/run /usr/local/bin/ansi
ll /usr/local/bin/ansi

• if not using Ubuntu (or a Debian product) * verify which file is used by the
  OS to establish your users environment at login

Permissions
As with any other AWS Service you will need to declare permissions. You have a
choice of asserting Access Keys on your machine or leveraging the power of Roles
and thus limiting the risks associated with unauthorized access and usage.

On your controlling machine you need to set some AWS permissions. Custodian will
by default look for the default AWS Credentials configurations. Thus creating
the following files and configuring them accordingly.

~/.aws/credentials

[sandbox-playground]
arn:aws:iam::#YourAccount#:role/custodianAccessRole
source_profile=default
region=us-east-1


~/.aws/config

[default]
region=us-east-1

Example usage:

custodian run --profile sandbox-playground --output-dir . --verbose asg.yml

Let's look at some code
Container:

# Dockerfile 
FROM python:3.7-alpine

RUN pip install c7n c7n-org \
    && mkdir -p /scripts/

Note the absence of ENTRYPOINT  and CMD  in this Dockerfile. This allows the
passing of commands making the usage more general. This will be better
understood as we get to the Usage section below.

Build script
#!/bin/bash 
docker build -t="custodian" .

## NOTE ## Make sure to update the path (/sr/custodian/run) to the full path of the directory in which you place these files. 
## The following will create a SymLink (Symbolic Link) to the run file saving you the hassle of referencing the full path 
## each time you execute the command.
ln -sf /srv/custodian/run /usr/local/bin/custodian

Run Script
The "$@" appends ALL the verbiage following the $ custodian  command. Reference
the Usage section below for examples.

docker run -v "${PWD}:/policies" -v "${HOME}/.aws/:/root/.aws/" -v "/tmp/custodian:/tmp" -e AWS_PROFILE -it --rm custodian "$@"

Yes, it would be more readable as below. However I have experienced too many
issues of it not being interpreted properly and use the above without issues.
Keeping in mind that I cross multiple platforms daily.

docker run \
  -v "${PWD}:/policies" \
  -v "${HOME}/.aws/:/root/.aws/" \
  -v "/tmp/custodian:/tmp" \
  -e AWS_PROFILE \
  -it --rm \
  custodian "$@"

Usage
$ custodian run --region us-east-1 --output-dir . --verbose custodian_aws_asg.yml

# NOTE the usage of the alias c7n
$ c7n custodian_aws_asg.yml

# NOTE the usage of the alias c7n-org
$ c7n-org run --config accounts.yml --output-dir . --tags path:/myEnv/Sandbox --dryrun --use custodian_aws_asg.yml

Aliases that I use
I use many aliases on my machines. These are meant to reduce the amount of
typing and increase efficiency not having to remember every key switch or the
unintended typos costing time to correct, etc. Here is an example alias as shown
in the Usage section above.

alias c7n='custodian run --output-dir=/tmp '
alias c7n-org='custodian c7n-org'

Example AWS User Data
#!/bin/bash
apt-get update -y --fix-missing
apt-get install -y unzip python python-pip python3 python3-pip
pip3 install awscli --upgrade

# Install Docker
curl -sSL https://get.docker.com/ | sh
systemctl enable docker; systemctl start docker

aws s3 cp s3://%% YOUR BUCKET %%/custodian_Container.zip /tmp/custodian_Container.zip
mkdir -p /srv/custodian/
unzip /tmp/custodian_Container.zip -d /srv/custodian/
mkdir -p /root/.aws/
mv /srv/custodian/aws/* /root/.aws/
cd /srv/custodian/ && chmod +x run && sh build
ln -s /srv/custodian/run /usr/local/bin/custodian

Use MFA (Multi-Factor Authentication) or Versioning
This AWS Document
[https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html]  discusses
this topic in more detail. One of my coworkers reported having encountered
extreme difficulty in trying to delete files from a bucket and the bucket itself
after affecting the MFA protection. While MFA will protect your bucket from
deletion the other option is Versioning. Versioning will allow you to store
multiple versions of your files.

An easy way to protect your bucket with the MFA Delete protection and/or
Versioning is via Terraform. Using Terraform you can apply versioning to your
bucket like this example: (Note the versioning).
In this example we are enabling versioning by using the enabled = true
statement and forcing MFA Delete protection with the mfa_delete = true
statement.

resource "aws_s3_bucket" "b" {
  bucket = "my-tf-test-bucket"
  acl    = "private"

  versioning {
    enabled = true
    mfa_delete = true
  }
}

I will include other IaC examples as time permits.
Future enhancements:

• Ansible
• Others?

Containers
I fully support Containers and containerization of services. I agree with the
portability viewpoint, but for me it is more of keeping a "clean" system. The
fewer services installed the fewer conflicts to work out and the fewer updates I
must worry about conflicting with installed services and so on. Next there is
the issue of bundled configurations minimizing conflicts of packages and updates
allowing less of an oppotunity of downtime while forced to troubleshoot a
package update. Although building containers on the fly does not protect you
from this. To truly appreciate this means of minimizing conflicts it would be
best to consider hosting your own Docker Hub or publish your containers
publicly.

What I tried
Initially I was trying to run a Container on my notebook with a Runner in an AWS
EC2. This will NOT work as the runner must be able to communicate with the
GitLab Server via an IP or FQDN. Next I tried running both the GitLab CE Server
and a Runner in the same EC2. In doing so I found a t2.micro is insufficient for
this purpose, although a t2.large is capbable of handling a small learning
configuration.

As I support IaC (Infrastructure as Code)
I built a Terraform Module that creates the AWS Environment and necessary
resources:
Terraform code used to create the AWS Environment is available here: GitHub
[URL].

Once you have your VM running with Docker installed, this User Data will
establish your GitLab-CE Container. Once completed you can access your new
GitLab-CE environment via http://your_domain.tld/. Be sure to append the URL
with your custom port number if you elect to use a port other than 80.

#!/bin/bash

apt-get update -y
apt-get install -y awscli unzip

Install Docker

curl -sSL https://get.docker.com/ | sh
systemctl enable docker; systemctl start docker

######### GITLAB SERVER
HOSTNAME=$(curl http://169.254.169.254/latest/meta-data/public-hostname)
docker run --detach
--hostname "$HOSTNAME"
--publish 80:80 --publish 2289:22
--name gitlab
--restart always
--volume ~/Library/Docker/gitlab/config:/etc/gitlab
--volume ~/Library/Docker/gitlab/logs:/var/log/gitlab
--volume ~/Library/Docker/gitlab/data:/var/opt/gitlab
gitlab/gitlab-ce:latest

######### GITLAB RUNNER
mkdir -p /srv/gitlab-runner/config
docker run -d --name gitlab-runner --restart always
-v /srv/gitlab-runner/config:/etc/gitlab-runner
-v /var/run/docker.sock:/var/run/docker.sock
gitlab/gitlab-runner:latest

This next step is not automated as I did not take time to figure out if an API
exists to extract this info, et al.
Following the deployment of the EC2, you will need to associate the Runner with
the GitLab Server:

HOSTNAME=$(curl http://169.254.169.254/latest/meta-data/public-hostname)
TOKEN=%% get from your gitlab server %%
docker run --rm -t -i -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register
--non-interactive
--executor "docker"
--docker-image ubuntu:18.04
--url "%% URL TO YOUR SERVER%%"
--registration-token "${TOKEN}"
--description "%% YOUR RUNNER DESCRIPTION%%"
--tag-list "test,AWS"
--run-untagged
--locked="false"

1. Be sure to set your --executor  accordingly docs
   [https://docs.gitlab.com/runner/executors/README.html]. I chose Docker.
2. Set the --docker-image  you will use each time your runner executes. Be sure
   to plan this accordingly. If you chose to use Docker for your Runners and
   you set Alpine here as the distro you will need to make sure your runners
   are configured to use Alpine packages, etc.
3. Set the --url  to point to your server. E.g. http://gitlab.mydomain.com.
   This is documented in your GitLab-CE Server. Login to your GitLab-CE server.
   On the lower left click: Settings | CI/CD | Runners. Scroll down "Specific
   Runners" and you will find the URL.
4. Set the --registration-token. You will need to access your GitLab-CE Server
   to obtain this. Login to your GitLab-CE server. On the lower left click:
   Settings | CI/CD | Runners. Scroll down "Specific Runners" and you will find
   the registration token shown below the URL.
5. Set the --description  to something meaningful. This will help you
   distinguish this runner from your other runners as you create them in the
   future.
6. Set the --tag-list  with tags that will be used to kick off this runner
   process.

During this event Microsof Azure Rapid Response Technical Support requested copies of IP Tables dumps etc to allow them to investigate. For experienced Linux Administrators this would seem simple enough - ssh admin@cluster -c 'iptables -L' > iptables.out, right? Not so quick on a Managed Kubernetes Cluster. One of seveal solutions to this problem is to use Node Shell to access the Kubernetes node.

The following demonstration is based on Azure Kubernetes Service. The kubectl commands are universal to any Kubernetes Cluster.

List the clusters in the Subscription.

# List the Clusters
az aks list

Obtain the credentials for the Cluster

# Get the Credentials
az aks get-cedentials --name <name of the cluster> --resource-group <resource group name>

List the nodes

kubect get nodes --output wide

Enter the Node. Essentially we are SSHing into the node via the Node Shell. As this demo is based on a "managed" deployment, I do not have the ability to directly SSH into the node. Albeit, yes - I could have created a pod with the proper private SSH key to accomplish the same, but in the interest of time ...

kubectl node-shell aks-cluster1-33854278-vmss000000

What has happened at this point is that a nsenter pod has been created for you on the node. When you exit the node-shell session the pod will be removed automatically.

Package the messages and syslog files up in a tar ball. Here I am using GZip compression. Note that for distinction I am using the pattern of <node name>.<log file type/name>.tar.gz. This identifies that we are creating a tar ball using gzip compression and which type of files for which machine.

tar czf aks-cluster1-33854278-vmss000000.messages.tar.gz /var/log/messages*
tar czf aks-cluster1-33854278-vmss000000.syslog.tar.gz /var/log/syslog*

Do not exit this terminal session yet.

Now we need to leverage the pod that was created so we can transfer off the archived files. To do this we need to get the name of the pod. So here we will get a list of pods in the default namespace. We need to identify the pod having a name starting with nsenter.

kubectl get pods --namespace default

NAME             READY   STATUS    RESTARTS   AGE
nsenter-9knomx   1/1     Running   0          14m

Now that we have the name of the pod we can copy the archives to your local workstation. Here we use the kubectl cp command. Note the path to the files created ealier are referenced following the : following the pod name. This is the same methodology used in Rsync and SCP.

kubectl cp nsenter-9knomx:/tmp/aks-cluster1-33854278-vmss000000.syslog.tar.gz aks-cluster1-33854278-vmss000000.syslog.tar.gz

Now you can list the files you downloaded.

#Linux
ls -l *.gz
-rw-r--r--  1 srhodes  staff  15280652 Apr 16 20:26 aks-cluster1-33854278-vmss000000.messages.tar.gz
-rw-r--r--  1 srhodes  staff  14671049 Apr 16 20:27 aks-cluster1-33854278-vmss000000.syslog.tar.gz

For non-*nix users, you can use dir in place of ls.