Optimising S3 Configuration

https://docs.aws.amazon.com/cli/latest/topic/s3-config.html

~/.aws/config

[profile development]
aws_access_key_id=foo
aws_secret_access_key=bar
s3 =
  max_concurrent_requests = 20
  max_queue_size = 10000
  multipart_threshold = 64MB
  multipart_chunksize = 16MB
  max_bandwidth = 50MB/s
  use_accelerate_endpoint = true
  addressing_style = path

$ aws configure set default.s3.max_concurrent_requests 20
$ aws configure set default.s3.max_queue_size 10000
$ aws configure set default.s3.multipart_threshold 64MB
$ aws configure set default.s3.multipart_chunksize 16MB
$ aws configure set default.s3.max_bandwidth 50MB/s
$ aws configure set default.s3.use_accelerate_endpoint true
$ aws configure set default.s3.addressing_style path

List CloudWatch Log Groups

aws logs describe-log-groups --query 'logGroups[*].logGroupName' \
  | jq -r .[]

Delete Matching CloudWatch Log Groups

• https://docs.aws.amazon.com/cli/latest/reference/logs/delete-log-group.html

aws logs describe-log-groups --query 'logGroups[*].logGroupName' \
  | jq -r '.[]|select(.|contains("StringIWantToMatch"))' \
  | while read -r logGroupName
    do
      aws logs delete-log-group --log-group-name "$logGroupName"
    done

Finding AMIs for Packer

• https://www.packer.io/docs/builders/amazon-ebs.html
• https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html

aws ec2 describe-images \
  --owners amazon \
  --filters \
    "Name=name,Values=amzn2-ami-hvm-2.0.*" \
    "Name=description,Values=Amazon Linux 2 AMI 2.0.* HVM gp2" \
    "Name=root-device-type,Values=ebs" \
    "Name=owner-alias,Values=amazon" \
    "Name=virtualization-type,Values=hvm" \
    "Name=architecture,Values=x86_64" \
  | jq '.Images[]|.Description'

Filtering

• AWS Documentation » Amazon EC2 » User Guide for Linux Instances » Resources and Tags » Listing and Filtering Your Resources
• AWS Documentation » AWS Command Line Interface » User Guide » Using the AWS CLI » Controlling Command Output from the AWS CLI
• Using Amazon Web Services Command Line Interface to Find Instances without a ‘Name’ Tag
• JMESPath Specification

Remember that the --filter argument may also accept a JSON file containing a filter specification:

$ cat filters.json
[
  {
    "Name": "instance-type",
    "Values": ["m1.small", "m1.medium"]
  },
  {
    "Name": "availability-zone",
    "Values": ["us-west-2c"]
  }
]
$ aws ec2 describe-instances --filters file://filters.json

Taken from the manual:

Inverse search

You can search for resources that do not match a specified value. For example, to list all instances that are not terminated, search by the Instance State field, and prefix the Terminated value with an exclamation mark (!).

Partial search

When searching by field, you can also enter a partial string to find all resources that contain the string in that field. For example, search by Instance Type, and then type t2 to find all t2.micro, t2.small or t2.medium instances.

Regular expression

Regular expressions are useful when you need to match the values in a field with a specific pattern. For example, search by the Name tag, and then type ^s.* to see all instances with a Name tag that starts with an 's'. Regular expression search is not case-sensitive.

Instances not Matching Type

aws ec2 describe-instances --region=eu-west-1 --output json | jq -r '.Reservations[].Instances[]|(select( (.InstanceType|startswith("m5.")|not) and (.InstanceType|startswith("t3.")|not) ))|[.InstanceId, (.Tags[]|select(.Key == "Name")|.Value), .State.Name, .InstanceType]|join(",")'

AMI

$ aws --profile production --region eu-west-2 ec2 describe-images --owners amazon --filters "Name=root-device-type,Values=ebs" "Name=name,Values=Windows_Server-2019-English-Full-Base-*" "Name=virtualization-type,Values=hvm" --query "sort_by(Images, &CreationDate)[-1].ImageId" --output text
ami-023b643326f4d6eff