Using the profile module to time Ansible playbook runs
This past weekend I spent some time revamping a few playbooks. One of my playbooks was taking a while to run, and I wanted to see how much actual time was spent in each task. Luckily for me, Ansible...
View ArticleUsing the Ansible async module to perform synthetic health tests during...
I’ve become a huge fan of Ansible’s async support. This is incredibly useful for performing synthetic health tests on services after a task completes. A common use case is patching a server that hosts...
View ArticleListing Kubernetes nodes by label
When you support large Kubernetes clusters, you need efficient methods to list pods, nodes, and deployments when you are troubleshooting issues. Kubectl has a number of built-in methods to do this. You...
View ArticleUsing grpcurl to interact with gRPC applications
The past couple of weeks I have been digging into gRPC and HTTP2 in my spare time. I needed a way to review the requests and responses, and an easy way to explore gRPC servers. I also wanted something...
View ArticleUsing the terraform console to debug interpolation syntax
I am a long time Terraform user. The number of providers that are available for Terraform, and having a resource for pretty much every cloud service makes it super appealing. But even with several...
View ArticleAdding default tags to AWS resources with Terraform
If you’ve worked with the various cloud providers, you’ve probably realized the value that comes with tagging resources. For billing and searching, I like to create a default set of tags that are...
View ArticleUsing Terraform for_each statements and dynamic nested blocks to simply AWS...
Over the past few months, I’ve been updating various Terraform modules to utilize the new features in 0.12. Among these, is the ability to iterate over dynamic blocks with for_each. Utilizing this new...
View ArticleFinding Kubernetes issues with Popeye
Kubernetes is an incredible platform, but there are a lot of things that can go wrong. This is especially the case when you are new to K8S, and are overwhelmed with configuration options, deployment...
View ArticleManaging multiple Kubernetes resources by label
Kubernetes labels are super useful. If you aren’t familiar with them, a label is a key/value pair assigned in the metadata section (either metadata.labels, or spec.template.metadata.labels) of a...
View ArticleDisabling cron jobs globally on CentOS machines
As a long time CentOS user, I’ve always winced when I took ownership of new systems with e-mail notifications enabled. Whether this was through setting MAILTO to a distribution list, or using the...
View ArticleTRIM support is enabled by default in Fedora 32
As a long time Fedora user, I like to keep up with the planning discussions that go into each release. These discussions are super useful for understanding what is coming to Redhat Enterprise Linux and...
View ArticleUsing node local caching on your Kubernetes nodes to reduce CoreDNS traffic
Kubernetes 1.18 was recently released, and with it came a slew of super useful features! One feature that hit GA is node local caching. This allows each node in your cluster to cache DNS queries,...
View ArticleImproving my Linux diff experience with icdiff
I recently came across icdiff. This little gem allows you to see the difference between two files, but what makes it special is its ability to highlight the differences (sdiff, which was my go to diff...
View ArticleCollecting Nginx metrics with the Prometheus nginx_exporter
Over the past year I’ve rolled out numerous Prometheus exporters to provide visibility into the infrastructure I manage. Exporters are server processes that interface with an application (HAProxy,...
View ArticleDecoding JSON Web Tokens (JWTs) from the Linux command line
Over the past few months I’ve been spending some of my spare time trying to understand OAUTH2 and OIDC. At the core of OAUTH2 is the concept of a bearer token. The most common form of bearer token is...
View ArticleUsing dockle to check docker containers for known issues
As an SRE, I’m always on the look out for tooling that can help me do my job better. The Kubernetes ecosystem is filled with amazing tools, especially ones that can validate that your clusters and...
View ArticleUsing the sslsplit MITM proxy to capture Docker registry communications
This past weekend I got to debug a super fun issue! One of my Kubernetes clusters was seeing a slew of ErrImagePull errors. When I logged into one of the Kubernetes workers, the dockerd debug logs...
View ArticleHow the docker pull command works under the covers (with HTTP headers to...
I talked previously about needing to decode docker HTTP headers to debug a registry issue. That debugging session was super fun, but I had a few questions about how that interaction actually works. So...
View ArticleControlling the inventory order when running an Ansible playbook
This week I was updating some Ansible application and OS update playbooks. By default, when you run ansible-playbook it will apply your desired configuration to hosts in the order they are listed in...
View ArticleDebugging Kubernetes network issues with nsenter, dig and tcpdump
As a Kubernetes administrator I frequently find myself needing to debug application and system issues. Most of the issues I encounter can be solved with Grafana dashboards and Prometheus metrics, or by...
View Article