Listen on Youtube

My pup and I missed each other while I was in Chicago this week!

🌘 🦊

It's one thing to do work in spite of nay-sayers and critics. It's another, greater challenge to continue on in a direction different from where the admirers and allure of "success" draws you.

Tim Kreider exploring the stubborn nature of painters De Chirico and Derain:

What I can’t help but admire about them is their indifference to critics and admirers alike, their untouchable self-assurance in their own idiosyncratic instincts and judgment. I admire their doggedly following their own paths, even if I’m not as interested in where they led. I admire their cussedness. It’s a value-neutral quality in itself, cussedness, amoral and inæsthetic, and not one you can really emulate, anyway—it would be like trying to imitate originality. But such artists’ careers demonstrate that it is at least possible to move through this world unswerved by its capricious granting or withholding of approval. They’re examples, good or bad, to their fellow artists as we all feel our blind, groping way forward—or, sometimes, back—through the dark of creation.

Kreider's closing metaphor on moving through "The dark of creation" is something that the richest creation requires returning to time and time again.

(Found while digging deeper on the always fantastic weekly newsletter by Austin Kleon, this week on writing.)

I've been spending time diving into instrumenting our serverless apps with Open Telemetry.

The situation: We've previously used a custom logging solution to monitor a few cron jobs. It's worked well for a time. As we scale, though, we've started looking for another solution.

An issue with most off-the-shelf monitoring services is the amount of vendor lock in you set yourself up for. Thus, the Open Telemetry protocol came about.

The gist is that it's a standardized way your application can emit:

• Traces: Path of a request through your app. Think multiple lambda functions. Broken up into individual spans
• Metrics: Measurement captured at runtime. Anything you'd like to quantify.
• Logs: What you emit from your logger. Can be tied to a trace.

These data points are then sent from the application to a collector that can send these to your backend of choice. Many vendors have supporting libraries, including big players like Data Dog and New Relic.

Concepts

There are a few concepts to understand before getting started. You'll likely be setting up several services to instrument your application.

The pieces are:

• Instrumented App: Your code wrapped in an SDK to create traces, metrics, and logs.
• Exporter: Responsible for sending your data to the collector. Can exist within your application.
• Collector: Can receive and send Open Telemetry data. This is where you will point your application to.
• Back End: Your vendor or self hosted solution specific service for storing and rendering your data.

Instrumenting The App

The fastest way to get started is with a zero code instrumentation. There are slick ways for the Open Telemetry (OTel) SDK's to integrate with a web server like Flask.

For an application without a framework, though, there's a bit of manual work to do:

The official OTel docs outline the packages to import. You'll have to choose between gRPC or http protocol as your means of exporting. (Though, it seems the gRPC implementation is mostly a dressed up HTTPS protocol.)

Once you have it setup, emitting traces is straightforward:

with tracer.start_as_current_span(fn.__name__, links=links) as span:
    span.add_event(f"{fn.__name__}() started")
    
    # Do work
        
    span.add_event(f"{fn.__name__}() complete")
    return result

Setting Up the Collector

You could point your OTel config directly to your backend. Much of the benefit, though comes from using this in a distributed system. Here, a dedicated collector comes in handy.

If you're in an AWS environment, you can take advantage of the AWS Open Telemetry Collector. Setup here is primarily a bit of configuration.

My OTel project is still a work in progress, so that's all for now! I'll leave the nitty gritty for another post.

Listen on Youtube

🦋 Adding color with those chord extensions!