Listen on Youtube

Been enjoying sightreading short and sweet 5-finger pieces like this.

A study of my destop companion 🦖

Of all the pages of official docs in Python, the page on sorting by Andrew Dalke and Raymond Hettinger may be my favorite. Clear, gradually continues on in complexity, and provides ample examples along the way.

Here's my situation this week: Simple enough, I needed to sort a list of dictionaries by a property on those dictionaries:

data = [
    {
        'name': 'Chris',
        'count': 5,
    },
    {
        'name': 'Lucy',
        'count': 3,
    },
    {
        'name': 'Clyde',
        'count': 3,
    },
    {
        'name': 'Miranda',
        'count': 10,
    },
]

To sort by the count, I could pass a lambda to access the property on the dict:

sorted(data, key=lambda x: x['count'])

Say that the counts are equal as it is with Lucy and Clyde in my example. If so, I would want to sort by the name.

Returning a tuple covers that:

sorted(data, key=lambda x: (x['count'], x['name']))

To reverse the order, there's a named property for that:

sorted(data, key=lambda x: (x['count'], x['name']), reverse=True)

Problem all sorted out!

match = next(x for x in db_data if x["PropertyId"] == parsed_incoming_data["PropertyId"])

Breaking it down:

• next() returns the first value of an iterator. In subsequent calls, it would return the following item.
• next requires an iterator. An iterator yields number of objects on the fly. This is different from a list which contains and stores values. Lists and tuples can be converted to iterators with the iter() method.
• In my example above, the list comprehension x for x in db_data above yields an iterator, covering our type requirement for next.
• We're filtering by matching another value: if x['PropertyId'] == parsed_incoming_data['propertyId]

Voilà! Filtering for a match in one line.

Listen on Youtube

Enjoying the space of these chords 😌

An outdoor study

As Jamison Dance put it in this week's episode of the Soft Skills Engineering podcast: "It takes just one more level of abstraction to be a great engineer!"

A task at work this week had me looking to decouple one of our components that uses a third party library. Let's say it's a bar graph that uses something like Chartist.js. We want to be able to reuse this chart in both full page settings as well as where it's a widget inserted into a page. In one case, clicking a bar would open a tooltip. In another, it may link to another page.

Normally, here are my considerations for doing that:

1. From the base component, elevate as much of the logic as possible. The child should essentially be a "view" component only concerned with rendering data.
2. Pass any interactivity down through callbacks such as "onClick", "onMouseEnter", etc.

That works fine in this example below:

const BarGraphContainer = (props) => {
    const dispatch = useDispatch();

    const onClickBar = (segment) => {
        dispatch(actions.openModal(segment));
    };

    return (
        <BarGraph
            onClickBar={onClickBar}
        />
    );
};

The Problem

In this instance, the base component is handling a third party library for initialization, setup, and much of the internal interactions. In some cases, I want to control firing off an event in that internal library (ex: opening a tool tip on click). But in other cases, I want an external behavior (linking away or opening a modal.)

Passing Context Up Stream

An interesting solution I came up with was one that I had seen in other libraries and ecosystems: Passing context to the callback.

When considering passing callbacks in react, a simple use case typically only passes an event object.

const onClick (e) => e.preventDefault();

However, if I need access to the internal API of our third party library, I can pass that up through the callback as well. Even batter, I can abstract most of the internal workings of the library with a wrapper function. Take a look at the example:

const BarWidgetContainer = (props) => {
    const onClickBar = (segment, graphContext) => {
        graphContext.renderToolTip(segment);
    };

    return (
        <BarGraph
            onClickBar={onClickBar}
        />
    );
};

Here, the renderToolTip function likely has a great deal of logic specific to the library I'm interfacing with. At a higher level, though, I don't have to worry about that at all. I can simply call for that functionality if needed from a level of abstraction.

Use Cases

As mentioned, the abstraction is great for providing flexibility without complexity. Consumers of the component can interface with the internals without getting into the weeds of it.

A major con, though, is the added decoupling. In most cases, this could be seen as an anti-pattern in React given the uni-directional flow that's preferred in the ecosystem.

Considering these points, we ultimate decided on another solution that allowed for the parent to child data flow. It makes the most sense for our situation since it keeps our code cleaner. Realistically, we're also only using this component in a hand full of use cases.

But why did I write this up, then? I'm keeping the pattern in my back pocket. Situations where I can see this being useful are in broader use cases. Say that instead of our internal React component, this were part of a larger library consumed by more users. The trade off of coupling for abstraction and flexibility might make sense in a more widely used library. That's likely why it's a frequent pattern in open source tools, after all.

It was a fun experiment this week! Saving this pattern for another time.

Listen on Youtube

Trying out stride piano! All without looking at my hands 🙌

Had my breath taken away by the background painting work of Merle Cox, Ray Lockrem, Robert Storms, and W. Richard Anthony in Disney's Fantasia's Night on Bald Mountain. Decided to explore the colors in a different context!

Sharing a bit of initial research for a serverless project I'm working on.

I've been wrestling this week with a challenge to coordinate a cron job on AWS Lambda. The single script running on Lambda is ideal. There's no server overhead, the service is isolated and easy to debug, and it's cost effective. The challenge, though, is how to scale the process when the run time increases.

Lambda's runtime limit at this moment is 15 minutes. Fairly generous, but it is still a limiter. My process will involve web scraping, which if done sequentially, could easily eat those 15 minutes if I have several processes to run.

The process looks something like this so far:

1. Initial function start, get a list of pages to crawl.
2. Sequentially crawl those pages. (separate lambda function called sequentially)
3. After page crawl, send an update on the status of the crawl and update the DB with results.

Simple when it's 3 or 5 pages to crawl for a minute each. But an issue when that scales up. Not to mention the inefficiency of waiting for all processes to end before sending results from the crawl.

This would be a great opportunity to lean on the power of the microservice structure by switching to concurrent tasks. The crawls are already able to be sequentially called, the change would be figuring out how to send the notification after the crawl completes.

To do this, each of those steps above can be broken up into their own separate lambda functions.

Once they're divided into their own serverless functions, the challenge is too coordinate them.

Self Orchestration

One option here would be to adjust my functions to pass state between functions. Above, the 1st lambda would grab the list of pages, and fire off all instances of the 2nd lambda for each page. The crawler could receive an option to notify and update the db.

It's a fine use case for it! With only three steps, it wouldn't be overly complicated.

To call a lambda function asynchronously, it simply needs to be marked as an "Event" type.

import boto3

lambda_client = boto3.client('lambda', region_name='us-east-2')

page = { ... }

lambda_client.invoke(
  FunctionName=f'api-crawler',
  InvocationType="Event",
  Payload=json.dumps(page=page)
)

Step Functions

Say it were more complicated, though! 3 more steps, or needing to handle fail conditions!

Another option I explored was using Step Functions and a state machine approach. AWS allows for the ability to orchestrate multiple lambda functions in a variety of patterns such as Chaining, branching, and parallelism. "Dynamic parallelism" is a patern that would suit my case here. Though, I may not necessarily need the primary benefit of shared and persistent state.

The Plan

For this use case, I'm leaning towards the self orchestration. The level of state being passed is not overly complex: A list of pages from step 1 to 2, and then a result of success or failure from step 2 to 3. The process has resources in place to log errors at each step, and there's no need to correct at any part of the step.

Next step is the implementation. To be continued!

Listen on Youtube

Walking that bass! Trying out playing without looking at my hands as well.

Followed this great video tutorial for the tune

Really enjoyed Netflix's Avatar! Such a feast for the eyes 🤩

Incase you missed it! Celebrating the arrival of spring! I've always loved how these two flowers speckle the hills and fields this time of year.

Purchase on 🤘 Bandcamp and Listen on 🙉 Spotify or any of your favorite streaming services!

Busy week, but I have a quick edge case to share:

We've upgraded to a newer version of Python for our microservices at work. While making the transition, I bumped into an error sending http requests through urllib3:

SSL: CERTIFICATE_VERIFY_FAILED

I've come across this issue before, having to manually add certs. The solution involved low level work and was gnarly to handle! Thankfully, though, with Python 3.6 and higher, the solution is more straightforward.

It's largely outlined in this Stack Overflow post by Craig Glennie. The gist is that on Mac, you actually have to run a post install script located in your Applications/Python folder.

If you follow along with Craig's post, you'll find that the documentation for it is even there in the Python README. Always good to read the manual!

Listened exclusively to early Hot Five and Hot Seven recordings of Louis Armstrong the other week. Nothing better to brighten your mood!