A couple weeks ago I wrote about making our reports take a couple seconds instead of 3 minutes. What I discovered later is that we didn’t actually have access to historical reports, because all the DynamoDB entries that pointed to the S3 data behind those reports had a TTL of one day. After asking around, the reason was simple: some partition keys were exceeding 10GB, and that’s the DynamoDB item collection limit per partition key (aka “all items with the same partition key”). So the...
19 days ago • 3 min read
My friend asked me yesterday if I know what AWS is. And it’s not someone I only talk to once in a while. We literally talk every day. I guess I always refer to my employer as just Amazon, so “AWS” never comes up. He recently acquired an app and needed to create a new AWS account, add a new user for his developer, and give him permissions for Lightsail (whatever that is). He managed to do the first two. The permissions part? Yep, he had no idea. I’ve written about IAM before here. My goal...
26 days ago • 2 min read
My recovery after the ACL surgery took longer than expected, so I didn’t return to work until December, and even then I never worked a full week. But now it’s the new year and we have to start on this grind again. If you subscribed recently, welcome! I write about AWS and engineering work the way it actually feels, plus the occasional lesson learned the hard way. Since I’ve been back, the most notable thing my team did while I was out was build a package with a bunch of “Skills” for Claude....
about 1 month ago • 3 min read
Last week, I ran into this tweet: the tweet It kinda triggered me. Why would someone pay $0.40 per secret per month when you could just use AWS Parameter Store and store them as SecureStrings FOR FREE? That’s what I use for oneiras.com, so I was determined to find out if I’d missed something. Am I unknowingly paying per secret? Or is there actually a reason to use AWS Secrets Manager instead? Turns out, there are a couple, but only if you really need them. The Big One: Automated Secrets...
3 months ago • 2 min read
Well, the global AWS outage happened just four days after I sent a newsletter about COEs and how “nobody gets blamed.” Great timing, right? I wish I could’ve been in the weekly global ops meeting to see the temperature in the room. That’s the one where teams present their recent issues and learnings. I can only imagine how lively that one must’ve been. Turns out the culprit was a DNS failure in the Amazon DynamoDB endpoint in the us-east-1 region. And while that sounds region-specific, it...
4 months ago • 1 min read
Someone pushes a new feature to prod the same day you go on-call. Hours later, your phone goes off - not a gentle buzz, but a full-blown siren that could wake up the entire neighborhood. You open the alert, and it’s for a feature you didn’t even touch. Maybe it’s unhandled NPEs, maybe something else. Doesn’t matter. You’re the one on-call, so it’s your problem now. When Things Break In those moments, it’s usually faster to just debug and fix it - even without full context. I’m pretty good at...
4 months ago • 2 min read
About eight years ago, when I was still a QA, Microsoft Azure “lost” our primary database. Without it, we were basically out of business - it was the main source of truth for, well, almost everything. I don’t remember exactly what the database held anymore, but I do remember the chaos that day. And the stress. A lot of it. Today, I saw a tweet about how the Korean government had all its data in a single location, with no backups. It reminded me: we all know this lesson, but we keep relearning...
4 months ago • 3 min read
When I was first learning how to code, I spent tens - maybe hundreds - of hours glued to online courses. Video after video. Tutorial after tutorial. Once I “made it,” though, I found myself more drawn to books. Maybe it was because my employer stopped paying for PluralSight. Maybe because I suddenly had an O’Reilly subscription. Or maybe because I realized video pacing never matches what I need - either way too slow (wasting hours) or too fast (forcing me to rewind endlessly, or drop the...
4 months ago • 1 min read
While recovering from ACL surgery (yes, that’s why there was no newsletter last week), I started re-reading Great at Work. I’d read it about a year ago, but almost nothing stuck. Maybe that explains why I haven’t exactly been "great at work." Chapter 2, "Redesigning Your Work," hit me at the right moment. Then a friend sent me this article: Altoids by the Fistful. The gist? We used to love building stuff—fun weekend projects, scrappy hacks, things that made us feel alive as developers. Now?...
5 months ago • 2 min read
