I’m currently working through the AWS Certified Solutions Architect – Associate (SAA-C03) curriculum. I’m about one third through with Stephane Maarek’s Udemy course and pretty much immediately I can see the value for federal IT environments. It’s eye opening.

Federal Cloud Is Here and AWS is Huge

Throughout the past decade, I’ve worked with mostly on-premises infrastructure. But the landscape has shifted. Cloud computing in the U.S. federal government is not only the norm, it’s almost considered a requirement. Even if you find savings or efficiency under the on-prem model, the policy pressure to go cloud is very real. A series of federal directives, from earlier Cloud First and Cloud Smart policies to recent OMB mandates, agencies are pushing to modernize legacy systems and migrate workloads to cloud environments (Don’t get me started on how they define legacy). For most agencies, this means AWS GovCloud or Microsoft Azure Government. Both platforms are tailored to address the compliance and security requirements of the public sector.

IT means operating in the cloud and not just reading about it, but working in it at the program management and architecture level, and with AI ramping up, it’s expected that program level managers even get into the weeds.

Finding Baked in Waste

Just a couple of weeks into the course, one thing becoming clear to me. The most immediate, practical value a solutions architect brings isn’t necessarily designing new infrastructure from scratch. It’s in auditing what already exists and identifying inefficiency. It’s everywhere!

A few concrete examples of the kinds of waste that are hiding in plain sight. You don’t have to know the specific terms to understand the waste.

1. Amazon EC2 offers instance types optimized for different use cases (varying combinations of CPU, memory, storage, and networking capacity). Right-sizing is the process of matching instance types and sizes to actual workload requirements. Organizations tend to provision instances for peak theoretical load and never revisit them. Right-sizing must become an ongoing process, not a one-time exercise.

2. Multi-AZ deployments (replicates data across multiple locations) and read replicas come at a cost. Running in multiple zones cost more, but may be required for production high-availability workloads. Some workloads can remain in a single zone if they are non-critical. How does the government define critical is the real question. Getting the stakeholders agreement that your application is not critical is the hard part. Defining the application as non-critical alone would save the costs without any other work involved - it’s free!
   

3. S3 Lifecycle policies allow you to transition infrequently accessed data to cheaper storage tiers automatically. If data is only retrieved once or twice a year, storing it in S3 Standard is throwing money into the trash. S3 Glacier or S3 Intelligent-Tiering would be more appropriate. Amazon S3 Storage Lens can identify cost optimization opportunities, and S3 Intelligent-Tiering can automate data lifecycle management. Define infrequent with your stakeholders. I bet they provision based on what-if and just-in-case scenarios. This is bad practice.

The underlying theme here is that it’s the human in the loop causing the costs to be high. Redefine what is critical. Stop worrying about edge cases. Be realistic about your needs. Behavior changes are the hardest changes to make in an organization, but they cost absolutely nothing, and can save thousands.

AI and ML Workloads

There’s another reason that makes this skillset increasingly important. AI and machine learning workloads are exploding within the government. This is driving demand for GPU compute, large-scale data pipelines, and complex storage architectures. What you need is the ability to evaluate the full infrastructure holistic view. Knowing where the data lives, how it moves, how frequently it’s accessed, what the compute pattern looks like, and whether the architecture actually matches the business need is a valuable skill to have. Don’t forget, convincing the decision makers will still be the hardest part. This soft skill can’t be underappreciated.

Understanding access patterns, retrieval frequency, data pipelines, and cost tradeoffs at the infrastructure level is where program managers with technical depth can differentiate themselves.

Where I Am and Where This Is Going

I’m inching towards the halfway point with Stephane Maarek’s SAA-C03 course and working hands-on in a live AWS environment. Certifications alone are worth absolutely nothing. I could drill TutorialsDojo practice exams, pass the SAA-C03, and still be useless in a real cloud environment. The credential is a door opener, something to compliment my other skills, nothing more. Hands-on projects, documented cost savings, architectural decisions is what you want in your portfolio.

That said, I’m aware this post stays at the conceptual level. I plan to go much deeper with the posts soon. I’m debating creating some hands-on deep dives that get into the granular mechanics of cloud cost optimization: how to actually find the waste, what the findings look like in the console (love the command line!), and what specific remediation steps produce real savings. Until then, there are plenty of other great resources for those looking to up skill.

Thank you for visiting my Substack at newsletter.markgingrass.com.

Is there anything wrong with hand and off complete ideas? Is there anything wrong with allowing computers to make big decisions that have real effects? Perhaps it's not as bad as you think.

Here’s the history of how we got here. Memory stored on clay tablets. Spread across populations through reading. Calculation to mathematics. Then to machines that did it faster than any human could. Then to networks that scaled all of it across the entire planet.

Every step was the same story with different tools. The printing press didn’t kill writers, it made literacy the baseline and pushed writers toward deeper meaning. The spreadsheet didn’t kill bookkeepers, it killed the need for rooms full of people doing arithmetic by hand and created the field of financial analysis. GPS didn’t kill navigators, it allowed precision as a survival skill and freed up cognitive bandwidth for everything else.

The pattern was always: tool handles the rote, human handles the judgment.

What’s breaking now is that we’re outsourcing the judgment itself because the tool is impressive, the pressures are real, and some bean counter said “we should be using AI for this.”

The human remainder

Here’s what every cognitive outsourcing event in history has in common: it created a premium on whatever was left. It commodities tasks that were once dedicated to skilled tradesmen.

When writing took over sheer brain memory, the premium moved to interpretation. What does this (the writing) mean and what do we do about it. When calculation moved to machines, the premium moved to knowing which calculation to run. When search engines indexed all human knowledge, the premium moved to knowing which question to ask.

AI is compressing code generation, document drafting, data synthesis, and pattern recognition. So the premium is moving again. It’s moving to judgment, relationships, and the ability to define what actually matters before anyone writes a line of code or drafts a single requirement.

If you’re a program manager in a federal agency right now and your response to AI is, “Let’s automate the reporting,” you may be optimizing for the thing that is becoming worthless: the artifact. In some cases, the artifact may never have had much value in the first place.

Think about what these artifacts usually are: documentation proving something was done according to regulation. And by “proving,” I mean it was written down, stored somewhere, and almost nobody will ever audit whether the paper matches reality. That should force a basic question: What value does this actually provide in practice?

If the answer is none, then the target should not be automation. The target should be removal. Get rid of the process. Get rid of the artifact. I know federal environments are not that simple. You cannot always delete a requirement because it annoys you. But you can challenge it, negotiate it, ask why it exists, or document that a section is not applicable instead of pretending it adds value. Try it out. Ask the receiving end why they need it, or if you can get by without it.

Where AI is worth it

AI is good for things that humans waste time on, of course.

It’s good for developers. GitHub’s research found that developers using AI coding assistants completed tasks roughly 55 percent faster than those working without one. 1Engineers can focus on the architecture, and less on the actual code. That’s a real trade with real productivity. It’s not much different than using C++ over assembly or machine language in computer science.

It’s good for the compliance artifact stack (EPLC, IT Boards, etc.), too. Almost nobody is reading that risk register. Nobody is scrutinizing the configuration management plan before they check the box. It’s going to live in a SharePoint folder until the contract ends.

When I say “nobody,” I do not mean literally no one. I mean very few people, if any, will ever read this material closely enough for it to matter. That is a larger problem, and probably a separate newsletter article. Not here.

If AI can generate a FISMA-compliant system security plan or a FedRAMP-ready continuous monitoring strategy in a fraction of the time it used to take, take the time back. Use it for something that matters until you can rid the entire process.

It’s good for getting something real in front of users faster. A working prototype in two weeks, two days, or even two hours, instead of six months means you kill bad ideas before they become programs of record. We all know management hates to admit sunk costs. In a space where programs have consumed hundreds of millions (or billions, ECSS2) of dollars before a real user ever touched the product, faster feedback loops are genuinely transformative.

These are real wins. They’re just not the wins the systems integrators are selling you.

The checkbox, good in theory, terrible in practice

Most compliance artifacts exist because a regulation says they must, not because anyone will use them. Trust me, I want to believe in the system. I want to believe that having everything “checked” guarantees a successful product. But the reality is different. Checkboxes are the last thing the product needs and the last task completed before submission. Look at the nearest public restroom. Check the cleaning log by the exit door. It may show that someone checked the box. But when was it actually cleaned?

The federal government now spends more than $100 billion on IT every year covering everything from legacy system maintenance to new acquisitions.3 That scale of investment has always attracted scrutiny: the GAO has flagged IT-related problems in its High Risk program since the early 1990s, and formally designated "IT Acquisitions and Operations" as its own High Risk area in 2015, a designation the office has renewed in every update since.4

If an artifact is never read, never used to catch a real problem, never changes a single decision, then it’s just theater. The box gets checked. The program moves forward. The document rots in a folder no one can find.

AI is perfect for theater. Generate it. Check the box. Move on. The real cost was never the artifact, it was the human spending three weeks on something no one reads. The only thing reading AI is more AI, turtles all the way down (get it?).

The acquisition cycle is its own trap

Here’s a problem that isn’t getting nearly enough attention: AI is moving at software speed, and federal acquisition is moving at regulatory speed.

The Federal Acquisition Regulation runs to thousands of pages. A competitive procurement from solicitation to award routinely takes twelve to eighteen months. A major IDIQ or GWACs contract vehicle can lock a program into a technology approach for five to ten years. An Authority to Operate (ATO), the security approval required before most federal systems can go live, can take six to eighteen months on its own.

AI capabilities are advancing on a timescale measured in months, even weeks. By the time you finish an ATO for a specific model version, that version has a handful of successors.

This is not an AI problem. And AI won’t fix it alone, because the constraint isn’t cognitive, it’s regulatory. It’s the protest timeline. “Other Transaction Authorities” exist to bypass some of the regulations. However, it takes takes someone with the knowledge, relationships, and institutional credibility to make the case for an exception (despite the justification written using AI, too). You still need to know who to call on.

That someone is a human. Specifically, one who has been around long enough to know when the rules bend and who needs to sign off on the bending of the rules.

What is the price tag

The pre-AI default was to acquire first, define later, lock in a contract, then spend years negotiating what the system should actually do. Agile was supposed to change that, and it's been gospel in federal IT circles for a decade. But genuine agile (iterative, user-driven, willing to kill bad ideas early) never really took hold in government. What took hold was the vocabulary and buzzwords.

Now: deploy the AI, then figure out what you need it to do (again backwards thinking).

Same trap. Faster, more expensive, with a better slide deck created by AI.

Requirements gathering can still be human work. Sitting across from someone and asking what they actually need. Finding the constraint they forgot to mention: the API that hasn’t been updated since 2007, the policy that blocks the entire workflow, the workaround that became standard practice three administrations ago and is now baked into every downstream system.

Healthcare.gov launched on October 1, 2013, and collapsed under load on day one.5 The technical failures were real, but the root cause was a requirements and governance failure. Dozens of contractors, no single integrator accountable for the end-to-end system, and a launch deadline that had become a political fixed point that no one was willing to move regardless of readiness. No AI tooling fixes a governance structure where nobody owns the outcome. That decision to hold the date, to accept the fragmentation, to not escalate, was a human failure at the leadership level.

You can’t prompt your way to what your users forgot to tell you. And you definitely can’t prompt your way to the hard call nobody wanted to make.

What you can’t prompt

When you have a real problem, you don’t run a query. You call the person who has done this before.

The one who reads the org chart the way Robert Caro reads a political biography, looking not for who holds the title, but for who holds the power (I loved the LBJ Series by Robert Caro). The one who turned a hard no into a yes in a meeting two years ago, on the strength of a relationship built over a decade.

Federal agencies are not flat organizations with clean reporting lines and transparent decision rights. They are layered institutions with career staff who have survived six administrations, political appointees who have eighteen months to make their mark, informal power structures built on years of working the same committees, and institutional memory that doesn’t live in any document. Knowing how to navigate that, who to brief before the formal meeting, who needs to feel included before they’ll say yes, which deputy’s chief of staff is actually running the budget process regardless of title, that is the real product. It took years to build. It doesn’t transfer to a model. In other words, the soft skills.

Persuasion is human. Getting the people in the room who aren’t decision-makers to pull in the same direction anyway, that’s the job underneath the job. No model has skin in that game.

The right measure

“I need a chatbot for constituent services” is not a requirement. It’s already a solution. Someone decided the answer before they understood the question.

The outcomes worth fighting for don’t sound like technology. They sound like: we caught $40 million in fraud before the money left the account. We cut veterans’ benefits processing time from nine months to six weeks. We gave case workers enough time back to actually see their clients instead of updating a database.

AI can contribute to all of those, and in some cases already is. The IRS has used machine learning to flag high-risk returns for audit selection. CBP uses AI to screen cargo manifests for anomalies that would take human analysts days to find. The Social Security Administration has been piloting tools to help process disability determinations faster, a backlog that has real human cost for people waiting on benefits they’re owed.

These work not because someone deployed AI at the problem, but because someone first defined what “working” actually means in each context. The outcome was specified before the tool was selected (so rare in the public space). That sequence matters. Reverse it, and you get a very impressive demo that solves a problem no one had.

You can’t prompt your way to knowing which outcome actually matters. You have to have been in the room. Talked to the caseworker. Understood what nine months actually costs a veteran waiting on a decision. AI can get you there faster once you know where you’re going. It has no idea what’s worth going there for.

What’s next

We outsourced memory. Calculation. Scale. Now we’re outsourcing whole ideas.

What’s left?

The model wasn’t in the room when that trust was built. It didn’t sit through the failed pilots, the budget kills, the vendor who promised a platform and delivered a PowerPoint.

You did. Knowing who to call, why the last approach failed, and which office will kill this if you don’t bring them in early, that’s more than institutional knowledge. That’s survival knowledge. It lives in people who stayed, who paid attention, who built something over time that outlasted the last reorganization.

Every organization that hollows out that function to save money on a headcount line while expanding an AI license is going to rediscover, the hard way, that the tool doesn’t know what it’s for. It will generate artifacts. It will check boxes. It will produce a demo that impresses.

And then someone will ask what problem it solved. And the room will go quiet.

The people trying to replace judgment with a prompt are going to find out the hard way.

The views expressed here are my own and do not represent any federal agency.

The draft was cleaner than mine, but it was hollow. Every sharp point got softened into a “balanced perspective.” Every real opinion got buried under caveats. It sounded polished, professional, and completely forgettable. Like most AI-written commentary, it used all the right words and said almost nothing.

The federal AI space has the same problem. Real money chasing solutions. No patience for basic questions: What problem are we solving? Who is this for? What changes if this actually works?

That’s how you end up spending serious money to automate confusion.

The Speed Trap

The program managers are usually the quietest people in the room. They see the weak requirements, the bad assumptions, and the missing data. They know what’s actually broken.

But by the time they’re in that room, the decision has already been blessed three levels up. The budget is allocated. The contract vehicle is picked. Staying quiet and buying the tool is the safest career move left.

So that’s what happens. Leadership gets movement. Vendors get paid. Agencies get a modern interface on top of a broken process.

A broken process that now demos better.

The Process Problem

This is where most of the waste lives.

Federal agencies are layering expensive technology onto workflows that should’ve been simplified, redesigned, or deleted years ago. Instead of asking whether a process deserves to exist, they ask how AI can be inserted into it.

One team uses AI to scrape updates from a tracker and generate a 40-slide deck. Leadership uses AI to summarize that deck into three bullets. Someone uses AI to draft a reply: “Thanks, keep going.” At every step, work is being done, time is being saved, and none of it creates value.

It’s the digital equivalent of putting a turbocharger on a shopping cart.

Automating waste doesn’t make it strategic. It just produces useless output faster and with more confidence. When something actually goes wrong, nobody trusts the polished dashboard. They call the person closest to the problem and ask what’s happening.

That should tell us everything we need to know.

The Ownership Trap

When agencies buy AI, they aren’t just buying software. Unlike a legacy system that fails the same way every time, an AI model degrades. It drifts. Inputs change, policies change, user behavior shifts — and a model that worked on day one quietly becomes wrong at scale.

If the contract is weak on data rights, the agency ends up paying for a capability it can use but can’t control. That’s how the duplication loop starts. One office pays to develop a model. Another office has the same need, but the implementation is locked in a vendor’s proprietary stack. It can’t be reused.

The government buys the same answer twice.

If you can’t inspect it, retrain it, or fix it without calling the original vendor, you don’t have a solution. You have a rental with a degradation clock running in the background.

The Talent Gap

Upskilling efforts in the federal AI space are mostly theater. The gap isn’t closed by hiring a few specialists or teaching staff to prompt a chatbot. Prompting isn’t strategy.

The people agencies need are the ones who understand how these systems behave in production — how they fail, how they drift, when to shut them off. People who can maintain a model through a recompete, an ATO renewal, a policy change, and two rounds of staffing turnover without losing the thread of what it was built to do.

That talent walks out the door. A contractor team builds something promising, leadership celebrates, and a few months later the expertise is gone. The PM is left holding a tool they can operate on a good day but can’t troubleshoot when the outputs start looking wrong.

The organization didn’t build capability. It outsourced the brain and called it progress.

The Work Comes First

Cut the process before you automate it. If a workflow improves when you remove two steps, those steps never needed a model — they needed a leader willing to delete them.

Define what you’re actually buying. Not “AI capability.” A result. Faster grant reviews. Fewer manual reconciliations. Shorter time from intake to decision. If the outcome can’t be stated in plain English, the acquisition is too vague to survive.

Secure real ownership. Data rights, retraining rights, documentation. If the agency needs to call the vendor to understand what the system is doing, it doesn’t own the solution. It’s just paying for access.

A prototype is easy. A production system that survives a policy change, an audit, and a change in administration is hard. That’s where the actual work gets done.

We keep buying complexity because clarity is harder to defend in a budget hearing. Saying “we deleted two approval steps and cut processing time in half” doesn’t land the same way as “we deployed an AI-enabled workflow modernization platform.” But one of those actually solved something.

The hard part isn’t buying the AI. The hard part is knowing when not to.

The views expressed here are my own and do not represent any federal agency.

You click yes.

Of course you click yes. Name one person who doesn’t.

This screen only exists so somebody can prove you clicked it. The ATO (Authority to Operate: the formal authorization a federal system needs before going live, issued after a security assessment package is reviewed) package has a control that says users must validate certificate trust. The system records the click and calls that a control. Nobody in that chain ever stopped to ask whether clicking yes changes anything.

That’s not security. That’s paperwork with a button. Worthless.

Click yes enough times and it becomes muscle memory. Which means the one time the warning actually matters, a legitimate attack could happen.

A warning everyone ignores isn’t a safeguard in any sense. It’s camouflage for bean counters. The auditor gets the screenshot. The attacker gets the goods.

Why the Popup Exists

Here’s how it usually goes. Someone stood up TLS inspection (Transport Layer Security: the protocol that encrypts network traffic between a client and a server). Nobody pushed the root certificate cleanly to managed devices. The browser complained. So the workaround became policy: click yes, carry on. Six months later, nobody remembers why the warning appears or whose job it is to fix it. It’s just part of the environment.

The popup isn’t adding any value.

This Is the Standard Playbook

The cert warning isn’t a one-off failure. It’s how federal IT security operates.

The annual security awareness training everyone clicks through in under an hour to clear the compliance flag? Same mechanic. The forced password change cycle that kept government sticky-note suppliers in business for a decade? Same mechanic. The enterprise-wide Zero Trust mandate in the strategy document while users manually bypass session hygiene every morning? Same mechanic.

In every case, a real security requirement got translated into a user interaction that nobody takes seriously. The box gets checked. The behavior doesn’t change.

It’s cheaper to make 80,000 people click through nonsense than to fix one broken enterprise service. So nobody fixes the service. Federal IT has spent two decades building for the audit and breaking for the mission. If a control generates a log, a screenshot, or a completion record, it survives budget cycles. If it requires fixing the underlying infrastructure, it gets deferred.

What the System Rewards

Nobody gets promoted for eliminating the popup.

The Authorizing Official signs a Risk Acceptance, documents the residual risk, and calls it managed. The control owner gets credit for the artifact. The help desk absorbs the tickets. The user absorbs the risk. The fix request dies in the intake queue.

This is what happens when control owners can pass audits without owning user behavior. The program office buys the system. Enterprise IT breaks the trust chain. Security signs off on the control language. Nobody owns what the user sees on the other end.

When the breach happens — and the logs show the warning appeared six hundred times and the user clicked through every single one — nobody is responsible. It was documented. The user was informed.

We Blame the Wrong People

We call it a workforce problem. Users don’t take security seriously. They click through warnings without reading them.

We designed it that way.

We built systems that produce compliance artifacts instead of secure behavior, then blamed the people who adapted to the environment we built. The annual training is forgettable by design — anything longer doesn’t clear the flag on time. The cert warning is ignorable by design — if it blocked access, it would generate more tickets than IT could handle. Every friction point that was too inconvenient got softened into a checkbox and called a control.

The workforce is doing exactly what the system trained them to do.

The Control Was Fake From the Start

If your security control depends on a user interpreting a certificate thumbprint, you don’t have a security control. You have a ritual.

A real control removes the decision from the user. It’s rare enough that when it fires, people notice. It doesn’t need to explain itself with a wall of hex digits no one will ever read.

The warning was fake the day it shipped. All it ever protected was the agency’s ability to say it tried.

The views expressed here are my own and do not represent any federal agency.