Complexity Killed the Code

While everyone is rushing to generate as much code as possible, no one is stopping to ask what code should exist

Recently I switched jobs. I went from a very large company to a smaller company. I went from everything self-hosted and on-premises (source control, CI/CD, ticket tracking, service deployment, feature flag management, internal package mirror, etc.) to everything being hosted on a cloud SaaS provider. It is very painful. Not because of anything that my new company has done wrong (they’re fabulous), but because the service providers are doing a lot of things wrong.

So, I checked the service providers out. I saw what they’ve been up to. Every single one that has had reliability issues, without fail, is generating as much code as they can with AI. Every single one.

The stable SaaS companies? Very little to no mention of using AI. Something is rotting in the software world. I’ll leave it for you to find out, because that’s not what I’m interested in writing about.

What I am interested in, is the secondary trend that I found, and am experiencing, and that I’m seeing in tech. The reliable systems were a simple architecturally. Their blogs talked about monolithic architectures. Or they were in the very beginnings of micro services, so their service was still mostly monolithic. Many had self-hosting options. Monorepos were common. The SaaS companies that had the most issues were also overly complicated from a software technology perspective. Tons of talks of microservices, aggressive scaling, eventing, micro frontends, micro repositories, chained deployment pipelines, etc¹. Anyone who architected these software “solutions” would be unable to deploy on anything less than an entire data center, much less a handful of servers (or even one server). Doesn’t matter how big the servers are. These people have designed a system with so many pods, so many clusters, so many dependencies that they exceed the operational capacity of an entire rack in just container overhead.

The scary thing was that there was no sign of the complexity getting better, only of it getting worse. And the reliability is going with it.

And yet, I can’t help but feel like just saying “make things simple” won’t help. People justify complexity a lot. Sometimes too much. And sure, the naive solution won’t always scale. But it’s hard to understand simplicity when our textbook examples of how to do basic things are overly complex.

The Textbook CRUD Application

I’m going to start with a textbook CRUD application. Most SaaS code grows from these. I’m going to assume we’re using middleware for session management, authentication, and route authorization. That means when I show an endpoint, I’m not going to show the code loading the session from a datastore and verifying it (or getting it from a JWT, or whatever). I’m just going to show the CRUD endpoint. We’ll also be able to assume that whatever the CRUD endpoint does, the user has access to since if they don’t have access then they were filtered out before it reaches the code. Same thing goes for rate limiting, CSRF mitigations, CORS, database migrations, etc. Basically, anything not in the database communication flow is assumed to be taken care of elsewhere.

This series of assumptions is to only make the example more focused. We want to focus on a small part of the application - the textbook example - rather than the entire application. We can show how the rest of the application fares when applying the same techniques later on. For now, let’s just focus.

The textbook example we’re going to focus on is simply CRUD endpoints for a todo app. Yes, a todo app. The simplest, most plain, brain-dead, overdone code that anybody could copy-paste from millions of first-year college students and get something reasonable. We’re going to start there, since apparently simplicity in that type of application is hard to find.

Also, we’re only going to do an API for todo items. To help solidify things, here’s the schema we’re working with in a Postgres database.

CREATE TABLE todo_items (
  id BIGSERIAL PRIMARY KEY NOT NULL,
  user_id BIGINT NOT NULL REFERENCES users(id),
  item_name TEXT NOT NULL,
  item_description TEXT NULL,
  done BOOLEAN NOT NULL DEFAULT false,
  deleted BOOLEAN NOT NULL DEFAULT false
);

-- session management and user data not shown to help us focus on todo items

The Textbook Method

So, let’s define the textbook method. The “best practice” of ways to make our application. I’m not going to mirror an “ideal” MVC or MVVM or WYAI² method. I’m going to do things the Enterprise, Gold-Standard, In-Production Patent-Pending methodology I’ve seen in so many codebases with parallels advertised throughout my entire career (MVC and MVVM being some of those parallels - and yes, there really isn’t much difference that’s actually significant in anyway). I’m talking about classes.

In this model, we usually have a class representing the database table. We also have a class representing the API schema we’re exporting to. If we need to do any logic/processing/filtering on the server side, we’ll have a “service model” class to represent the todo item internally - that way we can process “ephemeral” todo items that may be from a database record, or an event queue, or an API request³.

So, let’s create our models. I’m going to do them in Java because it’s where this type of code is widely accepted, even though it sucks. Feel free to use AI to translate it to TypeScript or C# or PHP or Go or whatever. Though I doubt you’ll need to because most of the ceremony is utter nonsense anyways.

package database.models;

@Entity
@Table(name = "todo_items")
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
@EqualsAndHashCode(onlyExplicitelyIncluded = true)
class TodoItem {
  @Id
  @GeneratedValue
  @EqualsAndHashCode.Include
  private long id;

  @Column(name = "user_id", referencedColumnName = "id")
  private long userId;

  @Column(name = "item_name")
  private String name;

  @Column(name = "item_description")
  private String description;

  @Column(name = "done")
  private boolean done;

  @Column(name = "deleted")
  private boolean deleted;
}

package api.models;

@Data
@Builder
class TodoItem {
  private String id;

  private String name;

  private String desc;

  private boolean done;

  private boolean del;
}

package service.models;

@Data
@Builder
class TodoItem {
  private long id;
  private long userId;
  private String description;
  private Boolean done;
  private Boolean deleted;
}

package utilities;

class TodoItemMapper {
  @Autowired
  private SessionManager sessionManager;

  public dbToService(db.models.TodoItem input) service.models.TodoItem {
    return service.models.TodoItem.builder()
      .id(input.getId())
      .userId(input.getUserId())
      .description(input.getDescription())
      .done(input.isDone())
      .deleted(input.isDeleted())
      .build();
  }

  public serviceToDb(service.models.TodoItem input) db.models.TodoItem {
    return db.models.TodoItem.builder()
      .id(input.getId())
      .user(userRepository.getById(input.getUserId()))
      .description(input.getDescription())
      .done(input.isDone())
      .deleted(input.isDeleted())
      .build();
  }

  public apiToService(api.models.TodoItem input) service.models.TodoItem {
    var builder = service.models.TodoItem.builder()
      .userId(sessionManager.getCurrentUser().getId())
      .description(input.getDesc())
      .done(input.isDone())
      .deleted(input.isDel());

    try {
      builder = builder.id(Long.parseLong(input.getId()));
    } catch (Exception e) {
      builder = builder.id(0);
    }

    return builder.build();
  }

  public serviceToApi(service.models.TodoItem input) api.models.TodoItem {
    return api.models.TodoItem.builder()
      .id(input.getId())
      .desc(input.getDescription())
      .done(input.isDone())
      .del(input.isDeleted())
      .build();
  }
}

package repositories;

@Repository
interface TodoItemRepository extends CrudRepository<TodoItem, Long> {
  TodoItem findByIdAndUserId(long id, long userId);

  List<TodoItem> findByUserId(long userId);
}

package api.routes;

import api.models.TodoItem;
// other imports excluded because they're self-obvious

@RestController
public class TodoItemController {
  @Autowired
  private TodoItemRepository todoItemRepository;

  @Autowired
  private SessionManager sessionManager;

  @Autowired
  private TodoItemMapper mapper;

  @GetMapping("/")
  public List<TodoItem> index() {
    return todoItemRepository.findByUserId(sessionManager.GetCurrentUser().GetId())
        .stream()
        .map(x -> mapper.dbToService(x))
        .filter(x -> !x.isDeleted())
        .map(x -> mapper.serviceToApi(x))
        .collect(Collectors.toList());
  }

  @GetMapping("/{id}")
  public TodoItem getById(@PathVariable Long id) {
    var item = mapper.dbToService(
      todoItemRepository.findByIdAndUserId(sessionManager.GetCurrentUser().GetId())
    )
    if (item.IsDeleted()) {
      throw new ResourceNotFoundException();
    }
    return item;
  }

  // so many more methods, I don't feel like continuing this pain
}

This is the “textbook,” “maintainable,” “easy” code. It’s so verbose. It’s so much code. There’s so much abstraction. There’s so much going on. And it’s slow. And it’s not maintainable. And it’s not easy to use. And it’s really just nonsense.

Almost all of that code is useless. And I’ll prove it to you.

Simple Code

Let’s replace all that code, with the following.

package api.routes;

import api.models.TodoItem;
// other imports excluded because they're self-obvious

@RestController
public class TodoItemController {
  @Autowired
  private JdbcTemplate jdbcTemplate;

  @Autowired
  private SessionManager sessionManager;

  @GetMapping("/")
  public List<Map<String, Object>> index() {
    String sql = """
SELECT
  id AS "id"
  item_name AS "name",
  item_description AS "desc",
  done AS "done",
  deleted AS "del"
FROM todo_items
WHERE deleted = false AND user_id = ?
""";
   List<Map<String, Object>> data = jdbcTemplate.queryForList(sql, sessionManager.GetCurrentUser().GetId());
   return data;
  }

  @GetMapping("/{id}")
  public TodoItem getById(@PathVariable Long id) {
    String sql = """
SELECT
  id AS "id"
  item_name AS "name",
  item_description AS "desc",
  done AS "done",
  deleted AS "del"
FROM todo_items
WHERE deleted = false AND user_id = ? AND id = ?
LIMIT 1
""";
   Map<String, Object> data = jdbcTemplate.queryForMap(sql, sessionManager.GetCurrentUser().GetId());
   if (data == null || data.isEmpty()) {
      throw new ResourceNotFoundException();
   }
   return data;
  }
} 

So much simpler. So much less code. And so much more composable. For instance, let’s say we want to make sure the fields we select are standardized across our methods. We can easily do that with string concatenation.

@RestController
public class TodoItemController {
  @Autowired
  private JdbcTemplate jdbcTemplate;

  @Autowired
  private SessionManager sessionManager;

  private static fields = """
  todo_items.id AS "id"
  todo_items.item_name AS "name",
  todo_items.item_description AS "desc",
  todo_items.done AS "done",
  todo_items.deleted AS "del"
"""

  @GetMapping("/")
  public List<Map<String, Object>> index() {
    String sql = "SELECT" + fields + "
FROM todo_items
WHERE deleted = false AND user_id = ?
""";
   List<Map<String, Object>> data = jdbcTemplate.queryForList(sql, sessionManager.GetCurrentUser().GetId());
   return data;
  }

  @GetMapping("/{id}")
  public TodoItem getById(@PathVariable Long id) {
    String sql = "SELECT" + fields + "
FROM todo_items
WHERE deleted = false AND user_id = ? AND id = ?
LIMIT 1
""";
   Map<String, Object> data = jdbcTemplate.queryForMap(sql, sessionManager.GetCurrentUser().GetId());
   if (data == null || data.isEmpty()) {
      throw new ResourceNotFoundException();
   }
   return data;
  }
}

And just like that, we have one place to update our field selection for our class. Similar things could be done with other aspects, however it’s probably not necessary.

Now you might be thinking “well, that’s great if you are doing basic queries - but I need joins and nested data structures - no way that works!” Well, it does. You just haven’t learned SQL. Here’s one of my favorite SQL queries I’ve ever done, all in Postgres, and it handles formatting nested data, lists, etcetera. This was for quizes, but I love it.

SELECT q.id as id,
        json_build_object('id', q.id, 'text', q.text) as question,
        json_build_object('id', m.id, 'type', m.type, 'url', m.url) as media,
        json_agg(json_build_object('id', a.id, 'text', a.text) ORDER BY a.pos ASC) as answers,
        ua."answerId" as "previousAnswer",
        ua."isCorrect" as "isCorrect",
        json_build_object('correctCount', uao."correctCount", 'totalAttempts', uao."totalAttempts") as summary
 FROM "quizSection" q
          INNER JOIN questions q ON q."questionId" = q.id
          LEFT JOIN media m ON q."mediaId" = m.id
          INNER JOIN answers a ON a."quizId" = q.id
          LEFT JOIN (
     SELECT ua."quizId", ua."userId", ua."answerId", a."isCorrect"
     FROM user_answers ua
              INNER JOIN answers a ON ua."answerId" = a.id
     WHERE ua."userId" = ?
 ) ua ON ua."quizId" = q.id
  INNER JOIN user_answer_overview uao ON q.id = uao."quizId" AND ua."userId" = uao."userId"
 group by q.id, q.id, m.id, ua."answerId", ua."isCorrect

This handles grouping quiz sections with the question data, potential media information, answers (sorted in a specific display order), what the user’s previous answer was, if the user got the answer correct, and the user’s total attempts (and number of correct attempts). It formats all of this as nested objects that I could simply JSON serialize and return to the user. It’s all in the format my API desired.

One query to aggregate a lot of data from different tables, and then simply serialize and return it. No mess of database models, service models, and API models. A single input parameter for the current user id. My server code was literally just the following:

async (req: Request, res: Response) => {
   res.send(await db.query(`SELECT q.id as id,
        json_build_object('id', q.id, 'text', q.text) as question,
        json_build_object('id', m.id, 'type', m.type, 'url', m.url) as media,
        json_agg(json_build_object('id', a.id, 'text', a.text) ORDER BY a.pos ASC) as answers,
        ua."answerId" as "previousAnswer",
        ua."isCorrect" as "isCorrect",
        json_build_object('correctCount', uao."correctCount", 'totalAttempts', uao."totalAttempts") as summary
 FROM "quizSection" q
          INNER JOIN questions q ON q."questionId" = q.id
          LEFT JOIN media m ON q."mediaId" = m.id
          INNER JOIN answers a ON a."quizId" = q.id
          LEFT JOIN (
     SELECT ua."quizId", ua."userId", ua."answerId", a."isCorrect"
     FROM user_answers ua
              INNER JOIN answers a ON ua."answerId" = a.id
     WHERE ua."userId" = ?
 ) ua ON ua."quizId" = q.id
  INNER JOIN user_answer_overview uao ON q.id = uao."quizId" AND ua."userId" = uao."userId"
 group by q.id, q.id, m.id, ua."answerId", ua."isCorrect`, req.session.user))
}

Could things be simpler? Absolutely! The database model was definitely a mess. Having a view built-in to the database, less normalization (like why is quiz section and question data separate when there is a one-to-one mapping? why is media and question data separate when there is at most a one-to-one mapping?), and more would help. Ideally the query from the server would simply become the following:

async (req: Request, res: Response) => {
   res.send(await db.query(`SELECT * FROM quiz_view WHERE user_id = ?`, req.session.user))
}

Then, all the complexity above would be handled in the database layer with the migrations. I could then do things like use materialized views to cache read-heavy workloads, add additional indexes based on the query plan, and even modify the table structure without breaking application code (the view just has to map). There wouldn’t be this nonsense of having a data object match the table structure just so we then map it to a service model so we can operate on it in-memory and then map it to an API model so we can remove the data that we don’t show the user.

Which also brings up another point of mine.

To the nay-sayers

Some people are going to argue that the SQL code above is not really all that simple, or that it’s hard to read, or debug, or whatever. Except it’s not. Compare the SQL with the “much simpler” Repository code:

@Repository
interface QuizeRepository extends CrudRepository<Quiz, Long> {
  List<Quiz> findQuizByUserId(long userId);
}

Now, tell me what those above functions do exactly. Tell me what the SQL plan is. Tell me if there’s a scan or an index lookup. Tell me what joins happen, and when. Tell me how many network trips happen. Tell me when those network trips happen. Is it when I access a field, or sub-field, or a list element? Is it all at once even or is it lazy? Tell me when the framework switches between lazy and eager. Tell me how to debug it when it’s getting the wrong fields, or not enough fields, or making too many requests. Tell me what happens if I lose my database connection. Tell me what happens if I lose my database connection after I accessed some fields, but not others. Tell me how it runs on the database. Show me the execution plan. Tell me how it operates under load. Tell me how to make it faster. If you can’t answer all of those from that snippet and a connection to your production database, then your solution is objectively worse than mine.

With my snippet you can get all of that information and more. You can even get the current worse case scenario and best case scenario. You can see every index used and not used. You can control how much is loaded and when. You can add in materialized views with ease.

At the end of the day, for both code snippets to be functionally equivalent, the SQL in the second example must be at least as complex as what I showed earlier. You just can’t see it. Instead, you see mindless classes, that offer no real value outside of extra memory allocations and memory copies for your garbage collector to clean up. You see annotations that are “magic” without thinking about the runtime cost (hint: most of those annotations use reflection - one of the slowest operations in any runtime/interpreter). You don’t see SQL, you see class method accesses. You don't even see your class data members, just getters and setters. Meanwhile that SQL - the thing your database runs and is the slowest part of your request - is generated on the fly, at runtime, and ran against your database without any sort of review at all. No DBA looked at it, no team member gave it a glance, and you did not even check if it’s using an index or not. You just shipped it.

At the end of the day, your users suffer because your query that you didn’t write - only generated - was slow. Your server suffers because you filled it with garbage memory and wasted millions of cycles per request copying data from class to class to class to class. Your garbage collector suffers from frequent, short-lived memory allocations blocking it from cleaning up the real stuff taking up memory (that’s how generational garbage collectors work - short-lived allocations block it from cleaning up long-lived allocations). Your database suffers because you’re hitting it with thousands of unvetted, unoptomized queries that you don’t even know what they do or where they came from or why sometimes they get everything at once and why sometimes they make hundreds of micro-queries per field.

All of this in the name of “abstraction.” Your abstraction brought in complexity, it killed your code, and AI cannot save you since it’s been trained to increase the poison - not lower it.

In the best case, you have DBAs pinging you at 2:00 in the morning asking what code is running such-and-such query that’s killing the database and you don’t know because you never wrote the query - and you have to find out. In the worst case, that urgent message is never sent because nobody cares, and your application joins the rest of the rot. Reliability goes from 99.99% to 99.9% to 99%, 98%, 95%, 92%, 90%, 85%, 70%, and down and down and down. Users start questioning your company. They look for alternatives. Some are forced off even though they once loved your product.

And so, the product dies. Slowly. Very slowly. Until something both breaks and a competitor gains advantage. Then, irrelevancy. Just like MySpace, and Yahoo Search. Maybe a rebrand happens, like Hotmail to LiveMail. Or maybe an aquisition like Sun Microsystems. Or just bankruptcy like Palm.

Of course, that may not matter much for you. There’s always another dying software that needs more poison, more bad abstractions, and more cycles wasted in the name of productivity and best practice. If, after all of this, you’re still fine with the complexity of the common abstractions, then go ahead. Use your AI tools as long as they’re affordable. Go nuts. I can’t convince you, so this post isn’t for you. If, on the other hand, you’re starting to question the textbook, then keep questioning it. Question the “best practice” because, at the end of the day, it’s almost all a cargo cult. Built to hide, not know.

Back to Microservices, AI, and SaaS

We just looked at a very basic “best practice” that kills code. Now, stack that on top of itself. We have controllers, views, models, and database models. Add in eventing with their own models, controllers, and tie those into your existing models and databases. Add in services with API requests. Now wrap those requests in a cache layer, wrap those cache layers in models, and integrate those models with your existing models and services.

Now do that again. And again. And again. And you have a little series of services that does one little part of a feature. Now keep doing it several dozen more times. Now you have most of a feature. Do it several dozen more times, and now you have a button that can send an email some time in the future for an authenticated user. Keep going. Now you can send emails immediately. Keep going. Do it a few thousand times. Maybe a few million times. Eventually, doing it enough times, you get GMail. Or maybe GitHub. Or NPM. Or any other SaaS⁴.

And yet, they didn’t need the microservices. They didn’t need the ORM, or the many model layers. Many of them don’t even need the eventing (or at least, not “proper” eventing - a database table with a status column and a single background job often suffices).

Even at scale, many companies don’t need microservices. It actually makes scaling harder. Now to scale up one service, you need to scale up everything that it touches - whether that’s event pipelines or direct API calls anybody that it puts load on needs to scale. But, to scale those services, you need to scale everything they touch. And so on.

And, if you ever want to do a sweeping change - like update from Java 8 to Java 11 - you now need to do that work thousands to millions of times. Every service needs to be updated, tested, fixed, and tested again.

Now do that for your frontend too. Do that with Angular. Or React. Try to keep your Node versions updated. Upgrade your build pipelines so you can upgrade your Node. Do that thousands of times.

And keep the network stable. Make sure you don’t overload the network. Make sure that you don’t have too much latency. Make sure you don’t route too much traffic from thousands of services to a single server.

And now debug things when they go wrong. Try to figure out what actually happened when a user clicked a button. Which services were called, when were they called, how were they called (API? gRPC? event?). Which ones failed. Which ones tried to fix the failure and made it worse. Which API gateway decided to drop the message before it hit the service - and why.

Now handle incidents. Try to get the right teams on an incident call. I’m sure they all kept your wiki updated - especially after that reorg last month. Try to keep track of which services are unreliable. Track the cascading failure. Which ones take down your site? What unexpected parts went down?

Now do this across time zones. Have some developers in Central India and some in on the US East Coast. They’ll never meet, or talk. Except maybe in an incident. How do they respond, under pressure, with complete strangers, in the middle of the night, after a long day of work, while the other people are about to go to lunch, and when the optics are bad?

Now add in some acquisitions. Little bubbles with completely different technologies, languages, patterns, design, and philosophy. Now apply pressure for them to conform or leave.

And finally, add in AI. Something designed to take all of the data of every instance of any of the above situations and amplify it. Automate it. But, do it in a way that is scientifically shown to be the most enticing and pleasing.

What do you expect the outcome to be? I expect it to mirror what we’re already seeing. Increased outages, especially from high profile companies. Increased severity of outages when they happen. Decreased public satisfaction with software. Increased concern about relying on SaaS. In short, I expect our complexity to kill our code.

1. It’s also interesting how the companies with the most complex code are the ones relying on AI the most, while the companies with less complex code don’t rely on AI as much. It’s almost like building a system that no person could ever hope to understand breeds a helplessness that heavy AI usage can prey on, while building a system that’s comprehensible builds a sort of resistance to AI.

2. Whatever-Your-Acronym-Is

3. And yes, this is very much real. I have worked on many production code bases with this as the simplest case. Usually it was a lot worse.

4. Almost all of them send emails now or in the future.