My 2023 Tech Summary
A reflection on the technologies explored in the past year and opinions on each.
Scalable Kubernetes in Production
At the start of the year, the platform that our team was working on finally launched to production. It was a platform for a major bank exploring bolstering their footprint by expanding to the cloud to deliver better experiences to the end users while easing the burden on the individual teams to create secure and scalable applications. A similar pattern is currently emerging from major enterprizes as the complexity of IaC is becoming more burdensome to maintain. Modern systems must have DevOps, SecOps, SupportOps, and all the other Ops integrated in a seamless way and the amount of knowledge required is now too great for any single team to know.
Terraform IaC
Heavy use of Terraform allowed us to create all of infrastructure for clusters and various components at will and create zero downtime deployments and canary testing of upgrades on the cluster level, ensuring safety and uptime for our teams. But Terraform is only a piece of overall IaC as there are limits to what one can do with a system for desired state reconciliation. Various strategies can be employed that mostly deal with some sort of a breakdown of the overall state into multiple components that are then managed by a workflow system. We found that with AWS EKS, Terraform performs really well and gave us many customization options that are missing from other tools, like eksctl.
Karpenter
We implemented cluster scaling with Karpenter and were very successful at letting the cluster manage its own high availability scaling requirements. We also were able to isolate workloads to specific nodes for security and stability reasons. The implementation was relatively simple to implement on AWS EKS and I look forward to other clouds implementing the concepts behind Karpenter. One of the best features was the speed at which new Nodes would become available, a MASSIVE improvement over manual scaling or autoscaling or Cluster Autoscaler.
Financial Data Security Requirements
Dealing with sensitive data is always a challenge and there are many security considerations and optimizations that must be implemented for a secure data transit and storage. These are some of the most important steps that a devops team must take initially and enforce for simple audit reporting and compliance to various security standards. Most of these can be solved by creating a platform that limits developers from creating bad scenarios and easing the burden of proof.
Consul Service Mesh
As part of the security footprint we utilized the Consul service mesh to enable transparent encrypted communication between services, clusters and on premise resources. There were many issues that we had to overcome and limitations that we had to impose on teams. The notable one was AWS NLB keepalive timeout that caused leader election in Consul to trigger and drop all existing connections from applications that needed them. Another was the communication protocols that were supported for endpoint communication. Because Consul operates at Layer 7, it really can’t deal with any communication protocol that is lower in the OSI model. This unfortunately meant that any outgoing HTTPS or TCP traffic can not be inspected for destination routing. And as our platform denied any incoming and outgoing connections that were not registered with platform, our troubles began at Kafka and continued with hardcoded HTTPS connections that could not be downgraded to be TLS over HTTP. Overall Consul solves some problems that an enterprise organization has with secure communication but introduction of tools that use eBPF to control communication is much more flexible and better for a platform.
Vault
Vault remains one of the most secure and flexible solutions for enterprize secrets management. Integration with Kubernetes is excellent and simple. The flexibility of the templating to inject the secrets directly into the containers and being able to write to disk at container init greatly increases any security posture of an application. One of the developer experience improvements that we implemented was to copy a secret file to any location on disk from initial secret injection, this was a limitation due to how Kubernetes manages file mounting onto the container.
Datadog
Datadog is a comprehensive monitoring platform for most projects. The amount of integrations and the flexibility of ingestion transforms and customization is simply amazing. Combined with alerting and incident management and security monitoring, makes this my primary recommendation for monitoring.
Pulumi
It’s ok. That's about the extent of what I can say about it. After dealing with advanced scenarios in Terraform, the ability to code in any language is really not that much of a selling point. Ultimately this is still a tool for desired state reconciliation, any language on top is basically a DSL in your preferred programming language and has the same limitations as Terraform. The company is trying to expand into other areas as DevSecOps but I do not have any desire to continue exploration of their product set.
Digital Ocean
Great alternative to other cloud providers. One of the limitations that I ran into was with Cilium. At the time DO used Cilium as the networking layer for Kubernetes and offered no customizations to upgrade or add additional features for the Cilium installation. The platform is still evolving and I hope to see more features geared toward enterprise requirements implemented in the future.
Cilium
eBPF is one of the most exciting things to happen to Linux kernel in a long time. The possible tooling that these concepts allow are already emerging and Cilium being one of the most exciting ones in the container space. I am a big fan of Cilium and I hope that local development for Kubernetes can catch up. Rancher Desktop is the local distribution that I use for Docker and Kubernetes but still has issues with MacOS and virtualization underneath, combined with experimental eBPF and kubeproxy replacement support, it's not the smoothest experience, but it's improving!
Github Projects, Actions and Workflow
I really enjoyed using the new Github Projects. I think that issue management and project management has greatly improved. The improved workflow engine is also very good. The separation of Actions from the container implementation is an interesting solution for a stepped workflow. I really like the flexibility of using the same Actions across multiple container definitions. While you can do the same thing on Jenkins Pipelines or GitLab CI/CD, I find Github Workflows easier to read and manage. I have only used it in personal projects and not sure how it would scale to an enterprise organization.
Backstage
Despite popular beliefs, Backstage is an opinionated platform. I think it is vital to any implementation of Backstage to read the documentation of the mental model behind the Software Catalog. Another important factor is that Backstage is still going through internal rewrites and upheavals of current APIs. A lot of the work that has been done so far will need to be rewritten to the New way of doing things. And finally you must understand that Backstage promotes Openness and Exposure which is a direct attack on most enterprize silos created by the current outdated managerial thinking. This is the biggest hurdle that any enterprise must overcome before even thinking about creating something like a Backstage based developer portal. The systems that are in place today were put there for good reasons and serve some purpose even in the most antagonistic environments but they must be rethought and rearranged and rules must be rewritten which sometimes is an exercise in futility if the upper management sees no problems in the present way of doing business and more importantly is not willing to put their neck on the line to change the status quo.
Frontend and Backend Application Development
The state of application development is abysmal. Everything from terrible decisions by committee to snake oil salesmen to drink my kool aid companies. There are so many choices and evaluating them for viability is a full time job. There is a terrible fixation on Server Side Rendering that is plaguing the current implementations as more and more frameworks are trying to solve the problem of SSR. But the interesting thing is that most people do not need to have SSR, and guess what, the industry has been here before, anyone remember PHP, ASP, ASPX, JSP? And all this to promote their own little silos and sales by reinventing the wheel and proclaiming their square design that doesn't work is better for everyone. And on top of this, is the fetishization of early founders and taking their opinions as gospel that everyone must follow. All these issues will inevitably lead to even worse application design with huge costs of maintenance in the future and even ChatGPT won’t be able to help.
Next.JS and React
Unless you want to host on Vercel and are willing to pay the framework developers to explain the stupid things you did, then I would re-evaluate if you need Next.JS. There are cases where its great, like Nextra is pretty awesome for documentation and generating a static site is a breeze. But full application development that needs Authentication and Authorization, you are better off with sticking with CRA and custom building everything yourself.
Angular
Angular has not kept up with the tooling. As our development tools improve, it becomes easier and safer to create large javascript applications. A lot of the burden that Angular puts on the developer with how it handle dependency injection and module importing is really unnecessary. State management and reactive rendering is also a big pain and it doesn't look like any of these issues are being addressed. It is a good enterprize framework to enforce certain practices but it comes at a cost of large and repeated code bases and slow speed of development.
Nuxt.JS and Vue
I think this is a developer passion project that got out of hand. While I think it is a good pursuit to create alternatives, the Vue community is so focused on selling you how to do Vue properly and how to make Nuxt.JS work on Netlify or Vercel that it leaves a very bad taste in your mouth. It suffers from the same problems as Next.JS and it’s just as awkward to build applications on top of. Maybe focus less on selling me courses and getting me to convert and more time on improving documentation and building a large set of examples for advanced use cases.
Remix
I liked Remix for the most part and could almost get over its obsession with SSR. I think at this point I was annoyed at full stack frameworks and it wasn't a fit for my application design. I am sure I could have made it work but then I would not be following yet another opinionated design and would have suffered by fighting something that was not a fit. Overall, if I was looking for a very specific way to build an application and needed a full stack answer, Remix is not the worst.
Blazor
One of my goals has been to check out the latest of dotnet features in dotnet 8. And in the middle of evaluating an application framework to use for a new simple application that I could easily code to be ready for cross platform and quickly deliver a mobile iteration, I came across the latest from the Blazor team, a .NET MAUI Blazor Hybrid application framework. After exploring Blazor for a bit and discovering the seamless compilation to WebAssembly and web socket delivery of data, I decided to create a small application to test out the features. One of the best parts was using a single programming language for most of the coding. And yet being able to call javascript and get data back which then can be serialized into c# and dealing the with objects once again. Getting back into the dotnet ecosystem was simple and a lot of the base technology has been improved quite a bit. Starting with something like Bit Platform speeds up the initial development efforts with sensible defaults, and I have customized it to my preferences on project layout, naming, and code styles.
Plaid
As part of my simple application, I wanted to connect to my various accounts and create a way to ingest all the data. One of the available tools is Plaid and connecting to the financial data is only one of the capabilities. So far I found their APIs straightforward, the documentation is mostly complete, only lacking in the full documentation of possible models returned for each call. The number of tutorials are also excellent, documenting everything from the OAuth flow to creating mobile apps.
Elsa Workflows
As part of the processing of various accounts connected through Plaid, I needed an async processing model to have flexibility in receiving new data through web hooks, manually refreshing the data, initial fetch of data and caching the data for faster retrieval and doing some data transformations. Elsa is a promising project that will allow the application to be fully self contained but also having an option of breaking out the workflow engine into its own application that can scale according to needs without having to rewrite the application.
Machine Learning
This guide by Andrej Karpathy inspired me to really dive deep into the language models and their training. I have done research in the past and understand the basic concepts behind the training and transforming data but this engaging lesson really took this surface level interest to a deeper desire for understanding. And with ML.NET, I can utilize my existing expertise in .NET and hopefully embed some simple models into some of the apps. For training the models it will be interesting to figure out a smooth way to be able to use cloud resources with machines specific for training models.
In conclusion, 2023 has been a busy year, and it doesn’t look like 2024 will be any easier. I look forward to Gateway API becoming a first class citizen and easing the ingress story for Kubernetes. Diving deeper into machine learning and understanding the technology beneath the readily available trained models. Maybe even finishing up one of the applications and releasing it.
