I actually build minimal Redis clones in every new language or runtime, or when I want to explore threading models.
It all started with https://github.com/rcarmo/miniredis (which I forked to add and experiment with pub/sub), and I just found myself doing it again and again because Redis is the quintessential network service:
By implementing it, you learn about socket handling, event loop for a specific runtime, threading models, data representation, concurrency (if you want to do a multi-threaded version), etc. None of my "ports" are fully functional, but they all helped me sort out some of the above plus build tools, packaging, dependencies, etc.
It's "hello world" for core cloud native microservices, if you will (and without having to do REST or JSON stuff).
Scrum is a way to take a below average or poor developer and turn them into an average developer.
It's also great at taking great developers and turning them into average developers.
Everyone just wants to take something easy off the board that you can get done in a day so you have something to report in tomorrows daily scrum. It's just everyone trying to pick the low hanging fruit. THere's no incentive to be smart and to take time to think about solutions, if nothing is moving across what are you even doing? You're letting the team down! The velocity is falling!
I think if you have hard problems to solve you solve them by giving them to smart people then leaving them the fuck alone. You don't constantly harras them everyday demanding to know what they did yesterday and what they plan to do today. With daily updates where is the incentive for the smart people to work on the hard problems? They now have the same incentive as the junior developer, find the easiest tickets to move across the board.
Sometimes I will want to just be alone and think about a solution for a few days. If I do that though I'd have nothing to say at the scrum. So instead I'll pick the user story where the colour on a front end was the wrong shade of green or a spelling mistake! See, I knocked out 2 stories in one day, before lunch! Go me!
And it all looks like progress because stupid cards are moving across a pointless board.
Scrum exists because managers don't trust developers to get things done without constant supervision and an ever present threat of looking bad for your team.
It takes a team of potentially really smart people and turns them into a code production line. It's depressing.
Damn, I use Prisma, it's a good way to have a unified database schema for which you can generate code in any language you want. It was very useful for converting a TypeScript project to a Rust one, I use prisma-client-rust in particular.
I defaulted to Prisma for edge apps I'm experimenting with (supabase db). Given it's not possible to connect via TCP to a database and I'm stuck with JS/TS for the edge, I opted for Prisma's data proxy and henceforth Prisma.
Using the ORM means I'm tied to this mess of an abstraction, migrations and scripts that require so many workarounds and experimentation to make it worthwhile when your app is not a simple CRUD or you want performance optimizations without ending up writing your own queries yet again. The generated TS API is the only pleasant feature, but doing things the Prisma way I noticed my app is now tied to this ecosystem and its quirks and I dread it. Being unable to check generated queries' SQL beforehand is also a huge issue that is not getting solved any time soon from what I've gathered on their github.
You could also use Prisma as a SQL client and leverage their data proxy, without needing to use the ORM and execute SQL migrations and queries directly, which is what I intend to do from now on. Please do let me know if there are any other generic alternatives database-wise (for "the edge"), I miss raw [Postgre]SQL. Supabase's client is the closest I can think of and it also generates types from your db, but it's obviously not a generic solution.
Lastly, one particular thing that disgusts me (I believe it's not on Prisma but due to how the edge works) is that you cannot use a local database for development. Perhaps ngrok will work, but then you'd need to constantly update your data proxy project settings manually - I need to experiment more on that.
I always find monorepo/polyrepo discussions tiresome, mostly because people take the limitations of git and existing OSS tooling as a given and project those failings onto whichever paradigm they are arguing against.
I'm pretty excited for new OSS source control tools that would hopefully help us move past this discussion. Particularly, Meta's Sapling[0] seems like a pretty exciting step forward, though they've only released a client so far. (MS released its VFS for Git awhile back, but unfortunately now is deprecated.)
Maybe diagrams made in, say, Visio, draw.io, or Inkscape are fine when you're the sole author of them, and you create typographic quality documentation, a business presentation, or otherwise need eyecandy.
In my practice across a few companies, diagrams made in code-oriented tools like Graphwiz, PlantUML, or Mermaid were vastly more useful for documenting real, evolving engineering projects.
* You can store them along with other documentation texts. Their source is always obvious.
* Version control! You can diff them and make sensible pull requests when updating some small aspect of a larger diagram, along with text, and possibly as a part of a code change. You can review, discuss, and approve the changes using tools familiar to every engineer.
* The amount of visual tweaking the language allows (graphwiz, quite some; mermaid, minimal) prevents bouts of bikeshedding and keeps the diagrams focused on their essential parts.
* Since the language is open, and no special tools are needed (except a text editor and a renderer on a web page), the diagrams are now everyone's concern and purview, as opposed to the person who happens to be good at graphics.
* The formats are not locked up; you don't need an authoring license, a subscription, a particular OS to run the editor, etc. You own your diagrams.
* Sometimes you can generate these diagrams from other data you have, pretty easily, and with consistent results.
These considerations appear to overweigh whatever visual imperfection issues these tools have.
That's how I thought until I started going deep into some technologies that have very strong technical foundations:
- Databases: Postgres, ClickHouse
- Back-end frameworks: Ruby on Rails
- Front-end frameworks: Elm
I lost count of the times I was reading the documentation of these projects and had my mind blow thinking "The person that came up with this concept was a genius! I'm glad I get to learn it."
World Wars 1 & 2 happened, shattering the remaining sense of aesthetic unity we had
that was followed by modernism which purity spiraled into the ubiquitous glass box, which was then followed by a post-modernist surface-level, purely negative rejection of modernism
the post-modernists were unable to return to any pre-modern sense of aesthetics due to their ideological commitment to the ur modernist impulse of pride (called innovation)
it is interesting to note that even individual aesthetic disasters like victorian homes, when combined in a neighborhood with common roof-lines, etc. form a charming aesthetic that has been singularly unachievable except in ersatz form since... deep questions of cultural unity lurk here...
I recently started measuring CO2, Radon, PM (particulate matter) 1.0, PM 2.5, VOC, humidity, temperature, and air pressure with the Airthings Wave connected device.
I have learned a lot while using it for a couple weeks. First, making a fire in your fireplace is great for ambiance but drags air quality down substantially. It eats up oxygen a lot and make the CO2 increase sharply. It also causes very high spikes in particulate matter (both 1.0 and 2.5 micron) from putting the burned byproducts into the air.
I also started improving the energy efficiency of our heating system by fixing spots in the house where cold air comes in. While this results in less energy used to heat the home, it causes CO2 to increase because there isn’t anymore large holes to bring fresh air in. This device helped me learn that CO2 is and energy efficiency are circular problems. The tighter my house is, the more I need to focus on ventilation - exhaust out and fresh air in. It sounds complicated but for me it just means opening windows throughout the house for about 15-30 minutes per day. That alone makes a major difference on everything - Radon, PM 2.5, PM 1.0, CO2, etc.
Lastly just want to mention that it’s amazing to me how fast CO2 levels can rise with just my husband and I in our living room watching a movie. Good ventilation is something I definitely recommend everyone start measuring and working on.
If you want a challenge, try "The Go Programming Language" by Alan A.A. Donovan and Brian W. Kernighan.
It's somewhat akin to what "The C Programming Language" (K&R) is to C. It's a fairly practical and on-point, and has a lot of (somewhat tricky) exercises, while still providing a complete overview of the language.
In the past I worked at a company that managed thousands of individual MSSQL databases for individual customers due to data security concerns. Effectively what happened is the schema became locked in place since running migrations across so many databases became hard to manage.
I currently work at a company where customers have similar concerns around data privacy, but we've been to continue using a single multitenant DB instance by using PostgreSQL's row level security capabilities where rows in a table are only accessible by a given client's database user:
For context, SQLite4 explored reimplementing SQLite using a key-value store on log-structured merge trees, like RocksDB and Cassandra.
I'd be interested to hear why they stopped. Presumably reimplementing SQL on a KV store was seen as not worth it, when applications that are satisfied with an embedded KV store backend (which is much faster and simpler to write!) already have many options.
Prometheus does support push. It's just that it's considered such an antipattern that it's been moved into a separate module (the Push Gateway) that you need to run separately.
Pulling has a few technical benefits, though. For one, only the puller needs to know what's being monitored; the thing being monitored can therefore be exceedingly simple, dumb and passive. Statsd is similarly simple in that it's just local UDP broadcast, of course, which leads to the next point:
Another benefit is that it allows better fine-grained control over when metrics gathering is done, and what. Since Prometheus best practices dictate that metrics should be computed at pull time, it means you can fine-tune collection intervals to specific metrics, and this can be done centrally. And since you only pull from what you have, it means there can't be a rogue agent somewhere that's spewing out data (i.e. what a sibling comment calls "authorative sources").
But to understand why pull is a better model, you have to understand Google's/Prometheus's "observer/reactor" mindset towards large-scale computing; it's just easier to scale up with this model. Consider an application that implements some kind of REST API. You want metrics for things like the total number of requests served, which you'll sample now and then. You add an endpoint /metrics running on port 9100. Then you tell Prometheus to scrape (pull from) http://example.com:9100/metrics. So far so good.
The beauty of the model arises when you involve a dynamic orchestration like Kubernetes. Now we're running the app on Kubernetes, which means the app can run on many nodes, across many clusters, at the same time; it will have a lot of different IPs (one IP per instance) that are completely dynamic. Instead of adding a rule to scrape a specific URL, you tell Prometheus to ask Kubernetes for all services and then use that information to figure out the endpoint. This dynamic discovery means that as you take apps up and down, Prometheus will automatically update its list of endpoints and scrape them. Equally importing, Prometheus goes to the source of the data at any given time. The services are already scaled up; there's no corresponding metrics collection to scale up, other than in the internal machinery of Prometheus' scraping system.
In other words, Prometheus is observing the cluster and reacting to changes in it to reconfigure it self. This isn't exactly new, but it's core to Google's/Prometheus's way of thinking about applications and services, which has subseqently coloured the whole Kubernetes culture. Instead of configuring the chess pieces, you let the board inspect the chess pieces and configure itself. You want the individual, lower-level apps to be as mundane as possible, let the behavioural signals flow upstream, and let the higher-level pieces make decisions.
This dovetails nicely with the observational data model you need for monitoring, anyway: First you collect the data, then you check the data, then you report anomalies within the data. For example, if you're measuring some number that can go critically high, you don't make the application issue a warning if it goes above a threshold; rather, you collect the data from the application as a raw number, then perform calculations (e.g. max over the last N mins, sum over the last N mins, total count, etc.) that you compare against the threshold.
In practice, implementing a metrics endpoint is exceedingly simple, and you get used to "just writing another exporter". I've written a lot of exporters, and while this initially struck me as heavyweight and clunky, my mindset is now that an HTTP listener is actually more lightweight than an "imperative" pusher script.
set -euo pipefail
should NEVER be used in a bash script. If you feel tempted to do so, then you are trying to make the shell scripting language be something that it is not.
This is the time for a full scale programming language such as Python or perhaps Groovy on the JVM or Go language. When you need to write robust code, use the tools that were created for writing robust code.
Bash just has too many quirks.
Note that this is related to the most common way that people build a Big Ball of Mud. You have a simple app and you need a couple of features so you add them on. Rinse and repeat. Before too long you have an app that does too much and was never designed/architected to do that much stuff. You are probably ignoring a number of techniques for integrating functionality in large apps such as message queueing, microservices, separate libraries or packages, multiple languages.
Shell scripts suffer the same trajectory towards too much complexity. When you see it happening, and before the task gets too complex, replace the script with an app and apply all the normal software engineering techniques to make it robust.
Tangentially related if you need search without the clustering and high availability story of elastic search and friends I highly recommend Xapian.
Its like the SQLite of search. Single library that provides the basic set of features you would expect in a quality search experience: facets, ranked search, boolean operators, stemming etc etc.
How does InfluxDB compare to TimescaleDB? My understanding is that the use case is pretty similar (time series/metrics), are they good at different things?
Turns out ONE answer is that DTLS is TLS for datagram connections (ie: UDP) and it’s pretty easy to forward along UDP packets at a proxy (as there is no need at all to track what sent or needs a retry).
It should be noted (if anyone reads this) that DTLS has protections and reliability on packets to establish the connection, but once it’s established it just goes back to being UDP although now encrypted, so you need to be ok with losses.
If you need messaging on top of that, CoAP adds in the counters and retries and reliability to UDP. Making UDP a little more like TCP but features to get this are one level higher in the CoAP packet. It’s a little weird, but gets you relatively reliable request/response HTTP-like (CoAP CONFIRMATION) and also event driven async MQTT-like (NON-CONFIRMATION). So that’s pretty cool.
Although near for me as GCP and Azure support CoAP but for some reason Amazon seems to refuse to and I’m locked in to Amazon for other reasons.
"By reference" means different things to different people.
The most accurate statement I've found is that JavaScript, much like Java, passes reference-by-value. Everything is passed by value, but if your value is an object, then the new scope gets a copy of the reference. If you modify the object by dereferencing (that is, `a.foo = "bar";`), your modifications to the object persist outside of your scope.
This is an important understanding to have when working with JavaScript.
You can use the X.509 Name Constraints extension to limit the power of your CA to just a single TLD. I personally just run on a public TLD with a wildcard cert (so my domains don't show up in cert transparency), but there is an alternative.
Apparently (I haven't tested this), Chrome only checks this on intermediate certs not root certs but you can mitigate around that by keeping your root cert offline. Will only work on "modern" browsers and operating systems.
It all started with https://github.com/rcarmo/miniredis (which I forked to add and experiment with pub/sub), and I just found myself doing it again and again because Redis is the quintessential network service:
By implementing it, you learn about socket handling, event loop for a specific runtime, threading models, data representation, concurrency (if you want to do a multi-threaded version), etc. None of my "ports" are fully functional, but they all helped me sort out some of the above plus build tools, packaging, dependencies, etc.
It's "hello world" for core cloud native microservices, if you will (and without having to do REST or JSON stuff).