If the first thing you failed to accomplish after 10 years was a medium complexity task, consider yourself lucky in some sense. Also, who determined the complexity level of this task, did you decide it was medium complexity, and if so is that corroborated by the team?
Moreover, most "bad" programmers don't necessarily think they are writing bad code, did you come to this conclusion yourself? If you did, it seems like maybe some other factors came into play, maybe you didn't have enough time, maybe the requirements were unclear. In either case, 1 failed task seems like a lot to evaluate 10 years of a career on; having recognized this behavior in myself, you might be catastrophizing this failure in a way that's not productive for you.
If you received this criticism from somewhere else, that's okay too, being "over 30" isn't some death sentence, there's plenty of time to reinvent yourself, start a new career or rebuild your skills from the ground up - if that's what you want to do.
Maybe part of this is just, you aren't sure you want to be a programmer and you are using this "failure" as a kind of escape hatch, which you know, is also fine. Or, maybe you aren't sure you want to devote the effort you might think it would take to actually improve as a developer
I would say, check in with yourself, and figure out what you want to do or at least what you don't want to do. I wouldn't change careers simply because you think you aren't good at this one. Being a PM, has a whole other set of challenges (but if you are actually curious, the idea that you have to be able to "code it the right way" should not be a blocker).
If you do decide you want to become a better programmer, realize it's not going to happen overnight, but conversely it's not going to take as long as you think. But you do have to start somewhere. Rather than ask on HN, I would go find someone whose technical opinions you trust (you mentioned talking with people in the industry - hey developer evangelist is also a role out there for you, heh), and ask them to review some code you've written and have them articulate those weak points, and start from there.
If you decide to start, just remember to start small. It doesn't have to be late night grinding sessions, 15-30 minutes a day reviewing a topic, solving a code challenge, reading a chapter of a book - can really add up after a few months.
At the time there were some motions towards a "better MapReduce," Scalding (ooh weee that had a learning curve) comes to mind, but I feel like Spark really lowered the bar for getting big data tooling and the ecosystem into mid level and pre-ipo orgs.
“When you discuss languages it almost becomes like a
religious argument. But really a language is just a tool.
It’s like arguing, well, which one is better: a hammer or a
screwdriver? You tell me what you want to do with it and
I’ll tell you which one is better.”
But isn't this comment from a time, like 1984, when programming languages were quite domain specific. Can we not have coherent discussions about General Purpose languages, where we can actually say some languages are better than others, and while specific use cases should be taken into account; that it isn't quite like arguing which is better a screwdriver or a hammer?
At this point a pipeline built on top of Stitch / Fivetran /
dbt is far more reliable than one built on top of custom-
built Airflow tasks.
I'd be curious if anyone who has used or integrated these products into their infrastructure could verify or comment on whether they are as effective as the author seems to suggest.
If you hire a data engineer who just wants to muck around in
the backend and hates working with less-technical folks,
you’re going to have a bad time.
I'm not sure this was the intent, but I found this somewhat dismissive. I think communication skills are indeed important and being able to effectively explain technical considerations to less-technical parties (or parties whose technical expertise is not aligned with Data Engineering), but I have encountered in my own experience an active disregard for those considerations by data scientists as orthogonal to their needs at best or at worst, details for which they cannot be bothered. This is underscored by the notion that we, as Data Engineers, "muck around in the backend." We do, and we have to, and it helps to like it.
There are a few other areas of input and contribution that a good data engineering team can provide, that I don't think get enough attention in the post:
1) Machine Learning Productionization
2) Being a source of data expertise (consulting) with other
developers (working on services or the main product) in
the organziation
Regarding 1, while the author seems convinced that the ELT/ETL tooling and ingestion pipeline building can be taken off-the-shelf, I don't if it is as likely that there is the same kind of mature tooling for machine learning model deployment/integration. Though, I believe that is changing, slowly.
Does any know of a community of practitioners in a field or discipline who are by and large usually impressed by output of the members of that field on a regular basis.
Not rhetorical, asking for myself. I guess, I want to know how much more toxic are we as a community (to the degree you can talk about all folks who read or might read HN as a single monolithic community).
I mean, I know a few artists, they are absolutely savage when it comes to the work of others. Is this a dev problem or human one?
from cursory experience in physics and biology is s pretty much the same: sometimes impressed, sometimes cynical, probably spending more time in the latter. IT's only normal and it's good: criticism pushes things forward
What's actually funny here, is that maybe you aren't old enough. The "Super-cool" functional way has been around as long as types and functions, Lisp and ML have been around since the what...50's and 70's respectively. Immutable state and referential transparency are boring. Get off MY LAWN kid!
