Which privacy-oriented companies do you prefer?

“Good luck, we don’t care.” is their stance, as far as I can tell.

Same model and same prompt won’t necessarily create the same result, unless I misunderstand how these audio models work.

It's possible to generate the same images and text from LMs by tweaking the settings, right? Are audio models different?

I wonder how many premium accounts Anna’s Archive had to use to scrape the whole thing. Surely Spotify has scrape protection and wouldn’t allow a single account to stream (download) millions of separate tracks.

I have a feeling they didn't use premium accounts since they downloaded at 160kbit/s, which is the highest quality that free accounts can get.

Premium gets 320kbit/s (or lossless)

I haven't looked at the code but I would be surprised if the premium account "requirement" is anything more than an if statement that can be commented out.

Junior devs become senior devs. No more junior devs = no more senior devs.

I’m concerned this will drive teens to dodgy apps and services that have lax data security and no oversight.

Discord isn’t banned, but Twitch is? Interesting.

Surely Discord harbours more bullying than Twitch (where image sharing isn’t even a feature).

Why wasn’t the rollback fixed within the second minute after they saw the 500s?

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up".

I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do that, it would appear that the code has to be written much more defensively than what they're doing it right now.

Yea agree.. This is the same discussion point that came up last time they had an incident.

I really don’t buy this requirement to always deploy state changes 100% globally immediately. Why can’t they just roll out to 1%, scaling to 100% over 5 minutes (configurable), with automated health checks and pauses? That will go along way towards reducing the impact of these regressions.

Then if they really think something is so critical that it goes everywhere immediately, then sure set the rollout to start at 100%.

Point is, design the rollout system to give you that flexibility. Routine/non-critical state changes should go through slower ramping rollouts.

Can't get hacked when you are down.

For hypothetical conflicting changes (read worst case: unupgraded nodes/services can't interop with upgraded nodes/services), what's best practice for a partial rollout?

Blue/green and temporarily ossify capacity? Regional?

- Push a version with the new logic but not yet enabled, still using legacy logic, able to implement both

- Push a version that enables new logic for 1% of traffic

- Continue rollout until 100%

Can also do canary rollout before that. Canary means rollout to endpoints only used by CF to test. Monitor metrics and automated test results.

That's ok but doesn't solve issues you notice only on actual prod traffic. While it can be a nice addition to catch issues earlier with minimal user impact, best practice on large scale systems still requires a staged/progressive prod rollout.

Yep. This is definitely an "as well as"

Unit test, Integration Test, Staging Test, Staging Rollout, Production Test, Canary, Progressive Rollout

Can all be automated can smash through all that quickly with no human intervention.

You can selectively bypass many roll out procedures in a properly designed system.

If there is a proper rollout procedure that would've caught this, and they bypass it for routine WAF configuration changes, they might as well not have one.

Not sure I buy it. Do 1% for 10 minutes. I mean it must have taken over half a day to code and test a patch. Why not wait another 10 minutes.

The update they describe should never bring down all services. I agree with other posters that they must lack a rollout strategy yet they sent spam emails mocking the reliability of other clouds

The irony is they support rolling out incrementally with some of their products for deployment.

They need that same mindset for themselves in config/updates/infra changes but probably easier said than done.

I believe they use Argo according to a previous post mortem.

https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-...

"Please don‘t block the rollout pipleline with a simple react security patch update."

The "half the internet is down, nothing we can do" excuse works great the first time, but doesn't fly the second time in a month.

What solutions are there for Multi DNS/CDN failover that don't rely on a single point of failure?