NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
Why AO3 Was Down (reddit.com)
zerocrates 2 days ago [-]
Is it faster to convert a column like this to unsigned? Obviously assuming you don't use negative IDs in the application.

That's much more of a "kick the can down the road" solution to only double your usable range, but if all positive the values in the rows shouldn't actually have to change, just the column metadata, so it could theoretically be more or less instantaneous. I guess in practice this doesn't happen; the server would rather use its generic "rebuild the table" alter method for changing a column type.

But it seems like you could reasonably do it if it's a signed-to-unsigned change and there's no negative values and there's an index on the column to make checking that fact fast. Or one of those third-party/lower-level type tools could let you do it without any checking.

afandian 2 days ago [-]
I don't know what DB was used in this csae, but Postgres doesn't have unsigned integers. It always struck me as hugely wasteful, as e.g. sequences start at zero by default.
masklinn 2 days ago [-]
At $dayjob we've actually used this property once or twice: if you need to merge two tables you can keep the positive ids for the first one, and use negative ids for the second one. It only works once, but damn if it's not effective when you need it, and it conveniently flags all the records with an id under some limit (positive and negative both) as "pre-transition" record when you're looking for patterns.
afandian 2 days ago [-]
I’ve also seen positive and negative ids for entities with different properties (can’t recall what). Felt like an unnecessary hack though.
kristianp 2 days ago [-]
It's mysql.
adamcharnock 2 days ago [-]
An interesting idea! I suspect a major speed up would come from the fact that the column is staying the same size. So (I assume) far fewer bytes would need to be moved around.
Groxx 2 days ago [-]
Ha, a site I worked on hit this limit for the "follow relationships" table - had to build a new compound key table to migrate to, with triggers to dual read/write, to unbreak everything. In a few hours of "wtf" -> "oh crap" -> "well I guess we gotta do it right this time" and quick coding.

And then I pulled apart PT-OSC to make it more... less incredibly stupid about resource use, so it wouldn't cause too much load while it backfilled. And let it run for about 6 weeks.

Good luck! It's a fun problem to have - excess success, and a light puzzle to solve :)

bilka 2 days ago [-]
Do you happen to have those PT-OSC changes around? We've already migrated bookmarks with the downtime (with PT-OSC), but there are more tables that would be nice to get migrated away from int without going into maintenance or shedding a lot of load.
Groxx 2 days ago [-]
No, it's long, long gone.

When I did it, the script was a bit of a mess of trigger setup, and then a backfill that only monitored replica lag, as if the status of the much less heavily used failover instance was somehow the most important part of a database. Hopefully that's no longer true, and none of this is necessary any more.

So I essentially split it in half, so I could keep only the trigger setup, and carefully read the queries the backfill would perform so I could duplicate it. And then wrote a very simple loop of "select N records, copy to new table, check how long that took. scale up by min(5%, 100), scale down by 30%, if outside target bounds".

Intentionally very polite to the main DB, because once the triggers are in place it really doesn't matter how long it takes. It dropped down to single digits at peak load on some days, so I think that was the correct choice.

p0w3n3d 2 days ago [-]
Hacker News helps me everyday break my information bubble. Archive Of Our Own is something that I wouldn't walk into when wandering through the internet
chii 2 days ago [-]
> I wouldn't walk into when wandering through the internet

it's interesting that some people are on the internet but is very well insulated! AO3 is very well known for me...

diggan 2 days ago [-]
> it's interesting that some people are on the internet but is very well insulated

Not sure I'd call it "insulated", the internet is just very, very vast, even when considering "just" the English-speaking web. Then you have all the other "versions" out there too that are kind of hidden to most people :)

Anecdotal, but also first time I heard about AO3, and I'd consider myself having broad interests and generally well-read, although my interests doesn't include fanfiction so maybe not so weird I haven't heard about it before.

jorvi 2 days ago [-]
Its very much a gendered thing. If you have lots of female (online) friends and late night topics with them ended up trending spicy, you might hear of AO3.

FWIW the vast majority of writing on there is decidedly mediocre. There is also an even more inferior alternative called Wattpad.

Funnily enough you learn that in general we aren't all that different in our tastes, it's just that what men like to watch, women like to read / imagine.

Edit: to paint the picture, this[0] was sent to me a while back :-)

[0]https://www.tiktok.com/@alexarowe11/video/746846214634761757...

ethbr1 2 days ago [-]
The great thing about crowdsourced content is that if you have mediocre at scale, there are some gems!

I remember when I first stumbled across the main Antimemetics Division storyline on SCP. https://scp-wiki.wikidot.com/antimemetics-division-hub

jorvi 2 days ago [-]
QNTM is actually 'remastering' this into a book and then publishing it.

There's also another great one about a spinning disc that ends up opening a portal when placed on a mirror. SCP-093.

ahazred8ta 2 days ago [-]
"But... we don't even have an anti-memetics division."

"Yeah, we get that a lot."

ethbr1 20 hours ago [-]
I always forget how good the anti-memetics division stories are.
diggan 2 days ago [-]
The world of "spicy reading" isn't new to me (male), just that website in particular.

I don't think it's as gendered as you paint it, but I'd also acknowledge it depends a lot on geographic location, probably looks different where I am compared to where you are, I agree with that we probably aren't all that different in tastes in general :)

Freak_NL 2 days ago [-]
Anecdata, but it does seem like a very gendered divide on the whole. The only reason I know of Archive of Our Own is because my wife is quite familiar with it. And I do consider myself well acquainted with various smut filled corners of the internet well beyond Literotica.
Ferret7446 2 days ago [-]
Literary vs visual porn is strongly gender divided. I would be very surprised if you were into erotica and weren't aware of AO3; I find it more likely that as a male you aren't actually very into erotica, as is the norm.

I'm sure you can find lots of studies about this, but just for a very easy data point, a few years old survey shows that AO3 is ~54% cis female, ~5% cis male, rest nonbinary/trans/etc.

diggan 1 days ago [-]
> Literary vs visual porn is strongly gender divided. I would be very surprised if you were into erotica and weren't aware of AO3; I find it more likely that as a male you aren't actually very into erotica, as is the norm.

Well, I've read my share of erotica, maybe I wouldn't say "very into erotica" but well-read enough it seems, compared to my peers.

I guess everyone who happens to read erotica automatically isn't interested in fan-fiction, so they never stumble across AO3, I can hardly be alone in this situation.

fouc 1 days ago [-]
Maybe that's more a statement about the type of content it has (or originally had)?

Fan Fiction isn't strictly about erotica after all, if you filter out the erotica part, the distribution might be a bit different..

I would assume webnovels.com wuxiaworld.com and r/ProgressionFantasy might be more male gendered, even though that some of type of content could easily be on ao3 too..

2 days ago [-]
Sharlin 2 days ago [-]
I’d say AO3 is the insulated part.

Having been an active internet user for longer than most AO3 users have been alive, the first time I heard about it was a few years ago in a student radio show about the fanfic genre and culture. Poorly written smut featuring popular culture characters has just never been my thing. Probably because I’m not that much of a fan of any specific fictional setting or franchise in the first place.

Ferret7446 2 days ago [-]
What do you mean by "active"? IMO, if you haven't spent at least few years on, e.g., 4chan, then you haven't been very active at all, you've just stayed in a bubble.

Who is a better judge of what is insulated or not, someone who spent decades living in a single city, or someone who spent a few years traveling everywhere?

(I bring up 4chan because by virtue of being practically unmoderated, it is about as unlike a bubble as you can get. Where else do you get people into origami, hardcore rape porn, international backpacking, nobel prize winning mathematicians, and pranksters running iphone microwaving campaigns rubbing shoulders?)

xeonmc 2 days ago [-]
I think in this case, the more appropriate adjective would be “quarantined”.
parlortricks 2 days ago [-]
this is the first i've heard of it
paganel 2 days ago [-]
First time I read about it, not exactly sure what it is, by quickly glancing at it. Looks like a collection of links to some fan-fiction stuff.
bananaflag 2 days ago [-]
It is currently pretty much the main repository of fanfiction.

I've been reading fanfiction on the Internet for two decades, so for me it would've been quite hard to miss it.

MangoToupe 2 days ago [-]
> Archive Of Our Own is something that I wouldn't walk into when wandering through the inter

Kind of crazy to hear. AO3 is so culturally massive I don't know how you miss it.

malnourish 2 days ago [-]
I have never heard of it, and I've been pretty active on the Internet since the early 2000s.
rsynnott 2 days ago [-]
The internet’s big. I’m aware of it due to a moderate tvtropes addiction (there’s a enough crossover that it’s hard to miss) but probably otherwise wouldn’t be familiar.
ahazred8ta 2 days ago [-]
Remember "Don't they know we're brothers??" from Supernatural? This was the site. <https://archiveofourown.org/tags/Dean%20Winchester*s*Sam%20W...>

XKCD on index number exhaution: https://xkcd.com/865/ Remember Data and the Borg? "Accessing... accessing... accessing... accessing..."

12_throw_away 2 days ago [-]
For anyone who feels like looking up exactly what this bookmark was pointing to: I did, and very much wish I hadn't!
nikanj 2 days ago [-]
I know I'll regret this, but how do you navigate to bookmark by column id?
camel-cdr 2 days ago [-]
The link is in the reddit comments.

It's Dead Dove though.

Sharlin 2 days ago [-]
Translated from fanfic to English: "a warning tag that signifies the story contains potentially disturbing or morally questionable content."
nikanj 2 days ago [-]
Oh my. Well, today I've learned something new about gen z and the internet
rsynnott 2 days ago [-]
AO3 is nearly 20 years old, and I think it was mostly LiveJornal refugees anyway. This is one that you very much can’t pin on gen-z.
2 days ago [-]
eknkc 2 days ago [-]
What in the name of fuck
heavensteeth 2 days ago [-]
I ask in complete earnest: is that your honest reaction to seeing it, or did you hype it up for your comment? Personally very little could evoke that kind of reaction from me. Maybe a little, "oh, that's an interesting thing to be turned on by" but for the most part, who cares?
ThrowawayTestr 2 days ago [-]
I can't remember the time when I was so innocent that forced mpreg breeding wasn't shocking.
Freak_NL 2 days ago [-]
I mean, it's fiction. If a writer can't explore the depths of human behaviour there, then where?

The only uncomfortable thing there are the explicit references to Harry Styles and Louis Tomlinson. I do take exception to using real people in fiction if you proceed to heap abuse on the characters which you model on those celebrities. (The story seems to use only the given names, but the tagging makes the link explicit.)

Obviously, you can refer to real world famous people in fiction — it would be silly to write a book about 2025 America and not mention that the president is Trump if it includes political themes — but there are limits.

rsynnott 2 days ago [-]
AIUI AO3 has an “anything not literally illegal goes, as long as it’s fanfic” policy, so does get a certain amount of this sort of thing.
ahazred8ta 2 days ago [-]
You remember the Supernatural line "Together? Don't they know we're brothers??" "They don't care." This was the site. Sweet cheese and crackers. <https://archiveofourown.org/tags/Dean%20Winchester*s*Sam%20W...>
MangoToupe 2 days ago [-]
New to the internet?
schoen 2 days ago [-]
A bookmark for every view of "Gangnam Style"!

https://arstechnica.com/information-technology/2014/12/gangn...

wging 2 days ago [-]
That article was from 2014, it has many more views now (about 5.6 billion).
RainyDayTmrw 2 days ago [-]
It's kinda impressive that they got to 2 billion rows - with indexes, no less - without falling over.
jiggawatts 2 days ago [-]
Point queries — typical of this kind of app - scales as log(n) in the number of rows. (Assuming a typical b-tree database index.)

This kind of workload cheerfully “scales” to your disk capacity.

RainyDayTmrw 1 days ago [-]
Locking would make that more complex, I believe?
jiggawatts 1 days ago [-]
Not necessarily. Typically a b-tree single-row modification updates only one leaf page. Higher levels change only 1/N times where N is the number of items per page, typically in the hundreds. So three levels up the chance of a lock conflict is only 1/N^3 or about one in a million. Four levels it is one in a hundred million, etc...
CubsFan1060 2 days ago [-]
I didn't see it mentioned, but the quick fix for this (assuming you don't depend on the order of id's) is just to alter your sequence to use the max negative int, and increment from there. Not a complete solution, but buys enough time to actually fix the issue.
kristianp 2 days ago [-]
They're using mysql and Rails. And Jira.

> Mysql2::Error: Out of range value for column 'id' at row 1 (Mysql2::Error)

https://otwarchive.atlassian.net/jira/software/c/projects/AO...

olivermuty 2 days ago [-]
Uh, the bookmark that broke it all was to a part of the internet I have yet to experience since getting online some 30 years ago. Alphas and betas and omegas, it was a wild ride.
Freak_NL 2 days ago [-]
The bookmark itself, for the curious:

https://archiveofourown.org/bookmarks/2147483647

That alpha/beta/omega thing is quite huge apparently, but not something you would ever encounter outside of specific subcultures (like Archive of Our Own):

https://en.wikipedia.org/wiki/Omegaverse

madaxe_again 2 days ago [-]
This is like seeing a brick wall 40 miles down a straight road and yet still managing to drive into it, and then blaming the wall.
ohdeargodno 2 days ago [-]
Ao3 doesn't have a dude getting slack alerts by a dozen monitoring agents. It's one of the last holdouts of the old, more personal internet. Hell, it's even certain that they forgot or even didn't know that the type was an unsigned int.

And that's perfect. Blame the wall too, because it was running just fine. It's a site to write (mostly porn), with better uptime and more daily users than most of the companies posted on HN daily.

camel-cdr 2 days ago [-]
I wasn't sure what the percentage of porn is, so I counted the number of works for each maturity rating:

    4,247,583: Teen And Up Audiences
    4,173,082: General Audiences
    2,816,083: Explicit
    2,271,446: Mature
    1,676,061: Not Rated
darkwater 2 days ago [-]
I guess that whoever maintains that infra simply hadn't thought of it or was not aware. It's not something you get for free in a monitoring system with some agent like disk usage for example. You need to know and remember you have a hard limit on IDs and be aware at which ID you are.
hinkley 2 days ago [-]
Meanwhile if I keep reminding people where the wall is and how fast we are approaching it I’m considered “negative”. That”s the real reason this stuff happens. If someone noticed, the got tired of harping on it and without the constant barrage everyone else immediately let it go out of sight, out of mind.
darkwater 2 days ago [-]
In a company, totally. But here it is a volunteer effort, I doubt it had happened.
alt187 2 days ago [-]
> This is like seeing a brick wall 40 miles down a straight road and yet still managing to drive into it, and then blaming the wall.

Not really, no. For example, if you drive into the wall, you may die.

Another experience that feels like death is working in a company that implements on-call rotations.

It would be too easy to draw out a parallel between how you approach a free fanfiction website (the website should mystically owe you five 9's uptime) and the mentality that metastased in the industry.

Instead, I'm gonna take this opportunity to point out that the AO3 downtime affected you, as a non-user, enough to vitrify the admin, where hardcore users laughed it off (because they're not entitled toddlers).

madaxe_again 2 days ago [-]
I don’t think I turned the admin into glass, nor vilified them - just pointed out that this sort of thing is readily avoided.

But sure, I committed a hate crime.

randallsquared 2 days ago [-]
> enough to vitrify the admin

Not sure it was that solid.

2 days ago [-]
notorandit 2 days ago [-]
> typical database column

Typical for 70s and 80s.

Honestly, designing a 21st century database is a different thing if compared to back then.

You can use 128 bit integers, provided that you really want to use integers. And maybe you put a timestamp along.

rsynnott 2 days ago [-]
The website appears to date from 2008. This was a _very_ common latent bug at that point, particularly because Rails would basically force you to implement it. I assume this got fixed at some point, but for a long time all ActiveRecord models had an autoincrementing ID, which had to be a signed 32 bit int. There were scary monkey-patching workarounds if you wanted something more sensible.

EDIT: And, yes, it is apparently Rails! https://fanlore.org/wiki/Archive_of_Our_Own#Timeline

throwawaysoxjje 2 days ago [-]
Nah I made the same mistake back in 2009 for a system that was storing behavior events during malware analysis.

You don’t often expect to have two billion of something until you do.

9dev 2 days ago [-]
It's not like those two billion things just materialise in your database, right? Someone must have watched that graph climb, and climb, and climb, approaching the limit.
detaro 2 days ago [-]
If they have that graph and remember the limit they choose 15 years ago... It's not something you think about constantly running a mostly stable code-wise site.
shakna 2 days ago [-]
Salesforce is a rather popular platform.

Its defaults are also either a 18-character ID, or a 32bit integer. So, unless you take the effort to actually fight Apex, you're gonna hit this problem sooner or later.

quickthrowman 2 days ago [-]
Doesn’t an 18-character alphanumeric ID give you 18^36 combinations? 1.54 x 10^45 seems like enough combinations.
shakna 2 days ago [-]
That's the point of the "or". You probably don't know which you're getting. It's what makes that particular design decision bite you more often.
Sharlin 2 days ago [-]
One of the first things I internalized about databases was "just always use BIGSERIAL for primary keys". There are very few good reasons not to.
2 days ago [-]
jarofgreen 2 days ago [-]
or use UUID/GUIDS, many databases (eg PostgreSQL) and frameworks (eg Django) support them.
dwedge 2 days ago [-]
Using uuids can cause lots of problems with indexing, fragmentation, row size and index size
j16sdiz 2 days ago [-]
let's use 128bit integer and handle them like floats in php!

and maybe put a 32bit timestamp along and pretend it can somehow store more than a 32bit integer can.

charcircuit 2 days ago [-]
>to fix it they have to migrate the entire database to use a different type for bookmark IDs... except of course this will take a while because there are two Billion Of Them Lol

You can shard them between 2 tables. Then migrate them to a single one later.

ohdeargodno 2 days ago [-]
There's no SLA for Harry Styles porn. Run the migration, lock the table for two days and redo the same in 13 years when you get to 4 billion bookmarks.
camel-cdr 2 days ago [-]
> There's no SLA for Harry Styles porn

But what about my good night's sleep? How can I go to bed without reading about my favorite blorbos?

ohdeargodno 2 days ago [-]
Real ones use bookmarks to find them ag- ah, shit.

Real ones back them up in a single .txt file

kijin 2 days ago [-]
In 13 years, the Unix timestamp will probably be a much bigger problem.
rsynnott 2 days ago [-]
I mean I’d assume they went for a 64bit integer. In a few million years, people who are into weird porn about whatever the temporally local equivalent of Harry Styles is (probably some sort of robot) will once again be mildly inconvenienced.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 13:16:17 GMT+0000 (UTC) with Wasmer Edge.