Welcome home! Please contact lincoln@icrontic.com if you have any difficulty logging in or using the site. New registrations must be manually approved which may take several days. Can't log in? Try clearing your browser's cookies.
“502” error on this site
Linc
Site ownerDetroit Moderator
Hi all, just a note to say I did notice the “502 gateway” error the site was throwing earlier and it’s been remedied. We did notice the same 3 weeks ago but were only able to put a temporary fix in place at that time and knew it would happen again. A longer fix was made this time.
If you’re curious, we’ve had challenges with our backup automation taking up too much space in the database server, and that server filling entirely is what triggered the downtime.
7
Comments
You are at it again (behind the scenes tinkering)
Second time I noticed some weird new editing features temporarily introduced …
… noticed the 502 gateway fault on occasion.
HAPPY Easter.
[don’t worry Jesus returns after Good Friday] Many thanks to those keeping us working …
I think it was this error again today and last night? It looked the same, a 502 bad gateway fault.
indeed
I had this fault but is gone now …
We had the same problems with the server backups taking up too much disk space. I got it running again but the underlying problem maybe isn't really fixed, so if it happens again just be patient and I'll kick it back online as soon as I notice. It's not a serious problem, just inconvenient that it keeps cropping up.
Noticed the “502 Bad Gateway” error seems to be back, and with disturbing frequency.
Hope you can work your magic @Linc…
Yes, same here..
Icrontic (which shares the server with NB) is getting swarmed by bots scraping the contents for AI nonsense, which is a growing problem across the Web. I've blocked a number of them for now, which hopefully will help. I have longer-term ideas but lack the time to implement them this week.
All seems good now.
It seems to all be running smoothly again… thank you.
I'm not getting 502 error anymore but it is still loading pretty slowly for me compared to other sites. Not sure if that is related.
Edit: Of course as soon as I post that it starts loading more normally.
The “502 Bad gateway” error seems to be increasing again, I’ve seen it about 5 times this morning in just checking the front page and making a couple of posts.
I was thinking Gateway had something to do with the rapture — maybe it did happen after all.
Or maybe it’s a scold from the celestial IT Central Command: Bad Gateway, no donut!
It’s back… over the past few days I’ve seen it quite often. Everything from using Search to Autosave to refreshing the Recent Discussions page seems to trigger it.
Perhaps @linc might take a look?
Yeah, we've been actively fighting against it on the Icrontic side (we have a small volunteer team that looks after our shared webserver). It's basically AI bots slamming the sites and causing issues. We rebooted a few things and the pace seems to have slackened as of yesterday afternoon (ET).
Did you have fresh issues this morning?
I had some issues this morning. AI bots slamming? Why does AI want to slam this forum? For training purposes?
Me too, this morning, 4-5 times before I gave up.
Yes, it's a widespread problem for the last couple years that is costing small sites a lot of trouble and money.
How can we protect ourselves? You say money? Maybe we can run a small GoFundme to save the forum and take the necessary measures?
One way (been getting 502 messages too) is to learn about the EFF who have been fighting/warning about this process for many years...
https://www.eff.org/event/effecting-change-enshittification
This morning again, intermittently. Another forum which I am a member of at one point had 2200 “guest” bots visiting, while the registered member count was something like 4 most of the time.
It depends on the hosting provider. We're on a cost-capped plan with DigitalOcean & have volunteers to fight against the issue, so we're relatively safe for now. I appreciate the consideration. It pays to have a sister site full of tech nerds.
I'm just saying if the bots really came for us hard... I am versed in the dark arts and have not yet begun to fight. 😈
I got a little more aggressive with banning bots this morning after some instability the last few hours.
Spent some more time this afternoon diagnosing things and determined that the server simply needs more CPU power to be able to manage both the increased overhead of our defensive software (fail2ban) and still serve both sites at the normal speed. I've given it a small bump in power and things are much faster now.
The site certainly seems more responsive… thanks Linc!