There was a temporary outage because a switch accidentally got disconnected from power. Apologies for the disruption.
Had issues with out upstream connectivity. They appear to be resolved now.
Jon's Minetest Server Notices reshared this.
Cable Replaced
The failing ethernet cable that's been plaguing the server has finally been replaced. Things should be a lot more stable around here. Sorry for all the issues lately.
Looks like the network cable that connects the server to the router is starting to fail. Have a replacement, but running it will be nontrivial. I expect it to be completed by the end of the weekend though.
We're having intermittent connectivity issues this morning. Seems to be an upstream issue.
Brief outage because the new server accidentally came disconnected from power.
Everything's back up. Anxiety levels returning to normal.
We are having intermittent connectivity issues. ISP is dispatching a tech within three days *groan*. Hopefully things will be resolved soon.
System was down for a good chunk of the day because the main drive suddenly and unexpectedly became full. System is back up and running, but taking a while to resynchronize with the rest of the network.
Hopefully things should level out in a few hours.
We are currently experiencing an upstream connectivity issue which is severely impacting our services. Hopefully this will be resolved shortly.
Changeover to Hetzner is 99% complete. We ran into some unforseen complications, namely a typo in the firewall config that took some time to diagnose and fix.
God, I never want to have to do that again.
Jon's Friendica Node Notices reshared this.
Reminder that we will be migrating to a new VPS today. Extended outages are to be expected.
Brief outage today as I somehow misconfigured the routing tables... ironically enough, trying to resolve a network congestion issue.
Everything's been restored, but the original issue hasn't been resolved yet. Will work on it some more later... hopefully without breaking the Internet again.
I've been troubleshooting some network congestion issues that have cropped up recently. 95% certain I know the cause and already have a partial solution in place, but working on a better one.
Edit: typo
There's been a lot of lag on this node lately. I've been tweaking the database to try to improve performance. Apologies for any inconvenience.
Always a bit of a nail biter doing a Friendica update. The database is getting larger and larger and requires more and more time to finish the mighration (and be certain that it worked) but we're now running on Friendica 2023.04.
Tomorrow (February 1st) the landlord will be in to do some work on the fuse box (replacing it for a breaker panel). This is scheduled to start at 09:00 EST (14:00 UTC) and should last for about four hours.
During this time, we will be without power, and this node will be down.
Apparently the electrician needs to come back tomorrow morning to finish up. There may be brief outages again.
We just had an unusually smooth upgrade to version 2023.01. As always, please report any issues.
Hey all,
We have been having some ongong issues with people's feed failing to load. There is a known bug that is scheduled to be fixed in the new version of Friendica coming out this month which I believe to be responsible for the problem. In the meantime, I'm seeing what I can do to alleviate the issue.
I apologize for any inconenience, and appreciate your patience.
Brief Outage
So, we had a brief unexpected outage today. I put off an upgrade on the VPS for too long and it came back to haunt me. Services may be a little lagged today as everything resynchronizes with the rest of the network.
Blank Page Issue Workaround
Hey all,
With the increased traffic on the network lately, we've been having some intermittent issues with people's feeds coming up as just a blank page. If you've been having problems with this, you can alleviate the issue by reducing the number of posts that show up per page on your feed.
You can find these settings by going to: Settings ➡️ Display ➡️ Content Settings
...and then changing "Number of items to display per page" and "Number of items to display per page when viewed from mobile device". I currently have mine set to 10.
So this appears to be a known memory issue in the Friendica code. A fix is already being worked on, and if nothing else, a new release is slated for some time in December.
For more details, see:
github.com/friendica/friendica…
Noticed that there had been a new stable release of the Friendica software that I hadn't noticed. Upgrading the software while the network is under such strain was a little harrowing. The database migration failed, but I was able to recover.
Everything should be good now. Please contact @Jonathan Lamothe (he/him) ❌ if you encounter any issue. I can also be emailed at jonathan@jlamothe.net.
We've been having some congestion issues on this node (perhaps due to the increase in traffic on the Fediverse in general). Sometimes feeds are just showing up as blank screens, but it seems to happen infrequently and only lasts for a couple minutes.
If you're having persistent issues, you can DM me, or let me know by email at fedi-issues@jlamothe.net, and I'll look into it.
Friendica 2022.06
And we are now running the latest version of Friedica. I had some apprehension about the upgrade since everything broke when I tried the release candidate a while back, but everything seems to be running smoothly.
Please feel free to reach out to me if you have any questions/issues.
After a brief power outage, we are back up and running again.
This node will be going down for an OS upgrade in the next couple of hours and will probably be down for the better part of the day.
See you on the other side.
And, despite some difficulties with the migration to the new OS, we're finally back up and running again.
As a side note: if I ever talk about changing the OS on the server again, kick me in the head repeatedly until I come to my senses. It's the humane thing to do.
like this
the thing I hate the most about debian is that not all tools are in the path by default, and I'm like WHYYYY
New Users Forum
!New User Forum
Hey all,
In an attempt to help new users to find each other, I've created a New Users Forum. This message has been posted to that forum. Please feel free to drop in and introduce yourselves.
New User Forum reshared this.
TOS Update
I've amended the Terms of Service to include trolling as a bannable offence. This still falls within the overarching theme of "don't be a dick".
Email Outage
Due to a misconfiguration, our outbound SMTP credentials were compromised. After an investigation, I was able to determine how this was done and correct the issue.
There is no evidence that the attacker was able to compromise user accounts, so you do not need to do anything. What this does mean is that outbound emails will not be working today, but should resume working by tomorrow. This means no email notifications and no password resets.
I apologize for any inconvenience this may cause.
Looks like we had a spambot. I blocked them but couldn't figure out why my mail provider kept blocking SMTP access. It's because I forgot to clear postfix's outbound mail queue, so as soon as I'd re-enable it, it'd just resume sending the old spam again.
In short, I'm an idiot.
Temporarily sending outbound email through Gmail's SMTP instead. It works, but ugh...
Invitation Link Not Working
Hey all,
I've managed to get the email system working (almost) perfectly now. The one thing that's still not working is the link to invite someone to the node. It will say that an invitation email has been sent when it actually hasn't. This seems to have to do with a bug in the Friendica code itself that I have reported upstream.
In the meantime, you can work around this by copying the text of the invitation and emailing it yourself, or you can just direct them to social.jlamothe.net to register their account.
I apologize for any inconvenience.
Ian Molton 🇺🇦🇵🇸🏳️🌈🏳️⚧️
in reply to Jon's Friendica Node Notices • • •