the time i fucked the servers
Not literally sadly, so not actual nice fucking
but during the whole lockdown covid era we went full into remote work, and i moved from user support to networking and server infrastructure stuff
support was fine because mostly dealed with users and shit issues and got kinda paid the same, so i was moved to servers and well, i started working into that over the years, the boss i had at that point was a fucking piece of shit, an homophobic asshole, he left after a while and that work became mostly mine.
i had two big clusters to manage at that point, full prod and another one that i was building to replace it, iirc 3 dell servers with dumbass ram amounts, a couple of storage things with lots of ssd terabytes and stuff.
we ran the redhat virtualization suite, this was a long time ago, can't really remember the name of that, but we paid for it and whatever, but the main prod server ran the opensource version of that, so no support or whatever.
One day i noticed the webui of that server being slow, glitchy, i wasnt able to delete disks and stuff like that, considered it a fluke, reseted the web server got it back, but weirder.
big issue raised from there, if i turn off a vm, i wasnt able to get it back to run again, it just fucking dissapeared from existance.
we did had backups, but mostly of DBs, content, media, whatever, but the vms itself were never backed up.
i was kinda working on that, getting everythging prod running on the new nodes and then get a paid backup software.
well, fuck me, because this serever web interface was broken, attaching the new iscsi disks worked, and i was able to copy some disks and vms over, some failed, i moved some over a fucking thumbdrive, others to my computer and then to the new server
i found my ways, i cloned machine filesystems live over sftp, cloned windows VMs with clonezilla without rebooting.
at some point during the year i was close to finish, some of the most critical bullshit was still runing there, a haproxy that was routing some sites, some wordpress sites
the whole shit was a mess, people before me created VMs for everything, one vm for one site, other for another, whatever, shit infraestructure.
one day, i was working on the secondary node remotely, i was running fedora and those servers ran centos, almost the same expertience over ssh and locally for me, got trusting and stuff.
i was fucking around outside my place during work hours and i was remoted to my computer in my phone
i was like im gonna reboot my computer
i rebooted one of the broken nodes
it was friday
because of course it was, fuck me.
i noticed what i did
and blood ran cold
i felt that shit feeling in my back
the server came back, some vms also did, but most were missing
shit was just gone to the ether idk
i just took an uber back home
while i was Omw (can you believe the mac tried to make that the fancy text replacement shit, hate those features) home and i saw the whatsapp blew up
they blamed the network guys, so i won some time, yeah fuck me i acted like "oh shit what happened im just a tiger lets go shopping"
most of the shit was down
and so it was, for like a some days
i made it to the office
boss was like wtf
and i was like the server is fucked yo
whats your plan
i was gonna be like, idk i quit fuck it
that was my though process the whole week i was fixing that.
so the thing is that the previous guy left the db of that webui running in a backup
so i reinstaled ovirt (remembered the name) and tried to restore that backup
error
error
invalid file
fuck me it was broken, i went months back in backups, yeah corrupt, it was just making a weird broken tar file, useless
so i was like idk what to do, but i attached the iscsi disks again, and i saw files appear on the disk explorer tyhing, no vms, no info, no nothing, but i started creating vms again, and bam, got one back running, i set up networking, vlans, other shit, got the server back
i did that for all the disk i found
create test check what it was
mostly it was fine
took some days, but got that shit running back again, and while i was doing that i was migrating VMs to the new fancy node
one day i was moving the last shit
hey one of the wordpress sites is broken, shows data but images are lost
yeah they were
gone, gonesky, missing, nada, ni una wea
backups?
lol
the backups had the database but no files for wordpress
fuck
yeah we dont have those files anymore
backups are useless
the files probably were on a snapshot of that vm
and that is probably lost forever idk
so what do we do? what's your plan?
i was like: idk, im not the one that gonna take the hit for it, it's gonna be you so fuck it we roll i guess, if they have the files we can restore them but if not well we are fucked
i was done at that point, i wwas gonna quit
the "head" boss was an asshole, he though yelling would make shit work faster
are you gonna fire me? i asked and he was like nah
so fuck you, let me work in peace
after this the focus on this work changed for me
i focused on improve, certificate and get the fuck out of there
i walked some months later after that issue, maybe a bit less than a year, a november and i was like, you know, fuck it, i quit, im done
and i left, i had friends there, from before
support and telephony there was amazing tbh
i got my share and the lifetime experience working with analog phones, wiring, resolving stuff that was phisical work, running around, drilling shit, moving computers
the last bits of real computering work before it all went to shit
the morale of this history is that bosses can be assholes and you can get them fired
the morale is also that computer world sucks now, there is almost nothing good about it anymore, is just rotten,dead husk of what used to be
i pray to that last netware server running hidden behind a wall keeping a contability system still working, last remainings of that era
the dead phone system replaced by a cloud saas solution, pray for you little Nortel system, little panasonic system
computers are dead and so is working with them
fuck that
kiwa.