So our ERP system crashed tonight so we had to do a DR event starting at 6 PM. Now there is some “crowdstrike” issue going on affecting end user laptops. So this is going to be fantastic. Still on the bridge call as I type this. I was up at 4 this morning because I couldnt sleep and I can hardly keep my eyes open now.
You’ll get that on those big jobs.
You must be doing as much work on your bridge as I am on mine 
Sounds like there were computer outages world wide last night?
Crowdstrike like you said but affected airlines and banks worldwide.
More Chinese hackers?
Haven’t seen the details but it sounded like a mess.
Sounds like there were computer outages world wide last night?
More Chinese hackers?
Haven’t seen the details but it sounded like a mess.
Crowdstrike is antivirus basically and they pushed an update that makes Windows computer blue screen on boot. That update is pulled but every PC needs to be manually fixed, locally, because they don’t boot anymore for network access or to be remotely accessible. It’s going to take a loooong time to fully resolve.
Congrats to the people who will get today off though, there’s going to be a bunch
This post must be directed at those in IT, because I have zero idea what “ERP” or “DR event” means. Call me a “DERRP” if you want.
Oh, wait- did u try shutting it off and turning it back on again? 
I feel for you. It has been a long time since I have had to deal directly with DRs. Since I am now a Data Engineer, other people are responsible for keeping systems running.
I primarily use SaaS tools now. In the past month, our database has been unavailable 4 times and our development tool once. All I could do is wait for updates from the vendor.
I hope you are sleeping now and have been given the rest of the day off.
This post must be directed at those in IT, because I have zero idea what “ERP” or “DR event” means. Call me a “DERRP” if you want.
Oh, wait- did u try shutting it off and turning it back on again?
You don’t need to be in IT to use an ERP system. It’s basically a system that handles all aspects of a business; from HR, payroll, sales, work orders, clock-in/out for shift workers ect.
They are supposed to make life easier.
See this is why I hesitate to tell people I do IT as part of my job… I have no idea what you guys are saying!! 
The kind of “IT” I do at my work is super basic but to my coworkers I’m a magical computer guru, and I’ll accept their admiration anyway. 
DR and HA…gotta be the worst or close.
Disaster Recovery – When you find out all the changes made since the last DR drill, aren’t there.
High Availability – When you find out all the changes made since the last HA drill, aren’t there.
Don’t think I’ve ever seen one go smoothly.
We have no idea what we do or do not have access to right now at work. So every person with a request we get to tell them we may or may not be able to help. Most of our inquiries are in federal databases and they are down. Its a deep breaths day today.
CaptMusky,
Your not alone. We are also having what we call Code Events (DR) this morning related to the IT Outage. We have an ERP release that is supposed to be deployed tonight so it is going to be a long day.
There is a global Microsoft issue affecting multiple operations. Flights at MSP have been affected since 5am.
Apparently it’s also affecting exchanges, banks, and hospitals too.
It sucks for everyone. Having worked in SaaS for a long time I’ve dealt with my fair share of outages. At my last place our application went down and we didn’t fail over for 8 hours. Lost a bunch of customer data and transactions in that span. Our dev and product teams worked a week straight to fix it.
I handled the customer side of our business so my team got our asses chewed the whole time. It was an issue with a piece of hardware at our data center that could apparently never fail…until it did. Nobodies fault but we all paid for it.
DR and HA…gotta be the worst or close.
Disaster Recovery – When you find out all the changes made since the last DR drill, aren’t there.
High Availability – When you find out all the changes made since the last HA drill, aren’t there.
Don’t think I’ve ever seen one go smoothly.
So true!
Retirement is so shaweeeeeet!! 
So are MacBook Pros. 
You’ll get that on those big jobs.
That got an actual LOL from me BC!
I got 4 calls and 2 texts telling me about it on my way in…might need to make a tee time!
I was on the bridge call until 2 AM. I was awake for 22 hours yesterday so needless to say I am dragging.
The strange thing for us is that this happened in two waves. First we lost ALL of our systems so we failed over to AWS from Azure which coincidentally we just did a DR test last weekend to move over to. ONce we got that all up and validated around midnight then this crowdstrike thing happened and battling it ever since. Its apparently a simple fix for the blue screen issue but we cannot modify anything on our PCs without calling helpdesk and they actually log in. You are supposed to delete 2 files, but again, we cannot do that without assistance. Ugh. I havent gotten the blue screen, but many have. Normally I am the one with PC issues.
Grounded flights, 911 outages and on and on. How vulnerable are we? I mean our power grid has been hacked, rail lines hacked. I am not saying this is a hack, but this is very scary.
If I was Crowdstrike I would definitely blame this on hackers instead of putting out a bad update. Seems like we need to diversify our operating systems.
Grounded flights
I was flying back from Orlando last night and we had to sit on the tarmac for almost an hour but they said it was because too many planes were in our flight path because of weather. Now hearing about this makes me think it was because of all these issues. So glad I didnt get stuck in Orlando’s airport!
If I was Crowdstrike I would definitely blame this on hackers instead of putting out a bad update. Seems like we need to diversify our operating systems.
Don’t think saying they were hacked would be a better look for a security company.
Don’t think saying they were hacked would be a better look for a security company.
Definitely NOT a good look. I wonder how their stock looks this morning LOL Edit, down 10.5%
Did anyone at Crowdstrike try turning off and back on again? Maybe call Al Gore and see if he can reset the Wifi router? 
Yeah, from I have read rebooting is when the bluescreen of death happens and then you are really stuck.
Yeah, from I have read rebooting is when the bluescreen of death happens and then you are really stuck.
That’s when you need to do it again, I thought you worked in IT?!? Sorry prolly shouldn’t joke with a guy on his 18th red bull and coffee!
All good BigWerm! I knew you were joking, but my sense of humor hasnt woke up yet LOL.
I think they should bomb large scale hackers.
I think they should bomb large scale hackers.
Back in 2018 we got hacked with malware because some idiot employee clicked a link in an email and it brought everything down. 30,000 global employees in countless countries and states and we were on our knees. They wanted millions to undo it but my company gave them the middle finger. In the end our division was offline for like 2 or 3 weeks but we lost nothing. Other divisions came up earlier but we are in flyover country with just cornfields so no prioritized.
Well we are back and fully functional. Production has been all weekend or since Friday. QA just came back online because that was still with Azure and not a priority, but now we have to failover to AWS to keep our systems inline with the host system. Going to be an event filled week!
You must be logged in to reply to this topic.
