Rockerduck Maintenance | ExchangeDefender [ Skip to content ](#main) [ ![ExchangeDefender](/assets/images/xd_logo_blue.png) ](/) Email Security ![Platform Overview](/assets/images/m-xd-pro.png)### ExchangeDefender PRO Enterprise-grade email security made simple for every business. [Sign Up](/sign-up) [Live Demo](/demo) [Email Security Stops spam, malware & phishing in real time. ](/email-security) [Encryption Secure your emails, attachments & sensitive data. ](/corporate-email-encryption) [Web File Sharing Easy, safe cloud file sharing with access control. ](/web-file-server) [Live Archive Locked-in email backup & continuity during outages. ](/livearchive-backup) [Emergency Inbox Keep email flowing without interruption if your provider fails. ](/inbox) [Spearphish Train your team with phishing simulations & risk tracking. ](/spearphishing) Managed Email ### Enterprise-Grade Managed Email Reliable, secure, and fully managed email operations for modern organizations. ![ExchangeDefender](/assets/images/managed_email_menu.png) [Microsoft365/Google Email Management Expert management and security for Microsoft 365 and Google Workspace email. ](/managed-email-operations) [AI Advisory Secure, practical AI consulting & implementation for SMBs. ](/management-ai) Company ### ExchangeDefender: 29+ Years of Trust We are a cybersecurity company focused on business email protection, with 29+ years of experience keeping organizations secure, connected, and resilient. ![ExchangeDefender](/assets/images/careers_1.png) [ About Us Learn our purpose, values, and the mission behind every decision. ](/about-us) [ Blog News, product updates and email security insights. ](/blog) [ Capabilities Discover the tech and expertise that keep your email secure around the clock ](/capabilities) [ Careers Be part of a team using modern tech to solve real threats and ship great work. ](/careers) [Partners](/partners) [Docs](/docs) [Contact Us](/contact) Talk to a Human Open menu Menu ---- Email Security [Email Security](/email-security) [Encryption](/corporate-email-encryption) [Web File Sharing](/web-file-server) [Live Archive](/livearchive-backup) [Inbox](/inbox) [Spearphish](/spearphishing) Managed Email [Microsoft365/Google Email Management](/managed-email-operations) [AI Advisory](/management-ai) Company [About Us](/about-us) [Blog](/blog) [Capabilities](/capabilities) [Careers](/careers) [Partners](/partners) [Docs](/docs) [Contact Us](/contact) Talk to a Human February 13, 2012 · [ExchangeDefender](https://www-preview.exchangedefender.com/blog?category=exchangedefender) Rockerduck Maintenance ====================== On February 9th – February 11th 2012 the ExchangeDefender staff performed maintenance on the Rockerduck cluster in which two separate individual ‘outages’ affected client access on the late half of the evenings and early half of the following morning on February 9th -10th and February 10th-11th . [![under-construction_l9wi2](http://www.exchangedefender.com/blog/media/827e76fb05e8_8705/under-construction_l9wi2_thumb.jpg)](http://www.exchangedefender.com/blog/media/827e76fb05e8_8705/under-construction_l9wi2.jpg)On the eve of Feb. 9th 2012 (~11:30 PM Eastern) we began upgrades on a failed/failing VPN device that is used to connect ROCKERDUCK:DAL and ROCKERDUCK:LA active directory and internal communication between sites. During the upgrade we began to notice random network related events in which communication seemed saturated and sluggish and randomly affected across the entire network. After various attempts (and configurations) to bring the new VPN router online we determined that the new VPN device was occasionally malfunctioning and flooding the network with ‘dead packets’. Unfortunately the massive flood of packets from the VPN device caused the Database Availability Group (DAG) on ROCKERDUCK to lose communication between nodes and eventually lose quorum. Once quorum was lost between nodes all databases between both sites were automatically dismounted as the DAG was considered unhealthy to Exchange. For the next few hours we worked to restore service to RD clients by replacing the failed VPN routers with our backup VPNs (new vendor) and restoring communication with Los Angeles. After communication was re-established clients were able to access their mailboxes. This outage affected all clients and lasted between the hours of midnight and roughly 3:15 AM. On the eve of Feb. 10th (~10:30 PM Eastern) we began work to finalize the VPN communication by consolidating both VPN devices in California to the one backup vendor VPN device. The reason we elected to replace the ‘working’ VPN device in California was due to the fear of the abnormal workings of the similar VPN device in Dallas. As part of our protocol to ‘down’ a data center in Exchange hosting we paused SMTP services on Rockerduck. After replacing the VPN device in California we resumed all services (including SMTP) and mail resumed normal flow. Around 5:30 AM Eastern we started to receive alerts about back pressured queues in Rockerduck which would amount to delivery delays. Upon investigation it was discovered that the issue was mail delivery between the EDGE server network and the HUB server network on RD. After two hours of investigating the issue internally (and opening a case with Microsoft) we were able to determine that our course of action would be reapplying the SP2 update to the edge networks. Once SP2 was reapplied to all EDGE nodes mail delivery returned on ROCKERDUCK by 9:15 AM Eastern. Finally there were about 5% of users who were left in a disconnected state through Outlook but had service through OWA (and some through active sync) between Saturday and Sunday as the database their mailboxes were housed was moved to Los Angeles for the content index database in Dallas to rebuild for RDDB9. Service was restored to these users by noon Eastern. Travis Sheldon VP, Network Operations, ExchangeDefender (877) 546-0316 x757 [ Previous post This Week in Development… February 10, 2012 ](https://www-preview.exchangedefender.com/blog/2012/02/this-week-in-development) [ Next post Happy Valentine’s Day! February 14, 2012 ](https://www-preview.exchangedefender.com/blog/2012/02/happy-valentines-day) Related Posts ------------- [ ![](https://storage.services.exchangedefender.network/exchangedefender-blog/blog/2026/04/xd10.png) 2 Apr ](https://www-preview.exchangedefender.com/blog/2026/04/announcing-exchangedefender-10)### [Announcing ExchangeDefender 10](https://www-preview.exchangedefender.com/blog/2026/04/announcing-exchangedefender-10) [Read more: Announcing ExchangeDefender 10](https://www-preview.exchangedefender.com/blog/2026/04/announcing-exchangedefender-10) [ ![](https://storage.services.exchangedefender.network/exchangedefender-blog/blog/2025/09/Top-7-IT-Solutions-Every-Small-Business-Needs-in-2025.jpg) 30 Sep ](https://www-preview.exchangedefender.com/blog/2025/09/top-7-it-solutions-every-small-business-needs-in-2025)### [Top 7 IT Solutions Every Small Business Needs in 2025](https://www-preview.exchangedefender.com/blog/2025/09/top-7-it-solutions-every-small-business-needs-in-2025) [Read more: Top 7 IT Solutions Every Small Business Needs in 2025](https://www-preview.exchangedefender.com/blog/2025/09/top-7-it-solutions-every-small-business-needs-in-2025) [ ![](https://storage.services.exchangedefender.network/exchangedefender-blog/blog/2025/09/Website-Active-Image-10.jpg) 23 Sep ](https://www-preview.exchangedefender.com/blog/2025/09/exchangedefender-rolls-out-advanced-reject-policies-for-safer-inboxes)### [ExchangeDefender Rolls Out Advanced 'Reject Policies' for Safer Inboxes](https://www-preview.exchangedefender.com/blog/2025/09/exchangedefender-rolls-out-advanced-reject-policies-for-safer-inboxes) [Read more: ExchangeDefender Rolls Out Advanced 'Reject Policies' for Safer Inboxes](https://www-preview.exchangedefender.com/blog/2025/09/exchangedefender-rolls-out-advanced-reject-policies-for-safer-inboxes) Search the blog Search Recent Posts ------------ - [ExchangeDefender New UI is Live](https://www-preview.exchangedefender.com/blog/2026/05/exchangedefender-new-ui-is-live) - [Announcing ExchangeDefender 10](https://www-preview.exchangedefender.com/blog/2026/04/announcing-exchangedefender-10) - [Merry Christmas](https://www-preview.exchangedefender.com/blog/2025/12/merry-christmas) - [Why IoT Devices Are a Hidden Security Risk for Your Business](https://www-preview.exchangedefender.com/blog/2025/11/why-iot-devices-are-a-hidden-security-risk-for-your-business) - [Top 5 IT Problems Small Businesses Face in 2025 — and How to Fix Them](https://www-preview.exchangedefender.com/blog/2025/10/top-5-it-problems-small-businesses-face-in-2025-and-how-to-fix-them) Categories ---------- - [Archiving](https://www-preview.exchangedefender.com/blog?category=archiving) (21) - [Business Continuity](https://www-preview.exchangedefender.com/blog?category=business-continuity) (13) - [Compliance](https://www-preview.exchangedefender.com/blog?category=compliance) (8) - [ExchangeDefender](https://www-preview.exchangedefender.com/blog?category=exchangedefender) (349) - [General](https://www-preview.exchangedefender.com/blog?category=uncategorized) (590) - [Hosted Services](https://www-preview.exchangedefender.com/blog?category=hosted-services) (86) - [Industry News](https://www-preview.exchangedefender.com/blog?category=industrynews) (29) - [Marketing](https://www-preview.exchangedefender.com/blog?category=marketing) (18) - [Phishing](https://www-preview.exchangedefender.com/blog?category=phishing) (5) - [PRO TIPS](https://www-preview.exchangedefender.com/blog?category=pro-tips) (44) - [Product Features](https://www-preview.exchangedefender.com/blog?category=product-features) (54) - [Security](https://www-preview.exchangedefender.com/blog?category=security) (65) - [Shockey Monkey](https://www-preview.exchangedefender.com/blog?category=shockey-monkey) (22) - [Spamfilter](https://www-preview.exchangedefender.com/blog?category=spamfilter) (5) - [Support](https://www-preview.exchangedefender.com/blog?category=support) (45) - [Webinars](https://www-preview.exchangedefender.com/blog?category=webinars) (44) Solutions --------- - [Stop Spam & Malware](/email-security) - [Hands-Free Email Backup](/livearchive-backup) - [Instant Email Recovery](/inbox) - [Send Private Emails](/corporate-email-encryption) - [Train Against Email Threats](/spearphishing) - [Share Files Safely](/web-file-server) Company ------- - [About ExchangeDefender](/about-us) - [Compliance & Legal](/our-compliance) - [Become a Partner](/partners) - [Support Portal Login](https://support.exchangedefender.com/) - [Microsoft 365 Email Management](/managed-email-operations) - [AI Advisory Service](/management-ai) ![ExchangeDefender](/assets/images/dark-xd-logo.png)USA: +1 (407) 465-6800 United Kingdom: 0800 8620149 Australia: +61 0390010641 Fax: +1 (954) 839-8737 [ ](https://www.facebook.com/ExchangeDefender) [ ](https://www.twitter.com/XD_Defender) [ ](https://www.linkedin.com/company/exchangdefender) [ ](/blog) © 2026 Own Web Now Corp - All rights reserved </> with ❤️ in Orlando [SLA](/sla) [TOS](/tos) [PRIVACY](/privacy-end-users) [AUP](/aup) We're Here to Help 👋 -------------------- Tired of AI 🤖 and hold music? Request a call or text (sms/telco charges may apply) and someone with a brain and a pulse will get in touch with you promptly. Close modal Website Name Email Address Company Phone Preferred Contact Method Text me Call me Call Date Call Time Time Zone Eastern Central Mountain Pacific London Paris Madrid Berlin Tokyo Dubai Sydney Request Human Contact Submitting... Close Start Your Free Year -------------------- Enter your details and we will set up your free year of email protection. Close modal Website Full Name Email Continue Submitting... Close Try ExchangeDefender for Free 💰 ------------------------------- We look forward to helping you secure your email. We offer a free 30-day no-risk trial, just fill out the form below and one of our onboarding techs will get you started. If you'd just like to take a look at the demo, click here for a [live demo site](/demo). Close modal Website Name Email Address Company Phone Preferred Contact Method Text me Call me Call Date Call Time Time Zone Eastern Central Mountain Pacific London Paris Madrid Berlin Tokyo Dubai Sydney 💰 Get your free month Submitting... 🏃 Explore a Live Demo Submitting... Close Train Your Team Before the Hackers Do 👋 --------------------------------------- ExchangeDefender Spearphish is our new phishing simulation and training tool, designed to help your team spot scams before they click. Want to be first in line? Request early access or get a personal demo — just drop your name and email below. Close modal Website Name Email Address Company Phone Preferred Contact Method Text me Call me Call Date Call Time Time Zone Eastern Central Mountain Pacific London Paris Madrid Berlin Tokyo Dubai Sydney Request Early Access Submitting... Close