At 14:16 CST, DNS Records that had been published to Cloudflare for several months were mysteriously removed, as indicated by the logs provided by Cloudflare.
At 14:17:22 CST, Voxtelesys monitoring systems detected Origin Errors being reported by Cloudflare across various API endpoints and portal.voxtelesys.net.
At 14:33 CST, Voxtelesys engineering pushed a mass DNS record change to the voxtelesys.net domain.
By 14:36 CST, the DNS change had propagated to mainstream public DNS resolvers.
This entire event lasted 20 minutes.
We have engaged Cloudflare's Enterprise Support and are awaiting a much-needed explanation to the logs provided to us.
We apologize for the inconvenience and will continue to monitor closely for any further issues.
Posted Feb 27, 2024 - 16:13 CST
Monitoring
A temporary fix has been implemented. If you are still seeing issues, please flush any DNS resolver cache. If the issue persists please contact support@voxtelesys.com.
Posted Feb 27, 2024 - 14:52 CST
Investigating
Voxtelesys NOC has received reports of intermittent issues with SMS and reaching the portal. Voxtelesys network engineering is currently investigating.
Posted Feb 27, 2024 - 14:31 CST
This incident affected: Messaging (SMS Messaging, MMS Messaging) and Portal.