Plesk Outage
We are currently experiencing an outage affected our Plesk servers.
Affected Servers :
Our engineers are working to restore the services momentarily.
UPDATE :
pl-sql1.hkdns.co.za has now been restored
UPDATE :
pl-2.hkdns.co.za has been restored now.
UPDATE :
All services restored.
Good morning
We will be performing maintenance on our PL-2 plesk server tomorrow evening.
Maintenance Window: 3 - 4 hours
Issue: Server maintenance
Affected Services: Websites and email hosted on the affected server will be inaccessible during the maintenance window.
Hello,
Our Data Centre operator will be conducting planned maintenance at our Samrand Data Centre on Tuesday, 5 December 2023 to service our electrical infrastructure and UPS backup power system.
During the maintenance window, the distribution board on our B-Feed will be shut down
Affected Services :
Our equipment is connected to both the A and B power feeds simultaneously and should not be affected.
Time frame :
Tuesday, 5 December 2023 from 18:00 until 00:00 (Duration 6 Hours)
Dear Hostking Customers
We want to extend our warmest wishes and express our gratitude for your continued loyalty and support. As the festive season approaches our support team will operate with a skeleton staff as of the 22nd December until the 3rd January 2024.
While we remain committed to assisting our customers, please anticipate potential delays.
Additionally, the cut-off date for web design orders will be the 12th December. The web design team will be temporarily closed from the 15th December and resume operations on the 8th January 2024.
We appreciate your understanding and cooperation during this festive period. Please join our live chat if you have any urgent inquiries or require assistance.
Yours in support
Hostking Staff
It seems we experienced a network issue to some servers in one of our cabinets in our JHB Data Center.
The issue seemed to stem from maintenance being performed by our Data Centre technicians where a urgent Power Feed needed to be replaced.
The issue has been resolved however some noticeable downtime occurred for some customers on certain servers hosted in the cabinet for about 1 - 3+ hours.
This should not reoccur as the feed is new and would not require any changes for 5+ years.
We apologize for the inconvenience caused
We have reported our Plesk SQL server pl-sql2.hkdns.co.za that is currently un-responsive.
Our engineers are working on the problem and will resolve access as soon as possible.
We are investigating why the cyclops.hkdns server is down
Network went down at +- 14:45 Today
Dear Resellers
We will be migrating some servers to a new server with newer hardware and better performance.
We have done quite a few on our normal shared hosting and found massive stability and increased performance.
We have scheduled this change over for tomorrow evening.
Note the IP address of the server will change to 156.38.171.133.
NB: Note customers using our NS server records or the customer NS records we provided will not be affected. But those using Cloudflare or any third party DNS may have to update their IP to the new one.
We are currently experiencing an outage on our servers this morning,which is currently being looked into.
Services should be restored within the next 20-30 minutes.
We do apologize for the inconvenience caused.
Dear Customer
All Customers on PL-1 Server will be migrated to a new High Performance Server.
We have noticed some instability in some Plesk servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains, please contact our support team for the appropriate DNS records.
The servers new IP address will be "156.38.171.132"
If you have any further queries feel free to contact our support team.
We are experiencing network issues with our Data Centre. They have engineers attending to the issue and will provide us with feedback once resolved
Outage :
The network issues are affecting packet loss to servers hosted in the Data Centre
Symptoms:
Customers may experience slow response times when accessing hosting services hosted in affected data centre.
Dear Customer
All Customers (Non-Resellers) on CP-16 Server will be migrated to a new High Performance Server with 64 Cores and NVME Disks.
We have noticed some instability in some servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains, please contact our support team for the appropriate DNS records.
The servers new IP address will be "142.132.152.171"
If you have any further queries feel free to contact our support team.
Dear Customer
All Customers (Non-Resellers) on CP-15 Server will be migrated to a new High Performance Server with 64 Cores and NVME Disks.
We have noticed some instability in some servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains, please contact our support team for the appropriate DNS records.
The servers new IP address will be "142.132.152.171"
If you have any further queries feel free to contact our support team.
Dear Customer
All Customers on DA-9 Server will be migrated to a new High Performance Server with 64 Cores and NVME Disks.
We have noticed some instability in some DA servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains, please contact our support team for the appropriate DNS records.
The servers new IP address will be "168.119.66.98"
If you have any further queries feel free to contact our support team.
Dear Customer
All Customers on DA-8 Server will be migrated to a new High Performance Server with 64 Cores and NVME Disks.
We have noticed some instability in some DA servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains we will have to communicate with you and manually migrate your accounts over.
If you have any further queries feel free to contact our support team.
We are investigating an issue affecting new registrations and updating/transfers affecting CO.ZA domains, we are in contact with ZACR who are assisting us with the matter.
We will provide feedback once resolved on their end.
------------------
Issue has been resolved. Seems the issue was with OpenSSL and PHP on the server which was not communicating correctly with the API of ZACR.
Dear Customer
All Customers on DA-6 Server will be migrated to a new High Performance Server with 64 Cores and NVME Disks.
We have noticed some instability in some DA servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains we will have to communicate with you and manually migrate your accounts over.
If you have any further queries feel free to contact our support team.
Good day
We are currently experiencing an outage affecting the "One Click Login to Direct Admin" via the Client Area. We have escalated this issue to our upstream vendors for further assistance. The current solution is to login directly to Direct Admin via the control panel url. We are able to resend the login details should you need.
Affected servers :
storm.hkdns.host
Interim solution :
Log in via https://storm.hkdns.host:2222/ with your appropriate username and password.
Dear Customer
All Customers on DA-7 Server will be migrated to a new High Performance Server with 64 Cores and NVME Disks.
We have noticed some instability in some DA servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains we will have to communicate with you and manually migrate your accounts over.
If you have any further queries feel free to contact our support team.
Dear Customer
All Customers on DA-4 Server will be migrated to a new High Performance Server with 64 Cores and NVME Disks.
We have noticed some instability in some DA servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains we will have to communicate with you and manually migrate your accounts over.
If you have any further queries feel free to contact our support team.
Dear Customer
All Customers on DA-5 Server will be migrated to a new High Performance Server with 64 Cores and NVME Disks.
We have noticed some instability in some DA servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains we will have to communicate with you and manually migrate your accounts over.
If you have any further queries feel free to contact our support team.
Dear Customer
All Customers on DA-3 Server will be migrated to a new High Performance Server with 64 Cores and NVME Disks.
We have noticed some instability in some DA servers and hence we will be migrating certain servers to better high performance servers.
This should not affect 99% of customers and they will hardly notice but for the 1% of customers that use Cloudflare or custom NS records on their domains we will have to communicate with you and manually migrate your accounts over.
If you have any further queries feel free to contact our support team.
We are currently migrating customers from da-1 and da-2 servers to our new storm EPYC+NVME servers.
This should drastically improve performance on all websites aswell as improve stability.
99% customers will not have any issues but for that use Cloudflare, EZOIC or any external Name Servers you may need to update the IP address to the new server 157.90.129.247
If you have any further queries regarding this feel free to message our support team.
Good Day
Upgrade of network switches on 16 March 2023
Good news: As part of our network infrastructure upgrade, we'll be upgrading network switches at our Cape Town Data Centre on 16 March 2023. This will increase the flexibility and capacity of our network, while simplifying maintenance and improving redundancy.
When:
Thursday, 16 March 2023 from 21:00 - 03:00 SAST
Server name:
We have emailed the owners of the servers that will be affected
Impact:
The total network downtime is expected to be a couple of seconds.
Action required:
As network connectivity may be affected, we suggest that you don't run any scheduled backups during this maintenance period.
We will be migrating our website to a more powerful server this evening at 1am gmt+2
This will result in new orders remaining in pending state until we have completed the migration to the new server
Please bare with us will we complete the migration.
Thank You for your understanding.
We have completed our investigation and have found the following:
The main /home directory on the server has become corrupted.
We are hence restoring a full backup of the disk onto a different server but at the same time currently running a xfs_repair on the partition.
Please accept our apology for the interruption in service.
Our System Administrator is currently currently attending to the issue.
--------------------
95% of accounts are fully restored. The Last 5% is taking some time as they are larger accounts.
-----------------------------
The outage has been resolved, we are performing audits on all restores.
Should you identify an issues, please contact our support team.
We received a major attack to our servers from multiple IP address ranges from various countries.
We are still monitoring it, and adding restrictions to those countries attacking our servers to keep the attack as low as possible and services up and running.
Our system admin team is monitoring the situation but all services should be restored and working.
Hello
Emergency Maintenance at our Cape Town Data Centre
We will be conducting emergency repairs on our generator at our Cape Town Data Centre on Friday, 23 December 2022.
Why is maintenance required?
On 22 December, the generator at our Cape Town Data Centre began overheating. To ensure uninterrupted operation, emergency maintenance is required.
When:
Friday, 23 December between 08:30 - 22:00 SAST (the duration of the maintenance will be 8 hours during this window).
Impact of load-shedding
Maintenance has been scheduled according to the current load-shedding schedule of Stage 3. Should the load-shedding stage increase, disruption to power is a possibility. We wish to assure you that we will do our utmost to prevent an interruption of service during the maintenance period.
Action required:
If the load-shedding stage is increased to coincide with the maintenance window, Self-Managed Server customers are advised to shut down their servers to avoid data corruption. In the event that the load-shedding stage is changed to Stage 4, load-shedding at Great Westerford will take place from 14:00 - 16:30 SAST. We will monitor the schedule and notify customers accordingly.
We have noticed that the server processes and tasks is slow this morning
We are investigating the issue and the server performance should return to normal when it is resolved.
Update :
The server had corrupted Windows files after an automatic update and we have patched them to restore server performance.
We apologize for the inconvenience caused.
There is an issue with card processing currently, it is being investigated, credit card and cheque card payments are being affected.
Please use the Instant EFT option in Flutterwave Instead or contact us on billing@hostking.co.za to assist.
An issue was reported around 08:30 this morning for our Reseller server rs-3-da.
We found a corruption on the root partition which caused intermittent issues with sites loading / email.
We noticed an outage with our cp-15 cPanel server this morning and reported it to our data-center sysadmin.
The system seems to have become read-only and we are running a full system scan to resolve the situation.
The server should be resolved shortly.
PAYFAST ALERT
Capitec Internet Banking down
New incident: Monitoring Capitec Internet banking is down or highly unresponsive (https://downdetector.co.za/status/capitec/).
This will affect any clients wishing to pay via Instant EFT from Capitec Bank. All other payment methods are unaffected.
We will monitor and advise once regular operation resumes. Time posted Oct 21, 09:49 SAST
Litespeed error reported on cp-15.hkdns.co.za cPanel
Corruption occurred on the kernel for the server.
The kernel was re-installed.
Resolved
Reported DDoS attack 10:55
Our servers was under attack today from distributed denial of service attack.
Our sys-admins managed to contain the attack within 40 minutes using the tweaks they did before which seems to have blocked it better this time.
Time of DDoS : 40 minutes.
We will be making a few more tweaks to the servers and we think this may lower it even more, and hope that eventually make the hackers ddosing some of our servers give up and go attack someone else.
We noticed massive packet loss to all services at 11:58AM and found that all servers were being attacked with 300M inbound traffic.
We investigated and found that it was a major DDOS attack and immediately reported the issue to the Data Centre.
They investigated, but took some time as it was a major attack across all our services.
They are investigating how this occurred and will provide feedback later this afternoon.
We apologize any inconvenience caused and will post updates received on this notice as we receive them.
------------------------
Update : Seems the IP addresses causing the ddos was from many countries but mainly from China and Russia. We created rules to make traffic from those countries be checked on higher levels.
Good day
We are currently experiencing issues affecting websites on our CP-16 server. Our Systems Admins are investigating and will have services restored momentarily.
Issue: There was metadata corruption on the disk
Fix : We ran an xfs_repair on the /home directory
Please feel free to contact us should you need any assistance
Its come to our attention that invoices generated for today were duplicated and some customers received duplicated versions of it.
Our developers have attended to the issue and it should be resolved though we are continuing to monitor the situation.
We apologize for any inconvenience caused.
We are working on resolving issues relating to logging into the server.
We do apologize for the inconvenience caused resulting from the downtime.
--------------------
Update :
It seems server and services were running fine but interface lost its permissions on the server. We reset the permissions after investigating and all is working once again.
We apologize for the inconvenience caused.
We are currently attending to an outage on our PL-4 server.
We do apologize for any inconveience caused resulting from the downtime.
Good day
We are currently experiencing issues affecting websites on our DA-1 server. Our sysadmisn are investigating and will have services restore momentarily.
Please feel free to contact us should you need any assistance
Please note that we are investigating intermitted connectivity on one of our Direct Admin Reseller Servers
Our onsite engineers are currently performing emergency maintenance and are working to restore service urgently.
Please accept our apology for the interruption in service.
Affected Services : Websites and email.
We have experienced an outage on our da-4 server where some databases were not working correctly.
The issue has been resolved by reinstalling db_gobernor on the server,which toke 36 minutes but trust the problem has been resolved and the sites that were affected should be loading once more.
we do apologise for the interruption in services experienced.
Good morning
Our cPanel servers are experiencing issues with Softaculous due to an update failure. We have contacted the vendor and have been informed this is due to maintenance being performed on their servers. We will restore services once the Softaculous has completed their maintenance.
We do apologize for the inconvenience this may have caused. Please note that normal access to various dashboard are not affected only access via Softaculous.
Please note that we are investigating intermitted connectivity on one of our Direct Admin
Our onsite engineers are currently performing emergency maintenance and are working to restore service urgently.
Please accept our apology for the interruption in service.
Affected Services : Websites and email.
Please note that we are investigating intermitted connectivity on one of our Direct Admin
Our onsite engineers are currently performing emergency maintenance and are working to restore service urgently.
Please accept our apology for the interruption in service.
Affected Services : Websites and email.
Hostking apologize for the interruption of mail service on our pl-2.hkdns.co.za Plesk server.
We are busy re-installing and re-configuring the mail server for our pl-2 Plesk server as we became aware of email accounts not working on the server.
We do apologize for the recent network outage which caused temporary interruptions in services.
We have increased the network throughput on our servers to accommodate for the increase in traffic.
Due to performance issues affecting this server. Our systems administrators will be migrating the server to a new node this evening.
Maintenance Window: 3 hours
Issue: Server migration
Affected Services: Websites and email hosted on the affected server will be inaccessible during the maintenance window.
We are currently investigatiing an issue with one of our nodes.
Our onsite engineers are currently performing emergency maintenance and are working to restore service urgently.
Please accept our apology for the interruption in service.
Please note that we are investigating intermitted connectivity on one of our cPanel servers.
Our onsite engineers are currently performing emergency maintenance and are working to restore service urgently.
Please accept our apology for the interruption in service.
Affected Services : Websites and email.
We have recently experienced high load on our PVE-3 VPS node. We are investigating the cause and the load should be dropping shortly.
Payfast is currently experiencing an outage,which is preventing payments being processed.
We have temporary enabled "sagepay" for instant payment methods.
We do apologize for the inconveinence caused and will update you as soon as this has been resolved.
We are currently experiencing technical difficulties with our live chat system and are working on resolving it as fast as possible.
We will be performancing maintenance on our phone system,which will be temporary unvailable.
We do apologiize for any inconvenience this may cause.
Kindly note we will still be available via live chat through our website.
Please note that Hostking has experienced a major DDOS attack in the last 24 hours.
The majority of the DDOS attacks originated from these regions, China, Ukraine and Russia.
We have worked feverishly through the night to restore services.
We were able to mitigate the effects of the DDOS by tightening the rules on our firewalls for the above-mentioned regions, we understand this may not be ideal for customers that may deal with these countries, we can assure you the changes will not affect email.
However, should you experience any difficulties, we ask that you please contact us directly.We sincerely apologize for the effect this has caused for many of our customers that were affected and can assure you that we strive to ensure a reliable uptime for your hosting services.
Yours in support
Hostking
Good day
Please note that Hostking has experienced a major DDOS attack in the last 24 hours. The majority of the DDOS attacks originated from these regions, China, Ukraine and Russia.
We have worked feverishly through the night to restore services. We were able to mitigate the effects of the DDOS by tightening the rules on our firewalls for the above-mentioned regions, we understand this may not be ideal for customers that may deal with these countries, we can assure you the changes will not affect email. However, should you experience any difficulties, we ask that you please contact us directly.
We sincerely apologize for the effect this has caused for many of our customers that were affected and can assure you that we strive to ensure a reliable uptime for your hosting services.
Yours in support
Hostking
We noticed that our Pl-2 server has been experiencing high load.
Our systems engineers are currently investigating a possible hard drive read error.
The server is active, however we may need to schedule a chkdsk to correct the errors on the filesystem this evening at 11pm.
We have noticed that SPAMHAUS has listed several of our cPanel servers on their blacklist.
Our server engineers are investigating.
SPAMHAUS has delisted our servers
We do apologize for the interruption in services experienced.
PL-4 experienced and outage today that caused a few sites from displaying an error message 503. We investigated the issue with Plesk support team and it seems to have been related to a Plesk repair script that is run whenever Plesk auto updates.
We corrected the issue by rerunning the script and letting it complete.
All should be resolved but kindly inform us if you experience any further issues.
We will also continue to monitor the situation on this server.
We are currently attending to an unexpected shutdown on the server and are currently running a chkdsk to repair any issues on the disks.
We do apologize for any inconvenience caused
We are currently experiencing an outage affecting Payfast submittions.
Payfast is currently attending to the issue.
We have switched to Sage Pay temporary to allow payments to be made.
We do apologize for the inconvenience caused.
We are currently experiencing an outage affecting our live chat and are currently investigating the cause.
We reported an outage on our pve-2 server node at 18:50 due a DDOS attack.
Admin is monitoring the services to ensure continued operation.
Service resolved at 19:45 and we continue to monitor.
Update:
The issue was related to the LVM partitioning encountering bad sectors. Xneelo replaced the faulty disk and restored the service.
Outage Status: Resolved.
We are experiencing intermittent network connectivity at our data centre.
The following servers have been affected:
pl-sql2.hkdns.co.za
cp-5.hkdns.co.za
Our engineers are currently investigating and should have the matter resolved shortly.
Letsencrypt have removed support for DST Root CA X3 Certificate. This unfortunately causes issues with some older devices and some other devices using the older SSL root cert. Please regenerate the SSL certificate via your control panel if you find this to be the cause. This is happening on many platforms as per Plesk and cPanel. They have provided links to fix the issue. We have hence updated as they stated to change to cPanel SSLs rather than Letsencrypt for cPanel Servers. However for Plesk it needs to be done manually. Links to the issue is described on their posts below on the following links:
LetsEncrypt:
https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
Plesk:
https://support.plesk.com/hc/en-us/articles/4407400262674
cPanel:
https://support.cpanel.net/hc/en-us/articles/4409770365335
Good Day
We experienced intermittent networking issues in one of our cabinets in the DC which affected a few servers, cp-1, cp-15, cp-5 , pl-1, pl-4 and pl-2 and our website
The issue is resolved
However, we have to perform maintenance at 1am though this should not cause any downtime during that window
Good day
There has been an outage on pve-1-cpt, we currently have our technicians attending to the issue.
We have experiencing an outage connecting to ZACR.
Affected services : The outage will affect domain registration and renewals for extension through ZACR.
Our engineers have escalated this matter to our vendor and are currently awaiting feedback.
We are currently experiencing load on some of our VPS servers and are currently in the process of investigating the cause. Some users might experience a slower connection until resolved.
Hello Clients
Please be advised, we are currently having an emergency maintenance on the server
|
||
|
We are currently experiencing a network outage related to our core switch in the CPT data Centre. Our Network Engineers are busy investigating. We currently do not have an ETA for resolution, but our Network Status page will be updated with new information as soon as we have more info.
Affected servers : VPS managed / unmanaged
Jul 21, 08:28 SAST
This will result in extended outages to the EPP registration services. This means that processing creates, updates, transfers and other domain operations will not be available for the following zones: co.za, web.za, org.za, net.za, .capetown, .durban and .joburg. Please note that the Whois/RDAP and DNS registry services will remain unaffected. During this maintenance window the ZACR Portal will be offline and the co.za legacy email based registration service will also experience delays
Dear Clients
We would like to apologize as the live chat system is down at the moment, we are still waiting on Tawk.to agents to resolve this
However, the ticket system is working or you can send an email to support@hostking.co.za
We apologize for the inconvenience caused
We are currently experiencing some DDos attacks on our PL-3 and PL-4 servers.
Our engineers are busy investigating and resolving.
Good day
Our Server CP-9 Experienced a massive ddos attack this morning but the issue has been resolved by one of our technicians.
Hostking Team
We received reports of mysql databases having issues. We also noticed some monitoring spikes on mysql.
The disk for mysql had some issues - we replaced the disk and resynced the data and all the mysql databases came up again.
We apologize for the inconvenience caused.
Today, mail administrators, organizations, and ISPs worldwide suddenly found that their outgoing mail was being rejected as it reported as being listed in the blacklist at bl.spamcop.net.
This is due to Spamcop a wellknown RBL service web hosts around the world use to send email are having a major outage.
We hence disabled their RBLs on our systems until they resolved it.
We initially thought it was DNS related but seems its related to SpamCops servers being down as it was only their service causing issues.
More information can be found on Google or the following links regarding the issue:
https://www.ecommercebytes.com/2021/01/31/seller-alert-your-emails-may-be-blocked-today/
https://www.webhostingtalk.com/showthread.php?t=1836446&p=10289418
We are experiencing disk errors on pl-sql2.hkdns.co.za
Our engineers are attending.
We are currently experiencing outages affecting our Plesk Servers.
Affected Servers : PL-2, PL-3 AND PL-4
Our engineers are working to resolve the issue.
cp-6.hkdns.co.za - cpanel - MySql / Mariadb outage.
Our Engineers are attending the issue and we will update here when the service is restored.
Please be aware that we are currently performing emergency maintenance on all our cPanel servers. We have experiencing increased DDOS activity which resulted in load and disk I/O in our shared hosting environment.
Effect services : Shared hosting cPanel servers,
Expected downtime 10 - 15 mins.
We do apologize for any inconvenience caused.
There seems to be an issue occuring in the CPT datacenter affecting some VPS servers.
Our engineers are attending and it will be resolved shortly.
Hello Clients
This is to inform clients on cp-4 to kindly be patient as we are having email issues at the moment. The emails are stuck in the queue and are slowly getting released.
Our Systems Admins are working their level best to solve this as fast as they can as the script that releases the mails is doing the outmost to get this sorted.
The issue is that as the emails come in and go out they need to be scanned, causing a further delay on the release of emails
We do apologize for the inconvenience caused. We are working our best to find a permanent solution to this once and for all
Hello
We are currently doing an emergency Maintenance on cp-1 cp-2 cp-3 cp-4 and pve-4 as we need to replace the raid controller battery urgently
Maintenance has been completed and servers are back up and running.
We do apologize for any inconvenience caused.
There was an outage on the Cape Town VPS cluster. The server seemed to become unstable. We were forced to reboot the node to get the server back up.
We further investigated the issue and it seemed we received a massive ddos attack of some sort to the Proxmox node. This has been resolved.
Please be advised we have experienced an outage that affected one of our Plesk server (pl-3.hkdns.co.za). Our systems administrators have resolve the issue and are currently investigating the cause.
We do apologize for any inconvenience this has caused.
-------------------------------------------------------------------------------------------------------------------
Resolution:
There was an issue with corrupted files caused by plesk update, redoing the update fixed the issue.
We are seeing Openserve traffic in all regions getting little to no throughput or authentication issues.Our upstream provider and Openserve are looking at the issues currently. We apologise for any inconvenience caused.
We got some reports of emails being stuck in queue even though support staff replied hours or days ago.
Random emails were blocked by our helpdesk ssytem we use.
We reported the issue to our Helpdesk Support Provider who provided us with a fix which we implemented.
We apologize for any inconvenience caused.
We have noticed an issue affecting all containers on pve-3-cpt and are in the process of restarting the lxc containers, which seems to be freezing from time to time.
We do sincely apologize but this will be resolved very shortly.
Regards hostking
We experienced a caching issue on our NGINX server hosting our website. Our customized configuration file was wiped during an auto-update which caused some unforseen issues.
This however has been corrected aswell as we temporarily disabled dynamic caching on the server until the weekend when we will do testing on the latest version of the software.
We apologize for any inconvenience caused.
PHONE LINES DOWN AT TIMES
In support of ongoing efforts to fight the spread of the COVID-19 virus, we have all our staff working remotely. We thus only have a handful of staff on phone system at times, but we are still servicing customers via email, tickets and live chat. For assistance please log a request via our Live Chat system for urgent requests on our website.
We would like to inform you that we updated our Domain Reseller Module with a critical updated version due to the new coding changes within WHMCS 7.10.
If you are currently using our Domain Reseller Module be sure to update it.
You can find the new updated version on our downloads page.
There is a current issue with the Yoast SEO plugin.
Existing installations making use of the update facility will get broken pages and 404/403 type errors.
Solution is simple. Remove the installed Yoast SEO plugin, and re-install it fresh to get rid oif the update bug.Here are two links below for more information regarding the issue, as well as a URL for the Yoast SEO update page :
https://wordpress.org/support/topic/yoast-seo-update-breaks-site/https://wordpress.org/plugins/wordpress-seo/#developershttps://wordpress.org/plugins/wordpress-seo/#developers
Update: Yoast has corrected their issues in their latest version of the plugin
An outage hosting some of our VPS servers in the cape town datacenter are currently down, data server technicians are currently attending to this.
We do apologize for any trouble caused throughout this duration, but we can assure our clients that we are working on an immediate resolution.
UPDATE
Both redundant power suppliers failed on the pve-cpt node and data center technicians have swapped out both PSUs, we do apologize for any inconvenience caused through this duration.
We are experiencing packet loss on vps cluster on vps servers, we still looking into it and dont have an eta at the moment
Attending:Hostking
Engineers
Additional Information:
Our Cape Town data centre was subjected to a large DDOS attack which took longer than expected to be mitigated. The cause of the delay is being investigated.
Symptoms:
Which may effect your website, emails and access to your control panel, our engineers are attending as a matter of urgency.
Incident detail :
At 20h11 our Cape Town network was hit by a series of 5 DDoS attacks. Our automated DDoS mitigation system failed to initiate the BGP network protocol update to divert traffic via our DDoS mitigation service provider.
Initial troubleshooting and remediation steps attempted to resolve the issue, however repeated attacks made it apparent that this was not the case. We therefore updated incoming traffic routes to permanently route via our DDoS mitigation service provider, fully resolving the issue. The attacks were fully mitigated by 22h20. Follow up investigation revealed that the DDoS attacks exceeded the ability of our core network devices to have the spare resource capacity to enact the BGP route updates.
Remedial Action :1. Our Juniper core network devices will be replaced with higher end models as an urgent priority. a. As this upgrade was already planned, it will take us a couple of weeks to complete. b. The units have been tested to confirm that they will have the capacity to trigger the BGP route updates at DDoS initiation.
2. We will continue to permanently route inbound traffic via our DDoS mitigation service provider until the upgrades are complete. This does not affect our network performance in any way.
We sincerely apologise for the negative impact this incident has had on your business.
We phoned the data center to see if there is an issue with networking and have been informed they are currently attending to an issue with their generators.
We will provide you an update as soon as this has been resolved,we unfortunately cannot provide an ETA at this moment.
We do apologize for the downtime.
We have experienced an outage at our JHB datacentre. The outage was caused by a Raid failure on our Plesk node. Our backups are scheduled to start running at 5:30PM during weekdays and since the raid failed early this morning. It had corrupted not only data on the server but also the latest backups for the 4th Feb 2020.
We do have backups for the 3rd Feb 2020 as well as full SQL backups made on pl-SQL2 server on the 4th Feb 2020 and we are working to restore all content urgently.
---Update---
We were able to restore data for the following servers:
pl-1 - pl-2 - pl-3 - pl-4 - pl-sql2
However we were not able to recover the data on the pl-sql1 server which has MSSQL Dbs for pl-1 server.
We have tried rebuilding the RAID using a replacement RAID card which should've worked however it only recovered the ones mentioned above but the pl-sql1 server refused to boot and gave corruption errors which makes us believe it was the first server affected in the outage a day before already.
If we can recover some data please request our support team to check for you and we will attempt to do so asap.
As communicated to customers we are in process of migrating all Windows Plesk Customers to Windows 2019 so we will not be affected by a resent hack that Microsoft will not be patching in earlier versions before 2019.
Hence the procedure is as follows:
We will do pre-check migration at 10pm Friday the 20th Dec 2019 for the following servers:
plesk01 -> pl-2.hkdns.co.za (156.38.224.12) - Completed
plesk02 -> pl-3.hkdns.co.za (156.38.224.13) - Completed (Last Server done so if any issues arise please inform us asap)
- Issues seem to arose with Plesk02 with Plesk's Migration tool. It synced website files and accounts fine but a few email accounts and databases it has not. We will resync these again throughout the day but most of the accounts have been successfully transferred over.
plesk03 -> pl-4.hkdns.co.za(156.38.224.14) - Completed
If using Cloudflare or any custom DNS entries please ensure it points to any of the IP addresses above depending on the server your account resides on. Contact support if required for more information or assistance.
Some Services like SQL IP may change so login the plesk panel to view IP or request info on this if required, though old server will still be live for months to come and we will email once we complete the SQL Server changeover.
Before we complete a final sync we will update this notice to inform any customers with custom DNS to check they are pointing to new IP address.
Also to inform customers if they would like to check their websites to confirm all is in order.
Further updates to follow.
We have received notifications regarding a new serious vulnerability within Microsoft Windows Server 2008R, Server 2012, Server 2012R2 and Server 2016.
As a result we have scheduled maintenance on our servers this evening.
Affected Servers :
All Plesk servers include Plesk database servers.
Impact:
Downtime can be expected and this will affect email and websites.
ETA:
We expect all servers to be restore within 1 - 2 hours.
Related Article :
https://support.plesk.com/hc/en-us/articles/360010138760
EDIT: All updates completed and servers restarted.
We completed the updates.
However will be required to move all accounts urgently to windows server 2019 soon as Microsoft has stated they have no solution to this issue as yet.
As this is a shared hosting server we have no choice but to perform the upgrade soon.
We will SMS and Email all customers on Plesk01 and Plesk02 servers when this process occurs.
----
We found our oldest Plesk Server having reports from customers that the server was hacked and we looked at it yesterday and this morning. We found random sites being affected aswell as some backups being infected. We however noticed Databases are intact and untouched.
This sites affected seem to be affected and encrypted with rx99 hack. We have 2 sets of AV on the server one Real Time scanning protection and another doing a weekly scan but neither has since picked up anything even after updating signatures.
We are cleaning sites as we can and attempting to restore from backups what we have available that were not affected on the DR weekly backups. For those customers affected we will inform you as we spot any infection.
We will be communicating with these customers individualy to move their sites to new server asap. Our system administrators are setting up a brand new server shortly.
Currently performing emergency maintenance on our plesk servers.
Our system administrators are actively working on the servers.
Affected Servers :
Plesk servers 1 and 2
Impact:
Downtime can be expected and this will affect email and websites.
We have performed an reboot on the server to run a scan disk to fix issues with server load caused by errors on the disks, and have also increased the memory on the server to assist in keeping server stable during daily tasks such as backups,mailscanner.
We do apologize for the downtime experience.
We need to do emergency maintenance as the disk on cp-7 seems to be failing
we will be replacing the disk but running a checkdisk to fix any errors
this may take it down for an hour or so
Our servers were DDOS attacks this morning which has resulted in extensive downtime our servers.
We have resolved this and servers should resume as normal going forward.
We do apologize for the inconvenience caused.
We will continue to monitor the servers.
We have experienced an outage in our cape town data center which is currently being looked into, and will provide you an update once has been resolved.
We do apologize for any inconvenience caused.
There is an outage currently in Data center affecting the following servers.
pl-1
cp-5
cp-6
cp-7
Engineers are currently attending to this issue and should be resolved shortly. We apologize for the inconvenience caused.
There is an outage currently in Data center affecting the following servers.
cp-8
cp-9
cp-10
cp-11
Engineers are currently attending to this issue and should be resolved shortly. We apologize for the inconvenience caused.
Resolved:
The issue was due to a network outage at our data centre. Our engineers have restored connectivity to the servers in question. Should you have any further queries feel free to contact us.
Please be aware that Plesk01 will be running a scheduled CHKDSK at 22:00 this evening.
Thank you for your understanding.
We are currently performing some emergency maintenance on Plesk01 server, this will be resolved shortly
We had a bbu and raid failure on one of our cpanel nodes. The issue caused a few minutes downtime but our engineers resolved it shortly after and alll should be working again.
We apologize for the inconvenience experienced during this time.
We are currently performing server maintenance which will result in experiencing some downtime
We do apologize for the inconvenience caused and are working at resolving this as soon as possible.
Update:
Plesk automatic update caused issue on the server.
we are pleased to state we fixed the issue with the help plesk admins and our system admin
We are currently performing server maintenance which will result in experiencing some downtime.
We do apologize for the inconvenience caused and are working at resolving this as soon as possible.
We will provide feedback as soon as this has been looked into and resolved.
More info on the outage on cp-6 is https://cloudlinux.statuspage.io/?utm_source=embed
Cloudlinux is attending to the issue
Please be aware we are performing maintenance on out virtual hosting environment adding an additional cpu to the server.
Effect services managed vps servers, Expected downtime 30-60 mins.
We are currently performing server maintenance which will result in experiencing some downtime.
We do apologize for the inconvenience caused and are working at resolving this as soon as possible.
We will provide feedback as soon as this has been looked into and resolved.
Update : 09:30 Running checks
Update : 09:55 Rebooting server
Upate :10:00 Reboot complete, running additional checks .
All issues on CP-4 seem to be resolved, we will continue to monitor.
Emergency Maintenance: Server upgrades in the Cape Town Data Centre
Start: 19 June 2019 approximately 17:30:40 SAST
Point of impact:TBA
Issue:Server upgrades to all rows in the Johannesburg Data Centre.
Attending: System Engineers
Symptoms: Websites and email hosted on the affected servers will be inaccessible during the maintenance window.
All servers hosting the SitePro web builder in cPanel will be temporarily unavailable during SitePros Scheduled maintenance.
This status will be updated once this has been completed.
We are currently performing server maintenance which will result in experiencing some downtime.
We do apologize for the inconvenience caused and are working at resolving this as soon as possible.
We will provide feedback as soon as this has been looked into and resolved.
Service is up and running,we will continue to monitor going forward.
Emergency Maintenance: Server upgrades in the Johannesburg Data Centre
Notice ID: #3830
Start: 18 June 2019 approximately 17:30:40 SAST
Point of impact:TBA
Issue:Server upgrades to all rows in the Johannesburg Data Centre.
Attending: System Engineers
Symptoms: Websites and email hosted on the affected servers will be inaccessible during the maintenance window.
We are currently experiencing outage at our datacentre.
The outage is affecting all Plesk servers. Our systems administrators are investigating and will have services restored momentarily.
Affected services : Websites, email and databases within the Plesk environment.
We experiencing an outage affecting connectivity to certain servers. Due to issue at data center
Affected Servers :
vps
plesk
Impact:
The outage will affect email and websites.
ETA:
System engineers are currently attending to it.
Emergency Maintenance: Multiple servers in the Johannesburg Data Centre
Notice ID: #3830
Start: 28 May 2019 approximately 02:40 SAST
Point of impact:TBA
Issue:Power outage on some of the rows in the Johannesburg Data Centre.
Attending: System Engineers
Symptoms: Websites and email hosted on the affected servers will be inaccessible during the maintenance window.
We have received alerts for hardware failure on one of our servers. We have re-scheduled maintenance for 28 March at 17:00.
Affected Servers :
Website Panel, Lin03, Lin09, Lin10 and the Hostking website (Client Area and ticketing system)
Impact:
The outage will affect email and websites.
ETA:
We expect all servers to be restore within 1 - 2 hours.
We experiencing an outage affecting connectivity to certain servers, due to hardware failure.
Affected Servers :
Client Area
Ticket system
lin05.hkdns.co.za
lin06.hkdns.co.za
lin07.hkdns.co.za
lin11.hkdns.co.za
lin12.hkdns.co.za
Impact:
The outage will affect email and websites.
ETA:
All servers except lin11.hkdns.co.za have been restored. We expect to have lin11 restored within the next 2 - 3 hours.
We are currently attending to a few server outages this morning.
We will update you the moment this has been resolved.
We apologize for the inconveniece caused.
Outage : Ticketing system
Our administrators are performing emergency maintenance on our ticketing system. We do apologize for the inconvenience this is causing.
ETA : 3 hours
Our vz-jhb-3 Node is being shut down temporarily to swap a faulty disk drive.
We expect the server to be back up soon.
------
02:15
Seems there was a issue with one of the physical nodes hosting some servers
We replaced the faulty disks and all websites should be working again.
We apologize for any inconvenience caused.,
We are investigating a network outage on our Plesk servers node.
Will update here soon as we have more info.
The network outage was resolved.
Good day,
Network Maintenance for Self-Managed Servers
We will be performing network maintenance in our Samrand Data Centre. During the maintenance, the network connectivity to your Shared hosting server may be disrupted. This will affect end-users and resellers.
When:
Wednesday, 30 January 2019 between 01h00 - 03h00 SAST.
Impact:
Your server(s) may experience up to 20 minutes of downtime during the maintenance window.
Affected server(s):
Websitepanel
cp01.hkdns.co.za
Lin01.hkdns.co.za
Lin02.hkdns.co.za
Lin03.hkdns.co.za
Lin04.hkdns.co.za
Lin05.hkdns.co.za
Lin06.hkdns.co.za
Lin07.hkdns.co.za
Lin08.hkdns.co.za
Lin09.hkdns.co.za
Lin10.hkdns.co.za
Lin11.hkdns.co.za
Lin12.hkdns.co.za
Lin13.hkdns.co.za
All managed and unmanaged servers
Action required:
None.
We will make every effort to minimise the impact on you.
Hi
We are currently experiencing network issues in one of our cabinets affecting some of our Plesk servers.
Servers should come up in the next minute or two as our engineers are bringing them back up.
---edit---
Our engineers have brought the network back up and we will continue to monitor the status.
We had major outages occur randomly on vz-cpt-1 node which contains a few of the big Managed cPanel VPS Servers. As these are managed servers, its our duty to ensure we keep them going as best we can.
We made a decision to quickly setup a new server and migrate the VPS servers off to new server before the server completely goes offline. Seems to be the RAID card killing the disks.
We will replace the raid card in that server at a later stage but our first priority was to move the VPS serves to a new server quickly.
UPDATE --
We replaced the disks on the original node but unfortunately it seems to definitely be the RAID card that is at fault as the disks have failed again.
We will be requesting hetzner to replace the RAID card for us if they can urgently on that server but that will incur downtime on that server. Fortunately this will not affect your VPS servers as we have migrated all customers VPS servers that were on that node to a new node in a different cluster.
Note we logged into each server to ensure all services were up and restarted services that were not.
Kindly inform us immediately if you experience any other issues.
Outage on server due to disk failure which is being resolved urgently.
We do apologize for any inconvenience caused due to downtime.
We are currently experiencing an outage on one of our Linux servers.
Server : lin08.hkdns.co.za
Affected service : MySQL , this will affect all customer websites that make use of MySQL databases.
Our systems administrators are currently attending to the issue.
Expected downtime : 1 hour
Dear Customers
We are currently exepriencing techincal difficulties our client area.
Our system administrator are attending to the outage.
We will provide further feedback once the issue has been resolved.
We have encountered an outage at our data centre. All services will be restored momentarily
Outage : Network Outage at Data Centre
Affected Servers : VPS Servers on node vz-cpt-1-ssd
Maintenance has been completed,we apologize for any downtime caused.
Please be aware we are performing maintenance on these following servers
winweb01.hkdns.co.za
winmail01.hkdns.co,za
winsql01.hkdns.co.za
wincp01.hkdns.co.za
lin03.hkdns.co.za
lin10.hkdns.co.za
lin09.hkdns.co.za
Time of maintenance 20:00
It will be down for 20 minutes.
We had a network Outage and we had to restart the servers
We are currently experiencing some outgates on lin07 , it seems to be a faulty disk. We are running a disk check and will replace if needed.
We are currently doing our 6 month updates on plesk servers. Each server will be rebooted for a minute or two.
We will be rebooting a VPS as we noticed massive packet loss and high load occuring on the server.
We will investigate thereafter once server is up.
Update: we are migrating all VPS clients to SSD servers
We've noticed high load occuring on one of our nodes affecting some Managed cPanel VPS servers.
We are attending to the issue and if not resolved by this evening we will be migrating the VPS servers affected to a new server.
-------------------------
Update:
We have setup a new server and are moving some VPS servers across to it.
We are currently experiencing outages with our Hetzner Datacenter. Their engineers are currently attending to this issue and should be resolved shortly
We apologize for the inconvenience caused.
Services are back up and running stable.We will continue to monitor for any further issues.
Issue : Loss of connectivity to the Hetzner Data Centre
Cause : The issue was related to BGP Sessions on their edge routers.
Resolution : The edge routers had to rebooted. Hetzner did continue to monitor and the problem was resolved
Currently there is a Outage on vz-cpt-1 server hosting some Managed cPanel and Unamanged Linux VPS servers.
The LVM Volume Group seems to be corrupted. We attempted a LVM repair with it not working we also attempted the normal fsck.
We are looking at alternative repair options and will also be using testdisk to see if we can repair the superblocks.
Will report back if we are successfull. If not we will restore the latest backup of 16th/15th of each VM on that server.
-------------
Only 2 VPS servers seem to have been badly affected. The rest are all up and running. 1 of the VPS servers has lost a few days of data and backup also corrupted due to its state. However we ran a repair and it came right up.
The other VPS we've restored the most recent backup.
All other VPS servers should be working as of the state as of the 16th.
We sincerely apologize for the inconvenience. This was an unfortunate issue and seems the host node software may have created the issue on the new kernel when using Thin Pool feature on LVM Groups.
Our systems administrators are scheduled to perform maintenance on one of our hosts this evening.
Affected Services :
Plesk-win01 and Phone system
We estimate that the down-time should not exceed 1 hour.
We are currently investigating a network issue on our Plesk01 server. We apologize for the inconvenience this is causing and are working to resolve it as fast as possible.
------------------------------------ Update
We found that the network card is definintely at fault when cpu spikes above 50% on all 24 cores of the server it seems the network card as all packets then do not pass through.
We decided to swop it out with a Intel Card and updated drivers after a reboot and server seems more stable, CPU usage has also dropped below 10% which is where it usually should be.
We will however have to continue monitoring it and added a few extra SNMP sensors and Nagios sensors to the server to monitor more stats. This may cause extra load as we use CACTI and Nagios to monitor many parts of each server.
Nevertheless we are confident the change should work and will keep this status update with any new updates.
----------------------
We came across one of the disks being faulty aswell on this same node.
This server has been running for 2 years now with no issues but seems we unfortunately will need to rebuild a new server and move the content across. Note we will plan a way in minimizing downtime as best we can and ensure we perform the maintenance and transfers during the evening at around 1am only.
----------------------
There is an outage currently in Data center with some of our servers. We are curretnly investigating.
The following servers have been picked up as currently affected:
lin02
lin08
lin01
lin04
We will try and get them up as soon as possible.
-------------------
Issue seems to be disks attached to RAID become faulty at the same time. 1 SSD and 1 Enterprise SATA disk failed.
We have replaced the disks and it may require rebuilding this evening but servers should be up and running already.
We apologize for any inconvenience caused.
We are currently resolving issues with our servers.
We apologize for the inconvenience during the downtime.
We will update you the moment the issue has been resolved.
The issue has been resolved,services should resume as normal within next few minutes.
We are currently experiencing issues with litespeed/apache on lin03 which we are currently attending to,sorry for the inconvenience caused.
We have resolved the issue,we will continue to further monitor for any problems.
We are seeing some intermittend connection issues at our Johannesburg Data-Centre.
Technicians are working on the issue and we should have it sorted soon,
Hi,
We are currently performing maintenance on our chat service. Please feel fry to contact us via email or contact our call centre
We will be rebooting the node VZ-CPT-1 that hosts some of our VPS's tonite at 22:00 in luei of the outage earlier.
Hostking team
-edit (UPDATE)
Outage has been resolved we did not have to reboot the node VZ-CPT-1.
Thank you.
-edit
There seemed to be an outage occurrence in data center affecting some openvz VPS servers on certain IP ranges.
We requested the DC technician take a look and noticed it came up on monitoring.
We are still investigating the cause to the issue.
hostking team
-edit
It seems one of the VPS servers on the server received a ddos and hence server network was affected. All should be fine now as we throttled the VPS network and are continuing to monitor it.
-edit
We have received reports aswell as intermittent issues of lin04 cpanel server having issues
We noticed mysql service crashing regularly. Our system admins have rebuilt mysql and added more memory but it is still occuring.
We are attending to the issue and we suspect one of the shared sites causing the issue. Once we find it we will inform the customer or possibly suspend the site temporarily till resolved.
Issue has been resolved.
We informed the client and they resolved the issue in their database.
We are performing some maintenance on our website. Please bare with us while we complete this.
We are optimizing our MySQL DB aswell as PHP limits on our website to increase responsiveness of the site.
We will be rebooting our Plesk - Windows hosting shared servers after emergency maintance. Raid failure on Plesk storage.
Card has been replaced.
Services will be down for a few minutes.
We experienced a intermittent network connectivity at our data centre for about an half an hour this issue has since been resolved and services is up and running now.
The network outage which was at our cape town data center has been resolved.
Servers are now backup and running.We appologise for the inconveinece caused.We are currently monitoring them.
There is an issue on lin01.hkdns.co.za
Some users might be experiencing temporary connection issues.
Our engineers are working on the issue and should have irt restored shortly.
There is a currently an issue with the RAID on plesk01. We are attending to the issue. Mail should still be fine.
Please bare with us while we attend to the issue.
----------------------
The issue was resolved. We had 2 disks in the RAID fail at the same which affect IIS sites, mail, databases were not affected. This unfortunately caused some delay in restoring site data and rebuilding the RAID array.
All services should be up and running once more.
------------------------------------
We are continuing to monitor the situation further.
------------------------------------
Unfortunately some data for the 3rd could not be retrievable. We apologize for any inconvenience caused for any sites added on that day. However please contact our support team who will still attempt to see if they can retrieve any data from the backups. Please note as per our terms though our backups are done daily in the evening. So any data during the day between working hours, on or before the crash for a few hours may be unretrievable.
We received reports of services dying on some VPS servers. We had to quickly reboot the box to get services running again.
It seems the issue is related to kernel update causing the issue early this morning.
Issue should be resolved now.
Kindly inform us if there are any further issues.
Note: We are planning on moving all servers on the VPS node to Faster SSD servers this coming weekend. Please stay tuned for that update.
We have had reports from Support Team that domains cannot be registered currentlly. We have investigated and find that the ZA registry currently has issues on their system.
We are awaiting feedback to our latest ticket with them to resolve this issue urgently.
We will provide feedback once received.
There was a outage in one of the cabinets affecting a few of our cPanel Shared hosting servers.
Outage lasted 12 - 15 mins.
Issue has been resolved.
Kindly accept our apologies and we will continue to monitor.
It seems we have encountered a issue with lin01 after a cpanel update it seems to suspend all the domains on that server we have logged a ticket with cpanel themselves and they are investigating this issue
We created a priority ticket with cPanel as the issue seemed to occur during the auto update of cpanel software.
They have responded to us with the following response:
"Thanks for holding, it appears this is a known issue in internal case #CPANEL-15431, in which, a empty line in /etc/passwd is causing this to occur.
I've removed the line and confirmed suspending and unsuspending affected accounts allows the site to load, though there are a large number of already suspended accounts, and accounts that would need this applied.
We should be able to create a loop to suspend and un-suspend just the accounts that are not currently suspended, would it be OK to proceed with this at this time."
-------------------------------------
They are now attempting to implement a temporary fix onto the server.
-----------------------------------------
cPanel have corrected the issue and have stated if the site has been viewed please ensure you clear your browsers cache, close browser and retry as all websites are back online. We can confirm this as we tested over many random sites and all are working. We will however continue to monitor the server going forward.
We had a network outage this morning for 20 minutes for Plesk01 Server in the Cape Town DC. We are attending to the issue.
We have added a temporary fix for now and traffic is working once again.
Please bare with us while we attend to this issue.
---------------------------------------------------------------
Update: We noticed after making some tweaks on the network aswell as the rebooting the network drivers on each server we found server to be more stable.
We are continuing to monitor the servers and will be performing additional stress/load testing this even aswell as creating a fake ddos attack to see how the server handles the load and traffic.
All in all seems server is more stable and we will continue to monitor it going forward.
--------------
We monitored it for the weekend and all ran fine. We will continue to do so and keep you updated if anything else changes.
There is currently an outage affecting JHB services. We are attending to the issue and will revert back once resolved. Seems to be affecting, lin06, winmail01, lin05 and lin07 thus far.
Update:
A faulty disk in RAID 10 was replaced to resolve the issue. It is currently rebuilding but all should be working nevertheless.
There was a 13 minute outage with LiteSpeed/Apache on lin04.hkdns.co.za this evening.
We restarted apache which seemed to resolve the issue
We apologize for any inconvenience caused.
Please be aware certain changes has been implemented for VPS Servers as currently Proxmox interface does not work. The developers are attending to this.
Additional IP address will need ot be requested through support.
Backups and Restores will also need to be done via support who will escalated to system admin as required. Note there may be a delay in this.
Rebuild feature will have additional templates at a later stage while we update the servers.
Thank you for your patience and time in this regard.
There is currently an outage with one of the VPS nodes. We are giving the server a full reboot as we see a kernel panic on the screen.
Should be up shortly.
------
Update: 12:22 - Server came up but went down again. Seems 2 disks in the RAID 10 Array has failed. We are replacing those failed disks and running a fsck at present which may take some time. Raid should but as its RAID 10 should be quite quick.
-----
We have rebooted the server after the fsck and all should be coming up now.
We will however be doing major maintenance this evening at around 1am to correct any possibly issues.
We've noticed some issues on plesk server with respect to email and hence upgrade its to its latest version 9.62 as per Mailenable support who recommended we do so quickly to fix many bugs in their older versions.
Please give us some time to reboot the server . should take a few minutes.
---------------------
Reboot was completed at around 21:15 however mailenable still ran and repaired itself for another 30 minutes though services were online.
If you experienced any issues we do apologize but all should be back to normal and server is now updated to latest version.
Plesk servers are currently facing a network issue, and we are working to restore the service.
-edit-
All services restored
Currently replacing 2 faulty disks in the RAID which are giving issues on one of the Virtual Server Nodes. This may affect a few VPS servers.
Please bare with us while we replace this should not take longer than 15 to 20 minutes.
We have looked at logs of LIN02 and LIN04 and have noticed some sporadic slowness with respect to mysql that may be causing issues for some clients or domains on these servers.
We will be upgrading the server from 12GB of Memory to 16GB this evening and adding more memory to MySQL ofr Innodb purposes.
We will continue to monitor thereafter and also look at increasing this for other servers aswell this coming weekend.
Hi
We noticed some issue with LiteSpeed on LIN04 occuring where SSL connections were being dropped for a few minutes. We reinstalled litespeed and are monitoring the is situation.
We will reboot the server in next few minutes which should create a few second timeout as we use SSD servers.
We will continue to monitor the server to ensure the issue is resolved and have already reported the issue directly to litespeed.
Their was a outage that occured on lin05 server due to high network usage. We disabled the account causing the issue and everything should be running normally once more. We apologize for any inconvenicence caused.
Emergency maintenance performed on the node.
Server was rebooted to apply new kernel.
Services restored.
We have been receiving very intense DDOS attacks this past week and our system administrators have worked long nights to try and stop this from continuing. We have now tested ratelimiting certain countries which seems to be working. For example Chinese IPs that hit our server to a specific port or from a certain IP to fast will be ratelimited longer than any other country. Most other countries are not ratelimited.
We are continuing to monitor this but as of now all seems to be working well and offending IPs are being blocked on the firewalls as they should be.
Recently we've noticed that TLS gets forced every evening on this server and hence causing issues when customers try to send emails the next morning. We've hence found a solution and will be implementing it this evening so that this does not reoccur.
We apologize for the inconvenience but rest assured it will be resolved.
------
We completely the reverting to Mailscanner at 6pm. All should be functioning as it was before.
Currently our Live chat system is undergoing some maintenance. Please send us any ticket or email if urgent to support@hostking.co.za or through client area.
Hi
We found an issue in cpanel settings:
"Require clients to connect with SSL or issue the STARTTLS" which was enabled after resetting the spam filter. We have removed this settings as it is be default enabled.
This should resolve all issues on the server.
We currently attending to issues occuring on the lin01 and lin05 servers. Please be patient while we resolve the issues.
Update: Issues was related to our LSI Raid card failing. We replaced the RAID card and all should be up and running within a few minutes.
We've been noticing on monitoring and getting reports of lin04 server being slightly slower than usual. We noticed some massive hits to server from random IPs from russia and china and may block those IP ranges completely for 24 hours.
We had to give some VPS Servers and Plesk servers on a few nodes a quick reboot as 2 VPS Nodes were having DDOS causing high load and blocking on the firewall was not helping much. We had to increase the number of TCP/IP connections and then block IPs on the servers firewall aswell.
All should be working once again.
We have been investigating a few issues with LIN04 and LIN02 servers with respect to email. Seems to be issues with port 26 aswel as the spam filtering.
We have made some changes and are still monitoring the situation.
Seems there may be an issue with the billing side at co.za and hence registrations are going through intermittantly.
We awaiting feedback from ZACR currently on the matter.
-------------------------
The ZACR has resolved the matter.
Seems we are experiencing a ddos attack on LIN04.
We have increased sensitivity of blocks hence should be blocking the IP ranges attacking the server on the firewall much faster but may inturn slow down connections for short period of time.
We noticed issues on 2 raid setups on 2 servers - LIN01 and LIN05 appearing on monitoring. We corrected the issues after noticing this and servers should be operating once again.
We apologize for inconvenience caused.
We will be applying a new kernel update to all KVM and LXC VPS Nodes.
This does not affect shared hosting. Only Unmanaged and VPS Client servers.
Downtime should not exceed a few minutes (+/-) 5 to 10mins.
Start: 2016-06-06 17:20 SAST
Resolved: TBA
Status: attending
Point of impact: Johannesburg data centre
Symptoms: Intermittent connectivity to their servers.
Cause of problem: TBA
Estimated time of repair: TBA
Attending: Hetzner Engineers
Seems there was a DDOS attack on the server currently causing very high usage. We are attending to it and have rebooted server so we can manage it.
We are rebooting the node which may affect some VPS customers for a few minutes.
Should come up shortly
------------------------------
We are monitoring server and seems fine going so far. Will keep posted if we spot anything else but everything should be up and running.
----------------------------
Seems the HW Raid Controller on this server is also having issues and may be why disk io went so high during the DDOS. We will be moving some of the Managed VPS servers off this evening and are making full backups of all servers.
Unmanaged VPS servers will remain for the time being as they are less resource intensive (disk usage lower).
Intermittent Connectivity: Cape Town Data Centre (2016-05-23, ATTENDING)
Start: 2016-05-23 21:25:00 SAST
Resolved: 2016-05-23 21:40:00 SAST
Status: attending
Point of impact: All services
Symptoms: Customers may experience slow or intermittent connectivity to our Cape Town Data Centre.
Cause of problem: TBA
Estimated time of repair: 30 Minutes
Attending: Hetzner Engineers
One of our servers seemed to have an issue starting after upgrading to new version of LiteSpeed 5.1.5 earlier this morning at around 5am.
We have attempted to reinstall the version once more and it seems to have come right.
Please inform us if there are any further issues.
We will be performing some maintenance on LIN02 for security and updates on the server.
We have no rebooted the server in over 180 days and hence will be required to do so for a short period.
Server should be up shortly.
We received monitoring errors of issues connecting to LIN03. We investigated and found that their was a network issue at the datacentre affecting only LIN03 server.
We informed the Data Center (Hetzner) who resolved the issue for us.
All should be resolved.
We are investigating an issue in the data center with a few servers in the same cabinet in the data centre.
Please bare with us as we resolve this issue.
--------------
Update - Issues seems to have been a network issue. Problem is resolved however but we will continue to monitor this.
There is possibly network issues at Hetzner with some of our IP ranges. We are discussing this and investigating with one of the engineers currently.
We have recieved a notice from Hetzner with respect to the Data Center DDOS attack it seems to certain IP ranges which is affecting Colocation and Truserv clients, More information as per their website from the data center.
DDOS - Network connectivity data centre (2016-05-03, RESOLVED)
Start: 2016-05-03 2:20:53 SAST
Resolved: TBA
Status: resolved
Point of impact: Truserv and Co-Location customers
Symptoms: Truserv and Co-location customers will have intermittent connectivity to their servers. Also experiencing high packet loss.
Cause of problem: DDos
Estimated time of repair: TBA
Additional Information: Target of the DDos attack has been identified and blocked accordingly. UPDATE: Another DDos attack started up 19:45 UPDATE: DDos attack mitigated at 20:20 UPDATE: Another DDoS attack started at 1:20am has been mitigated as of 1:35am but there is high packet loss now.
Attending: Hetzner Engineers
- Issue is now resolved and Offending IPs have been blocked on hetzner firewall
Seems to have reocccured and we are contacting hence once again to check.
Seems to have occurred again. Kindly note we are monitoring the situation with Hetzner and will provide feedback when received from data center
We noticed an issue where LIN03 servers mail authentication was not working. Seems to have been caused by a failed cpanel nightly automatic update.
We reran it via /scripts/upcp --force and it corrected permissions and reran the update.
All should be working now.
We apologize for any inconvenience caused.
Hi
Some servers has some network issue. Kindly note we will be giving them a quick reboot.
Please note we will be performing the first stage of the migration where we will be moving our existing servers into the same cabinet as our new servers, we will be using that has SSD disks within it.
SSD disks should make our servers much faster and more stable than before.
Kindly be aware there may be a 10 to 15 minute outage per server due to this move.
We thank you for your patience and understanding.
Hi
Seems we've picked up a few ddos attacks on LIN03 and LIN04 server occuring since yesterday.
We are working on this to resolve this issue.
Servers should be stable within the next few minutes.
We apologize for any inconvenience caused.
There was an outage for around 10 to 15 minutes on LIN04 server. Seemed one of the sites was DDOS'd and litespeed could not keep up with it. We quickly restarted the server now and disabled that website.
Server should be running fine now. We will look into solution shortly for the particular website affected.
We apologize for any inconvenience caused.
Please be aware we are performing maintenance on our website and client area. This may affect performance or live chat availibility.
We will try and get it up and working as fast as possible.
We need to reboot LIN03 server to apply updates.
We need to reboot the server quickly to have some kernel updates applied. It may take a matter of minutes.
We fixed an outage yesterday with respect to Winmail01 server on websitepanel.
We added the emails back to queue yesterday after the issue was corrected however noticed today that some emails did not go through.
We have restarted the queue today and it seems it is only picking them up now.
We apologize for any inconvenience caused.
Seems there was an Network Outage at 3:30am until 5:30am.
We contacted the engineers at the Data Center who looked into the issue stating there was an issue with one of the Data Centers Upstream PRoviders.
Hetzner switched over to a different upstream provider and the IP Ranges affected slowly started to come back.
All should be working once again. We are continuining to monitor the situation.
There seems to be an issue with web server of websitepanel currently.
We are investigating the cause.
Server should be up now but we will continue to monitor it and investigate to find the cause of the issue.
There was a sligh outage on LIN04.
CAgefs service stopped and sites PHP started to give issues.
We quickly restarted the service and reran cagefsctl --reinit to remount all the cagefs mounts.
All is working again though we will need to run an update this evening on cagefs - should not cause any downtime.
Seems we experienced high load on 2 servers and noticed immediately a high amount of connections to mysql. We blocked the port 3306 on the server and then found the IP addresses hitting the server constantly.
Seems to have been a overseas IP address. We have blocked the few IPs causing this and load as dropped.
We however are continuing to monitor this.
We have added more memory to MySQL config file for LIN02/LIN03 servers as they both were giving us memory errors recently this week and made the change on 21/01/2016.
We've since done some hardware changes aswell as config changes this morning.
We hope this resolves performance issues on this server.
Information from our Data Center Provider (Hetzner)
Slow or intermittent international connectivity (2016-01-21, ATTENDING)
Start: 2016-01-21 16:15 SAST
Resolved: TBC
Status: attending
Point of impact: JNB and CPT datacentres
Symptoms: Customers may experience slow or intermittent international connectivity at present.
Cause of problem: Undersea cable faults: WACS / Seacom
Estimated time of repair: TBC
Additional Information: We have failed over to our secondary upstream network provider which should alleviate the symptoms for the moment.
Attending: Various Network Engineers
Update: The Seacom fault was repaired at 18:46. WACS is still under repair, but is operating in a limited capacity. We are back on our primary upstream provider's network.
Seems Hetzner where our servers are stored had some DNS and Network issues occuring today.
This was resolved now but it may have affected:
Websites from loading
UptimeRobot and Pingdom Monitoring
Anything that require DNS resolution. Seems local queries were fine mostly international seemed to be affected.
We are still on the phone with one of their network admins trying to provide us with more information. But has stated it should be running fine now.
We applying quick fix to servers LIN02 and LIN03 to ensure server stays stable going forward. Should be down no longer than a fewm inutes.
Hi
We will be adding more memory to mysql my.cnf file aswell as make some tweaks to apache and litespeed this evening. Kindly note there may be 5 mins downtime during this process as it jsut requires restart of services.
We've had to reboot LIN02 aswell due to updates which had to be reverted. Mainly Cloudlinux PHP updates.
All should be working fine.
Hi
We are quickly rebooting LIN03 server to revert some kernel and yum updates that occured. Should be no longer than a few minutes.
Hi
There seemed to have been an issue with PowerDNS Mysql Service on our DNS service which caused some issues with DNS.
This should have caused some issues for many sites on VPS servers
The issue should be resolved now.
We apologize for any inconvenience caused.
We are running a full cpanel update after noticing a few accounts suspending for no reason and automatically unsuspended thereafter.
We suspect it to be a cpanel update that occured last night.
Should be no more than 15 to 30minutes for this to complete.
------------------------------------------------------------------------------------------------
We spotted some errors in cpanel and seems the full cpanel update has corrected these file issues.
We will continue to monitor it but are confident all is resolved now.
We apologize for an inconvenience caused.
We will be rebooting our Hostking Website at 10pm this evening to apply some important updates.
We need to restart cPanel servers to fight an attack. Should be up shortly.
Update. Seems to have been a South Africa IP address aswell. We have decided to remove certain subnets from the IP allow list even South African IPs now.
All should be working once more.
Clients have reported some issues with live chat since yesterday. We will be reinstalling the software quickly and it should be working shortly.
Hi
We are restarting MySQL on LIN03 currently - should be up shortly.
We are attending to an issue with MySQL on Server LIN03
It should be up shortly
We apologize for any inconvenience.
Currently the LIN01 Checkdisk is running and fixing a few disk errors.
Please bare with us while we get this back up and running.
Scan is complete and disk is working perfectly again.
We will be scheduling a replacement for the early hours of the morning.
We are quickly restarting mysql on each server. Should be no longer than 2 minutes each
There is currently an issue syncing to the TrendyTools servers. TrendyTools are attending to this issue and should resolve it shortly.
We apologizefor any inconvenience caused.
Planned Live Chat Outage : 9am to 12pm GMT+2, for deployment of new widget positioning, widget chat bubbles and server upgrades.
Update: Fully updated and working.
Hi
We noticed many sites affected and going down to Cloudlinux update early in the morning.
We have been and are attending to it and it should be coming up shortly
We apologize for any inconvenience caused.
---------------------------
Update
Issue has been resolved. Seems to be related to a cPanel update and CloudLinux issue. We have informed CloudLinux who are looking into a fix. We have however disabled our automated updates.
We need to quickly reboot a few servers to apply a new kernel and update for security reasons.
Downtime should not exceed a few minutes.
Hi,
Maintenance has been completed. Please note once every 6 months we perform these updates to servers. To keep them secure and stable.
Downtime is very little as it just requires a reboot.
Thank you for your patience.
We need to quickly reboot LIN01,WEB01 and WINSQL01 servers to apply a fix on the servers to stop a DDOS attack.
It should be down no longer than 5 minutes.
We've been informed by a client that random reboots were occuring on one of our Windows Servers managing VPS's. We investigated further and before it gets worse we have replaced the memory on the server.
Some clients may be done for few minutes while we do this. We have already completed a few that were critical but still a few are outstanding. Each VPS will only be done a few minutes.
Please chat to our Support Consultants for more information.
We noticed all cPanel servers giving mysql errors. We are attending to it and beleive it to be MAriaddb module PDO error.
Should be resolved shortly
There was an issue today that cause MySQL disconnections due to Cagefs.mp file being corrupted.
It was resolved and all users re-inserted into the Cage.
All was corrected.
We had to apply a few patches to kernel and firewall software for LIN01. Took a bit longer than expected but server is far better now.
Hi,
We have quickly rebooted and installed kernel updates for LIN01 and LIN02 servers.
All has gone successfully.
Thank you for your patience.
We are required to quickly update and install LIN01 server due to an important update that needs to be installed for security reasons.
Should not be done for longer than 5 minutes.
Dear Clients
Please be aware to speed up and ensure that servers are running optimally we will be moving some servers MYSQL and SQL Databases onto SSD drives. We are starting with the LIN01 and WIN01 Servers.
Servers may have to rebuild the RAID on OS and Website disks and may cause slight performance degradation for a day or 2.
WIN01 currently sits at 58% rebuild
LIN01 sits at 64%
LIN02 sits at 78%
WIN02 sites at 83%
If you have any questions feel free to message us.
We will quickly be rebooting the LIN01 Server to apply an important security update. Should not exceed 5 minutes.
We've noticed server LIN01 become slow the past few days. We've found a solution and hence will quickly be implementing it.
This will require a quick reboot.
We apologize for any inconvenience
Hi,
We are quickly rebooting each server due to maintenance being performed in the data centre.
All should be backup shortly
Sorry for any inconvenience caused.
We have been informed of issues and have investigated into live chat issues and disconnects.
Our engineers are rebuilding the database and will be updating the module soon.
It should be up by 10am.
We apologize for any inconvenience caused.
We had some attacks to our Live Chat software, we've hence scheduled emergency maintenance. Kindly email us if you require urgent support.
---------------------------------------------------------------------------------------------------------
We have successfully completed maintenance and upgrading of Live Chat software.
We need to performance maintenance on Server LIN01. Downtime should not exceed 30 minutes.
We experienced an issue where live chat support server was under high load and ticket system was affected by this aswell due to 1000s of log chats being sent from live chat server.
We have however resolved the issue and are closely monitoring the issue further to ensure this does not occur again.
We are investigating a issue or bug that currently exists in our billing system. Our developer is looking into this and we have unsuspended 4 affected client accounts that got affected today. This should be resolved shortly
A few reports of LIN01 server being slow has been reported and appearing on monitoring. We are attending to the issue and will resolve this asap.
Downtime should not exceed 5 - 10 minutes from now.
As per our email to VPS clients on VZ01 - We will be applying quick server maintenance to protect the host the VPS's reside on that has been reported earlier today.
Downtime should not exceed 10 minutes.
We have noticed an issue on the server that we are currently attending to. More information will be provided once completely resolved.
Update:
The seems to have been some sort of memory leak. We will be running a update tonight at 11pm to fix the cause.
We apologize for any inconvenience caused.
We will be performing some maintenance this evening on a few of the servers. We will be migrating these servers to a new cabinet in the data center and assigning each Dell Server with a backup server dedicated to it.
This should not take longer than 15 minutes.
We will be performing some maintenance on a few servers. It is a regular upgrade that needs to be done on a few of our servers. We will be taking them down one at a time for 15 minutes each.
Please contact us if you have any further questions regarding this.
We experienced slow response times on 2 of our Linux Cpanel Servers.
We have resolved the issue and all should be working again.
We apologize for any invonvenience caused.
We have scheduled Critical Maintenance to be done on the server to ensure optimal performance going forward.
There will be some downtime so we can perform the following:
1. Remove the current Backup software
2. Reinstall new agent to ensure faster backups on server.
Downtime should not exceed 1 hour.
It seems a few complaints of POP3 duplication is occuring on some domains.
We are attending to the issue and will resolve this shortly.
The issue has been resolved at 12:37.
We are investigating an issue a customer has brought to our attention with respect to email.
We are aware it may be affecting a few clients and are attending to the issue immediately.
--------------------
Update:
The issue has been resolved. It seems due to the issue with the server maintenance last night the CPanel Update that occurs each evening did not complete. We had to rerun it to complete it's update.
All should be working.
There seems to be a current outage with LIN01 server after the upgrage. Please be aware we are attending to the issue.
The issue was caused by a bad Hard Disk which has been replaced.
The server is fully opreational once more.
We will be performing emergency maintenance on LIN01 Cpanel server shortly.
Downtime should not exceed 10 - 15 minutes.
We are performing emergency maintenance on our live chat software.
This should not exceed 1hour and 30 minutes to complete maintenance.
We will be performing some maintenance to improve network performance and redundancy in our cabinet.
We will be rebooting our servers in the cabinet for 15 minutes.
Maintenance has been successfully completed.
There has been a network issue to a few of our servers, kindly note we have received confirmation of the issue being resolved.
Our engineers are now investigating as to the cause of the issue and will update this status page once enough information is found.
We apologize for any inconvenience caused.
-----------------
UPDATE:
We found a network issue on one of the ports of the switch which is currently being replaced.
We apologize for the inconvenience caused.
We experienced some issues with our Mail software due to a recent auto upgrade done last night. We have had to revert the changes in order for email functionality to work again.
All is resolved. We apologize for any inconvenience caused.
There is currently an outage with all domains worldwide with respect to registrations for .com, .net and .org.
We unfortunately cannot do anything with respect to registrations for any new domains and will refund any clients that purchase these domains.
More information can be found between many web hosting companies on the WHMCS forum: http://forums.whmcs.com/showthread.php?73951-Domain-checker-issue
We apologize for the inconvenience caused.
Update:
We've been informed that Verisign is aware of the issue and attending to it urgently. Once we received more feedback we will update the status
Update:
Verisgn have corrected the issue an Domain registrations for these domains are fine going forward.
Dear Clients,
Please be aware we are working on a small issue updating contacting details of domains. We have a developer looking at the issue and will update this when we are successful in fixing the error. The error returned looks like:
SaveContactDetails/contact-update(C0426031): Code (2001) cvc-minLength-valid: Value '' with length = '0' is not facet-valid with respect to minLength '1' for type 'postalLineType'. L9:C36
Status:Attending
We experienced a short outage for a few minutes on a few of our servers.
The outage was just over 10 minutes.
We apologize for any inconvenience caused.
Emergency maintenance is required on the following servers for 10 minutes:
Dear HostKing Client,
Stav serveru
Následující tabulka zobrazuje aktuální stav našich serverů. oto Vám pomůže zjistit jestli jsou naše služby funkční nebo ne.
| Server | HTTP | FTP | POP3 | PHP Info | Zatížení serveru | Doba bezchybného provozu |
|---|---|---|---|---|---|---|
| aries-rs| Directadmin |
|
|
|
PHP Info |
|
|
| cyclops| Directadmin |
|
|
|
PHP Info |
|
|
| mercury-plesk | Plesk |
|
|
|
PHP Info |
|
|
| my-vps | SolusIO |
|
|
|
PHP Info |
|
|
| orion | cPanel |
|
|
|
PHP Info |
|
|
| phoenix| Directadmin |
|
|
|
PHP Info |
|
|
| pl-1 | Plesk |
|
|
|
PHP Info |
|
|
| pl-2 | Plesk |
|
|
|
PHP Info |
|
|
| pl-3 | Plesk |
|
|
|
PHP Info |
|
|
| pl-4 | Plesk |
|
|
|
PHP Info |
|
|
| storm| Directadmin |
|
|
|
PHP Info |
|
|
| xavier| Directadmin |
|
|
|
PHP Info |
|
|
