Website attack or Misconfiguration? — Log Analysis holds the key.!!
Hello there Everyone. I would like to share a security incident which we took a while back for a client which was a web development firm with a relatively moderate network size.
The Incident
The client was involved in development of web applications for offshore clients on wordpress platform and had a decent amount of websites in development as well as in production. Suddenly, one day atleast 3 to 5 of the production sites started to crash and all the themes and plugins were removed and the sites went blank.
Initially, the client thought it to be some sort of malware attack and tried to scan the application using online scanners available for wordpress, but didnt find anything suspicious. After initial incident probing at client end, they re-launched the website from backups but within 24 hours the same issue hit the websites back again and the client reached out to us with the problem description.
Initial Approach
Initial approach involved indepth understanding of the business process of the client — the way in which the developers handle the website from initial development till deployment in production phase — both tehnical & non-technical parts, on-site visits and understanding the client network.
Then, after this initial kick-off, ‘divide and conquer’ approach was adopted as we found no hard evidence or symptom of any specific attack vector that could cause the aforementioned issue.
Code Review
The initial and the most painful part we began with was the manual review of code. So we took a copy of the most recent code backup from the client and firstly ran it through virustotal to rule out any possibility of known malware and as per our expectation, we didnt find anything malicious or suspicious.
So, the next step was gruelling manual code review — — arrrgggggghhhhhh!!!
File by file we sweeped through the core wordpress files looking for any signs of tampering and/or overwriting of code either in encoded or encrypted manner with the help of the developers from the client… After lots of coffee cups and frustration, we didnt find anything that we might call a ‘breakthrough..’
Initial Web Application Vulnerability Test.
Next up was a primary analysis of the web application for any security loopholes that could have been used by an attacker to exploit the application. So, we deployed the same installation of wordpress to our local environment and started our web app vulnerability test as per our beloved OWASP Top 10 standard.
Another couple of days of rigrous testing and focussing on the vulnerabilities that attackers love the most — file uploads, SQLi, XSS, Command Execution, etc. The application did passed all the tests and we observed no severe or critical or other exploitable vulnerability. P.S. — The Wordpress installation with all plugin and themes was most recent and up-to-date.
So, the web application vulnerability test was passed and the final resort for Log Analysis was to be taken up.
Enter Log Analysis
After days of working on this incident, we decided to look at apache logs — access and error logs, FTP and firewall logs of the client network and try gathering some solid leads.
By the time we were working on this case, the website crashing issue was recurring and thank goodness, the logs were available for reference.
We took the logs for the pre and post incident dates to look for any clues. The work of log analysis was smoothly done on the Splunk Platform of which we are big fans…
The Findings
We were quite amazed and amused by the findings from the log analysis. We were finding a pattern that matched the exact description from client and the website’s behaviour.
looking at the logs, it was getting clear that the script included by default in the wordpress installation, the ‘wp-cron.php’ was not disabled and as the website was hosted in a shared hosting platform, each time a request hit the server, the wp-cron.php was executed and it created a virtual DDoS for the website and hanged the server and made the website reset somehow.
This chain of events occuring after wp-cron.php was triggered, was visible for all the affected sites. Each time wp-cron.php was executed, it generated a huge load for the site and overhelmed the server resources and caused a Denial of service condition. Literally, the website ‘shot itself on its face’.
Each time this event triggered, the developers quickly came to action and started restoring the website and its contents. And everything matched up with the IP address of the developers machine, the access logs IPs, Firewall logs & FTP user logs that can be co-related and matched.
The Conclusion — Misconfiguration
Finally, a big sigh.!!!!!!!!
We found the root-cause of this nasty issue. It was plain misconfiguration from developer’s end. Somehow, developers overlooked to disable the default wp-cron.php file in the wordpress configuration file. This simple line could have avoided the painful cycle in the wp-config.php file
define(‘DISABLE_WP_CRON’, ‘true’);
After adding this line to the wp-config.php file, we wanted to make sure that we have pin-pointed the issue and thankfully, the test environment we set with wp-cron.php in disabled state worked and the website worked without any issues.
Major Observations at client side.
We would like to underline some of the most common issues that we encounter at client’s location with regard to security practices and mindset about security.
- Unauthorized downloads at workplace — Most of the employees like to mix up the work and personal tasks at the same time, at the same machine. Particularly for downloading applications or any other random stuff from the internet. Which has serious security implications. Most of the incidents we respond to have an involvement of an infected Machine in the network that opens up pandora’s box and a ton of misery for internal LAN environment.
- Shabby Firewall rule setting — most of enterprise level firewall products are installed in ‘out-of-the-box’ default setting with default administration passwords and insecure services like telnet enabled with default credentials that can be googled easily. Additionally, the firewall rules are never aligned with the actual requirement of business and left in as it is. Moreover, filtering of protocols — especially UDP protocols are often overlooked which requires proper care in line with the TCP protocol filtering rules.
- Unauthorized softwares / remote administration tools — For sake of convenience many businesses use remote administration tools like Teamviewer / AmmyAdmin / AnyDesk for remote administration as well as support from 3rd parties without any audit trails or relevant track keeping which remains like a potential landmine waiting to explode in any security incident. For such remote administration, a proper approval followed by provisioning and de-provisioning should be made necessary and part of daily business.
- Most of developers deploy web applications without any security testing and analysis in any form. The only test they do is the functional testing for site navigation and input fields. Which again too makes the site succeptable to future attacks. This also includes updates to the frameworks / themes / plugins,etc.
- Overlooking logs — Saved the best for the last. Almost all the attacks which we analysed and investigated our experience tells us that logs are the most important piece of puzzle. The access/error logs, Firewall logs do tells us a lot about what went wrong and actually from where it went down. But the sad part is that most of the clients we interacted with have either never checked for logs or even in some cases, they actually disabled the logging of server (Thats an overkill…!!!!), or even worse — some of them even dont know the log location at the server. It must be always remembered that logs are the best gatekeepers if you can configure it properly. We should always remember that a website will be subject to all sorts of obnoxious activity on the internet — starting from random scans to DoS, or bashing by an automated scanner or tool, it is the logs that keep an account of any unforeseen activity and can give us a hint on anything suspicious. Hence, having properly setup logging and in a secure manner and preferably in replication to another location can be your guide in times of need.
The lesson learnt
Security can never be confined to security products like Firewall / IDSes / Antivirus products based on the fancy claims.
Security is a ‘Continuous process’ and not a product comprising of People + Process + Technology and effective security works in layers.
We would urge all the readers to deploy the security for your home / office based on your specific needs and it is us ‘Humans’ that need to control our IT assets both — tangible and intangible. Other tools and products are just a way to simplify our taks and requirements.