JSONL: Definition, JSON Lines Format, Use Cases, and More

JSONL text format is also referred to as newline-delimited JSON. JSON Lines is an easy-to-use format for storing structured data that allows for record-by-record processing. It functions nicely with shell pipelines and text editors of the Unix variety. It's a great log file format. It's also a flexible format for sending messages between cooperating processes.

What is JSONL?

JSONL is a text-based format that uses the .jsonl file extension and is essentially the same as JSON format except that newline characters are used to delimit JSON data. It also goes by the name JSON Lines.

JSONL files can be imported and linked by Manifold. Additionally, Manifold offers JSONL export for tables. In the GeoJSONL format, JSONL is used.

There is just a single table in a JSONL file.
When working with very big files on devices with little RAM, reading a JSONL file dynamically parses it one line at a time.
The file itself can be any size, however, each line must not be more than 2 GB.

A JSON file generated in the JSON Lines format is known as a JSONL file. The structured data is described in plain language. The main usage of JSONL files is to stream structured data that needs to be handled one record at a time.

A JSON variation called JSON Lines helps developers to store structured data entries within a single line of text, enabling the data to be streamed using protocols like TCP or UNIX Pipes.

JSON Lines is a fantastic format for log files and a flexible way to transfer messages across cooperating processes, according to the website supporting the format (jsonlines.org). It also integrates well with shell pipelines and text processing programs that have a UNIX-style interface. JSONL files resemble .NDJSON files in structure.

Exporting to JSONL

Manifold offers JSONL output for tables. Binary fields are not exported or taken into account.

When importing files, the main distinction between JSON and JSONL is that a JSON file's total size is limited to 2 GB, but a JSONL file's size is unrestricted as long as no one line is higher than 2 GB.

JSON Lines Format

There are three requirements for the JSON Lines format:

UTF-8 Encoding
Unicode strings can be encoded in JSON using simply ASCII escape sequences, although this makes it difficult to see the escapes in text editors. To operate with plain ASCII files, the creator of the JSON Lines file may decide to escape characters. The likelihood of characters in JSON Lines files being mistakenly misinterpreted when encoded in a format other than UTF-8 is quite low.
Each Line is a Valid JSON Value
Objects and arrays will be the most typical values, although any JSON value is acceptable.
Line Separator is '\n'
This indicates that "\r\n" is also supported because JSON values implicitly ignore surrounding white space. Line separators can be the last character in a file, and they will be handled the same as if they weren't.

JSONL format and JSON format differ primarily in three ways:

JSONL employs UTF-8 encoding. This contrasts with JSON, which permits Unicode texts to be encoded using ASCII escape sequences.
Each line has a valid JSON value.
A newline ('\n') character is used to demarcate each line. This indicates that a carriage return, newline sequence, '\r\n', is also permitted because JSON values inherently disregard surrounding white space. Line separators can be the last character in a file, and they will be handled the same as if they weren't.

Use Cases of JSONL

The use of JSON Lines for real-time data streaming, such as with logs, is the first important point. For example, if data were being streamed over a socket (every line is a separate JSON, and most sockets have an API for reading lines).

Logs are stored as JSON Lines by Docker and Logstash.

Another example is the use of the JSON Lines format for lengthy JSON documents.

More than 2.5 million URLs have been fetched and analyzed in one of the company projects. They now have 11GB of unprocessed data.

When dealing with regular JSON, there is essentially just one course of action: load the entire dataset into memory and parse it. Although you can break an 11 GB file into smaller files without parsing the whole thing, search for a certain location inside JSON Lines, use CLI n-based tools, etc.

Three names for the same formats—JSON lines (jsonl), Newline-delimited JSON (ndjson), and Line-delimited JSON (ldjson)—are used to describe JSON streams in particular.

JSON Lines vs. JSON Text Sequences

Let's compare NDJSON with the JSON text sequenced in its corresponding media type "application/json-seq." It is made up of any number of JSON strings, each of which is encoded in UTF-8, has an ASCII Record Separator (0x1E) before it, and an ASCII Line Feed at the conclusion (0x0A).

Let's examine the JSON-sequence file representing the above-mentioned list of Persons:

{"id":1,"father":"Mark","mother":"Charlotte","children":["Tom"]}{"id":2,"father":"John","mother":"Ann","children":["Jessika","Jack"]}
{"id":3,"father":"Bob","mother":"Monika","children":["Jerry","Karol"]}

This is a placeholder for an ASCII Record Separator that cannot be printed (0x1E). The character represents the line feed.

The only difference between the format and JSON Lines is the special sign at the start of each record.

You might be wondering why there are two different forms when they're so similar.

For a streaming context, text sequences in the JSON format are employed. Thus, no corresponding file extension is defined for this format.

Although the new MIME media type application/json-seq is registered by the JSON text sequences format definition. This format is difficult to keep and edit in a text editor because the non-printable (0x1E) character could become jumbled.

JSON lines could be used consistently as an alternative.

JSON Lines vs. Concatenated JSON

Concatenated JSON is an additional choice to JSON Lines. Each JSON string is not at all isolated from the others in this format.

The preceding example is represented as concatenated JSON here:

{"id":1,"father":"Mark","mother":"Charlotte","children":["Tom"]}{"id":2,"father":"John","mother":"Ann","children":["Jessika","Jack"]}{"id":3,"father":"Bob","mother":"Monika","children":["Jerry","Karol"]}

Concatenated JSON is only a word for streaming numerous JSON objects together without any delimiters; it's not a new format.

Although creating JSON is not a particularly difficult operation, parsing this format takes a lot of work. You ought to implement a context-aware parser that recognizes different records and correctly differentiates them from one another.

How to Open a .JSONL File?

We'll walk you through the process of opening the .JSONL file on various operating systems in the section below.

How to Use Windows to Open a .JSONL File?

A step-by-step visual tutorial showing how to open a .jsonl file on Windows is provided below.

The GitHub Atom software must be downloaded first. You need to use this software to open the file. Other tools that can be used in opening this file are Microsoft Notepad and GitHub Atom.
The second step is locating the downloaded file. If you are unsure of where you downloaded a file, you should look in your /download/ folder because there is typically where it is saved by default.
After locating your file, do a right-click and select "Open with" in the third step.
You will be allowed to select the downloaded version of GitHub Atom after selecting the "Open with" option. Click "OK" after selecting your software. You have now successfully opened your file on Windows.

How to Use Mac to Open a .JSONL File?

On a Mac, opening the .jsonl file only requires 4 steps.

The GitHub Atom software must be downloaded first. The file will be opened using this software. Apple TextEdit and GitHub Atom are two other pieces of software that may be used to open this file.
Finding the downloaded file comes next. If you are unsure of where you downloaded a file, you should look in your /download/ folder because there is typically where it is saved by default.
After locating your file, do a right-click and select "Open with" in the third step.
The GitHub Atom software that you downloaded should appear in the fourth step when you select "Open with." Click "OK" after selecting the software. You have now successfully opened your file on a Mac.

Conclusion

The complete JSON Lines file as a whole is technically no longer valid JSON because it contains several JSON strings.

JSON Lines is a desirable format for streaming data. The JSON Lines structured file can be streamed since each new line denotes a unique entry. The same number of lines can be read to obtain the same number of records.

To handle the JSON Lines format, you don't need to create a unique reader or writer. JSON Lines can be read well even with basic Linux command-line tools like head and tail.

Further Reading:

Application Containerization

Containers-as-a-Service (CaaS)

Encapsulation

Machine Data

Structured Logging

Monitor Your JavaScript Applications with Atatus

Atatus keeps track of your JavaScript application to give you a complete picture of your clients' end-user experience. You can determine the source of delayed load times, route changes, and other issues by identifying frontend performance bottlenecks for each page request.

To make bug fixing easier, every JavaScript error is captured with a full stack trace and the specific line of source code marked. To assist you in resolving the JavaScript error, look at the user activities, console logs, and all JavaScript events that occurred at the moment. Error and exception alerts can be sent by email, Slack, PagerDuty, or webhooks.

Try Atatus’s entire features free for 14 days.