How large files open
JSON Digger does not load a file into memory. It indexes the file, noting where each object, array, and value is, and reads the parts you look at from disk. Indexing a multi-gigabyte file takes seconds, so JSON Digger shows the beginning of the file first and indexes the rest in the background.
This applies to JSON and NDJSON files. A plain text file is indexed by line.
Progressive indexing
Section titled “Progressive indexing”When you open a JSON or NDJSON file for the first time, JSON Digger indexes its beginning and shows the tree at once. The rest of the file is indexed in the background. On one test machine, the full index of a 1.1 GB file took 1.4 seconds; see Performance for more sizes.
While the background indexing runs, the status bar shows a spinner and indexing… N%, and its node and record counts grow until indexing finishes.
Results while indexing
Section titled “Results while indexing”Most features work before indexing finishes, some with partial results:
| Feature | While indexing |
|---|---|
| Tree | Shows the indexed part. An object or array that continues past it looks shorter until indexing finishes. |
| Search | Scans the whole file. The status line adds “may be partial, indexing in progress”; run the search again after indexing for full coverage. |
| jq, JSONPath, copy, validation | Work on nodes inside the indexed part. On a node that continues past it, such as the root of a JSON file, they fail with “value is still being indexed; try again when indexing completes”. On an NDJSON file, jq covers only the records indexed so far, without a warning: run it again after indexing. |
| SQL | Works on an NDJSON file, a root array, and an array inside an NDJSON record. An array inside a JSON file can be queried once indexing finishes. |
| Editing | Waits: “The file is still being indexed.” |
Quitting JSON Digger during indexing asks for confirmation, because the partial index is discarded.
When indexing finishes, the full index replaces the partial one. The counts update, the nodes you expanded stay expanded, and search matches found past the indexed part get their place in the tree.
The index cache
Section titled “The index cache”JSON Digger saves the index of each file it opens to its index cache. The next time you open the same file, it loads the saved index instead of indexing again, so even a file of several gigabytes opens in a fraction of a second. On one test machine, a 9 GB file that took 16.5 seconds to index opened in 53 ms from the cache.
When the file has changed since, JSON Digger indexes it again. No index is saved for a malformed file.
The jdig terminal viewer reads and writes the same cache, so a file indexed by one app opens fast in the other.
The cache has no size limit. To see where it is and how to reclaim its space, see Index cache.
Very large objects and arrays
Section titled “Very large objects and arrays”The tree shows the first 4,096 members of an object or elements of an array. A larger container shows a badge such as first 4,096 of 1,250,000 shown. Search, queries, and the Table view cover every child, not only the ones the tree shows.
In an NDJSON file, the same limit applies to records: the tree shows the first 4,096 records. When a search result or a query result is a record past them, JSON Digger opens it in the NDJSON view at its line instead.
Expanding a node with all its descendants, by double-clicking it or with Expand from Here, stops early on a very large node, and a notice says so.
For every size limit in one place, see Limits.