Skip to content
FrostyBee Docs LogoFrostyBee Docs

Data Quality

Analyze checks every row of a CSV or TSV file for two problems:

  • Ragged rows: rows with a different number of fields from the header row. In a file without a header row, Vaste compares with the field count that most rows have. A blank line counts as a ragged row.
  • Duplicate rows: rows that repeat an earlier row exactly. The first copy is not flagged, only the repeats. Rows that differ only in quoting or spaces are not duplicates.

Data quality works in Table view for CSV and TSV files. It is not available for tables made from NDJSON files: there, Analyze table in the More menu and the F8 keys show Analysis failed: File is not CSV/TSV.

Analyze the table

Section titled “Analyze the table”

Use any of these:

  • Select Analyze in the Table view toolbar.
  • Select More actions (…) → Analyze table.
  • Run Analyze table from the command palette.
  • Press F8 or Alt + F8 to go to the next problem row. If the table has not been analyzed, Vaste analyzes it first.

While Vaste works, the table state bar shows Analyzing… N% and a Cancel button. When it finishes, a notification sums up the result, for example Analyzed 250,000 rows in 3.2 s: 12 ragged, 40 duplicate. Running Analyze again checks the file again.

Read the results

Section titled “Read the results”

The table state bar shows a chip for each problem: N ragged rows and N duplicate rows. Hover over a chip for details: the expected field count for ragged rows, or how many different rows repeat for duplicates. A chip with no rows is dimmed.

In the table, a colored mark in the row number column flags each ragged or duplicate row.

For a file with more than 50,000,000 rows, Vaste checks only the first 50,000,000 for duplicates, and the duplicate chip adds first 50,000,000 rows checked. Every row is still checked for its field count.

Go to problem rows

Section titled “Go to problem rows”
Key Command
F8 Next ragged row
Shift + F8 Previous ragged row
Alt + F8 Next duplicate row
Alt + Shift + F8 Previous duplicate row

You can also select the text of a chip to go to the next row of that kind, or right-click a row and select Next ragged row or Next duplicate row.

Vaste starts from the active row, or from the rows on screen, and goes back to the start of the file after the last problem row. The row becomes the active row in the middle of the table. If the table has no rows of that kind, Vaste shows No ragged rows or No duplicate rows.

Go to the original of a duplicate

Section titled “Go to the original of a duplicate”

To see the row that a duplicate repeats, right-click the duplicate row and select Show original row, or make it the active row and run Show original row from the command palette. The original is the first copy of the row in the file. It becomes the active row in the middle of the table.

If a filter hides the original, for example Show only these rows on the duplicate chip, the table does not move. Vaste shows The original is row followed by the original’s row number in the unfiltered table. Remove the filter to go to it.

Show only problem rows

Section titled “Show only problem rows”

Select the funnel button on a chip, Show only these rows, to filter the table to ragged or duplicate rows. Select it again, Show all rows again, to remove the filter. You can also run Show only ragged rows or Show only duplicate rows from the command palette.

This filter combines with column filters: the table shows the problem rows that also match the column filters. Reset view in the table state bar removes it.

When results are kept

Section titled “When results are kept”

Vaste saves the results with the file. When you open the file again, the chips appear at once, without a new analysis. The results are dropped, and you need to analyze again, when:

  • the file’s content changes, for example after a reload that changed or added rows,
  • you change the CSV dialect or the header row, or
  • you reopen the file with another encoding.

Next steps

Section titled “Next steps”