Skip to content
Advertisement

Is there an easy way to stop d3.csvParse when it hits a blank line?

I have CSV that has data 20ish rows of data I want, a few blank rows, and then some other stuff that other people are calculating off the data that they care about. Am using d3.csvParse and it picks up the extra lines. Is there any way to tell it to stop when it hits a blank row or can this only be handled in post-processing?

data = d3.csvParse(await FileAttachment("Coverage Data.csv").text(), d3.autoType)

Advertisement

Answer

d3.csvParse provides the ability to pass a row conversion function (see dsv.Parse) which you can use to check for, and eliminate, blank rows.

You can define a blankChecker function which still incorporates d3.autoType and pass that function as the row argument. It can work for blank rows either in the middle, or at the end of, the input.

See below with a literal string as the input:

const csv = `foo,bar,baz
0,"a",10
1,"b",20
2,"c",30
,,
4,"d",40
5,"e",50
,,
,,
8,"f",60
,,
,,
,,
,,`;

const blankChecker = (d, i) => {
  const blankRow = Object.keys(d).every(k => !d[k]);
  if (!blankRow) return d3.autoType(d);
  console.log(`Row ${i} has all blank values`);
}

const data = d3.csvParse(csv, blankChecker);

console.log(data);
.as-console-wrapper { max-height: 100% !important; top: 0; }
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/6.5.0/d3.min.js"></script>

If you want to stop the parse on the first appearance of a blank row, you can define a similar function to do that. In the example below blankQuitter sets a processing flag to false if no further rows should be returned:

const csv = `foo,bar,baz
0,"a",10
1,"b",20
2,"c",30
,,
4,"d",40
5,"e",50
,,
,,
8,"f",60
,,
,,
,,
,,`;

let processing = true;
const blankQuitter = (d, i) => {
  const blankRow = Object.keys(d).every(k => !d[k]);
  if (blankRow) processing = false;
  if (processing) return d3.autoType(d);
}

const data = d3.csvParse(csv, blankQuitter);

console.log(data);
.as-console-wrapper { max-height: 100% !important; top: 0; }
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/6.5.0/d3.min.js"></script>
Advertisement