opencv.js perspective transform

I’m trying to use opencv.js to find a document in a provided image (detect edges, apply perspective transform, etc.

I’ve got a reasonable set of code that (occasionally) detects edges of a document and grabs the bounding box for that. However, I’m struggling to do the perspective transform steps. There are some helpers for this (not in JS) here and here.

Unfortunately I’m getting stuck on something simple. I can find the matching Mat that has 4 edges. Displaying that shows it to be accurate. However, I have no idea how to get some simple X/Y info out of that Mat. I thought minMaxLoc() would be a good option, but I keep getting an error passing in my matching Mat. Any idea why I can draw foundContour and get bounding box info from it, but I can’t call minMaxLoc on it?

Code:

//<Get Image>
//<Convert to Gray, do GaussianBlur, and do Canny edge detection>
let contours = new cv.MatVector();
cv.findContours(matDestEdged, contours, hierarchy, cv.RETR_LIST, cv.CHAIN_APPROX_SIMPLE);

//<Sort resulting contours by area to get largest>

let foundContour = null;
for (let sortableContour of sortableContours) {
  let peri = cv.arcLength(sortableContour.contour, true);
  let approx = new cv.Mat();
  cv.approxPolyDP(sortableContour.contour, approx, 0.1 * peri, true);

  if (approx.rows == 4) {
    console.log('found it');
    foundContour = approx
    break;
  }
  else {
    approx.delete();
  }
}

//<Draw foundContour and a bounding box to ensure it's accurate>

//TODO: Do a perspective transform
let result = cv.minMaxLoc(foundContour);

The last line above results in a runtime error (Uncaught (in promise): 6402256 - Exception catching is disabled). I can run minMaxLoc() on other Mat objects.

Answer

For anyone else looking to do this in OpenCV.JS, what I commented above seems to still be accurate. The contour found can’t be used with minMaxLoc, but the X/Y data can be pulled out of data32S[]. That should be all that’s needed to do this perspective transform. Some code is below.

//Find all contours
let contours = new cv.MatVector();
let hierarchy = new cv.Mat();
cv.findContours(matDest, contours, hierarchy, cv.RETR_LIST, cv.CHAIN_APPROX_SIMPLE);

//Get area for all contours so we can find the biggest
let sortableContours: SortableContour[] = [];
for (let i = 0; i < contours.size(); i++) {
  let cnt = contours.get(i);
  let area = cv.contourArea(cnt, false);
  let perim = cv.arcLength(cnt, false);

  sortableContours.push(new SortableContour({ areaSize: area, perimiterSize: perim, contour: cnt }));
}

//Sort 'em
sortableContours = sortableContours.sort((item1, item2) => { return (item1.areaSize > item2.areaSize) ? -1 : (item1.areaSize < item2.areaSize) ? 1 : 0; }).slice(0, 5);

//Ensure the top area contour has 4 corners (NOTE: This is not a perfect science and likely needs more attention)
let approx = new cv.Mat();
cv.approxPolyDP(sortableContours[0].contour, approx, .05 * sortableContours[0].perimiterSize, true);

if (approx.rows == 4) {
  console.log('Found a 4-corner approx');
  foundContour = approx;
}
else{
  console.log('No 4-corner large contour!');
  return;
}

//Find the corners
//foundCountour has 2 channels (seemingly x/y), has a depth of 4, and a type of 12.  Seems to show it's a CV_32S "type", so the valid data is in data32S??
let corner1 = new cv.Point(foundContour.data32S[0], foundContour.data32S[1]);
let corner2 = new cv.Point(foundContour.data32S[2], foundContour.data32S[3]);
let corner3 = new cv.Point(foundContour.data32S[4], foundContour.data32S[5]);
let corner4 = new cv.Point(foundContour.data32S[6], foundContour.data32S[7]);

//Order the corners
let cornerArray = [{ corner: corner1 }, { corner: corner2 }, { corner: corner3 }, { corner: corner4 }];
//Sort by Y position (to get top-down)
cornerArray.sort((item1, item2) => { return (item1.corner.y < item2.corner.y) ? -1 : (item1.corner.y > item2.corner.y) ? 1 : 0; }).slice(0, 5);

//Determine left/right based on x position of top and bottom 2
let tl = cornerArray[0].corner.x < cornerArray[1].corner.x ? cornerArray[0] : cornerArray[1];
let tr = cornerArray[0].corner.x > cornerArray[1].corner.x ? cornerArray[0] : cornerArray[1];
let bl = cornerArray[2].corner.x < cornerArray[3].corner.x ? cornerArray[2] : cornerArray[3];
let br = cornerArray[2].corner.x > cornerArray[3].corner.x ? cornerArray[2] : cornerArray[3];

//Calculate the max width/height
let widthBottom = Math.hypot(br.corner.x - bl.corner.x, br.corner.y - bl.corner.y);
let widthTop = Math.hypot(tr.corner.x - tl.corner.x, tr.corner.y - tl.corner.y);
let theWidth = (widthBottom > widthTop) ? widthBottom : widthTop;
let heightRight = Math.hypot(tr.corner.x - br.corner.x, tr.corner.y - br.corner.y);
let heightLeft = Math.hypot(tl.corner.x - bl.corner.x, tr.corner.y - bl.corner.y);
let theHeight = (heightRight > heightLeft) ? heightRight : heightLeft;

//Transform!
let finalDestCoords = cv.matFromArray(4, 1, cv.CV_32FC2, [0, 0, theWidth - 1, 0, theWidth - 1, theHeight - 1, 0, theHeight - 1]); //
let srcCoords = cv.matFromArray(4, 1, cv.CV_32FC2, [tl.corner.x, tl.corner.y, tr.corner.x, tr.corner.y, br.corner.x, br.corner.y, bl.corner.x, bl.corner.y]);
let dsize = new cv.Size(theWidth, theHeight);
let M = cv.getPerspectiveTransform(srcCoords, finalDestCoords)
cv.warpPerspective(matDestTransformed, finalDest, M, dsize, cv.INTER_LINEAR, cv.BORDER_CONSTANT, new cv.Scalar());

For reference, here is the class definition I was using for SortableContour. The code above is meant as a guide, not as something that can run on its own, however.

export class SortableContour {
    perimiterSize: number;
    areaSize: number;
    contour: any;
  
    constructor(fields: Partial<SortableContour>) {
      Object.assign(this, fields);
    }
  }