Google Builders Weblog: MediaPipe KNIFT: Template-based Characteristic Matching


Posted by Zhicheng Wang and Genzhi Ye, MediaPipe crew

Picture Characteristic Correspondence with KNIFT

In lots of laptop imaginative and prescient purposes, an important constructing block is to determine dependable correspondences between completely different views of an object or scene, forming the inspiration for approaches like template matching, picture retrieval and construction from movement. Correspondences are normally computed by extracting distinctive view-invariant options comparable to SIFT or ORB from photos. The power to reliably set up such correspondences allows purposes like picture stitching to create panoramas or template matching for object recognition in movies (see Determine 1).

Right this moment, we’re asserting KNIFT (Keypoint Neural Invariant Characteristic Rework), a basic goal native function descriptor just like SIFT or ORB. Likewise, KNIFT can also be a compact vector illustration of native picture patches that’s invariant to uniform scaling, orientation, and illumination modifications. Nevertheless in contrast to SIFT or ORB, which have been engineered with heuristics, KNIFT is an embedding realized instantly from numerous corresponding native patches extracted from close by video frames. This information pushed method implicitly encodes complicated, real-world spatial transformations and lighting modifications within the embedding. In consequence, the KNIFT function descriptor seems to be extra sturdy, not solely to affine distortions, however to a point of perspective distortions as nicely. We’re releasing an implementation of KNIFT in MediaPipe and a KNIFT-based template matching demo within the subsequent part to get you began.

Determine 1: Matching an actual Cease Signal with a Cease Signal template utilizing KNIFT.

Coaching Methodology

In Machine Studying, loosely talking, coaching an embedding means discovering a mapping that may translate a excessive dimensional vector, comparable to a picture patch, to a comparatively decrease dimensional vector, comparable to a function descriptor. Ideally, this mapping ought to have the next property: picture patches round a real-world level ought to have the identical or very related descriptors throughout completely different views or illumination modifications. Now we have discovered actual world movies an excellent supply of such corresponding picture patches as coaching information (See Determine three and four) and we use the well-established Triplet Loss (see Determine 2) to coach such an embedding. Every triplet consists of an anchor (denoted by a), a constructive (p), and a detrimental (n) function vector extracted from the corresponding picture patches, and d() denotes the Euclidean distance within the function house.

Figure 2: Triplet Loss Function.

Determine 2: Triplet Loss Operate.

Coaching Knowledge

The coaching triplets are extracted from all ~1500 video clips within the publicly out there YouTube UGC Dataset. We first use an current heuristically-engineered native function detector to detect keypoints and compute the affine rework between two frames with a excessive accuracy (see Determine four). Then we use this correspondence to seek out keypoint pairs and extract the patches round these keypoints. Word that the newly recognized keypoints might embody people who have been detected however rejected by geometric verification in step one. For every pair of matched patches, we randomly apply some type of information augmentation (e.g. random rotation or brightness adjustment) to assemble the anchor-positive pair. Lastly, we randomly choose an arbitrary patch from one other video because the detrimental to complete the development of this triplet (see Determine 5).

Determine three: An instance video clip from which we extract coaching triplets.

Determine four: Discovering body correspondence utilizing current native options.

Determine 5: (Prime to backside) Anchor, constructive and detrimental patches.

Arduous-negative Triplet Mining

To enhance mannequin high quality, we use the identical hard-negative triplet mining methodology utilized by FaceNet coaching. We first prepare a base mannequin with randomly chosen triplets. Then we implement a pipeline that makes use of the bottom mannequin to seek out semi-hard-negative samples (d(a,p) < d(a,n) < d(a,p)+margin) for every anchor-positive pair (Determine 6). After mixing the randomly chosen triplets and hard-negative triplets, we re-train the mannequin with this improved information.

Determine 6: (Prime to backside) Anchor, constructive and semi-hard detrimental patches.

Mannequin Structure

From mannequin structure exploration, we have now discovered comparatively small structure is ample to attain first rate high quality, so we use a light-weight model of the Inception structure because the KNIFT mannequin spine. The ensuing KNIFT descriptor is a 40-dimensional float vector. For extra mannequin particulars, please check with the KNIFT mannequin card.


We benchmark the KNIFT mannequin inference pace on varied units (computing 200 options) and checklist them in Desk 1.

Table 1: KNIFT performance benchmark.

Desk 1: KNIFT efficiency benchmark.

High quality-wise, we examine the typical variety of keypoints matched by KNIFT and by ORB (OpenCV implementation) respectively on an in-house benchmark (Desk 2). There are numerous publicly out there picture matching benchmarks, e.g. 2020 Picture Matching Benchmark, however most of them give attention to matching landmarks throughout massive perspective modifications in comparatively excessive decision photos, and the duties typically require computing 1000’s of keypoints. In distinction, since we designed KNIFT for matching objects in massive scale (i.e. billions of photos) on-line picture retrieval duties, we devised our benchmark to give attention to low value and excessive precision pushed use circumstances, i.e. 100-200 keypoints computed per picture and solely ~10 matching keypoints wanted for reliably figuring out a match. As well as, for instance the fine-grained efficiency traits of a function descriptor, we divide and categorize the benchmark set by object varieties (e.g. 2D planar floor) and picture pair relations (e.g. massive measurement distinction). In desk 2, we examine the typical variety of keypoints matched by KNIFT and by ORB respectively in every class, primarily based on the identical 200 keypoint areas detected in every picture by the oFast detector that comes with the ORB implementation in OpenCV.

Desk 2: KNIFT vs ORB common variety of matched keypoints.

From Desk 2, we will see that KNIFT constantly matches extra keypoints than ORB by a big margin in each class. Right here we acknowledge the truth that KNIFT (40-d float) is significantly bigger than ORB (32-d char) and this could have an effort on matching high quality. However, most native function benchmarks don’t take descriptor measurement under consideration so we’ll comply with the conference right here.

To make it straightforward for builders to attempt KNIFT in MediaPIpe, we have now constructed a local-feature-based template matching answer (see implementation particulars utilizing MediaPipe within the subsequent part). As a aspect impact, we will show the matching high quality between KNIFT and ORB visually in side-by-side comparisons like Determine 7 and 9.

Determine 7: Instance of “matching 2D planar floor”. (Left) KNIFT 183/240, (Proper) ORB 133/240.

In Determine 7, we select a typical U.S. Cease Signal picture from Google Picture Search because the template and try to match it with the Cease Signal on this video. This instance falls into the “matching 2D planar floor” class in Desk 2. Utilizing the identical 200 keypoint areas detected by oFast and the identical RANSAC setting, we present that KNIFT is profitable at matching the Cease Register 183 frames out of a complete of 240 frames. Compared, ORB matches 133 frames.

Determine eight: Instance of “matching 3D untextured object”. Two template photos from completely different views.

Determine 9: Instance of “matching 3D untextured object”. (Left) KNIFT 89/150, (Proper) ORB 37/150.

Determine 9 exhibits one other matching efficiency comparability on an instance from the “matching 3D untextured object” class in Desk 2. Since this instance entails massive perspective modifications of untextured surfaces, which is understood to be difficult for native function descriptors, we use template photos from two completely different views (proven in Determine eight) to enhance the matching efficiency. Once more, utilizing the identical keypoint areas and RANSAC setting, we present that KNIFT is profitable at matching 89 frames out of a complete of 150 frames whereas ORB matches 37 frames.

KNIFT-based Template Matching in MediaPipe

We’re releasing the aforementioned template matching answer primarily based on KNIFT in MediaPipe, which is able to figuring out pre-defined picture templates and exactly localizing acknowledged templates on the digital camera picture. There are three main elements within the template-matching MediaPipe graph proven under:

FeatureDetectorCalculator: a calculator that consumes picture frames and performs OpenCV oFast detector on the enter picture and outputs keypoint areas. Furthermore, this calculator can also be accountable for cropping patches round every keypoint with rotation and scale data and stacking them right into a vector for the downstream calculator to course of.

TfLiteInferenceCalculator with KNIFT mannequin: a calculator that masses the KNIFT tflite mannequin and performs mannequin inference. The enter tensor form is (200, 32, 32, 1), indicating 200 32×32 native patches. The output tensor form is (200, 40), indicating 200 40-dimensional function descriptors. By default, the calculator runs the TFLite XNNPACK delegate, however customers have the choice to pick the common CPU delegate to run at a lowered pace.

BoxDetectorCalculator: a calculator that takes pre-computed keypoint areas and KNIFT descriptors and performs function matching between the present body and a number of template photos. The output of this calculator is an inventory of TimedBoxProto, which incorporates the distinctive id and site of every field as a quadrilateral on the picture. Apart from the basic homography RANSAC algorithm, we additionally apply a perspective rework verification step to make sure that the output quadrilateral doesn’t end in an excessive amount of skew or a bizarre form.

Determine 10: MediaPipe graph of the demo


On this demo, we selected three completely different denominations ($1, $5, $20) of U.S. greenback payments as templates and tried to match them to varied actual world greenback payments in movies. We resized every enter body to 640×480 pixels, ran the oFast detector to detect 200 keypoints, and used KNIFT to extract function descriptors from every 32×32 native picture patch surrounding these keypoints. We then carried out template matching between these video frames and the KNIFT options extracted from the greenback invoice templates. This demo runs at 20 FPS on a Pixel 2 Cellphone CPU with XNNPACK.

Determine 11: Matching completely different U.S. greenback payments utilizing KNIFT.

Construct Your Personal Templates

Now we have supplied a set of built-in planar templates in our demo. To make it straightforward for customers to attempt their very own templates, we additionally present a instrument to construct such an index with person generated templates. index_building.pbtxt is a MediaPipe graph that accepts as its enter a listing path containing a set of template photos. Customers can use this graph to compute KNIFT descriptors for all template photos (which will likely be saved in a single file) by 1) changing the index_proto_filename discipline in the principle graph and the BUILD file and a couple of) rebuilding the APK file. For step-by-step directions on how we created the greenback invoice demo proven above, please check with this documentation.


We wish to thank Jiuqiang Tang, Chuo-Ling Chang, Dan Gnanapragasam‎, Howard Zhou, Jianing Wei and Ming Guang Yong for contributing to this weblog submit.