Skip to content

A WeChat MiniProgram Image AR using computer vision. No OpenCV, No Deep Learning. Only JavaScript Implementation.

License

Notifications You must be signed in to change notification settings

sanyuered/WeChat-MiniProgram-WebAR

Repository files navigation

  1. Chinese README

  2. Chinese Source Code Analysis

Updated

Date    Update
2022-07-23 New: Use an Experimental Worker on iOS. Please see the "Image AR and 3D Mask". Fixed Issue: "Image AR" cannot run on iOS for a while.
2021-12-11 New: Added a Image Tracker using OpenCV WebAssembly. Please see the "Image AR using OpenCV".
2021-09-07 New: Added a Color Tracker mode. Please see the "Color AR".
2021-08-15 New: Added a video mask mode for image AR. This is a CSS 3D transformation, which does not require three.js. Please see the "Image AR and Video Mask". Update: Replace the spirit geometry with a plane geometry.
2021-04-03 Update: The "access a camera" mode of Image AR is recoverd.
2021-03-15 New: The display of the AR mask is changed from 2D to 3D by three.js. Update: Because the "access a camera" mode is slow, it is removed.
2019-08-16 Update: The project structure has been modified. The color tracker and object tracker are removed. Fix "access a camera" mode that does not work properly on Android.
2019-08-06 Fix Issue: When function "wx.canvasToTempFilePath" is called frequently on Android Wechat, WeChat will be crashed.
2019-08-01 Update: The perspective transform is achieved.
2019-07-15 Update: The NFT(Natural Feature Tracking) is achieved.
2019-07-08 New: The affine transform is achieved.

Introduction on WeChat Web AR

This is a WeChat Web AR. On July 5, 2019, WeChat miniprogram supported AR. It was added a new API named "CameraFrameListener".

CameraFrameListener API

We can create AR effects with the new API. This demo demonstrates a AR tracker effect using "tracking.js" and "jsfeat" library.

The "tracking.js" brings computer vision algorithms and techniques into browser environment. The "jsfeat" is also a JavaScript computer vision library. We can do real-time image and face detection.

tracking.js and JSFeat

Index Page of the WeChat Mini-program

avatar

avatar

Image AR and 3D Mask

iOS Wechat version number 8.0.24 or above:It will use an Experimental Worker.

Android and other: It will not use an Experimental Worker.

Use the demo to scan a pattern image below.

avatar

A cat beard is on the pattern image.

avatar

A effect of translating and scaling.

avatar

A effect of rotating.

avatar

Image AR using OpenCV

This is the same as above. Supports image rotation.

The image is rotated by 30 degrees.

avatar

The image is rotated by 90 degrees.

avatar

Image AR and Video Mask

Use the demo to scan a rotating image below.

avatar

A video player is on and aligned with the image.

avatar

Color AR

Use the demo to scan a yellow color. Expect a effect below.

avatar

A effect of hiddening.

avatar

Custom the color of the highlighted area.

avatar

Face AR

Use the demo to scan a face. Expect a effect below.

avatar

A effect of translating and scaling.

avatar

Because the landmarks of the demo are simple and basic, only a effect of translating and scaling is on a rotating image.

avatar

How to replace the "cat beard" image

You may replace the default url of a image for 2D mask.

File: /package_image_tracker/pages/photo/photo.js and package_face_tracker/pages/photo/photo.js

// a url of sprite image
const modelUrl = '../../utils/cat_beard.png';

The width and height of the "modelurl" image should be 256 x 256, 512 x 512 and 1024 x 1024 etc.

How to replace the pattern image for Image AR

File: /package_face_tracker/utils/imageBusiness.js

const patternImageUrl = '../../../face_pattern.jpg';

a pattern image

avatar

How to put a image on other positions for Image AR

Select a track point on a pattern image, the point is used to set the "cat beard" image.

File: /package_image_tracker/utils/modelBusiness.js

// a index of a track point on a pattern image
const trackPoint = { 
    x: 185, // the width of the pattern image is 375
    y: 224, // the height of the pattern image is 375
};

How to put a image on other positions for Face AR

This is a map of the 31 keypoints of a face.

landmarks

avatar

For example, a number 27 and number 29 are the sides of the mouth.

File: /package_face_tracker/utils/modelBusiness.js

// index of the track points of the face
const trackPointA = {
    // index of a landmark
    id: 27,
    // X coordinate
    x: 155.69898111309, // the width of the face image is 375
};
const trackPointB = {
    // index of a landmark
    id: 29,
    // X coordinate
    x: 216.53075265284997, // the width of the face image is 375
};

Known Issues

The AR demo is very slow on iOS WeChat.

About

A WeChat MiniProgram Image AR using computer vision. No OpenCV, No Deep Learning. Only JavaScript Implementation.

Topics

Resources

License

Stars

Watchers

Forks