Vuforia open web cam by itself while we have two ways(https://developer.vuforia.com/library/articles/Solution/How-To-Access-the-Camera-Image-in-Unity) to access the source video background and get it rectified!
- result
a stereo rectify result! (the test calibration reslut is not good)
- use the Image Class
we could get the background image like:
Image image = CameraDevice.Instance.GetCameraImage(Image.PIXEL_FORMAT.RGBA8888);
then we could copy it to a texture and get the data
webcam_texture = new Texture2D (weight, height, TextureFormat.RGB24, false);
webcam_data = new Color32[webcam_texture.width * webcam_texture.height];
image.CopyToTexture(webcam_texture);
webcam_data = webcam_texture.GetPixels32 ();
this data is passed down to opencv dll and rectify with cv::remap() function
however, this way is far to slow!!! copying the image to texture takes lots of time and I ahieve only 15fps in my environment.
- use and OpenGL texture
Vuforia provide an example with shader program to modify the image in the background plane. we could encod the calibration map to a texture and work as coordinate lookup table, which works in GPU and would achieve high frame rate( in my environemnt, it's up to 60fps, up to the frame rate of my camera ).
calibration maps is create like:
C++: void initUndistortRectifyMap(InputArray cameraMatrix, InputArray distCoeffs, InputArray R, InputArray newCameraMatrix, Size size, int m1type, OutputArray map1, OutputArray map2)
we usually use a CV_16SC2, mapx is a 2 channel 16-bit short, x and y coordinate. mapy is a 16-bit short, the fraction of interpolation. we change this maps to a 32 floating point each way cause GPU would interpolate itself with a floating point coordinate.
C++: void convertMaps(InputArray map1, InputArray map2, OutputArray dstmap1, OutputArray dstmap2, int dstmap1type, bool nninterpolation=false )¶
we create a two channel RG float texture as the lookup table
texture = new Texture2D (width, height, TextureFormat.RGFloat, false);
Material mat = GetComponent<Renderer>().material;
mat.SetTexture ("_Texture2", texture);
opencv dll open the calibration files, convert them to 32float, and passed them back to c sharp script as float array.
map_x1 = new float[width * height];
map_y1 = new float[width * height];
IntPtr ptr = get_map_x1 (opencv_wrapper_ptr);
Marshal.Copy (ptr, map_x1, 0, width * height);
ptr = get_map_y1 (opencv_wrapper_ptr);
Marshal.Copy (ptr, map_y1, 0, width * height);
then we feed the texture with normalized coordinate
for (int i=0; i<width; ++i) {
for(int j=0; j<height; ++j){
Color color;
color.r = map_x1[j*width+i]/(float)width;
color.g = map_y1[j*width+i]/(float)height;
color.b = 0.0f;
color.a = 1.0f;
texture.SetPixel(i,j,color);
}
}
In Shader file
_Texture2 ("Texture 2 (RGFLOAT)", 2D) = "white"{}
uniform sampler2D _Texture2;
in the fragment
//coordinate encoded in texture pixel
float2 coord = tex2D (_Texture2, i.uv);
half4 c = tex2D(_MainTex, coord);
done!!!!
- Future work
I am newbee to unity and shader language. I am not sure whether fetching a positin far away from current vertex would be slow(as I remember, this is very slow in mobile GPU with GLES). if so, the coordinate access should be computed in the vertex part not the fragment. but what I have come gots good framerate, so enjoy!!
原文链接: https://www.cnblogs.com/flankechen/p/5238159.html
欢迎关注
微信关注下方公众号，第一时间获取干货硬货；公众号内回复【pdf】免费获取数百本计算机经典书籍
原创文章受到原创版权保护。转载请注明出处：https://www.ccppcoding.com/archives/229537
非原创文章文中已经注明原地址，如有侵权，联系删除
关注公众号【高性能架构探索】，第一时间获取最新文章
转载文章受原作者版权保护。转载请注明原作者出处！
