Image Manipulation (1)

Runtime, client-side, Promise-chain

canvas → context → base64-String

Promise.resolve().then( function() {
    return navigator.getPicture() // or whatever returns base64
).then( function(sBase64) { // manipulate image data
    return promisifiedAndManipulatedImage();
}).then( function(sBase64Modified) { // set it to UI5 control
    this.byId("idImage").setSrc(sBase64Modified);
}).catch(function (oError) { // never forget -> debugging!
    jQuery.sap.log.error(oError);
})

Image Manipulation (2)

(shortened for brevity)

promisifiedAndManipulatedImage: function(sBase64) {
    return new Promise(function (fnResolve, fnReject) {
        // wrap img.onload as promise
        var oImage = new Image();
        oImage.src = sBase64;
        oImage.onload = function () {
            // draw image data to an (invisible) canvas
            this.oContext.drawImage(this); // this.oCanvas needs to already live
            // retrieve ready-to-manipulate pixels
            var oImgData = this.oContext.getImageData();
            // iterate over every pixel in the image
            oImgData.data.map(function (data, i) {
                // MANIPULATION HERE...
            }
            this.oContext.putImageData(oImgData);
            // desired media-type and quality
            fnResolve(this.oCanvas.toDataURL("image/png", 1.0));
        }
    // ... fnReject for oImage.onerror() etc.
    });
}

Pattern Matching

Gaussian blur → Grayscale → FAST → BRIEF|ORB

• Binary Robust Independent Elementary Features (BRIEF) - tracking.js
• Oriented FAST and Rotated BRIEF (ORB) - JSFeat
• Features from Accelerated Segment Test (FAST) - tracking.js

Pattern Matching

Pro | Con

collecting items in locations

idea:
reference object

• match object in photo/live video
• record geo coordinates
• record time

→ provide as push-based AR scenario

push-based AR scenario

navigator.watchPosition()

• select recorded $timestamp
  (bind time to UI5 control)
• onCoordUpdate:
  $object at $lat+$long?
• (yes) blend $object into camera
  (bind space to UI5 control)

many many many issues (1)

navigator.

.*   ⚡accuracy

.getCurrentPosition, .watchPosition
☘ GPRS   ☔ WLAN

.heading flunky → ☔ VR img placement

many many many issues (2)

web-based AR "framework"

motion tracking, projecting

track movement across devices and
project target object

Desktop ←WebSocket→ mobile

motion tracking

Grayscale → Gaussian blur → Canny Edge → Lucas-Kanade

• navigator.getUserMedia()
• requestAnimationFrame(fnProjection)
• fnProjection =~ oCanvas
   .getContext("2d")
   .putImageData()

web socket image data transfer

// sender
var oImgData = oCanvas.getContext("2d")
                      .getImageData(0, 0, oCanvas.width, oCanvas.height);
WebSocket.send(JSON.stringify({oImgData});
// receiver
var oImgData = JSON.parse(oControlEvent.getParameter("data"))
                        .data;
var oNewImageData = oContext.createImageData(oCanvas.width, oCanvas.height);
_.map(oImgData, function (data, i) { // underscore helper
    oNewImageData.data[i] = data;
});
oContext.putImageData(oNewImageData, 0, 0);
                    

on mobile: websockets closed (at will?) on inactivity!

projecting

• ($canvas * 4).rotateClockWise()
• "bind" $canvas to WebSocket.attachMessage
• craft hologram device helper thingie

Summary

Image manipulation (+ matching)

• canvas, context(2d), base64-handling
• seamless UI5-integration
• matching: it's all math
• UI5 & Promises: awesome

⇒ rock-solid, production ready
(matching: not)

Record + AR-ify time + space

• UI5 binding with time and space - cool
• HTTPS!!!
• all WebRTC (navigator.getUserMedia)
  ☔iOS <= 10, IE (!Edge)

⇒ insufficient tooling, playground only

Extending UX → mobile

• UI5 event-binding works with WebSockets
• rotate dynamic canvas → intermediate Image
• sidestep UI5-lifecycle to avoid re-rendering issues

⇒ gorgeous nonsense ☺
no idea where someone can take this ☄