VR – Look to Select

The simplest and possibly most common is the Google Cardboard style of VR which is basically a phone put into a face mask. This kind of VR has no controller so people have to come up with creative solutions for allowing user input.

The most common solution is “look to select” where if the user points their head at something for a moment it gets selected.

Let’s implement “look to select”! we’ll add the PickHelper we made in . 

class PickHelper {
  constructor() {
    this.raycaster = new THREE.Raycaster();
    this.pickedObject = null;
    this.pickedObjectSavedColor = 0;
  }
  pick(normalizedPosition, scene, camera, time) {
    // restore the color if there is a picked object
    if (this.pickedObject) {
      this.pickedObject.material.emissive.setHex(this.pickedObjectSavedColor);
      this.pickedObject = undefined;
    }
 
    // cast a ray through the frustum
    this.raycaster.setFromCamera(normalizedPosition, camera);
    // get the list of objects the ray intersected
    const intersectedObjects = this.raycaster.intersectObjects(scene.children);
    if (intersectedObjects.length) {
      // pick the first object. It's the closest one
      this.pickedObject = intersectedObjects[0].object;
      // save its color
      this.pickedObjectSavedColor = this.pickedObject.material.emissive.getHex();
      // set its emissive color to flashing red/yellow
      this.pickedObject.material.emissive.setHex((time * 8) % 2 > 1 ? 0xFFFF00 : 0xFF0000);
    }
  }
}

To use it we just need to create an instance and call it in our render loop

const pickHelper = new PickHelper();
 
...
function render(time) {
  time *= 0.001;
 
  ...
 
  // 0, 0 is the center of the view in normalized coordinates.
  pickHelper.pick({x: 0, y: 0}, scene, camera, time);

In the original picking we converted the mouse coordinates from CSS pixels into normalized coordinates that go from -1 to +1 across the canvas.

In this case though we will always pick where the camera is facing which is the center of the screen so we pass in 0 for both x and y which is the center in normalized coordinates.

And with that objects will flash when we look at them

Typically we don’t want selection to be immediate. Instead we require the user to keep the camera on the thing they want to select for a few moments to give them a chance not to select something by accident.

To do that we need some kind of meter or gauge or some way to convey that the user must keep looking and for how long.

One easy way we could do that is to make a 2 color texture and use a texture offset to slide the texture across a model.

Let’s do this by itself to see it work before we add it to the VR example.

 const left = -2;    // Use values for left
const right = 2;    // right, top and bottom
const top = 1;      // that match the default
const bottom = -1;  // canvas size.
const near = -1;
const far = 1;
const camera = new THREE.OrthographicCamera(left, right, top, bottom, near, far);

And of course update it if the canvas changes size

function render(time) {
  time *= 0.001;
 
  if (resizeRendererToDisplaySize(renderer)) {
    const canvas = renderer.domElement;
    const aspect = canvas.clientWidth / canvas.clientHeight;
    camera.left = -aspect;
    camera.right = aspect;
    camera.updateProjectionMatrix();
  }
  ...

We now have a camera that shows 2 units above and below the center and aspect units left and right.

Next let’s make a 2 color texture.

function makeDataTexture(data, width, height) {
  const texture = new THREE.DataTexture(data, width, height, THREE.RGBAFormat);
  texture.minFilter = THREE.NearestFilter;
  texture.magFilter = THREE.NearestFilter;
  texture.needsUpdate = true;
  return texture;
}
 
const cursorColors = new Uint8Array([
  64, 64, 64, 64,       // dark gray
  255, 255, 255, 255,   // white
]);
const cursorTexture = makeDataTexture(cursorColors, 2, 1);

We’ll then use that texture 

const ringRadius = 0.4;
const tubeRadius = 0.1;
const tubeSegments = 4;
const ringSegments = 64;
const cursorGeometry = new THREE.TorusGeometry(
    ringRadius, tubeRadius, tubeSegments, ringSegments);
 
const cursorMaterial = new THREE.MeshBasicMaterial({
  color: 'white',
  map: cursorTexture,
  transparent: true,
  blending: THREE.CustomBlending,
  blendSrc: THREE.OneMinusDstColorFactor,
  blendDst: THREE.OneMinusSrcColorFactor,
});
const cursor = new THREE.Mesh(cursorGeometry, cursorMaterial);
scene.add(cursor);

and then in render lets adjust the texture’s offset

function render(time) {
  time *= 0.001;
 
  if (resizeRendererToDisplaySize(renderer)) {
    const canvas = renderer.domElement;
    const aspect = canvas.clientWidth / canvas.clientHeight;
    camera.left = -aspect;
    camera.right = aspect;
    camera.updateProjectionMatrix();
  }
 
  const fromStart = 0;
  const fromEnd = 2;
  const toStart = -0.5;
  const toEnd = 0.5;
  cursorTexture.offset.x = THREE.MathUtils.mapLinear(
      time % 2,
      fromStart, fromEnd,
      toStart, toEnd);
 
  renderer.render(scene, camera);
}

THREE.MathUtils.mapLinear takes a value that goes between fromStart and fromEnd and maps it to a value between toStart and toEnd. In the case above we’re taking time % 2 which means a value that goes from 0 to 2 and maps that to a value that goes from -0.5 to 0.5

Textures are mapped to geometry using normalized texture coordinates that go from 0 to 1. That means our 2×1 pixel image, set to the default wrapping mode of THREE.ClampToEdge, if we adjust the texture coordinates by -0.5 then the entire mesh will be the first color and if we adjust the texture coordinates by +0.5 the entire mesh will be the second color. In between with the filtering set to THREE.NearestFilter we’ll be able to move the transition between the 2 colors through the geometry.

Let’s add a background texture We’ll just use a 2×2 set of colors but set the texture’s repeat settings to give us an 8×8 grid. This will give our cursor something to be rendered over so we can check it against different colors.

const backgroundColors = new Uint8Array([
    0,   0,   0, 255,  // black
   90,  38,  38, 255,  // dark red
  100, 175, 103, 255,  // medium green
  255, 239, 151, 255,  // light yellow
]);
const backgroundTexture = makeDataTexture(backgroundColors, 2, 2);
backgroundTexture.wrapS = THREE.RepeatWrapping;
backgroundTexture.wrapT = THREE.RepeatWrapping;
backgroundTexture.repeat.set(4, 4);
 
const scene = new THREE.Scene();
scene.background = backgroundTexture;

A few things to notice and try.

  • We set the cursorMaterial‘s blending, blendSrc and blendDst properties as follows
  blending: THREE.CustomBlending,
  blendSrc: THREE.OneMinusDstColorFactor,
  blendDst: THREE.OneMinusSrcColorFactor,
  • This gives as an inverse type of effect. Comment out those 3 lines and you’ll see the difference. I’m just guessing the inverse effect is best here as that way we can hopefully see the cursor regardless of the colors it is over.
  • We use a TorusGeometry and not a RingGeometry
  • For whatever reason the RingGeometry uses a flat UV mapping scheme. Because of this if we use a RingGeometry the texture slides horizontally across the ring instead of around it like it does above.
  • Try it out, change the TorusGeometry to a RingGeometry (it’s just commented out in the example above) and you’ll see what I mean.
  • The proper thing to do (for some definition of proper) would be to either use the RingGeometry but fix the texture coordinates so they go around the ring. Or else, generate our own ring geometry. But, the torus works just fine. Placed directly in front of the camera with a MeshBasicMaterial it will look exactly like a ring and the texture coordinates go around the ring so it works for our needs.

Let’s integrate it with our VR code above.

class PickHelper {
  constructor() {
  constructor(camera) {
    this.raycaster = new THREE.Raycaster();
    this.pickedObject = null;
    this.pickedObjectSavedColor = 0;
 
    const cursorColors = new Uint8Array([
      64, 64, 64, 64,       // dark gray
      255, 255, 255, 255,   // white
    ]);
    this.cursorTexture = makeDataTexture(cursorColors, 2, 1);
 
    const ringRadius = 0.4;
    const tubeRadius = 0.1;
    const tubeSegments = 4;
    const ringSegments = 64;
    const cursorGeometry = new THREE.TorusGeometry(
        ringRadius, tubeRadius, tubeSegments, ringSegments);
 
    const cursorMaterial = new THREE.MeshBasicMaterial({
      color: 'white',
      map: this.cursorTexture,
      transparent: true,
      blending: THREE.CustomBlending,
      blendSrc: THREE.OneMinusDstColorFactor,
      blendDst: THREE.OneMinusSrcColorFactor,
    });
    const cursor = new THREE.Mesh(cursorGeometry, cursorMaterial);
    // add the cursor as a child of the camera
    camera.add(cursor);
    // and move it in front of the camera
    cursor.position.z = -1;
    const scale = 0.05;
    cursor.scale.set(scale, scale, scale);
    this.cursor = cursor;
 
    this.selectTimer = 0;
    this.selectDuration = 2;
    this.lastTime = 0;
  }
  pick(normalizedPosition, scene, camera, time) {
    const elapsedTime = time - this.lastTime;
    this.lastTime = time;
 
    // restore the color if there is a picked object
    if (this.pickedObject) {
      this.pickedObject.material.emissive.setHex(this.pickedObjectSavedColor);
      this.pickedObject = undefined;
    }
 
    const lastPickedObject = this.pickedObject;
    this.pickedObject = undefined;
 
    // cast a ray through the frustum
    this.raycaster.setFromCamera(normalizedPosition, camera);
    // get the list of objects the ray intersected
    const intersectedObjects = this.raycaster.intersectObjects(scene.children);
    if (intersectedObjects.length) {
      // pick the first object. It's the closest one
      this.pickedObject = intersectedObjects[0].object;
      // save its color
      this.pickedObjectSavedColor = this.pickedObject.material.emissive.getHex();
      // set its emissive color to flashing red/yellow
      this.pickedObject.material.emissive.setHex((time * 8) % 2 > 1 ? 0xFFFF00 : 0xFF0000);
    }
 
    // show the cursor only if it's hitting something
    this.cursor.visible = this.pickedObject ? true : false;
 
    let selected = false;
 
    // if we're looking at the same object as before
    // increment time select timer
    if (this.pickedObject && lastPickedObject === this.pickedObject) {
      this.selectTimer += elapsedTime;
      if (this.selectTimer >= this.selectDuration) {
        this.selectTimer = 0;
        selected = true;
      }
    } else {
      this.selectTimer = 0;
    }
 
    // set cursor material to show the timer state
    const fromStart = 0;
    const fromEnd = this.selectDuration;
    const toStart = -0.5;
    const toEnd = 0.5;
    this.cursorTexture.offset.x = THREE.MathUtils.mapLinear(
        this.selectTimer,
        fromStart, fromEnd,
        toStart, toEnd);
 
    return selected ? this.pickedObject : undefined;
  }
}

You can see the code above we added all the code to create the cursor geometry, texture, and material and we added it as a child of the camera so it will always be in front of the camera. Note we need to add the camera to the scene otherwise the cursor won’t be rendered.

scene.add(camera);

We then check if the thing we’re picking this time is the same as it was last time. If so we add the elapsed time to a timer and if the timer reaches its limit we return the selected item.

Now let’s use that to pick the cubes. As a simple example we’ll add 3 spheres as well. When a cube is picked with hide the cube and un-hide the corresponding sphere.

So first we’ll make a sphere geometry

const boxWidth = 1;
const boxHeight = 1;
const boxDepth = 1;
const geometry = new THREE.BoxGeometry(boxWidth, boxHeight, boxDepth);
const boxGeometry = new THREE.BoxGeometry(boxWidth, boxHeight, boxDepth);
 
const sphereRadius = 0.5;
const sphereGeometry = new THREE.SphereGeometry(sphereRadius);

Then let’s create 3 pairs of box and sphere meshes. We’ll use a Map so that we can associate each Mesh with its partner.

const cubes = [
  makeInstance(geometry, 0x44aa88,  0),
  makeInstance(geometry, 0x8844aa, -2),
  makeInstance(geometry, 0xaa8844,  2),
];
const meshToMeshMap = new Map();
[
  { x:  0, boxColor: 0x44aa88, sphereColor: 0xFF4444, },
  { x:  2, boxColor: 0x8844aa, sphereColor: 0x44FF44, },
  { x: -2, boxColor: 0xaa8844, sphereColor: 0x4444FF, },
].forEach((info) => {
  const {x, boxColor, sphereColor} = info;
  const sphere = makeInstance(sphereGeometry, sphereColor, x);
  const box = makeInstance(boxGeometry, boxColor, x);
  // hide the sphere
  sphere.visible = false;
  // map the sphere to the box
  meshToMeshMap.set(box, sphere);
  // map the box to the sphere
  meshToMeshMap.set(sphere, box);
});

In render where we rotate the cubes we need to iterate over meshToMeshMap instead of cubes.

cubes.forEach((cube, ndx) => {
let ndx = 0;
for (const mesh of meshToMeshMap.keys()) {
  const speed = 1 + ndx * .1;
  const rot = time * speed;
  cube.rotation.x = rot;
  cube.rotation.y = rot;
});
  mesh.rotation.x = rot;
  mesh.rotation.y = rot;
  ++ndx;
}

And now we can use our new PickHelper implementation to select one of the objects. When selected we hide that object and un-hide its partner.

// 0, 0 is the center of the view in normalized coordinates.
pickHelper.pick({x: 0, y: 0}, scene, camera, time);
const selectedObject = pickHelper.pick({x: 0, y: 0}, scene, camera, time);
if (selectedObject) {
  selectedObject.visible = false;
  const partnerObject = meshToMeshMap.get(selectedObject);
  partnerObject.visible = true;
}

Leave a comment

Your email address will not be published. Required fields are marked *