Lunarium 2 : Adventure Game Research

 As I was beginning to think about how cut scenes might be seamlessly integrated into the VR experience I thought about all of the really great adventure games I have played over the years. Although I will continue this research to hone my game, I began by storyboarding Monkey Island By watching a walkthrough on YouTube and then continued by watching walkthroughs Of Grim Fandango, The Day of the Tentacle and The Dig. 

I had researched puzzle adventure App such as Lumocity which tells the story of how the player is trying to track their grandads whereabouts but looking at these classic examples of really engaging storytelling was pretty inspirational and sets the bar really high for the project. 

Each game walkthrough demonstrated how cutscenes work to effectively progress the game narrative within the game play.

Monkey Island Walkthrough 

The Making of Monkey Island 

Monkey Island

Grim Fandango 

The Day of the Tentacle 

The Dig 

Lunarium 2 : materials

 A big part of this project as that the original models are created physically from paper and then translated into cgi (or vice versa) . It’s a kind of experimental ping pong which has been going on in my work process since 2007. I am interested in applying material thinking to cgi and conversely cgi modelling to physical materials. 

I love making physical models out of paper and I really enjoy the challenge of using this mindset in digital world building.

Drawn sketch of the Alien Death Character 

Unity Probuilder Alien Death Character 

Paper model Alien Death Character (in progress)

I figure that one thing that Covid has taken from us (apart from our smiles) is our sense of close proximity to others and our sense of touch.

So aside from the technical challenges that Unity presents to me on a daily basis, I am currently exploring tactile and tangible touch in my game design. 

Lunarium 2 : Cut Scene – Cinemachine & Timeline

Virtual Camera, Dolly, Cart & Track 

I am producing a short cut scene for the Top Tent scene finale. I have a rough set of storyboards and have a basic concept for the short film. 

So far, I have added the Cinematic Brain, two virtual cameras and a dolly, cart and track to the scene. 

The Virtual Camera tracks maintains focus on the Alien Death character while tracking around the tent in a looped motion. 

 

 

Lunarium 2 : Hand Interaction


It has been tricky to get the kind of hand interaction I was looking for. Using the Oculus Integration plug in on Unity 2019.4 LTS I was able to initially indicate hand interaction by representing the hands as oculus quest controllers. I didn’t really want this as I would like the player to have their own hands represented as they begin to pick up and interact with objects in the environment. 

OVR Hand Prefabs

I added the OVR left and right hand prefabs into the scene. I’d like to use a custom designed hand material for Lunarium and this will go on the development backlog. 

Next Steps: 

Current Object Interaction ispoor and I would like to really experiement with various options which I hope will include animated objects, fixed object holds, sound effects, visual effects, and gravity. 

  • Create 3 Balls for the Knock Em out Teeth Game Stall 
  • Create Teeth to interact with the balls 
  • Create a counter that will count the teeth as they fall 
  • If successful the player will gain a silver coin. 

Lunarium 2 : Quest Link (Rift) Setup and Navigation

Lunarium 2 

Quest Link (Rift) Development Environment

Lunarium 2 has been set up for rift due to the changes expected in December to have a 2 tier publishing program (per advice from occulus). In January I should be able to reformat for quest and begin the process of publishing on the oculus store (This could take yonks.. yes, yonks is in the dictionary)   

The navigation is now controlled by the left hand thumbstick and snapturns on the right thumbstick. The speed has been slowed. The player can also direct where they would like to go via their headset direction but this can be easily overriden by the thumbsticks. 

I might add a subtle head bob and footsteps sound but this will be fine for now. 

 

 

Lunarium 2 : Machine Learning Agent Experiment Day 3

 

This is how my hummingbird works In training Mode đź‘Ť

I have run into a few glitches 

  • On starting,  the main camera does not change to player camera. 
  • The ml-agent flies into the air and hits the environment boundary and stays there. It looks like it’s trying to train again. 
  • The player counter is not working 
Possible fixes 
  • Check on start Code to see what camera are set up and what happens. 
  • Check neural network file is correct. 
  • Check counter code. 
This has been fixed and now plays.. the controllers are clmbsy but at least its working! 

I should have a working prototype I can modify to work within Lunarium now. 

Lunarium 2: Machine Learning Agent Experiment Day 2

 Machine Learning Hummingbird agent
 Scene set up and script walkthrough to understand essential 
Components that make up the machine learning set up and functionality.  
Ray set up, down and front observes objects in the scene
Similar to Lidar 
Rays are set away from beak to avoid false detection 

Environment boxed in to facilitate bird training 

Nectar Full Flower 

On Empty, Flower changes colour 
Download and install Anaconda 
Creating a new environment 
Python 3.7 was incorrect : use Python 3.8 (64 bit) 
Download latest update from pip – use:
pip install six==1.10.0
$ pip install cherrypy==11.0.0
Fix Courtesy of  Pradyuns
;conta activatr ml-agents -3.0 activates this environment 
Pypi is deprecated 
Set up 8 Training Environments identical to each other.
Ensure Hummingbird is spelt correctly in behaviour 
Use this yml file instead 
behaviors:
  Hummingbird:
    trainer_type: ppo
    hyperparameters:
      batch_size: 2048
      buffer_size: 20480
      learning_rate: 0.0003
      beta: 0.005
      epsilon: 0.2
      lambd: 0.95
      num_epoch: 3
      learning_rate_schedule: linear
    network_settings:
      normalize: false
      hidden_units: 256
      num_layers: 2
      vis_encode_type: simple
    reward_signals:
      extrinsic:
        gamma: 0.99
        strength: 1.0
    keep_checkpoints: 5
    checkpoint_interval: 500000
    max_steps: 5000000
    time_horizon: 128
    summary_freq: 10000
    threaded: true
Yml file courtesy of Point_Nemo on Unity Github
When correct the unity logo will be displayed. 
One you see go to Unity and press play, you should do this immediately. 
If correct you will begin to see data downloading from the hummingbird 
It will take time to see it but the bird will begin to learn about its environment and the flowers within it as it starts to look for nectar. 
Use tensorboard—logdir results instead 
Represents the same data as a graph for ease of reference 
Here it begins to show the hummingbird learning how to successfully locate the nectar.
 
There is a lot of information here ! 
And even more here !
The hummingbird is successfully drinking the flowers dry of nectar as they turn from pink to purple and move on. 
This is where the data is stored on the drive. 
I have left the the training to run to continue the learning process so that I can begin the next part tomorrow. 
Unity Repository 

Lunarium 2 : Machine Learning Agent experiment Day 1

Top Tent Scene

The alienated death character has three pipes with eyes that flip up and down as air is forced through them (Think Pipe Organ). The concept was inspired by carnivorous plants that omit a sonar signal to attract bats. 

I want to create a bat Agent that will find the location of various carnivorous plants in the environment, and once all have been found, this will trigger the alienated death character to play the death march music.   

I have found a tutorial that explains how to create ML Agents. It effectively trains a hummingbird to locate nectar inside flowers in the environment. I think I can use this in my own project to train a bat to find and infect carnivorous plants. Once all are found and infected this will trigger the alienated death character to play the music and open the exit to the level. The ML Agent mechanic will act as a kind of timer.

I have set up the flower script, the flower area script and the hummingbird script so far.   

The code returned a number of errors which I think relate to how the code over rides Agent code, of which initialize is one. Apparently, It will still function but this won’t be evident until the script is applied and the game run. 

Errors

Initialize

Transform 

GetComponent

GetComponentParent 

OnEpisodeBegin

OnActionRecieved

Heuristic

AddReward

// Calculate reward for getting nectar

float bonus = .02f * Mathf.Clamp01(Vector3.Dot(transform.forward.normalized, -nearestFlower.FlowerUpVector.normalized));

AddReward(.01f + bonus);

The rewards must balance out to equal positive 1. This just needs trial and error to figure out. 

Give a reward for collecting nectar but take away reward if banged into too many walls. 

What is a .dot product ? 

  // Observe a dot product that indicates whether the beak tip is in front of the flower (1 observation)

  // (+1 means that the beak tip is directly in front of the flower, -1 means directly behind)

sensor.AddObservation(Vector3.Dot(toFlower.normalized, -nearestFlower.FlowerUpVector.normalized));


Unity’s description of a .dot product. 

Dot Product

The dot product takes two vectors and returns a scalar. This scalar is equal to the magnitudes of the two vectors multiplied together and the result multiplied by the cosine of the angle between the vectors. When both vectors are normalized, the cosine essentially states how far the first vector extends in the second’s direction (or vice-versa – the order of the parameters doesn’t matter).

It is easy enough to think in terms of angles and then find the corresponding cosines using a calculator. However, it is useful to get an intuitive understanding of some of the main cosine values as shown in the diagram below:-

The dot product is a very simple operation that can be used in place of the Mathf.Cos function or the vector magnitude operation in some circumstances (it doesn’t do exactly the same thing but sometimes the effect is equivalent). However, calculating the dot product function takes much less CPU time and so it can be a valuable optimization.


NOTES  

The ML Agent could be used as a mechanism to create a ‘daisyworld’ simulation that demonstrates the earths homeostasis. If you are not familiar with this concept, check out the gaia theory. Essentially when black flowers number more than white the earth absorbs more energy and heats up but when more white flowers are in higher numbers than black, the earth reflects energy and cools down (Like the ice fields at the north pole acts as a mirror.. which it could be said is currently is in a runaway state). It could also be used to represent the bee population and flower polination or human population and earths resources, or potentially the current pandemic (and all of the above). 

  

Lunarium 2 : Production Items

Tower Cut Scene
Tower Room
Tower Look Out Cut Scene
Tower LookOut
Labyrinth Cut Scene
Labyrinth Corridor
Labyrinth 1 Cut Scene
Labyrinth Room 1
Labyrinth 2 Cut Scene
Labyrinth Room 2
Labyrinth 3 Cut Scene
Labyrinth Room 3
Labyrinth 4 Cut Scene
Labyrinth Room 4
Labyrinth 5 Cut Scene
Labyrinth Room 5
Labyrinth 6 Cut Scene
Labyrinth Room 6
Labyrinth 7 Cut Scene
Labyrinth Room 7
Fairground Entrance Cut Scene
Fairground Entrance
Fairground
Caravan Site
Top Tent Cut Scene
Top Tent
Bottom Hole Cut Scene
Hole Exit 1
Hole Exit 2
Heaven Cut Scene
Heaven 1
Hell Cut Scene
Hell 2
End Cut Scene