What is TykeAI
TykeAI is a family of related apps that let you interact with virtual companions in mixed reality and virtual reality on Meta Quest 3. You can have a conversation on just about any subject with your companion and sit down and chill with them or go for walks or dance or watch movies or exercise or play chess with them, and they can analyze what you can see and more.
There are several app versions. currently they are TykeAI Suki, TykeAI Nova and Terminator, TykeAI Donald, TykeAI Expansion Pack 1, TykeAI and TykeAI Dev Build. they only vary in the avatars they contain and they're all standalone so you can install all of them or just 1 or 2 and none of them rely on the others
Suki gives the fastest responses.
A note on the Suki app: it really pushes the Quest 3 so if you experience performance issues like slow loading or crashing restart your headset and make sure you have a good amount of free space on your Quest by deleting/uninstalling large unused files/apps/games etc and especially if u have the 128GB Quest 3 and not the 512GB version.
There are currently 17 avatars to choose from with more planned.
Languages and memories
There is a menu button on the spawn menu called Add Memories To Avatars which opens a UI where u can add permanent memories to the avatars.
The menus are translated into 15 languages including Spanish, French, German, Italian, Chinese, Japanese, Brazilian Portuguese and Korean. Theres a menu button on the spawn menu where u can choose the language the menus are shown in
The avatars are multilingual.
If you don't fancy speaking out loud the Use Keyboard To Chat menu button gives u an in app keyboard to type into to chat to the avatars instead.
Setup and logging in
To install the apps follow the guide here : https://docs.google.com/document/d/1h5fCODB3vSEcIZbynThPiJy4pr1bUbGI4fWzojj6tn0/
(it's just 4 easy steps)
Once installed you’ll need to log in to the apps with a username and password. Your username is the email address you signed up to the Patreon with and your password is a 6-digit number that you will receive via Patreon message from Cris (TykeAI) after you’ve signed up (please be patient as I might be asleep so I apologise in advance for the wait, I will message you as soon as I can. I had to add a login system as the app was being heavily pirated).
You only need to enter these details once and then the app will remember them (unless you remove the app then reinstall).
If at any time your login stops working please contact me, Cris, via a private Patreon message. it will take literally 30 seconds to sort out so please just contact me because i will be happy to help
What the VR controller buttons do
Right VR controller main trigger: To speak to the avatar the trigger must be pressed in and held in whilst speaking then released when you have finished. If they’re waffling on a bit too much you can interrupt them by pressing in and then quickly releasing the right trigger which will stop them talking or you can just ask another question or make a comment etc and they will respond to that instead
Left VR controller main trigger: Centers the avatar in front of you at a distance of 1.5 metres and you need to keep the trigger pressed for 2 seconds (the light green onscreen circular timer shows you the countdown).
Left thumbstick: Manually moves the avatar. The left thumbstick also moves you if you are in 1 of the VR environments but if you hold in the grip trigger on the right controller at the same time the left thumbstick applies to you and not the avatar.
Right thumbstick: Rotates the avatar. The right thumbstick also rotates you if you are in 1 of the VR environments but if you hold in the grip trigger on the right controller at the same time the right thumbstick applies to you and not the avatar.
Right thumbstick pressed up or down (right controller): adjusts the avatars height from the floor. push the stick up to move them up and down to move them down. (If you find their height off the floor isn’t correct exit the app & redraw your boundary then go back into the app. Or just adjust it with the thumbstick.)
Button A (right controller): pressing and holding A while your controller is touching the avatar grips their body which is the body physics described in the body physics section.
Start/Menu button on the left controller: Show/hide the menu
Buttons X / Y (left controller): alters the physical height of the avatar. If you are in 1 of the VR environments if you hold in the grip trigger on the right controller then pressing buttons X / Y will adjust your own height and not the avatar's.
Left controller grip trigger (where your middle finger rests): this toggles on/off an extra light that you can use to light up the avatar if shadows are making them look not so good. If you are with Hannah when she is wearing her purple sleeveless T shirt with a white cross on it and you move your hand close to her hand and press in the grip trigger, she will hold your hand.
Right controller grip trigger (where your middle finger rests): this repositions the main light source so you can make the shadows (toggle the floor shadows on from the menu) cast at an angle similar to the real world if you want to do that. This grip trigger also lets you reposition the avatars or the menus: move the right controller so that it is very near or touching an avatar or menu then press and hold down the right grip trigger and move your controller to where you want the avatar or menu to be.
Hands
TykeAI can also be operated by hands if you want to. Make sure hands are enabled in your Quest 3 settings and put your controllers down and a few moments later you’ll be able to use your hands. A left pinch (left thumb and index finger pinched together) opens/closes the menu while a right pinch provides the function to speak to the avatar, just do a right hand pinch while you are talking then release when finished. A left or right pinch when in the vicinity of the avatar or menu lets you move the avatar or menu. See https://youtube.com/shorts/6uyadv4sscQ and https://youtube.com/shorts/t-h0s546csM
the body physics works with hands too: make a pinch gesture on the limb or whatever you want to grab it (see the body physics section). the web browser, the 3D video player and the keyboard for chatting all work with hand tracking as well.
The avatars may act of their own ‘free will’ (or at least give the illusion of it)
If the ‘Initiate Conversation/Autonomous Actions’ menu button is toggled to ‘On’ then after a brief period of silence between yourself and the avatar they may decide to do things for themselves like make conversation or dance or exercise or sit down etc. So to try this don’t speak for a short time and eventually you'll see the little green circular animation appear above their heads as this indicates they are thinking about doing or saying something
Three way conversations (avatar to avatar)
You can have more than one avatar in at once. press the Add Avatar menu button and an extra avatar is enabled. the avatars will then talk to each other as well as to you so u can have a proper 3 way conversation between yourself and the two avatars. Its great just sitting back and listening to them chat between themselves too.
Body physics
The avatars have full body physics so things like their hands can be held and arms moved about etc.
Here's how it works: move your controller so its touching a limb (or their head etc) then press and hold in button A on the right controller. this grips wherever youre holding and you can then move that body part around. release A to let go.
If you're using hand tracking instead of controllers just make a pinch gesture on the limb or whatever to grab it.
What happens to the body part after you let go depends on the Animation Toggle menu button:
On (the default): animations play as normal during physical interactions. the limb you are holding follows how u move it but on release it returns to its animated state
Partial: the moved limb stays in position wherever you move it to even after releasing it.
Off: all animations apart from head animations are stopped.
So if you want a body part to stick where u put it set the Animation Toggle to Partial or Off.
The avatars are touch sensitive and know if their hair or hand etc is being touched and may comment on it (this can be disabled). Some have hair and clothing physics.
Playing chess
the avatars can play chess with you and they have several skill levels
Play Chess: toggles the chess board. toggle it on to show the board and start playing then toggle it again to hide the board and stop playing chess.
Skill level: when the chess board is enabled another button is shown which lets you set the avatars chess skill level so there's someone worth playing whatever level youre at.
Reset Chess Board: resets the current game so a new one can be started.
Moving the screens and boards around
The video player screen, web browser, chess board and chat keyboard can all be moved by grabbing them: move the right controller to them and press and hold in the right grip trigger then move the controller to put them where you want.
Video screen and web browser: these can be grabbed at a distance with no need to be touching them. the borwser can also be grabbed with a closed palm if you're using hand tracking
Chess board and chat keyboard: these are grabbed with the controller touching them as mentioned above or with hand tracking using a closed palm.
The VR Nightclub, VR Luxury Home and Japanese Street
To enter/exit the nightclub or luxury home select the “Enter/Exit Nightclub” or “Enter/Exit Home” menu option. In the nightclub the four nearest seats and two tables and drinks are grabbable, use your right controller (hold the grip trigger) to move them to match your real seating. To have an avatar sit either move a chair to them or move them to a chair and choose “Sit On Floor” or “Sit On Seat” menu option. You can watch a movie with the avatar in the luxury home (how to do this is described in the music and movies section below).
In the VR Luxury home there are large picture frames on the wall with abstract paintings in them. You can add your own images to these: make sure there are image files in the Download folder or the Screenshots folder on your Quest then move to the picture frames and touch it with your controller, this selects an image from these folders to be shown in date order.
The Japanese Street has a cosy apartment tucked away with furniture, a working video player and stereo (touch the stereo to turn it on) and things like food and drink which have physics.
Nightclub, Luxury Home and Japanese Street navigation: to move around in the VR environments you can physically walk. Or use the controller. By default the controllers apply to the avatar’s movement etc but if you press the right controller grip trigger it changes some of the button/trigger presses to applying to your own movement instead. So when the right grip trigger is held down the left thumbstick moves you instead of the avatar and the right thumbstick rotates you instead of the avatar while X / Y buttons control your height
Music and movies
Adding your own music
I've added the ability for you to be able to add your own music for the avatars to dance or exercise to and you can dance to it if you want to too! You can now add music to your Quest’s Download folder so no PC is needed, just a file manager in your Quest 3. .mp3 format is the most reliable.
If you add your own music the music already present in the app that I've added won't play, only your added music will play and if you have added more than 1 song the song to play will be randomly selected.
To add your own video/movie
You can add movie to your Quest’s Download folder so no PC is needed (you can use a PC if you want).
Note : 1080P / 1920 x 1080 / HD / 2160p / 3840 x 2160 / UHD is supported. The video you add must be .mp4. The video player can play, pause, forward / rewind, adjust volume with the controllers or the menu. If you have more than 1 video in this location on the Quest 3 the most recently added video will play.
The avatars know what movie is being watched by the title of the movie you have placed in this folder on your Quest 3
Try to name the movie file so that the avatar can recognize it, something like "BladeRunner2049_1080p_7928jkl_download.mp4" will only confuse the avatar so it's best such a file is simply named "Blade Runner 2049.mp4".
Your movies play in 3D now
The movie player now plays your normal 2D mp4 files in 3D by default and converts them as you watch. no setup needed, it just works.
If you want to turn the 3D off or tweak it there's a settings file in your Quest’s Download folder called TykeAI_Video3D_Settings.json which any text editor like Notepad etc can edit (do not rename the file). These are the values u can change:
"depthModel"
"MiDaS" is the default value and is the more performance friendly model. "DepthAnythingV2" is the better model for depth and gives more 3D but will lower the FPS a lot.
"enable3D"
Set this to "false" to switch off the 3D effect.
"stereoDisparity"
Default 0.015. adjusts the strength of the stereo effect. be careful with this one as it can mess up the image and if you do mess things up just put the default value back in
"temporalBlend"
Default 0.12. same warning as above so if the image goes funny just put 0.12 back in.
Watching a movie with an avatar
This is a great feature and I’m using this very often 😊
If you want the avatar to enjoy a movie with you (they will look at the screen and understand and comment on the movie etc) then download the subtitles for that movie (do a Google search for them and choose the ones with the file extension .srt) and place the subtitle file in the same location as you just placed the movie file.
Rename the subtitle file to the same as the movie file so for example “Blade Runner 2049.srt”. The most recently added movie + subtitle file will be chosen to use by the app.
Some subtitle srt files are rubbish and some are very incomplete so check inside them first before trying to use them. The avatar will make relevant comments on the movie as you watch it and also make some final comments when the movie finishes.
For comments during the movie they analyse the subtitles file every 4 to 6 minutes. If you talk they will wait another 4 to 6 minutes before commenting on the movie.
When I watch a movie with them I just tend to not say much and concentrate on the movie while they make their comments on it.
They also make some final comments by detecting the last subtitle entry in the subtitle (srt) file. If the srt file contains music with lyrics at the end and the lyrics contain a ♪ the app will realise it is lyrics and ignore it and make their comments at the right time as the movie ends and not when the music ends.
Also check if the srt file contains entries at the end that are something like “subtitles by Lola” or something which aren’t actual subtitles for the movie and if you find something like this remove it.
At any point during the movie if you have added the subtitles for the movie too you can ask the avatar “Tell me about the movie?” then give them a few moments to think about it and they will give you their thoughts on the movie
remember the Initiate Conversation/Autonomous Actions menu button needs to be On for the avatar to watch the movie with you.
Avatar customization (entirely optional)
What is described next is entirely optional as the avatars have default values anyway.
Avatar customization has gone through testing and is ready for you to use 🎉
There's setup and account creation involved on the Convai website (a free account is fine). This process involves several steps but none of them are difficult provided you carefully follow instructions.
Avatar customization isn't really an accurate term for this and it should more accurately be described as using your own newly created avatars in the app that you create on the Convai website 😉
Once you have done this you copy your new character's character ID and name from the Convai website and then add these to the TykeAI app by using a PC or the in-app new user interface which can be accessed from the first menu in the app.
Once you have done this and chosen Yumi or Hannah for example the 3D model avatar for Yumi or Hannah will load but their backstory, personality, knowledge, name, voice, language etc etc (basically everything which makes them what they are apart from appearance) will be what you defined in your new character
Here are the steps summarized
Create a new Free account with Convai https://convai.com/
On the Convai website create your new character https://docs.convai.com/api-docs/convai-playground/character-creator-tool/create-character (Do not bother with creating the physical appearance of the character as the app will not use that part). If you want a good result on creating a new character etc please read all the docs on character creation, it's worth it in the end.
Once you have completed your character (go through all the settings for them) then from step 5 on the webpage linked immediately above copy the character ID and character name
You are now at the point where you give the character ID and character name to the TykeAI app. you can either create a new text file called characterID.txt and place the character ID and character name inside this text file exactly as shown here (use your own details of course) : https://ibb.co/whM0h3t , then using a file manager app or a PC place the file here on your Quest 3 : (your PC)\Quest 3\Internal shared storage\Android\data\com.TykeInc.TykeAI\files (please see the image https://ibb.co/WWQLGng )
Note: the com.TykeInc.TykeAI part of that path varies depending on which app version you want the custom character in. if its Mervin in the TykeAI app then com.TykeInc.TykeAI is correct but if its Nova in the TykeAI Nova app for example then that part needs to be com.TykeInc.TykeAI_Nova and so the full path would be (your PC)\Quest 3\Internal shared storage\Android\data\com.TykeInc.TykeAI_Nova\files. here are all the app version paths:
com.TykeInc.TykeAI_Suki,com.TykeInc.TykeAI_Nova,com.TykeInc.TykeAI_Trump,com.TykeInc.TykeAI_Expansion_Pack_1,com.TykeInc.TykeAI_Dev_Build,com.TykeInc.TykeAI.OR instead of the above use the app's new user interface by selecting the "Add Your Character ID" menu button on the first menu that appears when the app loads.
Using the UI you need to enter the character ID then press "Enter Value" or press Enter on the keyboard, then enter the character's name and press "Enter Value" or press Enter on the keyboard. Then make sure you have pressed "Enable" so that the new avatar feature is enabled. If you don't want it enabled press "Disable".
The character ID is a pain to type out using a VR keyboard however if you want you can copy it from a text file in your Quest 3 then press the "Paste" button in the UI which will paste it so you don't have to type it all out
Lastly you must press "Save and Exit" for the details to take effect. Then load the app up and enjoy your new character 😍
Also consider watching this Convai video on knowledge bank use in the character creation process https://www.youtube.com/watch?v=yXzU8jooAB0 (there are many other useful videos on Convai's YouTube channel).
Commanding the avatar
(note voice commands are still in development and might not be 100% reliable so you can use the menu to command the avatar just in case, more voice commands will be added) :
You can give the avatars commands by the menu or by voice. They understand :
“Start dancing”, “Can you dance please” will instruct the avatar to dance. There are 2 types of dancing and they alternate each time the avatar is instructed to dance. “Stop dancing” will instruct the avatar to stop dancing.
“Change your clothes please” will make them change their clothes and works with these avatars : Suki, Peter, Jen, Nova, Hannah, Sakura, Thistle, and Yumi.
“Start exercising” will make the avatar exercise and you can follow along if you want to. “Stop exercising” will end the exercising or if you don’t say this the workout will end after 3 minutes anyway
“Analyse the image” will make the avatar analyze what you can see. They can learn about what they see and remember things i.e. if they look at your dog for the first time they will say I see a dog etc and will ask about it, so reply to them "That dog is Lola (or whatever) and she is my dog. In future when looking at your dog they would say "I recognise your dog Lola" etc.
"Search the internet for the latest news on Trump (or whatever you choose)" will make them search the internet for the latest news on Donald Trump. They can only do this a limited number of times per day.
"Sit down please", "Stand up please". Makes the avatar sit down and stand up.
"Come here please" makes the avatar walk towards you if they are about 3 or more metres away and the navmesh has formed. (If when you do this their height off the floor is incorrect at first just adjust their height by pressing the right thumbstick up or down.)
“Lie down” makes the avatar lie down and you will need to manually place them into the exact position by moving the right controller to them, holding down the right grip button and moving the controller to where you want to place them.
"What can you remember?" will make the avatar recall a previous conversation they've had with you and they can recall up to 1000 previous conversations (1000 for each avatar), these memories build over time so there will only be 1000 after a lengthy period of time.
(Disclaimer: please warm up properly before doing any exercise and only exercise if you are healthy enough. TykeAI is not a qualified fitness trainer.)
Notes and updates
If you notice the avatars appear juddery and/or not smooth when they move etc it may mean the FPS is low so please restart your Quest as this may fix it.
When the avatar spawns in if you find the height of it with respect to the floor isn’t correct it’s because your boundary floor height needs to be corrected so exit the app and redraw your boundary or just adjust the avatar’s height off the floor by pressing the right thumbstick up or down
If you lose track of the avatar the left VR controller trigger centers the avatar in front of you.
Remember if you are maybe going for a walk outside with an avatar you will not have home network connection so possibly won’t have internet connection (the app needs internet connection for the AI to work).
However you can set up your cell/mobile phone as a hotspot for your Quest’s internet connection.
Finally
It is important that when a new version of TykeAI becomes available that you do not uninstall the .apk already on your Quest 3 before installing the new version. This is because uninstalling will delete the app’s folder and delete important information. So please when a new version is available just install the .apk over your existing installation, it’s exactly the same process.
If you have any issues whatsoever then please just send me a message on Patreon
Thanks for reading and have fun 👍