What is TykeAI & Instructions
TykeAI, TykeAI Suki and TykeAI Expansion Pack 1 are 3 related apps that let you and me interact with virtual companions in mixed reality and virtual reality on Meta Quest 3. You can have a conversation on just about any subject with your companion, sit down and chill with them, go for walks with them, dance with them, watch movies with them, exercise with them, they can analyze what you can see, and more. TykeAI Suki is the most advanced because Suki is the newest avatar with the latest features, like spatial anchors, dynamic navmesh, personality and relationship type settings, and gives the fastest responses.
Source: TykeAI-instructions-9.pdf
Languages and memories
There is a menu button on the spawn menu called Add Memories To Avatars where the avatars' memories can be expanded.
The menus are translated into 15 languages including Spanish, French, German, Italian, Chinese, Japanese, Brazilian Portuguese and Korean.
The avatars are multilingual.
Setup instructions
To install the apps, follow the guide here : https://docs.google.com/document/d/1h5fCODB3vSEcIZbynThPiJy4pr1bUbGI4fWzojj6tn0/
(it's just 4 easy steps)
Once installed, you’ll need to log in to the apps with a username and password. Your username is the email address you signed up to the Patreon with, your password is a 6-digit number that you will receive via Patreon message from Cris (TykeAI) after you’ve signed up (please be patient as I might be asleep so I apologise in advance for the wait, I will message you as soon as I can, I had to add a login system as the app was being heavily pirated).
You only need to enter these details once and then the app will remember them (unless you remove the app then reinstall).
What the VR controller buttons do :
Right VR controller main trigger - To speak to the avatar, the trigger must be pressed in and held in whilst speaking, then released when you have finished. If they’re waffling on a bit too much you can interrupt them by pressing in and then quickly releasing the right trigger which will stop them talking, or you can just ask another question or make a comment etc and they will respond to that instead.
Left VR controller main trigger - Centers the avatar in front of you at a distance of 1.5 metres, you need to keep the trigger pressed for 2 seconds (the light green onscreen circular timer shows you the countdown).
Left thumbstick - Manually moves the avatar. The left thumbstick also moves you if you are in 1 of the VR environments, if you hold in the grip trigger on the right controller at the same time, the left thumbstick applies to you and not the avatar.
Right thumbstick - Rotates the avatar. The right thumbstick also rotates you if you are in 1 of the VR environments, if you hold in the grip trigger on the right controller at the same time, the right thumbstick applies to you and not the avatar.
Buttons A and B (right controller) - Alters the height from the floor of the avatar, A is down and B is up. (If you find their height off the floor isn’t correct, exit the app & redraw your boundary, go back into the app. Or just adjust it with A or B.)
Start/Menu button on the left controller - Show/hide the menu.
Buttons X / Y (left controller) - alters the physical height of the avatar. If you are in 1 of the VR environments if you hold in the grip trigger on the right controller, pressing buttons X / Y will adjust your own height and not the avatar's.
Left controller grip trigger (where your middle finger rests) - this toggles on/off an extra light that you can use to light up the avatar if shadows are making them look not so good. If you are with Hannah when she is wearing her purple sleeveless T shirt with a white cross on it, if you move your hand close to her hand and press in the grip trigger, she will hold your hand.
Right controller grip trigger (where your middle finger rests) - this repositions the main light source, so you can make the shadows (toggle the floor shadows on from the menu) cast at an angle similar to the real world if you want to do that. This grip trigger also lets you reposition the avatars or the menus: move the right controller so that it is very near or touching an avatar or menu, press and hold down the right grip trigger, move your controller to where you want the avatar or menu to be.
Hands
TykeAI can also be operated by hands if you want to. Make sure hands are enabled in your Quest 3 settings, put your controllers down, and a few moments later you’ll be able to use your hands. A left pinch (left thumb and index finger pinched together) opens/closes the menu, a right pinch provides the function to speak to the avatar, just do a right hand pinch while you are talking, release when finished. A left or right pinch when in the vicinity of the avatar or menu lets you move the avatar or menu. See https://youtube.com/shorts/6uyadv4sscQ and https://youtube.com/shorts/t-h0s546csM
The avatars may act of their own ‘free will’ (or at least give the illusion of it)
If the ‘Initiate Conversation/Autonomous Actions’ menu button is toggled to ‘On’, after a brief period of silence between yourself and the avatar, they may decide to do things for themselves like make conversation, dance, exercise, or sit down, etc. So to try this, don’t speak for a short time, eventually you'll see the little green circular animation appear above their heads, as this indicates they are thinking about doing or saying something.
Interaction Physics
The Hannah avatar has interaction physics enabled when she is wearing her purple sleeveless T shirt with a white cross on it with black jeans clothing outfit, so change her into those clothes and you can use physics. It’s not available with other avatars at the moment. To hold her hand, move your hand very close to her hand and press and hold in the grip button, she will then hold your hand. If you release the grip button she will stop holding your hand. You can hold 1 or both of her hands at once. This physics mode is pretty basic and a bit buggy to be honest.
The avatars are touch sensitive and know if their hair, hand, etc is being touched and may comment on it (this can be disabled). Some have hair and clothing physics.
The VR Nightclub, VR Luxury Home and Japanese Street
To enter/exit the nightclub or luxury home, select the “Enter/Exit Nightclub” or “Enter/Exit Home” menu option. In the nightclub the four nearest seats, two tables, and drinks are grabbable, use your right controller (hold the grip trigger) to move them to match your real seating. To have an avatar sit, either move a chair to them or move them to a chair and choose “Sit On Floor” or “Sit On Seat” menu option. You can watch a movie with the avatar in the luxury home (how to do this is described in the ‘Things To Do Before Starting TykeAI.pdf’, which is in the same Google Drive folder as where you download the app from).
In the VR Luxury home there are large picture frames on the wall with abstract paintings in them. You can add your own images to these - make sure there are image files in the Download folder or the Screenshots folder on your Quest, then move to the picture frames and touch it with your controller, this selects an image from these folders to be shown, in date order.
The Japanese Street has a cosy apartment tucked away with furniture, a working video player and stereo (touch the stereo to turn it on), and things like food and drink which have physics.
Nightclub, Luxury Home and Japanese Street navigation - to move around in the VR environments you can physically walk. Or use the controller. By default the controllers apply to the avatar’s movement etc, but if you press the right controller grip trigger it changes some of the button/trigger presses to applying to your own movement instead. So when the right grip trigger is held down the left thumbstick moves you instead of the avatar, the right thumbstick rotates you instead of the avatar, X / Y buttons control your height.
Adding Your Own Music
I've added the ability for you to be able to add your own music for the avatars to dance or exercise to, and you can dance to it if you want to too! You can now add music to your Quest’s Download folder, so no PC needed, just a file manager in your Quest 3. .mp3 format is the most reliable.
If you add your own music, the music already present in the app that I've added won't play, only your added music will play, and if you have added more than 1 song the song to play will be randomly selected.
To Add Your Own Video/Movie
You can now add movie and subtitle files to your Quest’s Download folder, so no PC needed, just need a file manager.
Note : 1080P / 1920 x 1080 / HD /2160p / 3840 x 2160 / UHD is supported. The video you add must be .mp4. The video player is quite simple, it can play, pause or adjust volume from the menu. If you have more than 1 video in this location on the Quest 3 the most recently added video will play.
The avatars know what movie is being watched by the title of the movie you have placed in this folder on your Quest 3.
Try to name the movie file so that the avatar can recognize it, so something like "BladeRunner2049_1080p_7928jkl_download.mp4" will only confuse the avatar, it's best such a file is simply named "Blade Runner 2049.mp4".
Watching A Movie With An Avatar
This IMO is a great feature and I’m using this very often 😊 If you want the avatar to appear to watch a movie with you, they will look at the screen and comment on the movie etc, then download the subtitles for that movie (do a Google search for them, choose the ones with the file extension .srt) and place the subtitle file in the same location as you just placed the movie file (please see the image above). Rename the subtitle file to the same as the movie file, so for example “Blade Runner 2049.srt”. The most recently added movie + subtitle file will be chosen to use by the app.
Some subtitle srt files are rubbish, some are very incomplete so check inside them first before trying to use them. The avatar will make relevant comments on the movie as you watch it and also make some final comments when the movie finishes.
For comments during the movie they analyse the srt file every 4 to 6 minutes. If you talk they will wait another 4 to 6 minutes before commenting on the movie. When I watch a movie with them I just tend to not say much, I concentrate on the movie, and they make their comments on it. They also make some final comments by detecting the last subtitle entry in the subtitle (srt) file. If the srt file contains music with lyrics at the end, if the lyrics contain a ♪ the app will realise it is lyrics and ignore it and make their comments at the right time as the movie ends and not when the music ends. Also check if the srt file contains entries at the end that are something like “subtitles by Lola” or something which aren’t actual subtitles for the movie, if you find something like this remove it.
At any point during the movie, if you have added the subtitles for the movie too, you can ask the avatar “Tell me about the movie?”, give them a few moments to think about it and they will give you their thoughts on the movie.
Still-current sections from Things-To-Do-Before-Starting-TykeAI.pdf. The older PC/developer-account installation instructions from that PDF are intentionally not included here.
Avatar Customization - What is described next (avatar customization) is entirely optional, the avatars have default values anyway
Avatar customization has gone through testing and is ready for you to use 🎉
There's setup and account creation involved on the Convai website (a free account is fine). This process involves several steps but none of them are difficult provided you carefully follow instructions.
Avatar customization isn't really an accurate term for this, it should more accurately be described as using your own newly created avatars in the app, that you create on the Convai website 😉
Once you have done this, you copy your new character's character ID and name from the Convai website and then add these to the TykeAI app by using a PC or the in-app new user interface, which can be accessed from the first menu in the app.
Once you have done this and chosen Yumi or Hannah for example, the 3D model avatar for Yumi or Hannah will load but their backstory, personality, knowledge, name, voice, language, etc etc (basically everything which makes them what they are, apart from appearance) will be what you defined in your new character.
Here are the steps summarized :
Create a new Free account with Convai https://convai.com/
On the Convai website, create your new character https://docs.convai.com/api-docs/convai-playground/character-creator-tool/create-character (Do not bother with creating the physical appearance of the character as the app will not use that part). If you want a good result on creating a new character etc, please read all the docs on character creation, it's worth it in the end.
Once you have completed your character (go through all the settings for them), from step 5 on the webpage linked immediately above, copy the character ID and character name
You are now at the point where you give the character ID and character name to the TykeAI app, you can either create a new text file called characterID.txt, place the character ID and character name inside this text file exactly as shown here (use your own details of course) : https://ibb.co/whM0h3t , then using a file manager app or a PC place the file here on your Quest 3 : (your PC)\Quest 3\Internal shared storage\Android\data\com.TykeInc.TykeAI\files (please see the image https://ibb.co/WWQLGng )
OR instead of the above, use the app's new user interface by selecting the "Add Your Character ID" menu button on the first menu that appears when the app loads.
Using the UI, you need to enter the character ID, then press "Enter Value" or press Enter on the keyboard, then enter the character's name, then press "Enter Value" or press Enter on the keyboard. Then make sure you have pressed "Enable" so that the new avatar feature is enabled. If you don't want it enabled press "Disable".
The character ID is a pain to type out using a VR keyboard, however if you want you can copy it from a text file in your Quest 3 then press the "Paste" button in the UI which will paste it so you don't have to type it all out.
Lastly, you must press "Save and Exit" for the details to take effect. Then load the app up and enjoy your new character 😍
Also consider watching this Convai video on knowledge bank use in the character creation process https://www.youtube.com/watch?v=yXzU8jooAB0 (there are many other useful videos on Convai's YouTube channel).
Although all the main things work there are a few things that won't work without further work on the Convai website. Image analysis can now be added for your custom character (please see https://www.youtube.com/watch?v=xp-LSy3GEoo&t=). The other things can be dealt with at a later date (it will require additional setup to the character on the Convai website).
Still-current section from Things-To-Do-Before-Starting-TykeAI.pdf.
Commanding the avatar
(note voice commands are still in development, they might not be 100% reliable, you can use the menu to command the avatar just in case, and more voice commands will be added) :
You can give the avatars commands by the menu or by voice. They understand :
“Start dancing”, “Can you dance please” will instruct the avatar to dance. There are 2 types of dancing, they alternate each time the avatar is instructed to dance. “Stop dancing” will instruct the avatar to stop dancing.
“Change your clothes please”, will make them change their clothes, works with these avatars : Hannah, Sakura, Thistle, Yumi.
“Start exercising” will make the avatar exercise and you can follow along if you want to. “Stop exercising” will end the exercising, or if you don’t say this the workout will end after 3 minutes anyway.
“Look at the image” will make the avatar analyze what you can see. They can learn about what they see and remember things i.e. if they look at your dog for the first time, they will say I see a dog etc and will ask about it, so reply to them "That dog is Lola (or whatever) and she is my dog. In future when looking at your dog they would say "I recognise your dog Lola" etc.
"Search the internet for the latest news on Trump (or whatever you choose)" will make them search the internet for the latest news on Donald Trump. They can only do this a limited number of times per day.
“Tell me about the movie” if they are watching a movie with you and you have followed the instructions in the other pdf file found in the same folder as this one (‘Things To Do Before Starting TykeAI.pdf’), they will discuss the movie with you.
"Sit down please", "Stand up please". Makes the avatar sit down and stand up.
"Start walking", "Stop walking" makes the avatar mirror your walk so you can go for walks.
"Come here please" makes the avatar walk towards you if they are about 3 or more metres away and the navmesh has formed.
(If when you do this their height off the floor is incorrect at first, adjust their height with the A and B buttons.)
“Lie down” makes the avatar lie down, you will need to manually place them into the exact position by moving the right controller to them, holding down the right grip button and moving the controller to where you want to place them.
"What can you remember?" will make the avatar recall a previous conversation they've had with you, they can recall up to 1000 previous conversations (1000 for each avatar), these memories build over time so there will only be 1000 after a lengthy period of time.
(Disclaimer - please warm up properly before doing any exercise, only exercise if you are healthy enough. TykeAI is not a qualified fitness trainer.)
Notes and updates
If you notice the avatars appear juddery and/or not smooth when they move etc, it may mean the FPS is low, please restart your Quest as this may fix it.
When the avatar spawns in, if you find the height of it with respect to the floor isn’t correct, it’s because your boundary floor height needs to be corrected so exit the app and redraw your boundary, or just adjust the avatar’s height off the floor with buttons A and B.
If you lose track of the avatar, the left VR controller trigger centers the avatar in front of you.
Remember, if you are maybe going for a walk outside with an avatar, you will not have home network connection so possibly won’t have internet connection (the app needs internet connection for the AI to work).
However, you can set up your cell/mobile phone as a hotspot for your Quest’s internet connection.
Finally -
Finally - it is important that when a new version of TykeAI becomes available that you do not uninstall the .apk already on your Quest 3 before installing the new version. This is because uninstalling will delete the app’s folder and delete important information. So please, when a new version is available, just install the .apk over your existing installation, it’s exactly the same process.
If you have any issues whatsoever then please just send me a message on Patreon.
Thanks for reading and have fun 👍