Getting Started with Jacky

Welcome to Jacky, your virtual pet and intelligent desktop agent! Jacky wanders around your screen, schedules custom routines, triggers interactive commands, and responds to your applications.

Installation & First Launch

Download the latest release and run the Jacky application. On first launch, Jacky appears directly on your desktop and starts walking along the bottom of your screen and across window title bars.

Jacky also adds a system tray icon for quick access to Settings and Quit. To open the full menu, right-click Jacky directly on screen:

• 💬 Ask: Interact with Jacky using natural language prompts.
• 📋 Routines: Run a manual routine, or view scheduled automatic ones.
• ⚙️ Settings: Manage your companion's model provider, API keys, and configurations.

Installing Packages from the CLI

In addition to drag-and-drop, you can install .jacky packages directly from the terminal using the jacky add command. This works for skins, routines, and language packs:

# Install a skin
jacky add skin <user>/jacky
 
# Install a routine
jacky add routine <user>/weather-forecast
 
# Install a language pack
jacky add language <user>/japanese

The format is jacky add <type> <author>/<id>, where <type> is skin, routine, or language. The package is downloaded and extracted into your ~/.jacky folder automatically.

Companion Behavior & Boredom

Jacky is not just a static sprite; he is a living desktop companion with his own personality, window awareness, and boredom thresholds:

Window Awareness & Reactions

• Title Bar Navigation: Jacky walks along your active window title bars, peeks over them, sits on them, and can even interact with them (e.g., shaking, minimizing, dragging, or toppling windows based on permissions).
• App Comments: Jacky monitors which applications you have open and will make witty or funny comments about them.
• Window State Changes: Whenever you open or close a window, Jacky notices and reacts dynamically.

The Boredom Escalation Ladder

If you ignore Jacky or do not interact with him for a while, he will get bored and progress through four levels of escalation:

1. Callouts (20 mins idle): Jacky speaks up with short messages to catch your attention.
2. Erratic Walk (30 mins idle): He becomes restless, pacing around and randomly changing directions.
3. Self-Talk (40 mins idle): He enters a monologue state, using the LLM (if enabled) to think out loud and self-talk every 2 minutes.
4. Sleep (60 mins idle): He falls deep asleep, pausing all scheduled routines.

To wake Jacky up, simply click on him. This immediately resets his boredom level, triggers a happy animation, and resumes all active routines.

Screen Interaction (Experimental)

Jacky has the experimental ability to interact directly with your screen. When you issue a command like "click the Chrome icon" or "type hello world into Notepad", Jacky attempts to carry out these operations using a multi-phase coordinate mapping engine.

How it Works:

1. Intent Detection: Analyzes text commands to classify the target.
2. Screen Capture: Takes a screenshot of your active desktop.
3. Set-of-Mark Detection: Attempts to overlay bounding boxes on interactive elements using a local ONNX OmniParser model.
4. Targeting & Execution: The pet walks to the identified coordinates and triggers clicks or keystrokes.

What's Next?

Now that you know the basics, go ahead and build your first configuration or customization:

• Create a Custom skin
• Generate automated actions in the Routine Builder
• Package any skin, routine, or language pack with the Manifest Builder
• Set up streaming integrations in the Twitch Setup Wizard