# Pupil Capture screen surface

This small Python program fills the primary screen and displays four unique
`tag36h11` AprilTags in its corners. It has no third-party dependencies; the
standard Python installation used on Windows normally includes Tkinter.

## Run

Close or move anything important from the primary screen, then run:

```powershell
python show_apriltag_surface.py
```

Press `Esc` or `Q` to close it. For a preview in a normal resizable window:

```powershell
python show_apriltag_surface.py --windowed
```

On a multi-monitor setup, list detected displays (Windows display number,
resolution, position, physical diagonal) and target one of them:

```powershell
python show_apriltag_surface.py --list-monitors
python show_apriltag_surface.py --monitor 2
```

`gaze_exercises.py` accepts the same `--monitor`/`--list-monitors` flags and
uses the selected monitor's resolution and EDID-reported diagonal
automatically, so `--screen-diagonal` only needs to be passed to override an
inaccurate or missing EDID value.

By default, every marker is flush with both adjacent screen edges. Useful
options include `--tag-size 240`, `--margin 32`, and
`--tag-ids 10 11 12 13`. The requested size is rounded down to a multiple of
10 pixels so that the source grid is always scaled without interpolation.

## Define the surface in Pupil Capture

1. Start this program and look at the screen through the headset.
2. Enable **Surface Tracker** in Pupil Capture.
3. Confirm all four markers are outlined/detected in the World window.
4. Select **Add surface**, name it (for example, `screen`), then freeze the
   scene and edit the surface corners to match the usable screen area.
5. Keep this program running whenever the surface should remain trackable.

If detection is unreliable, increase `--tag-size`, reduce glare, and ensure
the world camera can see the markers. Do not reuse any of these four marker IDs
elsewhere in the camera's view.

For the next integration step, enable Pupil Capture's **Network API** plugin
and keep **Surface Tracker** active. Surface-relative gaze is broadcast on a
topic beginning with `surfaces.`; its normalized `(x, y)` coordinates use the
bottom-left as `(0, 0)` and the top-right as `(1, 1)`.

## Live gaze visualization

Set the `screen` surface Width to `2560` and Height to `1440`, keep Surface
Tracker and Network API enabled, and install the two network dependencies once:

```powershell
python -m pip install -r requirements.txt
```

Then close the marker-only program and run:

```powershell
python visualize_surface_gaze.py
```

This program displays the same four corner tags, subscribes to
`surfaces.screen`, draws a red gaze dot, and prints rows containing Pupil
timestamp, screen x, screen y, and confidence. Coordinates are converted to
the Windows top-left origin. It connects to `127.0.0.1:50020` by default;
use `--host 192.168.0.47` only when this script runs on another computer.

The displayed point passes through a 5-sample median filter, exponential
smoothing, and a 3-pixel dead zone. Terminal rows contain timestamp, raw x,
raw y, filtered x, filtered y, and confidence. For a steadier but slower point:

```powershell
python visualize_surface_gaze.py --smoothing 0.10 --median-window 7
```

For a more responsive point, use `--smoothing 0.30 --median-window 3`.

## Guided gaze exercises

With Pupil Capture configured the same way, run:

```powershell
python gaze_exercises.py
```

Press Space to begin and between blocks. The sequence contains 10 three-second
steady fixations, 10 rapid target acquisitions, 20 two-color selective-attention
trials, and three 10-second smooth-pursuit paths. A live yellow dot shows the
raw gaze when the program is run with `--cursor`; it is hidden by default. No
median or smoothing filter is applied, and every received sample that meets the
`--confidence` threshold is written to a timestamped CSV in `results/`, including
the target, gaze coordinates, pixel error, and whether gaze was on target. During
every color trial, green is the target and red is the distractor.

### Data collection for ML-based adaptive filtering

Each run writes `results/gaze_exercises_<timestamp>.csv` plus a sidecar
`..._meta.json` recording subject, session, screen, and distance metadata. Tag
every recording with `--subject-id` and, when varying setup, `--session-tag`:

```powershell
python gaze_exercises.py --subject-id you --session-tag laptop_50cm --screen-diagonal 15.6 --viewing-distance 50
python gaze_exercises.py --subject-id you --session-tag monitor_90cm --screen-diagonal 27 --viewing-distance 90
```

`--screen-diagonal` (inches) and `--viewing-distance` (cm) must match the
physical setup for that recording — they drive the pixel-to-degree conversion
used for both the logged error and velocity columns, so an incorrect value
silently skews every derived feature in that session.

CSV columns beyond the basic error metrics:

- `confidence` — Pupil Capture's per-sample confidence (already thresholded
  by `--confidence`, but the value itself is kept for weighting/filtering).
- `velocity_px_s`, `velocity_deg_s` — instantaneous gaze speed between
  consecutive samples; blank when the gap since the previous sample exceeds
  0.5s (block/trial boundaries).
- `accel_px_s2` — change in `velocity_px_s` between consecutive samples.
- `dispersion_px` — spread (bounding-box width + height) of the last 5 raw
  samples; low during fixation/pursuit, spikes during saccades.
- `event_label` — coarse ground truth derived from task design: `saccade`
  until gaze first lands within the hit radius of the trial's target, then
  `fixation` (or always `pursuit` during the pursuit block). This is a block
  design label, not a precise per-sample velocity-threshold classification —
  refine saccade on/offset from `velocity_deg_s` during post-processing if
  the paper needs tighter boundaries.

`subject_id` and `session_id` (subject + recording timestamp) are included on
every row so CSVs from different subjects, screens, and distances can be
concatenated directly for training/evaluation.

## Training the adaptive filter

Once you have one or more recordings in `results/`:

```powershell
python train_filter_model.py
```

This concatenates every `gaze_exercises_*.csv`, builds causal windowed
features (`gaze_features.py`), splits by whole trial (never by row, to avoid
leaking adjacent-in-time samples between train/test), trains a
`RandomForestClassifier` predicting `event_label`, and compares it against a
fixed 30 deg/s velocity-threshold baseline. It prints per-class precision/
recall, a confusion matrix, a saccade-recovery detection-latency comparison,
and feature importances, then saves `models/filter_model.joblib` +
`models/filter_model_meta.json`.

With only one subject/session recorded, the split holds out unseen trials
within that session — it does not yet test cross-subject generalization.
Once more subjects are recorded, re-run training; group the split by
`subject_id` (leave-one-subject-out) instead of by trial for the real paper
evaluation.

`adaptive_filter.py` defines the real-time filters built on that model:
`BaselineFilter` (the existing fixed median+EMA filter), and
`MLAdaptiveFilter` in `mode="hard"` (discrete fixation/saccade/pursuit
switching that discards history on a detected saccade) or `mode="soft"`
(continuous smoothing strength scaled by predicted saccade probability).

## Held-out evaluation: static fixation

`gaze_exercises.py` is what the model trains on, so it can't validate
generalization by itself. `eval_static_fixation.py` runs a task shape the
model has never seen — one unmoving target held for a long duration — and
reports RMS error, mean error, and on-target accuracy % for raw gaze, the
fixed baseline filter, and both ML-adaptive modes side by side:

```powershell
python eval_static_fixation.py --subject-id you --duration 60 --viewing-distance 70 --screen-diagonal 31.5
```

Press Space to start the 60-second hold. Results are saved to
`results/eval_static_fixation_<timestamp>.csv` plus a `_meta.json` summary
report. If `models/filter_model.joblib` doesn't exist yet, it records raw +
baseline only and prints a warning — run `train_filter_model.py` first for
the ML columns. A simplified eyes-as-aim game is a planned second held-out
evaluation task, to test generalization to fast target-acquisition demands
closer to real use.

Visualize the newest result file with:

```powershell
python visualize_results.py
```

Or select a particular run with `python visualize_results.py results\file.csv`.
The script opens a six-panel dashboard, prints summary statistics, and saves a
`*_dashboard.png` beside the source CSV. Use `--no-show` to only save the PNG.
