A discovery draft · for Dr. Lori Custodero
Watching Themselves Play
What happens when twenty-six musicians can see themselves from every angle, in real time, while they play — and how a room full of cameras lets us read it.
Living draft · the data is still opening up · version 20260802-02
Part I
The question that opened the door
Music is an art built on sound — yet the evidence keeps showing that the eye holds surprising power over it. Tsay (2013) found that people judging musical performances, even expert judges, were guided more by what they saw than by what they heard. Sight can even rewrite hearing itself (the McGurk effect). Across fields, seeing oneself is a strong lever: it speeds skill learning in surgery, sports, and dance, and simply watching oneself shifts behavior toward one's own standards. Musicians, though, have never really been able to see themselves while playing — a mirror shows one reversed angle; a recording arrives too late to shape the moment it captured.
So this study asks a question no one has been able to ask before: what happens when a musician can watch themselves from several angles at once, in real time, choosing freely what to look at — while they play? Twenty-six musicians each did exactly that. A researcher assisted with the equipment and occasionally offered a gentle prompt, but did not teach or correct: this is a musician reviewing themselves. The planned surveys and interviews were set aside (IRB-approved); instead the study reads what they actually did and said in the moment. What came back is richer than any questionnaire could have been — and learning to read it is half the adventure.
Part II
One session, opened layer by layer
Here is a single session — Jennifer, violin and piano — taken apart one layer at a time. Each layer is a real recording or a real measurement drawn from it. Read top to bottom and the whole method assembles itself in front of you.
The room — and how many ways she sees herself in it
Where everyone and everything sat, that day.
Two kinds of camera, and how the feedback works. Three 360° context cameras quietly record the whole room — the musician, the researcher, the instruments — so every interaction is on record. The four feedback cameras are iPhones (small, easy to place); through Live Multicam (Final Cut Camera on iPad, an all-Apple setup) their four feeds combine on the iPad the musician holds, and the iPad is mirrored by HDMI to a wall screen so she can see herself at room size while she plays.
One thing worth saying plainly, because it shapes everything downstream: the room is full of images of the musician. She appears as her actual body, of course — but also on the iPad in her hands, on the room-sized wall screen (and, in some sessions, a projector), in a standing mirror, and as the four live tiles, each of which is itself a picture of her from a different angle. This is exactly what makes the experience so rich for her: she is surrounded by herself. It is also the single most interesting puzzle for reading the recordings automatically — because the context cameras see all of those screens too. Where a person stands, the computer may find not one figure but several: the real body, plus its reflections and on-screen copies. Telling the one true body apart from its echoes is a real part of the work, and Part V returns to how it is being solved.
Putting every camera on one clock
The synchronization step — before anything can be compared.
Each device started and stopped on its own. The first job is to align them all onto a single shared clock by matching the sound they picked up in common. Only then can "the same instant" mean anything across cameras. Everything after this is measured on this one timeline.
The room, from the three context cameras
Three real 360° frames, the very same second (45:28) — drag any of them to look around.
The room seen three ways at the same instant. Each 360° camera captures the complete sphere continuously; here all three are frozen on one second — Jennifer playing violin, the researcher nearby, the piano, the mirror screen. Drag any frame to look all the way around.
What she chose to watch of herself
The feedback screen — same second as above.
The very screen she was watching, at that same instant — the four feedback feeds at once. She chose this four-view layout and could switch or zoom any tile.
- 1 — overhead, looking down at her hands and bow
- 2 — from the side: posture and bowing arm
- 3 — from the front: face and how she holds the instrument
- 4 — the whole room, wide
Finding the people, following them
Person tracking across the whole session.
The computer locates each person and follows them through the entire session, keeping the same identity minute to minute (each panel is a different moment, the same tracked musician). This is what makes it possible to ask "where was she, and in what posture, at any second" without a human scrubbing 77 minutes of video by hand.
Reading the body — the hip angle
Jennifer's own body, the same instant (45:28).
One simple, honest number, straight from the body's own joints. The computer finds her shoulders, hips, and knees, and measures the hip angle — the angle at the hip between the trunk (hip→shoulder) and the thigh (hip→knee), on the right side, the left side, and down the middle. When the trunk and thigh line up straight the angle is near 180° (the hip is open); as the trunk folds down toward the thigh the angle gets smaller (the hip closes). We deliberately do not call any value "standing" or "sitting" — the same angle can be either, and the interesting thing is how the angle moves, not a label. Because it compares the body only to itself (trunk against thigh), it does not depend on any "which way is up," so a tilted camera can't fool it. It is computed on every frame where those three joints are visible. (One honesty note: this is a flat angle, measured on the picture — a 2-D angle on a tangent plane at the hip, not a true 3-D one. It can't tell depth, i.e. the body leaning toward or away from the camera; reading real depth would take the three cameras together, which is on the horizon in Part V.)
Listening: music, talk, and both at once
The sound drawn out, and what the sound is.
The top row is the actual sound. Below it, a sound-recognition model listens in short windows and labels each as music or talking — run on several audio sources and cross-checked, so one wrong guess doesn't count. The dashed line marks when the iPad feedback came on (24:08). This is what makes the think-aloud tractable: the machine points to exactly where talk is, so transcription and reading start in the right place.
The words she spoke — one small example of coding
Automatic transcription, then a single illustrative pass of open coding.
Her spontaneous commentary is transcribed automatically. Here is a real stretch from around 44–47 minutes with a rough first-round open code on each line (what the utterance seems to be about) and the research question it hints at:
| time | her words (auto-transcript) | rough first-round code | RQ |
|---|---|---|---|
| 43:54 | "I'll focus on my facial expression. That I haven't done." | sets a new self-observation goal (expression) she'd neglected | RQ2 |
| 44:16 | "It's hard to look at facial expression — when you're making a facial expression, you're not really looking at a specific place." | a difficulty: unlike "watch my bow hand," an expression has no fixed spot on the screen to look at, so it's hard to self-monitor | RQ3 |
| 46:56 | "I like this view. I don't know why." | a felt, not-yet-explained preference for an angle | RQ1 |
| 47:00 | "Oh yeah, this would be helpful." | recognizes the practical value of a view | RQ3 |
(The 44:16 line, unpacked: she's saying facial expression is hard to watch yourself do — when you're being expressive you're not fixing your eyes on any one place, so there's no specific target to observe, the way there is for the bow hand or posture. The transcript here is automatic and un-corrected; the code is one rough first-round label, purely to illustrate. A more developed coded example is heard in Part IV.)
Everything on one clock
All the layers, stacked on the single timeline — and the exact moment shown above.
The whole session at a glance. Every row is one layer — the sound, what the sound is, the talking, and which cameras were rolling — on the same 77-minute clock. The red line is the instant shown in layers 3–6 (45:28, mid-violin, all four feedback tiles live). Any second of any session can be opened this way.
Part III
The twenty-six musicians
The musicians span a wide range of instruments — and many play more than one within a session (14 of 26 did). Voice counts as an instrument here; the range even includes the Vietnamese đàn tranh (a 17-string zither) and the suona (a double-reed horn).
And the sessions themselves vary in length — here they are on one time axis, each bar as long as that session actually ran, from David's hundred-plus minutes down to Yipeng's half hour (Jennifer, opened above, highlighted):
Twenty-six sessions, drawn to the same clock. Each is a full multi-camera recording with the same layers shown in Part II.
Part IV
What we are already seeing
The analysis has only begun, but real results are here — the kind of thing this study exists to find, and the kind you can look at now. The strategy is to take one aspect at a time across all the musicians — rather than finishing any single person — so each question is answered with the whole cohort, not one case.
1 · How many angles they watch at once
First, what the screen itself looked like. In Live Multicam a musician can hold a single view, a two-up, or a four-up — and within any layout a tile is either showing a live feed or sitting black (a camera not yet connected, or switched off). So the real question isn't just the layout, it's how many tiles were actually live. Here is a real example of every state we detect, pulled from across the cohort:
Every layout × live-tile state, from real sessions. Top row: a single view (black, then live) and a two-up (0, 1, 2 live). Bottom row: the four-up with 0, 1, 2, 3, then all 4 tiles live. Faces and device names are blurred; the black "Connection Available" tiles are exactly the ones we don't count.
Now we can measure, second by second, how many angles each musician actually watched at once — counting only the tiles that held a real image (the black ones above ignored):
How musicians split their attention (22 sessions with the detector run so far). Each row is one session; the bar fills the full width and is divided into the share of time spent watching 1, 2, 3, or 4 live angles at once. The number at right is the average.
Some watch everything at once; others take one angle at a time
Richard kept all four angles live the entire session (3.99 of 4 on average), with Colette close behind; Matt watched three at once throughout. At the other end, Nicole Collins, Khue, and Lina stayed mostly on a single angle. This split — panoramic vs. focused self-watching — is a direct, visible answer forming for RQ1.
2 · Reading the body from the hip — a direction we are holding, on purpose
Posture is the second thing we want to read, using the hip angle from Layer 6 — trunk against thigh, from the body's own joints. The machinery is built and the measure itself checks out. But we are deliberately not reporting posture results across the cohort in this draft — a few honest problems have to be solved first, and we'd rather flag them than show numbers that could mislead. This is the most exciting unfinished corner of the work, so it's worth being precise about exactly what stands between here and a real result.
First, the good news — the measure is sound. Because the three 360° cameras share one clock, we can compute the very same hip angle from each of them independently and check they agree. They do:
Three cameras, moving together. Jennifer's hip angle, measured independently from each camera. The three open and fold at the same moments — the pairs agree at r = 0.91, 0.91 and 0.82 (kept as three separate numbers, not averaged). They sit at different levels because it's a flat 2-D view (foreshortening), but the change is shared — so the measure captures real body motion. That's the encouraging part: the tool works.
- Whose body is it? The tracker detects every body in view — the musician, the researcher, people shown on the wall screen and the iPad, and reflections. Right now a chart would have to follow "the body on camera the longest," which is not guaranteed to be the participant. Until the musician is properly separated from the researcher and the on-screen copies, a per-musician posture line can simply be the wrong person. This is the big one, and it's the very next step.
- When exactly did "before" become "after"? The before/after-feedback comparison needs the moment the iPad screen came on — and a few of those screen-recordings are still missing, so some onsets are uncertain or unknown. Completing the iPad record (including reconstructing the screens that weren't recorded) comes first.
- It's 2-D, not 3-D. The angle is flat and single-viewpoint; it can't see the body leaning toward or away from a camera. Reading true depth means triangulating the three cameras together — the cross-camera agreement above is the first step toward exactly that.
None of this is a dead end — the per-frame angle, the three-camera check, and a fair before/after comparison are all built and waiting. The moment the musician can be told apart from everyone and everything else on screen, this turns straight into a real, per-musician posture finding. We're holding it until then rather than dressing up a provisional number.
3 · What they talk about — and how we read it
Reading across the transcribed sessions, a first pass at the recurring topics — the things musicians (and the occasional researcher prompt) actually put into words while watching themselves:
First-pass topics across 26 transcribed sessions. A rough automatic coding, to be refined by hand — but the shape is already clear: what to practice, which angle to watch, the hands, how it compares to what they expected, and their own posture come up again and again.
But a chart of topics is abstract. Here is what actually sits underneath one bar of it — a single real moment, read four ways. Bryan is playing his đàn tranh and talking about it at the same time — showing how he plucks. First, listen to the same few seconds three times: the room as recorded, then the two "stems" we separate it into — the instrument alone, and his voice alone.
① The room, as recorded
Everything at once — his playing and his voice mixed, the way the camera heard it.
② Just the instrument
The music "stem" — his playing, lifted out on its own.
③ Just his voice
The voice "stem" — his words, lifted clear of the music. This is what becomes the transcript.
Why split the sound? Musicians talk while they play, so their words are buried under their own instrument. Separating the recording into a music stem and a voice stem (an audio-source separation model) lets the transcription hear the words cleanly — the voice stem is literally what the transcript is made from. Here you can hear the đàn tranh plucks in ② and his words in ③ pulled apart from the mixed room in ①.
Now the words that voice stem produced, and the codes a first-round pass placed on them — the study's core analytic move, turning talk into evidence. A code is a short label the analyst attaches to a phrase for what it is doing. These are kept deliberately close to the words — we name the gesture he describes (plucking upward with the thumb), not a musical result the words don't establish:
"…sometimes with my thumb, I'll pluck up."
"…kind of pluck up and stuff."
Bryan, plucking his đàn tranh as he explains. The codes stay literal: his thumb, the upward pluck he's describing, his monitoring of his own playing, and the tentative, approximate way he puts it ("kind of… and stuff"). We resist reaching for a term like staccato — that's an interpretation the words don't yet support, and first-round coding stays honest to what was actually said. This is a "hands & technique" moment, one of the most common topics across the cohort (hands came up in 23 of 26 sessions).
And this is where that one moment rejoins the big picture. A code isn't used once — across Bryan's whole session (2247 coded utterances) the same codes recur and group into a handful of families. Here are Bryan's families, largest first:
What Bryan's talk is made of. The count is how many phrases received a code in that family; the examples are the actual code labels inside it. Self-monitoring & analysis is far and away the largest — noticing, analysis, reflection, evaluating — which is, in the end, the whole point of letting a musician watch himself.
4 · Do they talk more once they can see themselves?
Layer 7 showed Jennifer talking more after her feedback screen came on. Because every session has a shared clock and a known feedback-onset time, we can ask the same of everyone: for each musician, compare how much they spoke (seconds per minute) in the warm-up before the screen came on versus after. No threshold, no labels — just the sign of the change, one bar per session:
Change in talk-rate once the feedback screen came on, 24 sessions. Each bar is one musician; green to the right = talked more after seeing themselves, amber to the left = talked less. The number is the change in seconds-of-talk per minute. Jennifer marked.
Part V
How this works — and why it can scale
Everything above was read by a human only at the very end, to check it. The reading itself — finding the people, following them, measuring posture, separating music from voice, transcribing, labelling sound — is done automatically, then cross-checked. That is deliberate, and it is where the method itself becomes a contribution, not just the findings.
Two principles hold it together:
- Nothing rests on a single model's word. Each thing that matters is measured several independent ways and the answers are triangulated — several sound-recognition and transcription engines "vote"; posture methods were compared head-to-head on this study's own footage before one was trusted; a person's identity is checked across all three context cameras at once. When independent methods agree, we believe it; where they disagree, that disagreement is flagged rather than hidden.
- What a human would do by eye, made repeatable. A person could watch every session and note posture, glances, and talk — but not 26 sessions × ~5 cameras × an hour each, and not the same way twice. Turning that reading into a repeatable pipeline is what lets the study look at all twenty-six the same way — and, just as importantly, what would let the very same reading be pointed at a far larger group later, without starting over.
So it is fair to describe this as a rich case set of twenty-six that doubles as the proving ground for a method built to scale beyond them. The depth of each case and the breadth the method promises are not in tension; each is what makes the other worth doing.
- Telling the real body from its echoes. Because the room is full of screens and mirrors (Part I), the cameras sometimes see the musician several times over — once in the flesh, again on the wall screen, again in a reflection. The hip angle is measured on every visible body; separating the one true musician from its on-screen copies (and from the researcher) is what turns those measurements into a clean per-person result.
- Recovering a self-view that wasn't recorded. In a few sessions the iPad's own screen recording wasn't started. The view the musician saw is being reconstructed from the context cameras — locating the iPad in the room and un-warping and cropping that region back into a clean picture of what was on it.
- Recovering sound from a mic left off. Where a microphone wasn't switched on, the moment is re-aligned using the several other microphones and the musicians' movements across cameras — so the timeline stays whole even when one channel is missing.
Part VI
Research questions
The proposal's questions still point the right way, but the data — what musicians looked at and what they said — suggests a cleaner three-part arc: what they choose to see, what they notice, how they respond.
How do musicians choose to observe themselves — which angles, framings, and moments — and how do these choices vary by instrument, experience, and what they are working on?
Answered from the record of the views each musician selected (Part IV shows this beginning).
When watching themselves play in real time, what do musicians attend to and prioritize — in posture, technique, expression, or sound?
Answered from their spontaneous commentary and where they choose to look and zoom.
How do musicians make sense of and respond to what they see — especially when it differs from what they expected — and what do they find valuable or difficult about observing themselves this way?
Answered from their commentary and the adjustments they make as they play.
This folds the proposal's questions about reconciling discrepancies and about benefits and challenges into one (RQ3): in the data they surface together, in the same stream of talk. Keeping them apart would cut apart what the musicians keep together.
Part VII
Where this goes
The pieces are now in place: the recordings are synchronized and the layers (views, tracking, posture, sound, speech) are built. The plan is to move one aspect at a time across all twenty-six musicians — rather than finishing any single person — so each question is answered with the whole cohort, not one case.
The order of aspects:
- Posture — the hip angle (held, see Part IV·2). The measure is built and validated across the three cameras, but the cohort result waits on the step below — telling the musician apart from the researcher and the on-screen copies. That separation is the immediate next piece of work; once it's done, the per-frame angle and the fair before/after comparison already built turn straight into a real posture finding.
- Forearm and hand alignment. The same joint-angle treatment for the playing arm and hands — where technique lives — using the tracked keypoints already computed.
- Angle choice and view content (already producing the Part IV result) — extended and broken down by instrument and experience.
- What they say — full multi-coder grounded-theory coding of the think-aloud across the cohort (one pilot, Bryan, is already coded, as heard in Part IV), turning the first-pass topics into grounded themes.
Each aspect produces something concrete to look at and discuss — like the results in Part IV — with the literatures on visual dominance, self-regulated learning, and the costs of self-observation ready to make sense of them. The work can be steered along the way rather than judged only at the end. That is the spirit of this draft: a door held open, opening wider as the data comes in.