3.0.0.1 add NARRATE and EDIT modes

[JackMostow]

Enable story and prompt narration inside RoboTutor by adding NARRATE and EDIT modes to replicate the functionality of Project LISTEN's Reading Tutor.

NARRATE:

0. Start at first non-narrated sentence of story.

1. Synthesize pronunciations of any new words in next sentence. Add them to the pronunciation lexicon. (This should be part of normal Listener functionality but check to make sure it is.)

2. Display sentence.

3. Listen to user read sentence, with normal graphical feedback. Record it.

4. Check if ASR input contains sentence as a substring:

a. If yes, play back that portion, store audio and .seg with the sentence as its filename, and advance to next sentence.
b. If not, reinitialize position in sentence and prompt user to reread.

5. UI/UX for using hardware or on-screen keyboard to edit text

a. EDIT button at lower right brings up separate scrollable window as in Project LISTEN's Reading Tutor
-: requires new button
+: can use standard Edit widget
+: can navigate within text between pages
-: requires toggling between NARRATE and EDIT views with different text layouts

b. EDIT extends normal READ UI to allow text editing
+: more intuitive WYSIWYG interface to story page -- same screen supports both NARRATE and EDIT
+: to switch to EDIT, just start editing
+: no new button needed to enter EDIT mode
?: harder to implement than just using standard Edit widget?
?: how navigate between pages?

6. How to switch from EDIT to NARRATE

a. Tap Back
+: already exists
-: might be confusing because normally exits activity

b. Tap separate button, e.g. Go at upper right as in Project LISTEN's Reading Tutor
-: requires new button
+: visible target is more obvious
?: labelled how: mouth? Go?

c. Tap key on keyboard other than edit or navigation
+: no new button needed
-: not obvious which key to tap: ESC?

d. Return automatically to NARRATE (at first unnarrated sentence) after [3 seconds] <-- DO THIS
+: no button needed
+: may seem intuitive because expect to narrate after editing
-: could be disruptive if moves to different page

7. (How) visually indicate EDIT?

a. Display just as in ECHO, including highlighting current sentence when playing back narration
+: simplest
-: confusing?

b. Display all text the same as in regular editor, including cursor and text selected
+: avoid interfering with how editor indicates state of text

c. Display as in READ mode, but without highlighting current sentence
+: reduce interference with how editor indicates state of text

d. Display state indicator
-: requires additional display
-: easy to overlook if not apparent from the text itself
?: what form: blinking "EDIT" sign?

e. Blink entire text
+: requires no additional display
+: requires no change to text appearance
+: impossible to overlook
?: might be too distracting

8. Indicate which sentences lack narrations?

a. No
+: no implementation cost

b. Distinguish graphically by font, face, color, or underlining
+: shows what remains to narrate

9. IMPORT story: when, where, and how get (possibly illustrated) text to narrate?

a. Already in story matrix
+: no change to UI/UX
+: can update anything
?: (how) distinguish read-only from updateable?

b. Load new/draft stories from location specified in config.json
+: no change to UI/UX
+: no effort at runtime

c. Create new story in RoboTutor as a type of activity as in Project LISTEN's Reading Tutor
+: flexibility to have different story writing activities
?: how find in matrix?

d. Create new story in RoboTutor via button in debugger
+: easier to find because fixed location
-: requires new button

e. Use WRITE to enter/edit text
+: doesn't require keyboard
-: slower than skilled typing
+: could be kid activity
-: more misrecognition when target text is not known
?: how to navigate between sentences?

10. UPDATE: when, where, and how to save updated text and narration?

a. Automagically put into GitHub
-: might not have access

b. Save only locally; use some other process to retrieve updated assets

c. Save to location specified in config.json
+: no change to UI/UX
+: no effort at runtime
?: how cancel undesired changes?

d. Separate process uploads to server when WiFi available
+: factors out issue of how to update central asset

11. PICTURES: when and how to insert, replace, or move illustrations? --> DEFER FOR NOW
    a. Just say no: illustrate before importing text
    +: simplify EDIT
    ?: (how) move text between pages?

b. Illustrate before importing text, but allow moving text between (illustrated) pages

c. If text exceeds a page, duplicate illustration

d. If text exceeds a page, insert a page with no illustration

e. Let user move, delete, and insert illustrations
-: more to implement
+: more flexible

[Kevin DeLand]

Is this an interface for students, or for the people who are recording the narrations?

[JackMostow]

Narrators, at least for now. If we ever have authoring activities for kids, it would let them narrate, just as it did in Project LISTEN's Reading Tutor.