Organise a Meeting

Stephen, I'll see you at Ipswich tomorrow but need to organize a meeting with you before Friday. On Friday I will be going on holidays for a week down south and won't have a chance to see you next week.

Also I haven't competed the sign on tutorial for Social and Mobile Computing because I might be dropping the subject, pending an graduation check and I don't like having ghost accounts on the web. I feel it is going to be a same if I have to drop the subject because I'm doing Se rice Oriented Architecture as well this semester and the lecturer has already suggested the assignment will be to design a social web page using web services.

See you tomorrow

-SRE

Another Menu Hierachy Based on Interviews

After looking at the interview information and using informed design, the new menu hierarchy is thus;

Mode Select Screen
Select TV
Select Picture/Video/Audio Draw
Open Picture/Video/Audio Draw
Close Picture/Video/Audio Draw
Turn on TV
Play Playlist

Add Draw to Playlist
Add Album/Song/Movie to Play List
Remove Album/Song/Move From Playlist
Clear Playlist
Select Item in Playlist


Display Screen
Play
Pause
FF/Rewind
Skip
Skip Other Mode
Volume Up/Down
Zoom In/Out
Reset
Random
Repeat
Equalizer
Snap Shot
Full screen
Partial Screen

Access To
Volume Setting
Zoom Setting
Seek
Mode Priority

I have a set of gestures for the different functions. and there is some extra functionality based oin the interviews such as record and pirate (as in pirate a cd). I have only briefly gone through the gestures from the interviews briefly, so I'm working off the assumption I can differentiate between them effectively.

The Drawer analogue was one suggested from the interviews and fixes the problem with TV not needing a playlist. As it stands the system visually will be limited to suppling information to the user about the current state and not the available movements.

I'm thinking a beginner user representation showing hints towards the gesture which can be taken off when needed will supply the information about available gestures which can then let the user guess the right gesture. This comes back to the problem of making the gestures able to be recognized easily so the user can know quickly what the gestures are.

I've got some sketches of the system but since the gestures actully have nothing on screen they are a bit simplistic.

Gesture Plan and update

Been a while. Here are the results from the second gesture interviews

P1.
Play,
Pause
Stop
FF/Rewind
Skip
Pyhsical Eject
HUD
Mute
Volume

P2.
Play
Pause
Volume
Random
Burn
Next/Previous
Repeat
Pause/Play
Search/Seek
FF/Rewind
HUD
Visualisation
Equalizer
Volume
Mute
Focus
Reset

The Gestures suggested
Play:
Imitate Remote
Flag Dropping
Ref Blowing Whistle
Bounce a Ball
Putting a needle onto vinyl
Jumping
Clapping
Running In Circles
Pause
Time Out
Stop like a cop
Wheres the Remote
Pick Up
Random Movementy
Spirit Fingers
Bounces Ball
Imitate Remote
Mute
Shhh
Safe (base-ball)
kill it
Pledge
Cover ears
Next
Skip Rope
Swipe
Jump forward
Cast
Previous
Opposite of next
Seek
Rolling
Point to a particular point
Magnifying Glass
Scan Horizon
Fast Forward
Rolling (hurry Up)
Cycle Swipe
Rewind
Car Slow Down
Opposite action of Fast forward
Reapeat All (group action commonly above head and singular ussually below eye level)
Loop
Lasso (but throw at what)
Flight gear stick
Loop-de-loop
Wave Hands In air
Repeat 1
Point to a song
Attract attention (Hey you)
single action compared to double for all
Volume
Up/Down
Mix Vinyl
Cupping Ear
Turn Knob
Point to ear
Visualisation
Album View -> Albums in a rack and sorted into draws
Flicking through CDs
Opening a book
Grabing from a shelf
Equalizer
Instead of changing the levels, change based on instrument
Bass - pump
Treble - tweaters
Microphone
Guitar
Piano
Drum
Trombone
Focus
Small circles
Twist camera
Twist microscope
Random On
Shuffle Cards
Stir in a bowl
Scrumple
Random Off
Iron
Lay out cards
Flatten
Reset
Blank/Wipe screen
Wiupe Slate Clean
Scrumple then throw away
Clear Desk
Burn
Lighter
Light match
Start fire
Rip paper
Eye patch (pirate)
Arghhh(pirate)

The Menu Heirachy

The more I work on this the more I realize I actually need expert users to examine the design choices needed and the more I realize non-expert users are going to make the design more convoluted.

I am still going to have a chat with some people about the menu desing but don't expect much to come of it. I plan to talk tosome people morte about how to represent the movement on the screen. This stuff I just don't have a clue about because I've been doing the action to many time while testing the gesture code. This proximity has made it like saying a word to many time until it loses it' meaning. Their fresh viewpoint should be helpful. From them in need stuff like if I "Reel It In" like fishing, If I have curved edges to the screen would the curve be on the bottom (implying the screen is pulled up from the bottom) or curved at the top (impying the screen is picked up at the top of the motion). Further more does the object move (I.e. apple) or the scene move (3d desktop).

Righto thats enough I think I might do up another version of the previous glut menu to help explain this stuff o the people I an going to ask.

This is a version of the menu hierarchy based purely on what I think would work. It also included running 2 modes at once, this goal seemed to have drifted out of the scope as of late.

Important design choices made are:

1. The system will somehow show the arbitrary movements somehow (duffman in particular) to show the new menus.
2. The system will be be able to guess the best course of action for some actions. I.e. it can tell when to pause and play on the make it so action.
3. The double swipe action initiates the second mode version of the single swipe action.
   
4. The modes are represented as layers on top of each other.
5. The actions being performed will depend on the state to the remote. This is a problem because it into introduces more mode problems but they are one of modes. For Example as it stand performing duffman and holding the remote pointing high initiates the mode selection menu. Then the roll functions can be used to represent horizontal rotation instead of swiping since in that position the remote won't pick up swipeing because it's rotation around the y axis. But it might work.
1. As a side note I don't want to recreate apple's album chooser here it show the modes. I would like something else, but we'll see.
   
6. Even though this menu hierarchy is base on multiple modes. I would like to make a single mode version. One that make mode swapping simple but still separate from each other.
1. This version of the menu allows access to the basic functions of the second mode but a version without global access to the second mode's function might be worth testing. Also coding this should be to hard since mode swapping is an arbitrary function to a second menu therefore it should just require a new set of links and allocation of screen icons.
2. Single version would be interesting to contrast because it also could allow access to a couple more functions of the single mode on the main screen such a skip 5 sec, which aren't included in the multi modal version.
   

• Top Level
• Front Layer Next - Swipe Right
• Front Layer Previous - Swipe Left
• Volume Up / Zoom Out - Double Pitch Down
• Volume Up / Zoom In- Double Pitch Down
• Play / Pause / Choice Prompt Link - Make it so
• Repeat - Roll Left
• Random - Roll Right
• Increase Play Speed / Fast Forward - Double Right Hold
• Slow Play Speed / Rewind - Double Left Hold
• Continuous Menu Link - Reel
• Mode Select Link - Duffman (Hold)
• 2nd Mode Next - Double Swipe Right
• 2nd Mode Previous - Double Swipe Left
• Continuous Menu
• Volume Level - Change Pitch then Swipe
• Seek Position - Roll then Swipe "up" Sun Ray
• Mute - Double down
  
• Return Volume / Max Volume - Double Up
• Fast Forward - Double Roll Right (Hold)
• Rewind - Double Roll Left (Hold)
• Swap Mode - Duffman
• Main Menu Link - Cast

• Mode Selection Link
• Add Layer to Front - Reel
• Add Layer to Back - Cast
• Reset Modes - Duffman
• Next Mode - Roll / Swipe Right
  
• Previous Mode - Roll / Swipe Left
• Choice Prompt Menu
• Play / Pause - Swipe Right
• Stop / Swipe Left
  

Menu Design

This is a picture of the progression through the gestures based on the state of the Wiimote.
Initial is the holding it like a normal remote, High is oriented towards positive Y, Rolling is turn around z with forward orientation, Pitch = around the x with forward orientation and wiping is rotation around z with high orientation.


The Gestures are reduced to their basic form. I E Duffman is the progression from Initial to High to High Following Reel then Pump. It also represent the feasible menu hiearchy choices that can be made.

Having people round to night and am going to try and deside which tasks are best represented by each state and the grouping of the task to fit into each of the states.

Application Wrapper

Thoughts on how to implement the final program
1. Write the whole thing. I.E. Create a media player
2. Put a wrapper over the desktop the contrals the other applications installed on the computer

Link to how to put a glut window as the active background.
http://groups.google.com.au/group/comp.graphics.api.opengl/browse_thread/thread/681fd01ff6a70dcf/100d4918574ae32c?lnk=st&q=opengl+show+desktop&rnum=1&hl=en#100d4918574ae32c

Link to how to change YDock
http://www.dockex.com/yzdock

Post script:
Just found out that Y'a dock got a cease and disist from apple for distribution of the program. the program apparently is an exact copy of apple's dock. Ha lucky I got a version. Mind you I really wish it had a panel from showing the prgoms open so I could feasible get ride of the task bar.

Ramblings on menu Design

Standard menus use hierachial structure. Aftre watching a MS surface demo on Uyube, the idea of a menu that follows the user. In the demo as pictures are looked at the user can swipe the picture to the side to keep it in focus of the current surface mode. Applying this to task the user would be effectively selecting their own menu.

This leads the menu design towards 2 separate controls. One mode to select the task being used, and the second to control the tasks which have been selected. This appear to be an innovative way of controling the application and a fun way to explore a different men design.

Easier Than I though

5 min later...

Got Hello World running from inside the program. Now to get to winamp.

EDIT

5 more minutes later ...

I just got win amp to skip tracks using the Wiimote.
This is so much fun.


EDIT

Just for Reference

Previous track button 40044
Next track button 40048
Play button 40045
Pause/Unpause button 40046
Stop button 40047
Fadeout and stop 40147
Stop after current track 40157
Fast-forward 5 seconds 40148
Fast-rewind 5 seconds 40144
Start of playlist 40154
Go to end of playlist 40158
Open file dialog 40029
Open URL dialog 40155
Open file info box 40188
Set time display mode to elapsed 40037
Set time display mode to remaining 40038
Toggle preferences screen 40012
Open visualization options 40190
Open visualization plug-in options 40191
Execute current visualization plug-in 40192
Toggle about box 40041
Toggle title Autoscrolling 40189
Toggle always on top 40019
Toggle Windowshade 40064
Toggle Playlist Windowshade 40266
Toggle doublesize mode 40165
Toggle EQ 40036
Toggle playlist editor 40040
Toggle main window visible 40258
Toggle minibrowser 40298
Toggle easymove 40186
Raise volume by 1% 40058
Lower volume by 1% 40059
Toggle repeat 40022
Toggle shuffle 40023
Open jump to time dialog 40193
Open jump to file dialog 40194
Open skin selector 40219
Configure current visualization plug-in 40221
Reload the current skin 40291
Close Winamp 40001
Moves back 10 tracks in playlist 40197
Show the edit bookmarks 40320
Adds current track as a bookmark 40321
Play audio CD 40323
Load a preset from EQ 40253
Save a preset to EQF 40254
Opens load presets dialog 40172
Opens auto-load presets dialog 40173
Load default preset 40174
Opens save preset dialog 40175
Opens auto-load save preset 40176
Opens delete preset dialog 40178
Opens delete an auto load preset dialog 40180


WM_USER messages are sent using SendMessage(). In C/C++, you can send these messages by calling:

code:

int ret=SendMessage(hwndWinamp,WM_USER, data, id);

data is used by many of the messages, but not all. For messages where the meaning of data is not defined, simply use 0.

Here is a list of the currently supported ids that you can use from within Winamp plug-ins or from other applications (see plug-in only WM_USER messages, below, for more):

0 Retrieves the version of Winamp running. Version will be 0x20yx for 2.yx. This is a good way to determine if you did in fact find the right window, etc.
100 Starts playback. A lot like hitting 'play' in Winamp, but not exactly the same
101 Clears Winamp's internal playlist.
102 Begins play of selected track.
103 Makes Winamp change to the directory C:\\download
104 Returns the status of playback. If 'ret' is 1, Winamp is playing. If 'ret' is 3, Winamp is paused. Otherwise, playback is stopped.
105 If data is 0, returns the position in milliseconds of playback. If data is 1, returns current track length in seconds. Returns -1 if not playing or if an error occurs.
106 Seeks within the current track. The offset is specified in 'data', in milliseconds.
120 Writes out the current playlist to Winampdir\winamp.m3u, and returns the current position in the playlist.
121 Sets the playlist position to the position specified in tracks in 'data'.
122 Sets the volume to 'data', which can be between 0 (silent) and 255 (maximum).
123 Sets the panning to 'data', which can be between 0 (all left) and 255 (all right).
124 Returns length of the current playlist, in tracks.
125 Returns the position in the current playlist, in tracks (requires Winamp 2.05+).
126 Retrieves info about the current playing track. Returns samplerate (i.e. 44100) if 'data' is set to 0, bitrate if 'data' is set to 1, and number of channels if 'data' is set to 2. (requires Winamp 2.05+)
127 Retrieves one element of equalizer data, based on what 'data' is set to.
0-9 The 10 bands of EQ data. Will return 0-63 (+20db - -20db)
10 The preamp value. Will return 0-63 (+20db - -20db)
11 Enabled. Will return zero if disabled, nonzero if enabled.
128 Autoload. Will return zero if disabled, nonzero if enabled. To set an element of equalizer data, simply query which item you wish to set using the message above (127), then call this message with data
129 Adds the specified file to the Winamp bookmark list
135 Restarts Winamp

Gesture Pattern Recogniser Part 2

Got the program to recognise the set of simple gestures. It doesn't include the functions that rely on rotation before action, for example double roll right and left. The other gestures are recognised, with some difficulty. The recotgniser has a lot of room for improvement such as putting in a real pattern recogniser, but that looks a bit to Mathematical for the holidays. It can also be the base for generating the training set if that is the way the program heads in the future.

The decision made during the creation was to split the data for each movement 4 ways. Therefore each motion has 12 values. The the data was codified into -2, -1, 0,1 ,2 based on high and low break points. Then watching which values appeared when each motion was executed. Then personal choice for each action was used to decide on each of the actions qualifiers.

Now to see if I can't get it to open some external program..

Gesture Pattern Recogniser

After the last post I've been working on the pattern recognizer. The initial one show in the previous post was simple slap together job.

Trouble I had with the new one include customisable breakpoints.
Deciding when a gesture is running and sensitivity were the main concerns.

For refence the range is about -150 to 150.

Looked at 2 different ways of deciding when the gesture was started. The first version used a level (ussually 20, 30, or 40) of acceleration. This cause problem because the motion had to be jerk to get the higher values and the lower values were to easy to hit. Hence crap sensitivity.

The second versin uses rate of change in acceleration ( a jerk, i think is it's real name?). It's value is acctually only 1. It has to be low else the speed increase needed is insane. Had trouble keeping the motion going on only 1 axis but as soon as 3 were included it's not hard to create a fluid motion that keeps picking up values. Since a change is only needed on 1 axis top record all values. and the change is positive or negative.
I think the wrist itself actually helps this by give minor rotations around alternate axis.

Another problem was reducing the numbers. Pevious literature talks about noramlising the values. They used all the data and normalise properly. Both versions just took the average of set portions of the data. (I.E. quartiles if uding 4 as the spilt_size). The second version can change the split value in the code easily "# define".

The new version adoesn;'t actually look fopr patterns instead it stores each motion and which each of the spilt vaues X1, X2, X3, Y1,Y2... can be averaged. This works as a simple normalisation.

Since it simple int division the remained data value are ignored. This I think is acceptable because the simple gestures such as swipe only need to split the data into 2 to see the pattern. And for more complex actions, the data size increase so the remainder becomes less inportant.

The new version can rewset the trainer, read the averages so far. It doesn't save values, so it must be used in combination with writing the recogniser. Also it can't remove bad motion I.E. mistaken actions.

Might work on adding these things and then testing it with a few people to get the atual values to be used. Or aiming higher includeing a training program in the final product. Sounds good but the nuances of the motions are easiy confused if you don';t understand how the values are beiung read.

Found it particularly difficult to explain a jerking motion (for the initial program) to a friend from work. One the upside, once they had it, they understood the entire idea behind the gesture part of thesis. This finding from a hand's on approach might be useful later one.

Gesture Recognition

This is serious fun.

Gesture Recognition Ideas and Exploration

2stage Recognition of Raw Acceleration Signals
Relys on on or off states
Recognition Framework (
1 . Calibration,
2. Preprocessing
1. Motion area detection: interval of a gesture motion in time
2. Norm Normalization: Allowing for gravity
3. Gaussian Smoothing: Remove hand trembling
4. Resampling: Normalise in respect to motion speed
3. Feature point Extraction: determine the distance between the sampling points
4. Recognition:
1. Bayesian Network modeling of relationships
2. find model with highest model likelehood
5. Confusing pair discrimination
1. Determine between confusing models I.E. 0 and 6
2. Double intergration doesn't work becaue of double intergration error accumulation

Chapter 10. Gesture Recognition, Mathew Turk
Hummels and Stappers 4 aspects of a gesture wich may be important to its meaning
1. Spatial: where it occurs, location gesture refer to
2. Pathic: the path which a gesture takes
3. Symbolic: the sign a gesture makes
4. Affective: the emoitional quality of a gesture
Sturman suggested a taxonomy of whole-hand input that categorizes input techniques along 2 dimnentsions
1. Classes of hand actions: continous or discrete
2. Interpretation of hand actions: direct, mapped, or symbolic.
System Design Suggestioncs
1. Do inform the user
2. Do give the user feedback
3. Do take advantages of uniqueness of gesture
4. Do understand the benifits and limitations of the particular technology
5. Do usability testing of the system
6. Do avoid temporal segmentation if fesible
7. Don't tire the user
8. Don't use gesture as a gimmick
9. Don't incrase the user's cognitive load
10. Don't require precise motion
11. Don't create new, unnatural gestural language

Hidden Markov Models for Spatio-temporal pattern recognitions
Conclusions
1. Baum-Welch algorithim is a hillclimbing technique that is generally unable to fiund the gobal maxima
2. Baum-Welch is often very good
3. HMM perform better than Baum-Welch on simulated HMM data, but htese result do not nessecarily translate into improved perfromance in real world applications.
Other Points
1. Fully connected topology: There is not necessarily a defined starting stte and all state transitions as possible such the Aij |= 0 V i,j E[1,N]
2. Left-Right Topology: popular in speech recognition application. There is a defined stateing stae and only state transitions to higher-index state are allowed
3. Left-Right-banded topologies: transition structure contains on;y self-transitions and nnext state transitions, I.E. state cannot be skipped
3. Gestures have a natural state and finish state and thus it is reasonable to adopt the LR model


When to recognise
Many applications require an on off state to be selected before gesture recognition becomes avaliable. Stoke-it relys on the right mouse being pressed down. Hollar et al Wireless statis hand gesture recognition suggests movement from the hand from stble and the iPhoine activates when the screen is touched.


From AiLive movie
Problems come from speed, orientation, and direction, left/right handed. Ussually uses 3 motions