A different take for a curious mind?

Hi Taron,

The talent you show in your work: the art, music, writing, and software development is humbling. Your positive attitude which comes across in the posts here, on your blog and in your videos is inspiring. I've been web-stalking you a little bit recently - hope you don't mind I tend to do that with people that amaze me in one way or another. Thank you for giving so much of yourself in so many ways.

I came onto Verve by way of some other forum and after quickly running it and being overwhelmed by the many knobs I left exploration for later. When I finally took some time to watch your YT videos and actually made an effort to discover the UI, I was amazed at the power Verve provides. The fluid simulation you achieve is unbelievable and I can see that pretty much any kind of painting effect can be obtained. I'm hoping, though, that the UI can be made even more efficient yet at the same time easier to pick up for new or less frequent users. Based on the updates you've been giving, it looks like you will be addressing UX among other things. WRT this, can you tell me if in your ideal UI you see touch being very important in the way of being simultaneously used along with a stylus?

In most art software that has incorporated touch, it's used for panning, zooming and tool selection/configuration. It's for this last purpose that I think I've thought of a novel way to use touch. Seeing that you're the type of person that approaches UI/UX in a way that clearly shows creative thinking, maybe you could think about implementing what I will attempt to describe - at least to prototype it and see if it could be as useful as I can imagine it to be. The fact that Verve calls for a decent GPU is a plus for what I'm about to suggest since it would benefit from acceleration.

Let me preface the description by stating that I'm OCD about workflow efficiency - this is a driving factor for me in choosing apps that allow for movement to be minimized and for the focus to remain on the primary task. As a software developer myself (though not in graphics for quite some time), I've thought about implementing a prototype myself - I'd probably use Unity3D, but the stage Verve is at and your own personality (which I infer) makes it seem that you might be open to at least discussing new ways of interacting with software, if not to actually trying out some of the ideas.

So, assuming the stylus is held in one hand, how can we maximize what the other hand does with touch? How, at the same time, can we not obstruct the canvas unless it's for operating the UI? What you're doing in terms of showing a knob at the pentip is OK to a degree, but two of the problems that come to mind are reduced discoverability and too much memorization of keyboard shortcuts to bring up the various controllers. Pie menus (aka marking menu, radial menus) offer another solution, but the problem is two-fold there as well. First, they're usually meant to be invoked by a mouse or stylus. This breaks the flow of what the hand that is controlling the cursor is (or should be) mostly doing in a sketching/painting application (which is what I'm targeting) - artistic, flowing strokes and not picking through a menu (often multi-level). The second problem is that the breadth of features that need to be interacted with is so great that these kinds of relatively static menus/toolbars can't handle showing more than a limited subset at any one time so as not to overwhelm the user.

With the above issues in mind, would it not make sense to use the non-stylus hand to operate a fluid UI while the stylus hand simply picks and drags within that UI? But, isn't this what a toolbar or radial menu already allows for if operated using touch? Well, not really. Imagine if you will, a blank canvas. Single touch is discarded if pen near screen to avoid false strokes. If pen is away, single touch can be used for panning or, possibly, some other operation assigned to a double-tap or tap-and-hold, etc. Still the potential is limited. Two fingers are more useful when a pen is around, for both panning and rotating the canvas, but again, possibilities are limited without nesting.

OK, there's still nothing new here. How about three fingers (and maybe four, but for now let's talk about three). What if, when you place three fingers (and only three) on the screen/tablet, a translucent (or not) overlay is shown? It would not be centered at the geometric center of the triangle formed by the fingers, obviously, but rather would be a puddle-like "blob" that extends away from the three fingers (thinking thumb, index and middle or thumb middle and ring) providing touch and towards the hand holding the stylus. Hmm, still not very exciting. The magic would be in the implementation of this UI blob. The exact content and size of the blob would fluidly change depending on the relative positioning of the fingers to each other. A particular three-finger press can be pretty easily memorized in terms of the positioning and spread of the fingers, but what happens if you put down your fingers and get something other than the tools that you wanted?

Well, you don't lift them up and try again in geometric configuration hoping to get it right. Instead, you rotate them or change the distance between the fingertips and the blob overlay fluidly "scrubs" (or maybe morphs) between adjacent/related tools. At any time, part of the UI would be larger, easier to click on, while on the periphery you would see UI elements that can be brought into "focus" by continued movement or rotation of the fingers. Think fisheye or hyperbolic visualization where the outer boundary of the UI blob wouldn't necessarily be a regular shape, but rather puddle-like. I'm not quite sure what the ideal layout of the tools would be, but one possibility would have them arranged radially with sub-tool options further out from the three fingers. So again, the main idea is that a simultaneous rotation of the three fingers would scrub/switch the focused tool, while changing the position of the fingers relative to each other could maybe allow some other "dimension" of the UI to be navigated to, i.e. they could act as sliders.

Selection of a particular control (button, slider, widget) would most likely be best done with the stylus, but it might be reasonable for certain types of interactions to allow complete operation with the touch hand. This might work by scrubbing a particular tool into focus using rotation and, as mentioned above, using sliding motion of the fingers to reach sub-tools, but not to actually invoke them. Invocation could possibly be done by lifting one of the three fingers off the screen/tablet and, with the UI remaining as is in terms of content and selection, tapping the lifted finger once.

An implementation of the above would probably work best as a vector (rather than bitmap) overlay so that that it could be smoothly scaled as the fingers morph the UI and bring the various parts into focus. Of course there are many details that would need to be worked out by actual trial such as ergonomic layout, sub-tool navigation, the actual kinds of movements that would minimize the risk of RSI and so on.

I'm not sure I've given you a clear picture of what I'm imagining, but, without visuals, I've tried to paint a picture in words. It would be cool if you were to say that you've been thinking pretty much along the same lines. If not, well then, maybe you could still consider it for some time down the road. I'm curious to hear any feedback you might have or to provide details that I might've overlooked.

Cheers and best wishes,

Adrian

[Taron]

Hi Adrian,

I'm still reading your post, but right off the bet: On something like the Cintiq, you cannot use touch and pen at the same time! I was hoping you could, but it doesn't work. Particularly that device is acting poorly with touch, even while it has multi-touch. Responsiveness is shamefully weak. I was excited about hooking up multi-touch, though. Pity I only have the Cintiq to experiment and it simply sucks for that.

I like your ideas and wished there was a platform to support further exploration. By the way I'm using the keyboard, you can already tell that I like where you're going with this. I'd love to be able to allow customization to get as deep as it would take to have artists come up with their way of working with Verve.
For the Joslyn Museum of Art in Nebraska, I've worked with Nanonation on an implementation of my Verve (technology) into an interactive exhibit. It uses a laser frame based "touch screen", which can capture the horizontal and vertical rectangle covered by an obstruction, but only register up to two touch points. However, one challenge was to implement a squeegee, you know, like a window wiper, to push paint around. To do that I came up with a solution that cuts a gap into the lip of the squeegee so that it has two contact points rather than just one big bar. I then calculated the center between those two points and the line from one to the other so Verve could get the squeegees orientation and size at the same time. Works fantastically well! ...the other cool thing about this is that the rectangle the laser frame gives me is used for the brush size, making it stunningly realistic to use an actual brush on the screen. It's amazing!

SO, I'm familiar with and excited about multi touch. Sooner or later, once I have any funding, I will see that I get the latest generation of tablets and explore possibilities for those. I know there's a lot that could happen to lead to the next generation of intuitive content creation ( boy that sounds cold ). I'd be very thrilled to leap to the cutting edge on that for sure!


All in all, stalk away, I'm very flattered, happy and grateful about your enthusiasm and readiness to share it with us all here!
On that note: WELCOME TO THE FORUM, Adrian!

Pilou, does that image show the evolution of a hand after too many three-fingered gestures? Ouch!

Too bad about the Cintiq support for touch being so bad. Hopefully future Wacom tech will be an improvement, but I'm not holding my breath. I'm not very impressed with their advancement given their near position in the industry and the extremely high premiums (for what you get) that they demand from their users. To counter that kind of price-it-as-high-as-users-will-bear attitude, you have companies like Tactonic Technologies which has developed an unlimited multi-touch surface technology that is very, very inexpensive and which can be implemented at large sizes and applied to non-planar surfaces. In their previous embodiment as Touchco (which was acquired by Amazon), they were talking about something like $10/sqft in 2010. An interesting fact - Ken Perlin (of Perlin noise and general computer graphics fame) is one of the people behind Tactonic as well as Touchco. I've inquired to see how much their developer kit goes for and if it's reasonably cheap I'll probably get one to play with. Check out how you could make arbitrary controllers or how you could have touch/pressure sensitive stylus input that doesn't require a special pen.

On my Surface Pro 3 simultaneous touch and stylus works pretty well though it seems that in most apps, the stylus disables touch when it's near the screen. I'm not sure if this is inherent in the technology or imposed by the app, but this wouldn't matter too much for the kind of interaction I described as the UI overlay could just remain visible, but frozen in a particular state when the stylus is near. The selection would still change in response to stylus movement and a particular tool/sub-tool would still be selectable.

Do you have any videos of the work you did with Nanonation? Btw, how long have you been working on the algorithms you're using in Verve?

Thanks for the welcome here, Taron. Regards

[Taron]

Neat, sounds very interesting for sure!
I've written my first fluids over 10 years ago, I think, for the first fully digital fire in a Hollywood movie called "Gothika". After that I had a long gap. A year ago I had this funny idea to try using openGL for fluids and it worked immediately. Verve was born. So...in a way I can't exactly tell "how long", except that a combination out of the two would make it 1.5 years, hehe, with a 10 year gap.

Once I've got all my stuff up and running the way I like it and eh... we all like it, I'll be thrilled to move on to try the wackier stuff for sure! My goal, however, would be to create a really pleasant painting experience above all else. What ever feels pleasant then will be the thing I'd pursue.

Gothika was made by a very famous French guy!
But his american periode was very short!

[Taron]

Mathieu Kassovitz, yes, very hyper active at the time, that's for sure. He looks- if I may say so- adorable and sweet, but at the time he was kind of bouncing off the walls, haha. He came to me early on, and he came closer than Americans normally would've, haha, and he asked me, if I could make the fire elegant, I think, dancing delicately, I don't remember exactly. At the time it wasn't at all clear, whether it was really possible to begin with, haha, but somehow I pulled it off...damn near miraculously, I might add. And then a very different hell broke lose...it was a very harsh ride, this movie. But I blame foolishness outside of technical or artistic challenges and solely accuse idiotic politics to having had caused this to tumble about like a drunken barber about to shave. Somehow it all fell on the proper cheek, but it changed everything for me at the time. Considering where I am not and how I feel, I'm grateful for the curious madness of the universe every now and then, haha. Thou shalt cherish the unfathomable means by which we paddle through the currents of our river's arms.

http://www.anyclip.com/movies/gothika/sheriff-on-fire/

[Taron]

Yes!
It was so much cooler earlier on, but their panic and weird decisions created enough turmoil to drive me out of my mind. I hated the cheesy compositing around him, taking away the actual convincing power of the fire on him.
Funny you'd find the scene like that, hehe... curious!
...I've had my hands in many of the major shots, though, not necessarily to my pleasure, as the CG supervisor, who I really liked a lot, kept calling me over for many of them as well as having me do a bunch of more shots than I was ready to take on. It was almost relaxing, when I put the heat into and around Halle Berry just via After Effects. I helped timing out the big mirror shot, too, which was a complete mess before then, umpf. That was a fun thing, though, because I didn't have to do it myself, hahaha.

Ah well.... memories. Oh man, I really gotta shake off those bad feelings, though. Such layers of ugliness involved in the process. I can honestly say that this had been the most unfortunate project of them all. Followed by some of my most cherished moments, though, with some of the stuff I got to do after that. Weird things people wouldn't consider all that awesome, but things that meant a lot to me. But I also really started to despise childish politics, which were leading to outcomes that testified for my perception of them: foolishness.

Now I just have to pull myself together some more, building a bridge between Verve's improved insides and the interface concepts I have in mind. It's definitely a bit more than just "a leap".