Google Tech Talks
November, 7 2007
We present a user-friendly approach to unifying program creation and execution, based on a notion of “tangible values” (TVs), which are visual and interactive manifestations of pure values, including functions. Programming happens by gestural composition of TVs. Our goal is to give end-users the ability to create parameterized, composable content without imposing the usual abstract and linguistic working style of programmers. We hope that such a system will put the essence of programming into the hands of many more people, and in particular people with artistic/visual creative style.
In realizing this vision, we develop algebras for visual presentation and for “deep” function application, where function and argument may both be nested within a structure of tuples, functions, etc. Composition gestures are translated into chains of combinators that act simultaneously on statically typed values and their visualizations.
Speaker: Conal Elliott
Most of my research is aimed at supporting the creation of interactive synthetic media content, including computer animation, human-computer interaction, images, 2D and 3D geometry, and programmable shaders. In all cases, I aim to preserve or improve on the flexibility and performance of mainstream programming approaches, while greatly simplifying the creation process.
Synthetic media programs are almost always implemented in sequential, imperative (often object-oriented) languages. My research explores use of declarative languages, resulting in much simpler and more reusable and composable programs. These languages are also more amenable to execution on parallel architectures such as modern graphics processors, because declarative languages abstract away from order of execution, removing the accidental sequentiality found in imperative programs. Even on sequential machines, declarative formulations have much simpler mathematical semantics, which facilitates automatic optimization. They also tend to be spatially and temporally continuous (resolution-independent), allowing them to adapt naturally to machines with different speeds and display resolutions.
After exploring explicit programming of synthetic media content for several years, I’m now also keenly interested in supporting artists. The goal of my new new research agenda is to give artists access to the expressive power of computers and programming languages, while retaining an artist’s working style. I mean “artist” in a broad sense, in contrast to the verbal and sequential style of an engineer. (I don’t mean to suggest that people fit neatly into these two categories.) My ideal audience includes graphic designers, musicians, and children — really, the playful and curious in all of us.
This abstract has 2796
This abstract has 2820
hello good afternoon and welcome to this Tech Talk on tangible functional programming with Connell Elliott formerly of the Microsoft graphics Research Group where you worked for several years and now he's been working on this for several years so welcome Connell okay am I on is it mic on you can hear me okay cool well thanks for coming glad to be here I want to encourage interaction it's more fun for me to give a talk when people talk back so feel free to ask questions or raise objections or whatever yeah that makes no sense at all what you're saying whatever so just makes a lot more fun for me and for people who are video conferencing in from other sites if you want to ask a question or have something to say just unmute your mic and then dive in interrupt me so I can be talking today about what I call modern marriage of usability and composability it in a particular incarnation tangible functional programming so we have we live in this world of software and occurs to me that we have a rift a fundamental rift in this world we live in and that is between using software and making software okay so we have these two halves on the one hand there's using software and and and so all the values there are sort of user friendly values and so we have things like well it's humanly usable somebody can interact with it also that it's it tends to be concrete and visual so it's a it's a sort of a right-brain visual immediate world that's what we go for especially these days with gooeys on the other hand and the other side of the schism we have the the programmer friendly half and the values and properties they are quite different and there we go for composability that's what makes it really powerful so programmers have access to this very very powerful means of communication and composability is at the very heart of it that's how we make more complex things out of simpler things and the property is that that seemingly go along with that to support it are a syntax and abstraction as opposed to say something visual and concrete so why do I even bother bringing this up well it's because I think that there's serious drawbacks that each of these two halves have properties that are of immense value to the other and that we're losing those and so here are some examples of that which is that when I shift from a library and API over to an application I've lost tremendous amount of access to functionality those very fine-grained access to functionality in a library you can call any of the API calls within arguments you want and hopefully there's a type system that sort of mediates that make sure that you're talking some sense but there's this great fine-grain access and on the other hand in sort of going along with that is that applications generally are not composable so an application you can think of as you can think of a library as a means to an end all right so software libraries a means to an end in the application is the end but it's not only an end it's a dead end so I have this wonderful composability all this expressiveness all this inventiveness at the level of the library we wrap it into an application which is the thing we give away and now it's a dead end it's no longer composable for the most part so there's a terrible terrible loss then on the other hand looking at libraries well they're generally not very user friendly so they're friendly to a very small population sort of educated engineer or write left-brain sequential type thinkers and so there's a tremendous subset of the population I say most of the population which is left out in the dark and aren't able to communicate through this tremendously powerful medium which I would suggest is a more expressive and more profoundly important medium than the written word so the medium of computation this is very important to to get past this age of illiteracy we're only as kind of highly educated and not just highly but particularly educated priestly caste can express themselves and pass their ideas down and I would say that we're not only ting out some people but more importantly for hip 4 for human we're leaving out what these people would say and I would say solution is not teaching a visual basic or Java or something like that because because we actually change the way their brain works we change their creative process in doing that and they no longer say what they would have said so so that's sort of a picture of how I see the context where we are now and what's the dream well the dream is the dream is to take the best of both of these two halves and build this rich rich accessible entryway into the medium of computation that has these two properties of its usable and its composable and when I say composable I just mean distilling down that's the essence of what we programmers get to do that's what makes what we do so powerful so any questions I'm going to get into more technical stuff that's fine so here's a yeah how applications like automate the roll so is this an application builder just sign okay so the question is what about meta applications and you give a particular example which I'm not familiar with does it do great so so here's an example of a sort of app builder that gives you some sort of visual interface and interface to what are the entry points that are accessible and how to get information out of them is that sort of a example what you mean yeah okay so that's that's a great example of something that tries to bridge these two world so I'm not saying that we never do that well I'm going to describe a way to do that and I'm guessing and I'm only guessing because I don't know this application I'm guessing that the way they'll describe find simpler more systematic and more complete but I'm shooting the dark there okay so where else have we seen this at least an attempt to bridge these two worlds where else have we seen something that is both usable and composable self employs there's two talk hot there's been a bunch of visual systems which we first in the user interface systems individual program yeah so the world of visual program has addressed this sort of space and I want to suggest that they're doing something I was going to say wrong I'm doing something rather different and in about two slides I'll say what that is okay I'm going to move on now so I'll give you another hint where have we seen usability with composability so it's unix and unix-like programs you'll write small programs and compose them together feed the output at one end to the input of another and this idea was expressed by few different people but particularly McIlroy who invented pipes and who supervised the the group in which UNIX was born and so this is what Doug says this is the essence of the UNIX philosophy it's three things first write programs that do one thing and do it well so simplicity the second thing is write programs that work together that's composability so these are the two what's and then Doug offer to how and this is what UNIX does write programs that that handle text streams that are filters of text streams so it takes a text stream in gives a text stream out and why do that because that's a universal interface so this is Unix those composability it's usability but it's not very good you know usability or composability is it okay so for example if composability is based on text stream text stream filters what happens to my gooeys where am i gooeys and here's an answer from the art of UNIX programming despite popular mythology this practice that is texturing filters this practice is not as favored not because UNIX programmers hate gooeys it's because if you don't write programs that accept any MIT simple text streams it's much more difficult to hook the programs together okay so as emphasis is mine it's much more difficult to hook the programs together so here's a technique we want to be able to hook programs together in other words composition well here's a way to do it with text streams but we throw out a tremendously rich world when we do that so instead of sort of dumbing down our world to text streams let's have it be not trim it not more difficult to have it be easy to hook GUI programs together so how might we do that how do you compose GUI programs you sort of know how to compose text stream programs that's what UNIX showed us well my approach is a couple of a couple of main ideas you can see one is just a translation of the pipe idea so what's the pipe idea and here's where I think what I'm describing and what UNIX is is different from visual languages what's called visual programming languages so so they're two parts feed the output of one piece into the input of the next okay that's the same as these visual languages but the second one is different hide the intermediate data yeah question I think that was the earlier questions just like apples Automator and boy I wish I could answer that question I'll have to learn what Automator is maybe you can tell me later so hiding the intermediate data reduces the complexity down factum and I'll give you examples of that now so that's the first part is is to translate the pipe idea but the second part is is is I think a fundamental shift and contribution and that's that that's to loosely couple interfaces and functionality okay so we want something that's both usable and composable and kind of the key insight that I'll be showing you here the key trick is to have the user interface and the functionality together but separable so loosely couple them that way we can actually mutate the user interfaces when things are composed throw away things that are no longer throw away things that are just about the communication and I'll show you an example that I think I'll show you to show you an example now okay so this is a little rough prototype of a system called eros which embodies this composable and usability so here here are three little you call them apps if you want this one is a uniform scaling function so it takes in a scale factor and an image these two are inputs and this is the output and here's another one rotate okay here's one that's just an output just a checker pattern so what happens if I want to scale the checker pattern I want to combine these two pieces of functionality in Enochs sense what if I want to pipe this into this okay well what I'd like is the result is well I'll just show you right now start fusion and here's the result I've created a new thing which is the fusion of the other two now we'll take a look what's what's absent is is the input over here in the output here that's what happens when you pipe the output of one stage feeds into the input the other but neither is visible in the result that's what keeps the complexity manageable as opposed to a what's called graphical or visual programming languages another example would be if I want to feed this output into this input okay confuse these guys now we get something that has attributes of both and they're internally connected but the interface parts the output at 1 and the input of the other have vanished that's a little preview now in order to do that you saw that the user interfaces were actually mutated during composition that's possible because of this loose coupling alright so in addition to to improving on the old beautiful UNIX vision with by having gooeys here's some other things that were sacrificed in the old UNIX dream that don't need to be and one is graphics besides graphical user interfaces of graphics period another one that's really important is is to have a rich selection of data types not just one and that that's a lot more convenient and it's also safe so especially if you're static typing so it's much more convenient cuz it means I don't have to write parsers and unparticle application like you do in UNIX it's also safe because it means that if in a statically type setting anyway I can compose these two things and know that that is meaningful you know that I'm not going to a core dump or whatever and in addition this approach that I'm talking about functional as tangible functional programming is scalable and it's consistent and so in contrast in UNIX there are a lot of mechanisms that overlap in purpose with pipes and I've listed some here some may be obvious what the connection is some may be less obvious so standard input and output clearly that that's one mechanism arguments program arguments is another it's a way of getting information in besides standard input sockets is another example it's a communication mechanism threads is a concurrency mechanism so pipes that the the eunuchs idea of pipes involves not only communication but concurrency it's very elegant very simple so threads is an overlapping technology what I'm saying is that it is that our programming paradigms our programming structuring mechanisms have all this kind of stuff and I'm picking on Unix but I think all modern systems have essentially these things and then I'm throwing in the sea and the shell scripting language because they also are doing some of the same things that pipe does which is composition whenever I define a function whenever I call a function I'm doing composition so I have the second mechanism it's because the first one didn't scale nicely I skipped over this one multiple arguments so diff is an example it takes in two strings and gives a stream oops how does that fit into the pipe model it doesn't and so diff doesn't work like the other so it doesn't follow the dream so all of these can be fixed with what I'm calling a modern marriage as opposed to UNIX a modern marriage of usability and composability so so now I'm going to move on to the second part of the talk which is which is to give you a flavor of the eros user experience and I give you a preview of that before with the demo so to dive into that I want to say I'm taking a particular perspective on programming and the implication of this may surface later so you may have objection to this slide later on and so what I'm saying programming is for the purposes of this experiment is it's a way to express two things one is interface and the other is functionality okay so programming is about making these two things an interface to functionality and the functionality itself so from that point of view you can think of programming is an authoring process it's a creative process so programming environment is an app it's an authoring app and what it's making is while instead of music or video or three cool 3d models it's making interfaces and functionality now that's the case than what is code it's a command line UI – this app right so this is a very old fashioned sort of way to to author things with a command-line interface so I want to suggest let's move into the 21st century and look at more a concrete visual sort of GUI rich well a GUI rather than a command-line interface to this app so and the code interface a syntactic interface it's very handy we can do all kinds of things store it in a file run analyses over it print it out but it's also in essential so what I'm going to let me put this really simply the essence of programming has nothing to do with programs ok programs are just a means to an end the essence of programming is making interfaces and functionality ok so if that's what we're up to I'm going to pick a framework which is functional programming because and the reason for that is well there's a few reasons I love functional programming it's also a very simple sort of pure simplified and complete formalism for creating these interfaces and meanings now functional programming I can cite like show of hands who has sort of an idea of what functional programming is Oh quite a lot of you and say 80% who who has done some functional programming Wow Wow almost the same who actively does functional programming cool maybe that's why you came to this talk alright so another way to say what functional programming is it's I think a little more description is its value oriented programming so I'm functional programming everything is a value everything everything is a value including functions and if you look at it from a syntactic point of view so semantically everything's functions syntactically everything's expressions you don't need statements there's no need for sequencing because in a functional language you don't do anything so you certainly wouldn't want to do one thing before another so notion of sequencing is gone so everything is state is expressions and what I'm going to talk about force four sort of succinctness and purpose is on just three kinds of values and they're going to be atomic values which will be things like numbers and images so atomic doesn't necessarily means small and then secondly pairs so if I've got a couple of values I can pair them up and functions and really out of these building blocks you can build anything I'm not saying that you'd want to build everything out of these building blocks but if this is a sort of complete framework so eros has four key essential ideas that combine toward this vision of usability and composability and the first one is simply this to use gooeys as a visualization of type values so when I think of a GUI I'm not going to think of it as this is the GUI of my app I'm going to think of it as this is a visualization of my value so like the number the numeral three is a visualization of the number three okay so gooeys give another way to visualize things and in particular functions are very hard to visualize in strings but they're really fit very well with gooeys so so gooeys have sort of two parts one is a structure that doesn't change with time and the other is the dynamic content that does and the same thing is true when you look at typed values and you can translate this to you know imperative object Orient there are interfaces which are static and then there's the there's the thing itself the value or the function or the object that matches that meets that interface and that's more of a dynamic thing so what I'm saying is that is that gooeys has this have the same dualism that that that are our source semantic concepts out so and in particular and this comes came as a surprise to me or at least retrospect but maybe the surprising thing is why didn't I know this before which is that which is that interaction interaction and parameterization could be mapped to each other so so a function you I can think of something as a function meaning if I can create lots of things out of it or I can equally think of it as an interactive thing and I will I'll show you example that show you several examples that so this combination of the the GUI that shows the structure and then the dynamic thing together that combination I call tangible values and I want to emphasize that I use the word values in the way that a functional programmer would so tangible meanings might be a better way I don't think there is a word in the sort of mainstream imperative programming world that it's in academia we would call it denotations so it's the meaning of a program not a program so how would this look this idea of saying that that gooeys are visualizations of values and the structure of a GUI corresponds to the type well so there are three level that said there's atomic or base types pairs and functions so what a base types look like well they look like widgets an atomic value is an atomic GUI in other words a widget so here's an example these are visualizations of a boolean value true a numeric value pi and an image value disk so that part's very simple no we're going to have this so next two pairs so when we write pairs with textual sent actually usually use a comma and a pair of parenthesis now a lot of languages don't have pairs as first-class values but some languages functional languages do and certainly in math you do so the the visual syntax for a pair would is a horizontal juxtaposition of gooeys okay so here we see three values this one is of type pair of bool and number this one is of type a pair of number and bool and this one is of type pair of number and image so there's a visual syntax repairs and if you if you look they're actually three parts it has the the one half the other half and then a handle to the pair itself so this little vertical yellow slab is like the comma and the notation of pairs or the cross and the notation of of pair types so what about functions that's the third component to our calculus and functions we lay out vertically so a function has an input and output and we simply show the input above the output now something is going on here because an input is a different kind of a thing all right so far everyone so far all the other parts where we see values and input is something different so I skip over that for a moment so functions not only visualize vertically but they're interactive so a function is is in as video as an interactive GUI where the top the input is something that I get to interact with and the bottom is something that I get to see the results back and now what do you do about multi-argument functions well this is a very old idea goes back to probably the 40s which is that every function is a function of one argument in this simple formalism but if I want functions of multiple arguments whether a couple classic ways to do that one is to have a tuple as the argument say a pair and the others to do what's called currying and that's this is an example of currying over here so less than is a function of one argument which is the number and it gives me back a function of another argument which is the other number and that second function gives me a blue so we have the usual curried and uncared representation so I'm saying they map just very nicely onto this visual syntax so let's see I'll show you what some of that looks like well I'll just use this example so here we go these are curried functions let me start with this one so here's an input if you see if I hover the mouse cursor over I got input of type double here's another input of type well it's it's of type image actually says image bool because image is a kind of images of any type okay and as an output of image also so these inputs are it's our input and name only I just in this prototype don't have a way to okay so because it's because it's a very general notion of image that's parameterize over other types so we can have images of colors we could have images of bool which you might think of as being a region of space or a black-and-white image you can have colors with transparency you can also have images whose pixels so to speak our our 3d points or 3d points with normals now you get surfaces very cool okay so okay I'll move on so functions visualizes interactive gooeys so the point that I want to draw I want to drum this in what you see here is a single static unchanging value okay so I hope that maybe create some dissonance or if there if you're feeling some dissonance listen to it because this is important that in functional programming is it's a changeless world it's a stateless world they're just our values are nothing ever changes okay so what the heck's going on here then when you see this thing turning well what's going on is that you're looking at different slices of this function so the way to visualize functions is to give the user access to the to the domain to the parameter which is hit which is the user's way of saying which slice of this unchanging value wants to look at so it's unchanging in the way that the sine function the absolute value function is unchanging this has enough state just in the same way this doesn't have state yeah question with representative a sidebar no so question is are all inputs represented with sliders and the answer's no there's a lot of different a lot of representation so here's one that's not a slider this is an input ooo aisle number inputs know you can really roll any visualization you want as an input style or output style for any type yeah so examples you'll see in this talk will be mostly sliders maybe you might see a sort of type in or type out box for the same time thank you so that first idea is visualize a typed pure that is you know unchanging values as gooeys the second one is that to allow the user not only to play with these and explore these values look at them interactively but also compose them and that's what makes this as powerful as programming this is a key thing and so how does this work well it works by an idea which I'm calling fusion here so what I mean by fusion is I've got two things and I'll just show you this again let's get rid of that I've got I have two of these functions and what I want to do is I want to use them together make one thing that combines the two of them feeding the output of one into the input of the other and in the fusion process some of the visual on some of the outward visible aspects of the two will go away and specifically that the two matching the input in the output because now that connection between them is internal so that's what happens here I'll say start fusion and eros gives me some choices here are three things that have compatible type which one do you want and I'll choose this one so this is a fusion here now there's another way to play with this which is I wanted you to know that that fusion is an operation that makes a new thing it's not altering things okay just like pipe doesn't destructively modify right the two phases of the pipe it makes a new thing which is the nation so maybe a user friendlier way to do this is I'm just going to change the mode here now when I say start fusion and they click on here those two things go away so that sort of gives the illusion of them being fused together but I wanted you to understand that it really is a construct if not a destructive operation okay now so happens I don't really like this order of arguments now I've got a three argument function or a curried function takes a number an image and another number and it gives me an image and I'd rather have the two numbers together so I'm going to do another operation here which is to flip the arguments of this function so this is a function here these three things together as a function all of these is a function these two are a function so if you ever see something above something else that pairing is a function and here we see three such pairings okay so I want to flip the argument order here oh that's more what I like and now I'd like to actually put this checker in here so I can start another fusion and I get two offerings and I'll to pick this one there we go so now I can think of what I've made in two different ways and the context of authoring tools this is really intriguing to me one way to think about what I'm doing is this is an authoring tool or it's toward an authoring tool that has the expressiveness of programming and for instance if you think of this as being like something you use inside a Photoshop or the GIMP and obviously you wouldn't because it's so immature but where it's going is it's a tool with which in very direct way you can make not just an image using a filter so let me put it this way suppose you're an artist you want to use this tool well you're going to need to use some filters who do you ask for the filters you ask engineers those engineers either have made them or they haven't if they haven't made them do they may want to make them well if you're an artist an engineer doesn't work for another artist and his values are very different so it's very important to me that artists can make their own tools and what's powerful is not the image it's the parameterize image it's the tool again that's the disc difference we usability and composability that's why it's so important to put them together so if you thought of this as a sort of replacement or evolution of the GIMP then this is this is a GIMP in which artists make their own filters not just images so that's one way to think about another way to think about it is they make interactive images rather than just images that's pretty cool – thank you so that is key idea number two is is fusion and what happens in fusion well we select compatible input and output those two disappear everything else remains and fused into a single TV and that I think is what what distinguishes us from at least most visual languages so this is an example what this looks like if we have a here's a function that takes a that takes a numeric input and gives an image and here's something that is a new American output if we fuse those two together we get this result okay no longer has any input and has this one wired into it here's another example so so I call this TV fusion subsumes function application in other words function application is a special case of fusion well function composition or what they called piping and unix it is also a special case of function composition so here we've got this same function of a numeric input here we've got a function of a numeric output we compose these together what happens well the input and output that match go away everything else stays now when I'm getting to down here at the bottom is that if you look at if you look at types and if you look at the if you look at a function composition so I start out with something of this type here I try this guy is that every if we start with a function as is type and an argument has this type when you apply a function to an argument you get something of this type so these two matching pieces have gone away similarly if you do function composition here I get have something it's my function it takes a real number and here's another function that gives a real number function composition or gives you a result of this type so exactly what's happening with the UI elements is happening the types there's this direct correspondence and this is an example of what I mean by oh did I oh oh that that gooeys that gooeys and typed values have this parallelism they have a static part I have a dynamic part and the the composition the types correspond to the structure so whatever happens at the type level is mirrored at the gooey structure level so anyway the square root function is specified as so there's an underlying collection of primitives it's actually built on top of a language called Haskell a pure functional programming language there's there's a very simple way to take any Haskell function and a specification of a UI that's type compatible so there's a little language of you is which I'll show you in a bit very very simple language of you is and and building blocks for that and it's typed and so I can I can do I can build up UIs and the UI's have their types like the type of this UI is function from R to region type of that UI is function from R to R okay so so also though there's something that goes beyond function composition function application which is we can go we can reach deeply inside of functions we can reach deeply inside of pairs so here's an example this is something that takes a number and then its second argument curried argument is it take a pair and here's something that takes a number and gives you a single real and I want to hook these up together so that's something that I've never really I don't know what to call that it's not application it's not composition but that's something that so this notion of fusion generalizes it's it's quite a general notion that's what I'm trying to say and as always if you look at these types here you can see there's the input and output that go away okay so yes please to fudges yeah so the question is does this work have a relationship to fudge 'it's so fudge it's is it's a functional programming library that was done in Haskell by a couple of guys at Chalmers it's I'd say yeah it does have relationship to fudge it's I think it's a much cleaner separation between the content and the UI then fudge it's does boy almost anything interesting that's done with user interfaces and functional languages has something to do with budgets which is incredibly rich dissertation it's not a very informative answer maybe we can go into it more later but I have to say yes all right let's take a look okay so so back to the toys so I've shown you some of this already but I want to show you apply compose compose filters we did that okay now this is another thing that I think distinguishes this work from almost all visual programming language kinds of things which is typically what's in a visual programming language is is you got functions and values and functions are visualized as boxes and values or representatives flowing across arcs or you know segments arrows it's an inherently first-order visualization in other words the visualization itself makes a hard visually syntactic distinction between functions and values ouch what's powerful about functional programming is that functions are values okay so that's what I mean by its first order so so in this system is different in a couple of ways one is that visual languages with the boxes and arrows every time you do a composition you get something that's more complex than you had before so the complexity grows and grows and grows and visual languages are a very sparse rather than a dense notation all right so pretty quickly you get just swamped so you can't so you use up a lot of space not saying very much now why don't why doesn't a syntactic programming have that issue well it kind of does at the level of we are within a single definition but when I compose functions the interface stays small and that's what I was showing in the previous few slides every time I compose two things some of the interfaces of each go away so there's not only an accumulation that happens there's also there's also contraction of an elimination so that's why even though we do lots of compositions or types or interfaces don't have to get huge just our code does okay but the code gets to be isolated and manageable most visual programming languages don't have that property so this other property which is even more fundamental because it's a semantic limitation as this first-order Ness what I want to show you here is it is that this approach doesn't have its fully higher-order so what am I talking about so I've got some pieces here here's our checker it has no inputs just an output so it's of type image here is a function it Maps an image to an image it's a swirl function here is a translation this takes a vector and an image and gives me an image and again even though I'm never tweaking this it's sort of it's an input in theory that you could think of it you know I could pull out a paint program here and or I could load it in something it just doesn't happy wire happen to be wired up so here's an example the translate function okay it's kind of interesting I want to do something much more interesting here's something called hyper translate so translate takes a vector but then it takes an image and gives you an image hyper translate is much more powerful it takes a filter so translate when you apply it to a vector gives you a filter in other words it Maps images to images hyper translate takes in a filter and gives you back a filter so instead of moving an image it moves a an effect so hyper translate takes a vector and a filter which in this case is the identity and then it takes a region an image okay now let's do a fusion here so I'm going to say start fusion and what happens well look what lit up in particular this one so so this swirl it has it really has three parts the input the output and another output which is the whole thing so we have access not only to its input and output we've access to the whole function that's what this little handle is for so now I can perform a fusion okay so now that input that was a filter went away and my entire filter the swirl filter went away because that was the argument so that function was an argument so functions as value is critical here now I get this I get to drag the swirl effect around on this image here okay that gives you a little idea but let's give you more idea of what swirl does by fusing here okay so now I want to replace this region in put this image input with another one I'm going to pick this one here there we go so now all that I have left the checker is baked in the swirl is baked in the vector input is still free I could bake it in a but I'd rather not and we could see what it looks like to move the swirl effect across a checker now a lot of other things I could do in particular there's something called this was hyper translate I could do hyper scale so what hyper scale does is it scales a filter okay so if I scale filter what that means is scale the apply the filter to a kind of a bigger region what it really means mathematically is very simple it means inversely scale the input apply a plain old filter and then for words scale the output back so if I did that let's see how am i doing in time I think I want to not show that but I can show it later if anyone's interested so there we go now I'm going to breeze through the mechanics here and if people are really interested talk to me please I'm I love this part of it but I don't know this is the best audience what I don't know if this is what you'd be most interested in and also I have a paper that really focuses on this but here's the key idea and I said this before loosely couple the the interface with the functionality and so here's the simplest possible way I can imagine saying that so a TV a tangible value is simply a pair of what I call an output okay so this Haskell code this means its defining a type called TV but it's not really a type it's it's parameter Akande java it's parameterised by another type so it's a family of types so this says an out and a valued TV where this could be double image whatever function so a TV of type a is two things it's a pair one is it's a way of outputting an A that's the GUI and it's the a itself okay so this is what I mean by a loosely coupling is that is it instead of writing code that has the UI and the functionality all munge together right that would require brain surgery and there's white require skilled programmer to disentangle just never entangle them in the first place this is a very simple idea that works very well now every operation that's done or you see these sort of diffusion in particular every operation that's done is done on each of the two halves so when I want to do a composition of TVs in one of these fusions what I do is I simply fuse the UIs and I fuse the values so particular with function composition now let me do the application because it's simpler if I have a function application then I'm going to have a function value UI and a function value on the one hand I'm going to an argument value UI in an argument value on the other hand so when I want to compose them well I mean apply I apply the function to the argument that's very simple and then I do something that's analogous to it with the two you eyes so the key idea is combine them for convenience but keep them separable for component for composability okay now you couldn't do this in UNIX UNIX gives you compose ability but it still is generating and parsing these character streams right so this approach actually eliminates the generation of the output and the and the analysis of the input it directly wires the semantics and makes a direct semantic connection so how does this work very very simple remember I said there's three kinds of values of interest base type values pairs and functions so similarly there are three kinds of gooeys okay so this is a little algebra of gooeys and it has those three things here's a primitive is a widget okay put think of a put is a it's a type I'm not going to explain there's a lot of detail put is something that that understands how to put up a GUI element okay so if I know how to a particular a widget so if I know how to put up a widget then that gives me I can wrap that up with this thing called put and that gives me a way to visualize an atomic value so that's what out is a way to visualize something of type a now given two visualizers I can make a visualizer for the pair okay and you've seen that as horizontal juxtaposition I might have actually only shown you that for inputs but it's there for outputs as well and what is lambda this thing I'm going to call lambda well that's a way to build functions build visualizations of functions not functions themselves but visualizations of them so if I have opes this one's not an input it's not an output as input so if I have an input for alpha and an output for beta a and B then then combining those two things together stacking them vertically gives me a way to visualize functions from A to B so how do I visualize a function simple I give the user a way to tell me what input he's interested in put value who's interested and I get myself a way to tell him what the corresponding output is okay so there's a new thing that comes up with his input so input has another little algebra too it's just two things it's atomic widgets and pairs okay so this is the sort of complete little algebra a little set of building blocks for building you eyes of any of these values and the neat thing is that the structure is that the operations that happen at the semantic level for composition have corresponding operations at this level and now putting those two things together we get TVs we get to manipulate TVs with Simon thing manipulates user interfaces and values and I'm going to give you it here's another view on it and then I think I might just stop there and answer questions which is that you can also look at all of this as MVC so what I've just showed you so this slide is just the previous slide with some different names so I'm going to interesting to use these names classic names Model View controller so a model for a type 4 value of type a is just an a in other words this is just a name I'm introducing that it doesn't add anything except to remind you of this idea of Model View controller a view is one of these outputs and a controller is one of these inputs so again this slide says exactly the same thing the previous slide does so now in an MVC I'll call it set of a TV in MVC for a type a as one of these is a pair of a view for a type a and a model of type a and remember model a type a is just a okay so an MVC so this is interesting a little way to cut MVC it doesn't have three parts it has two parts MVC really has two things a model and a view but views are sometimes made out of controllers and views so that's what happens here so a view is as you can make out of a pair of views but a function value view you can make out of a controller interview and similarity controllers can be atomic or they can be pairs of controllers there's another way to dice this so I think though I'm going to skip past the rest which is very technical and just get on to the conclusion which is I think we can have we can have a rich marriage of usability and composability without a terrible compromise like everything is texts dreams as we can keep what's really there wonderful modern powerful convenient about modern programming languages and about modern user interfaces and put them together and and the heart of how I do it is with this of how I do an eros is this idea of fusion and it it doesn't have to make a compromise because I don't go for a universal interface and to make this work we want the GUI and the content to be together but separable okay they're together but they're never entangled and they're a bunch of things to explore one of them is what are the implications is idea of composable MVC another is is how does this translate to really the kind of classic setting of MVC which is object-oriented imperative I don't know and another one is if I say gooeys are types well that leads me in a really interesting direction which is I have a set of sensibilities about how to design programming interfaces I make them really simple really pure really composable well have people really thought about designing the user interfaces now if my goal for a UI I'm changing the game back to the original one if a good if a game of my UI is to make a really nice user interface for a customer alright for a user rather than just a way to visualize a pure value which is more like a kind of programming language angle but do the sensibilities that we developed for making really nice elegant composable simple powerful programming interfaces translate into UIs and i conjecture yes and so it is a thought experiment take a user interface for a realistic application now imagine that this user interface is a visualization of a pure functional value and the question is what the heck could that value be what's the type what are all these things doing here it's very complex I would never design a programming interface with that complexity and it's not only complex but it's it's specialized it's not very composable as long as general as I want so so my conjecture is that is is that if we go through this exercise it'll tell us we really want to redesign our UI for our application get into something that's more simple more composable I don't know like it'd be interesting to pursue so I'm going to stop I'm gonna stop right there take any more questions and okay mistake I look at what I've typed up and I say oh here I passed in X I should have passed in X minus one yeah with Fusion I don't know how somebody would determine that that happen if they were they used a bunch of things do you think on that you happy with the outcome so I'm gonna try and repeat what I think is the essence of your question then you can correct me tell me what it really was so I think you're asking is what happens to this this nice feature of syntactic programming which is I get not just your result like the running thing but I get to look at basically a recipe for it and in that recipe I can find the thing I want to change okay whereas what I show with Fusion all you're left with is the end result how do I go back how do I see what it was that I even did how to get back to it and change it I would say this I don't know what what appeals to me is to imagine is to ask the question of what's a really user-friendly way to do that ok so I want to stay in that I want to find an answer this question and stay in this world of concreteness directness rather than going into the world of abstraction and syntax is sort of very indirect way to get to meanings and so one idea is just just a weird fantasy idea is have an interface that's like a video camera right on you doing this experience and putting this thing together when you want to look at what you did and what you want to change go back and look at this movie of you doing it and edit the movie so I mean that's sort of sort of metaphorically but that would be the user experience it would say here's something that's sort of like a movie of you doing this go back and change the movie it's just an idea I've no idea if it's a good idea or bad one laughs you would all love you is a little bit it's one of these visual languages with boxes and connectors yeah we're used to having dials that they visit our things and so they have a dial pain where you've got all your knobs and registers and stuff and on the back is the wiring pane where you connect up the components some saying it's a one side was one side the dial where you've got the dials and pots of your gauges like a – moans wondrous stuff but that wiring on how okay gether oh I see so one side has the inputs and the outputs the other side is like this all these patch cables that that go through processors and connect up things okay tations of each of these and you can blow up one of them and look at the you know instead of just having an IC that you're wired in case you can actually blow off the IC and see what logic is going on inside of that so there are some concreteness there's the view of there's the kind of outer view inputs and outputs there's the interview of all the wiring which is kind of like the code in the type theory involved and there's an interesting connection between string diagrams directing operations for particles that sort of thing and flowcharts and the way things are hooked up together so the guy named John Pius is about medical physicist who doesn't which has direct connections to higher order Thai okay videoconferencing can can know what you're talking about okay so for the sake of other people who are listening it might not have heard the question you're pointing out that that in addition to the sort of LabVIEW there's also the world of string theory and techniques and theory there okay string diagrams is used and and that there's some relationship between string diagrams and high order types yes okay I didn't know about that I'd really love to get a pointer symmetric monoidal categories encompass both lambda calculus and string diagrams to have a common structure symmetric monomial categories is a place to look at connections between pyro types and string diagrams okay cool thank you we'll show you the structure program which on his question and how do I see this what am i doing so that connection might give us a way to hook them together in a diagram cool thank you okay I think we're out of time any other questions uh you know grab me afterward but Oh downstairs would be great so we can free up this space thank very much