Transcript

Transcript prepared by Bob Therriault, Igor Kim and Sanjay Cherian.
Show Notes

00:00:00 [Conor Hoekstra]

Are we recording right now?

00:00:02 [Bob Therriault]

We are recording right now.

00:00:05 [CH]

I didn't mean to interrupt. It's only nine minutes past the hour. Since when do we ever get off the ground that quickly.

00:00:10 [Marshall Lochbaum]

We get started fast when you're not around Connor.

00:00:13 [MUSIC]

00:00:22 [BT]Welcome to another ArrayCast. I'm your host today, Bob Therriault, as Conor is on the beach, I believe somewhere in Slovenia. And as a result, he's unable to join us for this episode. That means that we're going to go around the horn and talk to our panelists. And we will start and note this starting point. We will start with Marshall and then we will go with Rich and then we will go to Stephen.

00:01:03 [ML]

I'm Marshall Lochbaum. I'm current first panelist. But in the past, I've been a J programmer. I've been a developer at Dyalog and I also work on the BQN language now.

00:01:15 [Rich Park]

I'm Rich Park. I'm occasional panelist. I work programming and teaching APL at Dyalog Limited.

00:01:24 [Stephen Taylor]

And I'm Stephen Taylor and I bumped into APL and q sometime in my career. Not really recovered yet.

00:01:33 [BT]

And I'm Bob Therriault. And as you know from past episodes, I'm a J enthusiast. And today, I'm your host and you're stuck with me. So hope you're still going to be listening at the end of this. It's an interesting, what we feel it's going to be an interesting show. To start off with, let's begin with announcements and I believe, Rich, you have an announcement.

00:01:51 [RP]

Yeah, so by the time this comes out, registration for the Dyalog 23 user meeting in Elsinore, Denmark should be open. [01] So, largely for users of Dyalog APL, you can come see some interesting talks. We always host workshops where you can learn about different features of Dyalog or aspects of array programming. Yeah, should be a good time. That's all really for that.

00:02:20 [BT]

And I believe Stephen has an announcement just by his presence. How was your trip across France on your bicycle?

00:02:26 [ST]

Well, it was great. I went to visit my sister who moved to Burgundy a couple of years ago. And I just jumped on my bike and rode off to go and see how she was doing. Stopped off to see some friends and colleagues on the way there, on the way back.So I was six weeks away from the podcast, but I'm back in body at any rate.

00:02:47 [BT]

And we've just been podcast bombed by Conor who's appeared back in the show. So this will be very interesting. Let's see. Oh, there he is. A picture of the man himself. Has there been alcohol involved?

00:03:05 [CH]

Are we recording right now?

00:03:07 [BT]

We are recording right now.

00:03:09 [CH]

Like a pod-- I didn't mean to interrupt. It's only nine minutes past the hour. Since when do we ever get off the ground that quickly.

00:03:15 [ML]

We get started fast when you're not around, Conor.

00:03:17 [BT]

Yeah, we cut to the chase, man.

00:03:20 [CH]

I'm completely interrupted I thought-- I figured it was nine minutes past the hour. And I realized I've got 100 gigs of data. And my co-traveler has left. And I don't know how good the sound quality is going to be. But I figured I could drop in for the little intro/outro thing and then leave for the longer podcast at least.

00:03:39 [BT]

Yeah well that would be setting a precedent for our listeners that I'm not sure we would be happy about.

00:03:46

I don't know what to do now should I just disappear.

00:03:48 [BT]

No, no. How's, how's Slovenia?

00:03:51 [CH]

It's good here let me switch switch the camera around this is my view right now.

00:03:55 [BT]

And for the viewers we're for the listeners we're looking at a beach with Conor's feet on some kind of a cot and it looks like it's sunny and lovely.

00:04:05 [CH]

Yeah Yeah, and I was just in the water, but then I encountered a school of three jellyfish that were small, but then I got quite concerned. I was like, well, maybe this is a one-off. So then I swam a couple meters away and then there was another school of jellyfish. And then I immediately got out of the water and started Googling if these jellyfish could kill me. But as you can see, there are many people in the water, so they don't seem to be too concerned. But I was an actuary. Anyways, what were you guys chatting about.

00:04:32 [BT]

Is that all the announcements you have, Conor. The current state of jellyfish in Slovenia.

00:04:37 [CH]

We we that's that's all the updates I have.

00:04:42 [CH]

We we that's that's all the updates I have. I've completely messed up this. Are you in the middle of the intro or the outro.

00:04:46 [BT]

We have just finished the announcements. And in fact, you came in just at the right time because Stephen had just announced the results of his trip across France. So it's become a bit of a travel show.

00:04:57 [ST]

Are you getting any interesting comments on your T shirt there?

00:05:00 [CH]

OOh yeah, I'm wearing the Shakti shirt, no comments, no comments from the locals. I don't think they have any idea what Shakti DB is, unfortunately. [02]

00:05:09 [BT]

Well, I think at this point we'll, we'll let you go and enjoy your vacation. Thanks for popping in.

00:05:14 [CH]

Sorry to the listeners, we'll be there next time, and good luck with the episode, everyone.

00:05:25 [BT]

Thank you, Conor. Oh, that's somewhat perfect.

Anyway, I guess we should move on with the show at this point now that any listeners who were wondering what we're going to be doing, what we're going to be talking about today is learning the array languages. And, uh, the episode we had last, uh, last week was, or two weeks ago was actually a repeat of our very first episode because [03] I'm away on vacation by the time these episodes air and what we decided was that we would, we would rerun best bits from old episodes, but when I went back and looked, our podcast doesn't really lend itself to taking chunks out. It's very conversational. So instead, I decided that the very first episode was actually quite good and it's a good introduction. So if you happen to know anybody who's kind of wondering about whether they should get involved in the array languages, that's what that episode was for and we'll do announcements on that episode.But this episode is for people who might want to know more about how to learn array languages because we really haven't talked about that. They're different and there are different approaches to them and there are different resources for learning. So that's what we're going to talk about today. We also got a letter, but I'll dive into that a bit later because that's sort of a specialized thing. But I guess to start off with, does anybody have any thoughts about, what's the best way to learn an array language to get involved.

00:06:44 [ST]

Well, I think there are two answers to this. And one is how you learn the array language from scratch if you don't know any programming. And the other is how you learn the right language if you've already had your head colonized by scalar languages. I think they're very different processes because the second one, there's quite a lot of unlearning to do. I had the great privilege hundreds of years ago to be trained as an instructor in a working introduction to EPL, Ken Iverson's [04] very own How to Do It course, which was definitely an answer to the first question, how you teach people programming using an APL from scratch. And it's a great course, still is. You can find, I'd say, its descendants on things like TryAPL, which really make no assumption that you've got any prior experience with programming languages. It just shows you the stuff. But if you were now one of those rare people who actually came to this completely cold with no prior programming experience, then I think you'd find it an easy and natural way to absorb it, just like people used to do back in the '70s and the '80s. But most people who ask this question these days, "How do I learn APL or another array I've already learned at least three or four other coding languages of one kind or another. And it's a very different question. There's quite a lot of unlearning to do. Yeah, let me leave it at that for the moment.

00:08:33 [BT]

And I guess it depends on what paradigm you come from, [05] because if you come from object oriented, oriented, there'll be different challenges. And the array languages are a different paradigm. They're a different way of thinking about programming. So you sort of have to treat them like that. If you come from maybe one of the more traditional types, I don't know if that's the right word, like the C-type programming languages, that's probably common, but you're gonna have different challenges than somebody coming from object-oriented or actually I think these days the easiest transition might be functional programming languages because the array languages are quite functional.

00:09:08 [RP]

I mean, in the APL code I've seen written by different people, especially people who've not started that long ago. Well, I guess that's the contrast, isn't it really. Someone who I've seen the APL code written by and they were into functional programming before, they lean heavily on the combinators and trains and compositional ways of putting things together. So write a function that does this, than a function that does that. Much more classical APLs, it's still very procedural in flavor and they spend a lot of time modifying the contents of arrays, often with selected assignment. And what was I gonna say about this.

00:09:51 [BT]

I was gonna say procedural to me is what you would think of as a recipe for cooking. You do this, you do this, you do this, you do this. And functional is like you have different items in the kitchen that will do things for you and you put the thing in and then do it and then it comes back out.

00:10:09 [RP]

But then I think there's this other aspect to array programming, which is it's kind of in the functional sense, but it's more about thinking fundamentally about data transformations on very basic forms of, or very basic representations of what you're doing. And especially the way interpreted array languages work, If you can frame your solution in terms of primitive transformations, like a series of transformations of very small functions, you know, literally a few glyphs, then you're probably going to get better performance in those cases.

00:10:51 [ML]

Well, and at that point, the difference between functional and procedural style is kind of vanishes too, because if your program is just, well, this thing is this primitive applied to these other things.

00:11:02 [ML]

Well, and at that point, the difference between functional and procedural styles kind of vanishes too, because if your program is just, well, this thing is this primitive applied these other things. I mean, you know, you might write that more with trains if you're used to functional programming, but really you're writing the same thing. Because you're now focused on the primitives rather than the control flow. The control flow becomes linear in a lot of, I mean, in nice array programs. That's what happens. Not all problems can be solved that way.

00:11:20 [BT]

And one of the things you were, I've heard you talk about Marshall with BQN, is that it lends itself to a few different styles, maybe easier for people who are used to programming that they may be able to pick up BQN a little bit easier because it can be used in the same way, whereas some of the array languages, you feel like you're swimming upstream. [06]

00:11:43 [ML]

Yeah, definitely. BQN is definitely the best array language for object-oriented programming. For procedural, I mean, I think of APL and J as the more procedural-focused languages. Although K has a lot of. BQN is probably the worst for procedural programming. K is good as well. It depends on what version you get. Some of them have while loops, and some of them have different stuff. There's a lot of variation between Ks on how they handle this. And then, yeah, in terms of functional programming, BQN is pretty well-oriented for that. K is also very good, so I'm not going to pick a winner there. But I think they're both a bit better than APL or J in terms of how you can express functional stuff. One big thing I do see in people coming from functional languages is that they tend to use a lot of recursion, which is not necessarily a bad thing. Like I know John Scholes in APL was a big recursion fan. And so Dyalog APL supports tail calls for that reason. But I mean, if you're writing, if you're using recursion for everything, you're probably using it too much. And I mean, you should learn the array ways to do this. I mean, use each and use reductions and stuff. So in some styles of functional programming, that's already pretty common. In others, like particularly scheme and some Lisp stuff, that's not as common. So, and I mean, I think it's just a much easier way to approach stuff when it works. Call each on this array rather than recursing over some index or something.

00:13:14 [BT]

So I think Stephen touched on at the beginning, how would you get a person to start into an array language. Is it just throw a person at it or what would be the feeling. How would, if you had a friend who was saying, I'm kind of into these, um, I guess to start with, would you recommend a particular array language and you're allowed to recommend the one you're most familiar with, but, uh, you know, maybe think outside the box a bit.

00:13:40 [ML]

It's probably more about how you approach it than what language you choose. So even q, which I think of as kind of the furthest from the, you know, APL tradition has these very strong array roots. If you learn to program q well, you're an array programmer. But I mean, also I see a lot of people kind of hesitate when they're starting out because they don't know how to approach something the array way. So I think you have to balance it. On the one hand, you want to learn the array stuff eventually, but also you want to, you don't want to just get stuck and end up saying, "Well, this language isn't for me." So one thing you might might do is just starting out, learn the language. Even if you're just translating code from what you know into this language, just learn it. Be comfortable with working with it. And then you can look at the code you've written. I think this is actually much easier to look at what you've written and identify the things and say, well, I know this recursion here, this loop, whatever, isn't in an array style or this thing where I've stuck one element in an array, modified it over and over. And so then you can, you know, and asking on the forum at this point is a pretty good idea, [07] but or try to figure out for yourself, figure out, you know, how do I do this in a more array style. And then as you're able to relate these concepts to things that you know how to do, you'll kind of pick it up more gradually, and it'll be a smoother transition.

00:15:09 [BT]

And one of the things I know from working on the wiki and looking at past documentation and everything for J, there's a huge strong history of mentorship in the array languages. And I think actually the most appropriate person to talk to talk about that is probably Stephen.

00:15:26 [ST]

I don't think I know anybody who learned APL really from a book or cue. I learned it from somebody who showed it to me. And I've taught it to people that way. I if this is true, I kind of like it. And in religion, it's called apostolic succession. In my own religions and Buddhism, it's a strong thing. My teacher got it from her teacher, Sensai Orochi, in Tatokuji in Kyoto. And he traced a line all the way back to the Buddha. So that's kind of pleasant and satisfying.

00:16:04 [ML]

However, the Buddha did not know any array programming.

00:16:07 [ST]

No, no, I don't know where that snuck in, you know.

00:16:09 [ML]

No, that was a great discussion. I'm just making a joke.

00:16:15 [ST]

But what I'm really interested in is what is it that's going on in that, because mentoring, which I love doing, I love learning that way too, it doesn't scale. So it's particularly an issue in the Q world as KX is, [08] so to speak, going retail with its new strategy and providing web access. So for the last two or three decades, certainly in the KDB world, this technology has mainly lived inside the home company, KX First Derivatives, where people would learn from each other. So they've got a great, they've got lots of experience of bringing people in, putting them through a quick workshop and then mentoring them so they learn. And in specialist teams inside the customers, which for most of that time were large investment banks where some teams are very smart people and you could be recruited into that and pick up KDB and Q and get a real training in it. of the new technology that the KX is providing. And that makes it easy to get hold of data from KDB datasets through Python. It was always easy to use your SQL knowledge. That's where q came from. So you can get started in these things quite easily. And for many of the users, that will be enough. there won't be particularly concerned to get the extremes of performance out of KDB. But if you want that and you're not surrounded by people who can mentor you, then we've got a new issue to deal with, which we've never had to deal with before, is how do you learn remotely as a solo student.And I think forums of the kind that Richard and Adám run for APL or part of the solution. I'm very interested in the transition here. And as I think you know, I'm drafting a book on the subject of how you take on the vector style of programming in Q at any rate, on the assumption that you've been used to something else. You're not coming from APL, but from Python or C or so forth. So what's interesting to me here is the particular challenges, the things that people find. I've seen lots of instances of people trying to solve problems where they've used, say, an SQL-like query to hoist a whole load of data into memory, [09] and are then trying to solve the rest of their problem using SQL queries, which is like it's possible, but it's painful to watch. It's like trying to watch somebody playing the piano with their elbows. And you just know, you know, for goodness sake, just hack into this stuff with vectors. I know it's a table, but it's also, you know, a list of dictionaries. It's a collection of columns, the vector techniques will work there. So that's one stumbling block that people seem to bump up against. Another is the implicit iterations that are built into the primitives. And you kind of see that in the introductions. Oh, you can add two plus one, two, three, and it's. But most people don't seem to get or to explore just how far that iteration is built in implicitly. So you see quite a lot of QB code, which has got explicit eachs in there. It's like, you can just take those out, pal. You know, the law will happen. And as you get to realize this, You know, the code just seems to dissolve in front of you down to something really, really quite small. So that's a process I'm interested in. Another issue, which I'll merely flag here, 'cause I've been talking for long enough, is how you represent the data so that those implicit iterations get a grip. And I think that's quite a deep subject. Until you've started to really, you're at home and start to see things in terms of lists and operations between those, it's not at all clear to you how to arrange the data, what kind of data structures to use.

00:21:04 [BT]

That was one of the most powerful things I think I learned early on was that often if you're presented with any type of a problem, the first step isn't trying to solve the problem, it's trying to organize the data, which will make the problem very easy to solve. And the language is actually, especially I find in, well, I think all the array language, but especially I feel in J 'cause I use J more often, there's that first step of either adding something or subtracting something. If you're trying to work with markers or anything, you put that in at the right spot and then suddenly the problem is really easy. But without it, it's almost, it feels almost impossible, especially to do something like procedurally. But in terms of getting the message out, I know Richard, you've done a lot of videos and you've done the webinars with Dyalog. What's your feeling about sort of getting concepts out to people that way. [10]

00:21:56 [RP]

Depends, I guess what level. There are a couple of things, I think as Stephen was talking that came to mind. One is, do you start with arrays. You know, it seems so fundamental, especially when you get used to them. But actually as a rank beginner, it's quite a big concept to wrap your head around, which I think is what Stephen was alluding to with the, you know, suggesting the ways that people sort of write not exactly optimal code. And then a sort of different aspect is what things are you actually trying to do. What are you trying to solve. Because a lot of the stuff that we've done so far and a lot of the text out there that focuses on the array way of doing things are a lot of relatively small noodly problems that, you know, if you learn enough of them, you can sort of develop an intuition of how you would piece these together to solve the actual problems that you're interested in. But there isn't so much out there for people who want to use array programming to write, you know, decent sized applications. How do you actually go and solve more real problems. So yeah, you asked how do we get things out there. I'm sort of telling you what's missing, but the implication is the rest of it's there, right there.

00:23:17 [ST]

Say something about what you're, in your experience you think has been working quite well, 'cause you've done quite a lot there.

00:23:22 [RP]

Yeah, so I mean, text, like you said, Stephen, the mentoring doesn't scale that well. So text tutorials that are quite accessible, we do have. At least for APL specifically, There's tutorial.Dyalog.com and that's really great. Some people, if you've got programming experience before, you might find it a bit slow. But if you've never programmed before, it's really accessible for building up both the programming concepts and the array concepts together. Then there's course.Dyalog.com, which is more like example problems and sample solutions. Although again, largely focusing on the array techniques, more so than how do you actually put a program together. I know once upon a time, Stephen, you were working on this Dyalog APL cookbook. So I know that's on my list to revive at some point, but I think that was going in a good direction that we need to get back to. And another one that's quite good if you have got programming experience before is Stefan Krueger's book, "Learn APL." So I think that's xpqz.github.io/learnapl. Obviously links to all this will be in the show notes.

00:24:38 [ML]

And he's got a k-book too now, right. I haven't really looked at it. I don't know how detailed it is.

00:24:45 [RP]

Yeah, that's what I was gonna say. But yes, he does also have a k-book that he was working on. And he's got a nice section at the end where he presents this kind of like, okay, I've shown you how the bits of the language work, but how do you do stuff in an array style. And there he goes through a few problems comparing what a sort of conventional scalar loopy programmer might think of and then how do you do that in an array way. So sort of harking back to what Marshall said, a lot of I think the development is at first just have a go, you know, and solve the problem. That's satisfying in and of itself just to get the answer out. But then you will find yourself some days, weeks or months down the line looking back and going What was this. What have I written here. This is nonsense. And then you just, and then you slash it, boil it, boil it down, cut it up. Maybe just start again.

00:25:39 [ML]

Ohh 5 characters.

00:25:43 [RP]

Yeah, exactly. Yeah. That's very much the process that I think tends to work so far. It's just, just getting people in, do what you can and then evolve it from there. Yeah. I do think larger examples would be good. I want to also point out one thing if you're interested in sort of what these different we talked we've talked about programming paradigms about what they are Marshall's BQN docs has got some good pages on I guess specifically the functional programming stuff what is functional programming.

00:26:12 [ML]

Well I've got a list of kind of what paradigms BQN supports. [11]

00:26:16 [RP]

Yeah I think those are quite good descriptions of what they are.

00:26:20 [BT]

I was gonna say one of the things about BQN I know in terms of getting information out about approaching the language are your tutorials, Marshall.

00:26:30 [ML]

The 3 1/2 BQN tutorials.

00:26:32 [BT]

Yes, 3 1/2, But I've I've heard a lot of people say how good they are and.

00:26:36 [ML]

They're pretty long, so I mean, they're not. It's not like there's nothing there, but it doesn't get into multi-dimensional arrays or anything. Now the documentation is very thorough, and so I think you can smoothly move from the tutorials to the documentation, but I would still like to write more tutorials.

00:26:51 [BT]

But I, I, and, and I guess the, I have found the advantage of doing something that in written form, a text form is that a learner can sort of scan and move ahead at their own speed can be a drawback because often they'll skip over things they should be catching, but they, you know, like a video you have to sit and watch. And I found that's the one stumbling block is if you have, you know, do a half hour video and it's not focused at what the learner wants to learn. they may not come back and that's really you know you lose viewers that way.

00:27:27 [RP]

Yeah video is a double-edged sword for sure. Some people it's just sort of infotainment, it's they're not trying to get too deep into something, just want to see something cool. Text is more generally applicable probably.

00:27:39 [ML]

Yeah well and one thing that's very labor-intensive but very useful is having static diagrams of how stuff works. Because you might think an animation is always better than a diagram, but if you start working on this stuff, what you realize is when you're watching an animation, you see one part of it at a time, but if you have a well-designed diagram, everything's there at once. So you can see, you know, you've got everything laid out in front of you. You can look at one part of it at a time, but you can also see how that part relates to the other parts, and you can go back and forth until you get it. Where with an animation, you're stuck just looping it over and over again, and you know, if you want to figure out how 20% of it works, then 80% of your time is wasted. So having well-designed diagrams is a really nice resource, but they are very hard to put together. You have to simplify some things, really get to the core of what you're trying to explain. So those are pretty tough to make.

00:28:32 [BT]

And at this point, that's a brilliant lead into email that we got just prior to this episode from Grant. He starts off, "Hi all, thanks for the awesome podcast. We always love to hear that. My name is Grant. I'm new to Array Languages. I have started from the beginning of your podcast and I'm nine episodes into the 55. So those of you who've been following along, you know what Grant's got in store for himself. In any case, he says, one question I have for you is whether or not anybody has considered languages that are drawn, whether with Unicode art or something like Draw.io. Both are done with with plain text or compatible with get. In both cases, Unicode art or draw.io, as you code, you manually edit the source file and you could see edit the rendering as well, similar to writing markdown. So he's sort of talking about diagramming as coding. So the same challenges I think Marshall was talking about in trying to distill an idea down to a diagram, he's using a diagram to shape his idea. He says, "The reason I thought I'd bring this up "is when I'm learning these languages, or really when I'm parsing anything mathematical, such as a long equation, I always draw it out, like Marshall does in the BQN combinator tutorials. It seems to me that when we're writing the code, we think in charts and compress it into less digestible form, the linear string of characters, and then when we're reading the code, we diagram it to decompress it back to its digestible form. I'm wondering if we could skip back and forth and just write it in the more digestible form. I've seen Scratch and Blockly, [12] but from what I've seen, those are for teaching and aren't done in abstract languages. I don't see why we wouldn't combine the expressivity of these great languages, but do it in digestible format. Warning, opinion coming, it seems to me that sometimes us programmers and mathematicians prefer to make things hard so we can feel like we're doing real work. But really we're just making it harder on ourselves and we'd be more productive if we admitted to having merely human brains and that drawings and diagrams help us. I also think there's a little bit of enjoyment in taking a hard string of characters and solving the puzzle of figuring out what it means, but I think we'd get more mileage if we had that part of the puzzle explicit and got to focus on the puzzle of the problem domain. Anyway, I love you all. Oh, thank you, Grant. That's very kind. And I'm very thankful for you. This podcast is just what I needed, happy array programming. So thank you to Grant for providing that. Any comments on what Grant had to say.

00:31:07 [ML]

Well first I have to disclaim real quick. I did not come up with the idea of the combinator diagrams. These, I think Iverson was the one who started using them, and they're pretty prominent in his paper with Eugene Macdonald [13] where they introduced the function trains, and they've also been championed by Roger Hui over the years. I may have been the first to draw them with the curvy lines, but I think that's about all I did.

00:31:30 [BT]

What do you think about the idea of trying to use the diagramming as a way to understand coding.

00:31:36 [RP]

I think it's controversial to assert that the representation is like the same as the thought. Right. I guess notation as a tool of thought leads into this a little bit, but, um, he's describing the, uh, sort of what digestible form of a diagram. So that's very, but that's a representation of an idea versus a less digestible form of strings of text, which depending on your disposition, whether you like verbose, long, tall, structured loads of text or like dense, squiggly, flat lines of text is already a decision you have to make. Having said this though, coding, making programs of diagrams is not something I'm alien to. I've not really done it, but I know I've seen the idea in a few things, especially visual applications. So game engines, video editing or graphic software, 3D modeling software, even well, like embedded programming software. I think MATLAB Simulink is largely diagram based. So I think the idea, well, clearly it appeals to a lot of people. You can see how it would be really accessible. The other thing when me and Adám were talking about this on a previous APL show podcast [14] that I, that sort of came up with the idea of it helping you to perhaps structure large programs, because, you know, one thing, a code smell that's well known is spaghetti code, right. It's usually when you're doing procedurally things and you keep on doing conditional if statements and for loops to, well, the if statements to code around different cases and then go off to different parts of your code and it ends up all over the place. And the idea, I suppose, with some sort of visual diagram with lines connecting inputs and outputs is you could literally see that develop as your idea becomes a mess. And then hopefully you can mitigate that before too long. I don't really know how well that works in practice, but I can see, you know, you can see the idea there. The other thing that I wonder about this and why I think people have screwed their faces at me when I've tried to bring it up as a topic in the past, the idea of, why don't we do some diagrams or, you know, could we, could you do like coding the sort of high level structure of programmers diagrams. [15] If people imagining like at what level do you make a box. Like I guess a box is a function, is it. Where like the nodes on the edges of the box, so your inputs and outputs marked, you know, somehow or another or like are you gonna have a box for each primitive because that sounds like now just a bit of a waste. It sounds a bit tedious to get the plus box and like hook it up through different nodes to the slash box or something. So yeah deciding the level of abstraction is probably the hardest challenge no matter which tools you're using to try and put those abstractions into into practice.

00:34:42 [BT]

So since we have a practicing Zen Buddhist, are we looking at the moon. Are we looking at a reflection of the moon. I believe Stephen, you're muted, which may be entirely appropriate.

00:34:57 [ST]

I have nothing to say. And I'm saying it that is poetry. Or maybe we're looking at the finger pointing at the Moon. No question that diagrams can help explain things and I put into the chat and they'll be in the show notes some examples from code.kx.com/q where I think the diagrams really help but that's not what's proposed. What's proposed here is that the diagrams could do the heavy lifting. Basically we could do our work in diagrams. We start by by thinking that way and we wind up with those. And my real life experience with this hasn't been very happy. In the 1980s, I got enthused about the early software engineering movement, swallowed the Kool-Aid, started drawing functional decomposition diagrams. And the Kool-Aid that I was drinking led me to think that if I got the diagrams right, If I could sort out my program design at that level and show what gets passed backwards and forwards, then the code would kind of fall out. It would be kind of obvious at each point what to write. I think I got fired for that.

00:36:20 [BT]

Did the diagrams accomplish what they set out to do.

00:36:25 [ST]

I found myself endlessly redrawing diagrams. What do they call it. Analysis paralysis these days. Nothing like a rhyme to make something sound true. But I definitely experienced it. I think a few years ago, I came across a video of Tom DeMarco kind of apologizing for that work.

00:36:49 [BT]

Is the challenge there that the diagrams are too flexible. I wonder that myself. when I have constraints, I find I can, it's easier for me to work with those things if I'm just, you know, blank canvas.

00:37:03 [ML]

That's the whole idea of everything is an array. It's a constraint. It says you don't have to make the choice. It's an array.

00:37:10 [BT]

That's a good point. Yeah. And it also fits to, quite often a person's got any math training in the background. It fits to that style as well so that you've got constraints, but you're also constrained in an area that you might have some experience in.

00:37:24 [ST]

So do I start from a diagram now. That's Grant's premise. And I've got no doubt that he does, but I don't. I start by writing some, you know, at the top level, some high level coding. I want a program called this that's going to take the results from this expression and and it's going to pass it on something like this. And I'll work with these kind of placeholder programs and then start writing some of those and finding out, oh, that doesn't really quite work that way. I need to reorganize. But basically, no, I'd start and finish in a text editor.

00:38:03 [ML]

I'll say, I mean, I visualize a lot, but what I visualize is not the structure of my program. What I visualize is the arrays that it's dealing with, which goes back to the idea of you want to figure out your data representation, and then the code can often fall out. So, and of course, I mean, you can't just write the arrays and then have that turn into a program. That's pretty obvious. So I'm not saying the diagram thing won't work, but that's not how I think of my code either.

00:38:32 [ST]

I think I learned that most thoroughly from Morton, Morton Kromberg, at Dyalog. [16] I've heard him say it several times. times, you got to immerse yourself in the data, go eyeballing it. So what I understood by that was, you need to actually go and look at the stuff and, oh, no, that's got some nulls in, right. Are there some negative figures in that. Right. Some of those go out of range, right. And that leads me to what you were just talking about, Marshall, how am I going to represent this stuff.

00:39:07 [BT]

As an alternative, I tend to use what I've heard termed as bricolage, which is building, like constructionism. You're constructing parts. So you start with something and you take the data and you fire it through some kind of a primitive or a function. You see what comes out the other end, you go, "Oh, okay, I'll have to change that. " And you just sort of build it on. The interesting thing I find about working with J is to some extent you start constructing. And then the magical part for me is then you start removing things. And it's when, when you start removing things that you actually get for, if you take this approach, when you start removing things, you actually get much closer to what feels like a much more elegant solution. You're not doing anything that doesn't need to be done. And, and when you reach the end point of that, you've got something that feels really clean and smooth, and it does exactly what it's supposed to. But the process of getting there is incredibly messy. And I'm not going to say it follows straight lines, but for whatever reason, my mind tends to work that way when I'm trying to approach a problem.

00:40:10 [ST]

That is exciting to work on, isn't it. Very satisfying. And I've noticed that I often react that way when looking at QB code. The immediate reaction is like, "Oh, surely we don't need to do all that. " And start looking for, "Oh, we can take this out. We don't need to do that. " and so forth.

00:40:33 [BT]

Yeah I guess it to me it almost feels like when I'm going up that hill and adding things I'm the writer and when I'm coming down the other side and eliminating things I'm the editor and if I've got writing to work with editing is actually really a lot of fun because you're you've already got the concepts you're just trying to move them around. The writing for me is the hard part except that when you're just sort of slapping things at it it's not that hard to do and one of the things and we haven't mentioned it but with all these array languages they're interpreted which means that you don't have to wait for a program to be compiled before you find out what it's going to do. You're entering things it's responding to you with the answer and to be quite honest when I when I went to producing video as opposed to making films the big reason I did that was because as soon as you shoot video you know what you have and And that lack of patience, I have a lot of patience in other parts of my life, but when I'm creating things, I want to see what's happening as soon as I've done them. And that's certain, certainly followed me into the array languages for sure. And that interpretation, not having to wait to compile and get an answer back to me is, it makes all the difference when I'm working with them. [17]

00:41:50 [RP]

Controversial, Marshall.

00:41:51 [ML]

Well, yeah, I think a little bit differently from this because I make music and the way I think about it is that once I record things, then it feels very difficult to work with all of a sudden because it is all fixed. Part of this is just mistakes in the recording, but one thing is it's very hard to choose between two versions of something that are both bad in different ways.

00:42:20 [RP]

That's how I feel coding sometimes.

00:42:22 [ML]

Yeah, that definitely does happen too.

00:42:24 [BT]

Yeah, I think that's a really. . . and for me, myself, and my background, when I made the transition from making video to making live television, I know exactly what you're talking about. There's a huge amount of freedom that comes as soon as you go live.

00:42:40 [ML]

Yeah, because you can't. . . you're not going to take a different version.

00:42:43 [RP]

It just changes where the work goes, doesn't it. Because it's all. . . if you're doing a live thing or a recording, It's all in the preparation and the rehearsal. It's in the doing the planning beforehand and the practice so that you know when you hit go, it's likely to be pretty good. Whereas with video, it's quite fun to just.

00:43:03 [ML]

Yeah, well, and that's one thing I feel. I feel like it is easier to put the work in ahead of time rather than record and then try to sort things out.

00:43:12 [RP]

Sometimes a bit of both. And sometimes you get happy mistakes.

00:43:17 [ML]

Yeah, and of course I haven't moved all my work ahead of time. I still work with things that I've recorded and you know, tweak stuff and all that.

00:43:26 [RP]

This just reminded me of another challenge I think for, not just anyone learning array languages, but myself personally sometimes. Is this thing where you can have, you know, lots of similar, similarly valid approaches to write coding something. And a lot of array languages are quite agnostic in how you could use them. And I think a lot of beginners find that a massive challenge where one of the like the Zen of Python things is, you know, there should ideally be one way to do any given thing. So then if you search, you know, how to do X in Python, you know, there's one way to do it. So you just give them the answer. But when someone comes and asks, how do I do X, Y, Z in an array language. You have to go, well, you know, depends on, you know, what came before. what came before and what's coming after. And give me all the context and I can give you and I can give you a satisfactory answer.

00:44:24 [ML]

One thing I'll do a lot is that if I'm writing a line of code and I think of a way to do it that's pretty different, I'll copy the line and I'll write it a new way and I'll try to write it as well as I can. And then I'll look at these two lines and say, well, which one looks better to me. Which one's clearer. And then keep the one I like better.

00:44:42 [RP]

It sounds horrific. I think if you're listening to this and you come from programming where it does take a while, you know, to write all this, I mean, maybe you've got auto-complete, the IntelliSense in your IDE is probably pretty good these days. But at the same time, it's quite laborious and you're going to be really disheartened to think, "I've got to write this twice."

00:45:04 [ML]

Yeah, no, I absolutely spend as much time choosing between them as writing both.

00:45:09 [RP]

Yeah, you can lay down your several ideas for solving the same thing in not too much time. So like you said, Marshall, yeah, you'll spend most of that time comparing them and...

00:45:23 [ST]

And you get insight into what's going on that you'd otherwise miss. And if you're coming not from a Python or a Scala language, but from poetry, this will seem a perfectly natural thing to do to try three or four versions of a line and see which one works and feels best. In fact, this is something which I've done in classes, which is one way to begin scaling mentorship, is to run online classes. And where I've done this, it's been a lot of fun and exactly what Marshall's describing of trying different ways to solve the same problem, comparing them and and thinking about the different computational qualities and just which one looks or feels most satisfying or most clearly expresses to us what's going on. That's a large part of what we do 'cause it's work in digestible chunks. And I found that the format that Ken Iverson devised for his working introduction to APL all those decades ago works really well online. So briefly, yeah, when I learned I was going to be an instructor of the APL course, I was really excited because I was going to be the guy who stood up front and told people how everything works. I thought this was a great role for me in life. And one of the most important things I ever learned from Ken Iverson was don't do that. Ken was a great teacher. And from him, I learned that people just learn naturally, people will learn at their own speeds. And my job as a teacher is not to pontificate about what the answers are the right way to do it, but to help them learn. And those two main jobs of the instructor is rather like on a car with automatic transmission, you've got two pedals, one to go faster, one to go slower, as you've got to help people when they get stuck, dig them out of holes. And if they're ripping through the material too fast, toss them some curveballs and ask them some questions to really make them think. That's basically what an instructor's job is. So in the work introduction to APL, the instructor gives a little bit of an introduction, introduces a problem, and then people go away and work on pairs. In pairs on those problems. And this was long before pair programming was a term or a thing. And the great advantage of working in pairs is it's very rare for both people in a pair to get stuck. One person's always got some idea of where to go next. You want to get people who are, you want to get people to sort themselves out so they've got some similar levels of ability. Sorry, I'm getting distracted here. Okay, so you'll need to trim this bit out. We might have had a fox attack in the garden.

00:48:32 [BT]

Oh oh! A fox in the garden. Nothing worse than that when you have chickens.

00:48:36 [ST]

Mickey's dealing with it.

00:48:37 [BT]

One of the things you were talking about though, Stephen, I think carried on into J with the J labs, because although you're not pair programming in a J lab, that's exactly the technique that Ken uses (or has used and a number of other people use as well when he showed it), was that you have a few lines of instruction and then your job is to go in and use the actual interaction environment (the programming environment), to play around with it, and then to find things out about it and then you come to the next step and you find a little bit more instruction. And people think: well that's Jupyter Labs isn't it. [18] Yeah it is. J did this in the the early 90s, long before Jupyter Labs was around [chuckles]. And that's the sort of thing that I don't know that Ken always gets enough recognition for. He was a brilliant computer scientist, very good mathematician, but he was an excellent teacher too. And I think that's overlooked by people who are used to being computer scientists. They don't realize how good he was at another whole area of skill.

00:49:44 [ST]

Anyway, I figured out, I think mostly, how to do these online classes (it's a little bit of scaling). If you'd like to join one of those for q and do some practice and try some different expressions out and variations and then discuss them, just write to me (sjt@5jt.com) and we'll get you in the next class or one after. [19]

00:50:09 [BT]

Brilliantly brought in because I think at this point, I'll talk a little bit about J. If you're interested in doing anything with J, we have the J playground. It's online and it's in your browser, so essentially, you don't have to download anything. And I'm sure Rich will talk about tryapl.org too, which was the precursor to it. But for J it's the J playground. There are actually labs there that you can go through and they will teach you things and some of them are quite sophisticated. I think there's one on sort of [a] machine-learning kind of thing. It gets pretty sophisticated pretty fast, but there are some elementary labs too, if you're interested in picking it up. So that's J playground and there will be notes in the show notes and I'll toss it over to Rich to talk about what APL might have.

00:50:58 [RP]

Well, I guess, yeah, I've been thrown in here. So Bob mentioned TryAPL [tryapl.org], which is place you can go and try APL obviously, without having to install anything at all. You can just try the language out. There are some tutorials on there. I think we currently have a bit of a mind to do a rework of a lot of that stuff. There is an entire repository of Dyalog Jupyter notebooks, which are the source of those tutorials. There's actually more stuff over in that GitHub repository. And then we're thinking of having a little rejig of how that works. But something a bit more similar to what Stephen was talking about, or maybe a bit more lightweight is once a week in the APL Orchard (which you can go to from APL.chat) is the weekly APL Quest. I think there's an APL Wiki page about it as well, where you can find details on how to join. But basically it's just the phase one problems from past APL problem solving competitions. They can all be solved with a single line of APL and you know in advance which one is coming up, progressing linearly through them. Most of the time it's Adám there, being the instructor as it were. Sometimes I'm there. And it's exactly that, where during the week you have a go at solving the problems and then come on a Friday (at a time which changes throughout the year; I believe it'like 4pm UK right now, so that's like 3 o'clock UTC in the afternoon, but for reasons of being internationally appealing, the time changes throughout the year). And you come and sort of offer: what have you done, what was your approach, what are the solutions. And we see if we can think of lots of different ways of solving the problem or perhaps clean up some code if you're worried about it being a bit verbose or anything like that. Or you just want to try a new technique or learn a new primitive. That's every week. And then on top of that, Adám every week diligently produces a video as a result of those discussions where he basically boils down the main gist of the different approaches and then publishes that to Adám's APL YouTube channel. I think if you're just getting started and you're sort of dipping your toes in learning how the primitives work and how the things [are] put together and you want to actually try some programming exercises with a bit of guidance, then I think that's a great resource to come down to.

00:53:34 [ML]

Yeah, it is really interesting how [across] all the different languages, there's some things in common, but some huge differences about how they approach learning the language. So now BQN. It's really interesting to think how that affects who gets into the language and what the culture around the language becomes just based on how the people involved with it at the start chose to introduce it. So BQN you can try online via the page on the official BQN website ([it] is even called try.html). And there's another REPL, which is a very different format that's called BQNPAD (or "bacon pad" or however you pronounce it). It doesn't have these kinds of interactive tutorials. So it's a different approach to learning the language. And I think a lot of people have been successful with that. One thing I've noticed is that a lot of people who come to BQN are people who've, a few years ago tried J or APL and it didn't really click with them. They decided they just didn't feel connected to the language and didn't continue using it. So maybe it's a good thing to try, if you felt like the approach to learning those languages didn't work well. I've also found that people are having pretty good success moving [from] BQN to other languages, particularly K, so you shouldn't be worried about getting stuck with BQN. It seems J is pretty sticky. I don't know anybody who's really deep into J who's gone on to another language, and I have no idea why that is, but it's an interesting thing as well.

00:55:11 [BT]

Possibly we lack imagination! [laughs]

00:55:14 [ML]

Could be.

00:55:15 [RP]

They never installed the keyboard. Oh no, K has that too!

00:55:18 [ML]

I know that was an issue for me when I was a J programmer. The bigger issue was that it was harder to get a Dyalog license then, [20] so I just didn't bother to go through with that. But yeah, also there were things that I could read about APL, and I just thought, I don't want to pick up this character set. So that is one issue. But the character set is really not as big of a problem as you think it is for these languages that use one.

00:55:45 [BT]

Especially the online versions because essentially you can basically just click on it with your mouse and select things that way.

00:55:54 [ML]

Yeah, and you can hover over the key and see the keyboard shortcut that you would use. So pretty quickly you learn at least all the parts of the keyboard that you're using.

00:56:04 [BT]

And as we mentioned before, these languages are very terse, so it's not like you're having to type out a full word.

00:56:11 [ML]

Yeah, you're guaranteed to be doing less typing with the language, regardless of how you put in the characters. I still put it in with the backslash escape, which is slower than using a modifier character, but I don't care because I'm doing a third of the typing [chuckles] of someone who's using any other language. But yeah, in terms of the ways you can learn BQN: there's not these kind of interactive tutorials. One thing a lot of people have had success with is starting on the Advent of Code problems. There are a bunch of people who try out Advent of Code in BQN each year. [21] And there are also people who've started not at Christmas and have gone back to older versions of that and started learning BQN that way. Or of course, if you've got a problem that you want to work on independently, I think that's really great because it keeps you motivated because you want to solve this problem. But yeah, so the BQN stuff for learning is pretty text oriented: I've got my tutorials. There are a few blogs that cover some BQN topics that are linked to on the community page. That's a good way to see how other people use BQN. The other thing I should mention that it shares with APL (that it stole from APL!), is in APL, it's called APLcart, and in BQN, it's BQNcrate. [22] It's just a huge list of little pieces of code along with what you would do with them. And I think we've talked about this a few times here, but this is the place to bring it up again. What you do is you search for what you want to do, and then you'll see a bunch of snippets, and hopefully one of them does cover what you want to do. There's actually a pretty good chance because there are only so many common things, and having so much code, a pretty large fraction of them are covered. That's a cool resource for learning good ways of doing things where it's not immediately obvious from the primitives.

00:58:07 [BT]

And in the case of J, it's not as imaginatively named, but we have the J phrases. They're not as searchable (I think we're looking at doing that in the future), but certainly there are snippets of code that will do things that you might not think about. In fact, I've heard people say that it's very profitable if you're learning the language to maybe spend 15 minutes or half an hour a day just looking at the phrases and playing around with them because they're already short snippets of code that you might not have thought of those combinations and that might trigger things happening in your mind to solve things.

00:58:43 [BT]

Does anybody have anything else that they would like to add before we wrap all this up?

00:58:48 [ML]

Oh yeah, I did have a point on the diagrams. We seem kind of negative overall on writing the diagrams and writing code that way. But one thing that has been very successful is (the combinator diagrams are one piece of this) but representing code that's already written with a diagram. So I think a few languages have ways to take your code and convert it in different ways. Like in APL, there's the tree view. The BQN REPL on the website has this "explain" button; you'll click explain and then when you run your expression, it also includes a tree diagram of the evaluation order of everything. Another language that I think has done some work on this is KAP which is a newer array language. [23] It had some nice features for diagramming stuff that I can't remember exactly right now but they seem pretty cool. A lot of different languages have used that and it seems to help a lot of people.

00:59:46 [BT]

So in that case, I guess it's an automated form of documentation. You hit a button and it actually presents the program in a different form. So when Grant was talking about writing a diagram to create the code and then having somebody else to try and decode it (creating a diagram), it's that second step that this takes care of.

01:00:06 [ML]

So yeah, if you think in diagrams it helps you to read code. It doesn't help you to write code.

01:00:11 [RP]

That's maybe the thing that struck me. You can totally see the thinking process or the idea here that: okay, if I'm drawing diagrams to figure out how I should start writing code, and then I'm writing some code, and then once I've got the code all nice, I'm drawing diagrams to explain the code, then skip the middleman: it makes a lot of sense. But I think it's just glossing over that messy middle bit that turns out to probably be quite necessary. Or maybe it's more just we think it's too hard to write the diagram right first time [chuckles].

01:00:46 [ML]

Yeah, well, I mean that goes to there being too many options to do things. When you write the diagram, is the diagram that you wrote the one that you actually wanted to write? And maybe actually going through linear code is the best way to get it to be the diagram that you wanted to write, that's most clear.

01:01:03 [BT]

Well, and if you think about it, if consistently linear code produced some type of a diagram to show what you'd come up with, it's quite possible that as you do more linear code you'd start to develop those images within your mind of what the diagrams might be and it might make it easier to do your linear code. So you actually might take the first step and internalize it. You might not write the diagram but you might always be thinking in diagrams before you write the linear code [long pause].

01:01:44 [BT]

[laughs] And I'll just drop the mic now! If you'd like to get in touch with us, you can contact us at contact@arraycast.com. [24] We'd love to hear from you. Today we had a chance to feature an excellent email from Grant and I'm not just saying that because he told us how much he loved us (but we always like to hear that). We also do like to hear people who have critical takes on us. We're not expecting to be perfect. That's not the nature of programming, as you probably know, in your day to day lives.

01:02:15 [ML]

Programming code or audio programming [chuckles].

01:02:17 [BT]

Or audio programming [laughs]. That's right. Yeah. In any case, I think that's a wrap up of the show. I'd like to thank my co-panelists, Stephen Taylor, Rich Park and Marshall Lochbaum. And, with that, I will say happy array programming.

[everyone else together]

Happy Array Programming!

[music]