Transcript
Transcript prepared by
Adám Brudzewsky, Bob Therriault, and Sanjay Cherian
00:00:00 [Adám Brudzewsky]
So imagine you had that big matrix that I spoke about, first row, the numbers, second row, all fizzes, third row, all buzzes, and the fizz buzzes in the last row. We could rotate every column by an amount, 0, 1, 2, or 3 steps. And once we have done that rotation, we just take the first row, discard the last three, and now we've got the answers to all the fizz buzz values. So there are many ways.
00:00:23 [Bob Therriault]
Reminds me of a bike lock.
00:00:23 [AB]
Yeah.
00:00:25 [MUSIC]
00:00:35 [Conor Hoekstra]
Welcome to episode 99 of ArrayCast. I'm your host, Conor. And today with us, I have our four panelists. We're going to go around and do brief introductions. We'll start with Bob, then go to Stephen, then go to Marshall, and finish with Adám.
00:00:47 [BT]
Episode 99, who would have believed it? And I am a J enthusiast.
00:00:51 [Stephen Taylor]
I'm Stephen Taylor, APL and q.
00:00:54 [Marshall Lochbaum]
I'm Marshall Lochbaum. I started out with J. I worked at Dyalog for a while, then I made BQN and also Singeli.
00:01:00 [AB]
I'm Adám Brudzewsky. I've been doing APL all my life. I work professionally with APL at Dyalog.
00:01:06 [CH]
And as mentioned before, my name is Conor. I am an Array Language enthusiast of all the Array Languages. We love them all. And with that, I think we have four announcements. Three of them are from Adám, and then we will wrap up with a short announcement from me. Over to Adám.
00:01:21 [AB]
Okay, so the APL Challenge Round 2025 has just begun. [01] It is geared towards absolute newcomers to APL or Array Programming. So the challenge contains everything you need to complete it. You don't need to go and learn anything separately. It can be a nice entry point into Array Programming. So we'll have a link to that in the show notes. And then there was a recent Functional Conf 2025, and Aaron Hsu spoke on designing your data. We'll have a link to that presentation as well. And finally, there's now been announced the DyNA or Dyalog North America Meetup. And there are going to be two of them after each other. The first one is on April 7th. This is in New York City. And there'll also be one on a bigger one actually on in September. But we'll come back to that when we get there. This meetup is a live meetup in New York City. And it's going to be especially focusing on migrations from other APL flavors to Dyalog and on getting applications running on modern ways on the web and so on.
00:02:44 [CH]
Awesome. So all three of those things, especially Aaron's video, can recommend all of Aaron's videos. They're great. Not necessarily do I agree with everything he always says, but he is amazing at building convincing arguments. And with that, I think my last announcement, it's a short one. I have a rival podcast, TacitTalk. We are up to 18 episodes. At this rate, we are going to pass ArrayCast at some point. But that's because we've got AI and tech on our side, folks. 75% of the podcasts aren't actual live interviews. It started off with AI conversations, which a lot of people hate, brought to you by Notebook LM, which is a Google product backed by Gemini, I think 1.5. But I was looking into other technologies because I had read, I believe it was a personal view of APL, which is the 1991 paper by Iverson. And I had done the audio summary, but then I went and read it after the fact. And I realized that like the AI audio summary wasn't very good. It missed out a couple of really, I thought, insightful things at one point. We'll maybe talk about it in a future episode. It was that Tacit programming had some of the advantages of compiled programs, which I didn't understand. But anyways, past guests are mentioned in the paper. And I was like, wow, that AI summary is terrible. It's terrible, folks. I mean, it sounds convincing. But it missed out some of the most important, like insightful details, which makes sense, right? It's not going to mention all the details in some AI summary. So I went on a search for if we can get robots to make up a conversation, we should be able to get robots to just read it. Right? Long story long, I found an app called Speechify, which is a text to speech app. If any of you have listened to the text to speech readings of some of these papers, we're batting below 100. We're batting below 100 because sometimes, especially on tables, like for the first speech to text or text to speech, Speechify 1, I basically redid the previous book, which was the Michael Jenkins Array Theory and the Design of Meow. It reads out the tables or matrices of numbers as like plus minus plus minus one, one or two, one, two, three, four, or five, six, seven, which is not great. I'm sure some people are like, this is worse than the AI summary. But I was trying to explain to a couple of friends over the weekend that it's even if you can get 90% of the understanding out of this text, like a lot of the times I don't have time to sit down to read all the textbooks in the world. I would love to listen to 80% or 90% of a textbook. You have to put your ears into like filtering mode of like when it messes up on stuff. But yeah, so there's now three episodes out. 16 was the Array Theory and the Design of Meow. 17 was the Evolution of APL. Here's a quick question. I'm curious, how many folks have read end to end the Evolution of APL, which it's on J software as a standalone paper, but it was first published in the HOPL, the first edition of the HOPL conference, History of Programming Languages. I had not. So that's zero for one. Have the rest of you, I mean, I wouldn't be surprised if Marshall and Adám because you guys are like the history buffs.
00:05:47 [ML]
I think I went through it. I was, I think I was kind of like going back and forth and doing APL wiki stuff. So it's, I did not really read it in an entirely linear way. But I think I got through it all.
00:06:03 [CH]
It counts. One for two, one for two. And the lower the number, the more vindicated I am. So don't be, don't be afraid to say I haven't read it. I'm hoping this is not four out of five because it just makes me look bad. If it's two out of five. So yeah, Bob, Stephen, Adám, what about you guys?
00:06:19 [ST]
I do think I read the whole thing. But like Marshall, not in one, not in one swoop. I just really wanted chewing.
00:06:27 [CH]
So two out of three, Bob?
00:06:29 [BT]
I have looked at it. I don't think I've read all of it. I have the second one with Morten and Roger. I know I've watched all of that.
00:06:37 [CH]
And then Adám, it is looking bad for me, folks.
00:06:39 [AB]
I've studied it in depth. We actually had an internal book club in Dyalog where we went through all these old papers. And I can tell you the exact week that I was studying this paper as well.
00:06:54 [BT]
So AI stands for actual intelligence.
00:06:59 [CH]
Well, so here's the, here's the, uh, the follow up question. Does anybody know what I'm not? It's probably not in the HOPL publication, but it is on the J software publishing of it under the Creative Commons. Does anybody know what like the last half of that article is? Because like technically the, the article, the evolution of APL ends at the halfway mark. And then there's like an appendix with like a whole other half of something. My guess is no one's going to remember what it is, but I could, I was wrong.
00:07:28 [AB]
It's a conversation. It's from a conference or something that they, that they discussed.
00:07:33 [ML]
Yeah, I thought they had the questions at the conference.
00:07:34 [CH]
All right. So two out of five, maybe Stephen and Bob also. But that was the crazy thing is that like, you know, I have not gotten around to reading this stuff. And then as soon as I'm listening to it, I get to the halfway mark and then I'm now hearing someone read out Ken Iverson's dialogue with basically it's a q and A at the end. And there's some like, it reminded me of listening to the Ken Iverson audio podcast that we released because it was some of the answers were very terse. And it's like, ah, I don't understand the significance of this question next. It's anyways, it's amazing. And or at least I find it amazing. So it can turn these things. And that's the thing is, so I plan to do the HOPL 4 paper APL since 1978. That gets turned in to a four hour podcast that things like 150 pages. I'm probably going to edit it down because I think it captures by default like the appendices and whatnot. But the point being is, I guess it depends if you prefer to watch versus read versus listen. Due to the fact that I run a lot. I just have so many more hours of listening than I do of reading. And it's not perfect. But that's the thing. It's the worst it's going to be right now. So I imagine in five years, especially with the advent of LLMs and stuff, it'll actually be able it'll see, oh, here's a matrix. Let me just describe this matrix, not read it out character by character, but say or even like diagrams and charts and stuff like it will have, in my opinion, the ability to do like human level reading of anything, you know, audiobook or books, I guess audiobooks is the translation, but I'm pretty excited about it. This announcement has gone on way longer than I planned. But like I said, if you do go listen to it, grain of salt, it's not perfect, but it is a way to consume stuff that folks might not. There's a lot of idle time doing dishes, hard to read a book. Listening to audio though, definitely you can do it. Some people might say, get a dishwasher. I have a dishwasher, but I think I use less water. Anyways, announcement over, links in the show notes. That brings us to today's topic, episode 99. We're one away from 100, and I believe if I read the email and remember the chats correctly, we are going to be discussing and comparing selecting versus indexing versus any other kind of superficial way of doing that kind of stuff across the array languages. So I'm not sure how we want to do this. Do we want to throw it to one person to either give a summary of all or maybe one of the array? I mean, I assume Marshall's going to speak for BQN and Adám will speak for APL, et cetera, et cetera. How do we want to start off this conversation? Maybe a historical perspective would be a place to start. All right, who wants to take that?
00:10:18 [ML]
Well, I mean, I guess I can probably start it off by saying there is an interesting kind of historical note from the book, A Programming Language, [02] which did not really describe APL. There, the selection was done more like in mathematics with subscripts for arrays. And actually there was, I don't remember the exact scheme, but I think one axis used a subscript and another axis used a superscript. And I'm not sure if it made it into the book, but there were kind of thoughts that you'd put a subscript on the left and a superscript on the left and get up to four axes that way. And so then when, I don't know if this was Ken's idea, maybe it was someone else's, Phil Abrams or Adin Falkoff or somebody, but eventually they decided in order to make this a programming language on a computer where you just type in text, we have to use the square brackets. And so then it got this selection that looks a lot more like C, I think ALGOL uses the square brackets too, I don't remember, but would look more like programming languages at the time. But it is semantically very different. So Adám, do you want to describe how APL selection works?
00:11:37 [AB]
I can try at least. In the very simple case where we have a vector, and for simplicity, let's just give it a name X for now. And then we can put square bracket immediately, immediate right of that. If X is a vector, then superficially things can look a lot like you say, like other programming languages. You put a single number into the square bracket and you get out the corresponding element. That's an interesting question of index origin. We can come back to that, but it's kind of orthogonal to this whole question.
00:12:17 [ML]
Or maybe a non-interesting question or a debated to death question.
00:12:21 [AB]
There is another interesting question as to what happens if this is not a so-called simple vector. And we can come back to that as well. It's another subject. Somebody take note here of all the things we have to come back to. But then where APL really shines, of course, is working on whole arrays. And I believe it at least was unique at the time that APL could take multiple indices. You could have, say, a vector of indices and it would then compose a result consisting of the corresponding elements from X. And there's even a principle that the result of this indexing operation has the same shape as the shape of the indices. So even though we're indexing into a vector X, we could use, say, a matrix of indices, say a two by two matrix of the first row is one, two, and the second row is three, four. We're doing one based indexing for now. And that would give us a result where the first, that would be a matrix. In the top left corner, we'd be the first element of X and the bottom right corner of the result would be the fourth element.
00:13:41 [ML]
Yeah, I almost think it's kind of helpful to describe that backwards as like a substitution because sometimes it's useful to think about that. So maybe one example is if you had a Boolean array and you wanted to write a space for every zero and a star for every one, the standard way to do that in APL is to use the string space star and index by the Boolean array. There's still an uninteresting question of index origin. So the way you can describe that is maybe you're taking your index array and you're replacing every zero with a space and every one with a star. And it's the same if you have like a four element array, you replace every zero with something and every one with something else and every two with something else. So it's like a substitution in the index array.
00:14:24 [AB]
This is just the beginning.
00:14:26 [ST]
The way I've described that in that corresponding form in q is to say that the indexing is right atomic, which is to say that for every atom in the index expression, you get an atom, you get an element in the result.
00:14:43 [AB]
This is the beginning of it because we only spoke about X being a vector. Things get really interesting when X has multiple dimensions. There and this is all special syntax, which bothers some people, me included. Here we now can insert semicolons into the square brackets and that splits the square brackets, so to say, into sections. I don't think they really have official names and they're completely cut off. The semicolon is not a primitive of any sort. It's just syntax. So now we have open square bracket, section, semicolon, section, semicolon, section, close bracket, for example. And each of these sections can either be empty or have indices. And if it has indices, then again, it could be indices arranged in any rank. It could be a scalar, a vector, anything. So let's again take just a simple case where X is a matrix and that means we have square bracket and then a section, semicolon, section, close square bracket. Now in a, again, simple case, because this gets increasingly more complex, and if we say we have a vector of indices in both sections, let's say in the first we have 1 and 2 and the second section we have 3 and 4. This will give us a subarray, a subset of the elements from X. All of those that are in rows 1 and 2 and are also in columns 3 and 4. So it's the intersection of these indices. We are selecting a subset of the rows and a subset of the columns. If we leave a section empty, then that means all along that dimension. So it would be the same thing as writing 1, 2, 3, 4, 5 up to the length of that dimension. And finally, just to the beginning of all of this, if we have multidimensional set of indices, then it's basically-- it has to match as well with the result shape is the concatenation of the shapes of the indices given.
00:17:15 [ML]
Well, it's an outer product of all the indices after you take the empty ones and expand those to a full index vector.
00:17:22 [AB]
Yeah. Because you have this rule in outer product as well, that the result of an outer product has the shape of a concatenation of the shapes of the arguments to the outer product.
00:17:32 [BT]
So if you use, say for instance, you've got this multidimensional matrix, and instead of using the link or the semicolon between them, you just list the numbers, would it then do the rows of that matrix? Is that how it does it?
00:17:48 [AB]
No. Not in the original. And for now, I've only spoken about what APL came up with originally when it was linearized from Iverson notation. There's a whole lot more going on later. But this is just the simplest case that APL had from day one. It's already plenty powerful compared to, I think, all other languages at the time that used some kind of bracket indexing could only take a single element from something. Right. I think we covered this. This means, by extension, because of this whole thing we mentioned about the outer product, means that if you put in a scalar, then you decrease the rank of the result compared to the original and compared to x. So let's say we have a matrix and we do square bracket 2 semicolon 3. So both of these index parts have an empty shape. That means the result is going to have the concatenation of empty and empty that's also empty. That means we get a scalar out. So you can select a single element from a matrix. And if one of them is a scalar, the other one is a vector, then we're getting a vector out. Both of them are vectors. We get a matrix out, like we spoke about before. So now we are up to 1966 in the evolution of indexing in APL. And from there, there are a couple of ways we can go, because the next interesting thing that happened in the APL evolution was addition of nested or boxed arrays. So arrays of arrays, not just arrays of simple scalars. And here things got really interesting in multiple ways. One of those ways was that since an array is not just simple scalars, and since we have a rule that the result has to have the same shape as the indexing indices or the other part of those, then we have to deal with some enclosures. So let's say we have a vector of character vectors, which informally could be called a vector of strings. That's something you very often have. And you want to choose which string are we going to return, yes or no, or something like that, based on the Boolean. It could be. And so now we just choose a single one. So we're using a scalar. So let's say we have a vector of two elements. The first one is yes, and the second one is no. And we either index it with a 1 or with a 2. We just choose a 1. So now we think we would get out the vector yes. But we don't, because yes is a three-element vector. And we know that the indexer here was a scalar. So the result also has to be a scalar. Which means the result has to be enclosed. It has to be the enclosure, the scalarification, if that's a word, of the vector yes.
00:20:53 [BT]
In J, that's boxing.
00:20:55 [AB]
Yeah. You get a box out. If you have multiple indexes, people are not so bothered by it. If you write 1, 2, 2, 1, 2, like that, and you get a vector of vectors, not a problem. But people kind of expect the indexing operation to disclose. So there's one interesting issue there. And we can come back to things that were done about that. Another issue was, because arrays can be nested, you might want to reach into the array. You might want to do some scatter point indexing, where you want just this element and just that element, and not the intersection of all these rows with all these columns. And so the square brackets were extended to allow you to omit the semicolons. And we can detect a different mode of operation for the square brackets by the indexes being nested. And it just comes out nicely that, in the simple case, there's no difference between the normal indexing and the scatter point indexing. So there, if you have a nested content of the square bracket, then that specifies an individual-- those can be the full indices of every element that you want. So you can choose this element, that element. And then again, the rule applies that the result has the same shape as the indexer. So you can have an array of scatter points, an array of indices that goes straight into the array to choose from that. And then by extension to that, every one of those indices can itself be nested a further level. And in that case, it is a path, has to be a vector. It specifies a path into the array. So you might say, I want to step into element 2, 3, so row 2, column 3. And in that element, I want to step into the fourth element. And in that element, I want to step into the second layer, seventh row, first column. And in that, I want to just step into its element, because it doesn't have any dimension. It's just an enclosed thing. So that's called reach indexing. So we have these three forms of indexing, the normal indexing, and the scatter indexing, and the reach indexing. And really, scatter and reach is the same. It's just a question of how many levels do you go in. And when it comes to just a normal vector, then it all comes out to the same thing. That's a blue vector. You can't see the difference which one you're doing. And then there are even differences. Something Dyalog doesn't do, but I think NARS 2000 does, is it allows you to have an uneven path length, shall we say, on the reach indexing. So Dyalog requires you to either reach all the way in, and every index reaches the same level so you can assemble the array. And NARS 2000 allows every index to have a different reach, a different number of levels it steps in. And somehow, it manages to assemble the result back into something that makes sense. I'm not really sure how that works. At the same time, as nested arrays came onto the scene, operators were also generalized. And that meant you can now use a user-defined function or a mixed, non-scalar function with operators. And this led to the problem that you can't apply an operator with a square bracket. A square bracket essentially works like a function. It takes two arguments. The indexing, there's a little bit of an interesting thing with these sections that are semicolon separated, but that can be addressed in various ways. And then there's the actual data array. You can't apply this with an operator. You can't do an each on that. There's no symbols for the each at. So some work was done to create a function equivalent of the square bracket, which is complicated to come up with, really, because of those sections. You could use the nesting. So you have to just add another level of nesting. That was done. And then you have an index primitive, which the symbol for that is very much related to the square bracket. Basically, on a typewriter terminal, or any typewriter where you can type letters on top of each other, if you type the left square bracket and then in the same position you type the right square bracket, you get this very elongated rectangle, very tall and narrow. So that looks superficially similar to the APL symbol quad, which is just a rectangle with a golden ratio of dimensions. But it's been squished together. So it's called squish quad or squad. So it's often called SQUADindexing. But it's really squish quad. And there was the problem that how do we deal with these sections? How do we deal with sections that can be empty? Came up with the idea that we just add another enclosure level, one for each section. In J this was added with a different symbol, where you can-- originally the idea was that you can have just an extra enclosed list, extra level of enclosure, and an empty list. And that would mean it's an empty section. But then somebody got the great idea that, oh, if empty section means all of them, when it's an extra level of nesting, then you could also see that as meaning all but these. So all but none means all. And therefore, in J, if you add an extra enclosure level, it's an exclusion of indices instead of an inclusion of indices. A little bit arbitrary, but it works in practice. You get a lot of enclosures all over the place, if you need to type them out manually at least.
00:26:55 [BT]
The nice thing is that they actually also introduced the primitive ace or a colon. And that is exactly the same thing as an enclosed empty. And so you just, by putting the ace in, that a: in, you don't even have to put in the extra-- you're essentially introducing that extra level of enclosure.
00:27:17 [AB]
But that's just what it is. And it fits nicely as well that the a stands for all. So you want all along this axis. Ben also slowly started the ideas of leading axis orientation. It wasn't in full force, not for a long time. But it did come in.
00:27:33 [ML]
Oh, should we kind of pause and get more discussion on the nesting first?
00:27:37 [AB]
I wanted to mention one thing more, then I'll stop with this. That is a very common case, is this simple case, like we started with. Go way back to the beginning. We have a simple vector, x. And we have some list of indices. And we want to get out those elements. And now that we have this fancy indexing primitive, in order to do this very simple basic operation that we want to do all the time, we have to add this annoying extra enclosure. Because remember, we said that every segment gets its own enclosure. For a vector, there will only be one segment. So you have to be explicit about this enclosure. So it's very common in APL to see an enclosed vector of indices. Then that has to be parenthesized. And then the indexing primitive. And then some data. And that issue, that notational annoyance, has led to all kinds of other ideas and initiatives. But I'll stop here.
00:28:36 [CH]
All right. Over to you, Marshall. You have things to add, things to say.
00:28:40 [ML]
Well, see, this is the issue. Now I don't know where to start.
00:28:44 [CH]
Did somebody take notes about all the side things that I skipped lately, or we need to go back to?
00:28:50 [BT]
But we did get into the multiple dimensions a little bit. So I think we sort of covered that part of it. The one thing I was wondering about, when you're talking about the dimensions not being able to be applied to functions, what strikes me is you do the selection first, and then you apply the function to the new selection that you've indexed out. Is that what they used to do? So in other words, if you want to take the square of a certain number of elements of a matrix, you select those elements, you do the square, and you get the result. So you index before applying the function.
00:29:23 [ML]
I think he was saying more if you want to apply an operator to the selection itself. So like select rank one is one thing. Like if you've got a table and you want to get each column of it. So something that's-- if you have a list of which column you want out of each row. So you've got-- I want element one from the first row, and element five from the next one, and element two from the next one. Now in APL, we can write it as SQUADrank one. But with bracket indices, there's not-- I mean, because it's not a function, you would have to wrap it in a function first, and then loop with the function.
00:30:06 [BT]
I guess what I'm thinking about-- I'm thinking about the case with rank zero, where you'd apply it to the whole thing anyway, so there wouldn't be a difference.
00:30:15 [ML]
Yeah. I mean, so if APL's selection covers what you're trying to do, then it's all good. And if it doesn't, then it's not the best form for working with. It's not the best kind of thing to make your selection that you actually want out of.
00:30:31 [CH]
All right, so here's a here's a follow up question that Marshall doesn't know where to start, so I'll.
00:30:37 [CH]
All right. So here's a follow-up question that Marshall doesn't know where to start, so I'll give him a spot. I did not take notes on everything that Adám said. What I did do as he was speaking is I went and was trying to follow along both in tryAPL.org, [03] but also the APL wiki has a plethora of index-related pages. And I mean, if you go to the indexing page, the first line says, "This page is about the concept of extracting elements from an array. See index for the concept of a location in an array. See bracket indexing, index function, pick, and select," which are four different pages for the primitives designed for indexing. And if we go to -- which one is it? I believe it's the one entitled index parentheses function, which corresponds to what Adám said was SQUAD, aka Squish Quad. At the bottom it says, "Implementation support. This form of indexing is supported in J as the boxed left argument of from." And right underneath that it says, "See also pick and select," which were two of the other articles and I happen to know that in BQN they call it select. And so if we click on that page, it goes to -- even though it's called select, the title of the article is from because that's the name that J calls it. And then this reads, "From parentheses, we'll call it right horseshoe under bar, although it could be left. We don't know if it's right or left. Adám's saying it's right with a nod of his head, or the at symbol. And also called select or humorously sane indexing is a primitive function that selects multiple major cells, blah, blah, blah, blah, blah." Well, I shouldn't say blah, blah, blah, blah, blah. I'll finish the sentence. "That selects multiple major cells of its right argument using an array of indices given by its left." And then if I skip the second sentence and go to the last one, it says, "It appears in sax as at, extended Dyalog APL/Xyma APL, and Kap as the right horseshoe under bar. J, aka from, which is the left brace, and then as BQN with the symbol for select." All of that to say is that here's your starting point. Why is it so complicated? And I admittedly, I don't even understand the full overview. I just know that I like BQNs the best. And when I'm in APL, I usually typically go for SQUADbecause I tend to like the more functional code flow, data flow style. And bracketing, it's lispy. It switches the order of the reading. So I just always tend to go for SQuAD, which it doesn't even seem like SQUADcorresponds to this, it seems like a subset corresponds to from or select. And so why is it so complicated? And does BQN and all these other sax, extended Dyalog APL Kap, do they have the fully formed Pokemon, the highest evolution, all the mistakes are fixed, or is this still an open problem to be solved? All right, that was two or three questions in there. Go.
00:33:44 [ML]
Yeah, well, the main source of complication, of course, is, as you note, BQN is better. So there's this historical inversion where the better solution actually came later than the worst solutions. If I'm going to be serious, the-
00:33:59 [CH]
Well, that checks out. That checks out.
00:34:00 [ML]
One big source is that when you're working with arrays of indices, you have to connect one index to another, like if you have multiple dimensions in these arrays of indices. So we said that the original APL select does an outer product. It gives you, you know, you give it the indices in the rows and the indices of the columns. And so it'll select this entire row and select this entire column. And well, you can say the intersection, but the point is that you get every combination of this row and that column, this row and that column. But maybe you don't want to do this. So one thing you might want to do is instead to do something less like outer product and more like each. So say the indices correspond exactly to each other. So then you give it a list of specific places. You say, I want row one, column five, and also row three, column two, and so on. And you can't do that with an outer product because the outer product is designed to return, it's designed to do this subarray thing, which works very nice with the dimensional structure, but might not be what you want. So I guess once you go past the idea of an index that picks out one place in an array and you say, actually, I want multiple indices, then there are a lot of possibilities for what you want these indices to be doing. And so then what we get is a bunch of selection functions that try to combine multiple possibilities. And there are really a lot of ways to do this. Like Adám said, so when you have the simple version of scatter point indexing or reach indexing, the simple versions are all the same, but then when you do, they generalize it in different ways. So you've got all these functions that try to connect simple ideas together and all, like all of them, when you give it just one index into a vector, they should give you, they might enclose it or not, but they'll give you that specific element. But that concept generalizes in so many different ways that you get a really complicated, you get a mess of functions that you might want to do, and you want to kind of pack those together into something general and sensible that's useful. And there's really no right answer to that.
00:36:18 [AB]
So notice that APL, modern day APL, has two indexing functions besides for the square brackets, both this called index or SQUADindex, and there's also pick. And BQN as well has two, not the same two. It has the from or select together with pick.
00:36:38 [ML]
And I think it's worth noting that even k has two different kinds of selections. So it has, and it's going to depend on the dialect, but you have a selection with at, which that's the one that is, what do you call it? It loops over the right argument. So every scalar index gets turned into an element, but you can have a list of indices. But it's also got a form of indexing with dot that's more like APL's selection where it does an outer product on these indices. And so that corresponds to a multidimensional selection.
00:37:18 [AB]
There's also square brackets indexing in k and q.
00:37:22 [ML]
The square brackets, I think they just mean dot.
00:37:26 [AB]
Those are the two different syntaxes for the same thing. Just like you sort of have that in APL.
00:37:32 [ML]
Well, just like square brackets for SQUAD.
00:37:35 [AB]
Yeah, exactly. Cut down on things has multiple ways of indexing.
00:37:40 [ML]
Yeah. And it's it's hard to even say what is exactly the split there. I mean ad is.
00:37:48 [ML]
And it's hard to even say what is exactly the split there. I mean, add is a lot like just dot with only one element in its argument. So I mean, dot can index into any number of dimensions. But if you index into just the first dimension.
00:38:00 [ST]
You could think of it as syntactic sugar for apply or index the dot.
00:38:07 [ML]
Yeah. So I guess dot is the...
00:38:08 [ST]
For exactly that case you described, but it's just one dimension or one axis in play.
00:38:14 [ML]
So basically when you get down to one axis, it's more obvious that you can use these multiple indices. I guess at least it's more obvious what a list of indices should do. So there's really... Like if somebody gives you a vector and says, "I would like to index into this vector." And then they say, "What I would like to index by is this monstrous array." I mean, there's really only one thing you can do. You can say, "Well, I'm going to traverse into this array and find each number and each number is going to be in the index into the vector." But then if you have a multi-dimensional array, which in APL, that's what arrays are. And in k that's a vector of vectors. You can think of either as a plane vector or as a matrix and so on with more dimensions. Once you have multiple dimensions, you have to say, "Which number corresponds to what dimension and how do the dimensions interact and all?" And so that's kind of where the complication comes from.
00:39:18 [BT]
And I guess the whole thing is that indexing is really important because it's basically selecting what parts of a matrix you're going to work on or what you're going to end up manipulating. It's so important you have to have different ways of viewing it because the different ways allow you to do different things with different parts.
00:39:37 [ML]
Yeah. I mean, like if you don't have language support for scatter point indexing... So you can actually do it using index arithmetic. I guess actually the APL's base function is pretty good for this. So you take the shape of your array and then you use base and then you use your indices. So that would take you from a matrix of indices where an individual elements index is a column of that matrix. That would turn that into a vector of indices into the ravel of the array. And so then that converts it to one dimensional indexing. But I mean, one, that's just pretty annoying to write. Two, it doesn't actually do bounds checking for you, which is a really big issue. So you're going to like... If you have an out of bounds number there, it's just going to wrap onto the next row of the array and you'll get all these super weird bugs and be very confused. So of course you want the language to support this in some way. And I think it'll also probably end up much faster if the language, in certain situations at least, when the language is looking at the numbers directly and saying these are indices.
00:40:51 [AB]
You could make do just with, shall we say, pick. And then you can use various operators to loop.
00:40:59 [ML]
Yeah, well and BQNs, like I think the BQN specification actually, where it defines all the primitives, it starts with just a very simple pick that takes a number and a vector and gives you that element from the vector. So you can define it all. But of course, if you start at that low of a level, you're not really getting... You're not asking your language to do any array programming. So you don't get any of the utility of arrays at all.
00:41:28 [BT]
And if you do a pick on a matrix, do you get the first item, like that next level down? Or do you get... How does that work?
00:41:36 [ML]
Well, in my version, it's just not allowed. It's like the way the functions in the specification are defined so that they'll only ever use pick with those arguments.
00:41:45 [AB]
That's the minimal set, you mean? Eventually you can, right? Once it's built up.
00:41:49 [ML]
Yeah, yeah. And that's not actually what BQN calls pick, but that's the function that the specification uses to define things. So this kind of selection function, it only calls in this specific way.
00:42:04 [ST]
Do we have any fans of semantic indexing in the house tonight?
00:42:09 [ML]
What? What's it mean?
00:42:11 [CH]
You'll have to define what that means before I can tell you if I'm a fan of it or not.
00:42:15 [ST]
Oh, okay. So when I'm talking, I like to distinguish between, say, a list and a vector. A vector is an ordered sequence of items of the same kind. So you see a string, a character array as a vector. But a list is an ordered sequence of things which may be of different kinds. So if I refer to the index position of an item in a vector, this is like the third character in a sequence, the third item in a queue of requests coming in or whatever. But if I refer to the third item in a list of miscellaneous things, then its position has got some kind of meaning. So an example might be my name, my birthday, my address, my telephone number, the third item is my address. To a limited extent, I can keep that mapping in my head. The list item three is my address. As these things develop, we really want better notations for that. So I can say out of this list, I want the address. In non-array languages, we use objects. We name the items. They're not ordered lists. q has this neat thing, which basically says the index. A list is a mapping from an index to a value. A dictionary has got the keys explicit. [04] The keys of the dictionary make up the index. And you can think of a vector as a mapping from the natural numbers to the values. So you wind up in q with pretty much the same syntax working with dictionaries and with vectors. And the effect is when you're writing application code in that, you've got this kind of happy situation where you can index your lists with numbers or index them with vectors. And when you go up a dimension to tables, the second access, the columns, are typically named with strings. And the syntax supports that. So as an application programmer, I really like that because when I need to refer to lists tables rather than vectors or matrices, I've got semantically meaningful ways of indexing them. So that's long answer to your short question. What do you mean by semantic indexing?
00:45:03 [BT]
That sounds to me a lot like what Tali Beynon was talking about with his rainbow arrays. Very similar. He was talking about essentially the values to indices being a function. So in other words, your natural numbers map onto the values. That's a function.
00:45:21 [ST]
Which is in fact pretty much the way that q does it, saying there's an isomorphism between functions and arrays. They're both mappings from a domain to a range.
00:45:31 [AB]
Which then makes sense for the syntax. When we index with the square brackets, we tend to put the square brackets on the right of the data. When we do function application, we tend to put the arguments in some form of brackets (parentheses, whatever it may be) to the right of the function name. Not in APL syntax, but in general, although APLs tend to just let you remove the parentheses (the brackets). Not so for k-style languages, for user-defined functions. But if you unify indexing with function application, so that an array is just a function from indices to values, then the syntax all of a sudden makes more sense. It fits in better. Then maybe you can avoid that syntactic conflict that causes BQN and J to entirely reject square bracket indexing. k-style languages instead embrace this syntax and make that central to everything; everything is just an application. And then you have some primitives that also allow infix and prefix application. You can always use square bracket fabrication as well.
00:46:57 [ST]
I guess this has got a particular twist for array languages, because array languages wake us up if we're trained with scalar languages. A lot of things that we used to think as scalar programmers were natural applications of control structures (if-then-else), are easily represented as mappings. So instead of writing a lot of code going: if {this}, then {that}; switch case ... you set up a mapping from one set of values to another.
00:47:39 [AB]
But that requires you to compute everything in advance, right? Unless you have some lazy evaluation. [Steven agrees]
00:47:45 [BT]
I was thinking that also builds into the functional part of the array languages, because as soon as you're doing mappings, as opposed to if-then-else, now your mappings are functional. You're thinking about functions.
00:47:56 [ST]
I know the classic textbook or early textbook examples of how to do, say, something as simple as fizzbuzz go: if {this}, if {that}, if {that}, test, loop through. That approach, which is kind of fostered by the language tools you've got, obscures insights like fizzbuzz is working on a 15 element cycle. And you can do a simple test on your entire domain and say, where does it fit on this? You can do a 15 modulus and then a mapping to the results you want.
00:48:46 [ML]
Or even just reshape a length 15 pattern.
00:48:49 [ST]
Yeah, reshape the length 15 pattern so there's no test to do at all.
00:48:52 [AB]
You still need the original number in there for those things that are neither fizz nor buzz. [Steven agrees]. So it's not just a mapping, but I think maybe I can leave a link in the show notes to this as well. I think I have a fizzbuzz implementation somewhere, or I'll make one, where I do pre-compute everything so we have all the numbers. I think I create a matrix (I don't remember if it's three, four rows or something like that). So fizzbuzz on a single number, that's simple. The fun stuff comes in when you want to do fizzbuzz for a range of numbers from 1 to 100, to some given number. So I create a four row matrix. First row contains the numbers themselves. Second row contains only fizz: fizz, fizz, fizz, fizz the whole way. The third row contains buzz, buzz, buzz, buzz, buzz. And the fourth row contains fizzbuzz, fizzbuzz, fizzbuzz, fizzbuzz, fizzbuzz, like that. Now we can build up indices. So we want one element from every column.
00:49:48 [ML]
Well, this is a good time to mention yet another selection function that often goes by "case", which is just perfect for this. [Adám agrees] And what "case" says is that instead of the index corresponding to one particular array, you pass it in a series of arrays. In this case, we would have those four arrays that Adám described. And those all have the same shape. And you pass it in another index array that has the same shape as those. And the result will also have this shape. And it looks at the index array, and it takes the index, and that is an index into your collection of arrays (into your list of arrays). And it tells you which of those arrays that element should be taken from. So for fizzbuzz, you would have something like [this]: starting at 0, the first index would be 3. So that tells you to go to the fizzbuzz array. And then you get a few 0s. You get 0, 0. So that's 1 and 2. And then you get 1. So 3 is fizz. And then you go back to 0 for 4. And you do 2. So 5 is buzz. And so on. So "case" is the perfect selection function for that. What "case" does that wasn't in the original APL is bring it to a higher level where you've got a list of arrays that you're selecting from instead of just one array. But you can create indices, which I guess Adám was going to describe but it's just not quite as nice.
00:51:25 [AB]
Yeah. You could construct one giant index array that just does the whole thing in one swoop: that's another way to do it. Although yet another way to do it that you might not even think of (I think I'd like to bring this up): an indexing function that you might not think of is "rotate" [05]. And that's something I certainly do sometimes. Imagine that we have a character vector "A B C D" and we need to choose one of the letters based on the numbers: 0, 1, 2, and 3. Let's just do index origin 0 for now. Obviously, you can index using that number. But you could also cyclically rotate (I mean shift out but let's just say cyclically rotate) this character vector by 0, 1, 2, or 3 steps. And when you're done with that, you take the first element. So 0, you get the first element because you know there's no rotation. One step rotation, A moves over to the end and you have "B C D A". And now you get the first element and so on. So that also works. So imagine you had that big matrix that I spoke about: first row, the numbers; second row, all fizzes; third row, all buzzes; and fizzbuzzes in the last row. So now we can rotate every column (or if we had it in transposed form, we could rotate every row but APL has a primitive that makes it easy to rotate columns). We could rotate every column by an amount: 0, 1, 2, or 3 steps. And once we have done that rotation, we just take the first row and discard the last three. And now we've got the answers to all the fizzbuzz values. So there are many ways.
00:53:09 [BT]
Reminds me of a bike lock.
00:53:11 [AB]
What did you say?
00:53:11 [BT]
You flip a bike lock, the number, and then you just take that. You got the combination. Bang! you're done [others laugh].
00:53:16 [ML]
So you've got a bike lock that's 100 dials long. The first dial says: 0, fizz, buzz, fizzbuzz. And the next says: 1 fizz, buzz, fizzbuzz. And you've got to rotate all the dials to the right place.
00:53:29 [AB]
Yeah and then you can open the lock. So APL has rotational primitives (first and last axis rotation), that is really interesting in how it works on columns and rows. It can be useful at times. Because BQN and J have a single rotation function that works on consecutive axes instead, it can't rotate every column separately.
00:53:59 [ML]
The rotation has to be uniform.
00:54:01 [AB]
But you could rotate this many rows and this many columns and this many layers, this many rows, this many columns.
00:54:08 [ML]
All your dials on the bike lock are welded together and it's not really that secure.
00:54:13 [AB]
There are little plates that you have to slide into the right position [chuckles] for the holes to line up. Yeah, but here, too, you can imagine: if you have a giant matrix, and we want to select from the third row, fourth column element (let's do, again, index origin 0). So that means it's a (2, 3) we want to select. Now imagine if we did a rotation. So basically, think of it as the size of the matrix is a steel plate. And then you have this canvas that's spread out of it with all the elements. That's all squishy, stretchy. This canvas goes all the way; surrounds entirely the steel plate. Now you slide the canvas up and to the left; three steps to the left, two steps up. Everything moves around to the back. And it's magic so everything that comes up over the edge of the top left corner just appears magically in the bottom right corner. And then you select the first element in the top left corner. So effectively, your rotation with the "first" is the same thing as an indexing operation. All these things tie in with each other. This is also a valid way to select things.
00:55:30 [ML]
But not a valid way to solve FizzBuzz. [Adám agrees]
00:55:33 [BT]
And that's like a two-dimensional rotation.
00:55:36 [AB]
Something we haven't addressed at all here. I don't know if we should go into this or if that's for the follow-up episode. We've only talked about extracting data from an array and possibly duplicating it (where we haven't even mentioned that, that you can have duplicate indices, then you get multiple of the same thing. It's super useful). We did mention it in passing: Marshall said the thing about having a space and some symbol, and then you can do booleans and you get those symbols out. But there's also modifying our application and something we very often want to do is preserve your whole array, but you want to switch out some elements here and there. Or you want to apply a function to the elements here and there. Do we want to go into this as well?
00:56:21 [ST]
I'd like to reference a couple of papers on this one by Paul Mansour (I guess we'll put it in the show notes; it's not coming to my mind, but I reread it recently) in which he argued against the use of high-rank arrays and said you keep everything in a table. One good reason for doing that is we kind of think that way. We can just about envisage a rank three data structure. Rank four becomes a little too abstract to think with. I don't respect everything that Paul Mansour does; some of it I just revere. I don't find myself completely buying this argument. That's the second paper I want to reference. It's an article I wrote in Vector (the BAA journal some years ago) about an actuarial application in which the value that the actuaries were trying to calculate and work with gets partitioned in different ways and gets extended, for example, across time. It grows at assumed interest rates and so forth. It turned out it was really convenient to represent that in a numerical array with eight axes and the algorithms for doing the calculations and setting that up just shrank down to tiny bits of code. In fact, I think the last time I looked at this, it was a series of outer products. I think the only time I've seen in an application, an outer product "reduce".
00:58:11 [ML]
Why, I use this a lot.
00:58:12 [ST]
Well, I should come around more often, clearly [chuckles].
00:58:15 [ML]
In fact, if you want to implement selection on multiple axes, one really good way to do it is to find the strides of the selected array. So, for the lowest axis, it's one. For the axis above, it's the length of that lower axis (it's like a backwards product scan). You multiply the index for each axis by that axis's stride. Then you reduce the indices with a plus outer product. Then you use that to select into the ravel of the array of values.
00:58:54 [ST]
Okay, it seems to resemble in some ways the way I wound up describing or accessing this. I set up a series of indices for each axis and a constant to name the axes. I was able to write expressions which show which parts of the array were getting summed across which axes, relatively fluently. Well, Marshall, you've already leapt in and answered my implied question [chuckles], which was, has anybody else worked with this and tried to solve it? When I explained it to the actuary I was working with at the time, he frowned a bit. I said: "it must be hard to envisage a rank eight array." He said, "No, no, that's quite easy. I'm just thinking of it as an SQL table with eight columns or indices." Which I suppose would be the way that Paul Mansour would be looking to represent it but I can't see the code for manipulating the figures would be anything like as sweet as for the eight axis array.
01:00:10 [AB]
I don't feel like I have a huge problem with visualizing high rank arrays, but I don't think of them as a table with indices. I think of them as physical things, as a library. So if we work our way backwards where a matrix is represented by a page (we have rows and columns and then positions on the page, so a table on a page), then a rank three array is a book of pages, so it's a page number that comes first. Next up then is which volume in this series of books. I certainly deal with a lot of books that come in many, many volumes, so this is perfectly normal. If we have one series on each shelf, say in our bookcase, then we can also number our bookcases. And if you've been to a library, then you will find that there are bookcases numbered like this, and then they're set up in corridors which are themselves numbered. And that's the case on every floor of the building, in every building on the street, on every street along the avenue, on every avenue along the neighbourhoods, and every neighbourhood in the city, and the city in the county, and the county in the country, and the country on the continent, and the continent on the planet, and the stars and the galaxies.
01:01:45 [ML]
So there's a big square arrangement of planets, and each planet is square.
01:01:50 [AB]
Not square, rectangular [others laugh].
01:01:57 [ML]
Right angled. And yeah, the continents are rectangles.
01:01:57 [AB]
Have you ever been to an American city, Marshall? [06] This is exactly how it is!
01:02:08 [ML]
Not quite.
01:02:10 [AB]
Just in the corner [chuckles], Fifth Avenue and Second Street.
01:02:13 [ML]
Much of Manhattan is arranged like that. Not many cities that conform completely.
01:02:20 [BT]
And if you're in Quebec City or Nanaimo, you actually got a hub city. So they actually do radial like European cities do.
01:02:27 [AB]
And some cities also use house numbers that are not consecutive, but are more like percentages between cross streets. So you know that if it's number 267, then it's two thirds of the way between the Second Avenue and the Third Avenue, and on the cross streets on this street; this kind of thing. So right, somehow we managed to mess up the world and not build our cities this way but this is how I visualize it being that this system is simpler than what they deal with on a daily basis in a city. And surely people can find a way in the city without getting all confused: "what are you talking about? You've got multiple streets?" then they should also be able to figure out multi-dimensional arrays. Maybe it helps if we name the axes. That's something we've discussed here in the podcast before. We have names for the rows and the columns, maybe the layers as well. And then we start running out of words. But if we call them just rows and columns and pages and volumes and shelves and bookcases and so on, it's pretty straightforward.
01:03:42 [ML]
Yeah. So I guess (and you know, this gets into the round versus square planet question) what arrays really do for us is they allow us to encompass all this structure (and you know, on Adám's planet, the amount of letters on these pages is just like mind bogglingly huge) ... they allow you to consider that structure by using indices. And each index is just a list of maybe eight, maybe 12 things. That's because of the regularity. No matter what the second index is, the fifth index always has the same possibilities. So the array structure, even in this huge, multi-dimensional thing, all you have to understand to understand how it's laid out is: how many pages are in each book; what are the dimensions of each page; how many books go in a volume and so on. You only have to remember how each axis works instead of how this big outer product of all the axes works.
01:04:59 [AB]
Yeah. The restrictions on the arrays to be orthogonal (or rectangular, if you want) makes it much easier to reason about.
01:05:10 [ML]
Yeah. Which certainly doesn't apply to books.
01:05:11 [AB]
No, okay. Well, did I mention multiverses? [Bob laughs]. It's much easier to reason about. Imagine if instead everything was just lists of lists, which meant that at no point do you have any idea about how long this level is. That is in fact more or less how the actual world is: just because this shelf has three books on it doesn't mean the next shelf has three books on it. It could have many more. But that means you always have to check how long it is. You can't make any assumptions.
01:05:42 [ST]
That's the case with q's arrays. However, q basically doesn't complain if you index out of range.
01:05:53 [ML]
Yeah. So all the axes are really infinite. In a book you can say, what's the 10th letter of every line? And some lines are not going to have 10 characters on them and then it won't work. But if you put in a space for that, you can still do it because the lines still have a shared indexing scheme, even if they don't all have data for every index.
01:06:15 [AB]
Then you have to have some kind of null value to identify the ones that are missing. [Marshall agrees]
01:06:21 [ST]
Yeah, every data type except Booleans has got a null.
01:06:26 [CH]
Okay. Circling back to Adám's question from (we're going to estimate) 15 minutes ago: should we talk about the ying to this yang or the yang to this ying? Nobody knows how that's pronounced. We do know whether we should talk about it. And the answer is yes. I think we will defer that though to a follow-up episode, which brings me to the first of four comments/questions, which hopefully will wind down this podcast that I've been accumulating. And we're going to go in reverse chronological order. So it's a first in, last out situation we're in here. Number one, we brought up rotate at one point. I forgot that I wanted to mention this on a previous episode, but I remembered now. It is that I would like to commend BQN and also Kap (but Kap was just copying Dyalog APL or some APL from the past) for keeping the vertical rotate as the symbol for the first axis.
01:07:20 [ML]
Well, I was just copying Arthur Whitney's A or A+. So credit goes to Arthur Whitney.
01:07:28 [CH]
All right. Credit goes to Arthur Whitney. And I came across this because, I think Uiua uses a completely other symbol, but when I was taking a closer look at TinyAPL, it used the axis first or leading axis version, which is the horizontal one. And that may be a better decision, but as the dictator/host, I would just like to say there's a softer place in my heart for the vertical rotate and therefore [laughs] I appreciate that it exists in BQN. And was it A or A+?
01:08:02 [ML]
Both. I mean, A+ is just A plus a GUI, so the plus part doesn't change what language it is.
01:08:10 [BT]
And if you can overlook the period, J does the vertical as well.
01:08:12 [CH]
Gotcha.
01:08:13 [ML]
Yeah, it is vertically oriented. I will say, most of the time, even in an array language, most of your data ends up in vectors. So when you have a vector (and the vector is laid out in a line) the most reasonable glyph for that is the one that indicates a horizontal flip. Because you don't say, oh, I'm going to reverse my vector, so I should use the vertical flip. Which means that APL system, which makes perfect sense for matrices, kind of, when you go into the leading axis thing, it's not really like ... [sentence left incomplete].
01:08:49 [CH]
Yeah, the visual nicety is not there.
01:08:53 [AB]
Maybe vectors should be printed vertically [everyone laughs].
01:08:57 [CH]
I'm ending this tangent now, otherwise we will never get to my points two, three, and four. Maybe we will revisit which rotate glyph should be the one to rule them all in a language with only one glyph. Because it's built from a leading axis point of view first. The second one is also visual. So while this was like 30 minutes ago now, we were talking about "from" (aka select, aka, there was one other name for it). I saw on that Wikipedia page some of the languages that implemented it (Kap, extended Dyalog APL, the ones that use the right horseshoe underbar). And then that made me think that upcoming in Dyalog 20.0 is the "behind" operator (and it even addresses in that wiki article that it's easier to spell a lot of these reordering expressions with this "select", aka, right horseshoe underbar). But then, this is where things take a right turn. I was like: "okay, Kap has this". Let's go take it for a spin. I boot up my local instance of Kap and the underbar (I believe it's called bind left in Kap) and the selection glyph: they do not have the underbars at the same place in the glyph. And then I honestly for five minutes stopped paying attention because my soul was slowly dying. And then I, in the meantime, went and downloaded (because I only have BQN 386 locally) and got TinyAPL 386. I went and got Dyalog APL 386? I got all the 386s. BQN we can forgive because those glyphs aren't inside of it. But they're all over the place. TinyAPL, they're also all over the place. The only one that has consistently aligned underbars is the APL 386. However, the less-than and greater-than-equal-to, their underbars are a bit higher. So my question is (and it's like, what is the relevance? This is just important; we need to talk about this folks because it's the beauty of APL that is so important here): is this intentional in APL 386 because they're not operators and they are functions or verbs or whatever you want to call them? Or is this an accident that must be rectified?
01:11:20 [AB]
Okay. So I don't know when you went and downloaded the APL 386 font [07].
01:11:24 [CH]
Like 10 minutes ago.
01:11:26 [AB]
Okay, perfect. I didn't make an announcement when we started. I thought about it while I was making an announcement [others laugh]. The thing is that quietly, Dyalog has been sponsoring a font developer to completely redraw the APL font. APL 386 used to [have] my very slight fixes and added glyphs to the APL 385 font. And then others grew from there. The BQN 386 font is just an offshoot of that. And I also backported some changes there. But this is a complete redraw from scratch.
01:12:01 [CH]
I was going to say the self/swap (which is the tilde diarisis) it looks different. That little squiggle looks thinner. And I was like, that does not look like the 386 commute operator that I know. Are you saying that if I went and did this, that I have the new font now?
01:12:18 [AB]
Yeah. If you did it 10 minutes ago, then yes, you have the new one [chuckles]. Because it's being updated daily.
01:12:23 [ML]
Why is it not APL 387?
01:12:27 [AB]
[Others laugh] Because I don't think there's much ... [sentence left incomplete]. We've considered what to do with the naming; we're still considering it. We've gone back and forth. If you have an input on that, we could call it 387 instead. We may just use this as a placeholder name that some companies do when they're developing a font and then finally, when they choose to say that it's ready, then they do some community thing on what it should be called. And everybody thinks it should be called Font Face or something like that. [everyone laughs]
01:12:51 [CH]
Boaty McBoat Face.
01:12:52 [AB]
So that's still up in the air, what to call it. But just for the ease of it, we just slotted it in my 386 slot there.
01:13:03 [CH]
In my head, I was like: "if these are all 386 based, I'm surprised". I was just thinking: "oh, I guess Marshall forked it at some point and there's been additions over time." But the answer is that this is a whole new font, which is why the delta.
01:13:19 [AB]
It's completely different.
01:13:20 [ML]
Yeah. so I think if the differences are so large that somebody is just casually getting the font and noticing [it], first, some people might prefer the old 386 and then it becomes very difficult for them [Adám agrees]. It's just confusing if they've all got the same number.
01:13:36 [CH]
I interrupted you, Adám. You're about to make a second point.
01:13:38 [AB]
Yeah. I mean, it's perfectly valid. Other people also raised [that] maybe it should be called something else. It's actually more a coincidence that the new one is also called 386. It's more because I gave in the old 386 for the font designer to look at and work off and then he delivered his product named identically and I just didn't bother changing it. That's why it has the same name. But we can recompile it.
01:14:04 [ML]
So it is actually unnamed, but it's using a temporary name.
01:14:06 [CH]
If we're going to increment, like let's just round up. This is a vast improvement. Let's just call it 400!
01:14:13 [ML]
Whoa, whoa. That's a big step!
01:14:14 [AB]
[everyone laughs] 390 sounds great.
01:14:18 [ML]
I mean, this would be like if we just went on next week and we said, all right, BQN episode 115.
01:14:24 [CH]
That's true. That's true. Well, we would never do that. We would never do that.
01:14:27 [ML]
Not BQN. ArrayCast episode 115.
01:14:30 [AB]
So you're absolutely right that all the underlines are aligned. And I was very strict with the font designer on this. It took a long time to kind of teach him the principles of APL font design [chuckles], that all these elements need to be the same. If you adjust one thing, one place in the font, you have to adjust it everywhere else. And everything is just depending on each other. Every sub-glyph or every simple glyph is used in combination with other glyphs somewhere else. And one of the last things I did actually was ask him to move the underscores underneath the left and right shoe down to be on the level of all the other underscores. They used to be up, similar to less than-equal-to and greater-than-equal-to. And I did this because there are these mathematical operators (you could call them, mathematical symbols): less-than-equal-to, greater-than-equal-to. There's also a subset-or-equivalent-to (and it's a swapped argument form: a-superset-or-equivalent-to). And APL uses less-than-equal-to and greater-than-equal-to exactly like the mathematical symbols. But those set symbols are not used at all like that: that means something completely different. They're partitioning, selection. So the reasoning there is that if we move the underscore (and I did make experiments with that) if you move the underscore on less-than-equal to all the way down, so it's very far away from the less than symbol, it sort of doesn't really look like a single thing (less-than-equal-to) anymore. It's not a variation. I don't think of it as a variation of less-than. You could think of it like that. But it's a thing in itself. It's a single symbol. It's not an underscored symbol as opposed to all the other ones. So that's the reasoning. If it really bothers you, then it's not too late to change this [chuckles].
01:16:27 [CH]
No I think it's fine. As long as it was intentional. Because that's what I thought, is that I could see that these two are, we could call OG symbols if you want. But yes, they're mathematical in nature; they were there from the beginning. The more I stare at it, I do wonder because [of] this extra space now on the right or left horseshoe underbars. In the case where you have adjacent glyphs that are underbarred, it's obviously going to look weird if they're not aligned. I'm curious what the other cases are. But we will defer. We got some exciting breaking news: new APL font to be renamed, potentially (maybe not though; we'll find out in the future). It is available. I got it off the wiki. I'm in the Kap app right now, so I'm not looking at technically [the] Dyalog APL glyphs; I'm looking at all the glyphs that are in Kap. When they're all next to each other, the underbar ones, they look good. With that, I have two more points or questions, but I'm not sure we're going to have time. Maybe I'll ask one of them and I'll defer the other one. I remember at one time I said, I'll save a question when we were interviewing someone and then I actually got a DM saying, what was the last question? And I was like, I can't remember. I didn't write it down. Maybe if I went and re-listened to the episode.
01:17:57 [BT]
I think that was Brandon Wilson. [08] And he may have been asking to get ready for the next time he comes on [everyone laughs].
01:18:04 [ML]
We'll put them in the show notes.
01:18:05 [CH]
Yeah. But I'm pretty sure if I go listen to the first 15 minutes, I'll have the exact same question that I had originally because it went unanswered. But hopefully this will be a short answer. So Dyalog 20.0, I mentioned, is coming up with the behind operator. But then when I was playing around with it, it seemed like it was the selection glyph that actually made the three character spelling of sort, which famously Dyalog at the current moment in 19.0/2 or whatever we're on and future 20.0, we'll still not have. I think actually, Uiua has taken it out of experimental. So all of the "modern" [languages]: Kap, BQN, Uiua (we can include J; they've got two glyphs; We'll let it pass; two's fine). But right now, I think using
SQUAD in Dyalog, the latest Dyalog that's not out yet (so not 20) it's five characters. Is it the "behind" that can shorten it to three? Or are we missing the selection? Or can "behind" plus SQUAD get us to three? I think the answer is no, but I could be wrong.
01:19:06 [AB]
No, you need both "behind" and "select" in order to do this because the problem is what we spoke about. In the very simple case, we just have a permutation or a subset of indices and you want to index along the first dimension or into a vector. Because the design of the SQUAD indexing was that every segment in the square bracket corresponds to one enclosure in its left argument. Then when there's only one segment, that has to be enclosed. So in order to sort ... [sentence left incomplete].
01:19:44 [CH]
Well, you answered my fourth question right there anyways. It was that: can we explain the difference? Because I was trying to build that out in Kap versus Dyalog APL. And yeah, you still needed the "enclosed", which is what leads to that "enclosed" in my preferred spelling: jot, grade-up or grade-down. That's because of SQUAD. So yeah, you'd need an improvement on SQUAD in order to be able to spell. Technically some of you might be thinking, you don't actually need "behind". You can just use this with a fork where one of the symbols is the tack. That works as well.
01:20:15 [AB]
But that is a big difference though, because that means you can write "enclosed atop grade behind SQUAD" [Conor agrees]. You need the "enclosed atop grade" because you can't use the permutation vector from grade directly with SQUAD; it has to be enclosed. But this is just a derived function. This is not a train, which means you don't even have to parenthesize it inline.
01:20:36 [CH]
In, in certain contexts, yes.
01:20:48 [AB]
Yes, it does actually save you quite a lot. Yes, it's still five characters but it means that it is one symbol longer than if you had to spell out the word "sort". Then on the other hand, sort might need spaces next to it to separate it from. So we're getting there. That doesn't mean I don't want to do "select". Definitely on the roadmap. There's a bit of a design question. How exactly should it work? That's why it wasn't included into [version] 20.
01:21:06 [CH]
Well for what it's worth, I think I would prefer sorts. Because "select" helps in a lot of cases, but it only gets you partway to sort. I probably reach for sort more than I reach fo ... [sentence left incomplete]. Actually, I don't know. I don't know. I'd have to do some ... [sentence left incomplete].
01:21:23 [AB]
I mean, "sort" can do one thing. It does it well, but it only does one thing. Whereas "select" allows you to combine it with lots of things.
01:21:32 [CH]
But it is beautiful. It's a beautiful one thing.
01:21:34 [AB]
Sorting [chuckles]
01:21:37 [CH]
[Conor laughs out loud] It's like one of the major categories of ... [sentence left incomplete].
01:21:39 [AB]
Yeah, but sort doesn't do, for example, a sort-by. You can't take one set of data and grade it and use that permutation vector to sort something else.
01:21:49 [CH]
That's true. But that's what we got the grades for when we need them.
01:21:51 [AB]
Unless you have an actual sort primitive that in monadic form just does a sort on the array itself and the dyadic form does a sort-by. Which I think TinyAPL has. I think so.
01:22:06 [BT]
And J does.
01:22:07 [AB]
And J sort of does.
01:22:10 [ML]
Well, no, because J's monadic version is the grade.
01:22:12 [AB]
Yeah, so in J you have to add a self-commute. So it's only sort of ...
01:22:18 [BT]
But that's your second or third glyph.
01:22:20 [CH]
Oh, yeah, I always speak of J in glyphs, but they're digraphs. So technically it's three characters. Or trigraphs, yeah.
01:22:30 [AB]
But let's say you want to shuffle an array. Sort is not going to help you. But a "select" is going to help you.
01:22:41 [ML]
Well, in Uiua, unsort is shuffle [everyone laughs].
01:22:46 [AB]
Yeah, you say you prefer "sort" over "select". I think "select" is a pretty fundamental thing. And once you get a "select", then "sort" becomes three characters.
01:22:58 [CH]
Yes. And I think I prefer "behind"; I actually don't know. I don't know. I don't know. I definitely prefer "behind" to "select". I think I mean, I just prefer them all, to be honest. Take a page out of TinyAPL's book [09] and just add them all in. But I don't think there's any differences between Dyalog APL and the company behind that and TinyAPL's ability to add stuff. No differences whatsoever. But I'm very excited about "behind" coming. I mean, I've probably said that like 10 times by now on the podcast. Anyways, questions answered and we're very much approaching the hard stop for this recording. I think we said we were going to try and go shorter, and I just checked and we're at an hour and a half. So we failed! [everyone laughs]. No surprise. So with that, we will add all the show notes, the links to all of the pages we referenced, APL wikis, different languages in the show notes. If you've got thoughts, comments, questions, you can reach us at ...
01:23:57 [BT]
Contact@ArrayCast.com. We're willing and listening to your emails and you can send them in and if you have any comments you've got, questions you've got, we're happy to entertain them. We do read them all. We don't always respond to all of them, but they do all get read and thank you for the interaction. Thank you as well to our transcribers. This time round, it's Adám and Sanjay. Igor is taking a slight break; things have changed in his life. He will be back.
01:24:27 [ML]
He will not receive any gratitude in the meantime [others laugh].
01:24:29 [BT]
What's that?
01:24:31 [ML]
He will be back, but he will not receive any gratitude in the meantime!
01:24:32 [BT]
Oh, well, no. We've huge gratitude for what he's done. And thank you. That includes other people who have transcribed over the other times, I think going back to Rodrigo and I'm trying to think. Somebody else did the first transcription. I'm trying to remember who it was. But anyway, we're past the hour mark [laughs]. So, I guess I should wrap things up and say stay tuned for episode 100 coming up next.
01:24:56 [CH]
Yes, stay tuned for episode 100. We're in the midst of trying to lock down a special guest and you'll just have to wait and ... yeah, wait and listen, I guess (I was going to say wait and see). I guess that works here. You got to wait and see till episode 100 in t-minus.
01:25:13 [ML]
And see the title because sometimes people read those.
01:25:15 [CH]
This is true. This is true. Wait and see. It works perfectly. And wait and listen if you're going to be listening. With that, we will say Happy Array Cast!
01:25:23 [BT]
Happy Array Cast?
01:25:25 [AB]
Listen to us: Happier Array Cast.
01:25:27 [ML]
Smoothest plane you've ever landed!
01:25:30 [CH]
Listen folks, we can keep this in. You might have been able to tell my voice has been degrading in quality over the last hour and a half. And I had my bachelor party this weekend. I might have lost a few IQ brain cells, whatever you call it. We apologise. We'll do that again, folks. We'll do that [laughs]. With that, we say Happy Array Programming!
[everyone together laughs and says]
Happy Array Programming!
01:25:59 [Music]