Transcript

Transcript prepared by

Adám Brudzewsky, Bob Therriault, and Sanjay Cherian

Show Notes

00:00:00 [Brandon Wilson]

That mode of thinking, at least for me, brought these Iverson four points that he has in his Notation as a Tool of Thought paper. The subordination of detail, the suggestivity of expressions, things like this, made it much more tangible for me and able to actually use it. I felt like I understood what they mean.

00:00:20 [Music]

00:00:29 [Conor Hoekstra]

Welcome to episode 97 of ArrayCast. I'm your host, Conor, and today with us we have a special first-time guest who we will get to introducing in a few moments, but first we're going to go around and do brief introductions. We'll start with Bob, then go to Stephen, and finish with Rich.

00:00:43 [Bob Therriault]

I'm Bob Therriault, and I am a J-enthusiast.

00:00:48 [Stephen Taylor]

I'm Stephen Taylor. I work with APL and q and occasionally get enthusiastic.

00:00:53 [Richard Park]

I'm Richard Park. I do APL training and outreach at Dyalog Limited.

00:00:57 [CH]

And as mentioned before, my name is Conor. I am an array language enthusiast at large, love all the array languages, and super excited about chatting with our guest today. But before we get to introducing him, a small hint, we have a couple of announcements. I think we have two from Bob, then we'll have one from Stephen, and then I have a short follow-up from a listener after last episode with John. So over to Bob for the first two announcements.

00:01:22 [BT]

Okay, the first announcement, the smaller of the two I think, is that ArrayCast has been on Twitter as a way to let people know what's going on. Twitter, X, whatever you want to call it these days, we will no longer be on Twitter. [01] We're moving off Twitter. We have a Mastodon account. We're on Hachyderm. It's quite nice if you're looking for us that way. I'll still post things on the APLfarm. And actually the other one, I used to post on the Reddit, and the Reddit has, I guess there's no moderator. There's no way you can post anything anymore on the APL/J/K Reddit. So that's kind of weird, but I haven't been posting new stuff on because the newest stuff is my last post for the Ken Iverson episode, and that's two months old. But anyway, that's just a heads up. If you see us vanish from X, that's going to be permanent. We're not going to be on there anymore.

00:02:18 [CH]

For a follow up question to that, are we starting an ArrayCast account on Mastodon as well, or are you just going to be posting it from your personal one? Just so folks know to go either follow you or a new ArrayCast account.

00:02:29 [BT]

I guess it's my personal one. I'm just going back in my memory. I guess it's my personal one. I haven't started an ArrayCast account.

00:02:36 [CH]

So top of the show notes, folks, will be a link to Bob's Mastodon account if you have it. I, for the moment, am still, I'm actually on all three now. I've decided one of my New Year's goals is to test all of them out. I'm hoping that I can migrate to one. But yes, TBD. So maybe I will also start trying to post ArrayCast stuff from all the platforms so that if you're on any one of them or all of them, well, we'll figure that out. So follow us everywhere for where we are.

00:03:11 [BT]

Follow us anywhere, follow us everywhere. There you go. My second announcement is actually a fairly big one for me. With Eric Iverson's encouragement, I have wrapped up the work that I've done on the wiki and we'll put a link in the show notes. And it's got a new look. Essentially what we basically, what I did is I didn't really contribute very much to the content of the wiki. That's beyond me in a lot of areas. I have contributed in the past. But the big deal was trying to organize it so the information is more available. And what we've done is we've created something that's a little more friendly, I think, for newcomers. And also at the bottom of all the category pages, there's essentially a navigation link that'll take you into categorization of different pages. So if you're looking for information, it'll narrow down where you're looking for. And once again, I'll recommend Ed Gottsman's JViewer. If you really want to use the wiki intensively, that's a great tool for it. You need to be on JQt. But with that, you can do an incredible search across not only the wiki, but forum posts and all sorts of stuff. So that's really great if you want to go heavy duty. As it is now, the wiki should be friendlier and a little easier to navigate. And I'm certainly looking forward to feedback as the general users start to use it. In the past, with trying people out on it, it's been positive. But we'll see now that it's sort of hit the wall.

00:04:43 [CH]

Awesome. So a link to that as well. And maybe we'll do a follow up episode talking about that kind of restructuring in the future. All right, over to Stephen for your announcement.

00:04:54 [ST]

Yeah, well, this is us trying not to get too excited. But the story so far for new listeners is a few shows back, we had Ryan Hamilton on the show from TimeStore. And Ryan had posted on the TimeStore blog, an article reviewing the future prospects of the q Kdb language. And saying, if I can summarize, that given that the language could only be used with relatively expensive Kdb licenses, and that most users could get most of their work done using SQL or Python with the Kdb datasets, q and Kdb were kind of a language of last resort. And that if it stayed that way, the prospects for it as a language were not that bright. And he sketched out in a subsequent post, his recommendations for where he thought it should go. More recently, we had KX CEO, Ashok Reddy on the show. He came to tell us about how KX has gone back to being a software company. And he spoke very warmly about the company's commitment to the language, and indicated that we might see a community edition. And on the show, we talked to him a little bit about Dyalog's royalty based model in which you can basically use the Dyalog language commercially, and you only start paying for it when you start making some money with it. That was the story so far. A few days ago, Neil Kanungo on his KX blog, sketched out what looks the direction of travel, which incorporates the hints that Ashok was giving us. In particular, a curated library of open source modules, and free tier access to expand involvement and lower barriers to entry. So we're keeping our excitement quiet because none of this, of course, has happened. It's just the direction of travel. If this comes to pass, and means what we hope it means, then the implications for the future of q and Kdb as a language are potentially very big.

00:07:19 [CH]

Yeah, a very exciting announcement. I mean, I would love to have a q executable sitting on my computer that I don't need to refresh. I actually think currently I don't, because I have had two or three in the past, but every 12 months, if the license expires, depending on what you need to do, it may or may not be worth going through the hoops that you need to get it. And so if you can have, similar to Dyalog Limited now, that you can just download the interpreter once and you're good to go, that'll be very, very exciting. And there's been a lot of chatter on LinkedIn saying that this could sort of be the new age of q and Kdb+ if it does come to fruition. So we will be looking for the next update. And if you are interested in reading that post, we'll definitely have a link in the show notes for that as well. And last but not least, a follow up to episode 96, when we were interviewing John Ernest about Lil and Decker. I had expressed my surprise when he had implemented LILA, L-I-L-A, which was an implementation of Lil and Awk. And I got a message from Alex on LinkedIn saying that Conor, just listening to a Raycast episode 95, although was it 95? I think it might've been 96, while on a hike, you mentioned that it surprised you that Awk was used as an implementation language for LILA. This used to not at all be that uncommon. Chapter six of the original Awk book is literally called Little Languages. And then in a subsequent post, he mentions that it talks about all the different steps of kind of building a mini language. So apparently my surprise was not founded. If I had read this book, I would know that Awk actually at one point in time was used for implementing languages. Anyways, he sent me a link and so I will pass it along for the listeners if you want to check that book out. With that out of the way and our other three announcements, I'm excited to introduce Brandon Wilson, who is an APL developer for Dyalog Limited. Not only that, but he was an attendee of the Iverson College back in August of technically last year now. I was about to say this year, but we are now in 2025 and it was in August of 2024. And so I think that means you've heard his voice not once, but maybe twice on the two episodes that came out of Iverson College. And on top of that, we will link these in the show notes. He has given four different talks across the last two Dyalog conferences, which I have actually not seen any of them. We were chatting about this before we started recording. One of them, I think it might end up coming up today, was the Good, the Bad, and the Ugly of learning APL, which I was mentioning. If YouTube had recommended me that talk, which it typically does when I start watching talks from a certain conference, I definitely would have watched that because it has a very nice title and it's a topic I care about. So that is Brandon Wilson. But before we jump into a specific topic, let me throw it over to you, Brandon, because I believe you are a new APL, or a newer, I should say. You haven't been doing APL for the last, you know, since you were three years old like Adám, who's not here today. But maybe you can tell us about your backstory, how you got into programming, and how you found yourself not just programming in APL, but working with Dyalog APL.

00:10:31 [BW]

Thank you for that enthusiastic introduction, Conor. Yeah, so my path to APL started kind of slow. My background in industry is all doing infrastructure engineering. And so I've just, by the nature of that, I intersect a lot of different languages and a lot of different frameworks and constantly having to keep up with technology and different programming languages. And just, I have a natural curiosity for that anyway. And so I would spend time every year to pick up one or two new programming languages. And eventually that just led me into Dyalog array languages. And I started off with J, actually, and spent some time and got comfortable solving toy problems with it. And then in the midst of trying to go deeper, I started reading Iverson's papers, you know, the notation as a tool of thought, and started trying to use array languages in a new way, like to actually build software and see what is this thing that Iverson is talking about, and just hit a blank, basically. And so that actually sort of like flirted with array languages on and off during that period where I was sort of after hitting this wall and thought maybe I should go check out APL instead, since that's where everything started. And so I migrated from J over to APL and eventually came across Aaron Hsu's online talks, where he has a lot, you know, he kind of expands on the philosophy of APL and has a lot to say about what Iverson's promises about programming are. And it seemed everything he was saying was very bullish on the prospects that we could actually, that APL had a lot of new kind of revolutionary ideas about software engineering that we don't find anywhere else. And that was basically what I was looking for when getting into new languages was a new way to think about developing software. How could I use that in my job? How could I use that to provide value to customers and management and executives and things like this? And I hit this wall that I just like, I hit the same wall immediately with APL. You know, I could solve online, you know, advent of code problems. I could solve project Euler problems. Not a big deal, but constructing or finding a solution of iterating or finding, building software that iterates on a solution where you don't have a well-defined problem, which is usually an industry. It's like a vague notion of the kind of product or the kind of solution that you want. And you iteratively aim towards that. I didn't find anything new with APL that I didn't like. The Iverson talking about the subordination of detail. I didn't even know what that meant. It sounded good philosophically, but didn't know what it meant in software. And so I ended up tracking down Aaron Hsu at a conference at Functional Conf in I think 2022 and asked him if he'd be willing to do a mentorship. And he was down for that. And so we spent some time, we started meeting weekly and we picked a project. I wanted to build a YAML parser. [02] Just serendipitously happened to be right down Aaron's path of specialty. And so he jumped on that very enthusiastically and that went quite well. And it got me pushed past that barrier and we hadn't had enough to present at the Dyalog conference. And so Aaron encouraged me to go to that. And so I put a bit of a gamble. I didn't know what it was going to be like. So I put down, it takes a lot of effort to travel halfway around the world. And it's been a week at a conference that you've never been to and are not part of this like new weird function language or the industry at all. But I went there and one thing led to the next. And at that point, I knew that I wanted to go deeper and Dyalog eventually decided that it would be good that I had the skills and the enthusiasm at least to be a full time to APL developer. That's my quick summary of my convoluted path into APL a little bit.

00:14:50 [CH]

Well, so I mean, I'm sure the burning question in all of our listeners and my head is what can one expect to learn from Aaron? Obviously being a disciple like you were, you know, fallen in love with the APL way, but then you hit this wall and then you got this, you know, weekly mentorship. You know, what were the big things that you took away from? Because I imagine you were able to get past that wall with the help of Aaron, you know. Probably it's hard to condense that down into like something that's insertable into a podcast audio only format. But I mean, I guess I'm asking you to do your best and pass along some of the wisdom that Aaron was able to give you.

00:15:32 [BW]

Let me start from just like the experiential, what it's the experiential breakdown of what it felt like to just do this mentorship and try and write an YAML parser in APL. In particular, in the Co-Dfns, kind of Aaron's style of this linear data flow organization. I mean, coming from my background, the natural way to implement a parser would be to take the spec and especially for like a fairly well-defined specification, just take that and you can translate it directly into code. And so there's like for parsing and stuff, there's standard techniques that could be used and languages, you know, using all the problem solving tools that normal languages will give you having, you know, objects and structs and, you know, functional composition and all these tools to sort of break down that problem. And trying to do that, basically at first, I tried to do that in APL and just hated it. Or I was like, it would just felt like I was working against the language. It didn't feel like anything new and it felt like, you know, it would turn into a large thing. I could see it immediately turn into a bunch of code. And it's like, this doesn't look anything like Co-Dfns and it should feel different. And so the process was iterating on that. I would write some code and basically meet up with Aaron and be like, talk about why I hated it. And then he would go into philosophy mode and tell me about like what kind of, try and bridge the gap. I would tell him what I was thinking. And then he would have his way, he was trying to think about the problem and get me to, get me to try and internalize particular techniques and tactics that he was using in Co-Dfns that could sort of break down and get me through the particular, the slog that I was in. And it turned out to just iterate on that by, I would write a bunch, I would write, you know, maybe a hundred lines of APL or something at the beginning and just delete it all and then rewrite it again and delete it all and rewrite it again and delete it all and did that at least 20 times. And a lot of it was just staring at a blank page and I was just, I didn't know what to do. But it was like just mulling on the problem for a long time while I was dozing off to sleep or, you know, traveling around in the trains and things like this. And over time, like these concepts of, you know, having a very, carefully designing your data so that it makes it easy to use APL on the data to say the things that you want to say is kind of like the core concept that, that I feel like I picked up through that process. But the devil is all in the details, right? You know, it's like what, what kind of data structures you can use and how to encode them in just arrays, flat arrays, and what kind of like little idioms you can use on that, on that data and building up your intuition about, you know, something like something that comes up early. I think everyone here can relate to is if you're solving problems, it really helps to have like an initial pass on your data to uniform, like, like doing a make it uniform. And then operate out of the uniform data and then maybe do a final pass to like clean up any extra things that you don't want in it. Instead of having to deal with things like potentially empty arrays or deal with, you know, branches inside, inside while you're in the middle of processing, right? So you'll do like an initial normalization pass, things like that. But like in a more general sense of architecting your data structure so that the APL works very symbiotically with it. That's kind of what it felt like. That's still high level, right? It's not like getting in the weeds about it. On a more nuts and bolts level, what we, you know, things that we got into were about, so since, since this is like parsing is a problem where you just start off with an input and you're not, and it's, it's just a single function, right? It's just start, give it input and it'll throw out something on the backend, but they can give you an output. It's not a continually running server program or anything like this. So it's conceptually you could, you can just have that input just run from the top to the bottom in your function definition and just have this linear data flow and what that looks like. And so we talked a lot about how each, like basically if you have a lot, just a bunch of lines of APL back to back, how do you want to organize that? Right? No, instead of like a thing that a lot of APL enthusiasts I see do these days is naturally congregate towards bags of Dfns and building up like function, like building up a bunch of helper functions or building up a, breaking down the problem into functions and using those to compose other functions that'll, that'll solve the problem iteratively like that. So it'll be a, basically a bunch of Dfns that are defined. And then the solution at the end is like a one liner that combines some of these Dfns that they made together. And that's that kind of like mental model is still in my head, it's like closer to the functional programming paradigm. Whereas like this linear data flow style is much more shoves data in your face upfront. And it's all about not put, not doing any abstraction, don't define helper functions. And making all your choices so that at every single step of the way, all of you have all of the information at hand, right at that point that you need. So it's, it's, it's about data architecture along. And if what that looks like is if you're halfway down your function, you've done some processing on a bunch of data and you have like a new data structure, you have new, that, that data encodes new information because you've done processing on it. And so conceptually it's like you're, you're, you're piece by piece building up more and more rich, a more and more rich data structure. Instead of thinking of you're building up piece by piece functions to operate on some data. So it's still, it's very similar where you're like doing almost type theoretic thinking where you're, you think about the invariance and what each function, but in this, what you like functions would have where you'd have a type signature, but instead you're just thinking about what, what kind of, what is coming in on the front side of the line of APL and then what invariance you want that, that phase or that piece of your APL to encode. And then what comes out on the backside and what new guarantees that gives you. And so your next line of APL can operate on this essentially a new or richer data structure, but it's very, very heavy data first. And that mode of thinking, this sort of data driven design, your cognitive pipeline brings, at least for me, brought these Iverson four points that he has in his notation as a tool of thought paper, you know, where the, what was it? The subordination of detail, the suggestivity of expressions and things like this, that it made it much more tangible for me and able to actually like use it. I felt like I understood what they mean and I could come up with those concepts and like find those concepts in the APL that I was writing in this particular style. That's sort of the meat of what came out for me in the mentorship, if any of that makes any sense.

00:22:56 [CH]

Well, so, I mean, I've got two to three follow up questions and then I will, I will let, I will let the panel weigh in. The first one's a quick one. I did manage by going to the, your YAML parser Dyalog 23 video, I found a screenshot of the code and it doesn't look like all of it, but it is a wall of text. Is it available if people want to look on some like a open source thing, if not, okay, we can just send them to this video. But if people want to look at this, is it available?

00:23:26 [BW]

If it's not in the YouTube description, that might mean you need to be rectified. But yes, it's on GitHub and I believe Dyalog in their notes about the 23 meeting or the 2023 meeting, it gives a link to my, to the GitHub repo. It's called the YAML, D-A-Y-A-M-L. If you just search that on Google or something, it should be one of the top results.

00:23:47 [CH]

Yeah, I think the description of the YouTube video has a link to the Wikipedia YAML page, but nothing. So yeah, you might want to get Dyalog to update that. And so then that leads to the second question. So I mean, the folks that are listening to this and also the other panelists aren't looking at this, but if you've ever seen the Co-Dfns code, it is very representative of this kind of, and I say this with kindness, like a wall of text. Like when you were talking about the DAG of Dfns, I've never heard that before, a directed acyclic graph of Dfns, which is pretty representative of how I write.

00:24:24 [RP]

I think, he said bag.

00:24:26 [CH]

Oh, was it bag or DAG?

00:24:28 [RP]

Like a bag that you hold.

00:24:30 [BW]

I said bag, but a DAG of Dfns also kind of makes sense.

00:24:35 [CH]

A DAG does technically work, because I don't think you can end up with a Dfns with cycle, I don't know, topic for a different day or at a conference.

00:24:44 [BW]

Mutual recursion. You can have mutual recursion.

00:24:46 [CH]

Yeah, I guess as long as it completes. But that is very similar to how I would say I write array language code. I am building up small functions, and if I don't end up with a single line, there are definitely places in my code where that is kind of like the entry point, and it's like boom, boom, boom, boom, compose. So my question is, what are the advantages, because I with caution throw around the word readability, because readability is just a function of your familiarity with the style. But when you have those kind of composed named functions, it can give you a sense, or at least a newcomer to the code base, an English way of immediately being able to read this. And I've always been intimidated by Aaron's style, and I guess the style that you've adopted as well, because it's basically just symbols and single characters, and it's all just kind of on the page. And I've heard the arguments of that you can see it all, so as long as you're familiar with it, the changes that you can make. But anyways, I want to hear from you, because I've heard those from Aaron, is how do you find programming in this style, especially when your experience has come from, you said you're learning a couple languages a year, so you've got a breadth of experience when it comes to programming languages.

00:26:10 [BW]

Sure, sure. So the thing that felt like just experientially felt really new about writing APL in this manner, and actually let me give a quick caveat on the YAML parser, it's still sort of a proof of concept right now. It's like it doesn't have error handling or anything, but what you do see in that talk is the full parser, all of the entirety of it right there. So it's just like 20 lines of code. But what felt really novel about this particular style of coding was it brought back memories of creatively exploring ideas on a blackboard or a whiteboard with some of my other lab mates when I was in grad school. So we would be sitting there, you have the core problem that you're trying to solve, and you're writing equations on the board that make sense to each of you. If someone walks in, even another lab mate just walks in out of nowhere, and they look at the board, they don't have the context to immediately understand all of what you're doing in detail. Now, if they're familiar enough with the domain, they can probably understand the equations and start to get inroads there. But it overall felt very much like that. So in a way, it's very lingo heavy. So it's like, for parsing, if you're familiar with doing parsing, especially if you're really familiar with doing parsing in APL, then I think you could just read this, like the YAML parser, you could just open up Co-Dfns and read it. And it would feel very familiar, I think. And you could start to, because you could just make assumptions about what you expect the inputs to be, what you think you would know, what kind of things you should be doing first. Usually, comment parsing and string parsing happen kind of at the very beginning somewhere, because you have to deal with them really quickly. So they, you know.

00:28:10 [CH]

Actually, as soon as you said that, I did look for, you know, strings and comments, and you can see it very quickly.

00:28:18 [BW]

Exactly.

00:28:23 [CH]

I guess that's the thing, if you know what you're looking for. I was just saying this is closer to being less readable, but then as soon as I got even the slightest of a hint, I was like: "oh, okay, well, that's right there; that's not that hard to pick out". And I'm sure if you did that a couple more times, I'd be like: "oh, well, okay, that's probably there as well." And also, I've never implemented the YAML parser; I know the basics of YAML structure but I don't know what the full spec is. Maybe if I actually was like a pro at YAML and I knew all the ins and outs, I could probably pick out some more of those things just by thinking like: "oh, yeah, what else would you need in order for this to be capable of parsing YAML?"

00:29:01 [BW]

Right, that's been my experience of reading Co-dfns, the APL compiler from Aaron. After writing this YAML parser, I went into it without having notes on the side to explain it for me. Since I kind of knew what should be happening, I just went in there and it taught me actually a lot about how the Dyalog interpreter currently works and I've had a lot of surprises. It intimidating whenever you first look at this kind of style, because it's basically like walking up to a whiteboard with a bunch of equations on it and you have no context. I kind of think of this style as highly tailored for a particular dev style and a different incentive. If you're doing this in a company, what kind of dev team would you need? What kind of problem would you be trying to solve and how would you be trying to solve it? What kind of things are you wanting to guarantee where the style would fit really well? And I think it fits if you're trying to solve and iterate on a problem quickly with a very small team that can move very fast and iterate very quickly; because in that sense, you expect people to become domain experts very quickly. They're exploring the problem; they're building up the concepts themselves. It's not so important that you have very quick onboarding of new members. You don't expect people to come to this and be able to start hacking on it in a day, even if it's just like minor edits or something. Whereas if you expect to have a very large team with a lot of developers; or you have a lot of churn in your developers or something; or it's like an open source project where you just expect people to come in and make contributions haphazard and not have a lot of dedicated time on the project ... a style would waste a lot of effort people just getting up to speed to be able to do anything at all. But it's trade-offs, right? If you do bite the bullet of having a very steep onboarding curve and then everyone is really steeped in the problem space, then you can cut out a lot of the chaff that is trying to communicate to people who don't have that context. So it's like internal NASA discussions or things [03] (if you ever read any of their reports): just full of TLAs (three letter acronyms) all over the place. It's just this wild world. It's really effective communication if you're in on those meetings for like six months. Everyone is just being able to quickly shoot around ideas and there's specific meanings that are depending on the context of which those acronyms happen in. People know that. It feels when working on Co-dfns or something (which is my day to day work now, working on Co-dfns and using it) is the easiest way to represent that concept and talk about that concept is just the variable name. Like one of them is called "VB" because it's talking about "variable bindings". It has a lot of caveats; it has a lot of structure to how we're thinking about what variable binding means and just saying that concept, we give it a new name. That's the power of this kind of style I think; it allows you to very quickly build up your own understanding of a new domain or of a solution space, and then talk about that in an expert way and think about it in an expert way without having to manage all of the infrastructure around making things nice and easy to get up to that point. Does that kind of make sense?

00:33:10 [CH]

Yeah, yeah, it does. Yeah. I will withhold my third question because it's a, I think a good opportunity to go to our panel. I see that Bob has a question or a comment he'd like to say.

00:33:22 [BT]

A question and a comment. So my comment first is, this sounds to me a lot like mathematical theorem proving. You're moving step by step; you've got guarantees at each step about what you're trying to prove. Sometimes you're branching out and doing lemmas which are more involved but in the end you come out with a result that is, if you've followed the proof, you understand it actually at a much deeper level than somebody waving their hands and saying: "it's like this". Is that a similar way of approaching the challenge?

00:33:57 [BW]

That's, that's an interesting conceptualization of it. I think as a reader of this style, if you come in and are willing to go through the learning curve, it can kind of feel like that a little bit because there's a lot of moving pieces and you're reading fairly slowly, one line at a time or pieces of a line at a time instead of scanning large blocks of code like you would in Python or something. So in a way it does feel like that. Writing it does kind of feel like writing a proof, but proofwriting does not feel like a step by step mechanical process and it's a very different way. In practice it feels like an exploration of a domain. You don't know how to solve the problem; you don't know how to prove this theorem and you have intuitive notions that you're feeling out as you move along. My background is in pure math and in academia and stuff and so that process felt very similar. That's why it was striking to me because it's a very efficient way of exploring deep technical issues. And it's extremely lingo heavy and not beginner friendly in a way, but extremely efficient at doing what it does. And it feels the same as that because you're building up all of this lingo and cutting out all the chaff per se.

00:35:15 [BT]

Thank you for the comment on my observation [chuckles]. Second question is if you get partway through this process and for some reason it's not working (you can't have your guarantees or everything breaks down) do you just go back to the last step and work from there or do you go right back to the start and say: "I have to look at this differently?" Or I guess it just depends on the situation.

00:35:39 [BW]

So this is a fantastic question. It's a very deep question, I think. In terms of debugging code and the observability of our systems that we build in general, one of the really nice features that this style has is since it's just a linear flow, if you have a problem at any point in your code, you can cut out everything below the point where you see the problem immediately. Instead of having to do like a git bisect or something. Sometimes you do have to start from the top and kind of scan down but for the most part, if you're deep enough in the issue the problems that you see ... [sentence left incomplete]. So at any point in your code, all of the information that's relevant to the execution is encoded in your data structure. And the entire state of the system is explicitly encoded in that data structure. If you have a breakpoint there, you have access to the entire state of the system. You know how things should have kind of gotten there. So you can put breakpoints and just compare and diff your data, essentially using the snippets of code that you have in your head to know to explore the data, or just using the code directly from your code base, because it's doing the kind of operations that you want to be over your data anyway. That lets you explore and find out where the bugs come from. That's what that's what Aaron and I actually do to find some of these unexpected things that may come up in the data. Some of the tools that we have [lets you] explore your data structure. You can build ad hoc tables out of your data structure. We have like a big tree data; we have an overlay graph data structure on top of that tree; we have some of these things going on in there. It's all expressed as just flat arrays. But we'll take some of that data and just ad hoc create a table and scan the table and say: "okay, well, these nodes, they should be pointing here". You can just see a table that has all of these fields in it and it's like 10 records or something that are just like five columns wide. And you scan that with your eyes, and you can say: "oh, yeah, this kind of makes sense but that's unexpected". You can drill down into that and say: "look, we'll take that one field". And we'll just literally print out the subtree that really surrounding that or something and see if that makes sense. Or you can try and walk through the graph and mix and spread out all of the nodes that you get as you walk from a certain point to another certain point or things like that. It allows you to just organically find things. [It] engages your intuitions very deeply. Essentially, it's steeped in the problem space; they tend to be quite relevant and then just follow those very deeply, which is a very different experience at least than when debugging large, highly readable systems that employ a lot of like good practices because they tend to be highly factorized. A piece of behavior in the system that is unexpected, is by its nature, not something that's usually foreseen by the developers. That behavior arises out of a whole lot of disparate pieces of the system and the only way to observe or to figure out how those pieces interact is just to wind everything back and start at the beginning. It's really hard to breakpoint into [it and] step through the system and get all these pieces interacting to produce the issue the way that you want. Especially if it's threaded or things like this, right? So this ties into things like Observability 2.0. There's some discussion online about doing these things by taking some of these larger [modules], like a server or something, and inserting wide events and essentially taking little snapshots explicitly through your systems and shoving them into a big table database (like Datadog or something) so that post facto, if you have a problem, instead of just staring at graphs and trying to divine issues out of it (or staring at logs and trying to divine issues out of it) you can actually use your intuitions that are fuzzy at first and dig deep into the problem by digging into your database. Whereas the cool thing I think is that this method of APL just gives you that for free. You can build ad hoc tables in your data structure. Your state of system is just all right there.

00:40:17 [BT]

And I guess it becomes really important how you set up your invariants [04] and your guarantees because quite often if it's breaking, that goes back to what you evaluate.

00:40:26 [BW]

That's an extremely perceptive observation; yes. So between these phases of operating on your data (Aaron and I tend to think of it as phases) not everything is just one line of APL. So [there's] little blocks of APL. It'll be a single phase that does a single kind of semantic transformation. We'll think: "okay, the invariants that are coming in here; we'll leverage them in this way". That's what the block will be using and then out the backside, we'll have this new invariant that we can use in blocks below it. Whenever we're debugging, [that's exactly] how we think. It's really interesting how conceptually abstract we're able to stay while debugging instead of just really going down. Sometimes debugging will be just like really going down to assembly and following the bytes at a really low level. It's a conceptually very different space sometimes. But we're able to see something weird in the data and be like: "oh, that means that a variable binding that should be canonical is not canonical or something like that" (that's not gonna make sense unless you know what we're doing). But that should be a canonical binding because it has the wrong kind or something. And even though you won't know the exact chain of APL, or you don't know in detail everything that got you to that particular wrong thing, the high level concepts of semantics being wrong about that thing, trigger your intuition to go like: "that feels really wrong". Then you can zoom in from there to the code that is building up that that concept. Because if the concept feels weird and doing something wrong, maybe whenever you're trying to encode that in APL, and APL is trying to encode that concept, something's wrong there. It's very dumb in a way, but it's stripped down and it's simple but it's very highly honed simplicity in a way to allow those intuitions to engage in the maximum way we can.

00:42:39 [BT]

And since I got a great question, [which was] very perceptive (I feel validated for the day), I'll pass it over to Rich [Brandon laughs].

00:42:47 [RP]

Yeah, a couple of things. It sounds like you spent a lot of time in this kind of mode and in this world figuring out and sort of making it now a very effective way of working in this world. I guess you must acknowledge that it's a pretty niche way of working even within the APL community. So do you have any thoughts about why it's seen as so weird? [Brandon laughs] I mean, I guess there are obvious things like Conor's already mentioned. At first glance, it's a wall of text. I also wonder about the fact that you said you ended up working in a similar space that Aaron was already working in, doing parsing [Brandon agrees]. I guess you hope that this is something applicable to other places, but do you have any thoughts about how to get people into this?

00:43:41 [BW]

Sure. I mean, you're preempting other thoughts that I have here, so thank you for the question. One idea I have about this (and I think Bob and I were discussing this a few days back) but APL itself, and this particular style of APL in particular, starts to shine by the sum of its ... [sentence left incomplete]. It uses a lot of ideas and concepts and puts them together in a way to allow this kind of dilemma to happen. The whole is greater than the sum of their parts and that gap between the sum and what you're actually getting out is very big. If you try and take it piecemeal, every little piece is very easy to argue against: like high degree of tersity. It's very easy to come with arguments why that's not applicable or doing everything with a very hard line, data first, linear data flow style. It's easy to argue why you maybe not want to do that. But if you take all of the different concepts and shove them together in a bag, then it starts to sort of take off. I think that makes it hard. You have to sort of drink the Kool-Aid in order to like it to actually work, right? I think that is a big barrier for making it for adoption in general.

00:45:05 [RP]

Yeah. And I'm thinking as well. It's probably hard to have just a sip of the Kool-Aid, right? You can perhaps solve one of your toy problems, your advent of code or something, and maybe attempt to do it in this style, but maybe you'll just get a couple of terse lines or you'll get the thing (Stefan Kruger, I think in his book mentioned [it], and he's mentioned it in person) how he got into APL as he saw someone's advent of code solutions that was one page for the whole month.

00:45:36 [RP]

Yeah, yeah, yeah, yeah [chuckles].

00:45:38 [RP]

[He] finds that really impressive and stuff. And maybe you'll get a piece of that but maybe you won't see what you're describing, which is when you're deeply tackling a sort of larger problem, being able to zoom in and out; being able to, at any point freeze and see the sort of entire nature of the system within your representation.

00:45:58 [BW]

Exactly. Whenever I was doing this mentorship thing with Aaron, I tried to mimic the style, right? You know, like say: "I'm going to do data first design; I'm going to just start from the top of the bottom and like do one line at a time". I'm going to have single name variables. I tried doing this, but there are a lot of these micro decisions that really impact the ergonomics of how you go about writing APL, like the later APL. How you get to concepts. Do you want to reify a concept by creating a new field? A new variable that is a flag: is it a different kind of type thing? Or do you encode that by reappropriating one of your other fields? Or do you even need to reify that directly as a piece of data, or can it be already gathered by tweaking a little bit two pieces of data that you already have, where in combination, they can get at the third concept. There's a really a whole lot of data type or data structure engineering that goes on. And that's the primary focus of what it feels like to use this. [Which is] what makes it hard to build up the intuitions for it, I think.

00:47:18 [RP]

Right. That's more like ansatz; you're just trying a solution. [You] have a sort of guess. Do you have a lot of guesses and then you sort of follow them through and see how far they get you?

00:47:28 [BW]

It really tends to be thinking about your intuitions about how the problem behaves conceptually. And then Aaron and I will down and just talk about the problem for several hours, at a very high abstract level. One thing we're dealing with right now is scoping issues [05] and how they relate to namespaces. If you have a function like "ns.f <- g" or something and you have free variables in g. What happens to those free variables in f whenever you defined it. Does it grab it from the definition side or does it grab it at the execution site? Or something similar with a tradfn that is inside an op: how do the free variables behave? There's subtle issues like that. We're talking at that level. We're not talking about APL expressions or things like this; we're talking about these concepts because in our head, we have APL expressions that just say that concept or say the pieces of that concept that we need. Developing that facility does take time. It's that fluency in the lingo that takes time to build up. That gives you this sort of superpower in a way, it feels like a little bit, but also makes it really hard to get there.

00:48:54 [RP]

So [chuckles] on the idea of getting people from a lot of text to slightly more familiar or willing to try these ideas, Asher Harvey Smith was an intern at Dyalog [over] the last couple of years. And this year (yeah, he's great [laughs]; all our interns are great) he wrote a series of tutorials, or sort of one tutorial (attempting to break down for the mere mortals) the tree wrangling structures; the idioms, the phrases that are used in things like your YAML parser and in Co-dfns.

00:49:46 [BW]

So called Apter trees, I think, right? Yeah, yeah.

00:49:49 [RP]

Yes. Apter trees. I think it's really great. It's a nice introduction; it actually holds your hand a little bit. But if you look at it, you can see that he's clearly taken a much more bag of dfns style. He's named things [Brandon agrees] with actual names. The whole thing looks a lot more familiar at first glance [Brandon agrees again] than the YAML parser or Co-dfns code bases. What do you think of that?

00:50:20 [BW]

So I think it's one obvious [approach]. People will be kind of dubious whether APL or array languages are capable of all of the feats of other languages that have a really rich set of primitives for defining data structures, right? Trees are kind of like at the far end of the limit, away from arrays. Right? They're in a way, they're the most general data structure that you can have, algebraically or something. I think it's actually quite brilliant just to introduce and say: "hey, look, your syntax in your language does not have to mirror the cognitive way you're thinking about the data structure". It doesn't have to be one to one there and you can break them apart and show it just the way that Asher kept everything in very familiar land and said: "hey, look, you can use flat arrays and it contains all the information that trees do". It's actually pretty easy to use them. Just getting that message across, I think is quite valuable and was one of these big breakthrough moments for me to make that separation in my head between syntax and data structure. And the presentation style is great. It's a very approachable; Asher's voice kind of comes through.

00:51:50 [RP]

Is there still a gap for going from there to the wholesale Aaron Hsu [or] Whitney style terse [style]?

00:52:01 [BW]

Yes, there is. There's a large gap, right? And I mean, that's part of the pain points here is. Using Apter trees, I think I don't have a lot of experience using them in different styles of APL, so I couldn't comment much on that. But I would be interested to see other people trying to leverage them. I think that this linear data flow style is more about developing fluency of using APL; so developing fluency in designing your data so that you can use your APL to get at specific concepts that are relevant to the domain. I know that sounds very abstract.

00:52:44 [RP]

And that's as primitives?

00:52:47 [BW]

Kind of as little tiny expressions, right? So if you're dealing with trees, things that you want to think about are like insertions or deletions or getting subtrees or reparenting something; things like this. And all of those little expressions are already cached in my head. I know how to do that.

00:53:11 [RP]

Exactly. You haven't developed an insert function.

00:53:15 [BW]

Exactly!

00:53:22 [RP]

You haven't got a DSL for working with these tree structures. You've got a little library; a mini APLcart of tree manipulation ... [sentence left incomplete].

00:53:26 [BW]

Almost. Almost.

00:53:27 [RP]

Also a phrase that's been thrown around recently in some discussions is an "APL Soup".

00:53:35 [BW]

Apple soup [laughs]. It's kind of like that, right? It's interesting how that gets into territory, like your understanding of the APL phrases and what they can mean in different contexts. Like the phrases that we use a lot of times will be just more general, like concatenation, right? Concatenation is just concatenation. It can mean all kinds [of things]. It can mean pretty much anything you want it to mean. But if you're concatenating on the end of a parent vector, that usually signals that you're inserting a node. And so it's simple, just like "concatenate gets". If you're reading this kind of like Co-dfns, it just screams out: "oh, at this place we're inserting a bunch of nodes", so we expect other fields to get concatenated onto or inserted into in different ways and things like this. It's sort of this symbiosis between having those phrases available in your head and knowing the possibilities of those phrases. Then instead of immediately reaching for a cover function, because you have an expression that in your current data structure, to get at the thing that you want to do (if it's like kind of long and clunky) instead of building a cover sub function for it, you can flip that around and say: "well, no, my data structure just needs to be built better". And you can start asking: "what data can I put into the data structure to make it easier to say the thing that I want to say; to make this long phrase that is clunky, to make it really simple". Or instead of adding a new field, I can maybe merge two fields or add slightly different semantics: instead of having all positive integers, what if I just let some of them be negative and that would mean something different. And then instead of having to mix fields, I could just do like an absolute value thing and that would let me get a whole class of things together at once. There's always all kinds of techniques like that. But it's kind of like just saying: "I'm never going to do it". In your head, to learn this, it's really nice to just sort of play taboo and don't allow yourself to do any cover functions and just say: "how would I design the data to make everything not painful"? And really listening to those intuitions; when something feels dirty or painful or hard to get at, that's telling you that something's not ergonomic. You can start asking your questions: "how can we make it ergonomic around that?"

00:55:54 [BT]

So again, an observation and a question. First observation is I've heard Adám talk about the fact that APL is like a diamond. [Brandon agrees]. Everything fits exactly how it needs to fit. It goes back to what you're describing as the whole is greater than the sum of the parts. You take every part: "you go, I'm not sure where that ...", and you put it all together just exactly the right way. And it works and it's dynamic and suddenly you have a tool that is flexible and predictable and all the things we gain from APL. I'm wondering if when you're talking about a problem space (like a YAML parser or Co-dfns) whether you're taking it up a level. So you have to go through all the same arguments of finding all the same exact spots to fit it around exactly. And when you get that, now you have a really powerful tool. It's simple, but it has to fit exactly together. And that's the work you're doing in that next stage to come from the primitives up to what you've developed.

00:56:56 [BW]

00:56:56 [BW]

That's an interesting line of inquiry. I honestly, I don't quite vibe with it. I mean, I think of like the design of the primitives is kind of being like that where they were like honed over decades to get to the point where they are. And there's a lot of experimentation in what the primitives, what people thought they should, you know, what they should be. And there's, you know, well-known design decisions that are kind of regretted these days and things like that. And it feels almost like more human to me, actually. It's not like someone designing things in just the perfect way that they all just happen to slide together very comfortably in this nice way. It really feels like an externalization of just your conceptual framework. And what's interesting about reading Co-Dfns is like, when I first read it, I was surprised that I could see a piece of code and be like, this is like the code, the way it just felt like an ad hoc thing that was just pasted there temporarily. And it was like, Aaron, here, you're obviously cutting corners. And it seems like you're doing that because you've said, I just need to get past this point and I'm going to paste that in later, like the real solution. And he was like, oh yeah, that's exactly what I tried to do. And like those kind of human level design decisions have, I find, communicate very well in this style because of its fidelity for making, like it allows a whole wide berth for making design decisions about your data and the kind of APL you use instead of adhering to, whereas if you like adhere to a more Dfns style, where it tends to be a composition of functions, at least for me personally, like if I read the code, even namespace or something, it's harder to understand where those decisions were made or where something's like, like the design decision is just not as transparent to me at least.

00:58:57 [BT]

And I'm guessing that's because there's a linear flow. Right. And if at some point you've made a decision to step off the path and come back on again, you pretty, it jumps out at you.

00:58:59 [BW]

Right. Like there's this, in Co-Dfns, one of the, I think the most hilarious one is there's just literally a single line that deletes all control structures. Like we don't right now, we currently are just ignoring control structures because we know that they're not going to be too problematic to add in the data. It's not, it's not the big meat that we're dealing with right now. And it would just be too fiddly to put them in. So at this point, so we just, we just literally just delete them, but it just stands out so strongly because you're doing like before it, you're dealing with like some really intricate, you know, manipulations of like inserting like different kinds of data into the tree. And then after it is doing the same thing, but then it's just one line, they're just simple deletion. And you're like, why would you be doing that here? Right. And those kinds of, that I think is one of the really nice, like, that's one of the tools that you see in, or practicing mathematicians will, will do as well. They'll like do something in a really dirty way where you're using, like, instead of using a really nice algebraic backing, you'll just use matrices here and you know, it's going to be a bunch of ugly calculations, but you just crank through it real quick just to get, just to get your head around the problem. And it works. And you have a strong, you know, that the things that you're using, the thing, the concepts that are backing up your explorative tools are very solid. And so you can move forward with confidence, even if you know that they're not particularly, they're not exactly appropriate to how you want to express it in the end. It's that exploration possibility that is, that is there and very potent. And it feels like that to me in doing this in this particular style of APL.

01:00:42 [BT]

And your answer brings me back to my question, which is how is it for maintainability? I would guess if you have that depth of understanding, maintainability isn't really a big problem.

01:00:52 [BW]

It's yeah. I mean, we've, we've on the short amount of time that I've been basically, you know, doing this professionally with Aaron is like six months now. Right. We've made some fairly major changes in Co-Dfns. You know, like before they didn't have, had no Tradfn support. Now it has Tradfn support. And that was a very non, non-trivial change because Dfns are like scoped, Tradfns are dynamically scoped. [06] And that adds a whole bunch, like the interaction there is just extremely, you know, extremely subtle and figuring out how to exploring how that affects your system. Took some time to get right. But we were able to do that because of this style allowed us to sort of just play with the problem and be confident that that whatever changes we introduced could easily, easily be backed out or, and we could easily explore how it affects our data because, you know, inserting breakpoints and stuff and all of that kind of observability stuff comes on the table. All of that came together, comes together to make, in my experience the maintainability just off the charts better than anything that I've seen in any of my previous work of, you know, 10 years.

01:02:09 [BT]

I guess the one place where it would be a challenge would be the old win the lottery issue. How well, it would be difficult to bring a new person in to maintain something. There'd be a long onboarding process, but I guess the advantage of it, if you're working in a domain that needs to be structured, it needs to be resilient. That long process is worth it because you've got something that can be changed, can be adapted, but the internship is going to be a process for anybody new on the team.

01:02:40 [BW]

Exactly. It's, that's, that's exactly one of the, maybe the biggest downside you could potentially say or it's the biggest pill to swallow with this kind of thing is that the onboarding is huge. And it's, it's not just about like getting to good review with the style. It's about, it's really about honing the intuitions because it's really easy to screw it up, I think. Where you can, you can shoot yourself in the foot where like a little decisions here and there will accumulate and avalanche into just making the designs down the road really difficult. Where like, we'll, we will cognate about how we will, like the problem we're trying to solve now and we'll think about particular solution that we're doing and how that will affect other ways that we go about making solutions down the line. Like that, that kind of thing is easy to, it's easier to cognate about in this and honing those intuitions takes a long time and I'm still working on it. But if you, like I said at the beginning, if you wanted to have a huge group of people with a lot of churn, like an open source project with just piecemeal contributions, I think this would be a very challenging model without it already being part of, if it was in the Zeitgeist and everyone knew about this, then it may work. But if it's not, it's just, you've got to have that onboarding.

01:03:57 [CH]

So we've been talking, I mean, I still have that question that's waiting in the wings, but since, since the conversation that we've had since then, I now have a new, more pressing question or a question that I'm more interested in asking, I should say. Not that I'm not interested in asking the first one still, but we've talked a lot about the style of kind of the APL code or array code that you're writing. And I wonder if you have thoughts about just the, I don't want to say proliferation of styles, but just the different styles. So like in my head, there are a few different styles and I think it's, you know, something that only is apparent if you are in the array language community. I think everyone outside, they just see symbols and they're like, ah, like I literally tweeted out your YAML code or whatever the word is on Mastodon. And then a couple of the reactions were like, what? Is the polite way of putting it. But I think it's a misunderstood thing. And also I wasn't, didn't really realize it. I kind of thought that like the APL or array community was in agreement of like, you know, the way that you write code. But really since having, you know, been a part of the community for a few years now, it is very clear that not only is there not agreement, but there is the variety and like the distance between the styles is like so far. Anyway, so I'm just interested if you have thoughts on like, are there like styles that stand out? Is like, cause I think Co-Dfns or the Aaron style or whatever you want to name it is definitely one of them. It's this, you know, dense kind of try and keep everything on it. And you might, I know Bob, you mentioned Arthur's name earlier. That's his kind of style as well. Everything's on one screen, even if you are building up functions. And then we've talked to other people, like when we talked to Josh David, who used to work with Paul Mansour, they have like, he talked about their defunds only style and they take it to the extent that, cause I've looked at some of the source for their like Excel tools and whatnot, where like every function is in its own .aplf file. So like they have a lot less comments than anyway. So I'm interested to get your thoughts on like the different styles that are out there cause you've probably written and been exposed to a lot more APL code than I have.

01:06:25 [BW]

I wish I had, I mean, I've thought about this and wish I had more depth of experience with different types of APL. I mean, off the top of my head, apart from this linear data flow style, I can think of like a very, with some of the code that I'm dealing with these days, the Co-Dfns is like churning on. It's a bag of trad, it's like a bunch of Tradfns and the code tends to be kind of control flow heavy. So it feels imperative in many ways. They have like the more imperative kind of style, this bag of defunds, very functional kind of style. And then you can have code that feels almost like BASIC.

01:07:11 [CH]

Basic as in the language BASIC or basic as in the adjective?

01:07:15 [BW]

ANo, no, like the language BASIC, right? I'm not digging that hard. You know, you have with like labels and branches and it's interesting to me how they signal different ways of, and they signal different problems that they're trying to solve in sometimes. And then also the different incentives that are involved in like bringing that kind of code about. So like if you see like branchy code with like the right arrow and labels everywhere, a lot of that that I've encountered now tends to look like a state transition kind of program. It tends to behave like that. And it's easy to, in like having state transitions with branches and labels is very straightforward. It's like a direct encoding of a state transition or a state machine. And having like very Tradfn heavy code is, and with a lot of like an imperative and Tradfn heavy kind of thing, tends to, I think it tends to come about in a like a Mel Conway kind of way where you have different organizations that have to communicate. Is everyone here familiar with like Mel Conway's, "How Committees Invent" paper? Like the 1950s, I think. It's a brilliant like four or five page paper, how committees invent. And it makes, it's basically makes the observation that the structure of software systems tends to like the architectural organization of their systems tend to mirror the organization of the committees or the organizations that built that piece of software. So like if you have a compiler, for example, I think it brings up some example where like a compiler was built and it's conceptually broken into like three large phases or something. And it just turns out that each of those phases were the domain of separate teams that were building that. And there's all kinds of examples in the paper that it tries to point out saying, Hey, look, the structure of your program just mirrors the fact that if you have an information, a boundary, a communication barrier between people, then they're going to insert a barrier in the code that acts as a way, a media, a channel of communication. And so like having Tradfns code, I think, or having like this procedurally or yeah, procedural, very highly procedural code tends to also introduce a lot of sort of API definitions and things. And that tends to mirror like the organ, that kind of organization boundaries and, and a lot of historical crops that comes in through that. It's kind of how I feel about it, but I mean, I don't have, it's much deeper than an observation than those kinds of kind of intuitions around the different styles.

01:10:17 [CH]

Do you know if the Tradfn, the bag of Tradfns code, because I know that the bag of Dfns code [07] you can find online, like I think literally the URL is Dfns.Dyalog.com or something like that. And you can, that is what I think of like is the canonical kind of Dfns style where, and it actually looks quite nice is like you've got, I think maximum two, two character variable names, typically one. But sometimes the Dfns programmer likes to get a little crazy and add a second character that that variable name, but there's always a, almost every single line has a lamp comment that explains. So it's, it's not that you're dealing with you know, unreadable, like what does the letter L stand for typically the comment to the right. And because APL is so, so terse, you can fit like in kind of two columns, the code, the comments. And because you're reusing those variables, like it sounds like, well, why wouldn't you just name the variable, the thing you're putting in the comment? And it's like, well, because it enables you to do you know, terse writing of reusing that thing. Whereas if you have to reference it six times, it's like, well, now you're spelling out this 10 character, 15 character thing. Anyways, so you can find that stuff. Is there examples of the bag of Tradfns code that we can, or I was going to say we can link the listener, but I'm actually just personally curious for myself to go look at, or maybe Rich, you might know as well, because you guys both are exposed to APL code.

01:11:47 [BW]

I know Kamila has shared some slides with with code that's kind of in this direction, but it's fairly limited in scope. She's like, she's written some explicitly procedural trad, you know, algorithms to solve some problems. I think she showed them at IC last year, at Iverson College.

01:12:05 [RP]

I think also, I don't know enough about the specific how it works to say, comment on it much, but if you look at the Jarvis source code, which is a, it's a web service server in APL. That's a bunch of Tradfns and it's quite a control structure heavy.

01:12:24 [CH]

Sorry, when you say Java source code that.

01:12:25 [RP]

Jarvis, sorry, J-A-R-V-I-S is a, is a.

01:12:29 [CH]

Oh Jarvis. I was like, you're not trying to tell me that the Java language is implemented in APL. Are you Rich? Cause you're not going to get that one by me.

01:12:38 [BW]

Yeah, little did you know.

01:12:43 [CH]

Top 10 language in the world implemented in.

01:12:46 [RP]

I mean, the thing is, you know, as a, as an APL user of Jarvis, someone who's like written little web services with it. What's great is I never have to look inside of this. It's super like, um, it's very satisfying to be like, I have, I have my APL code, my function, and then I just put this thing on it. And then now I can send it web requests and it's, and it kind of just works in a way that you think should. I think if it came to talking about or trying to, cause it's very expecting everything to come in as JSON. Um, and we don't have to, but it's kind of expecting everything to come in as sort of JSON things, lists of JSON objects, typically, typically. And then they come in as these namespaces. And then once you get into the namespace representation of JSON, now you're talking about the kind of, uh, I think you were talking about Brandon where you've got this, it feels at first glance, sort of very intuitive representation of a nested tree structure. But then the code you have to write for that is very, I'm hopping over this piece. I'm hopping over that piece. I'm explicitly usually like a guard or an if statement checking, is this the case? Is that the case at like different levels?

01:14:00 [BW]

You can't, and you can't easily traverse that tree if you need to, like dynamically,

01:14:12 [RP]

Like if you wanted to do, um, if you want to, you know, list out all of the, I don't know, like all of the, again, at the top level. And if you've got very simple things you can in a way that's actually awesome. It's like, you know, the fact that you can have an array of namespaces and then when you dot into the array, as long as those namespaces all have the same names inside, which is very typical for like a web API thing, um, then you get the array of values for each of those things out. Um, so there, you know, yeah, the surface level is, it's very, very cool. But I imagine, well, I do know that you hit, you know, say you've got a missing value or this one in this case is slightly different. You hit some kind of edges that you end up having to work around.

01:14:53 [BW]

Yeah. I've been trying to think about that particular problem or the particular problem of what JSON manipulation looks like, JQ style, sort of JSON interaction [08] looks like if you did it in this Apter Tree style. And I'm not, I haven't completely fleshed out my thoughts on this, but I think that you're completely, you're, you're very, like, I completely agree that it's very nice whenever you have a very regular structure to, you know, beforehand, um, you know, that you have just a bunch of JSON objects essentially, you know, and you, and they're all uniform structure and you can get, and you can get into the magic. Um, on the other hand, I think if you have such a regular structure, if you're doing APL, just shove that into a table and it's way easier, right. Instead of having to like repeat..

01:15:41 [RP]

Or sometimes I'll be like, find the name, you know, check the name class of the name in this list of objects for each object, and then filter out the ones that don't have that name so you can all stick in a default value for those or something like this.

01:15:54 [BW]

Right, right. Which in my opinion, that's kind of painful, right? I've done it and I kind of don't like that kind of thing.

01:16:02 [RP]

The worst one is when you, when you. When I was first doing it and then it would be like Oh well, I can't use the dot think as one of the names. One of the one of these objects is missing the name. So I get the value error. So what I do instead is I make a little diefen and loop each on the put each on each on each of the objects with an explicit either even worse error guard.

01:16:11 [BW]

Right.

01:16:22 [RP]

Right. And then after it's like, hold on, I can just get the name classes for all of these and then filter out. And it's a little, like a little bit nicer, but you're right. It still feels, this is, I think this is like a hint into starting to get those intuitions of what feels a little bit, a little bit goopy, a little bit.

01:16:45 [BW]

Yeah. I mean, I'm going to say feel that strongly. Please. Like, I think like it works really well if you have really, really uniform data up and you know, you're very confident that it's going to be, have the structure that you need to get at. If it starts to get ragged data where you have a bag of a bunch of different objects and they have different structures to them, then you hit these kinds of issues, but everywhere all the time. Because if you have, what this would look like in an Apter Tree style is if something's missing a field, it'll like, the type will have a particular, it won't, it'll have like a zero for the type or something. Or and you would just, you would just filter that out with a mask because you don't care about it. And in the same way that like you'll pre-process input, like if you have a string and you need to split it on spaces, you'll just tack on a space on the front, which normalizes it so that you can do the standard trick, you know, generate a mask out of that and split, right? Things like that. I think the, like having all of those APL tools that we have, then we know how to operate over arrays. If you try and do this with namespaces, you're like in a completely different land and you have to have a whole new set of tools that have analogous kind of operations conceptually in many ways, but they're just actually not there. And you have to kind of build them up out of your own, you know, with Dfn, with cover functions and Dfns like this. And I don't know, I think that's kind of painful and a lot of unnecessary work.

01:18:19 [RP]

Yeah. On the flip side, I think it has been like, because of the convenience and the ease of seeing it, I think it's been useful for a lot of people who don't care about this stuff and just want to get, just want to get some stuff done and they just, and they'll write these things and maybe, you know, it works. So they're not bothered.

01:18:37 [BW]

This is why I really appreciate that Asher took the time to write his Apter Tree write up. I mean, I think that's the first step of if everyone like was really familiar with ApterTrees, then people wouldn't be reaching, I kind of suspect they wouldn't be reaching for the namespace representation as much. There are some cases where it might be appropriate, but...

01:18:55 [RP]

And right now it's definitely the default.

01:18:57 [BW]

Right, right. I mean, yeah, you have to put on an extra option to get the the matrix output on right, like a variant option, but I I I feel I really just think that boils down to familiarity and because everyone like outside.

01:19:03 [RP]

To get it, yeah.

01:19:13 [BW]

Yeah. But I, I, I really just think that boils down to familiarity and because everyone like outside of APL, the syntax is dotted notation for accessing it's like, or, you know, exalt or kind of things like that. It's that same mental model everywhere. And so, yeah. And it's pretty painful to use that to do complex queries on, on, I mean, like, have you read Jake, like really come fancy JQ code. There's like JQ implemented in JQ. Have you seen this? It's called JQ, JQ. And it like, you can do really fancy stuff using this particular model with like digging into a tree by having path notation and using filters to this. And it forces you into a lot of places where the performance is pretty terrible. And that, I mean, that affects the usability, I think, and where the after trees are like, one of the really nice things about the after trees is you don't, you won't have to be looping, like looping over a really large JSON, all the nodes in a really large JSON file would just be in this dotted node, you know, dotted namespace notation is horrendously slow. Where it would just be instant. You know, we have, we're dealing with like 1.5 million nodes. And all you know, in the current trees that we're dealing with in the current day to day work, and it's just like everything is instant. Unless you're, unless you're doing like really complex traversals over the entire tree. But most of the time, you don't have to do that. Whereas if you have the dotted notation, you're always having to do that. The actual thing you're working on is mostly vectors of numbers. Exactly. We I mean, we explicitly make that decision. So all if you have things that would normally be strings, or like keys, or something, you just have a, you just do intern string, index of selfie, or what might be thinking of something else we make a bunch of IDs for Yeah, exactly. Right. You have you have like a string table, and then just indices into that string table. And so now instead of dealing with string, you just have integers, single integers, things like that. So those are some of like the low level simple things that you can do for your to make your data structures really easy to use and fast. It's really nice to be able to just write equals whenever you're thinking in your head, I want a string to be equal to this other string, right?

01:20:59 [RP]

Our index of selfie. Oh, I might be thinking of something else. We we make a bunch of IDs for.

01:21:00 [BW]

Sorry. What what?

01:21:05 [BW]

Yeah, exactly. Right you have you have like a string table and then just indices into that string table and so now instead of dealing with string you just have integer single integers, things like that that those are some of like the the low level simple like things that you can do for your to make your data structures really easy.

01:21:23 [BW]

Easy to use and fast.

01:21:26 [BW]

It's really nice to be able to just write equals whenever you're thinking in your head. I want a string to be equal to this other string.

01:21:31 [BW]

Right.

01:21:31 [BW]

Instead of having to do like weird.

01:21:33 [RP]

Instead of having to do like weird match this and each includes one of them.

01:21:37 [BW]

Yeah, match each and then like, so it's so ugly.

01:21:41 [CH]

All right. Well, we have blown by the hour mark. And almost in depending on when I started the recording, we were approaching the 90 minute mark. So it's, it's probably time that we start to wind this down. But this has been absolutely awesome. Thanks for coming on, Brandon. And I mean, it was a yeah, it was a blast also getting to meet you in person back at Iverson College. And hopefully, we'll get to bump into each other, if not at an Iverson College at a Dyalog conference in the future. I hope.

01:21:58 [BW]

I hope sooner than later.

01:21:59 [CH]

Yeah, yeah. We had good times. I think Dyalog, I'm not sure if it's announced. I think it was announced. Maybe, maybe I'll be safe. But there's definitely Dyalog events happening this year. So but I'll save that for the Dyalog people for when the official announcements are made. But yeah, hopefully, we'll be able to bump into each other in person. If listeners have questions, thoughts, comments, they can send them to

01:22:33 [BT]

Contact@ArrayCast.com. You almost caught me sleeping there. It's because it's early in the morning here. And I've, you know, been on the edge of sleep. Not for the interest, but just for the fact that, well, I had to get up early. And Brandon said to stay up late. But anyway, Contact@ArrayCast.com. A shout out to our transcribers for all the great work they do. And to you listeners who get in touch with us and send us emails and everything. Contact@ArrayCast.com will be working. Twitter will not. Bye bye.

01:23:08 [CH]

And yeah, for those that are listening and that their interest is piqued, definitely check the show notes. Because we'll make sure to not just link to the YAML talk, but Brandon's other talks. And also, I managed to find the GitHub repo for the YAML code. And we'll also try and link to some of the other styles of code that we mentioned. Because I think it's, yeah, it's a less discussed topic is that there's, I don't want to say disagreement, but there are just different styles. And that just because you see the Co-Dfns style or the Aaron style, and you're like, oh, that's not for me. Similar to a lot of other languages, to be honest. I mean, the more modern ones tend to have single formatters. But C++, for instance, has a formatter that you configure yourself. So you can make it look the way you want to. I think APL and array languages, they fall into that category a lot more. Anyways, once again, thank you so much, Brandon. Hopefully we'll be able to do this. And I guess, yeah, my final question, we never got to. So we'll save it for part two of this conversation to happen maybe in the triple digit episodes, if you're willing to come back. Yeah, we'd love to have you.

01:24:15 [BW]

I will look forward to that. Thank you very much.

01:24:17 [CH]

Awesome. With that, we will say happy array programming.

01:24:27 [ALL]

Happy Array Programming

01:24:29 [MUSIC]