Transcript

Transcript prepared by Bob Therriault, Adám Brudzewsky, Sanjay Cherian and Igor Kim.
[ ] reference numbers refer to Show Notes

00:00:00 [Michael Higginson] 

Say you were making boiled potatoes and you know the traditional way to do it would be to like you take your potato. You wash your potato. You peel your potato. You boil your potato. You grab the next potato. You wash it, peel it and boil it. And then I said, you know in q and other array languages you would wash all your potatoes, peel all your potatoes, boil all your potatoes. And he said, ohh great, you know that's that's I get that that's really helpful. And then I said, but wait, there's you know there's this there's this other level but I don't think I'm at it's like I'm striving to get there that's why I keep pursuing this is like there's this other level where you know you boil your potatoes and you filter it for skins and dirt. 

00:0:52 [Music Intro] 

00:01:02 [Conor Hoekstra] 

Welcome to another episode of ArrayCast. I'm your host Conor and today with us we have a special guest who we will get to introducing in a couple of minutes, but before we do that, we'll go around and do brief introductions. We'll start with Stephen, then go to Bob, then go to Rich and then to Marshall. 

00:01:16 [Stephen Taylor] 

I'm Stephen Taylor, an APL and q programmer. 

00:01:19 [Bob Therriault] 

I'm Bob Therriault, a J enthusiast. 

00:01:22 [Rich Park] 

I'm Rich Park. I'm a APL teacher evangelist at Dyalog limited. 

00:01:27 [Marshall Lochbaum] 

I'm Marshall Lochbaum. I'm a former J programmer and APL developer and now the BQN developer. 

00:01:33 [CH] 

And as mentioned before, I'm your host Conor, research scientist at NVIDIA - C / other languages, but huge array language fan and we will get to introducing our guest who I'm super excited to talk to just after a couple short announcements. I believe we have three from Rich then one from Stephen and then one from Marshall. 

00:01:52 [RP] 

Yeah, so I'll go in order from closest to now to furthest away. So those of you who know or don't know Adám and I have got another podcast because we can never have too many array language podcasts, but we've rebranded as "The APL Show" [01] so you can go find out about that on APL dot Shows. And yes, the availability of that domain name was part of the decision to to call it that and and it turns out in searching actually, we're now on a bunch of other services that we finally figured out how to get the RSS feed coming out of GitHub page. So we're on, uh, Spotify Simplecast Google Podcasts, Samsung Podcasts, iTunes, Apple Music. It's unbelievable, isn't it, Bob? We managed to do that. 

00:02:36 [BT] 

Rusty's joined the club. It's a podcast! 

00:02:39 [RP] 

There's actually a podcast, but we're calling it show because we're contrarian, so but that's available now, and the latest episode where we ramble about some sort of higher level stuff about notations for describing structures and diagrams and things like that is out of on apparently everywhere now, everywhere where you can listen to. 

00:03:01 [CH] 

What's what's the name? 

00:03:03 [RP] 

It's the APL show. 

00:03:04 [CH] 

The APL show. 

00:03:07 [RP] 

APL dot show. 

00:03:08 [ML] 

Conor's not happy.

00:03:08 [CH] 

I'm literally on my cell phone right now searching for this on my my app. 

00:03:13 [RP] 

I mean literally, this is like yesterday the day before. 

00:03:16 [CH] 

So it might not have. It might not have updated. 

00:03:18 [ML] 

This is big news. 

00:03:19 [RP] 

Like I've gone on iTunes that I have to like filter down to podcasts to find it. You know it doesn't come up in the... there's too much detail about the other podcast. 

00:03:30 [BT] 

For the listener though, by the time you hear this, it will probably be out because that'll be another three or four days before it will be on. 

00:03:37 [RP] 

OK yeah, but going further out into the future March 12th there is... There's not been one of these in quite a while, so listeners might not be aware of what this is, but Adám used to host a fairly regular sort of live zoom panel with various figures from array, language, life and history called the APL Campfire. And there's going to be a special return of the APL Campfire on March 12th with Norman Thompson, who is the author of APL programs for the mathematics classroom and the co-author with Ray Polivka of the book APL 2 In Depth. So we'll link to the APL wiki page about that in the share notes, no doubt. And then lastly just another shout out for March 22nd. That'll be 10 days after. APLSeeds 2023 online meeting, which is an event hosted by Dyalog to enthuse people new and prospective APL users. People who might have heard of APL but not really know much about what it's about, come see a few talks and talk to a few APLers. The sign up is not ready as of speaking right now. Hopefully by the time episode comes out. Otherwise there will be a link to the Dyalog web page which will be updated with registration information as that happens. 

00:05:02 [CH] 

So many things I just learned about wow alright Appleseeds 2023 the APL Show and Campfire, of which we will talk about after the show you should put that stuff on your podcast. Alright, over to Stephen though. 

00:05:13 [ST] 

Well, we've we've fantasized on the on this podcast about Arraycon [02] - a conference for the array programming languages and that remains a fantasy, but this year in there will be a KX con and it's in Montauk NY 17th to 20th of May. I've got no details for you at this point. I'm just forecasting. MASSIVE!

00:05:41 [CH] 

Alright, awesome, we'll look forward to more details from that. Man, we're getting all all the announcements that I didn't know about alright. And finally over to Marshall. 

00:05:49 [ML] 

Alright, so my announcement you may know that BQN is uses a self hosted compiler that's written in BQN and this means that in order to run the compiler, you compile the compiler and then you have byte code which your BQN VM runs. Now, this requires you to start with the compiler working. So what happens in practice is that actually we just compile the compiler for you and give you the byte code for that which is not good for a number of reasons and a lot of people object to having this. I mean, you might call it a binary. It's not really binary, but it's byte code that you have to just take from us and trust. So what we have now that we've done is to build up a chain to to compiler compiler without relying on any on any pre compiled stuff. So what I did was take the compiler, simplify the syntax and but make sure it can still compile itself. And then I did that a few more times and got down to one with very simple syntax that I didn't mentioned and Dzaima then spent a day writing a compiler for it in C that works so CBQN runs that very simple compiler and builds up this chain and you get the full... And I don't know if it's set up to do that automatically, but you definitely can now download the CBN repository and the BQN sources and build your own compiler and compile the full one and verify that the one that we give you is actually the same. [03] And the other neat thing about this is that actually I have another BQN compiler that's equivalent to that simple C compiler. Some minor differences that that I verified don't do anything to the performance and the BQN one is actually twice as fast, so that's kind of. So that's the announcement, nothing. No big changes for any BQN users, but just an interesting thing that we did. 

00:07:54 [BT] 

BQN leaves the nest and enters the wide world. 

00:07:58 [ML] 

Well, it was the first array language to ever self host so there is that. Only one, as far as I know. 

00:08:05 [CH] 

For now for now, well what, we've got decades ahead of us so alright well that's all of our announcements. Be sure to check out the show notes for links to all of that stuff. Definitely the other APL podcast, but now we are going to introduce our guests who's been waiting in the wings. I'm super excited about this conversation. You've probably already read the title of this podcast, so you know today our guest is Michael Higginson, who if you follow the... I'm not sure there's two different Dyalog channels on YouTube. There's Dyalog limited and then Dyalog conferences or something. We'll put links to both, [04] but Michael gave a talk that I think was released around October, November of 2022. So if you're listening to this live that was two or three months ago and he was the winner of the professional entrant in the Dyalog APL contest. So for those of us that don't know or those of you that don't know there are. It's a student contest, but and I think you have to be enrolled in either a college or university across the globe. But you can still compete even if you are not a student and there is a professional sort of category and you can win that. And Michael was the winner in 2022, so a little bit about Michael. This is in his own words and this is on a blog that we will also put in the show notes that sort of talks about the winners of the contests. Immediately after receiving my bachelor's degree in computer science from the University of Toronto, so fellow Torontonian here, I joined a small company, founded and populated by a number of ex IPSA that's I.P. Sharps and Associates employees. I had a great time there and stayed for over 20 years. For most of those years I didn't know anything about APL except that it was different that in that it used a non standard character set and that it held in. It was held in some reverence by my mentors and colleagues, but around 2013 I started working with the q programming language and after a few battles with it, experienced a profound 'mental click' that irrevocably changed. Changed not just how I think about programming, but most other areas that I turned to my mind to as well as I was curious about other languages in the family and looking for new challenge, I decided to use Dyalog contest as a way to an excuse to learn APL, and it was a great experience. So I had the pleasure of meeting Michael back in September when we held the Toronto APL meet up so it was awesome getting to chat with him a bit there and I'm super excited to have him on the podcast. Thanks for being here, Michael, and maybe you can in your own words, once again, take us back to however far you wanna go back and you know give us a little bit more detail about your journey to q and then I guess to the APL contest. 

00:10:31 [MH] 

Sure, well, well thanks. Thanks everyone for having me. It's great to be here. I guess the traditional starting point in this journey is usually your first computer and so in that tradition I will let you know that my first computer was a Commodore Vic 20 [05] that I received as a Christmas gift as a little kid. And so the V20 was a small computer where the computer and the keyboard were sort of just one object and it had five kilobytes of RAM and for storage it had a cassette drive, so you it would take a an audio cassette and you could issue a command of the computer to load off the cassette player and it would say. Press play on tape and you would play and then you press the play button and then being a little kid I think I'd jump around the living room or whatever and and wait as as the cassette played, and then it would it would load the program and you know any of any member of a generation that had a a computer in the House as a child, the sort of the gateway to computing for me, of course, was video games. And that's really all I did on that. 

00:11:52 [MH] 

This was not the computer that I learned to program on, although I had heard you, there was a rumor in the school around that you could go to the library and get these take out a magazine, you know, I think there's a magazine called Compute and maybe there were others and and in that magazine would be the listings for what would be a game. So basically you could go to this and get this magazine and there'd be a basic program on the pages of the magazine and you could type it in your computer and then you could play a game. So this was this was very exciting to me, so I did that and sure enough it was there, but when I went to type in the when I went to type in the program every time there was an asterisk, you know, for a multiply or whatever the when I typed it into the Vic 20, it would be this big fat 6 pointed asterisk that took up, you know a whole character width and the asterisk that was printed in the in the magazine was a tiny little 5 armed asterisk, that was sort of up high and they looked so different to me that I thought well, that can't be right. So I left them out! I thought, I don't wanna put in the wrong character so I left it out and and you know, and this seemed epic. I'm sure it was just a couple of pages, but you know in for a little kid, this was like a major operation and so at some point I managed. You know, I got the thing to run. I hit run or whatever and the screen came up and I thought, OK, great I did it and then, you know, I think you were supposed to be a guy who could fly around the screen or something, and at the moment I tried to input something. Of course, I got a runtime error and and and that was very disappointing and I sort of moved on, although I often think if I had had someone in my life who knew more about this stuff and they said, you know. Michael, the game is actually to fix these errors I wonder, you know, maybe if my life, you know, how it would have played out? 

00:14:02 [MH] 

After that I got an IBM and XT clone.[06] So I sort of in the world of PCs and and DOS and then a 386 or whatever. And at some point I got a modem and and that was a real eye opening experience. Once learning that these computers could talk to each other and this was sort of pre Internet, but you know there were local things called Bulletin board systems and it was sort of like the Internet. And only one person could be on it at a time, but you could leave messages for each other on there, and and certainly some of my oldest friends still to this day are people I met through through those systems. So, I wasn't a programmer, but I was a computer user basically and so then I got to high school and but my high school did Co-op placements with with businesses and organizations and I. 

00:15:01 [CH] 

What really? 

00:15:02 [MH] 

Yeah, my high school. 

00:15:03 [CH] 

Wow that is like you don't hear about that until university days. That's an impressive. High school. 

00:15:10 [MH] 

Yeah, it was sort of interesting and so I guess what had what had happened was that the high school had placed a student in a government ministry here in Ontario and to work to help to be a Co-op student on their help desk on their Technical Support team. And I guess you know, I don't know. Maybe they didn't know what to do with him or whatever, but I guess they gave him some manuals for the system. And I think ultimately what he had done something he'd found out a basically a way to send a message to every user in the system, which was like every employee of the ministry. So that like some sort of strange, cryptic message that said something like I am watching you. Something like that came up on the screen of every employee of the ministry and basically, that didn't go over well and they wanted the I guess the the school and so he sort of lost his placement I guess over that and I guess the school wanted to smooth things over. So they went to me. I guess I was, you know, kind of smart and personable or whatever. And so they they sent me in to replace them to give a better impression. So I did and so that was sort of my kind of first technical you know job. I remember one of the projects I had to do is I went around and I had to install 486 upgrade chips into computers, the computers, the workstations being used by some of the programmers in the ministry and and I kind of remember these programmers. I thought of them as these sort of super beings like I knew how to use a computer, but they were doing something totally different than what I was doing. I know how to resize a window on Windows, which by the way was an application, not an operating system back then. And but you know they were just doing something really different, and what was sort of interesting to me is that they were sort of in a way they were like these super beings in that they could get a computer to do so many things. But in other ways they were sort of... They didn't know how to administer their own system like they kind of didn't know how to use a computer, but they know they knew how to program one. I just I thought that was really...

00:17:34 [RP] 

Some things don't change. 

00:17:36 [MH] 

Yeah no. I ended up being one of those guys. So anyway, I did that job and then you know, and then they extended and they gave me a summer job so they they asked me to stick around over the summer. So I guess I repaired the relationship between the school and the ministry, so I guess that was mission accomplished and so then I get to University of Toronto and I didn't really know what I wanted to do and I took philosophy and psychology and Greek and Latin and and... But I also took introduction to computer programming cuz I thought, well, this is something I ought to enjoy. I do like computers and I see their value, so I took this course, you know introduction to computer programming. I had a great teacher and we you know we were programming on the blackboard but the way he showed how to you know we would take up whatever problem we were trying to solve. Whatever it is you know, I don't know bubble sort or something like that and and I think OK, I sort of understand the problem, but I think how are we gonna solve that? We just wouldn't know how to solve it and you know we have this, you know and basically he would start just writing on the blackboard like the things that you know at a high level, the things that we're going to need to do to solve this problem. 

00:19:01 [MH] 

Like I you know I don't know we're going to need to create a list where we're gonna, you know, put the results and then we're gonna, you know, take two of them and compare them or whatever. He would just start writing down at a high level, you know, just paraphrase in English, the steps we're gonna do to solve the problem. And then we would convert those into code. At the time it was Turing, [07] which was an educational language developed at the University of Toronto and then the little fragments of English that we'd use to to, you know, work out how we were going to solve this problem. Then you leave those in as comments. Maybe you clean them up or whatever but you just you leave those in as comments. And something about this process for me seems so soothing like you, you know, and I basically realized ohh, you know you've got this. You know programming to me. You take this empty text file and you just start putting your worries and your problems into it, and then you just keep you refine them and you refine them and you refine them. And what you end up is a program that a well commented program that solves your problem. And there was something about that process or that approach to programming that just really clicked with me. You know, kind of made order out of chaos and confusion and and for whatever reason, that's what that's what made me say. OK, this is, I know, this is what I want to do. 

00:20:36 [MH] 

And then one of the courses I took was called Natural language processing and that of course was done in Perl and I kind of don't remember too much about really about Perl or the domain of that. You know, I know a bit, but but I remember one of the things that struck me was that Perl [08] had this regex operator like a regex matching operator as a built in, not as a function or whatever, but just as a syntactic piece of the language and it was so well suited to the problems that we were solving in the course or whatever, and so that stuck with me. I never really became a Perl expert, although Perl and you know, I ended up using Perl sometimes to solve, you know, one off problems in my programming career. I'll mentioned that later. 

00:21:31 [MH] 

So basically I got exposed to some languages, but C was really the only one I came out of with some competence at school. But in the summer between 3rd and 4th year I got a job as a research assistant at the AI lab at U of T, which probably sounded more glamorous than it was. But I was essentially I was a Java programmer now and we were building a visual knowledge modeling tool in Java, my first programming job for sure. And there was a guy there who was I think he was the original author. He was certainly the current maintainer of the program and he was another student, and so you know, we got introduced and I can't remember exactly how he started me off. But you know, we you don't get too far without needing to either create a source file or open one up and read it or whatever, and so, he's with me and he says, OK, open up this file or whatever and so, you know, as a Java source file and I opened it up in Notepad. [09] And he said, like, whoa, whoa, wait a minute like you're... we're not gonna be working in notepad all summer programming in Java and so he sorta sat me down and you know he was a vim user. 

00:22:55 [MH] 

So he started teaching me vim and so he he was sort of the guy who started putting programming tools in my hand, not languages, but sort of the tools around the languages and we you know. And I started forming opinions on syntax highlighting or whatever, and so that was kind of a really fruitful time and we did a we did a bunch of pair programming that summer which was just, you know, if you've ever done a lot of pair programming with someone whom you're on a wavelength with? It's just kind of a magical experience. It's just fun and motivating. An incredible way for knowledge sharing, and it's just really a lovely human experience to do that, and so it's 4th year now and now I need to... the end of undergrad is coming and I need to find a job and so through just kind of, you know, random luck in a networking phone call, a friend of my Mother's had worked at a company and and thought it was a good place to work or something, and so he sets up a phone call for me with a guy named Leslie Goldsmith [10] who worked at a company called Affinity System. 

00:24:16 [MH] 

And he and a guy named Hugh Hindman had founded this company and it was a small, you know, small to medium software consultancy in the Toronto area and Hugh and Leslie had worked at a company called I.P. Sharp and Associates. Which you know, I'm sure everyone on the panel knows, and most of your listeners know, but I.P. Sharp and Associates was a sort of a worldwide company with a heyday in the 70s I guess, and they were a timeshare company which is sort of pre Internet cloud thing and they and they built applications for for customers and so they were a consultancy and they did it all in APL and then they had their own flavor of APL and and a bunch of you know, luminaries from the field work there you know, including Ken Iverson, and after he left IBM and Arthur Whitney and I think even a a panelist on the Arraycast, might have worked there. 

00:25:27 [ST] 

I always regarded Leslie as one of the luminaries. 

00:25:31 [MH] 

Well, so well, so did I. Obviously I didn't work out at I.P. Sharp and Associates, but Hugh and Leslie had worked there. And there was this crew of other people who had had worked there too. Guys like Mark Dempsey and Dave Markwick and Raj Chauhan and Heather Bowen. She may have come from Reuters after the acquisition, I'm not sure, but there was this kernel, this crew of folks that had come from I.P. Sharp and associates. And so I joined this team and you know, they've been working together for 10 years at Affinity Systems. And you know some of them for 15 years or so before that, at I.P. Sharp and Associates. So you know some of them have been working, you know 25 years together by the time I got there. So I was really kind of slipping into an experienced team building, you know, really interesting and and and challenging projects and so like the the first project I started on was an online trading system for day traders. And so the piece that I built in that was the monitoring system for the back end of this thing, so you know like a centralized process and a little bit of library or whatever that would sit in all the other processes and report to them and it's it was a great like sort of bird's eye view of a complicated system and a great way to sort of get my feet wet and and you know keep in mind here like I'm this was in C++ I'm learning C++ on the fly, I've never seen an IDE. So you know I was using Vim and source files before I think this might have been my first exposure to source control or whatever. I didn't know what a memory leak was. I remember Leslie one time was demoing something I built on and that the demo went well or whatever and then I like stopped the application and he said, you know you've got memory leaks because you know at the bottom of the console of the IDE all these you know memory leak warnings came by and sort of I was, you know at this point still like Ohh I gotta look at that stuff? So I got that kind of monitoring system off off the ground and then and then I had to do a little bit of a refactor and so that was sort of interesting going and touching other people's code and then the next thing that needed to be built was what we called the quote Server, which was sort of like a feed handler is basically the piece of the system that receives all the real time market data from a market data vendor and shuttles it to you normalizes it and shuttles it where it needs to go or whatever. And so it's you know it's a high performance real time application. And you know that and that data are, you know, really critical to this to this trading system. And so you know, I'm not sure like if if most companies you know that needed to build one of those, let's say, well, like let's give it to the kid who doesn't know what an IDE is but but I do think that maybe at at IPSA that is what they would have done, and so you know the culture at Affinity systems, which I think was sort of modeled after the IPSA culture. Was yeah well we'll give it we'll give it to the kid, basically. 

00:28:52 [ST] 

Exactly!

00:28:58 [MH] 

And so that then became my project and you know and it's not like I was alone. I had some really, really smart people that wanted me to succeed and were, you know, going to help me do it, but from that point on I became the market data guy and and so we built that that application and then we you know we ended up doing a similar project just after that which was not aimed at traders but making market data real time market data available basically to consumers and we were doing that for a major newspaper in Canada. You know, and this you know, we were streaming real time quotes to Java applets over the browser in 2001. So this was, you know, I think kind of state-of-the-art at the time and and so and then we had to... Well, this this quote this market data problem grew and you know we started including options data and level two data and having to construct a level 2 for the Canadian exchanges, which was based on an order book and but it wasn't, I wasn't just the market data guy. There was, you know, so much variety. The types of projects that we did kept changing the next project. After that I think I was writing a device driver and we had some piece of robotic equipment for whatever and I had to install port sniffers and figure out how to drive this thing and then the next project after that was was in Java and we had to build this, I don't know sort of a general you know application framework almost sort of pre microservice architecture kind of thing and I could talk a lot about, but it was really interesting and detailed and I remember switching to Java to been many years since I've done Java. And you know the attitude was there was like don't worry about it like it's it's the same like it's just all the same, you know, and and as we subsequently went on and had to you know, do a C# project dealing with, you know, private health data or whatever, whatever it was like. Oh, it's the same you know C#. It's just the same. And so that was sort of a kind of a common theme is these languages are all the same and I remember at one point I started having these epiphanies, not just how the languages were all the same, but how the projects were all the same. They all tended. They all had data coming in. People who wanted something of it, and and I remember at some point. You know he's working up this sort of just sort of this grand unified architecture for everything idea I had of of just the this sort of way to solve all these problems were all the same and how you know one architecture could solve them all. And I was presenting it to Leslie over dinner one night and I remember. He, uh, he said, well, this is very APL-like this thing you're proposing, and I didn't know what he meant by that, but I did know it was a compliment. 

00:32:12 [MH] 

There was a thing I knew about and it was I knew it was a background that my colleagues shared, but it wasn't the solution of anything we were building and so so I never was I was never really exposed to it, but then at some point we had a project and the project was basically to store and provide access to and analysis of electrical consumption data for the province of Ontario. And so this was a database that you know for five million homes and businesses reporting every hour or 15 minutes or five minutes, or the target was every 5 minutes, so this this was a database that was going to grow and grow and grow and and we needed to be able to analyze that. And so this was certainly a big data problem, and so our first engagement was really a research project to recommend the technology to solve that Leslie with his I.P. Sharp background was aware of kdb+, [11] and the q programming language and we ended up basically we ended up recommending that he convinced those of us that we're working on the project, that this is what we should recommend to the client, which we did, and we built a proof of concept and how to Bake Off against another vendor which we won and so then we were hired to this electrical consumption data project at this point q is put in my hand and this you know this problem is is in front of me to solve and and really from that point on q that I just became a full time q programmer and I basically never went back. Oh, and I wanted to say then you know, unlike C# and Java, when they said this, this is don't worry about it's the same. It's just the same. When q was putting my hand they said OK, this is not the same this is this is going to be different, in case you didn't notice, just in case. 

00:34:25 [CH] 

in case you didn't notice, just in case. 

00:34:26 [MH] 

 And it wasn't. Although you know at first I was basically writing the same type of code. You know you can write some pretty Java'ish code in q and which is arguably a strength or it's certainly possible and, and I did that, and that's sort of, I think maybe one of the classics, sort of learning curves you have to go through, but at some point we kind of switched from project mode to product mode and and we required and and we we started building a product to solve problems like the one we had, we had just solved and so that was really interesting and and you know, building a sort of a layered product that that partners could customize and doing all of this in in q and kdb. And then and so I did that for a bunch of years and and then after 21 years that job, I decided to make a change and and move on and and I took a sabbatical to just kind of focus on some other areas of of my life and and and I started listening to this podcast in that time and I knew that, you know, because I known from all those experiences at Affinity that that these languages and that culture was a part of a sort of a greater community, and I'm sort of interested in that, so I started listening to this podcast. [12] And so through the podcast I heard about the competition, of course cause it cause it had been advertised on there, but but also you know I found myself listening and strangely having like I don't want to say shouting out, but maybe shouting out at times or whatever, like I remember, you know one episode where Conor and and Bob no, sorry Adám and Bob were debating, you know whose trains were better. You know Js or or Dyalog ATL's. And and sorry, Bob, but like Adám ended up convincing me like I like, I like the way they worked in APL. But I'd never written a train like, you know, I hadn't heard of 1 until you guys were talking about it, and so I had this experience of feeling finding myself with strong, passionate opinions about a topic I had no experience in. And I thought, well, this is dumb so. 

00:36:57 [BT] 

It works for me. 

00:36:59 [MH]

So I wanted to learn APL, basically, and so then I thought: "OK, well, the competition would be a great way to do that". This is sort of as an aside: so at one point I saw someone give a demo in q, where they used the q REPL [Read-Eval-Print-Loop] as PowerPoint, basically. They'd written a bunch of functions that could emit ANSI codes to the terminal and so they could, you know, go through slides and I think they could run code and then, you know, put the result up and leave it while running more code. And it was just this ... fascinating demo and, you know, it's like I forget what the topic was because all I could remember was what was shown. And so I started working. I wanted to see if I could make something like that. So I started doing that, but because this podcast had polluted my brain, I thought: "OK, I'm gonna try to do that all tacit in q and write the whole thing with no curly braces". And so I started doing that for a while and I started convincing myself: "OK, I can do this". Like I was getting somewhere. But then well, I thought: "this is dumb!" [others laugh]. It was definitely harder than it needed to be and what I really wanted to do was play around with APL and you know the deadline and the competition was coming up. So I sort of abandoned that project and then I started on the competition and we could, you know, go into detail on that. But just to get out of my background phase, that's the story from the kid who's never seen the computer to the APL competition.

00:38:51 [RP]

It was really cool. Well, it was great meeting you in Portugal. I got to see your Windows presentation live; that was really fun and interesting, but it's also been really fun hearing more details about different parts of your history that you didn't say there.

00:39:05 [CH]

Yeah, and (we mentioned at the top) for those that haven't seen that, it is available, recorded on YouTube. We'll put a link at the top of the show notes and both your talk, Michael, and (I'm going to forget the student winner, yeah, one of the student winners )

00:39:15 [RP]

Tzu-Ching Lee [13]

00:39:18 [CH]

Are both fantastic talks that are, you know, great if you're array language curious and you want to see ... maybe not a beginner, but sort of someone that's new to the language. I can't remember the other individual [who] also was new to the language and had some interesting insights and stuff.

00:39:43 [MH]

I'm glad to hear you say that, because in that presentation, my goal was to be useful to potential newcomers and try to break down what for me, in any way (I'm not claiming to be expert or anything) [is] the thinking process. I took one of the problems and tried to show how (given the tyrannical empty text file), how are we gonna solve this problem? I just sort of break down approaches (thinking approaches in a way) to get there. So I hope that might prove useful.

00:40:28 [RP]

Yeah, it's really good if you're looking for that perspective: how to approach the solving of the problem part more so than, maybe, how you would spell the squiggles? It's really good for that.

00:40:38 [BT]

That kind of goes back to your original story about your professor who started with, you know, what we need to do here. And breaking it down to a computer language. It's consistent with that, isn't it, Michael?

00:40:48 [MH]

Yeah, yeah, and I think for, you know, for whatever reason the way my psychology works is I'm just sort of perpetually stuck in the beginner's mind, and so, you know, my whole narrative is always like: "the first time I did this and the first time I did that and just how exciting it is to sort of learn new things and learn new ways".

00:41:12 [RP]

And I have a bit of a question there because you've programmed in largely a lot of traditional languages and then more recently array languages. One of the approaches to teaching computer science and programming in the sort of conventional computer science schools is to teach (well, especially like high school, sixth form, college kind of age) about, you know, breaking down (decomposing) a problem into the necessary steps until, I guess, you get it down to the point where: "oh, I know how to express that idea in this computer language". Did you find a difference? You know, the level at which you would start putting stuff to code or how much you need to break things down, or whether you even would break things down in different circumstances. You know back when you're solving problems in more conventional languages, and then whether that's different or similar with array languages?

00:42:10 [MH]

Well, yeah. I mean certainly you need to break problems down, depending on the size of your problem. I know a couple weeks ago you guys were talking about, sort of, leet code versus like production applications, [14] and I thought that was an important discussion because I know I've met a couple of people who are interested in this topic, but find it hard to ... cross that bridge from the sort of puzzles and brain teasers into, you know, how to build a whole complicated, bulletproof production-worthy application. And certainly, you know, it can be done. So obviously you need to break down a problem like that. I guess I would say one of the things about these languages is often ... [sentence left incomplete]. In a traditional way, you break it down until you know how to solve it, and then you build up the solution and test it or whatever, and then you get it working. And if it performs well enough, then you move on and whatever, and that's sort of the process. [On the other hand] I find in these languages frequently , I solve the problem 3 [or] 4 [or] 5 different ways, and then pick the one I like the best according to whatever, you know, whether that might be readability or performance or reusability or whatever. So that's definitely a different process. And this is maybe a tangent now, but relatedly, one of the big changes for me too is, of course, the workflow where, you know, in that traditional way, I talk about [how] you take an empty text file and you describe your problem into it, and then you refine it, and you refine it. And then you get it into code and then you try to get that text file to get the compiler's seal of approval and then see what the runtime gain looks like. Whereas in these languages, it kind of goes the other way. Which is to say, the REPL was a new experience for me, but I am building the code in the REPL and you try experimenting, getting it to work. Maybe seeing if I can get it ... I'm golfing it, I'm whatever it is, I'm trying it a bunch of different ways. And then when I've got something that I'm happy with, you know, I cut-and-paste that line into the text file and sort of cement it. Of course, I might come back and change it or whatever, but it's this sort of: I work and test and debug the code till I'm happy with it and then put it in the text file whereas in the traditional language I'm putting (typing) code in the text file and then, later figuring out whether I'm happy with it.

00:45:15 [ST]

You were speaking earlier about [how] this is not the same and also (in relation to q) not the same as Java, not the same as C# and you also mentioned that q has got features which allow you to use what you know in the languages that are the same, and get started. And it's definitely one of its virtues, that you can get productive more quickly.

00:45:44 [MH]

ABut also it's a pit, a trap too!

00:45:46 [ST]

Ah yeah, keep that thought, because that's exactly what I want to ask you about. Because if you approach k de novo, [15] you've got to pretty much grok the whole language before you can get getting anything going in it. Whereas q is a soft introduction. So in my day job I'm particularly interested in the transition to vector thinking because from one point of view, that soft introduction in q, the trap that you just spoke of, it defers that process. And as someone who's come through that process with a good deal of awareness (because you've got other programming languages behind you, and you've got a lot of support around you at the time from APL vector guys), I'm wondering what your thoughts are about that transition: what helps? what works? what makes it harder?

00:46:44 [MH]

Sure. Well, I guess I would say that for me it was a trap. You know I had to ... [sentence left incomplete]. The first major thing I wrote in q, I had to rewrite because it performed like I'd written it in Java and we didn't pick q to get Java level performance [chuckles] so, to some extent that can be a (if you're on a timeline or whatever) stressful experience but I would say, if you were looking for a bridge language you could do a lot worse than q. Invoking a function is "normal" [chuckles]. Quote-un-quote normal! You know, you take the name of the function and then some form of brackets and then the arguments to the function; some form of separators. Yeah, it's a little different than the traditional C-like ones, but it's the same sort of idea. And by the way your function can take any number of arguments, including zero or including nine or whatever. So the control structures are the same or quite similar. And it's funny enough: your case statement is a dollar sign or something like that. It's sort of syntactically different enough that it's feeling exotic and that's part of the trap I think. You think: "I must be doing this right because I figured out how to use each", but really, you're just writing loopy code or whatever. But it is sort of a soft landing, and one of the first things that it forces you to contend with is the change in precedence rules and which is sort of magic. Like to me losing BEDMAS [16] and all the extension, it's like taking a weight off that you didn't even know you'd been carrying for decades, having to remember those things. Like people, you know ... [sentence left incomplete]. BODMAS isn't that hard, but when you start getting into logical operators and bitwise operators, and pointer dereferencing and remembering what order ... just how to parse this. You have to look it up in a table or devote a lot of your brain to memorizing it and running it through the table you have in your head versus just scanning the line. Just reading it or writing it is such a great experience for me. Like that is such a magical thing. So the new programmers (like the programmer coming from a classical background and encountering q) they have to learn that and that's a great new thing, a great new experience, but of course they can still write this loopy code or whatever. I guess for me I'm not sure exactly how to get over that next hurdle, but I mean for me it turned my thinking inside out. You know, figuring out how to write code that solves the same kind of problems you're used to solving but having it look so different. I mean, it's a great experience. It's a challenge. I think a lot of that is sort of done historically through apprenticeship, word of mouth, orals, verbal storytelling or whatever.

00:50:49 [MH]

You know, for me, the main concepts you got to wrap your head around is sort of getting rid of the outer loop. You wanna push the loop further and further into your code, until ideally it disappears. Or if it doesn't disappear, you're looping over the smallest possible thing. So a classic sort of anti-pattern I would see in q is a big lambda-each over the things you wanna do it over and you may need an each in there somewhere. You may need to do something with a primitive that isn't vectorized or a function that isn't vectorized, but you wanna get that each so it's just over the things that need an each, and so it might kind of feel weird because you might need five or six or seven each's potentially if this is a big lambda thing you're doing, but the point is ... [sentence left incomplete]. Because once you have a big outer function and a single each over that, you've lost the opportunity to use the vectorized functions, any vectorized functions that occur within that lambda. So that sort of process of pushing your loops deeper and deeper into your code, until hopefully they evaporate is almost like a literal inside out thinking compared to the way I'm used to. And then same with branching too. I used to have in the traditional coding world, I had this sort of almost visual intuition to what a solution to a problem ought to look like. It's: "OK, this is going to be a for-loop with a condition where I'm going to iterate over the items and then I'm going to test some condition of interest over each of the items and if the condition is true, I'm gonna do this with it and if the condition is false, I'm gonna do that with it". So it's gonna be some sort of double-humped for loop kind of thing. And you would just know. Or maybe the problem is more complicated and it's a quadruple-bumped double for-loop or whatever. But I would just sort of instinctively know what the solution for a given problem ought to look like. And if I opened up a piece of code and it didn't look like that, I'd think: "hmm, I bet you something's off". And often I'd be right. And it was this kind of gut sense thing that I had. But of course in these languages you don't do the branching either, right? Or you try generally [not to]. q supports that sort of thing, but generally if there's a property of interest over the things you care about, you compute it for all of them at once, and one thing. And then you do whatever you want to do to the ones that have the property: you do it! And then you do whatever the other thing [is] to the ones that don't. So it's just four lines. Well what if one of these sets is empty? You just do it anyway! You know it'll all fall out, and it's really quick, to whatever, reverse a vector with nothing in it.

00:54:04 [BT]

Michael, when you moved over to APL, did you already have the epiphany fully formed from working with q, or were there other things that you found when you moved to APL that you suddenly thought: "oh there were areas of q" that you didn't like? I guess was APL more revealing to this process or did you get it all from working with q, just in the different paradigm?

00:54:31 [MH]

Well no, APL definitely revealed more, for sure. I mean because I had the experience of learning q with veteran APLers, I think I kind of got a lot out of it. So I was definitely coming to APL with a leg up for sure, but certain things took me further or refined it further. Like you know ... tacit. [17] I know there's tacit in q; it's limited compared to APL, but there was some tacit stuff and I had encountered some of that in the wild, but it wasn't really on my mind. It's podcasts that really just made me aware that it was a thing to have an opinion about, or whatever. But I started to embrace the terseness. Of course q is less terse than APL, but it's still pretty terse compared to Java, and very terse compared to how Java is frequently written. But one of the things that I did in q was there's a primitive in q called enlist and it's not the enlist in APL. It's the enclose from APL and this is something you use *all* the time. This is a very common thing to want to do: to basically turn a scalar into a list or a list into a list of lists with one item, or whatever. And so it's such a basic thing that you want to do and I found enlist, (a 6 letter word, often taking a variable afterwards, so needing a space) too long [chuckles]. And so you know, one day I'd had enough. I think I was maybe working on a weekend or something and so I felt a little more free to pursue a personal pet peeve and so I added to our utility library, a shortened version of it, ENL (for enlist) and I just sent an e-mail to the group: "I couldn't take it anymore; I did this, you know; let me know what you think". And the group actually adopted it. But the way I implemented it was as a lambda, as a function. Basically: curly brace enlist x close brace.

00:57:11 [MH]

So a function who passes its argument to enlist and returns that result. And then at some point I noticed Leslie (because I'm a diff reviewer by the way [chuckles] ) I noticed that Leslie had changed it. He changed the implementation to "ENL get enlist". So which is sort of like, duh! You're just creating a synonym or whatever, but at the time it kind of blew my mind like: oh, it's sort of the most basic tacit definition, you know. It's the degenerate tacit case, arguably, but it sort of blew my mind and that sent me on a whole new way of working. Functions as first class citizens is a great feature of q (and I'm sorry, I know your question was about APL and I will steer it there) but this idea, all of a sudden realizing functions as data? That was a big kind of mind expanding moment for me and the stuff I could do in q. And with its terseness, I could build tables in code, like literal tables and put functions as members of the table and drawing that in code as a 2 dimensional array. Basically then have that control the flow of the program, and just like really, really different interesting stuff. But Bob, you were asking about going to APL, and so certainly tacit was what brought me there and the interest in it and then obviously, you know, rank is a different concept and so those were good to play with.

00:59:14 [BT]

But I guess on the other hand, the thing that you lose is the higher order functions, right? Because I know BQN has that, but J, and APL do not.

00:59:23 [MH]

Like the first class like functions of data, yeah.

00:59:26 [ML]

Yeah, like putting a function in an array specifically is one that you just ... [sentence left incomplete]. Well, APL has these things called function arrays, but they're really weird. You have to create them from namespaces. I mean they're not the same at all.

00:59:39 [MH]

Right. yeah, they're not. Yeah, and it is probably (when I'm programming in APL) the thing I keep reaching for. And yeah I can, you know, put stuff in names. I know they can be kind of values of namespaces in a sort of dictionary kind of way, and there's some stuff, you know, there's classes, so there's like ways to encapsulate functions that are useful.

01:00:00 [ML]

Yeah, I mean, why do you have to encapsulate it? I want the function.

01:00:03 [MH]

Right. You know putting two functions in a pair and then using a boolean to get which one you want out of it and then just executing that, or whatever. That's a really nice feature of q. But if you don't mind, I did want to talk about one of the aha! moments I had in APL, at least with tacit, which I know is maybe a controversial topic for some people [chuckles]. But you know, when I got to the competition I, the idea to play with tacit is what said: "OK, I'm going to do this now". And so I got to the competition and there's the phase one problems which are basically, ten 1-liners and my initial goal was: "OK, I'm gonna do all of these tacit", because that's why I'm here. Which I failed to do, or some of them were just hairy enough that I just needed the safety of arguments, basically. But I did, but one of the questions [was] you had to write a function that basically looked [at] two lists of equal length and how many were not the same. Like if you line them up, [and] compared one to one and two to two and three to three, how many differences were there? So which is basically sum not equal, and so that was comfortable; that was easy; I knew how to do that. It was just called the two train or whatever it was. That was pretty easy, but the next one was: given 2 arrays return a list of things that are in just one array but not both. So what are the list of things that are in either one but not both. And so you know I worked and I worked. Like I thought about that and I tried it a few different ways. I'm sure there's a lot of ways to solve that problem. But the solution I used was: "union without intersection" ... tacitly. So just union without intersection [chuckles]. [18] Not a function that takes its arguments and passes its arguments to union and then passes it to intersection and then passes those results to without. Not that function! Just union without intersection. And it's like, you know, to me it was like, oh, this is like a line from a haiku, you know? It's like a metaphor for marriage or whatever. Like it's just this ... there's something to me that was kind of poetic about it, and I thought I really ... [sentence left incomplete]

01:03:10 [ML]

But it's got 8 syllables so ...

01:03:12 [MH]

Yeah, I know, I know! And the more I looked at it ... and if you look at it, you know the union symbol is sort of a wide symmetrical U and then tilde and then intersection is that inverted. And it's very kind of beautiful and there's symmetries to it. You rotate it 180 degrees and it still says "union without intersection". And if you like reflected on either axis, it says "intersection without union", which, (I don't know what that means) but it's ...

01:03:50 [ML]

Just going to be the empty result I guess!

01:03:52 [MH]

Yeah! And that was kind of interesting and then if you zoom in [to] the tilde ... if you look at the tilde, it's sort of [the] union and intersection fused together horizontally. You know, so it's like a fractal! You know you go in and there's just more of it. And so this was for me like ... I kind of ... [it] clicked. Like the the appeal, the sort of mystique or the why this is (for me at any rate), interesting. Like it was, you know, aesthetically pleasing and so that was that was really interesting to me, although I'm not sure that I'm a huge tacit guy. As I went on, and especially as I got into phase two problems where I needed to solve bigger problems than I would personally, I would find myself ... [sentence left incomplete]. Like I can't tell you how many hours I, you know, battled the interpreter to build a tacit expression for what I was trying to accomplish and succeeded at great personal cost [chuckles] and then ultimately decided (this isn't ... this is not to me ... to my taste at this moment, in my experience), did not seem like the best way to convey the idea. Or if I came back to it and needed to refine it (you know, fix a bug or whatever) what I wanted to come across or if I needed to debug it. So in the competition I wrote a lot of tacit stuff that was not in my submission at the end of it. But I was really there ... [sentence left incomplete] I wasn't really trying to win. I was trying to win as a way to learn APL. So I certainly succeeded in that, or getting a feel for tacit at any rate.

01:05:44 [ST]

Can I take you back to what you were saying in the learning and apprenticeship and you spoke very warmly earlier about pair programming.I reflected some years ago that pretty much all the APL I'd mastered or learned. I'd learned really from somebody or from working around people who knew stuff. I learned learned the language that way in the first place.And this is kind of a bit of a sideways jump, but as a young man I converted to Zen Buddhism. Which distinguishes itself amongst the Buddhist schools as a direct transmission of the Dharma of the truth from one heart to another. It's pretty much the basis of all the teaching I've done successfully has been based around pair programming.I was first trained as an APL instructor using by Ken Iverson. Using his working introduction to APL. Which was basically teaching people in pair programming before pair programming had a name.And when I reflected some years ago how a working from home and being dispersed around the world was kind of isolating me from all the great colleagues I used to have from whom I was always learning stuff.Oh, you're doing that, you know? Uh, we started a series of I don't know what we call.They're kind of week long APL house parties or array language house parties.Which is which we called Iverson College. [19] We started out in 2011. Just to get people back in working in the same room again and people come from all around the world to spend a week doing their work in the same room. So you could kind of talk about it afterwards. So this is a challenge for me, standing challenge is getting people over that hump you're talking about from the what Joe Kaplan on this podcast called The One Potato two potato approach.I was thinking in terms of vectors.I don't know any more effective way of doing it than really just like one to one pair programming as you as you described it, and unless I hear of a better way. If I can hitchhike a ride or I'm passing seagull off to kx con in May. I hope to be there and do what we call the Vector Dojo and set up that kind of pair programming so people can experience it.So that's my thinking.I wonder if you if if you can help me on that in any way or ideas or whatever.

01:08:29 [MH]

Well, I completely agree.I mean, I think you know I had, you know, that experience I had at the AI lab at UT. That.So there's a lot of pair programming in that and then basically pair programming with Leslie at Affinity. For, you know, the start of my job there. Completely made me who I am, you know, completely taught me how to be a programer.And you know, there's it's.You know whether you're trying to learn array or how to internalize the keystrokes of vim or whatever. It's it just such a way to connect with a person and and learn stuff. And it's so tailored to the person you know. If they don't get it, you don't move on, whereas you know a video or a book or a class. You know, doesn't work that way. And so you know you can. So yes. You know, to some extent I would say, you know, zoom like... there's some. There are certain physical sharing, you know, two office chairs at one desk and one monitor is sort of physically awkward, you know, and but other than that, like the human connection from being in the room. Talking, eye contact, real eye contact not not via a cam. Waving your hands, you know like just all that, you know interpersonal experience. I think is... you can't be beat like as as an educational as a way and especially for a code base. Too, you know, we're sort of talking about languages and tools. But also, you know you often have a code base and to, you know to it's to be given the tour through the code base you know. Let me walk you through this, let me show you search for this and these files or whatever is just a really great way to sort of hand off a code base as well or to bring someone into a code base.

01:10:54 [ST]

Well, maybe we can, maybe we'll both get to kx-con and be able to do something.

01:11:00 [CH]

All right, I have a list of like 7 things I need to ask Slash say. Because this is, well, the first thing I'm going to say is one of the most recent movies I watched in theaters was Avatar 2 and like this is like 10x 100x better. Like I've said it before on other podcasts that like I don't know. Listening to this kind of stuff, it's like better than sports and like entertainment and Marvel movies for me. Like hearing you talk about the poetry of the intersection union without, you know. It's just it's going to be great. I can't wait for our listeners to hear this because these kind of conversations. I just have that I was a couple times gonna interrupt and ask a question and then Bob would ask the question. I was you know what I'm just gonna treat this like I'm at a movie experience. But so, I guess, I'll start with my most recent thing that I want to say because we were talking about pair programming. I think one of the things too like that makes.. Because I've had good pair programming experiences and like less good. And the best ones are the ones where there's like a complete amount of like mutual respect between the programmers. Not that, like the bad experience I've had, there's there's been a lack of respect, but it's more like you are not worried that asking a question is gonna come across as like that I don't know what I'm doing. And I think that there is a ton of that today in the industry because salaries have gone up. Or, you know, the little bit of shift happening in the industry right now. But like I think a ton of people are completely afraid that if they do that kind of pair programming experience. Like day-to-day, we're fighting with compilers and trying to figure out how things work and people are very concerned that if people saw the way they worked that they would be exposed as a fraud. When really that's what we're all doing. Like I've talked to some of the most senior people in the industry that work at companies like Apple and Adobe. Executive senior people that still do technical work. And I remember one time I won't say his actually. I think he probably totally be fine with me saying this cause I might have it in some other contexts. Sean Parent [20] he works at Adobe. Extremely senior, you know, luminary of the C++ world. And I remember having lunch with him at a conference once and then he was telling me about this project he was working on and he needed to Stack Overflow. How to get some like Apple hardware to do and I was like, wait what? Did you repeat what you just said? Did you say you were on Stack Overflow. Like my view of this individual? Is that you know he's this kind of God like figure that lives in the clouds and you know they they tell stories of him. How like he just sits there for four hours without writing anything and like programs it in his head and then puts it down on the screen and. It just compiles cause he worked all the. Bugs in his anyways. But that's like the stories that they tell. And then you have lunch with him and he's like Oh no yeah. I was, you know. I found some stack overflow posts. It was like the third answer that actually did it anyways. The point being is that. I think pair programming there's not enough companies that do it, and I've even at certain companies have tried to suggest it and managers are worried it's gonna lead to like less efficiency because there's two people doing the same job and it's a real shame. Because there are other companies that it's their bread and butter. And if you go into it, knowing that like we hired you because we think you're smart, no question is stupid. You know there's things that I'm learning every day and everyone learning every day. Anyways, that's my little bit on pair programming.

01:14:10 [MH]

Sorry, can I just jump? I just wanna say one more thing about pair programming. Which I'm not sure if I used the word before, but I think it's really important, which is that it's fun. And like the laughs some like at affinity, you know? We would. This might be more in a, say, a whiteboarding session, than maybe a pair programming thing. But like if we were trying to solve a really hard problem like someone would always very dryly throw in a ridiculous suggestion for how to solve it. Like a really really bad way to do it, and it would be so fun and every once in a while that would actually spawn some sort of lateral thinking. That would open up some new possibility that it. You hadn't occurred with. But just fun and laughing with your colleagues. You know, is motivating and creates loyalty. You know people will stay longer somewhere they enjoy working, you know. That's sort of pretty pretty basic stuff there, but at any rate so. Yes, pair programming is is awesome and fun and and super useful and that whole concept of like egoless programming, which I think is part of what you're getting at. You know, not being afraid to look dumb, or, you know, we want the best solution, not, you know, from for people to think I: Can do the best solution.

01:15:36 [CH]

Yeah, it's like I recently posted a YouTube video and I'm a Rust beginner. [21] Like I've just started learning or maybe for a few weeks now, but you know. I made some mistake that you know a ton of people. Some people are really nice about it, but other people just like Oh my goodness like this is and I'm, you know, they're trying to be kind of like harmful. But I'm just like this is perfect, you know, I put stuff out in the world. People tell me and like you know, who knows when I would learn that. But like now I learned it faster because like honestly like good code review where people are pointing out mistakes or places where you can do something better. Oh hey, did you know that there's a convenience function? Where you can spell it, you know. 6 characters less I love that stuff, and it's fantastic.

01:16:12 [MH]

There's a I think in like musician circles or certainly jazz musician circles like there's a the you know, the. There's a theory that, like you always want to be the worst player in your band. That's how you get better.

01:16:26 [CH]

Yeah, that's I mean, I think. Wise his name alluding me Chris Lattner, the individual behind Swift and clang of LLVM and a bunch of other projects. I think that's one of his like quotes that you can hear on different podcasts is that he always is searching, you know, he was at Apple for a while, then Google for a while doing tensorflow for swift stuff. And if you listen to him on podcast, he's always saying that like: whenever I feel like I'm, you know, I'm not at the bottom of the totem pole in terms of some technology like I want to switch because I always want to be learning from people around me. Which I think is amazing to hear from someone, you know that senior and what he's been doing. Alright, I got a couple other comments and questions first one is another plug. I plugged this on my other podcast but so we we brought up the the poetry and the podcast is oxide and friends. It's by a guy named Brian Cantrell and friends and it started out as him and some some colleagues of his. I think a lot of them used to work at Sun Microsystems. I know Brian did, and I think Steve Klabnik, who I might be pronouncing his last name wrong. One of the co-authors or main authors of the main rust programming language book. They have an amazing podcast. My favorite podcast of all time is functional geekery. And I recall I was working at Amazon at the time down in Silicon Valley, and I would go for these runs at lunch and just like crush these podcasts and I learned about Elixir for the first time and Erlang for the first time, and F# for the first time. And just so many of these language that I had never heard of because at that point I was just a C++ programmer and knew a bit about Haskell. And it was just such I like I remember that time so fondly because I had 120 episodes to go back and I could just like listen to them in a a row. And I'm having the exact same thing, this time going back and listening to Oxide and Friends. [22] The reason I bring it up is because on two different episodes a couple of part I'll find them and put them in the show notes. Brian Cantrill who some of people, not know by name. But is the individual that had an article interview with Arthur Whitney. And so Roger Hui I think when he passed away they mentioned his name on the podcast in the episode that was close to that time period. And then a couple episodes later they mentioned that at some point, I think during the interview and it might not have made it into the interview with Arthur Whitney. Brian Cantrell asked him, what do you think the best analogy for programming is, you know, is it you know some management system or etc. like if you had to choose like sort of an analogy, what would it be? And Arthur Whitney's response was poetry, which I just thought was fantastic. So anyways, people folks should go listen to that podcast. Yes, and I guess I'll skip a couple of the other things. And maybe maybe we'll have a behind the scenes where I can ask these later because I'm going on too long. I think this has sort of been answered here and there, but maybe not directly so I just wanted to ask explicitly in case if there's anything that's been missed. I know when we talked back in September in person you were talking about sort of the things about q that every once in a while would come up. You'd be sort of those moments where you'd be shouting: Oh, I hope they mentioned this and and that sometimes we hadn't mentioned that. I think you know, putting the functions in a sort of a pair of functions and then executing one. And a couple of other things you said, but other other things about q that sort of you want to tell the listeners about like things you might appreciate in the q language that aren't possible in the other array languages, or that are more ergonomic. Just that in case there's something that you know hasn't been mentioned that you might want to.

01:19:49 [MH]

Well yeah, I mean I guess I think the main the what the you know the the thing. The event you're referring to that was definitely being able to put functions in data structures and use those data structures as as sort of ad hoc control structures for your program. So which I know I think Marshall you mentioned that when you were talking about BQN and the control structures episode so that that to me that is like a big feature of q that I use a lot you know. And for example, you know in a classical thing you might do in an application is you've got these events going off or messages that arrive from some you know upstream, whatever. And the first thing you gotta do is, OK, you deserialize it, or you deserialize enough of it to know like what sort of thing do I have? I mean in q you deserialize the whole thing I assure you. But you so you know, in C++ or whatever then what you what do you take this thing and you look it up in a big switch statement typically. This is the the code I would write. You look it up in the big switch statement and it's sort of every line is a, You know, if it's a this create a new this spelt differently. And you know and then off and then call do on it. So it's I've got a foo Message coming in so create a new Foo message object, populate it with the byte stream that's coming in, tell it to finish deserializing, and then do itself do whatever you know the programmer said we should do when the Foo comes in. So just you know, being able to like ike you know, index, you know if those if those literals that tell you what type of message it is. Are you know dense set of small numbers. You can just literally index into an array of functions and just you know, go off and running and it's just it's so fast and it's such a clean way to construct something like that.

01:21:55 [ML]

Yeah, so I've done that a bunch of times and one of the other cool things about it is that you don't have to have just one function. You don't have to have just a list of functions. You can even have a table of functions where you know one of the function processes it maybe even you have another function to you know, deserialize or something like that or another function to tell you some property of it. All that kind of stuff you so you can range your whole program very nicely into a big table of functions where you've got the type of element and the thing you want to do.

01:22:21 [MH]

Right and it's often like it's sort of. It can be a multi dimensional like. Here's the you know. Here's the pre processing processing post processing we wanna do for each of these types of messages, and so that's sort of a 3 by N table and each entry in that might actually be a chain of functions. So you know where each function if by convention returns, you know it's input or possibly modifying that. So you know, you might say, look pre processing I want to confirm that the sender had the right to send this type of message. I wanna audited, I wanna log it you know whatever. And so that's my so my pre processing chain which I can customize by function and then I go into you know processing and so you end up with these sort of control cubes and then the outer code. The sort of the top loop that runs your program is just sort of like, you know, take the thing and you know, find the cell and run it, run. Run it through the function. So that is to me really really neat and then that with the terseness is you know, I would, I think I mentioned this earlier, but I would literally I can you know. Someone, by the way, this is sorry, apologies for tooting my horn here. But someone once told me that you know he'd come across a table I'd written in q like a literal like a literal table like I was constructing. I was using code to create a constant table basically, but I draw it like a table, you know. I use whatever the separators between the rows and the fields and the rows or whatever like they be like the pretty little boundaries, the table or whatever... And you know they said it was the nicest you know table they'd come across, you know in q. So I felt I felt very pleased by that that was a former guest on your show actually. But then, like in the cells you know you can like, you can get a lot if you're doing something like that, you can put a lot of information in the cell because it can be so terse. So, so you can really just kind of get this bird's eye view of the program and just this, in a non-linear, you know. The code file with this non-linear representation of how the program works is kind of neat.

01:24:41 [ML]

And I guess we can point this out as one thing that APL is trying to remedy. I don't know if j has any plans. This array notation that q has. You've got your brackets and you put all your stuff in the middle and you can use either semicolons or newlines as a separator. This is pretty similar to what Dyalog is starting to roll out. Sort of as a as an array notation, but you'd still be missing it right now in a PO.

01:25:09 [RP]

Yeah, yeah, there's some history behind the the development that Adam is working on. [23] A more official proposal to send around the APL vendors and other community members who might want to have a say so that. If we implement it as an actual language feature, we're not gonna like make horrible mistake and want to go back later and regret those choices.

01:25:30 [BT]

And there there's certainly been discussions on the J forms, but I think most of it reverts back to Gerunds. Which I'm not sure is the solution we're all looking for. But often that's held up that way. But you know, it's being discussed.

01:25:44 [ST]

Well, Conor we got another guest here who's absolutely haunted by aesthetics.

01:25:51 [CH]

I was so. When you were telling when you're telling enlist. I was like, oh, this supports like whatever I was saying a couple episodes ago like so much,it's that like for these... And, that's the thing. It's like I know that there's people, maybe not the listeners of this podcast, but I'll talk to programmers at large at different conferences. And you know, you if you've listened to the my other podcast, Bryce, you know at one point he when we were having this discussion, he was like, what did characters ever do to you like you're so against the character count. And I just said like for primitive things. It's like you know who wants to go around writing for infix PLUS you know. And and then you know three like it's for something as fundamental as addition you know. If you're using it, especially in a domain where it comes up all the time, like it hurts my soul to have to, you know, write, you know in C++ it's STD, colon, colon, PLUS. Tt used to be bracket int bracket paren paren You know, with C++ 17 we got rid of like five of those characters but it's still like it's plus it's in it's math. In math, it's plus, and like Haskell, you can do it 3 characters. I think in q you with the parens you can do it with three character. And in certain languages like lisps, you can do it just with the plus and it breaks my heart when like, especially when you experience that experience it's like when you said when you go to APL from q and you're used to be able to used to be able to do something a certain way. And then you don't have that convenient tool that you're used to being able to grab, it's just it's sad and yeah. I some people will say, you know, uh, it's a silly argument. But I really do think that there is like the the brevity with with which you can express yourself and subordinate, you know syntax and details that don't matter. Affects the way that you write code and and the way like you said, all of this at the end of the day it's fun like it is when you when you write that three line, you know, solution every once in a while you're like it's so beautiful. How, why would I ever want to write this any other like it's yeah anyways, uh, Rich.

01:27:50 [RP]

To to be clear to clarify, I think there are two things we're talking about here as well at the same time, one is the ability functions of first class citizens that you can put into array structure. And then do array manipulations to choose which functions to apply on on your arguments. And then the other one is a literal array notation for in your source code, like writing, literally an array. Rather than having to construct it from primitives like you have to do most of the time. Just want to clarify that Dyalog is not putting in functions as first class citizens in case someone got got that idea from the way the conversation went.

01:28:27 [ST]

I can't leave this one alone. We have a 3 Canadian episode here. I'll be, I gotta quote another Canadian so I came so far for beauty and I left so much behind.

01:28:40 [CH]

I have no idea who that's by the way.

01:28:42 [BT]

That's Leonard Cohen. Yeah, Leonard Cohen. [24] No, I don't think we're in Leonard's league, but you know, you can always aspire.

01:28:51 [MH]

I didn't want to, but on the topic of maybe poetry, but certainly you know brevity like to me like you know, there's one of the arguments against brevity is a claim that it is counter... It's opposed to readability and I just wanted to, you know argue the other side of that for a moment and say, you know. I find with that, you know, one of the one of the great things about these languages. And certainly you know one and and getting going from q to APL was that there was another sort of gear of brevity I could get to and that you can... What I find is that, you know so. OK, two things. So one you know. For example, you know, what in a traditional programming language you would you know might take a paragraph or you know or a you know half page or whatever you know font size, these things matter. But you know a big chunk of code can be one line of code, and so yes, it's you really need to get, you kind of have to change your focal length if you want to see what's going on there. But there's but. You can get it on one line and then you can. And usually with enough space leftover that you can have a comment to the right of it that describes what not how you're doing what you're doing, but what you're doing or why you're doing doing it. And something about being in that sort of terse mindset makes me want to get that English comment as succinct as possible. You know, as simple as possible. But with, but no simpler you. Know how what is the just what is the poetic expression of why I'm you know the I'm whatever it is you're doing. You're gonna take the 1st and add 7 to it and blah blah blah you know you're you're doing something. And some people might argue, well the code is self documenting but I always think there's this other level where you're, you know why you're doing it, or what business problem you're solving, you know. Translating this into some other way, so you've got these sort of like 2 parallel narratives that are going on. Which is the the code and the comments, and that that brevity kind of creates space. So rather than interspersing them, you can have them side by side. Which is can be really useful. Like when I would in the competition when I would find a bug in. Something I was writing. You know I'd scan up the the comment side of the code to figure. They're like, you know, OK, this is. I've got a parsing problem here, so let me go find where I'm doing the parsing you know, and I can just go up that right hand side, find that and then switch back over to the left side of the code. To figure out where the bug is. So. And because things are so compact you can do other things, I just wouldn't really think to do in other languages. Like for example, you know this is sort of non standard styling. I suppose but. I will introduce a space somewhere where typically you wouldn't. You know if you if you say if you think consistency is king, you know you either put spaces around binary operators or you don't. And then that's just the way it is, but I would do things like I would put a space if I've got 2 lines of code and they have a relationship with each other. Like if they're mostly the same but slightly different, or something like that. They will line it up so that the similarities line up and so that the differences line up so that you you know you just you scan it, you see immediately. Oh, this is subtly different, but importantly different. And you know, if that's a one of these, double for loops you know, or something you're not. You're not really gonna be able to see those differences, but in these array languages you can do that kind of thing, and so there's like a different type of readability. That's that you get with such short all these. You know where the whole standard library is like is 1 character each or whatever you.. There's different opportunities to play with style and readability in ways that are that are not just pleasing, but I think like useful like add more clarity.

01:33:11 [CH]

I mean, you articulated this so well earlier, and that was one of the things I was thinking in my head. Is the because it's so terse you take the paragraph and you put it on a single line. And you mentioned earlier that that presents the opportunity to solve it several different ways and specifically you pointed out that there are there are different motivations you know. Does it perform. Am I going for performance here? Am I going for refactoring here? Am I going for the ability to parameterize this and reuse it somewhere else? And I think that's a lot of people they missed that. I gave a talk algorithms as a tool of thought [25] that that was the whole point is that I showed like 9 different ways to do this one short simple problem. And it's not about that like I'm going for the fastest one. I did do a performance analysis at the end, but like sometimes like you know, one of the ones I always run into is whether I'm gonna sort a list and then if I'm doing an N wise reduction with a non associative operator like subtraction. So I'm taking the difference between an increasing list of numbers. You know, if I wanna do the opposite where I'm sort of, I want I want positive numbers instead of negative numbers because that's a property of subtracting, you know, a left number from the right. I can either apply the self or the commute operator to the minus in APL. Or I can just reverse the list after sorting it. Or I could even try and get some sort function that passes it a custom binary operation with or the two different grades or whatever. Like there's very quickly three different ways that I can take my initial sort of and two N wise reduction with subtraction after sorting lists in APL. I can add reverse, I can add commute, or I can switch the grade operator. Like and it's not like: Oh well, performance matters sometimes. You know you're working with people that don't like the combinators, and they don't want to see a little selfie squiggly and be like what is this doing to my binary operation. Sometimes reversing after sorting is going to be more readable for anyways. It's just the ability to do that in APL or array languages is something that is not impossible. Like you can't rewrite something in C++, that's a single line in APL, like 3 seconds after you finish writing the other one, it's going to take you like a few minutes just to just to go and find the library with the function in it that doesn't require to use for. You to use for loops and stuff like that. Yes, so this is this is getting to the approaching the record of longest podcast recording. On that note, then, any final questions for Michael Slash. Michael is anything that we haven't covered, I. I feel like we should just bring you back cause we didn't even get into the contest side of things too much. You gave us a couple of highlights, but I feel like we could continue this for like a whole other hour or two. If we had the time but any. Final things you want to sort of say mention before we wrap this.

01:35:51 [MH]

Up, you know, I was thinking as I was preparing for this, I thought, you know, one memory I had was when a you know we had a PM. Like a like a project manager who worked with us who was just such a great guy to work with. Like he would he would literally you know at times. You know he was non technical, but like would sort of roll up his sleeves and become the guy that and how the the extraction tool for the data migration project worked and you know operator or whatever. So he was like he should have been a a programmer kind of thing and I remember at one point he was asking me what he understands that there's something there's different. It's a different way to think or a different way to program, could I help him understand what's different about it. And funnily enough, I also used potatoes in my example. And I it was something about I said OK, like I used a cooking metaphor and I can't. I said it was mashed potatoes or boiled potatoes or something. I said look if say, you were making boiled potatoes and. You know the traditional. Way to do it would be to. Like you, you take your potato. You wash your potato. You peel your potato. You boil your potato. You grab the next potato. You wash it, peel it and boil it. And then I said, but you know in you know in q or then you use array languages. You would wash all your potatoes, peel all your potatoes, boil all your potatoes. And he said, oh, great, you. Know that's, uh, I get that, that's really helpful and then I said but wait, there's this other level, but I don't think I'm at. It's like I'm striving to get there, that's why I keep pursuing this. Like there's this other level where you know you boil your potatoes and you filter it for skins and dirt you know and that's where we're all trying to get to.

01:37:51 [CH]

Alright, well I will start to wind this down because we are. I think we've probably hit the record here, but definitely thank you so much, Michael. For I know we've been trying to schedule to get you on for a a while, ever since you won the contest and the wait was well worth it. And yeah, I can't wait for our our listeners to hear this episode because I think they're going to absolutely love it. I will throw it to Bob to let people know you can contact us.

01:38:14 [BT]

contact@arraycast.com and we are always listening. People are coming up with some really good suggestions and we are pursuing those. MATLAB maybe in the future and also we got I think J schedule coming up because we've got the new version of J coming out. I think sometime in February. Not an announcement, but just letting you know you can reach us at contact@arraycast.com. Yeah, and we look forward to your interaction.

01:38:42 [CH]

And as always, if you want to like I, we've mentioned a couple times now. Michael's talk will be in the show notes other other resources people are you on. I'm not sure if you're on socials or anything like that where people.

01:38:53 [MH]

Not really, no. That the talk would be the the best that they're going to find.

01:38:56 [CH]

If he's not on social, that means we're just gonna have to bring him back. So subscribe and you'll hear the hear the next one whenever it's out. But once again thank you so much for coming on, Michael.

01:39:07 [MH]

That was my pleasure. Thanks for having me.

01:39:09 [CH]

And with that, we'll say Happy Array Programming.

[ALL]

Happy Array Programming!