Transcript
Transcript prepared by
Adám Brudzewsky, Bob Therriault, and Sanjay Cherian
00:00:00 [Yon Fernández de Retana]
So really my main motivation was bringing ideas from Perl and other text processing languages into an Array language and see how it turned out.
00:00:12 [Music]
00:00:22 [Conor Hoekstra]
Welcome to episode 102 of ArrayCast. I'm your host Conor and today with us we have our four panelists as well as a first time special guest who we will get to introducing in a few minutes. But first we're going to go around and do brief introductions. We'll start with Bob, then go to Stephen, then to Adám and finish with Marshall.
00:00:41 [Bob Therriault]
I'm Bob Therriault. I am a J enthusiast and there's lots of stuff happening in J.
00:00:48 [Stephen Taylor]
I'm Stephen Taylor. I'm APL enthusiast and when the wind's in the right direction, q as well.
00:00:54 [Adám Brudzewsky]
I'm Adám Brudzewsky. I do APL for a living. Lots of stuff happening there too.
00:00:59 [Marshall Lochbaum]
I'm Marshall Lochbaum. Worked with various Array languages and then I made BQN and that's that's my main one now.
00:01:07 [CH]
And as mentioned before, my name is Conor, host of ArrayCast, massive fan of all the Array languages and we're going to be talking about a new one today. I think it's been mentioned a couple times but we'll get to that after a couple announcements. I think we have three, if my memory serves correctly. The first one is from Bob, then we've got one from Marshall and then I believe one from myself. I know one from myself but there might be one other extra one if I'm forgetting. But we'll start with the one from Bob and then go to Marshall.
00:01:30 [BT]
The one for me is actually a little bit of a two-parter. [01] We mentioned the Array Portal on the last episode and or maybe two episodes ago and the fact it's a good way to find information about the Array languages, specifically APL and J right now, but I'm sure other languages are going to be added as Ed Gottsman gets ahead with the project. We'll have a link to how to get into the Array Portal but we got an email from Tom McGuire last week and Tom had been trying to create chat GPT in J and he'd been working on this for about six months, working out all sorts of things, learning about more chat GPT and then he heard about the Array Portal. He went in the Array Portal, he typed in chat GPT and he found that somebody had done it like a year ago. And so he sent in this email saying, "I've given up, I'm not doing J programming anymore because there's no point." And I actually emailed back to him and said, "Well, you're not giving up J programming." He goes, "No, I'm not giving up J programming." But I was pretty frustrated when I found out there was this already sitting there and I didn't even know about it. So that's kind of a plug for the Array Portal. But also now Tom has gone in and he's adapted what had been written before to be able to run on J and that's in the J Forum. So I'll put a link in for the J Forums. He's got a full listing about how to run it and what things you need to put in and you can actually run chat GPT on your personal computer in J, which is pretty cool. And that's my announcement.
00:03:01 [ML]
Mine is that back in November, I sat down with Christopher Augustus and Stephen Thames to record an episode of the ProglangCast. And now they've taken a lot of time making this episode shorter and it is now ready and released. So you can listen to this and actually watch this episode of me discussing all sorts of things about implementing languages, how I got into programming and Singeli and stuff on the ProglangCast now.
00:03:32 [CH]
And I can recommend it. I don't know if I was the first one to watch it. I was the first one to post, maybe the first and only one to post a comment. I mean, I haven't checked since so there might be more comments, but I very much enjoyed it. I learned some stuff, folks. You might think that as a panelist of ArrayCast, I know everything there is to know about Marshall, but apparently not. And the most exciting thing I learned is that I knew that music was a big part of your life. I did not know that you had, I've never heard of this Bandcamp website, but you have some tracks, specifically some BQN music. BQN tracks. And I went through and listened to them all. There's a couple of ban... I mean, they're all pretty good, but there's a couple of bangers. Joust was my favorite. I'm not sure if we have to need to have an offline discussion about maybe we've, I mean, we might've already started season two implicitly and we need to get a new jingle. I don't know. Anyways, we'll take it offline. I'm not sure if the royalty fees might be too much. They might be above the budget that ArrayCast has. But anyways, I recommend folks, and not just Marshall's episode. I think they've got like 10 or 15 episodes out so far. Many of them are on the Array languages, but they've got ones on Scheme, F#, other languages.
00:04:43 [ML]
Yeah. And they've got a mix of interviews and kind of programming exploration.
00:04:47 [CH]
Yeah. Highly recommend. It's a great, great resource. And we've also had them on the podcast. So if you haven't gone in the backlog and listened to that one, we will link it in the show notes. Last but not least, a quick announcement, Tacit Talk episode 27. We're flying up the numbers because we released like eight since the last time we had Kai on. All brought to you by Texas Speech, Speechify. Was with Aaron Hsu and Jon Smith. It was a continuation of, or a continuation, an extension, we'll say, of the conversation that I had with Aaron back in August while we were at Iverson College. We met up at the Eagle Pub and we'd recorded two episodes. One of them was about Tercity. I got a lot of great feedback about that conversation. It was mostly just listening to Aaron wax rhapsodic on the importance of Tercity and economical expression. And we got these comments from Jon Smith on the GitHub discussion. And long story short, or long story medium, I guess, we had him come on the podcast and basically have a conversation and go back and forth about, "Is Tercity a good thing or a bad thing?" Aaron in favor and then Jon kind of arguing the other approach. So that's online and it'll be in the podcast form probably by the time this episode, you're listening to this episode. And I think that's the end. So yeah, link in the show notes for all of that stuff. And with that out of the way, with the announcements out of the way, I'm excited to talk to our first time guest today. The name is Yon Fernández de Retana, AKA anaseto on the GitHubs and CodeBergs, where there is an array language that you may or may not have heard of called Goal, which stands for Go array language. It is an embeddable array programming language with a bytecode interpreter written in Go. And we're here to talk about that. So I think, I mean, I'm not super experienced with this language. I did in preparation for this interview, go take it for a spin. It does have an online REPL like all the best array languages do these days. So if you want, while you're listening to this, depending if you're on a run or doing dishes, but if you happen to be sitting at your desk in front of a computer, you can go check out the REPL and play around with this. And we're going to talk to Yon Fernández today. So yeah, we'll throw it over to you, Yon. Tell us maybe about your background in computer science or programming, if you have one, and how you got from there to implementing this programming language, array programming language, Goal.
00:07:22 [YFR]
Thanks for inviting me first. Actually, got into array languages quite late, like something like the first year after high school. And I originally studied mathematics in France. And my first programming language was actually OCaml, which is a bit strange because OCaml is like not the typical first programming language. But in France, it was used in some courses, in particular for people that did math and physics. And I did not do much of it because I was studying. But at the time, I learned functional programming and things like that. And it's only later when I discovered the world of open source and Linux and all that stuff that I kind of got interested in programming languages in general. And I remember, for example, that I was playing with Haskell at the time when I had to study for exams and things like that. But I always kind of like to play with other stuff, just as a hobby during my free time. And I kind of switched to a completely different language from like statically typed functional language. I went to Perl then. And it was a very different experience, like in many ways, people weren't interested in the world, the same kind of things. And like the orientation and the community was much more into like practical things. And that's something that's more in common with array languages. Because like in array languages, you have like a few types and it's a dynamically typed language usually. And you get that property that you usually directly are playing with like values and writing code. You're not writing types. So, that was something that I found interesting. I don't know. I had like a time I was exploring many different programming languages. But without like learning them very well. Like I had some fun with Lisp, Racket and other languages. And at some point, like I got to learn some J. And actually, J was my first array language. It was like I don't really remember how like what was the thing that made me want to check it out. I think I found some code on some forum or something. And I found it like how it's possible that so many things are done like in a single line. And I was surprised. And I started learning it. And something that I found interesting is that there was all that kind of vocabulary that was a bit linguistic. Like you have adverbs, you have verbs and like other things. Gerunds even in J. It's a bit of different take from linguistic perspective than Perl, which is also kind of inspired by like natural languages in some aspects. But in a different way. It was fun. And well, I hadn't really much time for J. Like I played with it during the summer for a few weeks or so. And then I had to continue studying and stuff. So it was only like later that I got into other array languages. At some point, I learned some k. But at the time, ngn/k wasn't still around. I think there was only Kona. It wasn't as major. So I didn't spend much time with it at the time. And it's only when BQN started to be out and all that I really started to spend more time on array languages. And that was after my PhD. And where I did things that had nothing to do with array languages. Like using proofs assistants and stuff like that. Even so, there was some relation with compilation. But it's not like I learned at the time how to write a compiler. I only knew how to proof like semantic preservation stuff and things like that. And so, something that I quite liked is the fact that array languages like Perl were like a complementary thing to all my more theoretical background in math and proof assistant and stuff like that. And well, if you have any questions, before I say more?
00:13:06 [CH]
I would encourage you to keep going. We've hit OCaml to Haskell to Perl to Lisp to Racket, then to J, then to BQN. And somehow you ended up through all of those languages choosing none of them to implement your array language. So I'm curious when Go shows up and why out of all these languages, Go [02] was chosen as the implementation language. This is like the beat drop of the song I'm listening to right now, of where you're going to be like, "And then I chose Go."
00:13:42 [YFR]
I mean, it's not a linear thing. But given that I don't have really a background initially in computer science and programming, I only got into that a bit late after doing math. And at some point, I needed to program some like more doing some programs. At some point, I implemented Markup Language and I did that in Go. My first version was in Perl, but it was a bit messy and there were some issues and I did a rewrite. And that's how I learned Go. It was, I think, in the beginning of my PhD, I got to learn Go that way. It was a Markup Language for writing novels and that's a member of my family uses, actually. That was my motivation. And later, because I had learned that language, I also did had some fun writing role-playing games, but the traditional kind, the ones that are like you play in your terminal and there are no graphics. And that's also, I think, something that kind of made me like k. It's the fact that I was used to look at like an exclamation mark and think, "Oh, wait, that's a potion." Like that's something giving characters different meanings and finding that to be like nature. Like in k, exclamation mark can be dict to make dictionaries or mod to make like different meanings. Like there are many characters like that that are repurposed, like to have new meanings different from the ones one would think. That's something that I think made me like using the ASCII character with respect to using new symbols. So, I also, Go actually has a couple of non-ASCII characters and that's something that also Role Plays often do. Like you have, they use ASCII characters when they can and when there are not enough symbols, you use some others as you need.
00:16:10 [CH]
So, you prefer a smaller set of symbols or in this case ASCII characters with more overloading than opposed to a larger set of ASCII characters or symbols with less overloading.
00:16:24 [YFR]
Yeah, you could say that. I mean, I kind of find that natural for me. Like it's a polysemic way to do things. So, it's kind of like in natural languages where have many meanings.
00:16:39 [BT]
We had Rob Pike on, I think almost more, maybe a year and a half ago, and he'd written an array language called Ivy in Go. Did you get a chance, or did that have any influence on you? I think he was using his more like a calculator, but did that have any influence on Goal?
Yeah. I mean, as a language, it didn't have really influence, but I had a look at the implementation because it's written in Go too. And the scanner where I tokenize the input is actually quite close to how it's written in Ivy. At least originally, now it's a bit more complicated because it's not exactly the same language, but the idea of how it's written is the same.
00:17:25 [BT]
I think you could do a lot worse than use Rob Pike's programming as a model for what you're putting together. So, I think that's a good move.
00:17:32 [YFR]
Yeah. I quite liked how it was written. So, I just compiled the style of that part.
00:17:38 [BT]
Yeah, yeah.
00:17:39 [YFR]
Of course, it's different because Ivy is a big number calculator, and it doesn't have a focus on the same kind of things. So, it has improved since back then, and now performance is a bit better. I think he used tree structure to avoid the quadratic complexity and things like mutating, which is a different approach than usually is taken in other languages. But in Goal, I took the classical approach of using a kind of ref count system for doing in-place optimizations when mutating arrays and things like that, despite the fact that they're not mutable from a user perspective. I think J also does the same or something similar.
00:18:32 [ML]
Yeah. So, to give a little detail, it's ref count is for reference counting. So, you keep each time you make a new reference to an array, you increment a counter in that array that says how many references there are to it. And if there's only one reference, well, you know that's the reference that you have right now. And so, then you can change the array freely because nothing else is going to see those changes.
00:18:57 [YFR]
In the case of Goal, it's a bit different from like usual because there's also a garbage collector. So, I only use the reference count system for arrays and other values don't need if they're completely immutable, they don't need to be handled.
00:19:20 [ML]
Yeah. So, you use the reference counts kind of as little as possible. Yeah. Yeah. BQN and Dyalog, [03] both because they have objects, they have to have a garbage collector too. But they kind of try to use the reference counts as much as they can.
00:19:32 [YFR]
Yeah. It's hard to do reference counting right. So, I try to limit the use.
00:19:42 [CH]
So, circling back to Go as the choice of implementation language, you were saying that you basically had programmed a markup language and also this role-playing game. And then in doing that, you know, you mentioned or when talking about that, you mentioned that you were kind of leaning towards k as the thing that you found most natural, which is I think probably why Goal is most k inspired. I mean, it takes, from reading the docs, it takes inspirations from, yeah, the scanning from Ivy and then also some primitives from BQN. But at the end of the day, when you chose Go as the implementation language, was that because you just had the most experience from the markup language and the game? Or was it it was kind of your favorite language at the time? Or what was, you know, when you were choosing what language you wanted to implement Goal in? And obviously, it's an embeddable language. So, it's a scripting language that you can embed in Go. Like, walk us through like what the thought process between Go versus any of the other languages that you had experience with over time.
00:20:53 [YFR]
Because I kind of had experience with Go, it was the easiest. But there are other reasons. There are not that many choices for implementation. Like, there are more, but there are not that many different choices. I didn't want to write it in a language like C without garbage collector and all, because it's more work. And I'm doing this as a hobby. So, and I want to, like, have a finished language that can be used. And I think I will have needed, like, much more time. I could have chosen, like, Java or some other. But given that I knew Go better, I used that. I kind of like some aspects of Go. In that it's a relatively simple language that has a focus on writing code instead of writing, like, types or declarations, classes, or I mean, that kind of stuff. Even if it's a verbose language, you're in, like, in array languages, you write code. You're more focused on manipulating the data and that kind of thing. Of course, it's very different. The thing that appeals me most in Go and also, of course, the fact that the implementation, the compiler and all is quite good. The tooling is good also. And it compiles fast. And I haven't had issues on my system. Like, I use usually OpenBSD. And I don't know. But I tried Java a few things at times, compiling things, Java things and stuff. And I had more issues. So, that also plays a role in why I chose Go, I think.
00:22:50 [CH]
Okay. That makes sense. And so, I guess maybe the next thing to do is walk the listener and us through what would you say, like, the biggest differences between k and Goal is, other than the fact that it's, you know, embeddable in Go, just from, like, a language point of view, I imagine that there's a -- I mean, I saw in your docs. I mean, I was about to say, I imagine. But I know for a fact. Because I went to your website and saw the headline. But there's a page in the docs that says differences between k and Goal. So, I'm sure especially the k developers, but the array language listeners in general will be interested. You know, what are the major differences between, you know, k's on average and Goal?
00:23:35 [YFR]
The main difference will be atomic strings. That is the fact that strings are handled as scalars instead of arrays of characters. So, for example, if you want to perform an operation on a list of strings in Goal, the primitives will apply to each string instead of applying pervasively at the level of character. So, I think that in practice, that's the thing that changed the most in typical script one can write. The way strings are handled is the most different. One thing I should mention is that at the end, it was inspired by k. The first versions that weren't really released were more like BQN-like. I wasn't even sure I was going to use ASCII instead of symbols and things like that. So, really, my main motivation was bringing ideas from Perl and other text processing languages into an array language and see how it turned out. I kind of had that idea for a long time that maybe it would be nice if there was an array language that handled strings like in a different way. The reason is I was used to Perl for writing typical scripts. And of course, the way typical tasks were done are very different from array languages. You want to put a string or remove some suffix, like dot something to have a list of five names or things like that. And so, I kind of felt that those kind of things for the kind of stuff I do are like built in, like should be built in, those kinds of operations. Even so, usually, things of built in sourcing that operates on numbers, like addition and stuff like that, there are some string handling functions that appear in almost every language. Like replacing some string, removing suffix or prefix or trimming spaces and stuff like that. They come in like every language. They usually are in the standard library or something, but they always are there. And so, why couldn't the same idea of using symbols for those kinds of operations, like it could be nice, like you don't have that in math or something. It's because in math you have notation for math related things. And that's what APL got. But I thought it would be like a fun idea to have basic string handling operations as primitives. And one thing, for example, when you have a list of five names, and for example, you want to append at the end of each five name an extension, like dot txt or dot csv. In Goal, you just write the name of your list and then plus and you write in between quotes, you put the string with the extension, and it will pervasively append the extension to all the five names. And that's something that actually even languages like Perl or other text processing languages don't have. That's something like you need to have both array language semantics where your operations apply pervasively to the level of either numbers or strings. So, you need both words to have that idea work right out of the box. You can do that in k or APL. You just need to append an adverb each and you get the same treatment. But you still have to have that in mind. Like you have to think about the fact that you have a list of characters instead of just thinking, oh, I have a string and I want to do something with that string or that list of strings. And I mean, it's difficult to say it's like a better way of handling this. But I kind of like it in practice when writing scripts and stuff like that. Which is the kind of things I usually do with Goal are not very complicated. Like I don't do much math stuff or things like that. So, that kind of having string handling features right in there with the primitives, it's kind of useful for me. That's what the idea. At the beginning, I didn't even plan to really make a complete language and go all the way. I just wanted to experiment with the idea of an array language with atomic strings. And I continued for two years and now Goal is more complete language.
00:29:08 [BT]
So, Yon, if the strings are atomic, if you want to work within the string, do you have something like ranks [04] that you can go in and break it apart or do you have a separate group of functions that do the work inside a string?
00:29:20 [YFR]
Yeah, you can get a slice of a string, but you can't get like an individual character as something different than a string. Like, if you can slice string and get one byte string or one or two byte string containing a code point or whatever, it's always like UTF8 encoded value. It cannot be split into something, into characters. You can split into strings that are each one character long.
00:30:01 [BT]
Yeah, then you've taken your original string, which is an atom, and you actually created, say, the string is four characters long. You'd have four atoms if you split it up that way. So, if you were trying to do manipulations, and I'm guessing you don't, your uses for the language, you don't need to do this too often. Because, I mean, you think about it, if you're doing mostly text processing, I'm not sure how often you're trying to go in and change the words or the text. You may want to be moving the text around. That's where the atomic area, I think, would make it really easy. If you're moving strings around as atoms, you're just sort of moving your way of thinking about text up a level. But if you did need to drop down a level, I guess what you do then is you break it apart into bytes and do your manipulations and put it back together again.
00:30:45 [YFR]
Yeah. You can convert to a list of bytes, and then you have numbers and you can process it like you would do in APL or other languages. But, I mean, I don't often do that in practice. But you can. It's the kind of thing you need to do in Advent of Code problems sometimes or things like that. But it doesn't come up very often in the scripts I write. But yeah, you can. Because it's important that you can do it if you have to.
00:31:22 [BT]
Yeah. I was going to say, you may not need to do it often. It's good that you can do it if you need to, because it's one of the functions you sometimes need to do. But you get the advantages with atomic strings, I would imagine, in a whole lot of other things that you're doing with that.
00:31:35 [YFR]
Yeah. I mean, most of the time I think of strings as atomic. In other languages, too, because even in, I don't know, Python or things, strings are usually like some kind of immutable array, and they think of it as a single thing, not a collection of character.
00:31:57 [AB]
This makes Goal's array model entirely identical to JSON, doesn't it?
00:32:03 [YFR]
JSON, I'm not sure. But yeah, it's similar.
00:32:08 [AB]
I mean, other than you have some implicit numeric types in Goal that JSON doesn't care about. But if I understand right, Goal doesn't care much. They're usually not to care about those, as opposed to k, where you tend to care more about those. So it seems to me that it's very similar, if not identical, to JSON, in that individual elements can be strings or numbers, and then arrays, which are just lists of lists, and then you've got dictionaries, but no tables like k usually has.
00:32:43 [YFR]
Well, actually, in Goal, there are not tables in the sense of a dedicated type. But in k, tables are actually internally just a dictionary of two arrays, a pair of arrays, a pair of keys, and a pair of values. And Goal represents tables as dictionaries, where the keys represent the column names, and the values are the columns. And there are several primitives that actually work in a special way for dictionaries that have keys that are lists of strings, and they operate in table mode or something like that. So you can do things similar to select and filter and stuff like that, to filter the columns of the table.
00:33:39 [AB]
But if the dictionary is really... then does that mean that the columns of tables, because you don't really have tables, they're really just dictionaries, the columns are not ordered? They are. Normally, I think, but in dictionary, the...
00:33:56 [ML]
So it in OK the table is not a table is like A2 dimensional type, so a dictionary.
00:34:03 [ML]
So in k, a table is like a two-dimensional type. So a dictionary, it's just keys and values. A table has columns where each column has a key. But there's also another dimension that's the rows, and those are just indexed. I mean, that's a whole different thing.
00:34:17 [YFR]
Yeah. I mean, it's different, but in practice, the only difference, not the only, but the main difference is that it's not enforced that all values are columns of the same length.
00:34:32 [ML]
Yeah. Okay. So you're saying represented as a dictionary where the values are lists.
00:34:37 [AB]
Yeah. But then don't you lose the ordering?
00:34:40 [YFR]
I mean, the keys are ordered and the values also, each value is a list.
00:34:46 [AB]
Yeah. The list, I understand, there's an order, meaning where if you take the nth element of every list, that's getting out one row, so to say, from the table. But you say even the keys are ordered. So a dictionary is considered an ordered collection.
00:35:01 [ML]
Yeah. It also has an ordering in k.
00:35:03 [AB]
Okay.
00:35:04 [ML]
And I mean, you're allowed to even have multiple copies of the same key and things like that, which may behave erratically, but it doesn't give you an error when you create it, at least.
00:35:13 [AB]
Yeah. Then it's not really the way I think of a dictionary, usually, if you can have duplicates in the keys. How do you...
00:35:21 [YFR]
Yeah. Dictionaries in k are quite particular. They're not the same as most other programming languages.
00:35:31 [AB]
I see. Okay.
00:35:32 [CH]
Yeah. I went and stored your help file, link in the show notes, if you want to take a look at that. And then I CADded it and grepped the regex, which I guess I probably could have done this better with goal itself if I actually had it locally installed, but ^s for a string, and then /b dot, basically. So it pattern matches on every expression that starts with s and then a character, a non-alphanumeric character, I guess. And then, so basically that gives you a screenshot of all the different functions, or I guess functions, maybe ASCII symbols that represent some kind of operation. And there's a ton of them. I'll quickly read through them, but concat, trim suffix, repeat, glob match, fields trim, count strings, cars/bytes, toString cast parts, rx, which I imagine is short for regex string. I could be wrong. Parse value format, rindex, index, two different substrings, two different CSVs, aka read and write, JSON, UTF-8, join and split, and then read. So a ton of stuff. And I'm on record during the Iverson College arguing that I hate having to write join. I just hate it. It's so common. I mean, you know, in Python, probably like split and join are like two of the most used functions. And it's only a few characters in your array language of choice. And not only does goal have them, but they are the slash and backslash. So overloaded kind of the fold and scan operations. So yeah, a ton of...
00:37:16 [AB]
That's just from k, right?
00:37:18 [CH]
Is that from k?
00:37:19 [YFR]
Yeah. Join, string, and split are already in k.
00:37:23 [CH]
Oh, maybe I should go... Maybe I haven't given kare.. uh k a fair shake. I haven't shaken it hard enough.
00:37:32 [ML]
I don't think kare is taken. If anybody needs a language name.
00:37:41 [CH]
Yeah, I mean, that's... Has anybody ever done... Not that this is probably the intended goal of goal, pun, initially unintended, but I will retrospectively make it intended. Has anybody tried Advent of Code in this? I guess if k has the join and splits. But also too, I mean, one of the... What was the name of it? Where it automatically converts... Yeah, parse I/N is the BQN function. And I don't know if it exists in the APLs as well, which is specifically... I don't know if it comes out of Advent of Code, where basically you're stripping the integers from a string or a sentence, which a lot of the times is basically what the Advent of Code problems require. It just... You have some sentence and you need to get out the list of integers that are consecutive sequences. It looks from my understanding that you have that as a primitive, basically, if you give it the...
00:38:40 [ML]
Yeah, well, and that's not built in to BQN at all. That's in... Might be here in libs, the library thing.
00:38:48 [CH]
Yes, but the point being is that..
00:38:49 [AB]
Sounds like something... What Quad VFI will do, basically, for free.
00:38:49 [CH]
What? What FI?
00:38:56 [AB]
I called VFA verify and fixed input.
00:39:00 [AB]
It's basically a field parser.
00:39:03 [AB]
Quad VFI, Verify and Fix Input. It's basically a field parser. So you could just give it the separation characters if it's not space, and then it will spit out all the values with zeros and the spots where it couldn't parse something as a number, but it also gives you a mask for whether it could parse or not. So if you just immediately take the result, which is a mask and the values, and just insert a replicate between them, then you get just the numbers.
00:39:25 [CH]
That's a lot of work. You made that sound simple, but that sounded like four or five different things going on there.
00:39:42 [AB]
That's not a lot of work. It's usually just the... Slash, slash. Slash, slash, Quad VFI and then your text.
00:39:43 [CH]
A replicate reduce?
00:39:45 [AB]
Yeah. Well, you're using reduce here just to insert, because it returns a two-element result.
00:39:49 [CH]
Right, and that's a fork? [05] No?
00:39:51 [AB]
No, no, it's it would be in the top because.
00:39:55 [AB]
No. It would be atop, because replicate reduction is one function, and quad VFI is another function that's in the top.
00:39:59 [CH]
Right. Okay. So, I mean, what is that? Quad VFI is four, six, I guess six characters. Maybe shorter actually than the BQN. What's the BQN function called?
00:40:11 [ML]
Two Nats, I think.
00:40:12 [CH]
Two Nats. Yeah, that sounds about right.
00:40:14 [ML]
So six characters.
00:40:28 [CH]
Six characters. So it sounds the same. The point is here, folks, we've gone off a little tangent, is that it's a single character. I mean, technically, if you include the three-character prefix that is necessary, quote, I, quote, in order to basically select the overload of extracting the integers from your string. But the point is, it's a quote, unquote, built-in in Goal.
00:40:37 [AB]
But does that discard... I didn't understand this. Does that discard anything that's not a number? No, it doesn't look like a number? Is it just literally converting something that looks... that's a stringified number back to being a number?
00:40:51 [CH]
Oh, that's a good point. I mean, that's a question for Yon. .
00:40:53 [ML]
Well, So what the what the BQ inversion does is it just treats everything other than a digit as as as separators. I mean there's no, there's no empty.
00:41:02 [CH]
A delimiter, yeah.
00:41:06 [ML]
Well, so what the BQN version does is it just treats everything other than a digit as... A delimiter. Separators. I mean, there's no empty field. So it splits on any number of separators. And so it pulls out all the groups of digits. And then there's another version that works with floating point values. So it'll recognize negatives and decimals and stuff.
00:41:18 [CH]
Well, we'll get Yon's answer. But in the...
00:41:20 [ML]
But yeah, the idea is it takes one string, gives you many numbers.
00:41:23 [AB]
And the APL one, it defaults to spaces. But you can give the left argument containing whichever delimiters you want. So you could just give it the alphabet if you wanted to ignore anything but digits. But Yon, can you answer this? The, quote, I, quote, dollar sign. Or I mean, we can just try it online, right?
00:41:45 [CH]
I mean, that's the thing is I just went and tried it and it does not... It does not seem to like anything other than the digits. So...
00:41:55 [YFR]
I mean, in Goal there are like several ways to parse a list of numbers or whatever. And several forms of like using the dollar primitive. One parses into integers. There's another that parses into floating point values. And there's also a special form that works as a partial inverse of the stringification function. And often can be used in things like Advent of Code because it recognizes spacing, separates, like in the language and so on. So it's a bit like writing arrays in k as literals. Like one space, two space, three space, 4.5 or whatever. And using this form, you can get it. And you can get also several lines at once because it parses as many lines as you give it to it. So, if you give it a file with each line has several numbers, you use that special parsing form with the "the" verb and you get a list of numbers. And if there is stuff that cannot be parsed, then you get an error, of course.
00:43:21 [AB]
I think I found... I looked through before this, went to record this, looked through some of the documentation, didn't read everything, but read quite a bit. And I think I found the most interesting was that the dollar sign is not a normal reshape the way I'm used to it from, I think, all the other array languages have basically the same definition of reshape. There's a slight thing with J that you have to revel first if you want the typical reshape. So people just get used to writing dollars and comma.
00:43:58 [ML]
Mostly you just... like that doesn't come up.
00:44:01 [AB]
No, I guess if there's already a list of elements and it doesn't matter.
00:44:07 [ML]
Yeah, I've decided that the J form is a lot more useful, even though I didn't use it in BQN.
00:44:13 [AB]
Oh, yeah. You've decided it's more... well, it's obviously more powerful, whereas the traditional APL form is a special case of that, which is just a typical one. But defining this APL form in terms of the J one is easy. The opposite is not true. But Goal is different. Goal doesn't do this.
00:44:33 [YFR]
I mean, ngn/k, for example, has a more traditional reshape. But I went with a cut shape with words like cutting lines or cutting columns, but you can't specify all the dimensions at once. I think some of the later cases do something similar, so they aren't released. So difficult to tell exactly, but I think some of the later cases do something similar.
00:45:06 [AB]
So I think I'm beginning to get it now. It's actually... I was over and over again looking through the Goal documentation, was thinking, oh, these are some good design choices. I like this. So that means if you want multidimensional reshaping, and obviously it's a list of lists in Goal because of the array model, then you just use the reshape multiple times.
00:45:34 [YFR]
Yeah. I think in some library file or something, there is a user-defined function that does the reshape, like using a fold over the reverse shape, the list of lists with the shape and cutting. I think one thing that is true about Goal is that I take the polysemic approach very far, maybe. Maybe too far. k in that aspect is already very polysemic because each verb does several things, depending on the type of the left argument. And Goal does the same thing, but because I have strings as a different type, too, there are more. I mean, it's a natural consequence of the choice I made with atomic strings. But it could be surprising, I think. Even so, I personally, I'm quite happy with how it is. It's something that can surprise. I mean, people already think that k does too many things sometimes.
00:46:47 [CH]
Stephen, you were going to, I think, ask a question.
00:46:49 [ST]
Yeah. I confessed at least a decade ago to only being in this business of programming languages for the poetry. And looking at Goal, I was looking at the things I thought I want to write. I was particularly attracted by the defining errors as a special data type, so that I can pass error results up and down through my functional programming results and arguments. And I don't need to special case all that. And I can write without exploding my flow of control with throwing exceptions and so forth. I thought that would clean up quite a bit of code. I like very much the field expression syntax, but my rough impression of qSQL, the SQL implementation blended in with k inside q, is that about half the work that it gets used for is pretty much covered by your field expressions, basically filtering stuff out. So always supposing you don't want to write an SQL server-like function close to the data, and you're trying to get this done in Goal, it seems you can get a lot of work done and avoid the SQL syntax by using the filter expressions and then do the transformations within the array language once you get it into the workspace. I know there are practical objections to that for the huge volumes of data that k and q aim to operate with. But in terms of keeping the language stripped down and being able to do pretty much everything you need within it, that I thought was pretty cute. But this is a bit of a confession. The thing that really first got me and sold me on Goal was your use of floor and ceiling to do upper and lower case in the text. And I thought that is so sweet. How the hell did we miss that?
00:48:56 [AB]
What do you mean by how did you miss that? I don't get it. k has.
00:49:01 [CH]
How did the other array languages not do that?
00:49:03 [AB]
Well, I mean, k does use floor to lowercase.
00:49:04 [CH]
Right.
00:49:07 [ST]
Okay. I didn't know that. q [06] doesn't.
00:49:08 [AB]
q doesn't? Well, q doesn't have a monadic floor symbol, right?
00:49:13 [ST]
Yeah.
00:49:15 [YFR]
That's something that ngn/k also does for floor. But Goal has also uppercase and ceil, not just floor. And yeah, I was thinking about the field expression thing. The funny thing is that when I started Goal, I was only thinking about atomic strings. There was no other thing in my mind. And I did that. And the first version didn't even have dictionaries yet. Well, at some point, when I started actually using the language for something like processing some simple CSVs I have and stuff, I kind of wanted to have something for processing like tables, but I did not want to have the full SQL stuff that q had because one reason is that it's a lot of work to put that into place. And the other is that it's not as minimalist in spirit because you get that other language in parallel with your array language. And so that's why I went with that special syntax to write expressions that are just simple syntax sugar for regular lambdas. But the variables you write, the names refer to actually the columns of the table or dictionary that you're passing to that function. And the thing is, there are some subtleties when defining that feature. I think there was a version of k in the later case that had some similar stuff for I'm not sure how it works internally exactly, but you could write I think it was with a colon and then you could write like a variable and it was interpreted as the colon of the dictionary. So you had an implicit application and field expressions in Goal do something similar and it gets like compiled to a lambda where each name gets replaced with an application on the default variable X. And that's how it works. And the difficulty was being able to refer to other kinds of variables and resolve like are you calling a user defined function or are you referring to the name of a column or are you referring to a local variable from the surrounding function? And the solution I use is to have a special prefix like for names, for variable names that tells whether this is a column name, a variable that needs to be a local variable that needs to be projected and passed as an extra argument to the function. Or if it's a global variable or local variable that has to be inserted as is without the prefix. And sometime to get used to at the beginning I think that's how I solve that thing. For example, I think the language Lil takes a bit different approach because they have lexical scoping and I think the names are resolved not sure exactly but around time so it can tell if it's a local variable or maybe it's actually a column name or something without requiring the use of prefix. The thing is in Goal it's just syntax sugar for lambda. So I had no other ideas for like resolving the names of variables and things while being able to tell to use the concise way to write queries referring to names without having to always prefix with the name of the table or dictionary. Yeah.
00:53:29 [BT]
When you started out with Goal and you decided to go with the atomic texts or text strings, did you do that because you were trying to explore atomic text strings or did you do it with a purpose in mind? Were you thinking of, you mentioned I think at one point you had a relative that was actually using this. Is that what you were going into that area for?
00:53:49 [YFR]
At the beginning it was just an experiment because I sometimes got a little frustrated with text handling in array languages. I know it can be done. But when you are already used to Perl and other languages that have built-ins that do this stream functionality or matching or whatever, I had that idea that what could happen if an array language had strings as a primitive type. I hadn't really thought about it more than that at the beginning. I mean, it's a hobby project. And as I go during the time, at the first I thought of dictionaries, it's complicated. I just have arrays like in BQN. But BQN has too much multidimensional complexity. I will go like in k. And I thought I will do something simple with just that new feature. And little by little I thought, well, the dictionaries are really useful. So, I mean, I'm going to add them. And then, oh, but it's not easy to use dictionaries to write queries without SQL queries in some cases. Because you have to refer several times to the table when you actually only want to work with the columns and things like that. So, I'm going to do something about that. And I haven't planned from the beginning most things. And sometimes I read something in the chat and it gives me an idea or some post about another language like LIL or whatever. And I say, I want that or I want something like that. And I just went following my feelings. It's not something like completely rational or prepared from the beginning.
00:55:58 [BT]
So, you're not finished with it yet. There's more to discover.
00:56:00 [YFR]
Maybe. I don't know. I mean, after a year I thought I had finished. And then I know I could tell you it wasn't finished at all. So...
00:56:11 [BT]
That's the danger of hobbies.
00:56:14 [YFR]
Yeah. Maybe I'm going to do something else for a while and then I come back. I often do that with my projects. So, I mean, lately I've been into like optimization and stuff like that. I was influenced by the BQN chat because they're always talking about like AVX, SSE and all that stuff. It got me curious. At first I had no idea about writing assembly and stuff like that. I had some background on that kind of stuff. But I had no experience with that. And I started like early January writing Go assembly using some kind of Mac. It's kind of generator, but it's quite close to the assembly. But it's a generator in Go. And I started with and in a couple of months at college, I wrote a lot of that. I'm not sure how good my code is, but it seems to I mean, it seems to that many algorithms go faster than before. So, I was quite happy with that. I haven't done AVX stuff still. Only SSE because my computer didn't have AVX. So, I did with what I had. Actually, recently got a better machine. I think it has AVX2. So, I could know maybe play more with that. It's not like I have a real need for more speed. So, it's not the focus of Goal. I mean, it's still nice when it goes fast. That's something I like about array languages is that you don't need to do things like so complicated, like just in time compilation or things like that to get good performance. And that's something I don't think there are other languages outside of array programming in the dynamic language space that have that property that you don't need a complicated interpreter for getting like decent performance, just writing an interpreter as a hobby. And you get something that goes quite well. That's also something that doesn't require writing in a low level language like C. [07] Because when you're writing an interpreter for a language like JavaScript or something like that, you need to do the non-tagging, pointer tagging and things like that. I mean, to get good score performance. And they do many things I have no idea about. I just read a bit about them. I think, I mean, some array languages also do that kind of thing. I think BQN does those non-tagging also. But the focus is not necessary for an as necessary for an array language to become practical. That's the feeling I had. And that's one of the reasons I choose also to do an array language that I don't need the same resources to get something that works.
00:59:44 [CH]
So do you know if there are a small community or large community of folks out there using Goal? And regardless of the answer, if folks want to join that community, regardless of the size, what is the steps that they need to take? I mean, I recall from looking into Ivy that there's like a Goal, I can't remember if it's build or run command. I imagine there's something similar for Goal if people want to get it on their machine.
01:00:12 [YFR]
Yeah. It's quite easy to install. You just need the compiler. There are no dependencies or fancy stuff to install other than that. And you're right to go build or go install. And that's all. And then from the community point of view, there are actually at least a couple of users that have written extensions. So Goal is written in Go. And Go in that aspect is a bit different than C because you can't really extend using a foreign function interface. You usually write either a derived interpreter or you embed Goal code into a larger Go program. And both are simple in those. But for example, there's an extension made by a user, who wrote a derived interpreter that has a SQL functionality, more time functionality, and some other stuff. And the community, everyone can do that easily enough. One advantage of Go is also that at least for me, it's an easier language to learn and write properly than C. So you can write an extension, like, without having to care about, like, the reference count, the memory, and things like that. For example, the Go API of Goal doesn't expose the in place optimization and ref count handling to the user. So they can, for example, introduce new types and things like that without caring about memory. And for example, if you define a new type for I have an example with zip files, for example, you define a type that represents a zip file system and satisfies some, like, interface. And then you can use from Goal using some built-ins that work on file system values.
01:02:32 [CH]
And is there a list of all the existing extensions I found on the GitHub? Or not the GitHub, the CodeBerg read me. There's a tooling section that mentions some projects. But it's unclear if that's the list of extensions that you just mentioned.
01:02:49 [YFR]
There are some that are in that repo. And they are not mentioned there. They are just on the repo in some directories. And they can be seen in the documentation of the Goal API. But, yeah, that's the extensions I did myself. But the others are just listed there. And as far as I know, there are two currently.
01:03:16 [CH]
Okay. We'll make sure to find links to each of those in case if folks want to not only take Goal for a spin, but to play around with the extensions and also the tooling as well. Any, I guess, final questions from the panelists? I mean, we've covered a wide swath of -- swath? Is that the correct word? Sounds like the correct word.
01:03:37 [BT]
Swath sounds good to me.
01:03:39 [CH]
Swath. It rolled off the tongue and didn't exactly sound right. Swath. You know, sometimes you say a word and you're like --
01:03:45 [ML]
The more times you say it, the worse it's going to get. Stop digging.
01:03:48 [CH]
Adám's got maybe a final question for Yon here. Or maybe just a comment on swath. We don't know. Time will tell.
01:03:58 [AB]
Do you use Goal just for one of scripting jobs? Or do you actually write programs that you'll run multiple times over a time period? And in which case, what's the biggest or most complex thing that you've written?
01:04:13 [YFR]
I have some quite short scripts, but as current jobs for climatic data, I have local climatic data from where I live. And I do some stuff with that. Then most scripts I write, they tend to be short, like for the various things I used Perl for before, I now use Goal usually. I have no big programs, more than a few hundred lines. I don't have anything more than a few hundred lines. I've done also like the Advent of Code a couple of years, but the two things are short, of course. For practical stuff, I just have a few scripts that run regularly, but nothing big.
01:05:00 [CH]
All right. Well, I'll be the first to say thank you, Yon, for coming on. It's always exciting to hear about a new array programming language.
01:05:10 [AB]
Relatively new.
01:05:12 [CH]
I mean, new for many of our listeners. I know Goal has been -- I think it's popped up a couple different times in our behind-the-scenes internal Slack where we discuss potential future guests. But yes, and I think I've seen Goal in the -- what do you call it? Bubble chart of influenced by and influencing languages. [08] So it's definitely been on my radar. But very cool to get to talk to the author of the language. So yeah, thank you so much for coming on. And I'm sure at least one of our listeners, if not more, will at bare minimum go to the online REPL, which as we mentioned at the beginning of the show, we'll be sure to link in the show notes if folks want to go around and play with this. And if any of our listeners happen to have thoughts, comments, questions that they would like either to ask the panel or to forward along to Yon, we will be sure to do that. You can email us at --
01:06:06 [BT]
Contact@ArrayCast.com is the way to get in touch with us, as Tom McGuire did to tell us he wasn't going to program in J anymore as a joke. But yeah, that's a good way to get in touch with us. We've had other really neat comments come to us, come to our ways in the last little while, which is nice. It's always nice to hear from people and some people suggest ideas for future shows and we certainly look at them. Sometimes it takes a while to get around to them. And we actually also have a list of people. It's funny, as the show goes on, more and more people become interested in it and we end up ending up with a longer and longer guest list. And thank you, Yon, for being part of that. Because hearing you extend the atomic text string out in a way that array languages weren't, it's that sort of thing that kind of pushes boundaries that, you know, as you say, you found out a lot doing it. But I think you found, as people look at what you've done, I think a lot of people will take that and maybe make decisions based on what they decide to do with their languages.
01:07:13 [YFR]
I hope so. And thanks also for inviting me twice because it's been fun.
01:07:20 [CH]
Yeah, it's been an absolute blast. With that, we will say happy array programming.
01:07:25 [All]
Happy Array Programming!
01:07:29 [Music]