History of Science and Technology Q&A (March 22, 2023)

Hello everyone. Welcome to another episode of Q and A about history of science and technology. And let's see, I see a number of questions that have been saved up here. There's one from Julie here. What's the history of AI? What's the first recorded example of artificial intelligence? Well, in terms of first recorded example, there are certainly plenty of mythological cases of, well, more like artificial. I'm not sure that there was so much of a distinction between artificial life and artificial intelligence back in sort of ancient times. But the idea of lumps of clay becoming alive and so on was certainly a thing. It's worth remembering that the concept of the fact that life only comes from other life was a comparatively recent realization. I mean, people thought that in a bunch of mud there will be spontaneous generation of living organisms and maggots and things like this. And part of the reason for that was until the 1600s, people didn't know about microorganisms. Before there were microscopes, there wasn't any knowledge of microorganisms. And although I think there was sort of an implicit. From the periods in the 1400s with the black Death and things like this, there was sort of some implicit knowledge of contagion and the fact that there were invisible agents of contagion, later germs and so on. But the idea that life could come from non life was still really a thing back, you know, until solidly into the 1600s, maybe even past that time. So this idea that life was not created creatable in an easy way was something that only sort of gradually came in. Now, of course, in today's world, one's interested in what's the origin of life on Earth. And back in the 1950s, for example, there were these experiments, famously the Miller Urey experiment, where you got a bunch of chemicals that sort of might have existed on the early Earth. And you put sort of a simulated lightning strike through it. And you discover you get other chemicals which are things sort of approaching the chemicals of life. I mean, one of the things that was discovered what in the 70s, 80s perhaps was that in interstellar space there are chemical reactions are very weird in that case because the density of material is really low. And what tends to happen is there'll be a molecule sitting on some grain out in interstellar space and some other molecule would sort of hit it and. And the things will have some reaction. But the result of all of that is that there are molecules of decent size that happen to be also molecules that appear in living systems. I think the mechanisms by which they come to be there are utterly different from the ones in living systems. But it's sort of notable that, oh, in interstellar space there are grains that have these molecules on them and so on. So and then the question of sort of how, how life arose from non life, which and we'll come back to intelligence in a moment, but how sort of life arose from non life, it's still a very much an open question what the details of that are and sort of how, you know, rna, DNA and so on came to be and how many alternatives there were. I mean, how much of what we see in life today isn't what was originally there? You know, we see in the Burgess Shale, for example, which comes from what, what is it maybe 500 million years ago, something like that. Examples of organisms from the, I guess Precambrian period when which were funky organisms that don't look much like anything we have today. I mean, it's kind of like, you know, the right handed versions of molecules in sort of the chiral structure of molecules won out in biological organisms. But maybe there were organisms that had the left handed versions of sugars and other molecules and those ones didn't win out and we don't know why. And the fact that there's five fold symmetry and lots of things, rather than sixfold, sevenfold, whatever else, is another thing where at some moment that won out in the history of life. But the question of sort of how the first organisms, that the first way in which different kinds of elements of organisms arose. My guess is that a lot of different mechanisms that exist in biological organisms sort of originated separately and then they all got combined together into this kind of package deal that is life. And I think we already see that in living organisms today. Our mitochondria that are the energy source for cells, those were separate organisms that became parasites in what became the whole line of development that leads to eukaryotes. And like us, one would guess that the things like the lipid bilayer of cell membranes was probably of a sort of a separate origin from things about the replication in DNA and so on. And these sort of came together as a package deal sometime a bit later, you know, all happening at least two and a half billion years ago, if not, if not longer ago. I think that the whole notion of having separate organisms as opposed to having a sort of mega organism that just sort of spreads out and has little pieces of it, that was probably a thing that had to be invented and maybe two and a half billion years ago or something now. But I think the whole Question of. Well, one can certainly imagine things like lipid bilayers forming from sort of not very biological kinds of things and more like bubbles that form and reform and, and so on. And obviously now we have sort of a technological version of something a bit like that in the lipid nanoparticles that are used, for example for MRNA vaccines. And there's increasing knowledge of sort of how to make things like, you know, cells artificially, the coating of cells. But in any case, I think that it's been a field, the whole origin of life field, field has been something where each different piece. People talk about clay, for example, where you can get a solid matrix and then molecules being swapped in and out and so on. And that, that becomes a good place to start doing some kinds of biological processes. But okay, so first thing is how did life come from non life? And that was all a bit muddled, still not known, but that's one piece. Then there's a question of intelligence. How do you get artificial intelligence? Well, I think this notion of sort of the disembodied intelligence idea, that is kind of a very ancient idea. I mean that's kind of related to the idea of souls and so on. The idea that there is a thing about brains that is not the concrete, not the explicit sort of biological tissue of a brain. It is something of the sort of the, the informational engram of a brain that's something of very ancient origin. The notion that there could be such a kind of an informational engram that is sort of in a sense a perfect immortal thing irrespective of its instantiation in actual biological tissue. But I don't think that the idea that that could be created artificially was really a thing. Because when people thought about machines in antiquity, for example, well, what were the machines? I mean there were siege machines for, you know, for all sorts of things in warfare. There was also this one example that we have of the Antikythera device, this clockwork computer that came from somewhere between 100 BC and 100 AD roughly found in this particular shipwreck off this island, off this Greek island. This particular device is a kind of many cogged small, you know, it's not a big thing, it's like size of a two hands or something. It's in a museum in Athens. Now the thing is a whole collection of cogs that seems to be something for computing. Well, was something for computing astronomical events and so on. I don't think anybody would have said of that. It's an intelligence like thing. Now I do know that Cicero wrote about the existence of such a device. And it was like many of these ancient writings, like, you know, the descriptions of Troy and Homer and things like this. People assumed for a long time that they were kind of mythical things. And it only became clear sort of in later archaeology that these things were real. I mean, this leaves out in the, you know, the question of Atlantis, you know, the city that vanished but now beneath the ocean type thing. Nobody's ever found a thing that is that and maybe it will be found. And that generally seems to be the case that these kind of ancient descriptions were about something. They weren't just sort of made up out of whole cloth, so to speak. But the Antikythera device was something. I'm not quite sure how Cicero described it, but he won't have used anything that we would recognize in sort of modern terms as artificial intelligence. Now, you know, by the 1600s there were lots of clockwork devices. I mean clock making was a thing and you could have all kinds of automata and they were called automata that would be kind of these models of people and animals and so on sort of all going around in a clockwork way, miniaturized into things like music boxes and so on. That was very much a known technology. Now again, very much a mechanical technology where not sure anybody would have thought of it as an intelligence. Now of course there were confusing things. When was it the 1800s, maybe 1700s, the so called mechanical Turk, you know, there was a chess playing machine that was, you know, a box, reasonably sized box that would sort of move chess pieces around. I'm not sure whether it was magnetically or some other way on a chessboard and it was sort of displayed around Europe and so on. And look, it's an intelligent machine. Well, it wasn't a machine. There was a small person inside the box who happened to be a pretty good chess player and who was manipulating the pieces. That's kind of why the name of the Amazon people doing tasks that are specified by computer, but it's actually humans behind the scenes. That's why it gets called Mechanical Turk. And hence the strange verb turking, referring to the way that people get sort of a mass of humans to do tasks that are sort of low level tasks in most cases, like labeling images. For machine learning and so on. So that was kind of a misfire in a sense in the history of artificial intelligence. A note are not really artificial intelligence, but looked like an artificial intelligence. So then coming much forward from that, I'm not sure that. Well okay, so when people were first starting to think about electronic computers, which was in the 1940s, the. The common description was giant electronic brains. So that was kind of the sort of the ready description of what was going on that people gave to explain what a computer was. Computers had started to come into existence with the eniac, with Atanasoft's machine, things like this. They were things that had arisen from the use of electromechanical technology. So first of all, there had been mechanical calculators, kind of clockwork calculator like devices. Then that became electromechanical, then they went electronic. Then there was the idea of a stored program computer where you could have a purely electronically specified thing that would determine the settings for effectively the switches in the computer to determine which operation would be done and things like this. And that was very much by the end of the 1940s, that was sort of a known thing. And then the first kind of production computers started existing in the 1950s. And that was, say. And when people were writing about these things, they were usually describing them as things like giant electronic brains, which kind of made it seem like the computers were doing something that was an intelligence like thing. And for example, it was a very common description. Computers were faster than thought. So they were thinking, but they were faster than thought. They were electronic, as opposed to the way that our brains are sort of the electrochemical processes in our brain brains. So I think from the very early days of computers, people had the idea that computers were somehow intelligent like brains. And the fact that computers were something very different from brains, I think, is something that only gradually arose. I mean, computers, as a practical matter, were first really used more for their database qualities, a little bit for their mathematical computation, but also for their database qualities. That was the whole sort of mainframe computer tradition. A lot of that was about kind of database uses. Some of it was about actually calculating things like trajectories of spacecraft or things for engineering purposes, of stress in structures, things like this. But I think by the time there were personal computers at the beginning of the 1980s, end of the 1970s, the question of what was a computer for? Well, it was for playing computer games, it was for doing word processing, it was maybe for doing spreadsheets. None of those things looked very intelligent. Like, they didn't look very mind. Like now there had been sort of a tradition among people sort of thinking about theoretical computation. Like Alan Turing, for example, in 1950, wrote this paper about what it would be like to have sort of artificial intelligence, what it would be like to take this computer, which he had at first imagined purely theoretically in 1936, as the Turing machine, the idea of a universal computer. By 1950, he was actually exposed to an actual computer in Manchester. And so electronic computers were starting to be a thing. And he was kind of extrapolating from that. What would it be like when computers were really intelligent, like people? And I think that that was his. What was it called? Intelligent machinery? I think something like that was a piece Alan Turing wrote in 1950, which famously introduced the so called Turing Test of if you're talking to a thing, it will be called a chatbot. Today, if you're talking to a chatbot, can you tell if the chatbot is a chatbot or a person, a mechanical turk person, perhaps on the other side of the partition. And I think just in the last few months, I think one has probably got to the point where that is rather difficult to tell. And one's way of telling would have to be very sort of quite subtle. You know, ChatGPT does a pretty good job of being. It's a little bit, you know, it's a little bit bland as a human in some ways, but it is quite human like. And I think it would pretty firmly pass pretty much any version of the turing test from 1950, but at the time. So Alan Turing defined a number of things in that very paper that were sort of tests for was a computer intelligent. One was this Turing test of the conversation type thing. Another was, for example, playing games like chess. Those were things which seemed like they were sort of things that would be characteristic of something intelligent rather than a mere calculational device type thing. Well, then, 1956, there was sort of an important thing which was a conference at Dartmouth, I think, about artificial intelligence with the word used. And that conference was organized, I believe, by John McCarthy, Marvin Minsky, perhaps Claude Shannon, all kind of major figures in this story. And they sort of. There were several different traditions that were coming together at that time. There was the tradition that came from finite automata. The idea of automata I'd mentioned were already these sort of mechanical, clockwork type things. Then there was the notion of switching systems. Then Claude Shannon, for example, was pointing out Boolean algebra is the same as switching systems. Then there was kind of the idea of finite automata. This was kind of a mathematicized version of. Of those kind of clockwork devices. And that was kind of one tradition that came in. Then There was the McCulloch Pitts neural net tradition that had been introduced in 1943 by McCulloch Warren McCulloch and Walter Pitts as their kind of idealization of what was observed to be going on in brains. What Romani Kajal had seen around 1900 as this sort of microscopic structure of all those neurons with all these kind of wire like tentacles coming out. That was another piece of the story was introducing this kind of idea of that there will be a sort of a formal model of that that was equivalent to Turing machines. So it was sort of known to be computationally capable, but it was represented in terms of these things that were kind of neuron like in sort of a formal way. So that was another piece of what sort of came into the. The AI world was. And this was all kind of connected to things like oh, I know telephone switching equipment, which again had these sort of long wires where things had to connect to things and so on. And then that was. John von Neumann had been involved in this, wrote a book. The last thing he wrote, I think in his life. He died in 1957 if I remember correctly. And his last book was called the Computer and the Brain was based on some lectures he gave at Yale. And in that he's talking a lot about McCulloch Pitts Neural Nets, about Turing machines, about sort of what can computers do and how brain like are they. And in fact in the early things that von Neumann wrote. Von Neumann was a consultant for IBM, among other things on the design of computers, famously making statements like the United States will never need more than five computers and some other things which weren't very successfully forward looking. He had reasons to say things like that. I mean maybe it's true that the US won't need more than five cloud providers. And that was kind of the way that it was being conceptualized at that time. The idea that that might be for example a widely distributed, a small or cheap personal computer was not really on the horizon yet. So maybe John von Neumann was right in the end. If there are only five large scale cloud vendors, in the end is he's justified. But in any case he was interested when he was first talking about computers. He described many of the devices in computers as organs. So he would talk about the and organ and things like this. A very biological metaphor for what was going on in an electronic computer. All a bit kind of confusing, kind of this is it thought, is it brains, is it electronic? Computers are brains doing, thinking like things. But 1956, this conference about artificial intelligence, sort of the other cast of characters involved with that, John McCarthy, these are people I knew. But John McCarthy was the person who developed LISP in the late 1950s. But he was kind of a. He came from a sort of formal mathematical tradition with recursive functions and things like this, but later was kind of turned into. That turned into kind of this symbolic AI traditional of being able to specify how intelligence works by things that were a bit like mathematical logic, where you would actually be describing kind of operations in a formal way, with kind of sort of formal, symbolic way. Marvin Minsky, another character who was involved in that 1956 conference, he done his PhD thesis at Princeton about neural nets. Actually, people didn't really know quite what to make of it. But he'd sort of started off in this. In this more neural net learning. I mean, to just explain. There's sort of the notion of you can compute things, you can do an explicit sort of symbolic computation. That's sort of plan A, Plan B is you can sort of vaguely learn things and you can sort of statistically approximate what's happening. That's kind of the plan B neural net type idea. In modern times, I would say that kind of the ChatGPT world of large language models is kind of the evolved version of statistical AI. Kind of the things that we do in Wolfram language and so on are probably the evolved version of symbolic AI. Very interesting to see those things be connected together, but that's a different story. But the thing that is, back in those days, sort of John McCarthy was sort of emerging into the symbolic AI, would eventually develop lisp and so on. Marvin Minsky started off in the. In the sort of statistical AI learning things, but then went into a sort of an even more symbolic kind of description of AI where he really was more, I would say, trying to give a conceptual framework for how one could think about thinking. And he would talk about things like frames. I never really completely understood what frames were. I asked him about them many times, but I'm not sure. They're kind of ways. And there were. I forget what they're called. K lines. Maybe they're different kinds of descriptions of sort of the thought process. I remember asking Marvin actually once one of the last books he wrote in his life was about emotion. At least I think that book came out. He certainly. I remember him talking about writing it. And there's sort of the question of what was the type of description? What was the kind of thing he was trying to do? Was it something where you could write a piece of software that would follow this and be a thing that would do emotion, like stuff? Or was it more as I asked him something More like what Sigmund Freud had tried to do, where you're giving kind of a conceptual framework for thinking about these kinds of things, which would be as a separate problem, go from the conceptual framework to an actual implementable kind of thing. I mean, it's a little bit like, you know, when it comes to natural selection and, you know, Darwinian theory and so on, that's also kind of a framework in many ways. It doesn't, you know, just saying natural selection operates, here's how it works. It's a kind of a description of a process, a description of a mechanism. But in and of itself, it doesn't let you start writing down equations. And that's a separate overlay to go go from that conceptual structure to the equations. And in a sense, sort of one approach to artificial intelligence, kind of the Marvin Minsky approach in his later years at least, was more of this what one might call a psychological approach, where you're trying to describe structurally what can happen and it's then sort of a separate problem to go implement that. In the case of Marvin, one of the things that sort of came out was this idea that sort of intelligence is all about making connections between things and so on. And that did emerge into a piece of actual sort of technology that can connection machine, parallel computer that I was a consultant to the company Thinking Machines that made that computer in the early 1980s. Unfortunately, that computer, while it had many aspirations to be involved with AI really ended up being involved with much more, in a sense, mundane, supercomputerish kinds of things, which it wasn't particularly architecturally suited to. But that was kind of a development of kind of Marvin's concept of, you know, things are. It's all about thinking, is all about connections between things. And pretty soon it was an actual thing with ASICS and giant black computers with big power cables and so on. But the thing that, so 1950s, these kind of things are sort of quite theoretical in many ways. In the neural net side of things, there was kind of the perceptron idea. A person called Frank Rosenblatt was the big person behind that. That was a kind of a hardware implemented neural net where you would actually have electrical wires and thresholding units and things like this. And particularly for vision applications, you could say you could do optical character recognition. You could say, what characters are these out of digits and things like this. And that got a certain distance. It was not so much pursued at a theoretical level. Well, it was not so much pursued at the level of developing large scale sort of computer Software along these lines more. It was a hardware implementation then Marvin Minsky and Seymour Papat. Seymour Papert was the originator of the Logo language, for example, wrote a book called Perceptrons which for many years killed the field because it basically was a book that mathematically analyzed perceptron neural nets and showed that sort of one layer neural nets couldn't do anything more than essentially linear algebra kinds of operations or linear programming kinds of operations. And so people said, oh, neural nets don't do anything interesting. So that kind of killed the statistical approach to AI for quite a long time. There continued to be a tradition of machine learning that name machine learning wasn't a thing people really used much back in the 1970s, even 1980s. But there were a number of techniques, self organized maps, associated memories, all kinds of different methods which were all really about approximate pattern matching of some kind. There was this kind of idea that was central to databases of oh, you've got this particular thing, you can find a hash code for it, you can go look that up, you can go find it in the database. That was that sort of precise information retrieval was a, was a well understood thing. The idea of kind of fuzzy information retrieval was something for which there were a whole bunch of different methodologies. Not so much neural nets. Neural nets weren't so much of a thing there, but they were different techniques that may yet come back for doing those kinds of fuzzy lookups. So that had been a tradition, 1960s, 70s, 80s kind of thing. Meanwhile, in the, in the world of AI, AI has come and gone multiple times. But there was a sort of another thing in the 1960s was language translation and text retrieval. So language translation had been this thing very much kind of a Cold War type sort of energy put into it for that reason. Famously there was this idea that oh, if there was some sort of high level Soviet American communication, maybe the translators who were in the room would kind of throw the whole deal by mistranslating things. So this was a US government kind of idea. Let's get machines that can do the translation because they'll be more reliable and there won't be any human in the room to kind of throw the deal. Of course it didn't work out that way at that time. And it's, you know, now language translation by machine is quite good. But I'm not sure that I think most people still to this now would be much happier to have the human making sure that nothing got horribly mistranslated rather than machine translation. But in any case, times were different then. And the giant electronic brains were sort of predicted to and expected to really kind of take over from humans. That was a thing from the 1950s that it's just brains are coming, electronic brains are coming. And there had been, you know, there were a bunch of demos. Another famous demo was Gray Walter's turtle. This was at the World's Fair, 1955. Maybe. It kind of was notable enough to make itself to get itself onto the COVID of Time magazine and things like this. It was a hemispherical turtle shell thing that sort of a hemispherical shell and it had wheels underneath and motors and it had some photoreceptor, some photosensors as eyes. And it was a thing that tooled around on the ground, you know, doing things like following lines, using. Using methods that essentially came from control theory, came from the kinds of things that Norbert Wiener had done in World War II. The things that had been done for, for anti aircraft guns and things like this as a way of given a stimulus, how do you respond to the stimulus? Something which was essentially an analog electrical process analyzed using control theory and analyzed with kind of the overlay of this notion of cybernetics. The, you know, cybernetics being defined at that time as the sort of the analysis of information and control, I think in humans and animals and machines, I think was sort of the definition. But that later on segued very much into the very directly mathematical and calculus like area of control theory. But Gray Walters turtle was a notable cover of Time magazine. Like Creature that really did things which today would seem quite elementary, like line following and so on. But that was another kind of artificial intelligence is coming type message from the 1950s. But okay, then we get to like the 1970s. Well, actually, I should explain another piece to this. In addition to the interest in language translation, there was also an interest in text retrieval. That really wasn't a funny field because it was sort of somewhere in between library science and maybe a little bit of computer science. Ish stuff. It was. Oh dear, I forget her name. A woman in Cambridge who was very instrumental in, in developing the ideas of, of text retrieval in the 1960s. But any case, the, I forget the name. The, in any case, the, the. The. The next question was sort of you got a big block of text and you know, how do you search for words? How do you retrieve things out of that block of text? There was starting to be decent blocks of text. I mean, there had been the Science Citation Index, which was originally done by hand, where that was Gene Garfield's creation in the 1950s, I guess, where it was like, there's all this scientific literature. How are you going to find things? He had this idea of just find everything that cites a given paper. And then those were printed things. Back when I first used them in the 1970s, there were printed science citation indices where you could go look up a paper and you could look up all the things that cited that paper going, going into the future, so to speak. And this whole, that, that whole thing turned into this giant sort of network, citation network and led to, you know, kind of the bibliometrics idea and so on. Gene Garfield told me, well, he was, he was kind of, by the time search engines existed, they were kind of a, a different approach to finding things. But Gene Garfield pointed out to me that he had invented PageRank, the kind of algorithm sort of popularized by Google and so on as a way to sort of estimate the importance of things. He'd invented PageRank as part of his work on bibliometrics in the effort to do kind of citation analysis on scientific literature. But any case. So there was another tradition which was the kind of text retrieval, information retrieval. Go search for this piece of text. Back in the day, I was all very disappointed when search engines came out because I had been used to old fashioned text retrieval systems. For example, the dialogue system that was eventually bought by Lockheed, I think that was a system that allowed you to search the abstracts of scientific literature. And it was something which had. It was a little bit of a hostile user interface, I have to say. But it had very kind of regular expression like search syntax. And you could do things like you could say if this word, a word break, you could put stars in the word to indicate arbitrary characters and so on. When search engines came around and it was just like the plain text, it's just the words, you can't put regex. Regular expression pieces. I was a little disappointed by that. But any case, that was kind of another tradition, was a text retrieval tradition. That's what by the way led in the end, the sort of a pretty direct line from that to things like the IBM Watson initiative. That had been a big text retrieval effort at IBM. IBM had been much involved in those kinds of things back from the 1950s on. I mean, IBM was sort of involved in the whole sort of large scale databases thing going back to their very early days in the early 1900s with people like Herman Hollerith and the kind of automation of the census, of the taking of the U.S. census where they'd been really punched cards. Punched cards have been invented in 1800 for looms by Jacquard. But by that time they were being used to. You sort of punch the information of the census and then you can literally have a mechanical thing that would pick up all the cards that have this hole punched in them corresponded to, I don't know, people of age, such and such or whatever it is. The notion of a tabulating machine was sort of the early. IBM's early devices were tabulating machines used for the census computation and so on. But any case. So, okay, we're now in the 1970s. We've had text retrievals come into existence. There's sort of the things like Lisp. That was still not very practical, but it was kind of this way of describing kind of symbolic operations with sort of at least a nod to kind of ideas of mathematical logic, which gave it least a nod to ideas like predicate logic and all the way back to kind of Aristotle and his kind of way of using logic as a way to kind of symbolically describe things about the world. Okay, so then we're in the 1970s. One of the things that became a big thing at that time was so called expert systems. It was a bit of a mushy concept, but kind of the idea was somehow encode in a software system kind of the knowledge of experts. And people had this kind of view that they would interview the experts. I remember kind of oil exploration experts being interviewed and it was like, oh my gosh, we're going to lose our jobs. Because after we've been interviewed once, all of our knowledge will have been sucked out and put in the giant electronic brain that is the expert system. And then that's all that will be needed and the expert system will be able to figure out everything that experts were previously supposed to do. Needless to say, it didn't work out at the time as had been hoped. In fact, there were a number of companies. In fact, my first company ended up as a result of venture capital pressure. My first company was originally called Computer Mathematics Corporation, later changed its name to Inference Corporation. Sounds rather modern nowadays, I suppose. But I thought it was kind of the. The concept was to sort of. It started off kind of as a company to commercialize my first big software system, SMP, that I developed in 1979-1981 that was kind of a forerunner of Mathematica and Wolfram language, a technical computation system. But under venture capital pressure, the company kind of decided to sort of add on a big effort in expert systems. And tried to make kind of a, a language for expert systems which I was not really involved in. And I have to say I never really understood. You know, I don't know whether it was any good at all. But any case, it was used for a number of early significant expert systems for the early credit reporting systems were built by my first company. And a bunch of things for, for NASA for checking things for, you know, is are we ready to launch the spacecraft type thing and a bunch of other kinds of. There are lots of inputs and you're kind of going to make rules that determine whether you do this or that thing. Honestly, it would have probably been better to have used S and P as the framework for setting up those kinds of rules and the actions from those rules. But, but that wasn't what was done at the time. One lives and learns. I have not in the last 40 something years. Notably, my company is not a venture capital investment based company. I kind of learned from that experience that if you have a vision for what you want to do, then you should be responsible for it and not have other people telling you oh no, no, we saw in the newspaper that there's another, there's a cool thing you can do. There were newspapers back in those days and you know, maybe the company should go chase this, this new shiny object. Not, not, not the best thing. I, I would say that company, just if you're curious, that company eventually in the 1990s actually went public in a rather undistinguished IPO and the, and was rapidly gobbled up by a long chain of other, other companies. But in any case, the, that was, that was another kind of moment was kind of the expert systems moment in the 70s, 19, late 1970s, 1980s and so on. Which brings us I suppose to beginning of the 1980s there was a, another burst of enthusiasm for neural nets, but. And the beginning of kind of using deep neural nets and so on. Multi layer neural nets that avoided the Minsky paput. Nothing interesting happens with perceptrons type thing. And people came up with pretty elaborate, increasingly elaborate kinds of constructions. But for neural nets, in the end, most of those elaborate constructions don't matter. And in the end the things that were sort of the early things that were done when you bash them hard enough with, you know, efficient enough, GPUs, CPUs, whatever, it's enough. But any case, back at that time there was sort of another burst of enthusiasm for neural nets. Beginning of the 1980s, I would say that some part of that enthusiasm segued into Kind of a. Let's use neural nets as a way to think about modeling of brains rather than let's use neural nets as a practical way to. To do something that would be sort of AI like. But any case, the, the thing that maybe I've told this the story separately but, but kind of by sort of the whole neural nets are a thing. Yeah, they were always a thing. People like me didn't really think they were going to do anything terribly interesting. I'd looked at them. We had packages for Wolfram language and so on. Then Mathematica that did neural net kinds of things. They were like, well, nice to have, but they're not that exciting. And it wasn't clear whether they were things that would be of technological software significance, whether they were mostly things that would be of significance for understanding actual brains. There had been some technology that had been built using neural nets, for example optical character recognition. The standard software that was doing that ended up being neural net based. That happened I would say by the 1990s for sure. But it was still the case that even in sort of object recognition things like this, that, you know, kind of hacks based on image processing outdid things that were based on neural nets. And that happened. That continued to be the case until 2012. 2012. Jeff Hinton and his students. Jeff Hinton had been a person who worked on neural nets back in 1980 when that sort of burst of enthusiasm at that time that had kind of trained the steep neural nets kind of left it training for a month or something. And when they kind of came back, turned out the thing had done pretty well at learning how to classify a few thousand different kinds of things in images. The so called ImageNet competition. And it won the ImageNet competition in 2012, I guess. And that started this big enthusiasm of deep learning and this whole sort of idea that there are going to be neural nets that do all these different kinds of things and first for object recognition, then lots of different applications that. Well, then, then I forget the ordering of them. Another big one had been with speech to text. Speech to text had been a thing that had really been started in the nineteen, well, sixties I guess to nineteen seventies. This whole speech recognition method. There was a lot of interest in doing speech recognition. IBM had a big effort in speech recognition, for example. Curiously, most of the people from that group ended up going to Jim Simons Renaissance Technology hedge fund. Perhaps because one of their early ideas for kind of predict the market was actually listen to the sounds on the trading floor at the Chicago Board of Trade to try and predict options prices moments before they would sort of show up as actual transactions. But in any case, that was also a general idea of speech recognition as an example of pattern recognition. And that became important in quant finance and so on. But there had been this long tradition of kind of speech to text. And typically what people did in speech to text was they would be the whole sort of audio processing part. There will be, let's break it into phonemes. Let's kind of match up, you know, this group of phonemes. It became more like a test retrieval story of this group of phonemes would be, would be put together to make, make a particular word and so on. And that was kind of a thing that was. Then there were companies like Dragon Systems was an early company that made, made speech to text systems. I think that was one that had managed to get speech to text to run on PCs. It was, it was always a big deal to have speech to text run on PCs because it was a very time intensive thing. And it also mattered a lot what the quality of the audio was. And then there started to be neural net systems for doing this. And pretty much the whole sort of structure of very elaborately built speech to text, statistical speech to text systems kind of fell away to something where you would train end to end with neural nets to do it. I mean, another thing that was a big deal. This must have been around. Well, actually this was already in 2010 when Siri first came out. Siri was originally a part of an outfit called sri. It was a sort of a research object. And actually it's kind of a core of. It was really the Wolfram Alpha computational knowledge engine. And then it had sort of overlaid on top of that a voice to text system that came from Nuance. And then the. But Nuance's system, which had come from I think, Dragon Systems, I think Nuance had kind of bought up a lot of the different companies that were in the speech to text business. Some people complained a lot about its business practices about doing that. But I think another one was Verbex Systems and a whole bunch of others whose names I probably don't remember now, but any case that it sort of rolled them all together and had this speech to text system that was the thing used for sort of the original Siri before Siri was part of Apple. But already by that time speech to text was starting to work fairly well. And that was before neural nets really started doing speech to text. And the big sort of innovation at that time was that you could run Multiple possible interpretations of the speech all in parallel, and you could kind of compare which one worked best. And that was really a matter for sort of how powerful the cloud servers were. But then by mid 2010s there were really end to end neural net speech to text systems. And that became quite a commoditized kind of thing. The notion of having, well, a series of different kinds of things became sort of things that neural nets could do. We're still not quite at the handwriting recognition stage. That's still a thing that's out there. We're not quite at the, at the good voice generation stage. That's still under development. The sort of, the big excitement of recent times has been text continuation or the kinds of things that ChatGPT does is fundamentally a text continuation system that have been again, a long series of language models that tried to continue text that date. Well, you can think of them dating all the way back to the 1940s and Shannon's work on information theory and on figuring out kind of the redundancy of English text through things about text retrieval and so on, that those were all through things about speech recognition. These all used language models. And that's something which now by sort of taking the whole web and lots of books and so on, it becomes possible, it's turned out it's possible to train a language model that also has sort of backup knowledge in it that can do very impressive things. I think that the thing that is a surprise with ChatGPT is its instruction following capabilities and the fact that it can sort of do what one tells it to do in some sense and not sort of get, get lost off in the, in the wilderness and so on. That's the thing which really has made it sort of pop at a Turing test level in a way that really hadn't happened with, with earlier language models. And I think it's sort of in some sense more a question of engineering than it has been a question of other things to really add that extra layer that's, that's made that really work so well. But any case, the, the, you know, so anyway, the sort of, by the 2010s, you know, neural nets sort of were a thing. Meanwhile, on the kind of symbolic side of things, it's kind of pretty much us with, I suppose, a large part of the, the effort there of building, well, first of all from language as a kind of computational representation of the world and being able to actually take kind of the knowledge that exists in kind of formal form and be able to make it something that you can compute things with. And then WolframAlpha from 2009, which was the first successful natural language understanding, kind of natural language to computational language to computation system, and in fact remains really the only one that does that in a sophisticated way. The, you know, people had tried to make sort of language understanding systems back. It had been kind of a common AI kind of idea to do that. But the problem was, what does it mean to understand language? What does it mean? What is the output? What is the thing that you get when you say, yes, I can understand language? And what we had when we built WolframAlpha was we had a target. We know we were taking sort of human language and we were converting it into this computational language from which we could do computations. I mean, I have to say, to give an example, I should emphasize that in the 1980s, for example, AI was a small field. There were not very many people doing it. You could kind of name. There was the MIT group, there was the Stanford group. There was maybe. I'm not sure if Carnegie Mellon was yet really a factor there, but there were. There were these groups in different places. There were a few different traditions. There was a cybernetics tradition in the Soviet Union. There had been some. There were different traditions in different places, but there really wasn't a large scale field of artificial intelligence. And there were these individual groups in different places. I remember visiting a group at Yale, this must have been the mid-1980s. And I can tell a weird story about that. Who give some sense of the state of artificial intelligence research. They said, yeah, I drop in and that's like, this is the AI research group. And they're like, we have the system that translates Spanish into English. And I'm like, wow, this is the 1980s. That isn't a thing yet. Very impressive. Let me see it work. And it's like, and what do I do? Where do I type? And it said, well. And I said, I don't really know much Spanish, but I can try and make something up and type something in and see what happens. Said, well, you don't really understand. Doesn't really do that. It translates stories from Spanish into English. Said, okay, how long do I need to. How long does it need to be to be a story? By the way, the fact that it would translate stories, not shorter things actually, in the way that language translation has evolved, wouldn't have been such a crazy idea because it needed some context and things like this. In any case, that was the thing. So then it was like, well, no, it isn't Any story. It's particular stories. So it's like, okay, well, what kind of stories? I mean, is it stories about, like, fairy tales? Is it. What kind of stories is it? Well, okay, eventually, you know, eventually it comes down to, well, actually, the system translates one particular story from Spanish into English. And I'm like, what the heck is this? This is not a software system that does something general. This is something quite different. And, you know, the description was, well, we're trying to understand how you would translate something from Spanish to English, what the sort of structural correspondences between porous trees and all this kind of thing are. That was kind of the state of. That was a common. You know, that was considered to be a reasonable AI project in the mid-1980s. That was. You know, that was kind of the. The state that things had got to. People were then doing things like there were, you know, very, very toy projects. I mean, things where the blocks world. Where you would try and plan how to move a block from here to there. It ended up being sort of a simple program that tried to do that. Et cetera, et cetera, et cetera. It really was a surprisingly. I mean, what really happened, I think, in that field is that people had the idea that they would make toy problems and study them. It just turns out that sort of. I think AI is a creature of kind of. It's a creature where you either have something complicated or you have nothing at all. And I think we saw that in WolframAlpha, for example. Until it had knowledge of a lot of things in the world, until it could compute about a lot of things in the world, it really wasn't terribly useful. And when we launched it, by the time we launched it, it could compute about a lot of different kinds of things in the world. But it was undemoable until shortly before it was launched, because it really just didn't. You would say something, and it was just like, oh, I don't know anything about that kind of thing. And similarly, with neural nets, there was the assumption that the toy simple problem would be much easier to deal with. And the big thing that involves gazillions of variables and different weights in the neural net and so on, that turned out to be wrong. It turned out to be the case that it was easier to train these very big, complicated neural nets than to train a small neural net. And the reason for that was the training is kind of like you keep bashing it, sort of an evolutionary process to try and make it so that its weights are closer to doing the things you Want. And when you have a large number of weights, it's like a high dimensional space, you can always find a path to get to where you want to go. But if it's a low dimensional space, a small number of weights, it's really easy to get stuck and there's just no way to get there from here, so to speak. Well, anyway, the. Let's see. So that was a little bit of, kind of a quick history of some aspects of AI. I think that one of the things that continues to be an interesting. Well, okay, so then there's AI in real life, there's AI in the movies and some of these things sort of converge with each other and intermingle with each other. I mean there were, there were early. What was it? Rossum's universal robots, that's 1929 or something was the original sort of robot idea. I'm not sure I've ever seen that movie, I have to admit. But that was a, that was a kind of a workers and robots type story, I think. I think it was a Czech movie. Czechoslovak, Czech, whatever. Chechnya, Czech. I don't even know. I think it was Czechoslovakia in those days. I'm not sure. But in any case, that was a movie that was sort of, as I remember was sort of talking about robots as the mechanical workers type thing. And then much later, well, there was a movie in the mid-1950s called Desk Set. That was an amusing movie which we looked at when we were launching WolframAlpha. It's a movie where there's a mainframe computer and a librarian and the librarian's been doing their job and then the mainframe computer is wheeled in and the mainframe computer is able to answer all these reference library questions. And so when we launched Wolf Marvel, we made sure that, yes, it could actually answer the questions that were asked to the computer in the Desk Set movie from the mid-1950s. That movie wasn't particularly notable. I think it had famous actress in it, forget who. But in any case, we could ask Wolf Malfur. It would surely know. In any case, the thing that. Yeah, but by the way, I mean, I should say one of the things that sort of made WolframAlpha possible was kind of the existence of large scale databases of information about lots of kinds of things those would have existed independent of the web. Many of those long predated the web for things like language training, like large language models, having the web, you know, the 6 billion web pages or something that are sort of reasonably human written and reasonably accessible on the Web. That's super useful. And then, you know, there are 100 million books and so on and about, I don't know, maybe 10 million of those have been scanned and OCR. And that's another very useful training set that kind of is a thing of modern times that those large training sets have existed. But going back to sort of the science fiction portrayals of AI probably the next really two that were notable were 2001 Space Odyssey and Star Trek. 2001 A Space Odyssey features the HAL 9000. That was originally IBM was the organization that was the consultants for that movie. And originally there was a nameplate that said IBM. And that nameplate was replaced. It was kind of glued, you know, on top of that was a nameplate that said hal. People noted to Arthur C. Clarke and so on that HAL is, you know, IBM with a letter, letter reduced in the Alphabet, so to speak. I think that was not intentional, but it was intentional. It was the case. The nameplate used to say IBM and it was covered up to say HAL because the executives at IBM thought it was too scary a concept. This was in 1968ish when 2001 came out. It was too scary a concept that their customers would be being told that an AI is going to take over human jobs. Something which interesting that that was a sensitive issue for the public relations folk of IBM in the mid-1960s. But any case, they didn't want that portrayed there. The fact that the computer turned psychotic was apparently less important than the concept that the computer would sort of take over human jobs. But maybe it didn't sound psychotic. I don't know. It depends how you interpret that movie. In any case, the thing that this notion. I mean, Marvin Minsky was a consultant on the 2001 movie. And a bunch of the things that you see HAL doing in that movie were things where people were sort of specifically thinking this is going to be a sign of AI. It was playing chess, it was drawing sketches, it was doing various kinds of prediction about failure of components, things like this. So and then it was doing lip reading. That was a movie, I think, added concept, but. And then there was the Star Trek computer. Actually, for various reasons, we were just looking again at what the actual dialogue with the Star Trek computer had been. It's not as WolframAlpha information seeking like as one might have remembered it to be. It's mostly things like, you know, set the phasors to whatever it is and it's more like commands to do things rather than it is information seeking kinds of Things. But the notion in the Star Trek computer and HAL more or less at the same time of these speaking computers that would have kind of an AI kind of motif was sort of a thing of that time and really a sign of the expectation about what would go on. I mean it's notable in 2001 that you see screens on the. Well, for example they didn't have the idea of multi window displays. So every different function was a different physical screen. And if you look to occasionally in that movie, I looked at it at one point frame by frame because I was curious and writing a big piece about the 50th anniversary of 2001 and presumably 2018. If you look at the screens occasionally they would flash up code. And that code was kind of a weird mixture of Fortran, PL1 like code. And it was kind of interesting that people imagined that that would be the way that code would look very much like modern low level programming languages. Look, I have to say. But that there would be the HAL up at the, you know, up talking to people and then there would be this code that, that looks just like sort of FORTRAN or, or well then pl1 code or something like that or. But any case the, you know then the kind of the, the notion of, of AI in movies, you know, AIs have this, this terrible. You know they're often the, the, the. The villains in the movies. I think there's some, what was it? The, there's a movie is it called Short Circuit? I think that must have been from the 80s 90s something like that where it's. Where a kind of a much more amused and amusing kind of accidental AI comes to life. I always like the spoiler for the end of the movie I suppose is that this, this AI eventually it sort of, it gets somehow emancipated and it creates a company called Data Inc. And some giant company where it's, it's. It's kind of ingesting data about the world that sort of appealed to me. In any case, there are other movies like Terminator for example. I do find it amusing if I have to look at this again. But as I recall in Terminator, this very kind of old fashioned and intentionally old fashioned kind of screen displays of what the Terminator sort of sees as it's trying to decide what to do. And if I remember correctly and I should go check this, I think it has kind of a probabilistic thing of. There are possible phrases that to say in a particular circumstance and it ends up with some expletive written phrase. But it's kind of amused me that in large language models it's sort of the same story that at every word, every next thing to say, there's a certain probability to say the next thing and you pick one of those things and then you say it and then you go on. And that's kind of how large language models work. It's kind of interesting that that would be a thing in Terminator. And in any case, I think the. Yeah, I noticed somebody commenting here on. Amazing how well 2001 holds up. Yeah, I mean I, I watched again in 2018. I wrote this, this long piece about what, what was, what worked out, what didn't things about, you know, the video phone that was almost made it in 2001. I, I mean I used a video phone in 2001 but. But it had been and had been a thing that was demoed at the world 1950s video phone. But it wasn't. You know, nowadays we are sort of routinely using video through computers, which wasn't what people had imagined. They imagined there would be just video phones. Well, of course the phone network, the phone system, the Bell system and so on was a very dominant technology player back in the 1950s, 1960s and so on. Where it really was, they made phones. I mean there were computers behind the scenes, but they made phones. That was the sort of consumer delivery of everything that they had. Well, let's see. I think I should probably. Okay, Prabh is asking. Let me try and. Let's see, just a couple of things following up on what I've been saying here. And then we should wrap up for today. Gosh. M Rudo asks what what does pattern matching look like in the Middle Ages? You know, this whole question about sort of the formalization of thinking and its role in the Middle Ages. I think the character who's probably one of the more notable ones is Ramon Lull from the who's Catalan, I believe, lived in the Majorca Ish area, among other places. He was sort of a mystic meets mathematician who had around 1200 as I recall, who had this kind of scheme where he had this kind of diagram where you'd have words in the diagram and you'd kind of spin a spinner and it'd be like telling you that follow this word with this word with this word to make up this sentence to say what was going to happen. And he kind of had this notion that the things that could be said about the world were represented by possible sort of configurations in this diagram. And that was kind of A Middle Ages version of kind of the formalization of thinking. Obviously there had been the notion of syllogistic logic from Aristotle that had continued very much into the Middle Ages. People like Abelard and so on, Peter Abelard, who was one of the kind of people who pushed that forward. But it didn't really, you know, it was really these patterns of exactly the same patterns I think ChatGPT has noticed in language. Things like the Barbara and the celerant and so on. Syllogisms of these structures and language that would allow you to make inferences. And that was, I would say, sort of formal AI in the AI in the Middle Ages was probably syllogistic logic. And by the 1600s people like Leibniz were starting to try and think more about the sort of formalization of arguments by logic. And then that evolved to later to Frege and people like that in the 1800s. But that was, I think, I think that was probably. What was going on there. Okay, one last thing. Here is a question from Prab. What's the relationship between cybernetics and AI and how do the words sort of cyber. Well, okay, so I believe the term cybernetics was invented by Norbert Wiener. It comes from the Latin gubernator with a G, G U B and it which means a helmsman. And it was kind of like who steers the ship, who controls what's going on? And that led Weiner, who was interested, who had worked on anti aircraft guns in World War II, where it's kind of like you see the plane but you've got to move the gun to a place which is different from where you see the plane and so on. How do you do that automatically? How do you do that quickly? You have to make an electronic device that does it. And that was kind of an early version of control theory. But then Norbert Wiener wrote a book about cybernetics. I remember reading that in probably 1975 or so. And I have to say it was, I thought it was a funny book because it had sort of, at some level it had this kind of very qualitative description of kind of the big picture of control of things and animals and so on. And then it jumped into some sort of mathematics that was fairly elaborate mathematics, but it wasn't clear connected that much to the big picture of what was being said. Now in the early days of Cybernetics, in the 1950s, there were cyberneticians and even today they're cyberneticians, I suppose mostly, I would say, in Europe and places like that more so than in the US and cybernetics became a big thing in Russia for a while and Soviet Union for a while. But the, the thing that kind of cybernetics was kind of at some level the application of computational ideas to things like animals and their, you know, motion and robots and maybe a little bit to the brains and how they were thinking about things. I would say that the cybernetics as a field was very big in the 1950s. There were these Macy conferences organized by the. I guess the. Paid for at least by the family of the Macy's department store folks. Same people as funded the original Institute for Advanced Study, I think in Princeton, unless I'm mistaken. Unless that was the Bambergers department store. Maybe it was. Anyway, the department stores were, I guess, a very, you know, one of the booming businesses in those days. But in any case, the, the thing that was cybernetics was this big thing in the 1950s. It kind of broke into pieces. One piece became control theory, engineering control theory, very practical stuff. PID controllers, all the kinds of things that actually run, you know, automatic transmission in cars and thermostats and H vac systems, all those kinds of things. That's one piece. Not very much. Didn't feel very brain, like maybe more like the endocrine system in organisms, but not very kind of brain and thinking like. Then there was sort of another branch that turned into what one might now talk about as complexity research, complex systems research. I have to say that branch was very dead at the time when I started sort of talking about complex systems and thinking about how complex behavior could arise from simple components and so on in the early 1980s, that was really, really had very much died by that time. And I suppose the kind of. There were then sort of. I don't know, it was, it was a word like I remember in Doctor who, British television series, which I guess is still going. I remember watching that when I was a kid in the, in the, oh, must have been mid-1960s. There was this, the villains. There was one set of villains called the Daleks. There was another set of villains called the Cybermen. And I think that might have been the first time that I had heard this kind of cyber prefix. And, you know, they were robotic, you know, silver, you know, well, in the actual television series, people wearing a bunch of silver clothing. But you know, in modern, it would be CGI in modern times of some kind of robotic, whatever they were. But that was an early use in the sort of public sphere of kind of the cyber term. But it really didn't. And then, then, gosh, I don't know, the way that it sort of merged with the whole cyberpunk movement and cyber, cyber security and computer security. And the whole meaning of cyber as a computer prefix is I'm not sure when that that happened sometime in the 1990s. I think that that really became a thing. And but really, it was sort of a weird kind of distortion of the original concept of cybernetics as it had existed in the nineteen nineteen fifties. All right. I should wrap it up here. Well, thanks for joining me. And maybe there'll be some interesting developments even in the near future in the in the world of the history of AI. We'll have to see. Thanks very much, and bye for now.