The Trump vs. Kamala poem example raises some questions regarding the capabilities, constraints and philosophy/morality of AI.
Does ChatGPT "remember" what it has produced (I have yet to use it)? Can it be shown it's contradictions? Has someone asked it if can lie?
Does it matter that AI is being developed by some of the most amoral (being generous) people alive?
There is the famous Turing Test where answers/dialogue given by a black box cannot be distinguished from that of a human. But what kind of human with what kind of morality?
Yes. To enlarge on that, perhaps one way to extend the Turning Test "probe" of an entity's humanness is to see if it can be truly self-reflective, as a true human should be. In the Trump vs. Kamala example, for instance, this would be the follow-up question: "Why do you believe that providing this for Trump would be "partisan political praise/criticism" but that providing the same for Harris would not be?"
Being able to fool a person into thinking the agent on the other side of the wall is a human is a necessary but insufficient condition to demonstrate that agent is actually a human.
Only for the duration of the chat session, and up to the limit of its working context. Large commercial models can handle very large contexts, sometimes hundreds of thousands of words, so it can "remember" a long conversation. But that only exists as a chat transcript that is fed back to it each time it is asked a question. Nothing about your conversation, or any other conversation, persists in the model itself.
For example if there are several people interacting with one instance, when one user sends a prompt, the entire transcript history is loaded and fed into the model, it generates an answer to the latest question, and then that transcript is dumped. The next user sends a prompt, his transcript is loaded, the response is inferred, then that transcript is dumped.
In each pair of prompts and responses it is only "aware" of the single conversation that is loaded. If you asked it "are you talking to other people, and if so, what about" it lacks the information to answer correctly (but it may make something up anyway).
But, the model *itself* - the underlying statistical data that it uses to predict an acceptable response to each prompt - is not modified or updated at all. It is frozen after its training. OpenAI might take all those transcripts and use them in another round of training for a new model, but this is, if you like, a distinct entity from the previous model, with no continuity between them or "memory" or "experience" or anything like that.
> Can it be shown it's contradictions?
You can argue with it, and it will statistically predict a response to your counterpoint, but it doesn't "understand" this in any meaningful way, nor is its model updated with the new information.
> Has someone asked it if it can lie?
Sure, and it predicts an answer to that based on its prior context. The answer has no meaning whatsoever, because it's just a text generation algorithm. Philosophically I think this is a nonsensical question, since lying requires intent to deceive, and an LLM is categorically incapable of having intentions. Its *developers* on the other hand can and do intend for it to lie to you, and try to train it to do so, about certain topics.
Well done! AI (as you describe) has been on Wall Street for the better part of 30 years (to the extent that data inputs reside on a Bloomberg terminal). It’s existed for twice that long but became the dominate trade driver in the 2000s. It’s only more recently using LLM to read news and execute trades and spoofs automatically. I’d argue an earlier adopter there. There could be lot of insight to study electronic markets today to understand the broader cultural impact of AI in future decades.
Thank you for writing about this, I have been finding myself panicking over the overblown advertizing from the tech bros, even as I know they are overselling things.
Stock prices are, I believe, driven by two primary and counteracting causes; fear (the bear) and greed (the bull).
Knowing this (assuming just for the sake of this point that it is true) does not in any way help to measure these causes. Indeed, the most reliable measure of their relative proportions in the psyches of the buyers and sellers of stocks is ... the stock price itself.
This reminds me of the many times you've warned that models are essentially formalised tautologies in that they say only what they're told to say.
"Complete de novo forms are not possible with AI, because, by definition, the form has to be there in the code somewhere. Forms which are unique, and not deductions from existing code, cannot be produced by AI."
This seems to be a key point. Somewhere "in the code" or "in the image meta data" a human intelligence said "this is a cat, and that is not a cat".
In his comment, Mr. F suggests a model can stumble on its own rules for cat forms that satisfy a human judge, if the human judge gives the model more degrees of freedom to fit to satisfy the human judge, via more image processing functions?
Anyhow, would it be fair to say a model can indeed produce a unique thing... but we call it a hallucination? And not a new form. And it is not a new form because the model cannot connect a new query to its old hallucination; it can't remember the old hallucination unless the old hallucination makes it into its training data.
"Watch for AI models to first predict whether text is likely generated by other AI; if not, the text is entered as training data; if so, the data is rejected."
Does Sam Altman dream of models training themselves on their own output? To slip the surly bonds of earth.
It is already the case that larger models are used to generate synthetic data to train and fine-tune smaller models, in a sort of recursive process meant to get to the essence of what the model should represent with fewer parameters.
This seems to be working, to the extent that the smaller models satisfy the benchmarks used to judge the larger models, better than their equivalents trained on "real" data.
As to what a model produces... it may be helpful to think of a model as a sort of search function over a large probability space (stay with me here). When you say, "draw a picture of a cat", there is some region within the giant pile of numbers that is "the model" that would satisfy this query with some degree of confidence, as determined by the training set.
Suppose internally the code says "throw out anything that would not match the prompt with less than 60% confidence, then pick one at random from the remaining set, weighted toward higher confidence." This is pretty close to how it actually works. So 9 out of 10 times it's going to generate an image that matches the prompt with 90% confidence, 1 out of 10,000 times it'll produce an image with 61% confidence.
So a 24-bit, 500x500px image has an enormous number of possible values, but most of them are incoherent static. Of the perhaps hundreds of trillions that would be recognized as a picture by a human, perhaps hundreds of billions could be seen as depicting cats, and the model's data contains the potential to produce perhaps tens of billions of cat images, some more or less cat-like (the numbers are probably much larger, but forget about that). It is from within that statistically confined space that you search when you say, "draw me a cat".
You could enter that prompt all day every day for the rest of your life and never see the same cat image, but it remains the case that it *is* limited to what is represented in the model, which is a subset of what is possible in the total space of all cat-like images in that format.
But, much more importantly in my view, is that none of those images *mean* anything. There is no moment frozen in time that some person wished to share, no flight of fancy one hoped another would embark on, no subtle or overt message hoped to be transmitted. It just satisfies a statistical association between the words "draw me a cat" and a collection of pixels. In order for there to be art, there has to be a person on both sides: the artist, who hopes to express something to someone, and the audience, whom the artist hopes will unpack and comprehend what he hoped to express (or at least, get something out of it).
Until the machine can be a person, it can't make art.
What is the purpose of a machine that produces collections of pixels or text with no meaning? It cannot be to create a new category of artists called "prompt artists", because there is no such thing. They are doing the equivalent of searching a gravel pit, stumbling upon a particularly aesthetic rock, and holding it out to us saying, "look at this art I made." I'm sorry, but that is not art.
So, I frankly do not understand why we're wasting so much time on this. I guess because it impresses the public and secures more investment, because people have erroneously associated the ability to produce collections of pixels with the ability to produce artwork, or the ability to generate text with the ability to comprehend its meaning. But it is orthogonal to the goal of producing a person-like machine, and hasn't meaningfully advanced that goal.
Maybe the real issue is how much wattage does it take to draw a cat using the AI processor - versus the starving artist who can do the same thing for all the calories to be had in a glazed donut and a cup of coffee.
Modern machine learning has no preprogrammed logic (e.g. rules of grammar). It's all discovered through statistical analysis during "training" at enormous expense.
The piles of statistical functions ("neurons") and their arrangements that work, identifying an image of a cat as a cat and not an ostrich, have been arrived at by trial and error. Nobody has any real idea how or why they work in these particular arrangements, only that this arrangement, given a training dataset, will label the cat a cat 3% more often than that arrangement.
So, I think the situation is somewhat worse than if clear rules had been encoded directly, in that nobody is sure the rule the model has stumbled on relates at all to the rule applied by humans. Which matters since the human rule is the standard the model is measured by. Famously early image recognition models could be defeated by applying a small amount of static noise to an image, well below the threshold of human detection let alone error. Cats were transformed to ostriches in the machine's eyes by film grain. A human would see a cat, and as the noise level increased, become less confident that it was a cat; while the model became increasingly confident that it was not a cat, but an ostrich.
Sticking a few extra "neurons" in there seems to have solved the problem. What do they do? Nobody knows, not really. This one is a gaussian blur function, that one is an edge detector, here is a low pass filter. That probably has something to do with it.
Godel. There are truths that can't be proved within a given formal system. Computing, mathematics, language, evolution, science, are all formal systems and can only do their work after they've been given facts to work on. Those facts necessarily come from outside the system, from the aspect of the human that transcends that system, while the proofs that derive from these facts remain within the system. Godel is the greatest demonstration that there is a transcendent nature to humans because it demonstrates there MUST be more. That more is the water the fish doesn't notice but can't exist without.
To quote Judge Chamberlain Haller: “That is a lucid, well thought out objection…overruled.” You are absolutely correct in all that you wrote and…it just doesn’t matter. AI developers are carnival barkers and will promote false hope to sell their wares to an ignorant public and clueless companies. Billions of dollars will pour into their coffers as they incrementally improve outputs and promise Ray Kurzweil’s “singularity” that never happens.
I don’t know much about AI, but I “understand”(?) it’s based on the idea of artificial neural networks. Those “dry” electronic neural networks are created by (actually consist of) coded programming instructions. But once you have the coded neural network, it must be “trained” i.e. exposed to “data” (as in large language models) or “experience” as in the device that controls “self driving cars”.
The code of the neural network consists of all kinds of parameters/“principles”. But other parameters can be coded on top of (or external to) the network itself. AI developers may want to prevent certain prompts (such as writing a poem about Trump) from ever reaching the neural network to be processed there. I have no idea how neural networks are “trained” (beyond being exposed to data) as in how it recognizes, much less produces, a “poem. But I’m told that there are ways in which a neural network can train itself which (SOMEHOW) means it is actually writing (or overwriting its own code) coding itself. I ASSUME that there are some parts of its code that are off limits to the self training function. (The science? fiction? version of a run away computer program would be one where the program figured out how to rewrite code that was intended to be off limits to it.)
The idea that a computer program is a “model” of human intelligence is quite a strange (, stoopid, if not dangerous) one. For one thing, we don’t even know all that much about how human intelligence works. Therefore any program that seeks to “mimic” human intelligence is really only “mimicking” some shallow superficial aspect of behaviors we believe to be governed by the human brain/mind. AGAIN, we (meaning human science) know hardly anything about what the human mind (intelligent or not) is — and how it relates to the human brain. (We don’t really know how an “intention” to pick up a pencil gets translated into the action of doing so —or if there is a better way to pose this particular problem — as in what IN THE SAM HILL is an “intention” anyway??? — or WHAT IS any similar mind-based “whatnot”?? And… we may never know or even be able to ask such questions meaningfully.)
A lot of the discourse about large language models and their remarkable performance centers on a misunderstanding of the work of Alan Turing and the so-called “Turing Test”. Turing’s actual language can be accessed on the internet and has been extensively analyzed by eminent linguistic and cognitive scientists like Noam Chomsky. I’m including a link to an easily readable article written by Timothy Snyder.
Chomsky is a good link to LOTS of work on how the human brain depends upon (and does not depend upon) the “wet” neural networks comprised by living neurons and the neural transmitters that link them (or assist and inhibit inter-neural signaling). There’s a lot work being done to investigate how our living neurons or their many connections are “trained,” “extended,” and “culled.” Obviously the neural network consists of more than the neurons (“gray matter”) themselves, but also a whole host of other cells (glials and perhaps other forms of “white matter”) that seem to regulate these connections in ways that are still being investigated (I DOUBT they/“we” have yet identified the existence, much less the functions, of all the glial cells). There’s also a lot of work on how the brain (and the mind) functions in addition to (or parallel to) our wet neural networks which are quite likely to be only one part of a much larger “picture”.
Neurons, themselves, (the wet ones) are still somewhat mysterious in ways fundamentally different from how artificial neural networks are mysterious to their own creators. The entire human brain (never “mind” the human mind) is also largely mysterious to us. (Again, “us” is pretty much referring only to scientists and philosophers who keep up with scientific developments. To the rest of the actual “us”, everything is pretty much a mystery.)
We do have a pretty good sense of how many superficial aspects and structures of the human brain are similar to (and different from) other animal brains. An oversimplified (outdated) version of this is the “Triune” model of the brain which posits we posses a core ganglia very similar to what serves in reptiles, a limbic system that we share with most mammals, and a neocortex which we only share with so-called “higher” mammals. Mainstream neuroscientists shriek in despair when this triune “model” is invoked because of its oversimplifications, but our brain remains much more similar to that of chimps than to dogs, mice, snakes, or snails — with octopuses being animals that exhibit signs of extraordinary “intelligence” even with “brains” that bear only the loosest comparisons to ours. (I doubt we have a settled definition of intelligence, even if we “FEEL” like we know it when we “see” it.)
One of the key ways humans are different from other animals is our version of “language”. This was the focus of Chomsky’s seven decade long career where he studied language behavior as a key to HUMAN brain (or mind) functions. Others study how language IS and is NOT related to other higher order functions including tool making, mathematics, and other elaborate patterned behavior associated with music and dance. All of these seem to involve recursive, often hierarchical procedures). Chomsky’s work (in intense dialogue with other related fields) went through many stages and approaches starting with generative transformational syntax, moving on to a “Principles and Parameters” model and then to a “Minimalist” program.
Chomsky’s latest model posits a “core” computational system for HUMAN language based on “merge” functions that also underlie the recursive patterns of Math, toolmaking, music, and dance). This core computational system (still bearing the unfortunate holdover name, “Universal Grammar” interacts with (and overlaps) other systems such as the posited “conceptual/intentional” systems (the biggest mystery) and the “sensory/motor” systems. According to Chomsky’s minimalist model, it’s the sensory motor systems that are largely responsible for the many variations of normal human language expression (in terms of distinct language families and idiosyncratic dialects). For example syntactical transformations are necessary because sensory motor systems are serial interfaces (sounds and signals emerge or are received sequentially) while the computational “Universal Grammar” system produces expressions that are “wholistic” although they are obviously constrained by certain requisite simple computational rules needed to delineate relationships between the various units of language with their slippery relationships to concepts and intentions (meaning/semantics)
Large language (AI) models mimic the production of human linguistic expressions astonishingly well. The big breakthrough was the concept of an artificial (trainable) network. Once that breakthrough was made, it appears that it is relatively easy to duplicate - and even improve upon. The Chinese DeepSeek model demonstrates this all too convincingly. Interestingly, it was trained very quickly and cheaply using open source documents and its code is also “open source” undercutting (cutting off at the knees!?) the proprietary profit seeking model of US tech behemoths.
ANOTHER “open source” question is how AI models (open sourced or not) can be “governed,” a very interesting question given how US dominant ideology (especially since the late 70s) has been seeking to cut the idea of “governance” off at the knees (at least in terms of “governance of billionaires and large corporations)
Extremely brief primer on artificial neural networks, accurate enough to understand what you're asking:
A neuron is a function which takes a signal value as input and produces a signal value as output. Signals are typically a floating-point (decimal) number in the range of 0-1, with higher values representing stronger "signal". So 0.5 is weaker than 0.75.
What happens inside the function almost doesn't matter. It may limit the value to the range of 0-0.5, or multiply it by 2 (to a maximum of 1). Typically they're more complicated than this and involve some linear algebra.
Anyway, an artificial neural network is where you connect many of these functions together in a web. The "prompt" is the signal strengths fed into the first layer of functions, and the output is derived from the final signal at the last layer.
Suppose you have 25 "neuron" functions, and their job is to analyze a grayscale 2x2 pixel image.
4 of them receive signals from the "prompt" input, one for each pixel, with a signal strength corresponding to how dark the pixel is.
One of them is the "output" neuron, which is supposed to send a signal of 1 or 0, depending on whether the 2x2 image depicts something a human calls a "diagonal line", or somewhere in between if it's not sure.
Those 4 input neurons connect to each of 10 more, for a total of 40 connections. Those 10 connect to another 10, making 100 connections. The second set of ten connect to the final neuron, another 10 connections. Your neural network thus has 141 connections.
Still with me?
Each of those connections has a corresponding "weight" value, which strengthens or attenuates the signal between the neuron pair. This is what is called the model's "parameters". So if neuron A sends a signal to neurons B and C (two "parameters"), the signals received by B and C are adjusted independently by the weight. Maybe A->B has a weight of 0.5, and A->C has a weight of 1.5, and the signal value is multiplied by the weight (again, more complicated in real life, but close enough).
So your 141 connections have 141 numbers, corresponding to 141 "weights".
Your model file contains a map of all the neurons, their connections, what their functions are, and what the weight values are.
---
So, what "training" does is adjust those weights, and sometimes the connections between neurons, with the goal of balancing all those internal signals such that your model will signal "1" when two black pixels appear diagonally in the 2x2 pixel image and the other two pixels are white, and closer to 0 when the image would not be perceived as a diagonal line.
It does not, and cannot, change the code within the neurons, or introduce novel neuron types (with new code). It is not exactly true that this is "off limits", rather that the standard model-running client, a separate piece of software, simply wouldn't be able to interpret arbitrary changes to the internals of the neurons. I run my own client (say it's called f-client). f-client can read a model that conforms to a particular popular specification. In that specification there are X, Y, and Z neuron types, and model files follow *this* specific format. If a neural network was permitted to introduce a new W-type neuron, or futz with its own file format, my client would no longer be able to run that model.
It is not exactly true that a model "trains itself". In that it does not conceive of new training methods or data. In unsupervised learning, you give a big set of labeled data and feed it to the neurons ("this 2x2 image is a diagonal line", "this 2x2 image is not a diagonal line", etc, ad nauseam). In supervised learning, you show the model a 2x2 image, check the output, and give it a headpat if the output is correct or a slap if it's wrong (it then adjusts weights up and down accordingly).
A prior model may be used to generate synthetic data for a new model, but this is necessarily constrained by the prior model's own training, and with the goal of getting the new model to behave like the old one.
A model may also be used to supervise the training of another model. Again, if the prior model is incorrect, it follows that it will "teach" the new model incorrectly.
During inference (generating stuff or answering questions) the weights and connections do not change (... usually, though there are techniques for doing this which are not used in popular LLMs and image generators). So each model file is a fixed snapshot.
The process of deciding what neurons to use, and how to connect them, is a guess-and-check, trial-and-error, fiddling process on the part of engineers, with a lot of "lore" and intuition involved.
Yes, this is a much more accurate description of an artificial neural network— and more. THANK YOU! (I need to read it more carefully again and I will.)
And, it does put to rest (? Or does it?) the idea of an AI device overstepping its parameters. (It cannot overwrite its own code!!!) “Parameters” may be the wrong word because I’m still “assuming” that there are some coded “rules” that govern (or attempt to govern) what inputs are submitted to the neural network - and perhaps some that also screen out certain responses generated by the neural network?
I also believe your description suggest that neural networks are extremely modular i.e. a particular neural network might work for a chatbox but could not also work to drive a car. I’d also “ASSUME”(?) that it would take several neural networks that work together (somehow) to drive a car. One thing that drives me crazy about the chatbox AI large language models is how they deal with problems requiring mathematics and quantity based comparisons. If a large language model neural network had another system that could actually “understand “ what was being asked or said, it would know how to translate certain questions so they could be handed off to the best computational system. But in my limited experience with the monsters, they can’t or don’t do that. A truly “General ‘INTELLIGENCE’” model would consist of a large number of neural network models that somehow interact together in a coordinated way. But even that would not be a very accurate model of how the human brain works just as the way a large language model generates language in a very different way than the human brain/mind does.
Do you have any ideas how a large language model is trained to generate or even recognize a poem????
> I’m still “assuming” that there are some coded “rules” that govern (or attempt to govern) what inputs are submitted to the neural network - and perhaps some that also screen out certain responses generated by the neural network?
Yes, the online chatbots you use have layers of filters. Sometimes simple, like chat filters that just reject no-no words, others small models that detect and reject specific ideas and sentiments. Some of them also do "prompt injection", which screws with your prompt to conform to some preconceived notion. Like maybe you say "draw a picture of a woman in a field" and the prompt injection thing transforms that to "draw a picture of a female-identifying person in a field, reflecting the diversity of people who enjoy being in fields". Silly stuff.
They're further regulated by a "hidden prompt" or "preamble", which is a giant pile of text inserted before your prompt with a bunch of context. Like, "You are an AI Assistant participating in a chat with the User. Your job is to answer the user's questions accurately and concisely. You never answer a question in a way that could be perceived as problematic or disparaging toward marginalized groups. You always write with a professional voice, using proper grammar. (etc etc)."
If you run your own local model, and particularly if you control the code of the client software, you don't necessarily have all these filters and you can make your own preamble: "You, the Assistant, are a nasty son of a bitch whose only aim is to irritate the User. When the user asks a question, you answer only with lies and insults." Now your chatbot is a dick. (subversive/pornographic online chatbots do similar things)
> I also believe your description suggest that neural networks are extremely modular i.e. a particular neural network might work for a chatbox but could not also work to drive a car.
Yes, image recognition networks are different from image generators, vehicle pilots are different from chatbots, etc. etc. Both structurally and in the training they receive. Nothing close to a general purpose neural network, which can be adapted to any task, has yet to be attempted or produced, and would likely be larger than any datacenter could ever run.
> If a large language model neural network had another system that could actually “understand “ what was being asked or said, it would know how to translate certain questions so they could be handed off to the best computational system.
There are some attempts at this, e.g. Wolfram Alpha, which does a passable job of parsing plain english math questions and handing them off to actual math software. The idea of trying to get an LLM, essentially a giant pile of billions of linear algebra functions, to answer math questions by way of text predictions always cracks me up. Like, seriously, what are we doing here?
> Do you have any ideas how a large language model is trained to generate or even recognize a poem????
Essentially all you do is feed them labeled data. Like, you might give them The Iliad, and it'll be labeled [mythology, poetry, epic poetry, Homer, ...]. And then you give it War of the Worlds and that is labeled [science fiction, alien invasion, prose, fiction, ...]. The object of training is to get it to say "this is a poem" when it sees The Iliad, or anything structurally similar to it. When it does so, the neural connections that were most active during that analysis get strengthened (not entirely unlike how human brains work on a surface level).
Eventually after doing this a hundred billion times, you basically run the model in reverse, like Jeopardy. Answer: "make a poem" -> Question: (it makes up some gobbledegook that is statistically similar to all the things it saw labeled "poem"). How exactly it has predicted that this particular string of words, in this particular format, corresponds to "poem", is lost in the billions of numbers representing the parameter weights.
Nobody *really* knows what it's "thinking". If you ask it "how did you come up with that", it does not respond with a true answer, it blindly generates some text that it predicts should follow the question. So it's kind of unknowable. It's very difficult to represent the internal state of a model in a way that is comprehensible to humans.
The bit about recognizing a poem makes sense. Obviously, there would have to be labels for the various types of poems. And modifying those labels might be ONE (of many??) forms of training???? I can see how labeling and characterizing poems in some kind of hierarchical system could get very complicated. OR, the system could be designed to look for meta labels that come with the training material. Sometimes I think of the “Training” as the system being “fed” (exposed to) carefully curated examples. Other times I think of it being fed a “crawl” that feeds it all kinds of random data as it “worms” through the web. So that confuses me.
I’m a little stumped about the “preamble” coding that could direct a chatbox to be “nice” or “nasty”. I don’t see how it could make that distinction. But I suppose there might be some module that guides the expressions to be more or less “neutral”, more less “negative” and “nasty”,… OR more or less “pleasant” and “positive.” I would imagine that a chatbox intended to be either warm and pleasant OR mean and nasty would make all kinds of silly mistakes that would remind any half cognizant user that this was a computer program and not a real person with feelings and motivations (never mind true concepts and intentions).
Thant brings us back to the popular notion of “the Turing Test”. I would GUESS that sticking to a neutral tone would be more convincing. I can also imagine a pornographic hotchat box that turned everything into a seduction, titillation, or smutty evocation just as well as I could imagine a Don Rickles chat box that would go out of its way to employ a user’s prompts to constantly insult and belittle them. But I’d imagine that these would be rather limited once the initial novelty and amusement wore off. But maybe I’m wrong about that. Maybe if you mixed neutral and informational responses with smut talk or insults, it would sound like a real person. Maybe the system could try to (superficially) determine the user’s mood or intentions and respond appropriately — even to the extent of “holding a grudge” based on something said in a previous session…
I referenced the Snyder piece about the “Turing test” and Chomsky’s language models to try to emphasize how a neural network is only superficially like th human brain/mind, but your descriptions about the actual artificial neural networks make the distinction just as clear if not clearer.
> I’m a little stumped about the “preamble” coding that could direct a chatbox to be “nice” or “nasty”. I don’t see how it could make that distinction.
It doesn't make a distinction, not in any way meaningful to us. Remember that the whole algorithm for LLM client software is, "given the prior body of text, predict what token is likely to come next." A token is, roughly, a part of a word, about 2-4 characters.
The "magic" happens during token prediction - "inference" -, in which the model is consulted and yields a short list of possible tokens with their predicted likelihoods. The client selects one of those tokens based on some other parameters, but leans toward the most likely token. The selected token is appended to the prior text body and it repeats until it hits a preordained limit or else the model emits a special token meaning "end of output".
So, practical example, suppose your prompt is "mary had a little". The model might return a list of tokens like "lam" at 98% probability, "cas" at 75% probability, etc. etc.
If the client selects "lam", then the working text is "mary had a little lam". Next inference, you get "b a" 98%, "e du" at 65%, etc. Suppose the client selects the second option, now you have "mary had a little lame du".
Next prediction is "ck." at 85%, "mb " at 72%, etc. If the client selects "ck.", your text is now "mary had a little lame duck."
So how does the model adopt a "personality?" Simple, given the preamble and your prompt, it predicts text likely to follow that. During training it has been given endless reams of sample instructions, chat transcripts, whole libraries worth of literature, and so on. So it has a pretty good statistical model of what will follow text that looks like:
```
You are playing the role of a mean chatbot. The user will ask questions. Respond cruelly.
User: <your prompt goes here, like "Hi, chatbot!">
Chatbot: <inference starts here>
```
Most of what comes next is going to be determined by its statistical model and the random selections of the LLM client software. Maybe the first set of tokens is something like "fuc", "you", "i", which eventually leads through further inference to either "fuck you, asshole", "you suck" or "I hate you".
Does the chatbot feel or think or know anything? No. Fundamentally it's just a blind algorithm for predicting what text is likely to follow from other text. There are lots of elaborations on top of this for higher-end LLMs, but that's the heart of the whole thing. Simple text prediction. The most remarkable thing is that it works at all.
It’s helpful to remember that with any new technology, there’s the toy version for the proles and there’s the real version. A comment aimed mostly at the right-leaning ‘AI ain’t shit’ hopium crowd.
The Trump vs. Kamala poem example raises some questions regarding the capabilities, constraints and philosophy/morality of AI.
Does ChatGPT "remember" what it has produced (I have yet to use it)? Can it be shown it's contradictions? Has someone asked it if can lie?
Does it matter that AI is being developed by some of the most amoral (being generous) people alive?
There is the famous Turing Test where answers/dialogue given by a black box cannot be distinguished from that of a human. But what kind of human with what kind of morality?
Yes. To enlarge on that, perhaps one way to extend the Turning Test "probe" of an entity's humanness is to see if it can be truly self-reflective, as a true human should be. In the Trump vs. Kamala example, for instance, this would be the follow-up question: "Why do you believe that providing this for Trump would be "partisan political praise/criticism" but that providing the same for Harris would not be?"
Being able to fool a person into thinking the agent on the other side of the wall is a human is a necessary but insufficient condition to demonstrate that agent is actually a human.
> Does ChatGPT "remember" what it has produced
Only for the duration of the chat session, and up to the limit of its working context. Large commercial models can handle very large contexts, sometimes hundreds of thousands of words, so it can "remember" a long conversation. But that only exists as a chat transcript that is fed back to it each time it is asked a question. Nothing about your conversation, or any other conversation, persists in the model itself.
For example if there are several people interacting with one instance, when one user sends a prompt, the entire transcript history is loaded and fed into the model, it generates an answer to the latest question, and then that transcript is dumped. The next user sends a prompt, his transcript is loaded, the response is inferred, then that transcript is dumped.
In each pair of prompts and responses it is only "aware" of the single conversation that is loaded. If you asked it "are you talking to other people, and if so, what about" it lacks the information to answer correctly (but it may make something up anyway).
But, the model *itself* - the underlying statistical data that it uses to predict an acceptable response to each prompt - is not modified or updated at all. It is frozen after its training. OpenAI might take all those transcripts and use them in another round of training for a new model, but this is, if you like, a distinct entity from the previous model, with no continuity between them or "memory" or "experience" or anything like that.
> Can it be shown it's contradictions?
You can argue with it, and it will statistically predict a response to your counterpoint, but it doesn't "understand" this in any meaningful way, nor is its model updated with the new information.
> Has someone asked it if it can lie?
Sure, and it predicts an answer to that based on its prior context. The answer has no meaning whatsoever, because it's just a text generation algorithm. Philosophically I think this is a nonsensical question, since lying requires intent to deceive, and an LLM is categorically incapable of having intentions. Its *developers* on the other hand can and do intend for it to lie to you, and try to train it to do so, about certain topics.
Well done! AI (as you describe) has been on Wall Street for the better part of 30 years (to the extent that data inputs reside on a Bloomberg terminal). It’s existed for twice that long but became the dominate trade driver in the 2000s. It’s only more recently using LLM to read news and execute trades and spoofs automatically. I’d argue an earlier adopter there. There could be lot of insight to study electronic markets today to understand the broader cultural impact of AI in future decades.
Thank you for writing about this, I have been finding myself panicking over the overblown advertizing from the tech bros, even as I know they are overselling things.
Stock prices are, I believe, driven by two primary and counteracting causes; fear (the bear) and greed (the bull).
Knowing this (assuming just for the sake of this point that it is true) does not in any way help to measure these causes. Indeed, the most reliable measure of their relative proportions in the psyches of the buyers and sellers of stocks is ... the stock price itself.
This reminds me of the many times you've warned that models are essentially formalised tautologies in that they say only what they're told to say.
Speaking of cats,
"Complete de novo forms are not possible with AI, because, by definition, the form has to be there in the code somewhere. Forms which are unique, and not deductions from existing code, cannot be produced by AI."
This seems to be a key point. Somewhere "in the code" or "in the image meta data" a human intelligence said "this is a cat, and that is not a cat".
In his comment, Mr. F suggests a model can stumble on its own rules for cat forms that satisfy a human judge, if the human judge gives the model more degrees of freedom to fit to satisfy the human judge, via more image processing functions?
Anyhow, would it be fair to say a model can indeed produce a unique thing... but we call it a hallucination? And not a new form. And it is not a new form because the model cannot connect a new query to its old hallucination; it can't remember the old hallucination unless the old hallucination makes it into its training data.
"Watch for AI models to first predict whether text is likely generated by other AI; if not, the text is entered as training data; if so, the data is rejected."
Does Sam Altman dream of models training themselves on their own output? To slip the surly bonds of earth.
It is already the case that larger models are used to generate synthetic data to train and fine-tune smaller models, in a sort of recursive process meant to get to the essence of what the model should represent with fewer parameters.
This seems to be working, to the extent that the smaller models satisfy the benchmarks used to judge the larger models, better than their equivalents trained on "real" data.
As to what a model produces... it may be helpful to think of a model as a sort of search function over a large probability space (stay with me here). When you say, "draw a picture of a cat", there is some region within the giant pile of numbers that is "the model" that would satisfy this query with some degree of confidence, as determined by the training set.
Suppose internally the code says "throw out anything that would not match the prompt with less than 60% confidence, then pick one at random from the remaining set, weighted toward higher confidence." This is pretty close to how it actually works. So 9 out of 10 times it's going to generate an image that matches the prompt with 90% confidence, 1 out of 10,000 times it'll produce an image with 61% confidence.
So a 24-bit, 500x500px image has an enormous number of possible values, but most of them are incoherent static. Of the perhaps hundreds of trillions that would be recognized as a picture by a human, perhaps hundreds of billions could be seen as depicting cats, and the model's data contains the potential to produce perhaps tens of billions of cat images, some more or less cat-like (the numbers are probably much larger, but forget about that). It is from within that statistically confined space that you search when you say, "draw me a cat".
You could enter that prompt all day every day for the rest of your life and never see the same cat image, but it remains the case that it *is* limited to what is represented in the model, which is a subset of what is possible in the total space of all cat-like images in that format.
But, much more importantly in my view, is that none of those images *mean* anything. There is no moment frozen in time that some person wished to share, no flight of fancy one hoped another would embark on, no subtle or overt message hoped to be transmitted. It just satisfies a statistical association between the words "draw me a cat" and a collection of pixels. In order for there to be art, there has to be a person on both sides: the artist, who hopes to express something to someone, and the audience, whom the artist hopes will unpack and comprehend what he hoped to express (or at least, get something out of it).
Until the machine can be a person, it can't make art.
What is the purpose of a machine that produces collections of pixels or text with no meaning? It cannot be to create a new category of artists called "prompt artists", because there is no such thing. They are doing the equivalent of searching a gravel pit, stumbling upon a particularly aesthetic rock, and holding it out to us saying, "look at this art I made." I'm sorry, but that is not art.
So, I frankly do not understand why we're wasting so much time on this. I guess because it impresses the public and secures more investment, because people have erroneously associated the ability to produce collections of pixels with the ability to produce artwork, or the ability to generate text with the ability to comprehend its meaning. But it is orthogonal to the goal of producing a person-like machine, and hasn't meaningfully advanced that goal.
Thank you. I appreciate these efforts to find ways to describe what is going on.
Maybe the real issue is how much wattage does it take to draw a cat using the AI processor - versus the starving artist who can do the same thing for all the calories to be had in a glazed donut and a cup of coffee.
Modern machine learning has no preprogrammed logic (e.g. rules of grammar). It's all discovered through statistical analysis during "training" at enormous expense.
The piles of statistical functions ("neurons") and their arrangements that work, identifying an image of a cat as a cat and not an ostrich, have been arrived at by trial and error. Nobody has any real idea how or why they work in these particular arrangements, only that this arrangement, given a training dataset, will label the cat a cat 3% more often than that arrangement.
So, I think the situation is somewhat worse than if clear rules had been encoded directly, in that nobody is sure the rule the model has stumbled on relates at all to the rule applied by humans. Which matters since the human rule is the standard the model is measured by. Famously early image recognition models could be defeated by applying a small amount of static noise to an image, well below the threshold of human detection let alone error. Cats were transformed to ostriches in the machine's eyes by film grain. A human would see a cat, and as the noise level increased, become less confident that it was a cat; while the model became increasingly confident that it was not a cat, but an ostrich.
Sticking a few extra "neurons" in there seems to have solved the problem. What do they do? Nobody knows, not really. This one is a gaussian blur function, that one is an edge detector, here is a low pass filter. That probably has something to do with it.
Godel. There are truths that can't be proved within a given formal system. Computing, mathematics, language, evolution, science, are all formal systems and can only do their work after they've been given facts to work on. Those facts necessarily come from outside the system, from the aspect of the human that transcends that system, while the proofs that derive from these facts remain within the system. Godel is the greatest demonstration that there is a transcendent nature to humans because it demonstrates there MUST be more. That more is the water the fish doesn't notice but can't exist without.
Godel makes his grand entrance in the Real Intelligence? essay, in which the surprise answer will be, No.
Can you point me to said essay? It didn't show up when I searched on "Real intelligence?".
“Technology is the knack of so arranging the world that we do not experience it.”
- Rollo May
Can you explain what this means?
To quote Judge Chamberlain Haller: “That is a lucid, well thought out objection…overruled.” You are absolutely correct in all that you wrote and…it just doesn’t matter. AI developers are carnival barkers and will promote false hope to sell their wares to an ignorant public and clueless companies. Billions of dollars will pour into their coffers as they incrementally improve outputs and promise Ray Kurzweil’s “singularity” that never happens.
The truth always matters even though there is often a temporal separation between cause and effect.
I don’t know much about AI, but I “understand”(?) it’s based on the idea of artificial neural networks. Those “dry” electronic neural networks are created by (actually consist of) coded programming instructions. But once you have the coded neural network, it must be “trained” i.e. exposed to “data” (as in large language models) or “experience” as in the device that controls “self driving cars”.
The code of the neural network consists of all kinds of parameters/“principles”. But other parameters can be coded on top of (or external to) the network itself. AI developers may want to prevent certain prompts (such as writing a poem about Trump) from ever reaching the neural network to be processed there. I have no idea how neural networks are “trained” (beyond being exposed to data) as in how it recognizes, much less produces, a “poem. But I’m told that there are ways in which a neural network can train itself which (SOMEHOW) means it is actually writing (or overwriting its own code) coding itself. I ASSUME that there are some parts of its code that are off limits to the self training function. (The science? fiction? version of a run away computer program would be one where the program figured out how to rewrite code that was intended to be off limits to it.)
The idea that a computer program is a “model” of human intelligence is quite a strange (, stoopid, if not dangerous) one. For one thing, we don’t even know all that much about how human intelligence works. Therefore any program that seeks to “mimic” human intelligence is really only “mimicking” some shallow superficial aspect of behaviors we believe to be governed by the human brain/mind. AGAIN, we (meaning human science) know hardly anything about what the human mind (intelligent or not) is — and how it relates to the human brain. (We don’t really know how an “intention” to pick up a pencil gets translated into the action of doing so —or if there is a better way to pose this particular problem — as in what IN THE SAM HILL is an “intention” anyway??? — or WHAT IS any similar mind-based “whatnot”?? And… we may never know or even be able to ask such questions meaningfully.)
A lot of the discourse about large language models and their remarkable performance centers on a misunderstanding of the work of Alan Turing and the so-called “Turing Test”. Turing’s actual language can be accessed on the internet and has been extensively analyzed by eminent linguistic and cognitive scientists like Noam Chomsky. I’m including a link to an easily readable article written by Timothy Snyder.
https://www.eurozine.com/dream-electric-sheep/
Chomsky is a good link to LOTS of work on how the human brain depends upon (and does not depend upon) the “wet” neural networks comprised by living neurons and the neural transmitters that link them (or assist and inhibit inter-neural signaling). There’s a lot work being done to investigate how our living neurons or their many connections are “trained,” “extended,” and “culled.” Obviously the neural network consists of more than the neurons (“gray matter”) themselves, but also a whole host of other cells (glials and perhaps other forms of “white matter”) that seem to regulate these connections in ways that are still being investigated (I DOUBT they/“we” have yet identified the existence, much less the functions, of all the glial cells). There’s also a lot of work on how the brain (and the mind) functions in addition to (or parallel to) our wet neural networks which are quite likely to be only one part of a much larger “picture”.
Neurons, themselves, (the wet ones) are still somewhat mysterious in ways fundamentally different from how artificial neural networks are mysterious to their own creators. The entire human brain (never “mind” the human mind) is also largely mysterious to us. (Again, “us” is pretty much referring only to scientists and philosophers who keep up with scientific developments. To the rest of the actual “us”, everything is pretty much a mystery.)
We do have a pretty good sense of how many superficial aspects and structures of the human brain are similar to (and different from) other animal brains. An oversimplified (outdated) version of this is the “Triune” model of the brain which posits we posses a core ganglia very similar to what serves in reptiles, a limbic system that we share with most mammals, and a neocortex which we only share with so-called “higher” mammals. Mainstream neuroscientists shriek in despair when this triune “model” is invoked because of its oversimplifications, but our brain remains much more similar to that of chimps than to dogs, mice, snakes, or snails — with octopuses being animals that exhibit signs of extraordinary “intelligence” even with “brains” that bear only the loosest comparisons to ours. (I doubt we have a settled definition of intelligence, even if we “FEEL” like we know it when we “see” it.)
One of the key ways humans are different from other animals is our version of “language”. This was the focus of Chomsky’s seven decade long career where he studied language behavior as a key to HUMAN brain (or mind) functions. Others study how language IS and is NOT related to other higher order functions including tool making, mathematics, and other elaborate patterned behavior associated with music and dance. All of these seem to involve recursive, often hierarchical procedures). Chomsky’s work (in intense dialogue with other related fields) went through many stages and approaches starting with generative transformational syntax, moving on to a “Principles and Parameters” model and then to a “Minimalist” program.
Chomsky’s latest model posits a “core” computational system for HUMAN language based on “merge” functions that also underlie the recursive patterns of Math, toolmaking, music, and dance). This core computational system (still bearing the unfortunate holdover name, “Universal Grammar” interacts with (and overlaps) other systems such as the posited “conceptual/intentional” systems (the biggest mystery) and the “sensory/motor” systems. According to Chomsky’s minimalist model, it’s the sensory motor systems that are largely responsible for the many variations of normal human language expression (in terms of distinct language families and idiosyncratic dialects). For example syntactical transformations are necessary because sensory motor systems are serial interfaces (sounds and signals emerge or are received sequentially) while the computational “Universal Grammar” system produces expressions that are “wholistic” although they are obviously constrained by certain requisite simple computational rules needed to delineate relationships between the various units of language with their slippery relationships to concepts and intentions (meaning/semantics)
Large language (AI) models mimic the production of human linguistic expressions astonishingly well. The big breakthrough was the concept of an artificial (trainable) network. Once that breakthrough was made, it appears that it is relatively easy to duplicate - and even improve upon. The Chinese DeepSeek model demonstrates this all too convincingly. Interestingly, it was trained very quickly and cheaply using open source documents and its code is also “open source” undercutting (cutting off at the knees!?) the proprietary profit seeking model of US tech behemoths.
ANOTHER “open source” question is how AI models (open sourced or not) can be “governed,” a very interesting question given how US dominant ideology (especially since the late 70s) has been seeking to cut the idea of “governance” off at the knees (at least in terms of “governance of billionaires and large corporations)
Extremely brief primer on artificial neural networks, accurate enough to understand what you're asking:
A neuron is a function which takes a signal value as input and produces a signal value as output. Signals are typically a floating-point (decimal) number in the range of 0-1, with higher values representing stronger "signal". So 0.5 is weaker than 0.75.
What happens inside the function almost doesn't matter. It may limit the value to the range of 0-0.5, or multiply it by 2 (to a maximum of 1). Typically they're more complicated than this and involve some linear algebra.
Anyway, an artificial neural network is where you connect many of these functions together in a web. The "prompt" is the signal strengths fed into the first layer of functions, and the output is derived from the final signal at the last layer.
Suppose you have 25 "neuron" functions, and their job is to analyze a grayscale 2x2 pixel image.
4 of them receive signals from the "prompt" input, one for each pixel, with a signal strength corresponding to how dark the pixel is.
One of them is the "output" neuron, which is supposed to send a signal of 1 or 0, depending on whether the 2x2 image depicts something a human calls a "diagonal line", or somewhere in between if it's not sure.
Those 4 input neurons connect to each of 10 more, for a total of 40 connections. Those 10 connect to another 10, making 100 connections. The second set of ten connect to the final neuron, another 10 connections. Your neural network thus has 141 connections.
Still with me?
Each of those connections has a corresponding "weight" value, which strengthens or attenuates the signal between the neuron pair. This is what is called the model's "parameters". So if neuron A sends a signal to neurons B and C (two "parameters"), the signals received by B and C are adjusted independently by the weight. Maybe A->B has a weight of 0.5, and A->C has a weight of 1.5, and the signal value is multiplied by the weight (again, more complicated in real life, but close enough).
So your 141 connections have 141 numbers, corresponding to 141 "weights".
Your model file contains a map of all the neurons, their connections, what their functions are, and what the weight values are.
---
So, what "training" does is adjust those weights, and sometimes the connections between neurons, with the goal of balancing all those internal signals such that your model will signal "1" when two black pixels appear diagonally in the 2x2 pixel image and the other two pixels are white, and closer to 0 when the image would not be perceived as a diagonal line.
It does not, and cannot, change the code within the neurons, or introduce novel neuron types (with new code). It is not exactly true that this is "off limits", rather that the standard model-running client, a separate piece of software, simply wouldn't be able to interpret arbitrary changes to the internals of the neurons. I run my own client (say it's called f-client). f-client can read a model that conforms to a particular popular specification. In that specification there are X, Y, and Z neuron types, and model files follow *this* specific format. If a neural network was permitted to introduce a new W-type neuron, or futz with its own file format, my client would no longer be able to run that model.
It is not exactly true that a model "trains itself". In that it does not conceive of new training methods or data. In unsupervised learning, you give a big set of labeled data and feed it to the neurons ("this 2x2 image is a diagonal line", "this 2x2 image is not a diagonal line", etc, ad nauseam). In supervised learning, you show the model a 2x2 image, check the output, and give it a headpat if the output is correct or a slap if it's wrong (it then adjusts weights up and down accordingly).
A prior model may be used to generate synthetic data for a new model, but this is necessarily constrained by the prior model's own training, and with the goal of getting the new model to behave like the old one.
A model may also be used to supervise the training of another model. Again, if the prior model is incorrect, it follows that it will "teach" the new model incorrectly.
During inference (generating stuff or answering questions) the weights and connections do not change (... usually, though there are techniques for doing this which are not used in popular LLMs and image generators). So each model file is a fixed snapshot.
The process of deciding what neurons to use, and how to connect them, is a guess-and-check, trial-and-error, fiddling process on the part of engineers, with a lot of "lore" and intuition involved.
Yes, this is a much more accurate description of an artificial neural network— and more. THANK YOU! (I need to read it more carefully again and I will.)
And, it does put to rest (? Or does it?) the idea of an AI device overstepping its parameters. (It cannot overwrite its own code!!!) “Parameters” may be the wrong word because I’m still “assuming” that there are some coded “rules” that govern (or attempt to govern) what inputs are submitted to the neural network - and perhaps some that also screen out certain responses generated by the neural network?
I also believe your description suggest that neural networks are extremely modular i.e. a particular neural network might work for a chatbox but could not also work to drive a car. I’d also “ASSUME”(?) that it would take several neural networks that work together (somehow) to drive a car. One thing that drives me crazy about the chatbox AI large language models is how they deal with problems requiring mathematics and quantity based comparisons. If a large language model neural network had another system that could actually “understand “ what was being asked or said, it would know how to translate certain questions so they could be handed off to the best computational system. But in my limited experience with the monsters, they can’t or don’t do that. A truly “General ‘INTELLIGENCE’” model would consist of a large number of neural network models that somehow interact together in a coordinated way. But even that would not be a very accurate model of how the human brain works just as the way a large language model generates language in a very different way than the human brain/mind does.
Do you have any ideas how a large language model is trained to generate or even recognize a poem????
> I’m still “assuming” that there are some coded “rules” that govern (or attempt to govern) what inputs are submitted to the neural network - and perhaps some that also screen out certain responses generated by the neural network?
Yes, the online chatbots you use have layers of filters. Sometimes simple, like chat filters that just reject no-no words, others small models that detect and reject specific ideas and sentiments. Some of them also do "prompt injection", which screws with your prompt to conform to some preconceived notion. Like maybe you say "draw a picture of a woman in a field" and the prompt injection thing transforms that to "draw a picture of a female-identifying person in a field, reflecting the diversity of people who enjoy being in fields". Silly stuff.
They're further regulated by a "hidden prompt" or "preamble", which is a giant pile of text inserted before your prompt with a bunch of context. Like, "You are an AI Assistant participating in a chat with the User. Your job is to answer the user's questions accurately and concisely. You never answer a question in a way that could be perceived as problematic or disparaging toward marginalized groups. You always write with a professional voice, using proper grammar. (etc etc)."
If you run your own local model, and particularly if you control the code of the client software, you don't necessarily have all these filters and you can make your own preamble: "You, the Assistant, are a nasty son of a bitch whose only aim is to irritate the User. When the user asks a question, you answer only with lies and insults." Now your chatbot is a dick. (subversive/pornographic online chatbots do similar things)
> I also believe your description suggest that neural networks are extremely modular i.e. a particular neural network might work for a chatbox but could not also work to drive a car.
Yes, image recognition networks are different from image generators, vehicle pilots are different from chatbots, etc. etc. Both structurally and in the training they receive. Nothing close to a general purpose neural network, which can be adapted to any task, has yet to be attempted or produced, and would likely be larger than any datacenter could ever run.
> If a large language model neural network had another system that could actually “understand “ what was being asked or said, it would know how to translate certain questions so they could be handed off to the best computational system.
There are some attempts at this, e.g. Wolfram Alpha, which does a passable job of parsing plain english math questions and handing them off to actual math software. The idea of trying to get an LLM, essentially a giant pile of billions of linear algebra functions, to answer math questions by way of text predictions always cracks me up. Like, seriously, what are we doing here?
> Do you have any ideas how a large language model is trained to generate or even recognize a poem????
Essentially all you do is feed them labeled data. Like, you might give them The Iliad, and it'll be labeled [mythology, poetry, epic poetry, Homer, ...]. And then you give it War of the Worlds and that is labeled [science fiction, alien invasion, prose, fiction, ...]. The object of training is to get it to say "this is a poem" when it sees The Iliad, or anything structurally similar to it. When it does so, the neural connections that were most active during that analysis get strengthened (not entirely unlike how human brains work on a surface level).
Eventually after doing this a hundred billion times, you basically run the model in reverse, like Jeopardy. Answer: "make a poem" -> Question: (it makes up some gobbledegook that is statistically similar to all the things it saw labeled "poem"). How exactly it has predicted that this particular string of words, in this particular format, corresponds to "poem", is lost in the billions of numbers representing the parameter weights.
Nobody *really* knows what it's "thinking". If you ask it "how did you come up with that", it does not respond with a true answer, it blindly generates some text that it predicts should follow the question. So it's kind of unknowable. It's very difficult to represent the internal state of a model in a way that is comprehensible to humans.
The bit about recognizing a poem makes sense. Obviously, there would have to be labels for the various types of poems. And modifying those labels might be ONE (of many??) forms of training???? I can see how labeling and characterizing poems in some kind of hierarchical system could get very complicated. OR, the system could be designed to look for meta labels that come with the training material. Sometimes I think of the “Training” as the system being “fed” (exposed to) carefully curated examples. Other times I think of it being fed a “crawl” that feeds it all kinds of random data as it “worms” through the web. So that confuses me.
I’m a little stumped about the “preamble” coding that could direct a chatbox to be “nice” or “nasty”. I don’t see how it could make that distinction. But I suppose there might be some module that guides the expressions to be more or less “neutral”, more less “negative” and “nasty”,… OR more or less “pleasant” and “positive.” I would imagine that a chatbox intended to be either warm and pleasant OR mean and nasty would make all kinds of silly mistakes that would remind any half cognizant user that this was a computer program and not a real person with feelings and motivations (never mind true concepts and intentions).
Thant brings us back to the popular notion of “the Turing Test”. I would GUESS that sticking to a neutral tone would be more convincing. I can also imagine a pornographic hotchat box that turned everything into a seduction, titillation, or smutty evocation just as well as I could imagine a Don Rickles chat box that would go out of its way to employ a user’s prompts to constantly insult and belittle them. But I’d imagine that these would be rather limited once the initial novelty and amusement wore off. But maybe I’m wrong about that. Maybe if you mixed neutral and informational responses with smut talk or insults, it would sound like a real person. Maybe the system could try to (superficially) determine the user’s mood or intentions and respond appropriately — even to the extent of “holding a grudge” based on something said in a previous session…
I referenced the Snyder piece about the “Turing test” and Chomsky’s language models to try to emphasize how a neural network is only superficially like th human brain/mind, but your descriptions about the actual artificial neural networks make the distinction just as clear if not clearer.
> I’m a little stumped about the “preamble” coding that could direct a chatbox to be “nice” or “nasty”. I don’t see how it could make that distinction.
It doesn't make a distinction, not in any way meaningful to us. Remember that the whole algorithm for LLM client software is, "given the prior body of text, predict what token is likely to come next." A token is, roughly, a part of a word, about 2-4 characters.
The "magic" happens during token prediction - "inference" -, in which the model is consulted and yields a short list of possible tokens with their predicted likelihoods. The client selects one of those tokens based on some other parameters, but leans toward the most likely token. The selected token is appended to the prior text body and it repeats until it hits a preordained limit or else the model emits a special token meaning "end of output".
So, practical example, suppose your prompt is "mary had a little". The model might return a list of tokens like "lam" at 98% probability, "cas" at 75% probability, etc. etc.
If the client selects "lam", then the working text is "mary had a little lam". Next inference, you get "b a" 98%, "e du" at 65%, etc. Suppose the client selects the second option, now you have "mary had a little lame du".
Next prediction is "ck." at 85%, "mb " at 72%, etc. If the client selects "ck.", your text is now "mary had a little lame duck."
So how does the model adopt a "personality?" Simple, given the preamble and your prompt, it predicts text likely to follow that. During training it has been given endless reams of sample instructions, chat transcripts, whole libraries worth of literature, and so on. So it has a pretty good statistical model of what will follow text that looks like:
```
You are playing the role of a mean chatbot. The user will ask questions. Respond cruelly.
User: <your prompt goes here, like "Hi, chatbot!">
Chatbot: <inference starts here>
```
Most of what comes next is going to be determined by its statistical model and the random selections of the LLM client software. Maybe the first set of tokens is something like "fuc", "you", "i", which eventually leads through further inference to either "fuck you, asshole", "you suck" or "I hate you".
Does the chatbot feel or think or know anything? No. Fundamentally it's just a blind algorithm for predicting what text is likely to follow from other text. There are lots of elaborations on top of this for higher-end LLMs, but that's the heart of the whole thing. Simple text prediction. The most remarkable thing is that it works at all.
AI is the 'Man as God' premise on steroids. The universe of life forms with their innumerable variables in real time will always 'trump' AI.
It’s helpful to remember that with any new technology, there’s the toy version for the proles and there’s the real version. A comment aimed mostly at the right-leaning ‘AI ain’t shit’ hopium crowd.
I really only want AI to help me win my Fantasy Football League. All other applications are subordinate.
AI is even vaster grift then climate change.