Not sure about the title of the article, but the topic is interesting.

https://www.atlasobscura.com/a...placement=newsletter

Nice article, thanks for sharing it! I'll include it in my list of popular news media articles that I share with students.

Related:

https://www.linkedin.com/posts...266619103080448-iqvP

"Language is low bandwidth: less than 12 bytes/second. A person can read 270 words/minutes, or 4.5 words/second, which is 12 bytes/s (assuming 2 bytes per token and 0.75 words per token)."

As, that’s super interesting, that’s for posting it!

I don’t agree with this:

quote:

Language is the icing on the cake

That makes it sound like there’s input (visual, sensory) and then there’s language, and these things are happening in isolation from each other.

But in fact, visual input is often received in tandem with linguistic input, and, crucially, linguistic development is happening well-before linguistic *output* is observable. We know this, for example, because when babies are signed to, they start to produce linguistic output earlier than babies who aren’t signed to — bc speech is harder to produce (in terms of the musculature systems involved) than signing.

There’s a lot of research in cognitive linguistics that looks at the connection between language and thought in a sort of chicken-and-egg way. IOW does language support cognition or does cognition support language? I don’t think there’s much consensus on an answer, because it’s very difficult to separate them. But it’s clear that language facilitates cognition in some very important ways.

So although this person wrote “ Language is the icing on the cake. We need the cake to support the icing” I think it might be better to acknowledge that we can’t probably understand the “cake” (i.e. the world around us) without the “icing” (i.e. the language we use to describe it, navigate it, change it, negotiate our roles within it).

And perhaps more importantly, once we get outside of our own heads, we need language to communicate our own understandings of the world with others.

So I think it’s a mistake to put language in anything other than the primary spot in this equation.

Which is to say, I would argue that language is the cake, not the icing.

But I’m a linguist, not (for example) a mathematician, so obviously I’m biased.

The other thing missing from LeCun’s comment is acknowledgement that speed and volume are not markers of quality.

Linguistic research on first language acquisition (i.e. infant language acquisition) has shown that babies are not learning primarily through imitation, and that their linguistic output both exceeds and transcends their linguistic input. We know this, for example, by looking at the mistakes that babies make (i.e. they make grammatical mistakes that no adults would ever make, but babies make those mistakes in ways that show they’re creating grammatical rules in their heads). Another sign is that babies (as all humans do) create with language.

There are things happening inside human heads that are qualitatively different from machine processeing of input. This is the “intelligence” feature and it’s why many people argue that the “intelligence” part of the name AI is misnomer. And I would add, to bring it back to this discussion, it’s why speed and volume (of input, of processing) are not the whole story.

See, you should post that stuff on LinkedIn to set LeCun straight.

I sort of considered it, but then I saw that there are already 500+ comments!

Separate from that, recently, sort of little by little, I've noticed that there seem to be more interesting discussions happening on LinkedIn and it makes me think I need to pay a little more attention to it.

And btw if I might add one other thing....

Input is not processed in a vacuum, and thinking does not happen in a black box situation. Another way of saying this is that none of these things -- input, cognition, learning -- are unidirectional. And I mean that to include input, despite the fact that we tend to think of input as unidirectional.

So with input (whether it's linguistic, visual, whatever), the receiver is performing operations on that input. Interpreting, categorizing, ignoring, forgetting, embellishing...

Humans are not like empty pitchers receiving input like water, you just pour it in and you get a pitch full of water that is unchanged from the way it was before it was poured into the pitcher.

That water (input) could get poured into the pitcher and you could end up with something that looks nothing like what you started with. (Ask any teacher, do students learn what they're taught...)

So what really matters is not the input, but what the person does with it.

Now, to bring it back to WTG's link, and Ax's...

from the article WTG linked:

quote:

What Pellegrino found is that, essentially, all languages convey information at roughly the same speed when all the factors are taken into account: around 39 bits per second. The higher the syllable-per-second rate, the lower the information density, which creates a trade-off that makes all languages around the same in terms of information rate.

I read somewhere (maybe Pellegrino's work, I'm not remembering now) that there's some speculation that cross-linguistic information rate ends up being more similar than it appears on the surface because the limits of human cognition are mostly the same, regardless of language. Above a certain speed, not only does linguistic input become incomprehensible in terms of not being able to audibly decipher it, but it also becomes incomprehensible in terms of not being able to keep up from the standpoint of cognitive processing.

Nevertheless, human cognition is remarkably fast, and, more importantly, we are able to do things with that input that thus far AI cannot. This is where the intangible magic of human cognition happens.

So the real question for LLMs is not how much faster can they process that input, but can they process that input in any ways that are different from the way they process now?

Maybe. (maybe that's the real question, maybe it's not, but it's fun to think about on a Sunday morning!)