> I myself shall certainly continue to leave such research to others,
and to devote my time to developing concepts that are authentic
and trustworthy. And I hope you do the same.
Haha.
Maybe someone with GPT-4 access can check if it has improved. But I find it essentially regurgitating its sources.
I am by no way an expert in questions I ask it. But I find the answers generally speculative and not helpful.
I have been asking bard and chat gpt once in a while "What's the internal representation of strings in Java"?
This is a good question to test its information retrieval capabilities. because your average blogspam site or tutorial doesn't cover the this.
They both somehow fail to convey me that it's a combination of char array and a cached hashcode.
At best, chat GPT told me a 3 paragraph sentence that java String was an array of UTF-16 characters, which is not quite the case since JDK 9 as I know. There's no mention of compact strings.
Other day I asked what's the GCC pass which deduplicates similar function definitions. Both chat gpt and bard made up their own pass name.
I am no expert in these topics, nor in ML or IR. But I don't believe LLM is the way towards information retrieval.
If it's ingesting everything it reads, it would be worse than pagerank, right?
Granted, it does normie stuff pretty well, like writing a data class or making HTTP request. But as soon as you need something deep, it is worse than useless, because it confidently claims incorrect stuff.
> Maybe someone with GPT-4 access can check if it has improved. But I find it essentially regurgitating its sources.
My wife's a librarian in a teaching hospital. A recent senimar reported the same problem all over the country -- people coming in asking for papers which don't exist. Chat GPT is making up sources and quoting them. That's whatever version the free option at chat.openai.com, or whatever app someone downloads, uses.
> Granted, it does normie stuff pretty well, like writing a data class or making HTTP request.
I find it really handy to find useful libraries in an unfamiliar language. I needed to deal with some network addresses in python, it introduced me to the ipaddress module. I'm not a software developer, I certainly don't keep up with the latest developments (which I'd define as anything post matts-script-archive), so things like that are valuable.
> I find it really handy to find useful libraries in an unfamiliar language. I needed to deal with some network addresses in python, it introduced me to the ipaddress module. I'm not a software developer, I certainly don't keep up with the latest developments (which I'd define as anything post matts-script-archive), so things like that are valuable.
That's correct. It has been many times helpful to me as well. But that's fundamentally because a google search for the same thing will be SEOd to death by blogspam sites listing top 100 python libraries.
Libraries ChatGPT told me about were libraries I could have found faster with a search engine and libraries it made up. It can be handy if a library doesn't have examples though.
People keep trying to use those LLMs as if it were a search engine but it's not. The value in ChatGPT is its faithful recreation of human common sense.
Use it to hold a conversation, to ask feedback on a text you wrote, to come up with ideas for something. Don't use it as Google and be dismissive when it's not the universal information retrieval tool it's not meant to be
Right. It's almost as if your average human has little creativity, indeed less than what ChatGPT can demonstrate, even though that magical "creativity" is supposed to be the current Big Thing that separates us from the machines. (In a year, once everybody is forced to agree that LLMs exhibit real creativity, not just regurgitation of their training data, the next Big Thing is going to be something else.)
Sooner or later most of the people will use it as a better replacement for Google search or Google on steroids. Before the advent of ChatGPT researchers especially, have been clamoring for better Google search with more contexts, intuitive and relevant feedbacks.
With the new ChatGPT (Plus) features introduction for examples web online search and plug-ins, ChatGPT has becoming a very powerful and viable better alternative to Google search.
I asked GPT-4 the sun over Japan question which GPT-3 got wrong, this was 4's response:
The sun is directly overhead (at the zenith) when it's at a 90-degree angle relative to the observer. This phenomenon typically occurs in the tropics, between the Tropic of Cancer (23.5 degrees North) and the Tropic of Capricorn (23.5 degrees South). These are the maximum latitudes at which the sun can appear directly overhead.
Japan is located between about 24 and 46 degrees north latitude. This means it is generally too far north for the sun to ever be directly overhead.
On July 4th, the sun is still moving northward from the Tropic of Cancer (which it reaches at the summer solstice, around June 21st). As a result, even in the southernmost parts of Japan, the sun will not be directly overhead, though it will be relatively high in the sky. In more northern areas, the angle will be even more pronounced.
If you want to know more precise details, such as the exact angle and time for a specific location in Japan on July 4, you'd need to consult a detailed almanac or a sun position calculator that takes into account the exact coordinates and date.
---
It no longer hallucinates an incorrect answer, though I don't have the knowledge to know if it's correct in saying that there isn't one.
It seems to mash up the concepts of moving northward from the equator (which the sun does before June 21st) and moving southward from the Tropic of Cancer (which it would be doing on July 4th).
It's right that only places between the tropics will ever see the sun directly overhead. I asked it the same question and it was kinda subtly misleading:
"The Sun is directly overhead, or at the zenith, at a given location only when that location is on the Tropic of Cancer (23.5 degrees north latitude) during the Summer Solstice (around June 21st each year) or on the Tropic of Capricorn (23.5 degrees south latitude) during the Winter Solstice (around December 21st each year)."
There are other situations besides these - you don't have to be -on- one of the tropics for the sun to be overhead, you have to be -on or between- them.
In Java, strings are internally represented by an instance of the `String` class, which is an immutable class that means once created, a `String` object cannot be changed.
The `String` class stores character data as an array of `char` data type. Prior to Java 9, this array was encoded as UTF-16. This means that each character in the string is typically stored in 2 bytes (16 bits), which can represent a range of Unicode characters.
Starting with Java 9, the `String` class uses a `byte` array, plus an encoding-flag field, to store string data. This change was part of the "compact strings" optimization, which is aimed at reducing the memory footprint of `String` objects. The flag tells the JVM whether the string is encoded as UTF-16 or Latin-1 (ISO-8859-1), depending on the string's content. If a string can be represented using Latin-1, it will be, because Latin-1 uses only 1 byte per character, which can save memory compared to the 2-byte UTF-16 encoding.
It's worth noting that these implementation details are internal to the `String` class and aren't usually something you need to worry about when writing Java code. They could also potentially change in future versions of Java.
People need to stop drawing conclusions based on GPT-3. If you're seriously interested in evaluating LLMs you should definitely try GPT-4. Most of my issues in 3 are non-existent in 4.
Once I had tried Bing Chat, it had prompted to open in edge and I ignored. After your comment I went back and tried it again. It certainly seems to find some more details after I specify the prompt well.
I admit it's a clear improvement. (Thank you for pointing out.) But it still gave some unsatisfactory answers in short interaction.
I asked to write a comparator to sort in decreasing order, for which it returned verbose (anon. class) form of `b - a`, which can go wrong with large negative integers.
Surprisingly, it did fairly well when asked to write a custom `JavaFileObject` class for `JavaDoc` toolprovider API, which is fairly obscure and I knew only because I worked with it. (There were errors in code, but they could be spotted by someone who can figure out the code's meaning).
So my conclusion from short interaction, bing chat finds stack overflow pages more reliably than Google does (LoL), but still gets affected by crap of society like geeksforgeeks.
Best results come with a conversation style prompt chain.
When I have it write code, I always say "identify any corner cases that are not properly handled". If you see an obvious problem, you could tell it to correct it "b - a will cause overflow for large negative numbers. Use an alternative."
I see it as an extremely broad, but maybe not incredibly deep, tool, that's most useful when you're not at the extreme specifics/esoteric knowledge.
And, bing chat is quite a bit different than ChatGPT 4 (no search), which is quite a bit different than raw GPT 4 completions.
As a counter-anecdote, most of my issues with GPT 3/3.5 are just as present in GPT-4 in slightly milder form.
Any code GPT4 produces needs to be checked with a fine tooth comb, any stated "facts" need to be double checked, any solutions to a given problem need to be examined by someone who's already familiar with the subject matter.
I can't deny its usefulness as a rubber duck, though.
My first question about any criticism of an AI response is "ChatGPT or GPT-4?"
If it's ChatGPT I'll offer to run the prompt through GPT-4 for them. It's better 100% of the time, and it negates their complaint probably 90% of the time.
I know it's not perfect, but GPT-4 is categorically different from ChatGPT.
Indeed. 3.5 didn't recognize my reference to Bomb #20.
But 4 did, and could describe the scene. And it even appreciated its relation to our discussion on epistemology when I said that I sometimes "hear" its voice when reading its replies!
I asked your question to Claude but added a line specifying that I wanted more technical details. It actually included a line mentioning a cache but that's about it.
Here is Claude's response:
---
Strings in Java are implemented as objects of the String class. Under the hood, Java strings use the following internal representation:
- A char array to store the characters of the string
- An integer storing the length of the string
- Some other internal fields for synchronization and cache (since Java 5)
When a string is created in Java, a char array is allocated to store the characters of that string. The length integer is set to the actual length of the string.
For example, if you create a string like this:
```java
String s = "hello";
```
Java will do the following:
- Allocate a char array of size 5 to store the 5 characters 'h', 'e', 'l', 'l', 'o'
- Set the length field to 5
- Initialize the char array with the string characters
So the internal representation in this case would be:
- A char[5] array initialized to {'h', 'e', 'l', 'l', 'o'}
- length = 5
When string operations like concatenation are performed, a new char array is allocated to store the resultant string and the length is updated accordingly. The String class then manages this char array and length field.
So in summary, Java strings use a char array and length field as their internal representation, and the String class encapsulates and manages this representation. The char array approach provides an efficient implementation for string operations in Java.
That's fairly underwhelming? I don't think it's correct for any version of the JVM. Really old versions had offset/length fields, but never a length field on its own. (Back then, substring() didn't make full copies.) For mainline Java, since version 9, strings can use a byte[] to store characters if the string consists entirely of characters in the Latin-1 subset. String literal assignment does not allocate, and concatenation allocates a new String object and internal array.
What is being described is closer to a typical implementation of C++ std::string, without the small-strings optimization. And it's written rather convincingly.
Here is Claude+'s[1] response (note that's "Claude Plus", not just plain "Claude"). It mentioned hash code. I don't know Java, so can't comment on its accuracy.. but here it is, for what it's worth...
In Java, strings are represented by the String class. Under the hood, strings are represented by char arrays.
When a string is created in Java, it is stored in the heap memory. For example:
```java
String s = "Hello";
```
This will create a char array like this:
`['H', 'e', 'l', 'l', 'o']`
And the String s will point to this array.
The String class is immutable, meaning the char array cannot be changed once created. So when you do:
And s will now point to this new array. The old array will eventually be garbage collected.
Under the hood, String uses a char[] value field to store the characters. For example:
```java
public final class String {
private final char value[];
}
```
The array is marked final, meaning it cannot be changed after initialization. This is how String achieves immutability.
For small strings, the value array points to a char[s.length] array in the String object itself. For larger strings, value points to a separate char[] on the heap. This cutoff is determined by the MIN_VALUE_SIZE which is currently 15 characters.
The String class also has a hash field which caches the hash code for fast lookups in hash tables. There is also a serialVersionUID used for serialization.
Other details:
- String is backed by char, using UTF-16 encoding.
- String pool is used to reuse similar strings and save memory.
- Substring creates a new backing char array and shares it with the original string.
- String is Comparable and has compareTo(), startsWith(), endsWith(), etc methods.
- Escape sequences can be used in String literals, e.g. "\n" is newline.
- Much more! Strings have many useful methods in the Java API.
Hope this helps provide lots of technical details on the internal representation of strings in Java! Let me know if you have any other questions.
Perhaps we shouldn't expect these models to know everything about everything. What sources did you yourself use to learn this knowledge and did the training data incorporate them? It's a bit like asking a software engineer law questions, you can only draw from what you've studied. I feel as though what's missing is the ability for the model to understand what it doesn't know or cite sources. It's not like humans know everything either.
It's unreasonable for the user to be able to guess what the software can do when it's a wide-open text interface and gives you no guidance. An ideal UI would be one where you can ask any question and if it's not something the computer can do, it would tell you, and perhaps give you some hints for what it can do. That is, you should be able to learn its limitations by playing with it.
There are some things ChatGPT will refuse to do, but there are also a lot of missing error messages. This is because the LLM doesn't know what it knows. All error messages need to be trained in.
One example of a category where the error message is missing is asking why it wrote something. It's reasonable to ask, but it doesn't know:
It’s interesting to me how people approach an AI with simple knowledge retrieval requests. We’ve had search engines for a while and being able to search for facts isn’t a particularly interesting use case. It doesn’t take anything like intelligence to regurgitate existing facts.
But that's the only thing they are good at, being smarter search engines (and that's why they should be backed by real search results, like Bing does it)
The only thing? You seem to have had a very limited exposure to what ChatGPT can do. Indeed it seems that some people have so little creativity that they can simply not think of asking these things anything except "a smarter Google" questions.
So what, by your estimation, are LLMs best for? Because they seem good for serving up relevant bits of information from vast amounts of information. Why do you think it's the worst thing they are good at?
Because it's the most basic use. In a single prompt you can have the LLM serve up relevant bits covering multiple perspectives, contrast and compare the perspectives, analyze their effectiveness in a given problem domain, and then produce meaningful output towards a solution. Information retrieval is just step 1.
Consider a prompt like the following:
"Given the task: 'TASK GOES HERE', break it down into intermediate steps or 'thoughts'. Consider multiple different reasoning paths that could be taken to solve the task. Explore these paths individually, reflecting on the possible outcomes of each. Then, consider how you might backtrack or look ahead in each path to make global decisions. Based on this analysis, develop a final to do list and complete the first course of action."
What should be expected then? It difficult to determine what the negation of "we shouldn't expect these models to know everything about everything" is.
Well chatgpt is often framed as an information retrieval tool or coding helper.
I don't have deep knowledge about these things I asked, I am just an undergrad student, and still I rarely find a technical answer by chatGPT satisfactory or helpful. I just don't see it as useful as it is framed.
In Java, strings are internally represented as objects of the String class, stored in a character array (char[]). Each character in the string is represented as a Unicode character, using UTF-16 encoding. This allows Java strings to handle a wide range of international characters.
> At best, chat GPT told me a 3 paragraph sentence that java String was an array of UTF-16 characters, which is not quite the case since JDK 9 as I know. There's no mention of compact strings.
For what it's worth, I got both ChatGPT and Bing chat to include compact strings in their replies to my query. I think people need to be much more specific in their prompts instead of expecting the LLM to read their minds exactly as to how much detail should be provided.
I'm quite disappointed in Knuth's dismissal. As the "king of algorithms", he of all people, should appreciate the leap. To put it another way, imagine if one of his students came to him with this question: I want to write an algorithm that given any of those 20 sample questions and others similar to them, gives this kind of answer. I couldn't find such algorithm in your books. What am I missing?
Haha.
Maybe someone with GPT-4 access can check if it has improved. But I find it essentially regurgitating its sources.
I am by no way an expert in questions I ask it. But I find the answers generally speculative and not helpful.
I have been asking bard and chat gpt once in a while "What's the internal representation of strings in Java"?
This is a good question to test its information retrieval capabilities. because your average blogspam site or tutorial doesn't cover the this.
They both somehow fail to convey me that it's a combination of char array and a cached hashcode.
At best, chat GPT told me a 3 paragraph sentence that java String was an array of UTF-16 characters, which is not quite the case since JDK 9 as I know. There's no mention of compact strings.
Other day I asked what's the GCC pass which deduplicates similar function definitions. Both chat gpt and bard made up their own pass name.
I am no expert in these topics, nor in ML or IR. But I don't believe LLM is the way towards information retrieval.
If it's ingesting everything it reads, it would be worse than pagerank, right?
Granted, it does normie stuff pretty well, like writing a data class or making HTTP request. But as soon as you need something deep, it is worse than useless, because it confidently claims incorrect stuff.